Liking cljdoc? Tell your friends :D

io.github.bsless/jsonista.streams

Clojure library for fast streaming encoding and decoding of JSON alongside jsonista

Dependency

Deps

io.github.bsless/jsonista.streaming {:mvn/version "0.0.2"}

Leiningen

[io.github.bsless/jsonista.streaming "0.0.2"]

Usage

Require

(require '[bsless.jsonista.streams :as js])

Values Stream

read-values returns a reducible Iterator over the JSON values and is completely lazy:

;; Just allocate an object
(read-values (json/write-value-as-string [{:a 1} {:a 2} {:a 3}]))

Since the resulting object is reducible, we can reduce over it without deserializing the entire payload:

(into
 []
 (map :a)
 (-> [{:a 1} {:a 2} {:a 3}]
     json/write-value-as-string
     (read-values json/keyword-keys-object-mapper)))
;; => [1 2 3]

Similarly, we can use write-values to directly emit values without creating the entire blob in memory. This can be useful for quickly getting data out of a process instead of accumulating a large blob, consuming heap and resources.

(.toString
 (doto (StringWriter.)
   (write-values [1 2 3])));; => "[1,2,3]"

We can also change the format of the output writer:

(.toString
 (doto (StringWriter.)
   (write-values
    [1 2 3]
    json/keyword-keys-object-mapper
    {:write-value-as-array false
     :root-value-separator "\n"})))
;; => "1\n2\n3"

A newline separated JSON might be preferable for some APIs and consumers.

Putting it together

By using lazy encoding and decoding together, we can construct flows which only ever hold one decoded object in memory:

(let [original-data [{:a 1} {:a 2} {:a 3}]
      input (json/write-value-as-string original-data)
      stream (read-values input json/keyword-keys-object-mapper)
      flow (->Eduction (map :a) stream)
      writer (StringWriter.)]
  (write-values writer flow)
  (.toString writer))
;; => "[1,2,3]"