clj-jq

A library to execute jq expressions on JSON data within a Clojure application. It is a thin wrapper around jackson-jq: a pure Java jq Implementation for Jackson JSON Processor.

Available jq functions can be found here.

This library is compatible with the GraalVM native-image.

Alternatives

There is another jq library for Clojure: clj-jq. This library works by shelling-out to an embedded jq instance. The problem with this approach is that it has fixed costs for every invocation. Also, it creates difficulties to use this library with the GraalVM native-image.

Use cases

The library is intended to be used for stream processing.

(require '[jq.api :as jq])

(let [data "[1,2,3]"
      expression "map(.+1)"
      processor-fn (jq/stream-processor expression)]
  (processor-fn (jq/string->json-node data)))
=> [#object[com.fasterxml.jackson.databind.node.ArrayNode 0x8b78e9e "[2,3,4]"]]

NOTE: the input is of the JsonNode type while the output ßis a Collection of JsonNodes.

Loading JQ modules from the filesystem

File with contents:

; Prepare a module files
cat /path/to/script.jq
=> def increment(number): number + 1;

(let [data "[1,2,3]"
      expression "include \"scripts\"; map(increment(.))"
      processor-fn (jq/processor expression {:modules ["/path/to"]})]
  (processor-fn data))
=> "[2,3,4]"

Multiple modules can be provided.

Variables for the expressions

The library supports both: compile and runtime variables.

(let [expression "[$cvar, $rvar]"
      processor-fn (jq/stream-processor expression {:vars {:cvar "compile"}})]
  (processor-fn (jq/string->json-node "null") {:vars {:rvar "run"}}))
=> [#object[com.fasterxml.jackson.databind.node.ArrayNode 0x154d64f4 "[\"compile\",\"run\"]"]]

Inlined example

((jq/processor "map(.+1)") "[1,2,3]")
=> "[2,3,4]"

One-off script execution

(let [data "[1,2,3]"
      expression "map(.+1)"]
  (jq/execute data expression))
=> "[2,3,4]"

How to join multiple scripts together

Joining jq scripts is as simple as "piping" output of one script to another: join jq script strings with | character.

(let [data "[1,2,3]"
      script-inc "map(.+1)"
      script-reverse "reverse"
      expression (clojure.string/join " | " [script-inc script-reverse])]
  (jq/execute data expression))
=> "[4,3,2]"

Transducer API

Transducers are composable algorithmic transformations. They fit really well with the JQ model: expression produces a sequence of 0 or more JSON entities.

(require '[jq.transducers :as jq])

; Duplicate every value
(into []
      (comp
        (jq/->JsonNode)
        (jq/execute "(. , .)")
        (jq/JsonNode->value))
      [1 2 3])
=> [1 1 2 2 3 3]

; Several JQ expressions in a row
; NOTE: It is efficient because no serializations/deserializations are done between executions of expressions 
(into []
      (comp
        (jq/->JsonNode)
        (jq/execute ". + 1")
        (jq/execute ". + 1")
        (jq/JsonNode->value))
      [1 2 3])
=> [3 4 5]

; JSON string to JSON string
(sequence (comp
            (jq/parse)
            (jq/execute ".foo |= 2")
            (jq/serialize))
          ["{\"foo\":1}"])
=> '("{\"foo\":2}")

; A convenient "common-use-case" transducer
(into [] (jq/process "(. , .)") [1 2 3])
=>  [1 1 2 2 3 3]

A custom Jackson ObjectMapper can be provided for most transducers. E.g. you could take an ObjectMapper from the jsonista library and supply it to transducers.

For available transducers check here.

Performance

On MacBook pro M1 Max laptop:

(use 'criterium.core)
(let [jq-script (time (jq/processor ".a[] |= sqrt"))]
  (quick-bench (jq-script "{\"a\":[10,2,3,4,5],\"b\":\"hello\"}")))
=>
"Elapsed time: 0.257709 msecs"
Evaluation count : 334260 in 6 samples of 55710 calls.
Execution time mean : 1.812567 µs
Execution time std-deviation : 24.996676 ns
Execution time lower quantile : 1.788598 µs ( 2.5%)
Execution time upper quantile : 1.849297 µs (97.5%)
Overhead used : 1.840854 ns

Future work

• [ ] Expose interface to provide custom function, like here

License

Copyright © 2022 Dainius Jocas.

Distributed under The Apache License, Version 2.0.