EQL

This repository contains a specification for EQL and a support library to help the development of tools using it.

Table of Contents

1. What is EQL?
2. Specification
3. Library

1. What is EQL?

EQL is a declarative way to make hierarchical (and possibly nested) selections of information about data requirements.

EQL doesn’t have its own language; it uses EDN to express the request, taking advantage of the rich set of primitives provided by it.

1.1. EQL for selections

An easy way to get started is to think of a map and try to describe its shape. For example:

{:album/name "Magical Mystery Tour"
 :album/year 1967}

By describing the shape, we mean describing the structure but without the values, the previous example can be described using EQL as:

[:album/name :album/year]

Like using select-keys to specify which fields to extract from a map. Now let’s see what it looks like for nested structures:

{:album/name   "Magical Mystery Tour"
 :album/artist {:artist/name "The Beatles"}}

; can be described as:

[:album/name
 ; note the use of a map to express nesting
 {:album/artist
  [:artist/name]}]

It works the same to represent nested sequences:

{:album/name   "Magical Mystery Tour"
 :album/tracks [{:track/name "The Fool On The Hill"}
                {:track/name "All You Need Is Love"}]}

; can be described as:

[:album/name
 {:album/tracks
  [:track/name]}]

Although with just EQL you can’t know if a key value is expected to be a single item or a sequence, you can have this information setup out of band using Clojure specs, if you do so you can instrospect the spec and detect that, this is not a feature of EQL in any way, just a suggested approach in case you need to know if the response of a key is a single item or a sequence.

1.2. EQL for operations

EQL also supports mutations, which are like side effect calls to an API. Mutations can appear on EQL transaction, and they look like Clojure function calls, example:

[(call-mutation {:data "value"})]

More details in Mutations.

1.3. Datomic Pull Syntax comparison

On top of the Datomic Pull Syntax expression, EQL also supports:

Parameters
Mutations
Union Queries

Check the links on each for more details.

The attribute with options feature from the Datomic Pull Syntax is not present in EQL; instead, we provide the parameterized attributes that can handle arbitrary data to go along the base property.

1.4. GraphQL comparison

Similar to GraphQL, EQL works as a language to client libraries to communicate data requirements and operations, there is a good set of intersection in features between both, as:

a language to describe arbitrarily nested structures
support for mutations (operations to side effect the world)
support for parametrization
union queries for query branching (select query according to some custom definition based on the data)

GraphQL has a type system in its definition, and it is required for a GraphQL system to work. EQL has no such thing, and it dictates only the syntax but not the semantics. Some features in GraphQL just doesn’t make sense in EQL, like fragments, since EQL is already a data format (EDN), it’s easy to compose data as you would do in a regular Clojure program, for that reason many features are not necessary because EQL is a parseable data format with all the Clojure tools already available to operate on top of it. Also read the AST section for more information on how to programmatically manipulate EQL data structures.

2. Specification

The following sections explain the features provided by this library, including code examples. We show how you can work with an AST representation of an EQL expression, which is often an easier form to manipulate programatically. Of course, we’ll also cover how to convert between EQL and an AST representation (bidirectionally).

In the end of this section you will also find the Clojure Spec formal specification for the EQL syntax.

2.1. Query / Transactions

An EQL transaction is represented by an EDN vector.

Examples:

[] ; empty transaction

; ast

{:type :root, :children []}

A transaction that only contains reads is commonly called a query, but notice that at the syntax level, it has no difference.

2.2. Properties

Properties in EQL are expressed as Clojure keywords; they can be simple or qualified keywords, and they express the property been requested.

Examples:

[:album/name :album/year]

; ast

{:type     :root
 :children [{:type :prop, :dispatch-key :album/name, :key :album/name}
            {:type :prop, :dispatch-key :album/year, :key :album/year}]}

2.3. Joins

Joins are used to describe nesting in the request transaction. They are represented as EDN maps, always with a single entry, the entry key is the property to join on, and the entry value is a sub-query to run.

Examples:

[{:favorite-albums
  [:album/name :album/year]}]

; ast

{:type     :root
 :children [{:type         :join
             :dispatch-key :favorite-albums
             :key          :favorite-albums
             :query        [:album/name :album/year]
             :children     [{:type :prop, :dispatch-key :album/name, :key :album/name}
                            {:type :prop, :dispatch-key :album/year, :key :album/year}]}]}

Nested joins example:

[{:favorite-albums
  [:album/name :album/year
   {:album/tracks
    [:track/name
     :track/duration]}]}]

; ast

{:type :root
 :children
 [{:type         :join
   :dispatch-key :favorite-albums
   :key          :favorite-albums

   :query        [:album/name
                  :album/year
                  {:album/tracks [:track/name :track/duration]}]

   :children     [{:type :prop, :dispatch-key :album/name, :key :album/name}
                  {:type :prop, :dispatch-key :album/year, :key :album/year}
                  {:type         :join
                   :dispatch-key :album/tracks
                   :key          :album/tracks
                   :query        [:track/name :track/duration]
                   :children     [{:type :prop, :dispatch-key :track/name, :key :track/name}
                                  {:type         :prop
                                   :dispatch-key :track/duration
                                   :key          :track/duration}]}]}]}

2.4. Idents

Idents are represented by a vector with two elements, where the first is a keyword and the second can be anything. They are like lookup refs on Datomic, in general, they can provide an address-like thing, and their use and semantic might vary from system to system.

Examples:

[[:customer/id 123]]

; ast

{:type :root
 :children [{:type :prop, :dispatch-key :customer/id, :key [:customer/id 123]}]}

Note that this time in the AST the :dispatch-key and :key got different values this time, the :dispatch-key been just the ident key while the :key contains the full thing.

It’s common to use an ident as a join key to start a query for some entity:

[{[:customer/id 123]
  [:customer/name :customer/email]}]

; ast

{:type     :root
 :children [{:type         :join
             :dispatch-key :customer/id
             :key          [:customer/id 123]
             :query        [:customer/name :customer/email]
             :children     [{:type :prop, :dispatch-key :customer/name, :key :customer/name}
                            {:type         :prop
                             :dispatch-key :customer/email
                             :key          :customer/email}]}]}

2.5. Parameters

EQL properties, joins, and idents have support for parametrization. This allows the query to provide an extra dimension of information about the requested data. A parameter is expressed by wrapping the thing with an EDN list, like so:

; without params
[:foo]

; with params
[(:foo {:with "params"})]

; ast

{:type     :root
 :children [{:type         :prop
             :dispatch-key :foo
             :key          :foo
             :params       {:with "params"}
             :meta         {:line 1, :column 15}}]}

Note on the AST side it gets a new :params key. Params must always be maps, the map values can be anything. Here are more examples of parameterizing queries:

; ident with params

[([:ident "value"] {:with "param"})]

{:type     :root
 :children [{:type         :prop
             :dispatch-key :ident
             :key          [:ident "value"]
             :params       {:with "param"}
             :meta         {:line 1, :column 15}}]}

; join with params wrap the key with the list

[{(:join-key {:with "params"})
  [:sub-query]}]

{:type     :root
 :children [{:type         :join
             :dispatch-key :join-key
             :key          :join-key
             :params       {:with "params"}
             :meta         {:line 1, :column 16}
             :query        [:sub-query]
             :children     [{:type         :prop
                             :dispatch-key :sub-query
                             :key          :sub-query}]}]}

; ident join with params

[{([:ident "value"] {:with "params"})
  [:sub-query]}]

{:type     :root
 :children [{:type         :join
             :dispatch-key :ident
             :key          [:ident "value"]
             :params       {:with "params"}
             :meta         {:line 1 :column 16}
             :query        [:sub-query]
             :children     [{:type         :prop
                             :dispatch-key :sub-query
                             :key          :sub-query}]}]}

; alternate syntax to add params on joins (wrap the entire map, AST result is the same)

[({:join-key
   [:sub-query]}
  {:with "params"})]

{:type     :root
 :children [{:type         :join
             :dispatch-key :join-key
             :key          :join-key
             :params       {:with "params"}
             :meta         {:line 1, :column 16}
             :query        [:sub-query]
             :children     [{:type         :prop
                             :dispatch-key :sub-query
                             :key          :sub-query}]}]}

You’ll need to use quote and unquote in CLJ files for calls, otherwise the lists will be evaluated as Clojure calls. Quote is not necessary in EDN files.

2.6. Query Meta

Metadata can be stored on a query. The AST will encode the metadata so that transformations to/from an AST can preserve it.

(with-meta [] {:meta "data"})

; ast

{:type :root, :children [], :meta {:meta "data"}}

2.7. Unions

In EQL unions are used to specify polymorphic requirements, that means depending on some condition a different query might be chosen to fulfill the requirements. For example, a messaging app may have a single list, and each entry on the chat log can be a message, audio or photo, each having its own query requirement. Here it is in code:

; message query
[:message/id :message/text :chat.entry/timestamp]

; audio query
[:audio/id :audio/url :audio/duration :chat.entry/timestamp]

; photo query
[:photo/id :photo/url :photo/width :photo/height :chat.entry/timestamp]

; list query
[{:chat/entries ???}] ; what goes there?

Now to express this polymorphic requirement as the sub-query of the :chat/entries list we can use a map as the join value, and each entry on this map represents a possible sub-query. The way this information is used is up to the parser implementation; EQL only defines the syntax. Here are some examples of how it could be written:

; in this example, the selection is made by looking if the processed entry contains
; some value on the key used for its selection
[{:chat/entries
  {:message/id [:message/id :message/text :chat.entry/timestamp]
   :audio/id   [:audio/id :audio/url :audio/duration :chat.entry/timestamp]
   :photo/id   [:photo/id :photo/url :photo/width :photo/height :chat.entry/timestamp]}}]

; in this case, we give a type name and use as the key, this usually requires some
; out of band configuration to know how to pull this data from each entry to use
; as the comparison
[{:chat/entries
  {:entry.type/message [:message/id :message/text :chat.entry/timestamp]
   :entry.type/audio   [:audio/id :audio/url :audio/duration :chat.entry/timestamp]
   :entry.type/photo   [:photo/id :photo/url :photo/width :photo/height :chat.entry/timestamp]}}]

; ast for the first example

{:type :root
 :children
 [{:type         :join
   :dispatch-key :chat/entries
   :key          :chat/entries
   :query        {:message/id [:message/id :message/text :chat.entry/timestamp]
                  :audio/id   [:audio/id :audio/url :audio/duration :chat.entry/timestamp]
                  :photo/id   [:photo/id
                               :photo/url
                               :photo/width
                               :photo/height
                               :chat.entry/timestamp]}
   :children     [{:type :union
                   :query
                         {:message/id [:message/id :message/text :chat.entry/timestamp]
                          :audio/id   [:audio/id :audio/url :audio/duration :chat.entry/timestamp]
                          :photo/id   [:photo/id
                                       :photo/url
                                       :photo/width
                                       :photo/height
                                       :chat.entry/timestamp]}
                   :children
                         [{:type      :union-entry
                           :union-key :message/id
                           :query     [:message/id :message/text :chat.entry/timestamp]
                           :children  [{:type :prop, :dispatch-key :message/id, :key :message/id}
                                       {:type :prop, :dispatch-key :message/text, :key :message/text}
                                       {:type         :prop
                                        :dispatch-key :chat.entry/timestamp
                                        :key          :chat.entry/timestamp}]}
                          {:type      :union-entry
                           :union-key :audio/id
                           :query     [:audio/id :audio/url :audio/duration :chat.entry/timestamp]
                           :children  [{:type :prop, :dispatch-key :audio/id, :key :audio/id}
                                       {:type :prop, :dispatch-key :audio/url, :key :audio/url}
                                       {:type         :prop
                                        :dispatch-key :audio/duration
                                        :key          :audio/duration}
                                       {:type         :prop
                                        :dispatch-key :chat.entry/timestamp
                                        :key          :chat.entry/timestamp}]}
                          {:type      :union-entry
                           :union-key :photo/id
                           :query     [:photo/id
                                       :photo/url
                                       :photo/width
                                       :photo/height
                                       :chat.entry/timestamp]
                           :children  [{:type :prop, :dispatch-key :photo/id, :key :photo/id}
                                       {:type :prop, :dispatch-key :photo/url, :key :photo/url}
                                       {:type :prop, :dispatch-key :photo/width, :key :photo/width}
                                       {:type :prop, :dispatch-key :photo/height, :key :photo/height}
                                       {:type         :prop
                                        :dispatch-key :chat.entry/timestamp
                                        :key          :chat.entry/timestamp}]}]}]}]}

2.8. Mutations

Mutations in EQL are used to represent operation calls, usually to do something that will cause a side effect. Mutations as data allows that operation to behave much like event sourcing, and can be transparently applied locally, across a network, onto an event bus, etc.

A mutation is represented by a list of two elements; the first is the symbol that names the mutation, and the second is a map with input data.

[(call.some/operation {:data "input"})]

; ast

{:type :root
 :children
 [{:dispatch-key call.some/operation
   :key          call.some/operation
   :params       {:data "input"}
   :meta         {:line 610, :column 17}
   :type         :call}]}

Mutations and parameters are very similar, their main difference is that once uses symbols as keys, and the other uses one of the read options (properties, idents, joins).

The EQL notation does not technically limit the combination of expressions that contain both query and mutation elements; however, implementations of EQL processing may choose to make restrictions on these combinations in order to enforce particular semantics.

2.8.1. Mutation Joins

A mutation may have a return value, and that return value can be a graph; therefore, it makes sense that EQL support the ability to describe what portion of the available returned graph should be returned. The support for mutation graph return values is done by combining the syntax of a join with the syntax of a mutation:

[{(call.some/operation {:data "input"})
  [:response :key-a :key-b]}]

; ast

{:type :root
 :children
 [{:dispatch-key call.some/operation
   :key          call.some/operation
   :params       {:data "input"}
   :meta         {:line 612 :column 18}
   :type         :call
   :query        [:response :key-a :key-b]
   :children     [{:type :prop, :dispatch-key :response, :key :response}
                  {:type :prop, :dispatch-key :key-a, :key :key-a}
                  {:type :prop, :dispatch-key :key-b, :key :key-b}]}]}

3. Library

The package edn-query-language.core provides a suite of specs to validate queries and ASTs. It also provides generators for the query and helper functions to common query operations.

3.1. Clojure Specs

The EQL library provides specs to validate and generate queries.

3.1.1. Validation

You can validate the query syntax using clojure.spec, here is an example:

(s/valid? ::eql/query [:sample :query]) ; => true
(s/valid? ::eql/query [#{:set}]) ; => false
(s/valid? ::eql/query ['(call/op {})]) ; => true

s is alias for clojure.spec.alpha

You can use spec explain feature for more details:

(s/explain ::eql/query [#{:set}])
; In: [0] val: #{:set} fails spec: :edn-query-language.core/mutation-expr at: [:mutation :mutation] predicate: seq?
; In: [0] val: #{:set} fails spec: :edn-query-language.core/mutation-join at: [:mutation :mutation-join] predicate: map?
; In: [0] val: #{:set} fails spec: :edn-query-language.core/property at: [:prop] predicate: keyword?
; In: [0] val: #{:set} fails spec: :edn-query-language.core/join at: [:join] predicate: map?
; In: [0] val: #{:set} fails spec: :edn-query-language.core/ident at: [:ident] predicate: vector?
; In: [0] val: #{:set} fails spec: :edn-query-language.core/param-expr at: [:param-exp] predicate: seq?
; In: [0] val: #{:set} fails spec: :edn-query-language.core/special-property at: [:special] predicate: #{(quote *)}

I suggest you check the sources for the specs for more details on parts that compose it, they will stay consistent and can be used to validate parts of the transaction as well.

3.1.2. Generation

EQL also provides built-in generators, the main intended usage for it is to write generative tests for parser implementations.

Basic example to generate random queries:

(gen/sample (s/gen ::query) 10)
=>
([]
 []
 [(:?./*_ {}) :z/ZH]
 []
 [#:J{:w {:c/!V [#:YY{:u [:u1/X?!
                          #:r94{:*+ [#:aG{:YA 2} :t!o/Ya1 :XL/HR #:!-Q{:b_ []}]}
                          :OP/E]}
                 :.qE/Nd-],
          :j./!T [[:p/h*y :f?1]
                  #:s*{:-W []}
                  (NG_
                   {[] #{}, [4] (0.5 :_ -3 -Ch), #{} #{}, #{-1 {##-Inf ?.1/e?A}} {}})],
          :z/s+ []}}
  :-_/_
  :H/E
  :Y/xD]
 [:?7/w :iO/! (:r/!N {{-2.0 false} [], [] [], [:P7] [0 J1]})]
 [:+Bi/-K :!8*/r0 :?/Cio]
 [:*.-/R* :+BT/W :-l8/c :Ih/V [:RE/- "0>WwI`u"] :H/vT]
 [:z+8/g]
 [])

gen is alias for clojure.test.check.generators

Although fully random queries can be interesting to test some parser edge cases, in many situations you will may want to constraint how the query is generated, with this in mind EQL provides a way to enable this kind of customization. To get a sense of what you can customize you can take a look at the default implementation for each default generator, any of those keys can be tuned to constraint how the query is generated.

To demonstrate how to use this, let’s customize the generator to limit the properties it generates to a fixed set we pre defined:

(gen/sample (eql/make-gen {::eql/gen-property ; (1)
                       (fn [_] (gen/elements [:id :name :title :foo :bar]))}
              ::eql/gen-query) ; (2)
  10)
=>
([]
 []
 []
 [[:X/q6 1] :name :title]
 [({:title [(L {#{} [], () [], #{-5} ()})
            (:name {{#{} {}} :., {} {}})
            {:name [:bar :title]}]}
   {[*+-] #{0.5625 #uuid"edf051fb-ab28-42d0-a941-152c4e87b060"},
    #{#uuid"712e7415-5148-400b-99db-cfb79004700e" -1/2} (),
    {} (:F/le9 #uuid"5ad52713-d13a-4888-bd92-2d1541c0387b" "" true)})
  {(:foo
    {[(2.0 false) z/NO] [I./j #uuid"eef64a1d-8055-4ae7-95be-06bdc4f9cefd"], {} [""]}) [:id
                                                                                       ({:id [:name
                                                                                              *]}
                                                                                        {})]}]
 [:id :id]
 [{:foo [:name * [:mO/D MZ_/e0Z] :bar :foo]}]
 []
 [:bar]
 [:foo])

1	We send a map to `eql/make-gen` to override some of the generator settings, any non defined keys will fallback to default implementation
2	Select which generator to use, this is useful to generate only sub-parts if needed

One more example changing many definitions:

(let [system (assoc generators
               ::gen-params
               (fn [_] (gen/map (gen/elements [:param :foo/param]) gen/string-ascii))

               ::gen-property
               (fn [_] (gen/elements [:id :name :title :foo :bar :other :price :namespaced/value]))

               ::gen-ident-key
               (fn [_] (gen/elements [:user/by-id :other/by-id]))

               ::gen-ident-value
               (fn [_] gen/string-ascii)

               ::gen-mutation-key
               (fn [_] (gen/elements '[do-something create/this-thing operation.on/space])))]
  (gen/sample ((::gen-query system) system)))
=>
([]
 [{:other []}]
 []
 []
 []
 [{:price [{[:user/by-id "!"] []} :title]} :id]
 [:bar {[:other/by-id "@"] [:foo :other :name]}]
 [:name :id]
 [:price :title :id :name]
 [:foo
  ({:bar [[:user/by-id ""] :price {:id [:other]} :other]} {})
  :other
  :namespaced/value
  {:name [:name
          {:bar [:name
                 :bar
                 :namespaced/value
                 ({[:user/by-id "AeA$;"] [:foo]}
                  {:foo/param "_+y9ihY", :param "Y@p5Bd5B"})
                 :id
                 :namespaced/value
                 :name]}]}
  :id])

If you wanna see an even more advanced usage, you can check Pathom connect generator, which uses the Pathom connect index to generate queries that are valid according to the user property graph.

3.1.3. Removing specs on Clojurescript

If you are not using the specs provided by EQL you can free some build space by eliding then. To do that you need to set the Clojurescript compiler options with:

{:closure-defines {edn-query-language.core.INCLUDE_SPECS false}}

3.2. AST Encode/Decode

To convert between query and AST, EQL provides the helper functions eql/query→ast and eql/ast→query. Here are some example usages:

(eql/query->ast [:foo])
; => {:type :root, :children [{:type :prop, :dispatch-key :foo, :key :foo}]}

(eql/ast->query {:type :root, :children [{:type :prop, :dispatch-key :foo, :key :foo}]})
; => [:foo]

3.3. API Docs

Check the complete API docs at EQL cljdoc page.