autonormal

A library for storing graph data in a Clojure map that automatically normalizes nested data and allows querying via EQL, optimized for read (query) performance.

Use cases

The primary use case this library was developed for was to act as a client side cache for pathom APIs. However, you can imagine any time you might reach for DataScript to store data as entities, but where you need fast nested / recursive querying of many attributes and don't need the full expressive power of datalog, as being a good use case for autonormal.

Project status

While feature complete, it has not been used in production yet.

Usage

Normalizing

A db is simply a map with a tabular structure of entities, potentially with references to other entities.

Autonormal currently makes a default conventional assumption: your entities are identified by a keyword whose name is "id", e.g. :id, :person/id, :my.corp.product/id, etc.

(require '[autonormal.core :as a])

(def data
  {:person/id 0 :person/name "Rachel"
   :friend/list [{:person/id 1 :person/name "Marco"}
                 {:person/id 2 :person/name "Cassie"}
                 {:person/id 3 :person/name "Jake"}
                 {:person/id 4 :person/name "Tobias"}
                 {:person/id 5 :person/name "Ax"}]})

;; you can pass in multiple entities to instantiate a db, so `a/db` gets a vector
(def animorphs (a/db [data]))
;; => {:person/id {0 {:person/id 0 
;;                    :person/name "Rachel"
;;                    :friend/list [[:person/id 1]
;;                                  [:person/id 2]
;;                                  [:person/id 3]
;;                                  [:person/id 4]
;;                                  [:person/id 5]]}
;;                 1 {:person/id 1 :person/name "Marco"}
;;                 2 {:person/id 2 :person/name "Cassie"}
;;                 3 {:person/id 3 :person/name "Jake"}
;;                 4 {:person/id 4 :person/name "Tobias"}
;;                 5 {:person/id 5 :person/name "Ax"}}}

The map structure of a db is very efficient for getting info about any particular entity; it's just a get-in away:

(get-in animorphs [:person/id 1])
;; => {:person/id 1 :person/name "Marco"}

You can assoc/dissoc/update/etc. this map in whatever way you would like. However, if you want to accrete more potentially nested data, there's a helpful add function to normalize it for you:

;; Marco and Jake are each others best friend
(def animorphs-2
  (a/add animorphs {:person/id 1
                    :friend/best {:person/id 3
                                  :friend/best {:person/id 1}}}))
;; => {:person/id {0 {:person/id 0 
;;                    :person/name "Rachel"
;;                    :friend/list [[:person/id 1]
;;                                  [:person/id 2]
;;                                  [:person/id 3]
;;                                  [:person/id 4]
;;                                  [:person/id 5]]}
;;                 1 {:person/id 1
;;                    :person/name "Marco" 
;;                    :friend/best [:person/id 3]}
;;                 2 {:person/id 2 :person/name "Cassie"}
;;                 3 {:person/id 3
;;                    :person/name "Jake"
;;                    :friend/best [:person/id 1]}
;;                 4 {:person/id 4 :person/name "Tobias"}
;;                 5 {:person/id 5 :person/name "Ax"}}}

Note that our animorphs db is an immutable hash map; add simply returns the new value. It's up to you to decide how to track its value and keep it up to date in your system, e.g. in an atom.

Adding non-entities

Maps that are added are typically entities, but you can also add arbitrary maps and add will merge the map with the database, normalizing and referencing any nested entities. Example:

(def animorphs-3
  (a/add animorphs-2 {:species {:andalites [{:person/id 5
                                             :person/species "andalite"}]}}))
;; => {:person/id {,,,
;;                 5 {:person/id 5
;;                    :person/name "Ax"
;;                    :person/species "andalite"}}
;;     :species {:andalites [[:person/id 5]]}}

Querying

This library implements a fast EQL engine for Clojure data.

(a/pull animorphs-3 [[:person/id 1]])
;; => {[:person/id 1] {:person/id 1
;;                     :person/name "Macro"
;;                     :friend/best {:person/id 3}}}

You can join on idents and keys within entities, and it will resolve any references found in order to continue the query:

(a/pull animorphs-3 [{[:person/id 1] [:person/name
                                      {:friend/best [:person/name]}]}])
;; => {[:person/id 1] {:person/name "Marco"
;;                     :friend/best {:person/name "Jake"}}}

Top-level keys in the db can also be joined on.

(a/pull animorphs-3 [{:species [{:andalites [:person/name]}]}])
;; => {:species {:andalites [{:person/name "Ax"}]}}

Recursion is supported:

(def query '[{[:person/id 0] [:person/id
                              :person/name
                              {:friend/list ...}]}])

(= (-> (a/pull animorphs-3 query)
       (get [:person/id 0]))
   data)
;; => true

See the EQL docs for more examples of what's possible!

Features

• [x] Supports Clojure and ClojureScript
• [x] Auto normalization
• [x] Full EQL query spec
  • [x] Props
  • [x] Joins
  • [x] Idents
  • [x] Unions
  • [x] Recursion
  • [x] Preserve query meta on results

Prior art

• Fulcro
• MapGraph
• EntityDB
• DataScript and derivatives
• Pathom
• juxt/pull

Copyright

Copyright © 2020 Will Acton. Distributed under the EPL 2.0.