datahike document index (konserve-lmdb.datahike)

Experimental. ZFS is not required; this is pure konserve-lmdb + datahike. Behind the :datahike deps alias, and the API may change.

datahike (like Datomic) restricts attribute values to scalars, and its AVET indexes those scalars — but it cannot look inside a structured value. This namespace turns a datahike attribute into a queryable EDN document: the whole document is stored as one boring blob, and selected nested paths are indexed so datalog can query into it. It makes datahike usable as a document store, with the documents still living in the same database.

How it works

Declare the attribute :db.type/bytes + :db.secondary/only true and store the boring-encoded document under it. Under :db.secondary/only, datahike keeps only a content hash in the primary EAVT/AEVT/AVET and hands the full value to this index — so the document lives only here, and the primary stays lean. (This holds under schema-on-write: a :db.type/bytes value passes validation, and hasch over the bytes is content-stable, so identical documents dedup and a retraction re-finds them by hash.)

The index is its own konserve-lmdb store (one per branch), navigable so reads project zero-copy. Three key ranges:

Key	Value	Purpose
[:idx <path> <value> <eid>]	<hash>	ordered path index — the LMDB B-tree; range/equality queries into documents
[:doc <hash>]	the decoded document	whole-doc read + zero-copy field projection
[:e <eid>]	<hash>	eid → doc (backup, update, retract)
[:rc <hash>]	refcount	blob GC

<hash> is (str (hasch/uuid bytes)), the same hash datahike stores in the primary. Only the [:idx …] range is order-preserving (LMDB's strength); the [:doc …] range is content-addressed (dedup, random keys).

Setup

(require '[datahike.api :as d]
         '[konserve-lmdb.datahike :as kd])

;; 1. the document attribute, secondary-only, + a covering index
(d/transact conn
  [{:db/ident :doc/data
    :db/valueType :db.type/bytes
    :db/cardinality :db.cardinality/one
    :db.secondary/only true}
   {:db/ident :idx/docs
    :db.secondary/type kd/index-type          ; :konserve-lmdb.datahike/ordered
    :db.secondary/attrs [:doc/data]
    :db.secondary/config {:path "/data/klmdb-docs"
                          :index-paths [[:address :city] [:name] [:age] [:tags]]}}])

;; 2. store documents (encode-doc gives content-stable bytes)
(d/transact conn [{:doc/data (kd/encode-doc {:name "Ada" :age 30
                                             :address {:city "Berlin"}
                                             :tags ["clojure" "lmdb"]})}])

:index-paths are the nested paths you can query on; a path landing on a sequential value indexes each element (so [:tags] supports contains-queries).

Querying

Each capability maps to one konserve-lmdb primitive.

Filter — query into the document (scan-keys, ordered)

;; equality on a nested path -> entity ids
(d/q '[:find [?e ...] :where
       [(konserve-lmdb.datahike/doc= :idx/docs [:address :city] "Berlin") [[?e]]]] db)

;; range on a nested path, ordered, binding the value
(d/q '[:find ?v ?e :where
       [(konserve-lmdb.datahike/doc-range :idx/docs [:age] 30 60) [[?e ?v]]]] db)

;; contains: [:tags] indexed each element
(d/q '[:find [?e ...] :where
       [(konserve-lmdb.datahike/doc= :idx/docs [:tags] "clojure") [[?e]]]] db)

Join — guided by the EntityBitSet

doc= returns a RoaringBitmap of entity ids; datahike stores it and narrows the scan of every other clause on that entity var — the same join-guidance used with other secondary indices. A doc= clause composes with primary-attribute clauses and with other doc= clauses for free:

(d/q '[:find [?n ...] :where
       [(konserve-lmdb.datahike/doc= :idx/docs [:address :city] "Berlin") [[?e]]]
       [?e :person/name ?n]] db)          ; :person/name is an ordinary attribute

Project in datalog — bind an un-indexed field (project, zero-copy)

;; for every Berlin document, bind its (un-indexed) :name, read zero-copy
(d/q '[:find [?v ...] :where
       [(konserve-lmdb.datahike/doc-project :idx/docs [:address :city] "Berlin" [:name]) [[?e ?v]]]] db)

Read one field / the whole document (project / decode)

(def idx (get-in @conn [:secondary-indices :idx/docs]))
(kd/project-field  idx eid [:address :city])          ; zero-copy, no full decode
(kd/project-fields idx eid [[:name] [:age]])
(kd/read-doc       idx eid)                            ; whole decoded document

Aggregate — group-by / count / sum / … (project-reduce, one pass)

;; whole-collection reduce, projecting only the named fields
(kd/aggregate idx [[:age]] (fn [acc [a]] (+ acc (double a))) 0.0)

;; group-by + aggregates over the whole collection (one project-reduce pass)
(kd/query-aggregate idx {:group [:city] :agg [[:count :city] [:avg :age]]})
;; => [{:city "Berlin" :count 2 :avg 40.0} ...]

;; ...optionally filtered first, via the ordered path index
(kd/query-aggregate idx {:where {:path [:address :city] :eq "Berlin"}
                         :group [:dept] :agg [[:count :dept] [:sum :salary]]})

Aggregating a document field is a full zero-copy scan of the collection — not an indexed lookup. There is no AVET for a field buried inside a blob, so this is the price (and the ability) of the document model: project-reduce visits every document but reads only the named fields. For heavy analytical aggregates over a few columns, a columnar engine (e.g. stratum) is the right tool; this is for aggregating documents you are already storing. -columnar-aggregate is also implemented, so datahike can route an aggregate down here once its planner learns about document-path columns (not wired in the current build).

Because every document shares the [:doc …] prefix, a whole-collection aggregate is one contiguous zero-copy scan. A filtered aggregate/projection (WHERE + SELECT) instead does per-matched-document point reads, since matched hashes are scattered — still zero-copy per field, just not a single scan.

Durability & branching

The index implements IVersionedSecondaryIndex, so its store persists across connects and is branched with the database: a datahike d/branch! physically forks the index store. By default this is a hot, consistent LMDB copy (O(n), never disturbs the source). Add a :fork config to make it an O(1) ZFS clone instead — copy-on-write, blocks shared until the branches diverge:

:db.secondary/config {:path "/klmdb-lab"
                      :index-paths [...]
                      :fork {:pool "rpool" :prefix "klmdb-lab"
                             :mount-base "/klmdb-lab"
                             :mount {:helper "/opt/klmdb/klmdb-zfs-mount.sh"}}}

With :fork, each branch's store is a ZFS dataset under the delegated sandbox and a branch is zfs clone (see FORKING.md for the privilege modes). -sec-value returns the full document, so a :db.secondary/only attribute round-trips through export-db/import-db.

Limitations (experimental)

• Cardinality-one document attributes; attribute-ident mode (not :attribute-refs?).
• Indexed path values must be konserve-lmdb-orderable scalars (string, keyword, symbol, long, double, boolean, uuid, instant) or sequences of them; a path landing on a map is skipped.
• :db.secondary/only means plain [?e :doc/data ?v] returns the hash, not the document — read documents through the index (read-doc / doc-project / -sec-value), which is the point of keeping them out of the primary.
• Consuming an upstream entity-filter is implemented, but this datahike build does not push one into an external engine's -search yet (it uses the produced bitmap to guide the primary scan instead). Forward-compatible.
• Valid-time (:db.valid/*) is not pushed down; datahike's generic post-filter applies if a query sets :datahike/valid-at.