konserve-lmdb

Note: This library is currently in beta. The API may change in future releases.

A high-performance LMDB backend for konserve using Project Panama FFI (Java 22+).

Features

• Zero-copy reads via LMDB's memory-mapped architecture
• CBOR on the wire via boring — an interchange format any CBOR reader consumes, not a private encoding
• Projection: read one field out of a range of values without decoding them (below) — 49x PostgreSQL JSONB on the case it is built for
• Two API levels: Full konserve compatibility or Direct API for maximum performance
• Ordered access: range scans over the primary keyspace (format v2)
• Lock-free operations: LMDB provides MVCC, no application-level locking needed
• Extensible type handlers for custom serialization

Requirements

• Java 22+ (for Project Panama FFI)
• liblmdb native library

Installing LMDB

The library auto-detects common system paths, so usually just installing the package is enough.

Ubuntu/Debian

sudo apt install liblmdb0

macOS (Homebrew)

brew install lmdb

Arch Linux

sudo pacman -S lmdb

Building from Source

LMDB is a small, dependency-free C library that compiles in seconds:

# Clone the official LMDB repository
git clone https://git.openldap.org/openldap/openldap.git
cd openldap/libraries/liblmdb

# Build (produces liblmdb.so and liblmdb.a)
make

# Install system-wide (recommended)
sudo make install

Custom Library Path

If the library is in a non-standard location, set KONSERVE_LMDB_LIB:

export KONSERVE_LMDB_LIB=/path/to/liblmdb.so

Enabling native access

Loading LMDB is a restricted operation, so without a JVM flag you get:

WARNING: A restricted method in java.lang.System has been called
WARNING: Restricted methods will be blocked in a future release unless
         native access is enabled

Add the flag to your :jvm-opts:

:jvm-opts ["--enable-native-access=ALL-UNNAMED"]

This is not merely cosmetic. The JVM currently warns; a future release will refuse, and the store will fail to open rather than print a warning. Every alias in this repository's own deps.edn sets it.

Usage

Add to your dependencies:

Konserve API (Full Compatibility)

(require '[konserve-lmdb.store]  ;; Registers the :lmdb backend
         '[konserve.core :as k])

(def config
  {:backend :lmdb
   :path "/tmp/my-store"
   :id #uuid "550e8400-e29b-41d4-a716-446655440000"
   ;; Optional:
   :map-size (* 1024 1024 1024)  ;; Default: 1GB
   :flags 0})

(def store (k/create-store config {:sync? true}))

For API usage (assoc, get, multi-assoc, keys, etc.), see the konserve documentation.

Features:

• Lock-free operations: LMDB provides MVCC, no application-level locking needed
• Multi-key operations: Atomic bulk operations supported
• Efficient key enumeration: keys uses metadata-only decoding for GC efficiency

Direct API (Maximum Performance)

For performance-critical code, use the Direct API, which bypasses konserve's metadata tracking.

It is bytes in, bytes out — you choose the codec. The store encodes keys, because it owns ordering; you encode values, because you own their semantics. Any encoder works: boring, nippy, protobuf, or raw bytes you assembled yourself.

(require '[konserve-lmdb.store :as lmdb]
         '[boring.core :as boring])

(def store (lmdb/connect-store "/tmp/my-store"))

;; Direct put/get - no metadata wrapper, fastest possible
(lmdb/put store :key (boring/encode {:data "value"}))
(boring/decode (lmdb/get store :key))
;; => {:data "value"}

;; Zero copy: decode-fn sees a ByteBuffer over LMDB's own mapping, inside the
;; read transaction. Consume it -- it is not valid after this returns.
(lmdb/get-decode store :key #(boring/decode (boring/segment-source %)))

;; Batch operations - single transaction
(lmdb/multi-put store {:k1 (boring/encode "v1") :k2 (boring/encode "v2")})
(lmdb/multi-get store [:k1 :k2])
;; => {:k1 #object[[B ...], :k2 #object[[B ...]}

(lmdb/del store :key)

(lmdb/release-store store)

Important: Direct API and Konserve API use different storage formats and are not interoperable. Data written with lmdb/put cannot be read with k/get and vice versa. Choose one API for each store.

Ordered Access (format v2)

Keys are stored in value order, so range and prefix scans mean something. konserve's protocols have no range query, so this comes through konserve-lmdb's own surface.

(s/scan store {:from "a" :to "m"})     ; lazy [key value] pairs, :to exclusive
(s/scan-keys store {:prefix ["user"]}) ; keys only -- reads no value pages
(s/scan store {:reverse? true :limit 10})            ; latest 10
(s/scan store {:key-pred #(str/starts-with? % "x")}) ; pushed to the cursor

scan and scan-keys return a Range, which is Seqable, IReduceInit, Counted and Sequential — so reduce, transduce, into, first, take, filter, some and count all work on it directly. There is no parallel range-filter/range-count/range-some family to learn, and nothing to close.

(reduce (fn [acc [k v]] ...) init (s/scan store {:prefix ["user"]}))
(transduce (map second) + 0 (s/scan store))
(count (s/scan-keys store {:prefix ["user"]}))   ; walks keys only

Range spec

key	meaning
:from / :to	bounds, as user keys
:from-inclusive?	default true
:to-inclusive?	default false
:prefix	tuple keys starting with these elements
:reverse?	walk descending
:key-pred	predicate on the decoded key, applied at the cursor
:limit	max entries
:chunk-size	entries per read transaction (default 512)

:from-inclusive? and :to-inclusive? cover all nine of the usual range types (:closed, :closed-open, :open, :open-closed, :at-least, :greater-than, :at-most, :less-than, :all), and :reverse? doubles them.

:key-pred is a pushdown: a key it rejects never has its value copied out of the mapping, let alone decoded.

Prefix scans are exact: a scan for ["user"] does not match ["users" 1], because the string terminator lands where "users" has its s.

Two things worth knowing:

• A scan is not a consistent snapshot. It runs as a series of short read transactions, so writes landing between chunks are visible. This is deliberate: LMDB reuses freed pages only once no older read transaction remains, so a scan holding one long-lived transaction pins the freelist and the database grows instead of being reclaimed. Tune with :chunk-size.
• These refuse on a v1 store rather than return a plausible wrong set. v1 keys sort by (type, length, content), so "z" sorts after "aa" and negative integers sort last.

Keys: supported types and ordering

Because keys are stored in value order (format v2), the encoding only accepts types it can lay out order-preservingly. Supported key types:

Type	Notes
nil
boolean	false < true
integer	any Long/Integer/Short/Byte, widened to Long
double	Double/Float; NaN is rejected (it has no order)
string	ordered by Unicode codepoint (UTF-8 byte order)
keyword, symbol	ordered by their namespace/name string
java.util.UUID
java.util.Date	epoch millis (a java.sql.Timestamp key truncates to millis)
java.time.Instant	epoch seconds + nanos (full nanosecond precision)
sequences	vectors/lists/seqs, compared element-by-element

Anything else — maps, sets, ratios, bigints, char — throws at write time rather than being silently coerced. This is deliberate for a beta: a key you cannot order back is a key you cannot range-scan, so it fails loudly instead. (konserve's own EDN keyspace is wider; these are the types this store can order. Support for more can be added later without a format change, since each gets its own tag.)

Two ordering caveats worth knowing:

• Types are segregated by tag — there is no cross-type numeric order. All integers sort before all doubles, so 5 < 5.5 < 6 does not hold if you mix Long and Double in one keyspace. Likewise a Date and an Instant at the same instant do not sort together. Keep one type per key position.
• Strings order by codepoint, which is UTF-8 byte order. Clojure's compare uses UTF-16 code-unit order; the two disagree only on supplementary (astral-plane) characters. The codepoint order is the one scans see.

Projection (reading one field without decoding the blob)

project walks a key range and pulls one field out of each stored value without materialising the value. The store's Range picks the rows; boring's navigator picks the columns.

(require '[konserve-lmdb.store :as s])

(def store (s/connect-store "/tmp/db" :navigable? true :index 16))

;; [user-key projected] for every key under the prefix
(s/project store {:prefix ["user"]} [:profile :address :city])

;; several fields in one pass, reduced as it goes
(s/project-reduce store {:prefix ["user"]} [[:profile :city] [:revenue]]
                  (fn [acc _k [city revenue]]
                    (update acc city (fnil + 0.0) revenue))
                  {})

path is a get-in-style path; an integer step indexes a vector. Both refuse on a store that is not navigable rather than quietly falling back to a full decode, because a silent fallback turns the whole point of the call into a performance mystery.

The two options, and what they cost

option	wire	navigable	size
(default)	stringref	no	1.00
:navigable? true	plain	yes	~1.17
:navigable? true :index N	plain + index frame	yes, with jumps	~0.73

:index N is the stride — how many entries share one index anchor. It also re-enables stringref for the value, which is why an indexed blob comes out smaller than a navigable one: measured 5245 B against 7229 B on a crawled-page record, 27% less.

When it wins, and when it does not

The cost of reaching a field is proportional to the number of CBOR items before it, because a CBOR container carries an element count rather than a byte length — so stepping over a value means walking its subtree. An index replaces that walk with a jump.

Measured directly, sweeping only the number of items before the probed field and holding everything else still (clojure -M:bench-crossover):

items before the field	no index	:index 16	PostgreSQL
32	0.80 µs	0.73 µs	1.00 µs
160	1.68	1.80	1.41
288	2.38	1.69	4.75
512	4.02	1.75	7.33
2048	15.36	1.94	29.77
8192	56.84	2.79	108.96

Three numbers come out of that, and they are the ones to design against:

• A walk costs ~7 ns per CBOR item. That is the slope of the unindexed column, and it is what you pay for everything sitting in front of your field.
• An indexed read is flat: 1.7–2.8 µs from 160 items to 8192, while the walk goes 1.7 → 56.8. The index decouples the cost from the document.
• The crossover is a few hundred items. Below ~200 the two are within noise of each other; the index pulls away durably from ~300. Under that, the frame open costs more than the walk it saves.

So the rule is about field position and document size, not about document count:

• A field near the front is nearly free with or without an index.
• A field behind bulk — after a large array or map — costs the whole bulk without an index, and roughly nothing with one.
• Small documents gain little; the index frame has to be opened.
• Touching every field is faster with konserve/get and a plain decode. Projection pays per lookup what a decode pays once.

Where we lose

Around 150–200 items the sweep shows Postgres ahead — 1.41 µs against our 1.68 unindexed and 1.80 indexed. That band is real and it is not an artefact: the document is still small enough to sit inline in the row, so Postgres has no detoast to pay, while it is already big enough to cost us a walk. Its cost steps up sharply just above it, exactly where pg_column_size stops tracking the raw size and the datum goes out of line.

If your documents are ~1–2 KB and you read one field from each, Postgres JSONB is a reasonable choice and this store has no structural advantage to offer.

Measured, 300 rows, a record whose probed field sits after a pad-element array, ZFS, power-saver pinned:

pad	no index	:index 16
400	10.15 ms	3.61 ms
3000	55.99 ms	1.48 ms

The indexed cost barely moves as the document grows 400x, because it stops tracking document size at all.

Against PostgreSQL JSONB

Same records, same machine, count(doc->'tail'->>'city') against project, comparing Postgres' own EXPLAIN ANALYZE execution time (the JDBC round trip is not a property of JSONB and floors around 14 ms here):

pad	ours	Postgres
400	3.61 ms	24.34 ms	6.7x
3000	1.48 ms	73.07 ms	49.5x

The advantage is not better navigation. JSONB is, if anything, structurally better at it: its JEntry array stores a length per element with an absolute offset every 32nd, so stepping over a value is a pointer add, and it binary-searches sorted object keys. Two designs converging is evidence ours is sane, not that it is faster.

The advantage is that we never materialise the document. PG_GETARG_JSONB_P is a full detoast — there is no DETOAST_DATUM_SLICE on any of the path entry points in jsonfuncs.c — so every extraction decompresses and rebuilds the whole value before its O(1) navigation runs. Its cost tracks document size whatever field you asked for; ours tracks how much sits in front of that field. The index is what keeps the second number from growing.

Reproduce with clojure -M:bench-jsonb and clojure -M:bench-composite; both print the filesystem and the CPU power profile they ran under, because a governor change between two arms moves the numbers more than most of the effects being measured.

Versioned Stores (experimental)

Opt in at creation. Writes then append a new version instead of overwriting, which buys copy-on-write semantics, reads as of a point in the past, and history.

(require '[konserve-lmdb.versioned :as v])

(def store (s/connect-store "/tmp/vstore" :versioned? true))

(k/assoc store :a 1 {:sync? true})
(k/assoc store :a 2 {:sync? true})
(k/get store :a nil {:sync? true})        ;; => 2, newest version wins

(s/history store :a)                      ;; => ([<hlc-2> 2] [<hlc-1> 1]) newest first, decoded
;; read a past value with a pin taken earlier (see with-pin / current-sequence):
(s/as-of   store :a a-pin)                ;; => the decoded value as of that point

The store-level s/history, s/as-of and s/latest return decoded values; the lower-level v/* variants under konserve-lmdb.versioned return the raw [coordinate bytes] if you need them.

It still satisfies PEDNKeyValueStore — "newest version wins" is a faithful implementation of a mutable document interface. -dissoc writes a tombstone rather than removing, so a reader holding a pin from before the delete still sees the value; the key reads as absent and is not listed by keys.

Versions are ordered by a hybrid logical clock (HLC). The version coordinate is a 64-bit value: the high 48 bits are wall-clock milliseconds, the low 16 a logical counter advanced as max(now, last) + 1. It carries wall-time — which is what makes time-based as-of/since/retention possible — while keeping every guarantee a plain counter gave. A bare clock could not be used directly: two writes in the same millisecond would collide on one key and the second would overwrite the first, and a skewed clock across processes could place a later write at an earlier position. The HLC's logical low bits and monotonic advance remove both hazards. The key layout is unchanged (an 8-byte ordered coordinate), so it is seamless over a store written before the HLC.

Garbage collection

Every write grows the store and nothing shrinks it until GC runs. That is inherent to copy-on-write, so GC is part of the feature, not an optional extra — especially since LMDB never returns freed pages to the OS, making pre-GC bloat a permanent high-water mark on the file.

;; :max-pin-age-ms bounds how long a pin can hold back collection.
(def registry (v/pin-registry {:max-pin-age-ms 60000}))

;; Hold a pin while reading the past; GC will not collect under it.
(v/with-pin [pin registry current-seq]
  (v/as-of (:env store) :a pin))

;; Mark runs in read transactions, split across key ranges and marked
;; concurrently. Only the sweep takes the write lock, in short batches.
(v/gc! (:env store) registry current-seq
       {:parallelism 4     ; mark threads
        :batch-size  1000  ; deletions per write transaction
        :cutoff      50000}) ; bound one pass over a bloated store
;; => {:scanned 1200 :collected 1150 :skipped 0}

Mark and sweep are separate, which is what keeps the pause small. Marking never blocks the writer; the ordered keyspace lets it be split across ranges and run in parallel, with split points snapped to group boundaries so no version group is ever half-judged. The sweep re-checks before deleting a whole group, since the single writer may have committed a new version between mark and sweep — those groups are skipped and reported in :skipped.

:cutoff bounds a single pass, so a first collection over a badly bloated store can be run in slices rather than one enormous transaction. Repeated slices converge to the same result.

Returning space to the OS

gc! frees pages for LMDB to reuse; the file itself never shrinks. To actually reclaim disk:

(n/compact-to! (:env store) "/path/new")

This copies the environment omitting free pages. It runs in a read transaction so it does not block writers, but the copy is a snapshot — writes landing during it are not included. Needs room for both copies, and per lmdb.h it fails outright if the environment has suffered a page leak.

This copy-and-swap is inherent to LMDB: a copy-on-write B-tree with no background compaction cannot shrink in place. An LSM store like RocksDB compacts in the background instead, and pays read amplification for it.

The rule: delete every version strictly older than the newest version at or below the watermark, where the watermark is the oldest live pin (or the current sequence when none is live).

This is much simpler than tombstone collection in a CRDT, where you cannot drop a tombstone until you know every replica has seen the delete — that needs causal stability, hence coordination. Here there is a single writer and a total order, so the safe point is a local computation.

Three things to know:

• A leaked pin stalls the watermark, and since LMDB never returns space that is an unrecoverable disk commitment. Use with-pin, which releases in a finally, and set :max-pin-age-ms so a pin leaked by a crash eventually lapses. An expired pin then fails loudly on use rather than quietly serving state GC may already have collected.
• The pin registry is in-process. A pin held in another process is invisible, so multi-process deployments must not run GC unattended.

Migrating a v1 store

Do you have to migrate? No. A store created before format v2 keeps working untouched — it stays a v1 store forever, with one warning logged at connect. Its keys remain v1 (buffer) keys for the whole store; new writes are never silently mixed into v2 (v1 and v2 keys are byte-indistinguishable, so mixing would corrupt ordering). New values, however, are written in the current boring format, so an actively-written v1 store ends up with old values in the legacy layout and new ones in boring — both decode fine, nothing breaks. What a v1 store cannot do is the ordered-key features: scan / scan-keys / project and versioned history all refuse on it. Migrating is how you opt into those.

To get ordered access, migrate:

(require '[konserve-lmdb.migrate :as mig])

(mig/migrate-store! "/path/old" "/path/new"
                    :type-handlers my-buffer-handlers  ; to READ v1 custom types
                    :registry      my-boring-registry  ; to WRITE them back
                    :progress-fn   (fn [n] (println n "entries")))
;; => {:entries 12345 :verified true}

This copies to a new store and leaves the source untouched — not caution, necessity. An in-place migration would mix old- and new-encoded keys in one database with no way to tell them apart (every v2 tag is also a valid v1 tag), so a crashed run would be unrecoverable.

Swap and delete the source once you are satisfied.

Configuration Options

Environment Flags (combine with bit-or):

(require '[konserve-lmdb.native :as n])

;; Example with flags
(def config
  {:backend :lmdb
   :path "/tmp/my-store"
   :id #uuid "550e8400-e29b-41d4-a716-446655440000"
   :flags (bit-or n/MDB_NORDAHEAD n/MDB_NOTLS)
   :registry registry})  ; boring tag-registry for custom types

Store options (to connect-store):

• :navigable? true — store values so project can read a field without decoding the value. Costs ~17% more bytes on its own.
• :index N — additionally seal an index frame with stride N, so reaching a field behind bulk becomes a jump rather than a walk. Re-enables stringref for the value, so the blob comes out ~27% smaller than plain navigable. Prefer 16 over 1: measured 1.48 ms against 3.35 for the same scan, with a smaller blob. See Projection.
• :index-min N — smallest container that earns an index node. Leave it unset; boring's default is chosen from measurement and this store used to override it with a value that excluded the one container that mattered.
• :registry — a boring tag-registry for custom types. This is the current path: it governs how boring encodes and decodes your values (see Custom types).
• :type-handlers — legacy. A konserve-lmdb.buffer registry, consulted only to decode old v1 (buffer-encoded) blobs that carried custom types. New writes never use it. Leave it unset unless you are reading a pre-boring store.
• :versioned? true — append a new version on every write instead of overwriting, enabling history / as-of / latest (see Versioned Stores). Fixed at creation: a store cannot be reopened with the other setting, because it changes what dissoc means.
• :format :v1|:v2 — force the key format instead of detecting it. A new store defaults to v2; an existing one keeps what it was created with. Opening a v1 store with :format :v2 throws rather than corrupting it — migrate instead.

Environment Flags:

• n/MDB_NORDAHEAD - Don't use read-ahead; reduces memory pressure for large datasets
• n/MDB_RDONLY - Open in read-only mode; allows concurrent reading while another process writes
• n/MDB_NOSYNC - Don't fsync after commit; faster but less durable (use for ephemeral data)
• n/MDB_WRITEMAP - Use writeable mmap; faster for RAM-fitting DBs but less crash-safe
• n/MDB_MAPASYNC - Async flushes when using WRITEMAP; requires explicit sync for durability
• n/MDB_NOTLS - Disable thread-local storage; needed for apps with many user threads on few OS threads

Flags can be combined with bit-or:

(lmdb/connect-store path :flags (bit-or n/MDB_NORDAHEAD n/MDB_NOSYNC))

LMDB Best Practices & Caveats

LMDB is a powerful but low-level storage engine. Here are important considerations:

Do NOT Use LMDB On Network Filesystems

LMDB uses memory-mapped files and POSIX locking. Never store LMDB databases on NFS, CIFS, or other network/remote filesystems - this will cause data corruption.

Database File Growth

LMDB's database file never shrinks automatically. Deleted data frees pages internally for reuse, but the file size remains. To reclaim space, copy the database with compaction:

mdb_copy -c /path/to/db /path/to/compacted-db

Map Size Configuration

Set map-size large enough for your expected data. LMDB pre-allocates virtual address space (not physical memory). On 64-bit systems, setting 100GB+ is safe and recommended for growing databases:

(lmdb/connect-store path :map-size (* 100 1024 1024 1024)) ; 100GB

Long-Running Processes

For servers running continuously, be aware that:

1. Stale readers - If a read transaction is abandoned (e.g., thread dies), it prevents space reuse until detected. LMDB has mdb_reader_check() but it's not exposed here yet.

2. Keep transactions short - Long-lived read transactions prevent freed pages from being reclaimed, causing database growth. The konserve and Direct APIs handle this correctly with short-lived transactions.

MDB_WRITEMAP Warning

While MDB_WRITEMAP is faster, it has risks:

• Buggy code can corrupt the database by writing to mapped memory
• Filesystem errors may crash the process instead of returning errors
• Use only when performance is critical and you have good backups

Thread Safety

LMDB environments are thread-safe. You can share a single store across all threads. However:

• Write transactions are serialized (one at a time)
• Read transactions provide MVCC isolation
• Don't pass cursors between threads

Custom types

Values are encoded with boring (CBOR). To teach the store a custom type, build a boring tag-registry and pass it as :registry. A registry is an immutable value; each register-tag returns a new one. A handler is a CBOR tag number plus a write-fn (value → something encodable) and a read-fn (decoded content → value):

(require '[boring.core :as boring])

(def registry
  (-> (boring/tag-registry)
      ;; tag, class, write-fn, read-fn
      (boring/register-tag 40001 java.net.URI str #(java.net.URI. %))))

(def store (lmdb/connect-store "/tmp/store" :registry registry))

Records need no write-side registration (a record encodes under its own type name); register a constructor with boring/register-record only so the reader can rebuild it. See boring's register-tag / register-record docstrings for the full API, and konserve-lmdb.pss/boring-registry for a worked example that folds persistent-sorted-set (and datahike) handlers into one registry.

Legacy (v1 only). Pre-boring stores used a konserve-lmdb.buffer ITypeHandler registry passed as :type-handlers. New writes never use it; it is consulted only to decode old buffer-encoded blobs. If you are reading a store created before the boring format, pass that registry as :type-handlers — otherwise ignore it.

Performance

Benchmarks comparing konserve-lmdb against datalevin's raw KV API.

Test setup: 1000 entries, ~50 bytes per value (map with UUID, timestamp, counter)

Re-measured against format v2, on power-saver:

Operation	Native	Direct	Konserve	Datalevin	Direct vs Datalevin
Single Put	195K	168K	60.9K	122K	1.38x
Single Get	506K	450K	314K	293K	1.54x
Batch Put	1.22M	851K	91.8K	344K	2.47x
Batch Put (imm)	1.22M	851K	163K	344K	2.47x

Operations per second, measured with criterium

Do not compare these against the previously published table (557K / 1.43M / 3.52M for Native). Every arm came out ~3x lower on re-measurement — including datalevin, which we did not change — so what moved is the machine, not the store. The old table recorded no CPU power profile and was almost certainly taken on performance; this one is on power-saver. The harness now prints the profile so this cannot recur.

What IS comparable across the two runs is the ratios, and they hold: Direct beat datalevin on all three operations then and does now, by a wider margin on batch put (1.47x → 2.48x).

One thing worth a look: the Konserve API's share of Direct fell (batch put was 29% of Direct, now 10%). That may be real — metadata tracking now goes through boring — or it may be the same machine-state effect amplified. It has not been isolated.

Why the Konserve API costs more, and how to get it back

Every konserve write is a read-modify-write of metadata: meta-up-fn is handed the value's existing metadata, so the store must read and decode it before it can write. Timed per component, 1000 keys, one arm per JVM:

	µs/key	share
read old metadata	7.0	19%
decode old metadata	8.3	22%
encode metadata + value	8.3	22%
batched write	8.1	22%
konserve machinery	~5.9	16%

The write itself is only 22%. The Direct API skips the first three rows entirely, which is the whole gap — it is not indexing (the benchmark store is plain) and not the codec.

An immutable batch skips the read. A content-addressed value is written once, so there is no prior metadata worth merging — mark the batch and the read-modify-write disappears:

(k/multi-assoc store nodes nil (assoc opts :meta-all {:immutable? true}))

	µs/key	batch put
ordinary batch	52.6	91.8K ops/s
:meta-all {:immutable? true}	27.0 (1.95x)	163K ops/s (1.78x)

End to end that takes the Konserve API from 11% of the Direct API's batch throughput to 19%. It stays below datalevin's raw KV batch (344K), and it should: datalevin's number carries no metadata at all, which is the comparison the Direct API answers (2.47x).

This is the shape datahike's node storage uses. For genuinely mutable values the read is required by konserve's contract and the cost is real.

Key findings:

• Direct API is faster than datalevin for all operations
• Konserve API adds ~1-2µs overhead per operation for metadata tracking
• Batch operations are 3-10x faster than sequential puts

Projection

A different axis: not operations per second, but the cost of reading one field out of many values. See Projection for the model; these are the numbers.

Scan-and-extract, 300 values, the probed field sitting after a pad-element array, against PostgreSQL 14 JSONB with the same documents. The Postgres column is its own EXPLAIN ANALYZE execution time, excluding the JDBC round trip:

pad	blob	no index	:index 16	Postgres	vs Postgres
400	10 KB	10.15 ms	3.61 ms	24.34 ms	6.7x
3000	76 KB	55.99 ms	1.48 ms	73.07 ms	49.5x

The indexed column barely moves as the document grows; the other two grow with it. That is the whole claim.

Narrow-then-project, 2000 crawled-page records, selecting one domain by key prefix and pulling a field out of each match:

matched	no index	:index 16	Postgres	vs Postgres
2000	43.00 ms	27.84 ms	63.57 ms	2.3x
1000	18.06 ms	10.05 ms	62.98 ms	6.3x
500	8.02 ms	2.19 ms	13.17 ms	6.0x

Both were run on ZFS with the CPU pinned to one power profile, which the harnesses print — a governor change between two arms moves these numbers more than most of the effects being measured.

Run benchmarks yourself:

clojure -M:bench              # KV throughput, against datalevin
clojure -M:bench-jsonb        # projection against PostgreSQL JSONB, swept by size
clojure -M:bench-composite    # narrow-then-project against PostgreSQL JSONB

The two JSONB benchmarks need a reachable PostgreSQL; see the namespace docstrings for the connection settings and for what each one is and is not evidence of.

API Reference

Store Management

Multimethod API (konserve.core):

• (k/create-store config) - Create/open an LMDB store via multimethod dispatch
• (k/connect-store config) - Connect to existing LMDB store (same as create for LMDB)
• (k/store-exists? config) - Check if store directory exists
• (k/delete-store config) - Delete store and all data

Direct API (konserve-lmdb.store):

• (lmdb/connect-store path & opts) - Create/open an LMDB store directly
• (lmdb/release-store store) - Close the store
• (lmdb/delete-store path) - Delete store and all data

Direct API (High Performance)

Bytes in, bytes out — the caller owns value encoding.

• (put store key value-bytes) - Store bytes at key
• (get store key) - Get value bytes by key, or nil
• (get-decode store key decode-fn) - Zero-copy read; decode-fn gets a ByteBuffer valid only for the call
• (del store key) - Delete key
• (multi-put store kvs) - Store multiple [key value-bytes] pairs atomically
• (multi-get store keys) - Get multiple values as bytes

Ordered access & projection (konserve-lmdb.store)

Format v2 only; these refuse on a v1 store.

• (scan store range-spec) — ordered [key value] pairs over a range (a Range: seqable/reducible/counted)
• (scan-keys store range-spec) — ordered keys only, reads no value pages
• (project store range-spec path) — [key projected] over a range, without materialising values (needs :navigable?)
• (project-reduce store range-spec paths f init) — reduce a multi-field projection over a range

Versioned / temporal (konserve-lmdb.store)

:versioned? true stores only; each write appends a version keyed by an HLC coordinate.

• (latest store key) — current decoded value, or nil
• (as-of store key pin) — decoded value as of a pin (from current-sequence or versioned/with-pin)
• (history store key) — every [sequence value], newest first, decoded
• (current-sequence store) — allocate and return the current sequence (take a pin, or feed gc!)
• (gc! store registry opts) — collect versions no live pin can reach
• (compact! store path) — copy the store to path omitting free pages (the only way to shrink the file)

Migration (konserve-lmdb.migrate)

• (migrate-store! from-path to-path & opts) — copy a v1 store to a new v2 store (source untouched)

Konserve Protocols

The store implements all standard konserve protocols:

• PEDNKeyValueStore - get-in, assoc-in, update-in, dissoc
• PBinaryKeyValueStore - bassoc, bget
• PKeyIterable - keys enumeration
• PMultiKeyEDNValueStore - multi-get, multi-assoc, multi-dissoc
• PLockFreeStore - indicates MVCC-based concurrency

Experimental namespaces

Beyond the core store, a few experimental namespaces ship behind their own deps.edn aliases so their dependencies stay opt-in. Their APIs may change.

Namespace	What it does	Alias
konserve-lmdb.fork	Physical, O(1) fork of a whole store via filesystem copy-on-write (ZFS today). A fork duplicates every key in constant time, the copies sharing blocks until they diverge — a cheap read replica, throwaway experiment, or live backup. See doc/FORKING.md.	:zfs
konserve-lmdb.pss	A boring tag-registry carrying persistent-sorted-set (and datahike) handlers, to pass as :registry — so a PSS can be stored and navigated in place.	:yggdrasil
konserve-lmdb.yggdrasil	A yggdrasil adapter that maps its clone=branch model onto ZFS forks, turning a store into a branchable, durable history.	:yggdrasil

These are not required for normal use and are not on the default classpath.

Development

# Run tests
clojure -M:test

# Run benchmarks
clojure -M:bench

# Format code
clojure -M:ffix

# Build jar
clojure -T:build jar

License

Copyright © 2025 Christian Weilbach

Licensed under Eclipse Public License 2.0 (see LICENSE).