The index store on disk — a KvBackend (vaelii.impl.kv) over a durable
write-ahead log.
The index is derived state (small next to the records, and reindex can rebuild it
from the records alone), so the disk KV keeps the whole key→value map in RAM — a Long at
each counter key, a set at each set key, exactly the shape MemoryKvBackend holds —
and durably logs every mutation to a kv.log of length-prefixed nippy frames. Reads
are the in-RAM map, so kv-members / kv-intersect are the same reference /
set-intersection operations the memory backend does — the disk only buys durability.
Logical (op) logging. A frame is the write op itself — [:add-to-set k m], [:remove-from-set k m], [:put k v], [:delete k], [:increment k], [:decrement k] — not the resulting value. A
set-add logs the one added member, O(1), so a bulk load of N members into one root
writes O(N) WAL bytes; new-value logging re-serialized the size-i set on the i-th add
and cost O(N²). On open the log replays by folding each frame through apply-op, the
same function that applies a live op. compact! rewrites the log as one [:put k v]
op per live key, so every frame — ordinary or post-compaction — is a uniform op and
the reader needs no snapshot-vs-delta discrimination; it also bounds replay length and
reclaims the delta frames (compaction is this store's snapshot cadence).
Crash-safety: scan-log truncates a torn tail on open (a partial op frame is dropped
whole, never half-applied), and compaction rewrites the log crash-safely
(files/recover-log-compaction!). All log writes hold the backend lock (the RAF file
pointer is shared).
The index store on disk — a `KvBackend` (`vaelii.impl.kv`) over a durable write-ahead log. The index is derived state (small next to the records, and `reindex` can rebuild it from the records alone), so the disk KV keeps the whole key→value map in RAM — a `Long` at each counter key, a set at each set key, exactly the shape `MemoryKvBackend` holds — and durably logs every mutation to a `kv.log` of length-prefixed nippy frames. Reads are the in-RAM map, so `kv-members` / `kv-intersect` are the same reference / `set-intersection` operations the memory backend does — the disk only buys durability. **Logical (op) logging.** A frame is the write op itself — `[:add-to-set k m]`, `[:remove-from-set k m]`, `[:put k v]`, `[:delete k]`, `[:increment k]`, `[:decrement k]` — not the resulting value. A set-add logs the one added member, O(1), so a bulk load of N members into one root writes O(N) WAL bytes; new-value logging re-serialized the size-i set on the i-th add and cost O(N²). On open the log replays by folding each frame through `apply-op`, the same function that applies a live op. `compact!` rewrites the log as one `[:put k v]` op per live key, so every frame — ordinary or post-compaction — is a uniform op and the reader needs no snapshot-vs-delta discrimination; it also bounds replay length and reclaims the delta frames (compaction is this store's snapshot cadence). Crash-safety: `scan-log` truncates a torn tail on open (a partial op frame is dropped whole, never half-applied), and compaction rewrites the log crash-safely (`files/recover-log-compaction!`). All log writes hold the backend lock (the RAF file pointer is shared).
(close! {:keys [dir log lock] :as b})Compact if the deltas have earned it, flush durably, record the length the WAL closed at, and close it. The next open skips the torn-tail walk while that length still agrees.
Why compact here. Opening this store is a replay: every frame is thawed and folded
through apply-op, so the open costs the frame count, not the live-key count. A
bulk load leaves those far apart — 5.81M frames against 2.01M live keys on a 300k-fact
KB, a 0.65 dead ratio — and a compaction collapses each key's delta chain to the one
[:put k v] a replay actually needs. Closing is the moment to pay for it: the writer
is done, and the cost lands once instead of on every subsequent open.
Gated on the same switch and threshold the background tick uses
(vaelii.impl.disk.durability), so a store closed just after a compaction does not
rewrite its log for nothing, and one knob turns both off.
Compact if the deltas have earned it, flush durably, record the length the WAL closed at, and close it. The next open skips the torn-tail walk while that length still agrees. **Why compact here.** Opening this store is a replay: every frame is thawed and folded through `apply-op`, so the open costs the *frame count*, not the live-key count. A bulk load leaves those far apart — 5.81M frames against 2.01M live keys on a 300k-fact KB, a 0.65 dead ratio — and a compaction collapses each key's delta chain to the one `[:put k v]` a replay actually needs. Closing is the moment to pay for it: the writer is done, and the cost lands once instead of on every subsequent open. Gated on the *same* switch and threshold the background tick uses (`vaelii.impl.disk.durability`), so a store closed just after a compaction does not rewrite its log for nothing, and one knob turns both off.
(compact! {:keys [data log log-path lock frames]})Crash-safely rewrite the WAL as one [:put k v] op frame per live key — the
snapshot that collapses every accumulated delta chain. Every frame in the log,
post-compaction or ordinary, is thus a uniform op replayed by the same apply-op
fold, so the reader needs no snapshot-vs-delta discrimination. Holds the lock, so
writers wait; the RAM map is untouched, so reads never block.
Crash-safely rewrite the WAL as one `[:put k v]` op frame per live key — the snapshot that collapses every accumulated delta chain. Every frame in the log, post-compaction or ordinary, is thus a uniform op replayed by the same `apply-op` fold, so the reader needs no snapshot-vs-delta discrimination. Holds the lock, so writers wait; the RAM map is untouched, so reads never block.
(dead-ratio {:keys [data frames]})The delta-accumulation ratio: 1 - live-keys / frames-written, the fraction of the
WAL that a compaction would collapse. Under logical logging each op is one frame, so
a key touched K times since the last snapshot carries K-1 frames a [:put k v]
snapshot would fold away; frames drops to the live-key count on compact!, so this
reads 0 immediately after and climbs as deltas accumulate against the live footprint.
The delta-accumulation ratio: `1 - live-keys / frames-written`, the fraction of the WAL that a compaction would collapse. Under logical logging each op is one frame, so a key touched K times since the last snapshot carries K-1 frames a `[:put k v]` snapshot would fold away; `frames` drops to the live-key count on `compact!`, so this reads 0 immediately after and climbs as deltas accumulate against the live footprint.
(fsync {:keys [log lock]} fsync?)fsync the WAL. fsync? false drains to the page cache without the fsync.
fsync the WAL. `fsync?` false drains to the page cache without the fsync.
(open-kv-backend dir)Open a DiskKvBackend rooted at dir/index, replaying kv.log into the RAM map
(recovering any interrupted compaction and truncating a torn tail first).
Open a `DiskKvBackend` rooted at `dir/index`, replaying `kv.log` into the RAM map (recovering any interrupted compaction and truncating a torn tail first).
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |