The record store on disk — an implementation of RecordStore over per-kind
log/idx pairs (vaelii.impl.disk.files).
Three int-keyed kinds — sentexes, justifications, provenance — each a .log of
length-prefixed nippy frames plus a .idx of fixed 24-byte slots mapping handle →
log offset. A frame holds its record's fields positionally
(vaelii.impl.disk.codec), so the type tag and field names are not rewritten into
every one of them; a frame written before that codec still reads, as its own shape. A record is paged from disk on get: read the slot, read the frame
it points at, thaw it — two positional reads, no seek, and the records do not sit in
RAM. What does sit in RAM per kind is the set of live handles (so enumeration is
O(1)), rebuilt from the idx on open, and a bounded LRU of hot records in front
of the read (vaelii.disk.cache, 0 to disable).
That live set is a compressed bitmap (vaelii.impl.handle-set's LiveHandleSet), not a
PersistentHashSet<Long>. Handles are minted in assertion order, so a live set is a
strided run of longs with holes where records were deleted, and the boxed set retains
48–75 bytes a handle for it — 9.47 GB at 100M records, the second-largest resident row
in the engine (docs/density.md). A bitmap answers all four things the
set is asked (membership, iteration, cardinality, a first handle) at about a bit a
handle. What it costs is a monitor: the bitmap is mutated in place and is not
thread-safe, so a read of the live set takes the kind lock, exactly as a write does —
see the two-monitors section below.
next-id is a monotonic counter recovered as max(the counters blob, 1 + the highest slot id across the record kinds) — the highest slot id is stable across
deletes (a tombstone keeps its slot) and across compaction (slot ids are preserved),
so a handle is never reused even if the counters blob is stale after a crash. Within
a session every write holds the same bound (clear-counter!), because a record can
arrive carrying its own :id and nothing re-reads the slots until the next open.
A premise is exactly a sentex whose :strength is non-nil (the strength lives on
the record, as on every backend), so the premise set is derived from the durable
records — rebuilt on open, maintained in lockstep by mark/unmark — rather than
stored separately. Rebuilding it does not mean reading them: every write records
the answer in its idx slot's flags, so the open reads the set off the slot walk it
already makes. A slot that does not carry the bit sends that one handle to its
record, and the record is authoritative wherever both speak (rebuild-premises!).
The premise's strength rank rides the same slot flags (bits 2..3), so
premise-strength — read once per premise on every recover — answers off the 24-byte
slot instead of paging the whole record for one keyword. A slot carrying no rank (a
non-premise, or one older than the bits) falls back to the record, the same
no-format-bump story as the premise bit itself (f/slot-strength).
Recovery on open: finish any interrupted compaction, truncate a torn log tail, then
tombstone any slot whose frame now extends past the log (validate-idx-tail!).
Crash-safety rests on the write ordering (append the frame, then point the slot at it)
and on files' crash-safe compaction. Where it stops is the slot itself: 24 bytes do
not divide a page, so a crash can leave one spliced from two writes, and a splice still
pointing inside the log is reported as a thaw failure on that handle rather than caught
here (f/validate-idx-tail! says what it does and does not cover).
The tail is located from the frame lengths
(files/log-tail-offset) and nothing is decoded to find it — and a clean close!
records each log's length, so an open whose log is still that long skips even the walk.
The marker is consumed here, so it never describes a store in use; every disagreement
falls back to the walk.
Every RAF touch holds the owning kind's lock. A write or force! must, because the
file pointer is shared (see files' shared-pointer invariant); a read is positional
and need not, but still does, because that is what serializes it against a concurrent
append and its slot write.
Two monitors, and which resident field sits under which. Three threads touch this
store — the writer, the durability daemon (fsync, every few seconds) and the
compaction executor — so the resident state is not the writer's alone and a field
mutated outside a monitor is one a reader can catch mid-pair.
The kind lock covers that kind's log, its idx, and the resident state derived
from them: live-ids, live-bytes, the hot-record cache, compacting and
failed. A store, a kill, a batch and the compactor's reconcile each take it once
and do both halves inside it, so an id is never live to a reader while its slot says
tombstone, and the compaction delta set is never cleared under a writer folding an id
into it.
It covers live-ids on the read side too, which the other three do not need: the
live set is a bitmap mutated in place, so a tally or an enumeration taken beside a
concurrent addLong reads a structure mid-edit. A read that hands the set onward
takes a hs/live-snapshot inside the lock and lets go of it — the copy costs the
bitmap's size, not the corpus's, which is what makes holding the writer's monitor for
an enumeration affordable at all.
counters-lock covers the three that move together and belong to no kind: the
handle counter, the counters.nippy blob, and synced-seq — read and written by
fsync on the daemon's thread and by clear-records! on the writer's. Its own
monitor rather than a kind's, because a whole-file blob rewrite held inside a kind
lock would put a record append behind it every tick that minted a handle.
premises is a LiveHandleSet too, and sits under the sentexes kind lock for reads
and writes alike, because a premise is a sentex handle. It is not folded into
store!'s acquisition: a premise joins the premise set after its record lands, in a second
acquisition, so a failed write leaves no handle in premise-ids whose get-sentex is
nil. The compactor's drop-lost! removes from it only when compacting the sentexes
kind, under the lock it already holds.
The record store on disk — an implementation of `RecordStore` over per-kind log/idx pairs (`vaelii.impl.disk.files`). Three int-keyed kinds — sentexes, justifications, provenance — each a `.log` of length-prefixed nippy frames plus a `.idx` of fixed 24-byte slots mapping handle → log offset. A frame holds its record's fields **positionally** (`vaelii.impl.disk.codec`), so the type tag and field names are not rewritten into every one of them; a frame written before that codec still reads, as its own shape. A record is **paged** from disk on `get`: read the slot, read the frame it points at, thaw it — two positional reads, no seek, and the records do not sit in RAM. What does sit in RAM per kind is the set of live handles (so enumeration is O(1)), rebuilt from the idx on open, and a **bounded LRU of hot records** in front of the read (`vaelii.disk.cache`, 0 to disable). That live set is a **compressed bitmap** (`vaelii.impl.handle-set`'s `LiveHandleSet`), not a `PersistentHashSet<Long>`. Handles are minted in assertion order, so a live set is a strided run of longs with holes where records were deleted, and the boxed set retains 48–75 bytes a handle for it — 9.47 GB at 100M records, the second-largest resident row in the engine (`docs/density.md`). A bitmap answers all four things the set is asked (membership, iteration, cardinality, a first handle) at about a bit a handle. What it costs is a monitor: the bitmap is mutated in place and is not thread-safe, so a read of the live set takes the kind lock, exactly as a write does — see the two-monitors section below. `next-id` is a monotonic counter recovered as `max(the counters blob, 1 + the highest slot id across the record kinds)` — the highest slot id is stable across deletes (a tombstone keeps its slot) and across compaction (slot ids are preserved), so a handle is never reused even if the counters blob is stale after a crash. Within a session every write holds the same bound (`clear-counter!`), because a record can arrive carrying its own `:id` and nothing re-reads the slots until the next open. A premise is exactly a sentex whose `:strength` is non-nil (the strength lives on the record, as on every backend), so the premise set is derived from the durable records — rebuilt on open, maintained in lockstep by mark/unmark — rather than stored separately. Rebuilding it does not mean *reading* them: every write records the answer in its idx slot's flags, so the open reads the set off the slot walk it already makes. A slot that does not carry the bit sends that one handle to its record, and the record is authoritative wherever both speak (`rebuild-premises!`). The premise's **strength rank** rides the same slot flags (bits 2..3), so `premise-strength` — read once per premise on every `recover` — answers off the 24-byte slot instead of paging the whole record for one keyword. A slot carrying no rank (a non-premise, or one older than the bits) falls back to the record, the same no-format-bump story as the premise bit itself (`f/slot-strength`). Recovery on open: finish any interrupted compaction, truncate a torn log tail, then tombstone any slot whose frame now extends past the log (`validate-idx-tail!`). Crash-safety rests on the write ordering (append the frame, then point the slot at it) and on `files`' crash-safe compaction. Where it stops is the slot itself: 24 bytes do not divide a page, so a crash can leave one spliced from two writes, and a splice still pointing inside the log is reported as a thaw failure on that handle rather than caught here (`f/validate-idx-tail!` says what it does and does not cover). The tail is located from the frame *lengths* (`files/log-tail-offset`) and nothing is decoded to find it — and a clean `close!` records each log's length, so an open whose log is still that long skips even the walk. The marker is consumed here, so it never describes a store in use; every disagreement falls back to the walk. Every RAF touch holds the owning kind's lock. A write or `force!` must, because the file pointer is shared (see `files`' shared-pointer invariant); a read is positional and need not, but still does, because that is what serializes it against a concurrent append and its slot write. **Two monitors, and which resident field sits under which.** Three threads touch this store — the writer, the durability daemon (`fsync`, every few seconds) and the compaction executor — so the resident state is not the writer's alone and a field mutated outside a monitor is one a reader can catch mid-pair. - The **kind lock** covers that kind's log, its idx, and the resident state derived from them: `live-ids`, `live-bytes`, the hot-record cache, `compacting` and `failed`. A store, a kill, a batch and the compactor's reconcile each take it once and do both halves inside it, so an id is never live to a reader while its slot says tombstone, and the compaction delta set is never cleared under a writer folding an id into it. It covers `live-ids` on the **read** side too, which the other three do not need: the live set is a bitmap mutated in place, so a tally or an enumeration taken beside a concurrent `addLong` reads a structure mid-edit. A read that hands the set onward takes a `hs/live-snapshot` inside the lock and lets go of it — the copy costs the bitmap's size, not the corpus's, which is what makes holding the writer's monitor for an enumeration affordable at all. - **`counters-lock`** covers the three that move together and belong to no kind: the handle `counter`, the `counters.nippy` blob, and `synced-seq` — read and written by `fsync` on the daemon's thread and by `clear-records!` on the writer's. Its own monitor rather than a kind's, because a whole-file blob rewrite held inside a kind lock would put a record append behind it every tick that minted a handle. `premises` is a `LiveHandleSet` too, and sits under the **sentexes** kind lock for reads and writes alike, because a premise is a sentex handle. It is not folded into `store!`'s acquisition: a premise joins the premise set after its record lands, in a second acquisition, so a failed write leaves no handle in `premise-ids` whose `get-sentex` is nil. The compactor's `drop-lost!` removes from it only when compacting the sentexes kind, under the lock it already holds.
(abort-compaction! {:keys [kinds]})Tell an auto-compaction of store to abandon its rewrite: set :aborted on every kind,
under the kind lock every other write of that atom takes. Idempotent. Called by
backend/close-dir!.
A rewrite still copying frames stops at its next check (abort-check-frames); one at the
reconcile discards its temps. A rewrite past its commit marker is not reachable: the
reconcile holds the kind lock until the install is done, so an abort lands before the read
or after the install, and the next open would finish it anyway (recover-compaction!). The
flag is written whether or not a rewrite is running, so an abort that lands between the
executor's registry check and the compaction's snapshot sticks, and the snapshot carries an
existing flag across. clear-records! keeps the when c form, since its store goes on
being used. It does not wait; durability/await-compaction-quiescent! does.
Tell an auto-compaction of `store` to abandon its rewrite: set `:aborted` on every kind, under the kind lock every other write of that atom takes. Idempotent. Called by `backend/close-dir!`. A rewrite still copying frames stops at its next check (`abort-check-frames`); one at the reconcile discards its temps. A rewrite past its commit marker is not reachable: the reconcile holds the kind lock until the install is done, so an abort lands before the read or after the install, and the next open would finish it anyway (`recover-compaction!`). The flag is written whether or not a rewrite is running, so an abort that lands between the executor's registry check and the compaction's snapshot sticks, and the snapshot carries an existing flag across. `clear-records!` keeps the `when c` form, since its store goes on being used. It does not wait; `durability/await-compaction-quiescent!` does.
(close! {:keys [dir kinds dict fault] :as store})Flush durably, record the log lengths this session closed at, close every RAF, then remove
the dirty marker (a clean shutdown). The lengths are read after the fsync and written before
anything closes, so the marker names what is durable and the next open skips a log's tail
walk while its length agrees (f/log-tail-offset-from).
Every handle is released whatever the flush did, and only the dirty-marker removal is
conditional. A store whose fault latch is already set closes without throwing: it writes no
clean marker, leaves the dirty one, releases every handle and logs at :warn. A flush that
fails here throws. Afterwards the latch is :closed or faulted, so a later call is refused
by name.
Flush durably, record the log lengths this session closed at, close every RAF, then remove the dirty marker (a clean shutdown). The lengths are read after the fsync and written before anything closes, so the marker names what is durable and the next open skips a log's tail walk while its length agrees ([`f/log-tail-offset-from`](files.clj)). Every handle is released whatever the flush did, and only the dirty-marker removal is conditional. A store whose fault latch is already set closes without throwing: it writes no clean marker, leaves the dirty one, releases every handle and logs at `:warn`. A flush that fails here throws. Afterwards the latch is `:closed` or faulted, so a later call is refused by name.
(compact! {:keys [kinds premises]})Compact every kind's log (reclaim the dead frames left by deletes and premise
re-stores). Preserves every live record and its handle — except a handle whose frame
the log cannot give back, which is dropped rather than re-stored empty
(compact-kind!).
Compact every kind's log (reclaim the dead frames left by deletes and premise re-stores). Preserves every live record and its handle — except a handle whose frame the log cannot give back, which is dropped rather than re-stored empty (`compact-kind!`).
(dead-ratio {:keys [kinds]})Max dead-byte ratio across the kinds — the durability daemon's compaction trigger.
O(kinds): each kind answers off its maintained live-bytes and its log's length.
Max dead-byte ratio across the kinds — the durability daemon's compaction trigger. O(kinds): each kind answers off its maintained `live-bytes` and its log's length.
(fsync {:keys [dir kinds counter synced-seq dict counters-lock epoch fault]})fsync every kind's log + idx, and rewrite the counters blob when the handle counter has moved since the last tick.
The token dictionary is fsynced first, under the sentexes kind lock, so every record durable
after the tick has durable tokens. The counters blob is a whole-file rewrite and is written
only when synced-seq differs from the counter; a skip costs no handle
(recover-next-id takes the max of the blob and one past the highest slot). This runs on
the durability daemon's thread: the counter, the blob and synced-seq move together under
counters-lock, which clear-records! also takes, and the counter is read inside it. A
failed fsync latches the fault (:fsync-failed, files/with-io-guard), because a retry may
succeed over pages the kernel dropped. A faulted store's tick is a no-op.
fsync every kind's log + idx, and rewrite the counters blob when the handle counter has moved since the last tick. The token dictionary is fsynced first, under the sentexes kind lock, so every record durable after the tick has durable tokens. The counters blob is a whole-file rewrite and is written only when `synced-seq` differs from the counter; a skip costs no handle (`recover-next-id` takes the max of the blob and one past the highest slot). This runs on the durability daemon's thread: the counter, the blob and `synced-seq` move together under `counters-lock`, which `clear-records!` also takes, and the counter is read inside it. A failed fsync latches the fault (`:fsync-failed`, `files/with-io-guard`), because a retry may succeed over pages the kernel dropped. A faulted store's tick is a no-op.
(open-record-store dir)(open-record-store dir
{:keys [cache-capacity tokenize?]
:or {cache-capacity (config/disk-cache-capacity)
tokenize? (config/disk-tokens?)}})Open a DiskRecordStore rooted at dir/records. Recovers each kind, rebuilds the
live-id sets, the premise set (sentexes with a non-nil :strength), and the id
counter, and drops a dirty marker (removed on a clean close!).
:cache-capacity sizes the per-kind hot-record LRU, defaulting to the
vaelii.disk.cache property; 0 runs with no cache (what the fetch benchmark measures
against, and the per-fetch cost with no cache hits). :tokenize? writes sentex bodies
as ids from a durable token dictionary rather than in full, defaulting to the
vaelii.disk.tokens property; it is a write choice only — frames written either way
always read.
Open a `DiskRecordStore` rooted at `dir/records`. Recovers each kind, rebuilds the live-id sets, the premise set (sentexes with a non-nil :strength), and the id counter, and drops a dirty marker (removed on a clean `close!`). `:cache-capacity` sizes the per-kind hot-record LRU, defaulting to the `vaelii.disk.cache` property; 0 runs with no cache (what the fetch benchmark measures against, and the per-fetch cost with no cache hits). `:tokenize?` writes sentex bodies as ids from a durable token dictionary rather than in full, defaulting to the `vaelii.disk.tokens` property; it is a *write* choice only — frames written either way always read.
(reasoning-fingerprint {:keys [kinds epoch]})The fingerprint a reasoning image is validated against, as
{:sentexes fp :justifications fp}, each a kind-fingerprint. Belief reads both kinds
— the sentexes, whose slots also move when a premise mark re-stores one, and the
justifications — so a change to either moves the fingerprint, where slot-fingerprint
alone misses a justification stored over sentexes that did not change.
The fingerprint a reasoning image is validated against, as
`{:sentexes fp :justifications fp}`, each a `kind-fingerprint`. Belief reads both kinds
— the sentexes, whose slots also move when a premise mark re-stores one, and the
justifications — so a change to either moves the fingerprint, where `slot-fingerprint`
alone misses a justification stored over sentexes that did not change.A thunk run once inside compact-kind!'s rewrite phase — after the
snapshot, before the first frame is copied — with the kind map as its argument. Nil
in ordinary operation, and nothing in the engine ever sets it.
It exists because the race window this store's lifecycle turns on is between the snapshot and the reconcile, and it is a race window a test cannot reach by sleeping: a rewrite of a test-sized log is over in microseconds, so "close the directory while a rewrite is open" reproduces by luck or not at all. A barrier here holds the rewrite at exactly that point, so the handshake can be asserted rather than hoped for.
A thunk run once inside `compact-kind!`'s rewrite phase — after the snapshot, before the first frame is copied — with the kind map as its argument. Nil in ordinary operation, and nothing in the engine ever sets it. It exists because the race window this store's lifecycle turns on is *between* the snapshot and the reconcile, and it is a race window a test cannot reach by sleeping: a rewrite of a test-sized log is over in microseconds, so "close the directory while a rewrite is open" reproduces by luck or not at all. A barrier here holds the rewrite at exactly that point, so the handshake can be asserted rather than hoped for.
(slot-fingerprint {:keys [kinds epoch]})What the sentexes idx currently says, as {:count :max-handle :digest}: the fingerprint a
derived-index snapshot is validated against (vaelii.impl.disk.index-snapshot). It is read
off the slots in one sequential pass over a 24-byte-per-record file and decodes nothing. It
detects a record added, deleted or re-stored, and a wipe by the :epoch clear-records!
mints. It consults usable!, since a half-copied idx would describe no version of the
records.
What the sentexes idx currently says, as `{:count :max-handle :digest}`: the fingerprint a
derived-index snapshot is validated against (`vaelii.impl.disk.index-snapshot`). It is read
off the slots in one sequential pass over a 24-byte-per-record file and decodes nothing. It
detects a record added, deleted or re-stored, and a wipe by the `:epoch` `clear-records!`
mints. It consults `usable!`, since a half-copied idx would describe no version of the
records.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |