The record store on disk — an implementation of RecordStore over per-kind
log/idx pairs (vaelii.impl.disk.files).
Three int-keyed kinds — sentexes, justifications, provenance — each a .log of
length-prefixed nippy frames plus a .idx of fixed 24-byte slots mapping handle →
log offset. A frame holds its record's fields positionally
(vaelii.impl.disk.codec), so the type tag and field names are not rewritten into
every one of them; a frame written before that codec still reads, as its own shape. A record is paged from disk on get: read the slot, read the frame
it points at, thaw it — two positional reads, no seek, and the records do not sit in
RAM. What does sit in RAM per kind is the small set of live handles (so enumeration
is O(1)), rebuilt from the idx on open, and a bounded LRU of hot records in front
of the read (vaelii.disk.cache, 0 to disable).
next-id is a monotonic counter recovered as max(the counters blob, 1 + the highest slot id across the record kinds) — the highest slot id is stable across
deletes (a tombstone keeps its slot) and across compaction (slot ids are preserved),
so a handle is never reused even if the counters blob is stale after a crash. Within
a session every write holds the same bound (clear-counter!), because a record can
arrive carrying its own :id and nothing re-reads the slots until the next open.
A premise is exactly a sentex whose :strength is non-nil (the strength lives on
the record, as on every backend), so the premise set is derived from the durable
records — rebuilt on open, maintained in lockstep by mark/unmark — rather than
stored separately. Rebuilding it does not mean reading them: every write records
the answer in its idx slot's flags, so the open reads the set off the slot walk it
already makes. A slot that does not carry the bit sends that one handle to its
record, and the record is authoritative wherever both speak (rebuild-premises!).
The premise's strength rank rides the same slot flags (bits 2..3), so
premise-strength — read once per premise on every recover — answers off the 24-byte
slot instead of paging the whole record for one keyword. A slot carrying no rank (a
non-premise, or one older than the bits) falls back to the record, the same
no-format-bump story as the premise bit itself (f/slot-strength).
Recovery on open: finish any interrupted compaction, truncate a torn log tail, then
tombstone any slot whose frame now extends past the log (validate-idx-tail!).
Crash-safety rests on the write ordering (append the frame, then point the slot at it)
and on files' crash-safe compaction. Where it stops is the slot itself: 24 bytes do
not divide a page, so a crash can leave one spliced from two writes, and a splice still
pointing inside the log reads as a thaw failure on that handle rather than being caught
here (f/validate-idx-tail! says what it does and does not cover).
The tail is located from the frame lengths
(files/log-tail-offset) and nothing is decoded to find it — and a clean close!
records each log's length, so an open whose log is still that long skips even the walk.
The marker is consumed here, so it never describes a store in use; every disagreement
falls back to the walk.
Every RAF touch holds the owning kind's lock. A write or force! must, because the
file pointer is shared (see files' shared-pointer invariant); a read is positional
and need not, but still does, because that is what serializes it against a concurrent
append and its slot write.
Two monitors, and which resident field sits under which. Three threads touch this
store — the writer, the durability daemon (fsync, every few seconds) and the
compaction executor — so the resident state is not the writer's alone and a field
mutated outside a monitor is one a reader can catch mid-pair.
live-ids, the hot-record cache, compacting and failed. A store, a
kill, a batch and the compactor's reconcile each take it once and do both halves
inside it, so an id is never live to a reader while its slot says tombstone, and the
compaction delta set is never cleared under a writer folding an id into it.counters-lock covers the three that move together and belong to no kind: the
handle counter, the counters.nippy blob, and synced-seq — read and written by
fsync on the daemon's thread and by clear-records! on the writer's. Its own
monitor rather than a kind's, because a whole-file blob rewrite held inside a kind
lock would put a record append behind it every tick that minted a handle.premises needs neither on the write path: every mutation is one swap! on one atom,
and the pair that matters — a handle in premise-ids whose record is gone — is a
kill! the writer makes, on the thread that would read it back. The one mutation from
another thread is the compactor's drop-lost!, which takes the kind lock beside the
live-ids drop it belongs with.
The record store on disk — an implementation of `RecordStore` over per-kind log/idx pairs (`vaelii.impl.disk.files`). Three int-keyed kinds — sentexes, justifications, provenance — each a `.log` of length-prefixed nippy frames plus a `.idx` of fixed 24-byte slots mapping handle → log offset. A frame holds its record's fields **positionally** (`vaelii.impl.disk.codec`), so the type tag and field names are not rewritten into every one of them; a frame written before that codec still reads, as its own shape. A record is **paged** from disk on `get`: read the slot, read the frame it points at, thaw it — two positional reads, no seek, and the records do not sit in RAM. What does sit in RAM per kind is the small set of live handles (so enumeration is O(1)), rebuilt from the idx on open, and a **bounded LRU of hot records** in front of the read (`vaelii.disk.cache`, 0 to disable). `next-id` is a monotonic counter recovered as `max(the counters blob, 1 + the highest slot id across the record kinds)` — the highest slot id is stable across deletes (a tombstone keeps its slot) and across compaction (slot ids are preserved), so a handle is never reused even if the counters blob is stale after a crash. Within a session every write holds the same bound (`clear-counter!`), because a record can arrive carrying its own `:id` and nothing re-reads the slots until the next open. A premise is exactly a sentex whose `:strength` is non-nil (the strength lives on the record, as on every backend), so the premise set is derived from the durable records — rebuilt on open, maintained in lockstep by mark/unmark — rather than stored separately. Rebuilding it does not mean *reading* them: every write records the answer in its idx slot's flags, so the open reads the set off the slot walk it already makes. A slot that does not carry the bit sends that one handle to its record, and the record is authoritative wherever both speak (`rebuild-premises!`). The premise's **strength rank** rides the same slot flags (bits 2..3), so `premise-strength` — read once per premise on every `recover` — answers off the 24-byte slot instead of paging the whole record for one keyword. A slot carrying no rank (a non-premise, or one older than the bits) falls back to the record, the same no-format-bump story as the premise bit itself (`f/slot-strength`). Recovery on open: finish any interrupted compaction, truncate a torn log tail, then tombstone any slot whose frame now extends past the log (`validate-idx-tail!`). Crash-safety rests on the write ordering (append the frame, then point the slot at it) and on `files`' crash-safe compaction. Where it stops is the slot itself: 24 bytes do not divide a page, so a crash can leave one spliced from two writes, and a splice still pointing inside the log reads as a thaw failure on that handle rather than being caught here (`f/validate-idx-tail!` says what it does and does not cover). The tail is located from the frame *lengths* (`files/log-tail-offset`) and nothing is decoded to find it — and a clean `close!` records each log's length, so an open whose log is still that long skips even the walk. The marker is consumed here, so it never describes a store in use; every disagreement falls back to the walk. Every RAF touch holds the owning kind's lock. A write or `force!` must, because the file pointer is shared (see `files`' shared-pointer invariant); a read is positional and need not, but still does, because that is what serializes it against a concurrent append and its slot write. **Two monitors, and which resident field sits under which.** Three threads touch this store — the writer, the durability daemon (`fsync`, every few seconds) and the compaction executor — so the resident state is not the writer's alone and a field mutated outside a monitor is one a reader can catch mid-pair. - The **kind lock** covers that kind's log, its idx, and the resident state derived from them: `live-ids`, the hot-record cache, `compacting` and `failed`. A store, a kill, a batch and the compactor's reconcile each take it once and do both halves inside it, so an id is never live to a reader while its slot says tombstone, and the compaction delta set is never cleared under a writer folding an id into it. - **`counters-lock`** covers the three that move together and belong to no kind: the handle `counter`, the `counters.nippy` blob, and `synced-seq` — read and written by `fsync` on the daemon's thread and by `clear-records!` on the writer's. Its own monitor rather than a kind's, because a whole-file blob rewrite held inside a kind lock would put a record append behind it every tick that minted a handle. `premises` needs neither on the write path: every mutation is one `swap!` on one atom, and the pair that matters — a handle in `premise-ids` whose record is gone — is a `kill!` the writer makes, on the thread that would read it back. The one mutation from another thread is the compactor's `drop-lost!`, which takes the kind lock beside the `live-ids` drop it belongs with.
(close! {:keys [dir kinds dict] :as store})Flush durably, record the log lengths this session closed at, close every RAF, then remove the dirty marker (a clean shutdown).
The lengths are read after the fsync and written before anything closes, so the
marker names exactly what is durable; the next open skips a log's tail walk while its
length still agrees (f/log-tail-offset-from).
Every handle is released whatever the flush did. The flush can fail — a full disk
— and the caller above releases the directory's lock on a failed close as deliberately
as on a clean one, so a throw that skipped the closes would hand the directory over
with every RandomAccessFile still held. The dirty marker is the one step that stays
conditional: it says the store closed cleanly, and an unclean close is what it exists
to record.
Flush durably, record the log lengths this session closed at, close every RAF, then remove the dirty marker (a clean shutdown). The lengths are read **after** the fsync and written before anything closes, so the marker names exactly what is durable; the next open skips a log's tail walk while its length still agrees ([`f/log-tail-offset-from`](files.clj)). **Every handle is released whatever the flush did.** The flush can fail — a full disk — and the caller above releases the directory's lock on a failed close as deliberately as on a clean one, so a throw that skipped the closes would hand the directory over with every `RandomAccessFile` still held. The dirty marker is the one step that stays conditional: it says the store closed cleanly, and an unclean close is what it exists to record.
(compact! {:keys [kinds premises]})Compact every kind's log (reclaim the dead frames left by deletes and premise
re-stores). Preserves every live record and its handle — except a handle whose frame
the log cannot give back, which is dropped rather than re-stored empty
(compact-kind!).
Compact every kind's log (reclaim the dead frames left by deletes and premise re-stores). Preserves every live record and its handle — except a handle whose frame the log cannot give back, which is dropped rather than re-stored empty (`compact-kind!`).
(dead-ratio {:keys [kinds]})Max dead-byte ratio across the kinds — the durability daemon's compaction trigger.
Max dead-byte ratio across the kinds — the durability daemon's compaction trigger.
(fsync {:keys [dir kinds counter synced-seq dict counters-lock]})fsync every kind's log + idx, and rewrite the counters blob when the handle counter has moved since the last tick.
The token dictionary is fsynced first, under the sentexes kind lock — that lock is what stops a record being appended between the two fsyncs, and so is what makes every record durable after this tick one whose tokens are durable too.
The counters blob is written only when it changed. Persisting it is a whole-file
rewrite — a temp, an fsync of it, an ATOMIC_MOVE and an fsync of the directory — and
the durability daemon ticks every three seconds for the life of the process, so writing
it unconditionally charges a KB nobody is writing to those four operations a tick
forever. synced-seq holds what the file was last left holding; equal to the counter
means the file already says what there is to say. A skip can never cost a handle:
recover-next-id takes the max of the blob and one past the highest slot in the idx, so
a blob behind the counter is behind only on handles that were minted and never stored.
This runs on the durability daemon's thread, and the counter it reads is bumped by
the writer's. A counter that moves between the read and the blob write costs nothing —
a blob one handle behind is what the paragraph above is about — but a clear-records!
landing there costs the wipe: the blob would be stamped with the pre-wipe high-water
mark and synced-seq would agree with it. So the three that move together move under
counters-lock, which the wipe takes for the same three; the counter is read inside
it rather than before it, which is what makes the read part of the same step.
fsync every kind's log + idx, and rewrite the counters blob when the handle counter has moved since the last tick. The token dictionary is fsynced **first, under the sentexes kind lock** — that lock is what stops a record being appended between the two fsyncs, and so is what makes every record durable after this tick one whose tokens are durable too. **The counters blob is written only when it changed.** Persisting it is a whole-file rewrite — a temp, an fsync of it, an `ATOMIC_MOVE` and an fsync of the directory — and the durability daemon ticks every three seconds for the life of the process, so writing it unconditionally charges a KB nobody is writing to those four operations a tick forever. `synced-seq` holds what the file was last left holding; equal to the counter means the file already says what there is to say. A skip can never cost a handle: `recover-next-id` takes the max of the blob and one past the highest slot in the idx, so a blob behind the counter is behind only on handles that were minted and never stored. **This runs on the durability daemon's thread**, and the counter it reads is bumped by the writer's. A counter that moves between the read and the blob write costs nothing — a blob one handle behind is what the paragraph above is about — but a `clear-records!` landing there costs the wipe: the blob would be stamped with the pre-wipe high-water mark and `synced-seq` would agree with it. So the three that move together move under `counters-lock`, which the wipe takes for the same three; the counter is read inside it rather than before it, which is what makes the read part of the same step.
(open-record-store dir)(open-record-store dir
{:keys [cache-capacity tokenize?]
:or {cache-capacity (config/disk-cache-capacity)
tokenize? (config/disk-tokens?)}})Open a DiskRecordStore rooted at dir/records. Recovers each kind, rebuilds the
live-id sets, the premise set (sentexes with a non-nil :strength), and the id
counter, and drops a dirty marker (removed on a clean close!).
:cache-capacity sizes the per-kind hot-record LRU, defaulting to the
vaelii.disk.cache property; 0 runs with no cache (what the fetch benchmark measures
against, and the honest per-fetch cost). :tokenize? writes sentex bodies as ids from
a durable token dictionary rather than in full, defaulting to the vaelii.disk.tokens
property; it is a write choice only — frames written either way always read.
Open a `DiskRecordStore` rooted at `dir/records`. Recovers each kind, rebuilds the live-id sets, the premise set (sentexes with a non-nil :strength), and the id counter, and drops a dirty marker (removed on a clean `close!`). `:cache-capacity` sizes the per-kind hot-record LRU, defaulting to the `vaelii.disk.cache` property; 0 runs with no cache (what the fetch benchmark measures against, and the honest per-fetch cost). `:tokenize?` writes sentex bodies as ids from a durable token dictionary rather than in full, defaulting to the `vaelii.disk.tokens` property; it is a *write* choice only — frames written either way always read.
(slot-fingerprint {:keys [kinds]})What the sentexes idx currently says, as {:count :max-handle :digest} — the stamp a
derived-index snapshot is validated against (vaelii.impl.disk.index-snapshot).
It is read off the slots, so it costs one sequential pass over a 24-byte-per-record
file and decodes nothing. That is the whole point: an image exists so an open reads
bytes rather than records, and validating it against a content digest (walking every
record through fingerprint/accumulator) would put
all of them back on the open path. What it detects is every way the record set can
change under a snapshot — a record added, deleted, or re-stored (a re-store appends a
new frame, so the handle's offset moves).
It consults usable!: this is a claim about the record set, and a half-copied idx
would answer with a fingerprint describing no version of the records that ever existed
— which a derived image would then be stamped with, or validated against.
What the sentexes idx currently says, as `{:count :max-handle :digest}` — the stamp a
derived-index snapshot is validated against (`vaelii.impl.disk.index-snapshot`).
It is read off the **slots**, so it costs one sequential pass over a 24-byte-per-record
file and decodes nothing. That is the whole point: an image exists so an open reads
bytes rather than records, and validating it against a content digest (walking every
record through `fingerprint/accumulator`) would put
all of them back on the open path. What it detects is every way the record set can
change under a snapshot — a record added, deleted, or re-stored (a re-store appends a
new frame, so the handle's offset moves).
It consults `usable!`: this is a *claim about the record set*, and a half-copied idx
would answer with a fingerprint describing no version of the records that ever existed
— which a derived image would then be stamped with, or validated against.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |