The record store on disk — an implementation of RecordStore over per-kind
log/idx pairs (vaelii.impl.disk.files).
Three int-keyed kinds — sentexes, justifications, provenance — each a .log of
length-prefixed nippy frames plus a .idx of fixed 24-byte slots mapping handle →
log offset. A frame holds its record's fields positionally
(vaelii.impl.disk.codec), so the type tag and field names are not rewritten into
every one of them; a frame written before that codec still reads, as its own shape. A record is paged from disk on get: read the slot, read the frame
it points at, thaw it — two positional reads, no seek, and the records do not sit in
RAM. What does sit in RAM per kind is the small set of live handles (so enumeration
is O(1)), rebuilt from the idx on open, and a bounded LRU of hot records in front
of the read (vaelii.disk.cache, 0 to disable).
next-id is a monotonic counter recovered as max(the counters blob, 1 + the highest slot id across the record kinds) — the highest slot id is stable across
deletes (a tombstone keeps its slot) and across compaction (slot ids are preserved),
so a handle is never reused even if the counters blob is stale after a crash. Within
a session every write holds the same bound (clear-counter!), because a record can
arrive carrying its own :id and nothing re-reads the slots until the next open.
A premise is exactly a sentex whose :strength is non-nil (the strength lives on
the record, as on every backend), so the premise set is derived from the durable
records — rebuilt on open, maintained in lockstep by mark/unmark — rather than
stored separately. Rebuilding it does not mean reading them: every write records
the answer in its idx slot's flags, so the open reads the set off the slot walk it
already makes. A slot that does not carry the bit sends that one handle to its
record, and the record is authoritative wherever both speak (rebuild-premises!).
Recovery on open: finish any interrupted compaction, truncate a torn log tail, then
tombstone any slot whose frame now extends past the log (validate-idx-tail!).
Crash-safety rests on the write ordering (append the frame, then point the slot at it)
and on files' crash-safe compaction. The tail is located from the frame lengths
(files/log-tail-offset) and nothing is decoded to find it — and a clean close!
records each log's length, so an open whose log is still that long skips even the walk.
The marker is consumed here, so it never describes a store in use; every disagreement
falls back to the walk.
Every RAF touch holds the owning kind's lock. A write or force! must, because the
file pointer is shared (see files' shared-pointer invariant); a read no longer has
to — the read primitives are positional — but still does, because that is what
serializes it against a concurrent append and its slot write.
The record store on disk — an implementation of `RecordStore` over per-kind log/idx pairs (`vaelii.impl.disk.files`). Three int-keyed kinds — sentexes, justifications, provenance — each a `.log` of length-prefixed nippy frames plus a `.idx` of fixed 24-byte slots mapping handle → log offset. A frame holds its record's fields **positionally** (`vaelii.impl.disk.codec`), so the type tag and field names are not rewritten into every one of them; a frame written before that codec still reads, as its own shape. A record is **paged** from disk on `get`: read the slot, read the frame it points at, thaw it — two positional reads, no seek, and the records do not sit in RAM. What does sit in RAM per kind is the small set of live handles (so enumeration is O(1)), rebuilt from the idx on open, and a **bounded LRU of hot records** in front of the read (`vaelii.disk.cache`, 0 to disable). `next-id` is a monotonic counter recovered as `max(the counters blob, 1 + the highest slot id across the record kinds)` — the highest slot id is stable across deletes (a tombstone keeps its slot) and across compaction (slot ids are preserved), so a handle is never reused even if the counters blob is stale after a crash. Within a session every write holds the same bound (`clear-counter!`), because a record can arrive carrying its own `:id` and nothing re-reads the slots until the next open. A premise is exactly a sentex whose `:strength` is non-nil (the strength lives on the record, as on every backend), so the premise set is derived from the durable records — rebuilt on open, maintained in lockstep by mark/unmark — rather than stored separately. Rebuilding it does not mean *reading* them: every write records the answer in its idx slot's flags, so the open reads the set off the slot walk it already makes. A slot that does not carry the bit sends that one handle to its record, and the record is authoritative wherever both speak (`rebuild-premises!`). Recovery on open: finish any interrupted compaction, truncate a torn log tail, then tombstone any slot whose frame now extends past the log (`validate-idx-tail!`). Crash-safety rests on the write ordering (append the frame, then point the slot at it) and on `files`' crash-safe compaction. The tail is located from the frame *lengths* (`files/log-tail-offset`) and nothing is decoded to find it — and a clean `close!` records each log's length, so an open whose log is still that long skips even the walk. The marker is consumed here, so it never describes a store in use; every disagreement falls back to the walk. Every RAF touch holds the owning kind's lock. A write or `force!` must, because the file pointer is shared (see `files`' shared-pointer invariant); a read no longer has to — the read primitives are positional — but still does, because that is what serializes it against a concurrent append and its slot write.
(close! {:keys [dir kinds dict] :as store})Flush durably, record the log lengths this session closed at, close every RAF, then remove the dirty marker (a clean shutdown).
The lengths are read after the fsync and written before anything closes, so the
marker names exactly what is durable; the next open skips a log's tail walk while its
length still agrees (f/log-tail-offset-from).
Flush durably, record the log lengths this session closed at, close every RAF, then remove the dirty marker (a clean shutdown). The lengths are read **after** the fsync and written before anything closes, so the marker names exactly what is durable; the next open skips a log's tail walk while its length still agrees ([`f/log-tail-offset-from`](files.clj)).
(compact! {:keys [kinds]})Compact every kind's log (reclaim the dead frames left by deletes and premise re-stores). Preserves all live records and their handles.
Compact every kind's log (reclaim the dead frames left by deletes and premise re-stores). Preserves all live records and their handles.
(dead-ratio {:keys [kinds]})Max dead-byte ratio across the kinds — the durability daemon's compaction trigger.
Max dead-byte ratio across the kinds — the durability daemon's compaction trigger.
(fsync {:keys [dir kinds counter dict]} fsync?)fsync every kind's log + idx and persist the counter. fsync? false drains
writes to the page cache without the fsync (the durability daemon's non-durable tick).
The token dictionary is fsynced first, under the sentexes kind lock — that lock is what stops a record being appended between the two fsyncs, and so is what makes every record durable after this tick one whose tokens are durable too.
fsync every kind's log + idx and persist the counter. `fsync?` false drains writes to the page cache without the fsync (the durability daemon's non-durable tick). The token dictionary is fsynced **first, under the sentexes kind lock** — that lock is what stops a record being appended between the two fsyncs, and so is what makes every record durable after this tick one whose tokens are durable too.
(open-record-store dir)(open-record-store dir
{:keys [cache-capacity tokenize?]
:or {cache-capacity cache-capacity
tokenize? tokenized-default?}})Open a DiskRecordStore rooted at dir/records. Recovers each kind, rebuilds the
live-id sets, the premise set (sentexes with a non-nil :strength), and the id
counter, and drops a dirty marker (removed on a clean close!).
:cache-capacity sizes the per-kind hot-record LRU, defaulting to the
vaelii.disk.cache property; 0 runs with no cache (what the fetch benchmark measures
against, and the honest per-fetch cost). :tokenize? writes sentex bodies as ids from
a durable token dictionary rather than in full, defaulting to the vaelii.disk.tokens
property; it is a write choice only — frames written either way always read.
Open a `DiskRecordStore` rooted at `dir/records`. Recovers each kind, rebuilds the live-id sets, the premise set (sentexes with a non-nil :strength), and the id counter, and drops a dirty marker (removed on a clean `close!`). `:cache-capacity` sizes the per-kind hot-record LRU, defaulting to the `vaelii.disk.cache` property; 0 runs with no cache (what the fetch benchmark measures against, and the honest per-fetch cost). `:tokenize?` writes sentex bodies as ids from a durable token dictionary rather than in full, defaulting to the `vaelii.disk.tokens` property; it is a *write* choice only — frames written either way always read.
(slot-fingerprint {:keys [kinds]})What the sentexes idx currently says, as {:count :max-handle :digest} — the stamp a
derived-index snapshot is validated against (vaelii.impl.disk.index-snapshot).
It is read off the slots, so it costs one sequential pass over a 24-byte-per-record
file and decodes nothing. That is the whole point: an image exists so an open reads
bytes rather than records, and validating it against fingerprint/of-records would put
all of them back on the open path. What it detects is every way the record set can
change under a snapshot — a record added, deleted, or re-stored (a re-store appends a
new frame, so the handle's offset moves).
What the sentexes idx currently says, as `{:count :max-handle :digest}` — the stamp a
derived-index snapshot is validated against (`vaelii.impl.disk.index-snapshot`).
It is read off the **slots**, so it costs one sequential pass over a 24-byte-per-record
file and decodes nothing. That is the whole point: an image exists so an open reads
bytes rather than records, and validating it against `fingerprint/of-records` would put
all of them back on the open path. What it detects is every way the record set can
change under a snapshot — a record added, deleted, or re-stored (a re-store appends a
new frame, so the handle's offset moves).cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |