Low-level file primitives for the on-disk backend.
.log files hold length-prefixed nippy frames.
Frame layout: [len: i32 big-endian][nippy-bytes]. The offset returned from
append-record! points at the len prefix..idx files are fixed-width arrays of 24-byte slots, indexed by id.
Slot layout:
bytes 0..7 offset (i64, -1 = empty, -2 = tombstone)
bytes 8..15 length (i64)
bytes 16..19 flags (u32; bit 0 = the premise bit is meaningful, bit 1 = premise)
bytes 20..23 gen (u32, per-write increment)
Reads rely on the OS page cache; writes overwrite the slot in place.
A slot is read in one positional channel read, not a seek plus four primitive
readLong/readInt calls — a RandomAccessFile is unbuffered, so each of those is
its own syscall and moving 24 bytes cost six of them (measured: 52% of a warm
record fetch, lein bench-records)..nippy files hold whole-blob metadata (counters, the premise set). Callers
rewrite them atomically via write-nippy-atomic!.Shared-pointer invariant. A seek→read/write pair on a RandomAccessFile uses that
object's single shared file pointer, so any access to a store's live RAF must hold the
owning kind lock; the disk adapters take a per-kind lock around every such touch (write,
force!) for exactly this reason. The read primitives here (read-slot,
read-record, read-record-sized) are positional FileChannel reads instead — they
name the file offset in the call and never touch the pointer, so they neither disturb a
concurrent seek nor need one of their own.
Compression: frames freeze under the nippy compressor named by the
vaelii.disk.compress system property (lz4 | zstd | none); default none.
nippy reads the compressor id from each frame header, so mixed frames thaw
correctly. Durability: vaelii.disk.fsync=dsync opens logs rwd (O_DSYNC) so
every append is synchronous — off by default (the durability daemon fsyncs on a
tick instead).
Low-level file primitives for the on-disk backend.
- Append-only `.log` files hold length-prefixed nippy frames.
Frame layout: `[len: i32 big-endian][nippy-bytes]`. The offset returned from
`append-record!` points at the len prefix.
- `.idx` files are fixed-width arrays of 24-byte slots, indexed by id.
Slot layout:
bytes 0..7 offset (i64, -1 = empty, -2 = tombstone)
bytes 8..15 length (i64)
bytes 16..19 flags (u32; bit 0 = the premise bit is meaningful, bit 1 = premise)
bytes 20..23 gen (u32, per-write increment)
Reads rely on the OS page cache; writes overwrite the slot in place.
A slot is read in **one positional channel read**, not a seek plus four primitive
`readLong`/`readInt` calls — a `RandomAccessFile` is unbuffered, so each of those is
its own syscall and moving 24 bytes cost six of them (measured: 52% of a warm
record fetch, `lein bench-records`).
- `.nippy` files hold whole-blob metadata (counters, the premise set). Callers
rewrite them atomically via `write-nippy-atomic!`.
**Shared-pointer invariant.** A seek→read/write pair on a `RandomAccessFile` uses that
object's single shared file pointer, so any access to a store's live RAF must hold the
owning kind lock; the disk adapters take a per-kind lock around every such touch (write,
`force!`) for exactly this reason. The *read* primitives here (`read-slot`,
`read-record`, `read-record-sized`) are positional `FileChannel` reads instead — they
name the file offset in the call and never touch the pointer, so they neither disturb a
concurrent seek nor need one of their own.
Compression: frames freeze under the nippy compressor named by the
`vaelii.disk.compress` system property (`lz4` | `zstd` | `none`); default none.
nippy reads the compressor id from each frame header, so mixed frames thaw
correctly. Durability: `vaelii.disk.fsync=dsync` opens logs `rwd` (O_DSYNC) so
every append is synchronous — off by default (the durability daemon fsyncs on a
tick instead).(append-record! log-raf value)Append a nippy-serialized value to the log; return the byte offset of the [len]
prefix. The length prefix + payload are packed into one buffer and written with a
single .write (a per-record append is the hot path; four one-byte writeInt
syscalls would dominate). Caller must hold the log's write lock.
Append a nippy-serialized value to the log; return the byte offset of the `[len]` prefix. The length prefix + payload are packed into one buffer and written with a single `.write` (a per-record append is the hot path; four one-byte `writeInt` syscalls would dominate). Caller must hold the log's write lock.
(assert-format! root)Gate the store directory root on its format.edn sentinel: a known version is
ok, an unknown one throws, and a missing sentinel is stamped with the current
version (a pre-sentinel directory is by definition today's layout).
Gate the store directory `root` on its `format.edn` sentinel: a known version is ok, an unknown one throws, and a missing sentinel is stamped with the current version (a pre-sentinel directory is by definition today's layout).
(clean-marker-path root)(close! c)(compact-temp-paths log-path idx-path)Sibling temp + commit-marker paths for crash-safe compaction of a (log, idx) pair. The marker keys off the log path, so one marker governs the pair.
Sibling temp + commit-marker paths for crash-safe compaction of a (log, idx) pair. The marker keys off the log path, so one marker governs the pair.
(create-dirty-marker! root)Durably drop the dirty marker (fsync the file AND its dir).
Durably drop the dirty marker (fsync the file AND its dir).
(delete-compact-temps! marker log-tmp idx-tmp)Remove the commit marker then both temps (marker first, so a crash mid-cleanup can't leave a marker without its temps). Idempotent.
Remove the commit marker then both temps (marker first, so a crash mid-cleanup can't leave a marker without its temps). Idempotent.
(delete-log-compact-temps! marker tmp)Remove a single-log compaction's marker then temp (marker first). Idempotent.
Remove a single-log compaction's marker then temp (marker first). Idempotent.
(dirty-marker-path root)(dirty-marker-present? root)True when a previous session's dirty marker survives — that session never ran a clean close!, so the open path should run its crash checks.
True when a previous session's dirty marker survives — that session never ran a clean close!, so the open path should run its crash checks.
Slot offset meaning 'nothing stored at this id'.
Slot offset meaning 'nothing stored at this id'.
(ensure-dir! path)Create the directory at path (and any missing parents); return the File.
Create the directory at `path` (and any missing parents); return the `File`.
(force! raf meta-data?)fsync raf's channel; meta-data? includes file metadata (mtime, length).
fsync `raf`'s channel; `meta-data?` includes file metadata (mtime, length).
On-disk layout version for a backend directory. Bumped only on an incompatible layout change; a directory written before the sentinel is adopted as version 1.
On-disk layout version for a backend directory. Bumped only on an incompatible layout change; a directory written before the sentinel is adopted as version 1.
(log-compact-paths log-path)Temp + commit-marker paths for crash-safe compaction of a single .log.
Temp + commit-marker paths for crash-safe compaction of a single `.log`.
(log-length log-raf)(log-tail-offset log-raf)The truncation point after a torn trailing write, computed from the frame lengths
alone — read a prefix, skip n, repeat — without thawing anything.
This is scan-log's return value without decoding the log, and decoding is the whole
cost: finding one offset in an 11M-record log means thawing eleven million frames whose
values are then discarded, and it makes the open path depend on whether this build can
decode what is in the log. That dependency is not hypothetical — a record class rename
turned one unreadable store into eleven million exceptions, thrown from a scan whose
only question was how long is the log.
Nothing is thawed, deliberately, and the length chain is enough. A frame is
appended before the idx slot that points at it, so a torn tail frame is one nothing
references, and the idx is the authority on what is live. Truncating the tail is
therefore tidiness rather than repair, and validate-idx-tail! — which still runs —
is what actually reconciles a slot against a log that lost its end. A frame whose
payload is damaged but whose length is intact stays, to fail (if anything references
it at all) at that one record, which is the same one-record blast radius the token
dictionary's own damage check settles for. The alternative is worse in exactly the
case that bit us: thawing to decide truncation means a build that cannot decode
deletes the log it cannot read.
A non-positive length terminates the walk rather than being stepped over. A frame payload is a nippy value and is never empty, so a zero can only be space that was never written — and on a filesystem that zero-fills past a tear, stepping over zeros four bytes at a time would walk to EOF and pronounce the whole torn tail intact.
The prefixes are read through a window rather than one seek+readInt apiece: a
RandomAccessFile is unbuffered, so per-frame reads make this syscall-bound (measured
at 5.8M frames: 5.4s that way, 23ms this way). The window is filled by positional
channel reads, which name the offset and never touch the shared file pointer — so this
neither disturbs a concurrent append nor needs the kind lock.
The truncation point after a torn trailing write, computed from the frame **lengths** alone — read a prefix, skip `n`, repeat — without thawing anything. This is `scan-log`'s return value without decoding the log, and decoding is the whole cost: finding one offset in an 11M-record log means thawing eleven million frames whose values are then discarded, and it makes the open path depend on whether this build can decode what is in the log. That dependency is not hypothetical — a record class rename turned one unreadable store into eleven million exceptions, thrown from a scan whose only question was *how long is the log*. **Nothing is thawed, deliberately, and the length chain is enough.** A frame is appended before the idx slot that points at it, so a torn tail frame is one nothing references, and the idx is the authority on what is live. Truncating the tail is therefore tidiness rather than repair, and `validate-idx-tail!` — which still runs — is what actually reconciles a slot against a log that lost its end. A frame whose payload is damaged but whose length is intact stays, to fail (if anything references it at all) at that one record, which is the same one-record blast radius the token dictionary's own damage check settles for. The alternative is worse in exactly the case that bit us: thawing to decide truncation means a build that cannot decode *deletes the log it cannot read*. A **non-positive** length terminates the walk rather than being stepped over. A frame payload is a nippy value and is never empty, so a zero can only be space that was never written — and on a filesystem that zero-fills past a tear, stepping over zeros four bytes at a time would walk to EOF and pronounce the whole torn tail intact. The prefixes are read through a **window** rather than one `seek`+`readInt` apiece: a `RandomAccessFile` is unbuffered, so per-frame reads make this syscall-bound (measured at 5.8M frames: 5.4s that way, 23ms this way). The window is filled by *positional* channel reads, which name the offset and never touch the shared file pointer — so this neither disturbs a concurrent append nor needs the kind lock.
(log-tail-offset-from log-raf clean-length)The truncation offset for log-raf, skipping the walk entirely when clean-length
(this log's entry in the clean marker) equals its current length. Returns
[offset scanned?] so a caller can report whether the fast path was taken.
The truncation offset for `log-raf`, skipping the walk entirely when `clean-length` (this log's entry in the clean marker) equals its current length. Returns `[offset scanned?]` so a caller can report whether the fast path was taken.
(max-slot-id idx-raf)The highest slot id the idx file can address, or -1 if empty. Stable across
deletes (a tombstone keeps its slot), so 1 + max never reuses a handle.
The highest slot id the idx file can address, or -1 if empty. Stable across deletes (a tombstone keeps its slot), so `1 + max` never reuses a handle.
(open-idx path)Open an idx file for random read/write. Always rw — idx slots are
reconstructible from the logs, so they need no per-write durability.
Open an idx file for random read/write. Always `rw` — idx slots are reconstructible from the logs, so they need no per-write durability.
(open-log path)Open a log file for append + read, positioned at EOF. Under
vaelii.disk.fsync=dsync the channel is rwd so every append is durable.
Open a log file for append + read, positioned at EOF. Under `vaelii.disk.fsync=dsync` the channel is `rwd` so every append is durable.
(open-log-read path)Open a private read-only RAF on an existing log. The copy-on-write compactor reads the log's immutable (already-written) region through its own handle, so it needn't hold the kind lock while it does the O(live) rewrite — a fresh RAF has its own file pointer, so this does not touch the shared-pointer invariant of the store's live RAF.
Open a private read-only RAF on an existing log. The copy-on-write compactor reads the log's immutable (already-written) region through its own handle, so it needn't hold the kind lock while it does the O(live) rewrite — a fresh RAF has its own file pointer, so this does not touch the shared-pointer invariant of the store's live RAF.
(premise-flags premise?)The flags word for a slot whose record is, or is not, a premise.
The `flags` word for a slot whose record is, or is not, a premise.
(read-clean-marker root)The name -> length map a clean close left, or nil (absent or unreadable).
The `name -> length` map a clean close left, or nil (absent or unreadable).
(read-nippy-file path)(read-nippy-file path default)Read a whole-file nippy value, returning default when the file is missing,
empty, or unreadable (a rare torn blob thaws to default + a warning rather than
throwing — these blobs hold reconstructible metadata).
Read a whole-file nippy value, returning `default` when the file is missing, empty, or unreadable (a rare torn blob thaws to default + a warning rather than throwing — these blobs hold reconstructible metadata).
(read-record log-raf offset)Read a nippy value from the log at offset; nil for a negative or past-EOF offset.
For a caller that holds the slot, read-record-sized is one read instead of two.
Read a nippy value from the log at `offset`; nil for a negative or past-EOF offset. For a caller that holds the slot, `read-record-sized` is one read instead of two.
(read-record-sized log-raf offset length)Thaw the frame at offset whose payload is length bytes — one positional read,
of the payload alone. The slot already recorded the payload length, so a caller
holding the slot needs neither a seek nor a re-read of the length prefix.
Thaw the frame at `offset` whose payload is `length` bytes — **one** positional read, of the payload alone. The slot already recorded the payload length, so a caller holding the slot needs neither a seek nor a re-read of the length prefix.
(read-slot idx-raf id)Read a slot by id: {:offset :length :flags :gen :tombstone?}, or nil when the
slot is past EOF (unwritten), empty (offset=-1), or a zero-filled gap
(offset=0 AND length=0, left by .setLength growing the idx past its high-water
mark — no real record has length 0, so this is unambiguous).
One positional read of the 24 bytes, decoded in memory: a short read is the
past-EOF test, so this costs neither a .length call to make that test nor a seek
and four primitive reads to satisfy it.
Read a slot by id: `{:offset :length :flags :gen :tombstone?}`, or nil when the
slot is past EOF (unwritten), empty (offset=-1), or a zero-filled gap
(offset=0 AND length=0, left by `.setLength` growing the idx past its high-water
mark — no real record has length 0, so this is unambiguous).
One positional read of the 24 bytes, decoded in memory: a short read *is* the
past-EOF test, so this costs neither a `.length` call to make that test nor a seek
and four primitive reads to satisfy it.(recover-compaction! log-path idx-path)Finish or discard a crash-interrupted compaction of the (log, idx) pair, BEFORE they are opened for normal use. Returns :replayed, :discarded-incomplete, or :none. Idempotent.
Finish or discard a crash-interrupted compaction of the (log, idx) pair, BEFORE they are opened for normal use. Returns :replayed, :discarded-incomplete, or :none. Idempotent.
(recover-log-compaction! log-path)Finish or discard a crash-interrupted single-log compaction, BEFORE the log opens. Returns :replayed, :discarded-incomplete, or :none. Idempotent.
Finish or discard a crash-interrupted single-log compaction, BEFORE the log opens. Returns :replayed, :discarded-incomplete, or :none. Idempotent.
(remove-clean-marker! root)Drop the clean marker — called on open, so it never describes a store in use.
Drop the clean marker — called on open, so it never describes a store in use.
(remove-dirty-marker! root)Remove the dirty marker after a clean fsync+close. Idempotent.
Remove the dirty marker after a clean fsync+close. Idempotent.
(replay-temp-onto-raf! target tmp-path)Install the freshly-built tmp-path over an already-open RAF: truncate, copy the
temp's bytes in, fsync, leave positioned at EOF. Used by the live compactor.
Install the freshly-built `tmp-path` over an already-open RAF: truncate, copy the temp's bytes in, fsync, leave positioned at EOF. Used by the live compactor.
(scan-idx! idx-raf on-slot)Walk the idx from the start, invoking (on-slot id offset length flags) for every
live slot (not empty, tombstone, or zero-filled gap). Reads in 96 KiB chunks (a
per-id read-slot on a large idx pays one seek each); re-seeks per chunk so an
on-slot that itself seeks idx-raf cannot derail the scan.
The flags ride along because this walk is the only one an open makes: whatever a slot can answer without reading its record has to be answered here or not at all.
Walk the idx from the start, invoking `(on-slot id offset length flags)` for every live slot (not empty, tombstone, or zero-filled gap). Reads in 96 KiB chunks (a per-id `read-slot` on a large idx pays one seek each); re-seeks per chunk so an `on-slot` that itself seeks `idx-raf` cannot derail the scan. The flags ride along because this walk is the only one an open makes: whatever a slot can answer without reading its record has to be answered here or not at all.
(scan-log log-raf f)Scan an append-only log, calling (f index value) for each valid frame. Returns
the byte offset of the first unreadable tail byte — the truncation point after a
torn trailing write.
A caller that wants only that offset wants log-tail-offset, which reads the length
prefixes and skips the decoding entirely.
Scan an append-only log, calling `(f index value)` for each valid frame. Returns the byte offset of the first unreadable tail byte — the truncation point after a torn trailing write. A caller that wants only that offset wants `log-tail-offset`, which reads the length prefixes and skips the decoding entirely.
Fixed slot width in .idx files: 8 (offset) + 8 (length) + 4 (flags) + 4 (gen) = 24.
Fixed slot width in `.idx` files: 8 (offset) + 8 (length) + 4 (flags) + 4 (gen) = 24.
(slot-count idx-raf)The number of fixed-width slots the idx file currently holds.
The number of fixed-width slots the idx file currently holds.
(slot-premise flags)What flags says about its record: true, false, or nil for a slot that does
not say — which is what sends a caller to the record itself.
What `flags` says about its record: `true`, `false`, or **nil** for a slot that does not say — which is what sends a caller to the record itself.
(token-log-present? root)True when root already holds a (non-empty) token dictionary — so a store must open
it to decode its frames, whether or not it still writes tokenized ones.
True when `root` already holds a (non-empty) token dictionary — so a store must open it to decode its frames, whether or not it still writes tokenized ones.
Slot offset meaning 'this id was deleted'.
Slot offset meaning 'this id was deleted'.
(tombstone-slot! idx-raf id)Mark slot id as a tombstone (offset=-2, other fields zeroed).
Mark slot `id` as a tombstone (offset=-2, other fields zeroed).
(truncate! raf)Truncate a log or idx RAF to empty and reposition at the start — the whole-file
wipe clear-*! needs.
Truncate a log or idx RAF to empty and reposition at the start — the whole-file wipe `clear-*!` needs.
(truncate-log! log-raf new-len)Truncate the log to new-len — used after scan-log detects a torn tail frame.
Truncate the log to `new-len` — used after `scan-log` detects a torn tail frame.
(validate-idx-tail! idx-raf log-raf window)Tombstone any of the last window idx slots whose frame (offset + 4 + length)
extends past the log's current length — a torn trailing write where the idx page
was persisted but the log pages were not. Returns the count repaired.
Tombstone any of the last `window` idx slots whose frame (offset + 4 + length) extends past the log's current length — a torn trailing write where the idx page was persisted but the log pages were not. Returns the count repaired.
(write-clean-marker! root lengths)Durably record lengths (a name -> log length map) as this session's clean close.
Call after the final fsync and before closing, so the lengths are the durable ones.
Durably record `lengths` (a `name -> log length` map) as this session's clean close. Call after the final fsync and before closing, so the lengths are the durable ones.
(write-commit-marker! marker info)Atomically create the compaction commit marker and fsync it (and its dir). Its existence is the commit point.
Atomically create the compaction commit marker and fsync it (and its dir). Its existence is the commit point.
(write-nippy-atomic! path value)Write a value to path by writing a unique temp file, fsyncing it, atomic-renaming
over path, then fsyncing the parent dir — durable, not merely atomic (a power loss
can otherwise leave the rename visible while the data blocks are still zeros). The
per-call unique temp suffix keeps a racing fsync tick from consuming another
writer's temp.
Write a value to `path` by writing a unique temp file, fsyncing it, atomic-renaming over `path`, then fsyncing the parent dir — durable, not merely atomic (a power loss can otherwise leave the rename visible while the data blocks are still zeros). The per-call unique temp suffix keeps a racing fsync tick from consuming another writer's temp.
(write-slot! idx-raf id offset length flags gen)Write a slot for id, growing the idx file if needed (the gap is zero-filled and
read back as unwritten).
Write a slot for `id`, growing the idx file if needed (the gap is zero-filled and read back as unwritten).
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |