Low-level file primitives for the on-disk backend (docs/storage.md).
.log files hold length-prefixed nippy frames, [len: i32 big-endian] [nippy-bytes]. The offset append-record! returns points at the len prefix..idx files are fixed-width arrays of 24-byte slots indexed by id:
bytes 0..7 offset (i64, -1 = empty, -2 = tombstone)
bytes 8..15 length (i64)
bytes 16..19 flags (u32; bit 0 = the premise bit is meaningful, bit 1 = premise, bits
2..3 = a premise's strength rank, 0 when unrecorded)
bytes 20..23 gen (u32, reserved, written as 0)
A slot is read in one positional channel read; writes overwrite the slot in place..nippy files hold whole-blob metadata (counters, the premise set), rewritten
atomically by write-nippy-atomic!.A seek then read or write on a RandomAccessFile uses its single shared file pointer, so
any access to a store's live RAF holds the owning kind lock. The read primitives
(read-slot, read-record, read-record-sized) are positional FileChannel reads, which
never touch the pointer.
Frames freeze under the nippy compressor named by vaelii.disk.compress (lz4 | zstd |
none, default none); nippy reads the compressor id from each frame header.
vaelii.disk.fsync=dsync opens logs rwd so every append is synchronous; the default
leaves fsync to the durability daemon.
Low-level file primitives for the on-disk backend (docs/storage.md).
- Append-only `.log` files hold length-prefixed nippy frames, `[len: i32 big-endian]
[nippy-bytes]`. The offset `append-record!` returns points at the len prefix.
- `.idx` files are fixed-width arrays of 24-byte slots indexed by id:
bytes 0..7 offset (i64, -1 = empty, -2 = tombstone)
bytes 8..15 length (i64)
bytes 16..19 flags (u32; bit 0 = the premise bit is meaningful, bit 1 = premise, bits
2..3 = a premise's strength rank, 0 when unrecorded)
bytes 20..23 gen (u32, reserved, written as 0)
A slot is read in one positional channel read; writes overwrite the slot in place.
- `.nippy` files hold whole-blob metadata (counters, the premise set), rewritten
atomically by `write-nippy-atomic!`.
A seek then read or write on a `RandomAccessFile` uses its single shared file pointer, so
any access to a store's live RAF holds the owning kind lock. The read primitives
(`read-slot`, `read-record`, `read-record-sized`) are positional `FileChannel` reads, which
never touch the pointer.
Frames freeze under the nippy compressor named by `vaelii.disk.compress` (`lz4` | `zstd` |
`none`, default none); nippy reads the compressor id from each frame header.
`vaelii.disk.fsync=dsync` opens logs `rwd` so every append is synchronous; the default
leaves fsync to the durability daemon.(append-record! log-raf value)Append a nippy-serialized value to the log; return the byte offset of the [len]
prefix. append-record-sized! for a caller that also wants the payload length.
Append a nippy-serialized value to the log; return the byte offset of the `[len]` prefix. `append-record-sized!` for a caller that also wants the payload length.
(append-record-sized! log-raf value)Append a nippy-serialized value to the log as one frame; return [offset payload-length] — the byte offset of the [len] prefix, and how many payload bytes
follow it, which is what an idx slot records (write-slot!), so the caller needs no
second length read to learn it. All-or-nothing (append-bytes!). Caller must hold
the log's write lock.
Append a nippy-serialized value to the log as one frame; return `[offset payload-length]` — the byte offset of the `[len]` prefix, and how many payload bytes follow it, which is what an idx slot records (`write-slot!`), so the caller needs no second length read to learn it. All-or-nothing (`append-bytes!`). Caller must hold the log's write lock.
(append-records! log-raf values)Append every value in values to the log, each its own frame, as one write:
the frames are packed into a single buffer and land together or not at all
(append-bytes!), so a batch of ops is never half on disk. Returns the offset of
the first frame, or the log length when values is empty. Caller must hold the
log's write lock.
Append every value in `values` to the log, each its own frame, **as one write**: the frames are packed into a single buffer and land together or not at all (`append-bytes!`), so a batch of ops is never half on disk. Returns the offset of the first frame, or the log length when `values` is empty. Caller must hold the log's write lock.
(append-records-sized! log-raf values)Append every value in values to the log, each its own frame, as one write
(append-record-sized!'s batch form). Returns a vector of [offset payload-length] in
input order, the pair an idx slot records. All or nothing for the batch
(append-bytes!). Caller must hold the log's write lock.
Append every value in `values` to the log, each its own frame, as one write (`append-record-sized!`'s batch form). Returns a vector of `[offset payload-length]` in input order, the pair an idx slot records. All or nothing for the batch (`append-bytes!`). Caller must hold the log's write lock.
(assert-format! root)Check the store directory root against its format.edn sentinel: a known version is
ok, an unknown one throws, and a missing sentinel is written with the current version. A
damaged sentinel is refused (:unreadable-store), not rewritten, and one past
manifest-bytes is refused as :manifest-too-large.
Check the store directory `root` against its `format.edn` sentinel: a known version is ok, an unknown one throws, and a missing sentinel is written with the current version. A damaged sentinel is refused (`:unreadable-store`), not rewritten, and one past `manifest-bytes` is refused as `:manifest-too-large`.
(check-fault! fault)Throw the latched refusal when fault holds one.
Throw the latched refusal when `fault` holds one.
(compact-temp-paths & targets)Sibling temp and commit-marker paths for a crash-safe compaction of targets, the files
rewritten and installed together (a record kind's (log, idx) pair, or the KV backend's one
log). One marker governs the set, keyed off the first target. Returns {:temps [[target temp] …] :marker marker} in the order named.
Sibling temp and commit-marker paths for a crash-safe compaction of `targets`, the files
rewritten and installed together (a record kind's (log, idx) pair, or the KV backend's one
log). One marker governs the set, keyed off the first target. Returns `{:temps [[target
temp] …] :marker marker}` in the order named.(create-dirty-marker! root)Durably drop the dirty marker (fsync the file AND its dir).
Durably drop the dirty marker (fsync the file AND its dir).
(delete-compact-temps! marker temps)Remove the commit marker then every temp in temps (marker first, so a crash
mid-cleanup can't leave a marker without its temps). Idempotent.
The fsync is what makes the removal durable, so it is owed only when a directory
entry actually went away — .delete answers that, and the common call finds nothing
to delete. clear-records! runs this per kind on every wipe, so an unconditional
fsync would charge a whole-store flush per kind to a wipe that had no compaction in
flight.
Remove the commit marker then every temp in `temps` (marker first, so a crash mid-cleanup can't leave a marker without its temps). Idempotent. The fsync is what makes the *removal* durable, so it is owed only when a directory entry actually went away — `.delete` answers that, and the common call finds nothing to delete. `clear-records!` runs this per kind on every wipe, so an unconditional fsync would charge a whole-store flush per kind to a wipe that had no compaction in flight.
(dirty-marker-present? root)True when a previous session's dirty marker survives — that session never ran a clean close!, so the open path should run its crash checks.
True when a previous session's dirty marker survives — that session never ran a clean close!, so the open path should run its crash checks.
Slot offset meaning 'nothing stored at this id'.
Slot offset meaning 'nothing stored at this id'.
(ensure-dir! path)Create the directory at path (and any missing parents); return the File.
Create the directory at `path` (and any missing parents); return the `File`.
(fault-reason t write?)The latch reason for t, thrown by an access to a live channel, or nil when t does
not stop the store. A closed channel stops it on any access; any other IOException
only on a write (write? true) or an fsync (write? :fsync), since a read that
fails leaves nothing on disk changed.
The latch reason for `t`, thrown by an access to a live channel, or nil when `t` does not stop the store. A closed channel stops it on any access; any other `IOException` only on a write (`write?` true) or an fsync (`write?` `:fsync`), since a read that fails leaves nothing on disk changed.
(fault-refusal {:keys [reason path cause] :as f})The :store-unusable refusal for the latched fault f, carrying :reason and
:path, with the throwable that latched it as the cause.
The `:store-unusable` refusal for the latched fault `f`, carrying `:reason` and `:path`, with the throwable that latched it as the cause.
(force! raf meta-data?)fsync raf's channel; meta-data? includes file metadata (mtime, length).
fsync `raf`'s channel; `meta-data?` includes file metadata (mtime, length).
On-disk layout version for a backend directory. Bumped only on an incompatible layout change; a directory written before the sentinel is adopted as version 1.
On-disk layout version for a backend directory. Bumped only on an incompatible layout change; a directory written before the sentinel is adopted as version 1.
(fsync-dir! path)Best-effort fsync of the directory holding path, so a file's creation or deletion is
durable. Swallowed on filesystems that reject directory fsync, and warned once: a
commit marker's directory entry is the atomic commit point of a compaction.
Best-effort fsync of the directory holding `path`, so a file's creation or deletion is durable. Swallowed on filesystems that reject directory fsync, and warned once: a commit marker's directory entry is the atomic commit point of a compaction.
(fsync-file! path)fsync the file at path, then the directory holding it (fsync-dir!).
fsync the file at `path`, then the directory holding it (`fsync-dir!`).
(index-layout-decision root current populated?)Check an index directory root's layout.edn sentinel against current
(kv/index-layout-version): :current when the version matches, :stale when it does
not (an absent sentinel over a populated? log, a damaged one, and one past
manifest-bytes included), and :unwritten for a fresh, empty directory. :stale is the
caller's cue to clear the index, rebuild it from the records and write-index-layout!.
This reads and never writes, since a base mounted under :base may not be written to.
Check an index directory `root`'s `layout.edn` sentinel against `current` (`kv/index-layout-version`): `:current` when the version matches, `:stale` when it does not (an absent sentinel over a `populated?` log, a damaged one, and one past `manifest-bytes` included), and `:unwritten` for a fresh, empty directory. `:stale` is the caller's cue to clear the index, rebuild it from the records and `write-index-layout!`. This reads and never writes, since a base mounted under `:base` may not be written to.
(latch-closed! fault path)Latch fault as :closed unless it already holds a fault — what a clean close leaves,
so a call on the store after it is refused by name rather than by a closed channel.
Latch `fault` as `:closed` unless it already holds a fault — what a clean close leaves, so a call on the store after it is refused by name rather than by a closed channel.
(latch-fault! fault path reason t)Latch fault with reason unless it already holds a fault, and return the refusal
for whichever fault it holds. The first latch logs at :error, once: every later call
is told through the refusal.
Latch `fault` with `reason` unless it already holds a fault, and return the refusal for whichever fault it holds. The first latch logs at `:error`, once: every later call is told through the refusal.
(log-tail-offset log-raf)The truncation point after a torn trailing write, computed from the frame lengths alone
(read a prefix, skip n, repeat) without thawing anything: scan-log's return value
without decoding the log. A frame is appended before the idx slot that points at it, so
truncating the tail is tidiness, and validate-idx-tail! reconciles a slot against a log
that lost its end. A non-positive length terminates the walk, since a payload is never
empty and a zero is space never written. The prefixes are read through a buffer by
positional channel reads, which never touch the shared file pointer, so this needs no kind
lock. See docs/storage.md, "Durability + crash-safety".
The truncation point after a torn trailing write, computed from the frame lengths alone (read a prefix, skip `n`, repeat) without thawing anything: `scan-log`'s return value without decoding the log. A frame is appended before the idx slot that points at it, so truncating the tail is tidiness, and `validate-idx-tail!` reconciles a slot against a log that lost its end. A non-positive length terminates the walk, since a payload is never empty and a zero is space never written. The prefixes are read through a buffer by positional channel reads, which never touch the shared file pointer, so this needs no kind lock. See docs/storage.md, "Durability + crash-safety".
(log-tail-offset-from log-raf clean-length)The truncation offset for log-raf, skipping the walk entirely when clean-length
(this log's entry in the clean marker) equals its current length. Returns
[offset scanned?] so a caller can report whether the fast path was taken.
The truncation offset for `log-raf`, skipping the walk entirely when `clean-length` (this log's entry in the clean marker) equals its current length. Returns `[offset scanned?]` so a caller can report whether the fast path was taken.
The most an EDN manifest may hold (a dump's meta.edn and index.edn, a store's
format.edn, an index directory's layout.edn and records.edn, a belief image's
manifest.edn, a corpus's report.edn, a machine's catalog.edn). Each is the first
thing read about a directory nobody has promised anything about, so the read is bounded.
The most an EDN manifest may hold (a dump's `meta.edn` and `index.edn`, a store's `format.edn`, an index directory's `layout.edn` and `records.edn`, a belief image's `manifest.edn`, a corpus's `report.edn`, a machine's `catalog.edn`). Each is the first thing read about a directory nobody has promised anything about, so the read is bounded.
(mark-index-rebuilding! root)Write root's layout.edn as mid-rebuild, so an open that lands before
write-index-layout! reads :stale and rebuilds again.
Write `root`'s `layout.edn` as mid-rebuild, so an open that lands before `write-index-layout!` reads `:stale` and rebuilds again.
(max-slot-id idx-raf)The highest slot id the idx file can address, or -1 if empty. Stable across
deletes (a tombstone keeps its slot), so 1 + max never reuses a handle.
The highest slot id the idx file can address, or -1 if empty. Stable across deletes (a tombstone keeps its slot), so `1 + max` never reuses a handle.
(open-idx path)Open an idx file for random read/write. Always rw — idx slots are
reconstructible from the logs, so they need no per-write durability.
Open an idx file for random read/write. Always `rw` — idx slots are reconstructible from the logs, so they need no per-write durability.
(open-log path)Open a log file for append + read, positioned at EOF. Under
vaelii.disk.fsync=dsync the channel is rwd so every append is durable.
The seek is guarded: a throw between the open and the return leaves a
RandomAccessFile nothing holds a reference to and nothing will close, and the caller
that answers a failed open by releasing the directory lock would then give the lock
back over a handle this JVM still has.
Open a log file for append + read, positioned at EOF. Under `vaelii.disk.fsync=dsync` the channel is `rwd` so every append is durable. The seek is guarded: a throw between the open and the return leaves a `RandomAccessFile` nothing holds a reference to and nothing will close, and the caller that answers a failed open by releasing the directory lock would then give the lock back over a handle this JVM still has.
(open-log-read path)Open a private read-only RAF on an existing log. The copy-on-write compactor reads the log's immutable (already-written) region through its own handle, so it needn't hold the kind lock while it does the O(live) rewrite — a fresh RAF has its own file pointer, so this does not touch the shared-pointer invariant of the store's live RAF.
Open a private read-only RAF on an existing log. The copy-on-write compactor reads the log's immutable (already-written) region through its own handle, so it needn't hold the kind lock while it does the O(live) rewrite — a fresh RAF has its own file pointer, so this does not touch the shared-pointer invariant of the store's live RAF.
(premise-flags premise?)(premise-flags premise? rank)The flags word for a slot whose record is, or is not, a premise, carrying the
premise's strength rank (0 when not a premise, or when the rank is unrecorded) in
bits 2..3.
The `flags` word for a slot whose record is, or is not, a premise, carrying the premise's strength `rank` (0 when not a premise, or when the rank is unrecorded) in bits 2..3.
(read-clean-marker root)The name -> length map a clean close left, or nil (absent or unreadable).
The `name -> length` map a clean close left, or nil (absent or unreadable).
(read-edn-manifest f)The EDN manifest in f, read under manifest-bytes, or a refusal (:manifest-too-large)
naming the file and the bound. The bound is on the bytes read, not on File.length, which
answers 0 for a FIFO. Content the EDN reader cannot parse is refused as
:malformed-manifest; the caller decides which refusal means a broken KB.
The EDN manifest in `f`, read under `manifest-bytes`, or a refusal (`:manifest-too-large`) naming the file and the bound. The bound is on the bytes read, not on `File.length`, which answers 0 for a FIFO. Content the EDN reader cannot parse is refused as `:malformed-manifest`; the caller decides which refusal means a broken KB.
(read-nippy-file path)(read-nippy-file path default)Read a whole-file nippy value, returning default when the file is missing, empty or
unreadable (a torn blob thaws to default and a warning). The file's bytes are read first and
thawed with thaw-from-in!, because write-nippy-atomic! writes a headerless typed stream.
A VirtualMachineError is rethrown as itself, so a heap failure never returns default.
Read a whole-file nippy value, returning `default` when the file is missing, empty or unreadable (a torn blob thaws to default and a warning). The file's bytes are read first and thawed with `thaw-from-in!`, because `write-nippy-atomic!` writes a headerless typed stream. A `VirtualMachineError` is rethrown as itself, so a heap failure never returns `default`.
(read-record log-raf offset)Read a nippy value from the log at offset; nil for a negative or past-EOF offset.
For a caller that holds the slot, read-record-sized is one read instead of two.
Read a nippy value from the log at `offset`; nil for a negative or past-EOF offset. For a caller that holds the slot, `read-record-sized` is one read instead of two.
(read-record-sized log-raf offset length)Thaw the frame at offset whose payload is length bytes — one positional read,
of the payload alone. The slot already recorded the payload length, so a caller
holding the slot needs neither a seek nor a re-read of the length prefix.
Thaw the frame at `offset` whose payload is `length` bytes — **one** positional read, of the payload alone. The slot already recorded the payload length, so a caller holding the slot needs neither a seek nor a re-read of the length prefix.
(read-slot idx-raf id)Read a slot by id: {:offset :length :flags :gen :tombstone?}, or nil when the
slot is past EOF (unwritten), empty (offset=-1), or a zero-filled gap
(offset=0 AND length=0, left where a slot written past the idx's end grew it over
unwritten ids — no real record has length 0, so this is unambiguous).
One positional read of the 24 bytes, decoded in memory: a short read is the
past-EOF test, so this costs neither a .length call to make that test nor a seek
and four primitive reads to satisfy it.
Read a slot by id: `{:offset :length :flags :gen :tombstone?}`, or nil when the
slot is past EOF (unwritten), empty (offset=-1), or a zero-filled gap
(offset=0 AND length=0, left where a slot written past the idx's end grew it over
unwritten ids — no real record has length 0, so this is unambiguous).
One positional read of the 24 bytes, decoded in memory: a short read *is* the
past-EOF test, so this costs neither a `.length` call to make that test nor a seek
and four primitive reads to satisfy it.(reconcile-slot-flags! idx-raf id premise? rank)Rewrite id's slot flags to match its record's premise-ness and strength rank when
the two would answer differently — a torn flags page across a crash can leave a slot
speaking a stale bit or rank. Returns true if the slot was rewritten. A slot that is
silent, or already agrees (an unrecorded rank included, since that falls back to the
record), is left untouched. Only the flags word moves — offset/length/gen are written
back unchanged — and the record is the durable truth, so this is a repair toward it,
never a new fact: idempotent, and safe to interrupt, since the next dirty open repeats
it.
Rewrite `id`'s slot flags to match its record's premise-ness and strength `rank` when the two would answer differently — a torn flags page across a crash can leave a slot speaking a stale bit or rank. Returns true if the slot was rewritten. A slot that is silent, or already agrees (an unrecorded rank included, since that falls back to the record), is left untouched. Only the flags word moves — offset/length/gen are written back unchanged — and the record is the durable truth, so this is a repair toward it, never a new fact: idempotent, and safe to interrupt, since the next dirty open repeats it.
(recorded-reasoners root)The reasoner names recorded in the store directory root (reasoners.edn), as a set,
or nil when the directory records none. Read through read-edn-manifest.
The reasoner names recorded in the store directory `root` (`reasoners.edn`), as a set, or nil when the directory records none. Read through `read-edn-manifest`.
(records-identity root)The record store root's index was last built against, or nil when the directory has
never been recorded. Read through read-edn-manifest, so a file past the bound or cut
mid-form is refused by name.
The record store `root`'s index was last built against, or nil when the directory has never been recorded. Read through `read-edn-manifest`, so a file past the bound or cut mid-form is refused by name.
(recover-compaction! & targets)Finish or discard a crash-interrupted compaction of targets, BEFORE they are opened
for normal use. Returns :replayed, :discarded-incomplete, or :none. Idempotent.
The marker plus a complete set of temps is the commit, whether the set is a record kind's (log, idx) pair or the KV backend's single write-ahead log: the originals stay authoritative until every temp is installed, so a crash before the marker keeps them and drops the orphan temps, and one after it replays what the marker vouches for.
Finish or discard a crash-interrupted compaction of `targets`, BEFORE they are opened for normal use. Returns :replayed, :discarded-incomplete, or :none. Idempotent. **The marker plus a complete set of temps is the commit**, whether the set is a record kind's (log, idx) pair or the KV backend's single write-ahead log: the originals stay authoritative until every temp is installed, so a crash before the marker keeps them and drops the orphan temps, and one after it replays what the marker vouches for.
(remove-clean-marker! root)Drop the clean marker — called on open, so it never describes a store in use.
Drop the clean marker — called on open, so it never describes a store in use.
(remove-dirty-marker! root)Remove the dirty marker after a clean fsync+close. Idempotent.
Remove the dirty marker after a clean fsync+close. Idempotent.
(replay-temp-onto-raf! target tmp-path)Install the freshly-built tmp-path over an already-open RAF: truncate, copy the
temp's bytes in, fsync, leave positioned at EOF. Used by the live compactor.
Install the freshly-built `tmp-path` over an already-open RAF: truncate, copy the temp's bytes in, fsync, leave positioned at EOF. Used by the live compactor.
(scan-idx! idx-raf on-slot)(scan-idx! idx-raf on-slot r)Walk the idx from the start, invoking (on-slot id offset length flags) for every live
slot (not empty, tombstone or zero-filled gap), in 96 KiB chunks. The flags ride along
because an open's walk is the only one that can answer what a slot knows without reading
its record. r, when non-nil, is a vaelii.impl.progress reporter ticked with the slots
each chunk covers.
Walk the idx from the start, invoking `(on-slot id offset length flags)` for every live slot (not empty, tombstone or zero-filled gap), in 96 KiB chunks. The flags ride along because an open's walk is the only one that can answer what a slot knows without reading its record. `r`, when non-nil, is a `vaelii.impl.progress` reporter ticked with the slots each chunk covers.
(scan-log log-raf f)(scan-log log-raf path f)(scan-log log-raf path f r)Scan an append-only log, calling (f index value) for each frame in order. Returns the
byte offset of the first byte past the last frame read, the truncation point after a torn
trailing write. The walk follows log-tail-offset's length chain. A frame whose payload
does not thaw is a torn tail when it is the chain's last (the walk stops at its offset and
logs it), and damage otherwise: the walk throws :damaged-frame naming path, the offset
and the frame's index, with the thaw's exception as cause. A VirtualMachineError is
rethrown as itself. r, when non-nil, is a progress reporter ticked with the bytes each
frame takes.
Scan an append-only log, calling `(f index value)` for each frame in order. Returns the byte offset of the first byte past the last frame read, the truncation point after a torn trailing write. The walk follows `log-tail-offset`'s length chain. A frame whose payload does not thaw is a torn tail when it is the chain's last (the walk stops at its offset and logs it), and damage otherwise: the walk throws `:damaged-frame` naming `path`, the offset and the frame's index, with the thaw's exception as cause. A `VirtualMachineError` is rethrown as itself. `r`, when non-nil, is a progress reporter ticked with the bytes each frame takes.
Fixed slot width in .idx files: 8 (offset) + 8 (length) + 4 (flags) + 4 (gen) = 24.
Fixed slot width in `.idx` files: 8 (offset) + 8 (length) + 4 (flags) + 4 (gen) = 24.
(slot-count idx-raf)The number of fixed-width slots the idx file currently holds.
The number of fixed-width slots the idx file currently holds.
(slot-premise flags)What flags says about its record: true, false, or nil for a slot that does
not say — which is what sends a caller to the record itself.
What `flags` says about its record: `true`, `false`, or **nil** for a slot that does not say — which is what sends a caller to the record itself.
(slot-strength flags)The strength rank a premise slot carries in bits 2..3, or 0 when it carries none — a non-premise, or a slot older than the strength bits — which sends the caller to the record for the answer.
The strength rank a premise slot carries in bits 2..3, or 0 when it carries none — a non-premise, or a slot older than the strength bits — which sends the caller to the record for the answer.
(token-log-present? root)True when root holds a tokens.log at all, an empty one included — so a store
must open the dictionary to decode its frames, whether or not it still writes tokenized
ones. Emptiness is not absence: a tokens.log a crash tore to zero bytes leaves the
sentexes log's tokenized frames citing ids no dictionary holds, and opening the (empty)
dictionary is what turns each such frame into a typed :damaged-dictionary tombstone on
rebuild-premises!'s walk. Opening the dictionary only at a non-empty length would
leave open-record-store with a nil dictionary and the tokenized decoder dereferencing
it.
True when `root` holds a `tokens.log` at all, an **empty** one included — so a store must open the dictionary to decode its frames, whether or not it still writes tokenized ones. Emptiness is not absence: a `tokens.log` a crash tore to zero bytes leaves the sentexes log's tokenized frames citing ids no dictionary holds, and opening the (empty) dictionary is what turns each such frame into a typed `:damaged-dictionary` tombstone on `rebuild-premises!`'s walk. Opening the dictionary only at a non-empty length would leave `open-record-store` with a nil dictionary and the tokenized decoder dereferencing it.
Slot offset meaning 'this id was deleted'.
Slot offset meaning 'this id was deleted'.
(tombstone-slot! idx-raf id)Mark slot id as a tombstone (offset=-2, other fields zeroed).
Mark slot `id` as a tombstone (offset=-2, other fields zeroed).
(truncate! raf)Truncate a log or idx RAF to empty and reposition at the start — the whole-file
wipe clear-*! needs.
Truncate a log or idx RAF to empty and reposition at the start — the whole-file wipe `clear-*!` needs.
(truncate-log! log-raf new-len)Truncate the log to new-len — used after scan-log detects a torn tail frame.
Truncate the log to `new-len` — used after `scan-log` detects a torn tail frame.
(validate-idx-tail! idx-raf log-raf)Tombstone every idx slot whose frame (offset + 4 + length) extends past the log's current
length, a torn trailing write where the idx page persisted and the log pages did not.
Returns the count repaired. It catches a slot that outruns the log and only that: a slot
spliced from two pages that still lands inside the log surfaces later as a thaw failure
on that handle (:type :malformed-record), since the 24-byte slot has no room for a
checksum. Rides scan-idx!'s chunked walk, so every live slot is tested.
Tombstone every idx slot whose frame (offset + 4 + length) extends past the log's current length, a torn trailing write where the idx page persisted and the log pages did not. Returns the count repaired. It catches a slot that outruns the log and only that: a slot spliced from two pages that still lands inside the log surfaces later as a thaw failure on that handle (`:type :malformed-record`), since the 24-byte slot has no room for a checksum. Rides `scan-idx!`'s chunked walk, so every live slot is tested.
(with-io-guard fault path write? & body)Run body against a store's live channels, latching fault and throwing its refusal
when what body throws stops the store (fault-reason). Anything else travels
unchanged.
Run `body` against a store's live channels, latching `fault` and throwing its refusal when what `body` throws stops the store (`fault-reason`). Anything else travels unchanged.
(write-clean-marker! root lengths)Durably record lengths (a name -> log length map) as this session's clean close.
Call after the final fsync and before closing, so the lengths are the durable ones.
Durably record `lengths` (a `name -> log length` map) as this session's clean close. Call after the final fsync and before closing, so the lengths are the durable ones.
(write-commit-marker! marker info)Atomically create the compaction commit marker and fsync it (and its dir). Its existence is the commit point.
Atomically create the compaction commit marker and fsync it (and its dir). Its existence is the commit point.
(write-index-layout! root current)Write root's layout.edn naming current — the commit mark of a layout
rebuild (index-layout-decision).
Write `root`'s `layout.edn` naming `current` — the commit mark of a layout rebuild (`index-layout-decision`).
(write-nippy-atomic! path value)Write value to path as a headerless nippy stream (freeze-to-out!), through
write-atomic!'s temp, fsync and rename.
Write `value` to `path` as a headerless nippy stream (`freeze-to-out!`), through `write-atomic!`'s temp, fsync and rename.
(write-reasoners! root names)Record names as the reasoner set of the store directory root.
Record `names` as the reasoner set of the store directory `root`.
(write-records-identity! root identity)Record which store root's index describes.
A :disk-log index over :disk records needs no such record: the two share the directory,
so the files cannot describe anything else. An index over records on a server can —
the directory says nothing about which database, and the coverage check that would catch
a mismatch compares record counts, which two unrelated databases match easily. What
gets through is worse than an empty index: reads answer out of another store's handles,
and a re-assert of a sentence this store does hold mints a second handle for it.
Record which store `root`'s index describes. A `:disk-log` index over `:disk` records needs no such record: the two share the directory, so the files cannot describe anything else. An index over records on a **server** can — the directory says nothing about which database, and the coverage check that would catch a mismatch compares record *counts*, which two unrelated databases match easily. What gets through is worse than an empty index: reads answer out of another store's handles, and a re-assert of a sentence this store does hold mints a second handle for it.
(write-slot! idx-raf id offset length flags gen)Write a slot for id — the 24 bytes packed in memory and written with one
positional channel write, the mirror of read-slot: a RandomAccessFile is
unbuffered, so a seek plus four primitive writes is five syscalls for 24 bytes. A
slot past the file's end grows it, and the gap is read back as unwritten
(read-slot's zero test).
Write a slot for `id` — the 24 bytes packed in memory and written with **one positional channel write**, the mirror of `read-slot`: a `RandomAccessFile` is unbuffered, so a seek plus four primitive writes is five syscalls for 24 bytes. A slot past the file's end grows it, and the gap is read back as unwritten (`read-slot`'s zero test).
(write-slots! idx-raf slots)Write many slots — each [id offset length flags gen] — as one positional write per
run of consecutive ids, where write-slot! is one write per slot.
A slot is 24 bytes at id * 24, so a run of consecutive handles is a contiguous range
of the idx and a batch of them is one syscall rather than one apiece. Handles are
minted in order and a preserved dump numbers its records in order, so the common case
is a single write for the whole batch; the runs are what keeps it correct when it is
not. Caller must hold the idx's write lock.
Write many slots — each `[id offset length flags gen]` — as **one positional write per run of consecutive ids**, where `write-slot!` is one write per slot. A slot is 24 bytes at `id * 24`, so a run of consecutive handles is a contiguous range of the idx and a batch of them is one syscall rather than one apiece. Handles are minted in order and a preserved dump numbers its records in order, so the common case is a single write for the whole batch; the runs are what keeps it correct when it is not. Caller must hold the idx's write lock.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |