A mapped snapshot of the columnar index: the compacted index is written to disk once
and mapped back on open, so the fact-scaled postings live in the page cache instead of
the heap. The index is derived state, so any doubt about a snapshot discards it and runs
reindex. See docs/storage.md, the image
section under :disk-snapshot.
Under <dir>/index/:
trie.csr — the trie's six CSR sections (fcounts foffsets fedge-tok
fedge-tgt fleaf-off fhandles), each a raw little-endian int run behind a header
naming the counts.roots.csr — every root family as CSR over dense-roots' packed long keys (sorted
keys, an offset column, one shared handle run), plus the scope table the count tries'
leaves decode through: one (predicate, position) pair, (predicate, position, context) triple or (context, 0, context) scope per entry, as three columns with -1
for a pair's context, indexed by dense-roots' argfam-id.roots-fallback.nippy — the vocabulary-scaled counters no routed family claims. Its
entry count and byte length ride the meta and are checked on open (read-fallback).tokens.log — the token dictionary the int edges cite, in vaelii.impl.disk.tokens'
format.Resident on open: the CSR skeleton (fcounts foffsets fedge-tok fedge-tgt), the
roots' key and offset columns, the scope table, the token dictionary and the fallback
blob. Mapped: fleaf-off, fhandles and the roots' handle run.
The meta records the format, kv/index-layout-version, a byte-order tag and
record-store/slot-fingerprint; all must agree on every open, else the open discards the
snapshot with a reason (:absent :layout-changed :records-differ :entries-truncated
:unsupported-platform). A commit writes the sections to temps and fsyncs them, deletes
the meta, renames the temps into place and writes the meta last, so a crash leaves no meta.
The commit renames over a mapped file, which Windows refuses: enabled? is false there.
A **mapped snapshot** of the columnar index: the compacted index is written to disk once and mapped back on open, so the fact-scaled postings live in the page cache instead of the heap. The index is derived state, so any doubt about a snapshot discards it and runs `reindex`. See docs/storage.md, the image section under `:disk-snapshot`. Under `<dir>/index/`: * `trie.csr` — the trie's six CSR sections (`fcounts` `foffsets` `fedge-tok` `fedge-tgt` `fleaf-off` `fhandles`), each a raw little-endian `int` run behind a header naming the counts. * `roots.csr` — every root family as CSR over `dense-roots`' packed `long` keys (sorted keys, an offset column, one shared handle run), plus the scope table the count tries' leaves decode through: one `(predicate, position)` pair, `(predicate, position, context)` triple or `(context, 0, context)` scope per entry, as three columns with -1 for a pair's context, indexed by `dense-roots`' `argfam-id`. * `roots-fallback.nippy` — the vocabulary-scaled counters no routed family claims. Its entry count and byte length ride the meta and are checked on open (`read-fallback`). * `tokens.log` — the token dictionary the `int` edges cite, in `vaelii.impl.disk.tokens`' format. Resident on open: the CSR skeleton (`fcounts` `foffsets` `fedge-tok` `fedge-tgt`), the roots' key and offset columns, the scope table, the token dictionary and the fallback blob. Mapped: `fleaf-off`, `fhandles` and the roots' handle run. The meta records the format, `kv/index-layout-version`, a byte-order tag and `record-store/slot-fingerprint`; all must agree on every open, else the open discards the snapshot with a reason (`:absent` `:layout-changed` `:records-differ` `:entries-truncated` `:unsupported-platform`). A commit writes the sections to temps and fsyncs them, deletes the meta, renames the temps into place and writes the meta last, so a crash leaves no meta. The commit renames over a mapped file, which Windows refuses: `enabled?` is false there.
(discard! dir)Delete a snapshot's commit marker — what a caller does when it knows the image is dead and would rather the next open not have to work that out.
Delete a snapshot's commit marker — what a caller does when it knows the image is dead and would rather the next open not have to work that out.
(drift dir roots-now)How far roots-now has moved from the image at dir, as a ratio of the image's own
count — or 0.0 when this JVM has neither written nor read one, since there is nothing
to have drifted from and a refresh would be writing an image against no baseline.
How far `roots-now` has moved from the image at `dir`, as a ratio of the image's own count — or 0.0 when this JVM has neither written nor read one, since there is nothing to have drifted from and a refresh would be writing an image against no baseline.
(due? dir roots-now)Has dir's image drifted past vaelii.index.snapshot-drift, with the interval floor
vaelii.disk.compact-min-interval-ms elapsed since the last attempt? False when
vaelii.disk.auto-compact=false, which is read before state-key resolves a path. A
drift threshold of 0 means any drift, not off. See docs/storage.md, "The cadence".
Has `dir`'s image drifted past `vaelii.index.snapshot-drift`, with the interval floor `vaelii.disk.compact-min-interval-ms` elapsed since the last attempt? False when `vaelii.disk.auto-compact=false`, which is read before `state-key` resolves a path. A drift threshold of 0 means any drift, not off. See docs/storage.md, "The cadence".
(enabled?)Can this platform serve a mapped image? True, or a throw — never false.
Which KBs get an image is kb/backend-axes' answer, not this one's: :index :snapshot
names the representation and kb/snapshot-mode? reads the axis. What is left here is
the platform, and it refuses rather than degrades. An operator who named
:disk-snapshot and silently got :disk-columnar would be told nothing about the
rebuild they think they no longer pay for, which is the whole of what the name bought.
Can this platform serve a mapped image? True, or a throw — never false. Which KBs get an image is `kb/backend-axes`' answer, not this one's: `:index :snapshot` names the representation and `kb/snapshot-mode?` reads the axis. What is left here is the platform, and it refuses rather than degrades. An operator who named `:disk-snapshot` and silently got `:disk-columnar` would be told nothing about the rebuild they think they no longer pay for, which is the whole of what the name bought.
(forget-image! dir)Drop dir's cadence state — what a close does, so a directory reopened in this JVM
measures its drift against the image it actually finds rather than against one this
process happened to write earlier.
Drop `dir`'s cadence state — what a close does, so a directory reopened in this JVM measures its drift against the image it actually finds rather than against one this process happened to write earlier.
The snapshot's own layout number, beside kv/index-layout-version (which says what the
entries mean). Bump when a section's shape or order changes, or when a packed key
cites a different scope. 4 packs each [key ctx] leaf of a count trie other than the
argument trie under its context's scope [ctx 0 ctx]. 5 packs the taxonomy's supporter
and length-binding count tries, their nodes interned whole, and every name set, so the
fallback blob holds counters alone.
The snapshot's own layout number, beside `kv/index-layout-version` (which says what the *entries* mean). Bump when a section's shape or order changes, or when a packed key cites a different scope. 4 packs each `[key ctx]` leaf of a count trie other than the argument trie under its context's scope `[ctx 0 ctx]`. 5 packs the taxonomy's supporter and length-binding count tries, their nodes interned whole, and every name set, so the fallback blob holds counters alone.
(load! dir store fingerprint-fn)Map dir/index into store. Returns {:index :mapped …} or {:index :rebuild :reason r}
with r one of :absent :layout-changed :byte-order :unsupported-platform
:records-differ :entries-truncated :duplicate-tokens :unreadable. The caller
reindexes on any :rebuild. :duplicate-tokens also repairs the log (load-dictionary!).
A VirtualMachineError in a failed map's cause chain (safe/vm-error) empties store
and is rethrown as itself.
Map `dir/index` into `store`. Returns `{:index :mapped …}` or `{:index :rebuild :reason r}`
with `r` one of `:absent` `:layout-changed` `:byte-order` `:unsupported-platform`
`:records-differ` `:entries-truncated` `:duplicate-tokens` `:unreadable`. The caller
reindexes on any `:rebuild`. `:duplicate-tokens` also repairs the log (`load-dictionary!`).
A `VirtualMachineError` in a failed map's cause chain (`safe/vm-error`) empties `store`
and is rethrown as itself.(note-attempt! dir)Restart dir's interval floor without moving its drift baseline. A refresh calls this as
it begins: only a completed save moves the baseline (note-image!), while the clock
advances whether the save lands, throws or declines (:unchanged, :empty). A no-op
for a directory with no cadence state.
Restart `dir`'s interval floor without moving its drift baseline. A refresh calls this as it begins: only a completed save moves the baseline (`note-image!`), while the clock advances whether the save lands, throws or declines (`:unchanged`, `:empty`). A no-op for a directory with no cadence state.
(note-no-image! dir)Start dir's cadence clock with no image on disk. A load! that maps overwrites the
count; one that rebuilds leaves it. It starts a clock and never resets one: a second
open-kb over a directory whose image is current keeps the baseline. forget-image!
drops the state at close.
Start `dir`'s cadence clock with no image on disk. A `load!` that maps overwrites the count; one that rebuilds leaves it. It starts a clock and never resets one: a second `open-kb` over a directory whose image is current keeps the baseline. `forget-image!` drops the state at close.
(save! dir store fingerprint-fn)Write a mapped snapshot of store (a columnar IndexStore) under dir/index, with
(fingerprint-fn) as its records fingerprint. Returns {:index :saved …} or {:index :skipped :reason r}.
fingerprint-fn is a thunk so load! can decline before paying for it; it is called
first, before any byte moves. The save compacts the trie, then writes every section to a
.tmp, drops the meta, renames the temps into place and writes the meta last. A throw
before the meta drop leaves the previous image readable and deletes the temps. An empty
index saves nothing and drops any surviving meta.
Write a mapped snapshot of `store` (a columnar `IndexStore`) under `dir/index`, with
`(fingerprint-fn)` as its records fingerprint. Returns `{:index :saved …}` or `{:index
:skipped :reason r}`.
`fingerprint-fn` is a thunk so `load!` can decline before paying for it; it is called
first, before any byte moves. The save compacts the trie, then writes every section to a
`.tmp`, drops the meta, renames the temps into place and writes the meta last. A throw
before the meta drop leaves the previous image readable and deletes the temps. An empty
index saves nothing and drops any surviving meta.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |