A mapped snapshot of the columnar index, which pages its cold tail to disk instead of holding all of it in heap.
:disk-columnar keeps durable records and rebuilds the derived index on every open.
That rebuild is O(records) — measured at 5.6 s for 313k, ~30 min at 100M — and the
rebuilt structure is then wholly resident, which is the wall the scale plan names.
This writes the compacted index to disk once and maps it back, so an open reads bytes
and the fact-scaled postings live in the page cache rather than the heap.
The index is derived state: reindex rebuilds every entry from the records. That
is what makes this cheap — no write-ahead log, no op log, no crash-consistent mutation
protocol, no bucket directory. It needs a snapshot that can be thrown away and
rebuilt whenever it is in doubt, and "in doubt" resolves to reindex in every case.
It is also why there is no directory to page. A flat-map index keys every trie node by
a boxed vector of its whole path prefix, so an out-of-core design over it has to page
the keys themselves; the columnar trie has no keys at all — a node's identity is its
int id and its position in the parallel arrays is the directory. columnar/compact!
already produces exactly the arrays this writes.
Under <dir>/index/, four things:
trie.csr — the trie's six CSR sections (fcounts foffsets fedge-tok
fedge-tgt fleaf-off fhandles), each a raw little-endian int run behind a
header naming the counts.roots.csr — every root family (context/functor roots, the argument roots, the
term, rule and exception indexes) as the same CSR shape over dense-roots' packed
long keys: sorted keys, an offset column, one shared handle run. Plus the scope
table the argument roots decode through — one (predicate, position) pair per entry,
indexed by the scope id their packed keys carry (dense-roots' argfam-id). The
table is vocabulary-scaled and rides this file because this file's key column is its
only reader: written in one pass, discarded as one unit, so the two cannot drift.roots-fallback.nippy — everything the routed families do not claim: the term and
slot rosters, whose members are names rather than handles. Both are
vocabulary-scaled. It is still index truth — the slot roster is what the
predicate-agnostic argument reads descend through — so its entry count and byte
length ride the meta and are checked on open like the CSR sections' lengths, and its
load is strict (read-fallback).tokens.log — the durable token dictionary the int edges cite, in
vaelii.impl.disk.tokens' format (append-only, id = append position, content-keyed,
first-writer-wins, never reused). That module is reused rather than a second
dictionary format minted: persisting the trie's int edges is precisely the seam
vaelii.impl.tokens names as its durable variant.One file per structure, not one per section. A structure is mapped or discarded as
a unit, so per-section files would multiply the crash window by six for nothing; the
section table in the header already names the offsets map needs.
scale-100m.md's rule is never page the walk — the worst measured index pathology was
the leading-variable trie fan, 18,512 lookups for one query, and a disk seek is worse
than the round trip that pathology was made of. So the load is deliberately asymmetric:
fcounts foffsets fedge-tok fedge-tgt), the
roots' key and offset columns, the scope table, the token dictionary, and the
fallback blob. Every one of them is path- or vocabulary-scaled.fleaf-off / fhandles and the roots' handle run. Each posting is
touched only when its own term is queried. Cold by construction.No handle family is an exception to that split. The resident half is where the index's shape lives and the mapped half is where its mass lives, and the line between them is the line between what the vocabulary sizes and what the facts do.
With mmap the OS page cache is the residency policy, which is the point — but only
because the skeleton stays hot.
The failure to fear is a stale snapshot that passes its check: one can be perfectly
self-consistent and describe a KB that no longer exists. So the stamp covers the
records, not the snapshot's own bytes, and it is checked on every open — never
behind a flag. Three things must agree or the snapshot is discarded and reindex runs:
the format and kv/index-layout-version, the byte-order tag (the sections are always
little-endian, so this guards a future format that changes the order, not this machine's
architecture), and record-store/slot-fingerprint. The
decision carries a reason from import's vocabulary — :absent :layout-changed
:records-differ :entries-truncated :unsupported-platform — because a rebuild
nobody can explain is a rebuild nobody notices.
The commit is an atomic rename over a file this process has mapped, which Windows does
not permit, so :index :snapshot is refused there (enabled?) and an image
found on disk is discarded as :unsupported-platform rather than read and never
refreshed. Nothing else in the durable store is implicated: the logs are appends and
the slots are positional writes.
A commit is one atomic step: the sections are written to temps and fsynced, the meta is deleted, the temps are renamed into place, and the meta is written last. Its presence is the commit point, so a crash anywhere leaves no meta, and no meta means reindex.
1.72–1.84× off the index's resident heap and a 1.5× faster open, on a corpus whose vocabulary is fixed — and resident heap that still grows with the facts, because the token dictionary is fact-scaled and the CSR skeleton is path-scaled. The acceptance property it was built for does not hold, which is why it is off by default.
A **mapped snapshot** of the columnar index, which pages its cold tail to disk instead of holding all of it in heap. `:disk-columnar` keeps durable records and rebuilds the derived index on every open. That rebuild is O(records) — measured at 5.6 s for 313k, ~30 min at 100M — and the rebuilt structure is then wholly resident, which is the wall the scale plan names. This writes the compacted index to disk once and maps it back, so an open reads bytes and the fact-scaled postings live in the page cache rather than the heap. ## The design is a snapshot, not a store The index is **derived state**: `reindex` rebuilds every entry from the records. That is what makes this cheap — no write-ahead log, no op log, no crash-consistent mutation protocol, no bucket directory. It needs a *snapshot* that can be thrown away and rebuilt whenever it is in doubt, and "in doubt" resolves to `reindex` in every case. It is also why there is no directory to page. A flat-map index keys every trie node by a boxed vector of its whole path prefix, so an out-of-core design over it has to page the keys themselves; the columnar trie has no keys at all — a node's identity is its `int` id and its position in the parallel arrays *is* the directory. `columnar/compact!` already produces exactly the arrays this writes. ## What is written Under `<dir>/index/`, four things: * `trie.csr` — the trie's six CSR sections (`fcounts` `foffsets` `fedge-tok` `fedge-tgt` `fleaf-off` `fhandles`), each a raw little-endian `int` run behind a header naming the counts. * `roots.csr` — **every** root family (context/functor roots, the argument roots, the term, rule and exception indexes) as the same CSR shape over `dense-roots`' packed `long` keys: sorted keys, an offset column, one shared handle run. Plus the scope table the argument roots decode through — one `(predicate, position)` pair per entry, indexed by the scope id their packed keys carry (`dense-roots`' `argfam-id`). The table is vocabulary-scaled and rides this file because this file's key column is its only reader: written in one pass, discarded as one unit, so the two cannot drift. * `roots-fallback.nippy` — everything the routed families do not claim: the term and slot rosters, whose members are *names* rather than handles. Both are vocabulary-scaled. It is still index truth — the slot roster is what the predicate-agnostic argument reads descend through — so its entry count and byte length ride the meta and are checked on open like the CSR sections' lengths, and its load is strict (`read-fallback`). * `tokens.log` — the durable token dictionary the `int` edges cite, in `vaelii.impl.disk.tokens`' format (append-only, id = append position, content-keyed, first-writer-wins, never reused). That module is reused rather than a second dictionary format minted: persisting the trie's `int` edges is precisely the seam `vaelii.impl.tokens` names as its durable variant. **One file per structure**, not one per section. A structure is mapped or discarded as a unit, so per-section files would multiply the crash window by six for nothing; the section table in the header already names the offsets `map` needs. ## The residency split `scale-100m.md`'s rule is *never page the walk* — the worst measured index pathology was the leading-variable trie fan, 18,512 lookups for one query, and a disk seek is worse than the round trip that pathology was made of. So the load is deliberately asymmetric: * **resident** — the CSR skeleton (`fcounts` `foffsets` `fedge-tok` `fedge-tgt`), the roots' key and offset columns, the scope table, the token dictionary, and the fallback blob. Every one of them is path- or vocabulary-scaled. * **mapped** — `fleaf-off` / `fhandles` and the roots' handle run. Each posting is touched only when its own term is queried. Cold by construction. **No handle family is an exception to that split.** The resident half is where the index's *shape* lives and the mapped half is where its mass lives, and the line between them is the line between what the vocabulary sizes and what the facts do. With `mmap` the OS page cache is the residency policy, which is the point — but only because the skeleton stays hot. ## Validity is the whole design The failure to fear is a stale snapshot that passes its check: one can be perfectly self-consistent and describe a KB that no longer exists. So the stamp covers the **records**, not the snapshot's own bytes, and it is checked on **every** open — never behind a flag. Three things must agree or the snapshot is discarded and `reindex` runs: the format and `kv/index-layout-version`, the byte-order tag (the sections are always little-endian, so this guards a future format that changes the order, not this machine's architecture), and `record-store/slot-fingerprint`. The decision carries a reason from `import`'s vocabulary — `:absent` `:layout-changed` `:records-differ` `:entries-truncated` `:unsupported-platform` — because a rebuild nobody can explain is a rebuild nobody notices. ## The platform The commit is an atomic rename over a file this process has mapped, which Windows does not permit, so `:index :snapshot` is **refused** there (`enabled?`) and an image found on disk is discarded as `:unsupported-platform` rather than read and never refreshed. Nothing else in the durable store is implicated: the logs are appends and the slots are positional writes. A commit is one atomic step: the sections are written to temps and fsynced, the meta is **deleted**, the temps are renamed into place, and the meta is written last. Its presence is the commit point, so a crash anywhere leaves no meta, and no meta means reindex. ## What it measured 1.72–1.84× off the index's resident heap and a 1.5× faster open, on a corpus whose vocabulary is fixed — and resident heap that still grows with the facts, because the token dictionary is fact-scaled and the CSR skeleton is path-scaled. The acceptance property it was built for does **not** hold, which is why it is off by default.
(discard! dir)Delete a snapshot's commit marker — what a caller does when it knows the image is dead and would rather the next open not have to work that out.
Delete a snapshot's commit marker — what a caller does when it knows the image is dead and would rather the next open not have to work that out.
(drift dir roots-now)How far roots-now has moved from the image at dir, as a ratio of the image's own
count — or 0.0 when this JVM has neither written nor read one, since there is nothing
to have drifted from and a refresh would be writing an image against no baseline.
How far `roots-now` has moved from the image at `dir`, as a ratio of the image's own count — or 0.0 when this JVM has neither written nor read one, since there is nothing to have drifted from and a refresh would be writing an image against no baseline.
(due? dir roots-now)Has dir's image drifted past vaelii.index.snapshot-drift, with the compaction
interval floor elapsed since the last one?
The floor is vaelii.disk.compact-min-interval-ms, shared with the record store's
compaction rather than given a knob of its own: both answer the same question — how
often may a background rewrite of a derived structure interrupt this KB — and two knobs
would be two answers to it.
vaelii.disk.auto-compact=false turns the mid-life refresh off, leaving the image to
the close and to the JVM-shutdown hook. Same knob as the record store's background and
opportunistic compaction, on the same argument the shared floor rests on: a refresh is
an opportunistic compaction of a derived structure — save! calls columnar/compact! —
so an operator who has said not to do those must not still be paying one per drift
threshold, on the writer's thread, with nothing saying so. The caller that wants the
switch is the batch that fills a whole KB in one run and closes cleanly: it needs exactly
one image, at the end.
The drift ratio cannot say that, and 0 is the trap. As a threshold zero means
"any drift at all", so it is the most eager setting in the range and rewrites the image
on every write past the floor — 400 asserts under 0 and a floor of 0 measured 401
images. It is the pathology the threshold exists to bound, not the off switch.
One state read, one state-key. This is asked per write on a :disk-snapshot KB
and state-key resolves a canonical path, so the switch is read first — an off switch
that still paid a realpath per write would not be off for the writer that asked for it
— and the entry is then read once, with the drift computed off it rather than reaching
for the same key again through drift.
Has `dir`'s image drifted past `vaelii.index.snapshot-drift`, with the compaction interval floor elapsed since the last one? The floor is `vaelii.disk.compact-min-interval-ms`, shared with the record store's compaction rather than given a knob of its own: both answer the same question — how often may a background rewrite of a derived structure interrupt this KB — and two knobs would be two answers to it. **`vaelii.disk.auto-compact=false` turns the mid-life refresh off**, leaving the image to the close and to the JVM-shutdown hook. Same knob as the record store's background and opportunistic compaction, on the same argument the shared floor rests on: a refresh *is* an opportunistic compaction of a derived structure — `save!` calls `columnar/compact!` — so an operator who has said not to do those must not still be paying one per drift threshold, on the writer's thread, with nothing saying so. The caller that wants the switch is the batch that fills a whole KB in one run and closes cleanly: it needs exactly one image, at the end. **The drift ratio cannot say that, and `0` is the trap.** As a *threshold* zero means "any drift at all", so it is the most eager setting in the range and rewrites the image on every write past the floor — 400 asserts under `0` and a floor of `0` measured 401 images. It is the pathology the threshold exists to bound, not the off switch. **One state read, one `state-key`.** This is asked per write on a `:disk-snapshot` KB and `state-key` resolves a canonical path, so the switch is read first — an off switch that still paid a `realpath` per write would not be off for the writer that asked for it — and the entry is then read once, with the drift computed off it rather than reaching for the same key again through `drift`.
(enabled?)Can this platform serve a mapped image? True, or a throw — never false.
Which KBs get an image is kb/backend-axes' answer, not this one's: :index :snapshot
names the representation and kb/snapshot-mode? reads the axis. What is left here is
the platform, and it refuses rather than degrades. An operator who named
:disk-snapshot and silently got :disk-columnar would be told nothing about the
rebuild they think they no longer pay for, which is the whole of what the name bought.
Can this platform serve a mapped image? True, or a throw — never false. Which KBs get an image is `kb/backend-axes`' answer, not this one's: `:index :snapshot` names the representation and `kb/snapshot-mode?` reads the axis. What is left here is the platform, and it refuses rather than degrades. An operator who named `:disk-snapshot` and silently got `:disk-columnar` would be told nothing about the rebuild they think they no longer pay for, which is the whole of what the name bought.
(forget-image! dir)Drop dir's cadence state — what a close does, so a directory reopened in this JVM
measures its drift against the image it actually finds rather than against one this
process happened to write earlier.
Drop `dir`'s cadence state — what a close does, so a directory reopened in this JVM measures its drift against the image it actually finds rather than against one this process happened to write earlier.
The snapshot's own layout number, beside kv/index-layout-version (which says what the
entries mean). Bump when a section's shape or order changes.
The snapshot's own layout number, beside `kv/index-layout-version` (which says what the *entries* mean). Bump when a section's shape or order changes.
(load! dir store stamp-fn)Map dir/index into store, or say why it cannot be. Returns {:index :mapped …} or
{:index :rebuild :reason r} with r one of :absent :layout-changed :byte-order
:unsupported-platform :records-differ :entries-truncated :duplicate-tokens
:unreadable.
The caller reindexes on any :rebuild — which is always legal, because the index is
derived state and this is a cache of it.
:duplicate-tokens is the one rebuild that also repairs as it declines
(load-dictionary!), and so the one a reader should not expect twice over one
directory.
Map `dir/index` into `store`, or say why it cannot be. Returns `{:index :mapped …}` or
`{:index :rebuild :reason r}` with `r` one of `:absent` `:layout-changed` `:byte-order`
`:unsupported-platform` `:records-differ` `:entries-truncated` `:duplicate-tokens`
`:unreadable`.
The caller reindexes on any `:rebuild` — which is always legal, because the index is
derived state and this is a cache of it.
`:duplicate-tokens` is the one rebuild that also **repairs** as it declines
(`load-dictionary!`), and so the one a reader should not expect twice over one
directory.(note-attempt! dir)Restart dir's interval floor without moving its drift baseline — what a refresh does
as it begins.
The baseline belongs to an image that is actually on disk, so only a completed save
moves it (note-image!). The clock is a different claim — how recently a refresh was
attempted — and it has to advance whether or not one landed: a save that throws, and a
save that declines (:unchanged, :empty), both leave the drift exactly where it was,
so a cadence stamped only on success reads due again on the very next write. That turns
a broken directory into an image-sized write per assert. Stamped here, it retries once
a floor, which is the cadence the floor exists to impose.
A no-op for a directory with no cadence state: due? cannot fire without one, so
minting a baseline here would be inventing an image nothing wrote.
Restart `dir`'s interval floor without moving its drift baseline — what a refresh does as it begins. The baseline belongs to an image that is actually on disk, so only a completed save moves it (`note-image!`). The *clock* is a different claim — how recently a refresh was attempted — and it has to advance whether or not one landed: a save that throws, and a save that declines (`:unchanged`, `:empty`), both leave the drift exactly where it was, so a cadence stamped only on success reads due again on the very next write. That turns a broken directory into an image-sized write per assert. Stamped here, it retries once a floor, which is the cadence the floor exists to impose. A no-op for a directory with no cadence state: `due?` cannot fire without one, so minting a baseline here would be inventing an image nothing wrote.
(note-no-image! dir)Start dir's cadence clock with no image on disk — what an open does before it
knows whether one is there. A load! that maps overwrites this with the real count;
one that rebuilds leaves it standing, which is the truth: nothing usable is on disk.
Without this a directory that has never held an image would have no baseline, so
drift would answer 0.0 forever and a writer that ran for weeks and was killed would
reopen onto nothing — which is the failure the cadence exists to close, and the one a
fresh directory is most exposed to.
It starts a clock and never resets one. Every KB constructed over a directory runs
this, and they share one index — so a second open-kb over a directory whose image is
current would otherwise reset the baseline to zero, drift would read 1.0 against an
image that is exactly right, and the first write past the interval floor would rewrite
a whole CSR for nothing. A close is what drops the state (forget-image!), which is
the point at which what is on disk stops being knowable from in here.
Start `dir`'s cadence clock with **no** image on disk — what an open does before it knows whether one is there. A `load!` that maps overwrites this with the real count; one that rebuilds leaves it standing, which is the truth: nothing usable is on disk. Without this a directory that has never held an image would have no baseline, so `drift` would answer 0.0 forever and a writer that ran for weeks and was killed would reopen onto nothing — which is the failure the cadence exists to close, and the one a fresh directory is most exposed to. **It starts a clock and never resets one.** Every KB constructed over a directory runs this, and they share one index — so a second `open-kb` over a directory whose image is current would otherwise reset the baseline to zero, `drift` would read 1.0 against an image that is exactly right, and the first write past the interval floor would rewrite a whole CSR for nothing. A close is what drops the state (`forget-image!`), which is the point at which what is on disk stops being knowable from in here.
(save! dir store stamp-fn)Write a mapped snapshot of store (a columnar IndexStore) under dir/index, stamped
with (stamp-fn). Returns {:index :saved …}, or {:index :skipped :reason r}.
The stamp arrives as a thunk rather than as a record store: what an image is valid
against is a fingerprint of the records, and which store computed it is none of this
namespace's business — record-store/slot-fingerprint is what the disk KB passes.
A thunk, so load! can decline before paying for it.
Compacts the trie first — the CSR arrays compact! produces are the on-disk layout,
so there is no serialization step, only a write. An empty index saves nothing and
drops any meta that survives, since a stale one describing a wiped KB is the one thing
worse than none.
A failed save costs the new image, never the one already there. Every section is
written to a .tmp beside its target and the meta — the commit mark — is dropped only
once all of them are complete, so a throw anywhere before that leaves the previous
image whole and readable, and takes the temps it had written with it: a section is
the size of the index, and one left behind by a save that never committed is a file
nothing reads and nothing else deletes. The stamp is taken first, before a byte
moves, because it is the one input that lives in another component: on the shutdown
path that component can already be closed, and taking it late would trade a working
image for none.
Write a mapped snapshot of `store` (a columnar `IndexStore`) under `dir/index`, stamped
with `(stamp-fn)`. Returns `{:index :saved …}`, or `{:index :skipped :reason r}`.
The stamp arrives as a thunk rather than as a record store: what an image is valid
against is a *fingerprint of the records*, and which store computed it is none of this
namespace's business — `record-store/slot-fingerprint` is what the disk KB passes.
A thunk, so `load!` can decline before paying for it.
Compacts the trie first — the CSR arrays `compact!` produces *are* the on-disk layout,
so there is no serialization step, only a write. An empty index saves nothing and
drops any meta that survives, since a stale one describing a wiped KB is the one thing
worse than none.
**A failed save costs the new image, never the one already there.** Every section is
written to a `.tmp` beside its target and the meta — the commit mark — is dropped only
once all of them are complete, so a throw anywhere before that leaves the previous
image whole and readable, **and takes the temps it had written with it**: a section is
the size of the index, and one left behind by a save that never committed is a file
nothing reads and nothing else deletes. The stamp is taken *first*, before a byte
moves, because it is the one input that lives in another component: on the shutdown
path that component can already be closed, and taking it late would trade a working
image for none.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |