A mapped snapshot of the columnar index, which pages its cold tail to disk instead of holding all of it in heap.
:disk-columnar keeps durable records and rebuilds the derived index on every open.
That rebuild is O(records) — measured at 5.6 s for 313k, ~30 min at 100M — and the
rebuilt structure is then wholly resident, which is the wall the scale plan names.
This writes the compacted index to disk once and maps it back, so an open reads bytes
and the fact-scaled postings live in the page cache rather than the heap.
The index is derived state: reindex rebuilds every entry from the records. That
is what makes this cheap — no write-ahead log, no op log, no crash-consistent mutation
protocol, no bucket directory. It needs a snapshot that can be thrown away and
rebuilt whenever it is in doubt, and "in doubt" resolves to reindex in every case.
It is also why there is no directory to page. A flat-map index keys every trie node by
a boxed vector of its whole path prefix, so an out-of-core design over it has to page
the keys themselves; the columnar trie has no keys at all — a node's identity is its
int id and its position in the parallel arrays is the directory. columnar/compact!
already produces exactly the arrays this writes.
Under <dir>/index/, four things:
trie.csr — the trie's six CSR sections (fcounts foffsets fedge-tok
fedge-tgt fleaf-off fhandles), each a raw little-endian int run behind a
header naming the counts.roots.csr — the secondary roots / term / rule / exception postings as the same CSR
shape over dense-roots' packed long keys: sorted keys, an offset column, one
shared handle run.roots-fallback.nippy — the term roster and anything else the routed families do not
claim. Vocabulary-scaled and irregularly shaped, so it is a blob rather than a
column.tokens.log — the durable token dictionary the int edges cite, in
vaelii.impl.disk.tokens' format (append-only, id = append position, content-keyed,
first-writer-wins, never reused). That module is reused rather than a second
dictionary format minted: persisting the trie's int edges is precisely the seam
vaelii.impl.tokens names as its durable variant.One file per structure, not one per section. A structure is mapped or discarded as
a unit, so per-section files would multiply the crash window by six for nothing; the
section table in the header already names the offsets map needs.
scale-100m.md's rule is never page the walk — the worst measured index pathology was
the leading-variable trie fan, 18,512 lookups for one query, and a disk seek is worse
than the round trip that pathology was made of. So the load is deliberately asymmetric:
fcounts foffsets fedge-tok fedge-tgt), the
roots' key and offset columns, and the token dictionary. Read into heap on open.fleaf-off / fhandles and the roots' handle run. These are the
fact-scaled mass (the roots alone are 69 MB of a 186 MB compacted 300k-fact index) and
each posting is touched only when its own term is queried. Cold by construction.With mmap the OS page cache is the residency policy, which is the point — but only
because the skeleton stays hot.
The failure to fear is a stale snapshot that passes its check: one can be perfectly
self-consistent and describe a KB that no longer exists. So the stamp covers the
records, not the snapshot's own bytes, and it is checked on every open — never
behind a flag. Three things must agree or the snapshot is discarded and reindex runs:
the format and kv/index-layout-version, the byte-order tag (an image whose endianness
differs is refused rather than read wrong), and record-store/slot-fingerprint. The
decision carries a reason from import's vocabulary — :absent :layout-changed
:records-differ :entries-truncated — because a rebuild nobody can explain is a
rebuild nobody notices.
A commit is one atomic step: the sections are written to temps and fsynced, the meta is deleted, the temps are renamed into place, and the meta is written last. Its presence is the commit point, so a crash anywhere leaves no meta, and no meta means reindex.
1.72–1.84× off the index's resident heap and a 1.5× faster open, on a corpus whose vocabulary is fixed — and resident heap that still grows with the facts, because the token dictionary is fact-scaled and the CSR skeleton is path-scaled. The acceptance property it was built for does not hold, which is why it is off by default.
A **mapped snapshot** of the columnar index, which pages its cold tail to disk instead of holding all of it in heap. `:disk-columnar` keeps durable records and rebuilds the derived index on every open. That rebuild is O(records) — measured at 5.6 s for 313k, ~30 min at 100M — and the rebuilt structure is then wholly resident, which is the wall the scale plan names. This writes the compacted index to disk once and maps it back, so an open reads bytes and the fact-scaled postings live in the page cache rather than the heap. ## The design is a snapshot, not a store The index is **derived state**: `reindex` rebuilds every entry from the records. That is what makes this cheap — no write-ahead log, no op log, no crash-consistent mutation protocol, no bucket directory. It needs a *snapshot* that can be thrown away and rebuilt whenever it is in doubt, and "in doubt" resolves to `reindex` in every case. It is also why there is no directory to page. A flat-map index keys every trie node by a boxed vector of its whole path prefix, so an out-of-core design over it has to page the keys themselves; the columnar trie has no keys at all — a node's identity is its `int` id and its position in the parallel arrays *is* the directory. `columnar/compact!` already produces exactly the arrays this writes. ## What is written Under `<dir>/index/`, four things: * `trie.csr` — the trie's six CSR sections (`fcounts` `foffsets` `fedge-tok` `fedge-tgt` `fleaf-off` `fhandles`), each a raw little-endian `int` run behind a header naming the counts. * `roots.csr` — the secondary roots / term / rule / exception postings as the same CSR shape over `dense-roots`' packed `long` keys: sorted keys, an offset column, one shared handle run. * `roots-fallback.nippy` — the term roster and anything else the routed families do not claim. Vocabulary-scaled and irregularly shaped, so it is a blob rather than a column. * `tokens.log` — the durable token dictionary the `int` edges cite, in `vaelii.impl.disk.tokens`' format (append-only, id = append position, content-keyed, first-writer-wins, never reused). That module is reused rather than a second dictionary format minted: persisting the trie's `int` edges is precisely the seam `vaelii.impl.tokens` names as its durable variant. **One file per structure**, not one per section. A structure is mapped or discarded as a unit, so per-section files would multiply the crash window by six for nothing; the section table in the header already names the offsets `map` needs. ## The residency split `scale-100m.md`'s rule is *never page the walk* — the worst measured index pathology was the leading-variable trie fan, 18,512 lookups for one query, and a disk seek is worse than the round trip that pathology was made of. So the load is deliberately asymmetric: * **resident** — the CSR skeleton (`fcounts` `foffsets` `fedge-tok` `fedge-tgt`), the roots' key and offset columns, and the token dictionary. Read into heap on open. * **mapped** — `fleaf-off` / `fhandles` and the roots' handle run. These are the fact-scaled mass (the roots alone are 69 MB of a 186 MB compacted 300k-fact index) and each posting is touched only when its own term is queried. Cold by construction. With `mmap` the OS page cache is the residency policy, which is the point — but only because the skeleton stays hot. ## Validity is the whole design The failure to fear is a stale snapshot that passes its check: one can be perfectly self-consistent and describe a KB that no longer exists. So the stamp covers the **records**, not the snapshot's own bytes, and it is checked on **every** open — never behind a flag. Three things must agree or the snapshot is discarded and `reindex` runs: the format and `kv/index-layout-version`, the byte-order tag (an image whose endianness differs is refused rather than read wrong), and `record-store/slot-fingerprint`. The decision carries a reason from `import`'s vocabulary — `:absent` `:layout-changed` `:records-differ` `:entries-truncated` — because a rebuild nobody can explain is a rebuild nobody notices. A commit is one atomic step: the sections are written to temps and fsynced, the meta is **deleted**, the temps are renamed into place, and the meta is written last. Its presence is the commit point, so a crash anywhere leaves no meta, and no meta means reindex. ## What it measured 1.72–1.84× off the index's resident heap and a 1.5× faster open, on a corpus whose vocabulary is fixed — and resident heap that still grows with the facts, because the token dictionary is fact-scaled and the CSR skeleton is path-scaled. The acceptance property it was built for does **not** hold, which is why it is off by default.
(discard! dir)Delete a snapshot's commit marker — what a caller does when it knows the image is dead and would rather the next open not have to work that out.
Delete a snapshot's commit marker — what a caller does when it knows the image is dead and would rather the next open not have to work that out.
(enabled?)Is the mapped index snapshot on? vaelii.index.snapshot — the property is the switch,
and the validity check is not gated on it: a snapshot that exists is either valid or
discarded, never trusted because a flag said so.
Is the mapped index snapshot on? `vaelii.index.snapshot` — the property is the switch, and the *validity* check is not gated on it: a snapshot that exists is either valid or discarded, never trusted because a flag said so.
The snapshot's own layout number, beside kv/index-layout-version (which says what the
entries mean). Bump when a section's shape or order changes.
The snapshot's own layout number, beside `kv/index-layout-version` (which says what the *entries* mean). Bump when a section's shape or order changes.
(load! dir store stamp-fn)Map dir/index into store, or say why it cannot be. Returns {:index :mapped …} or
{:index :rebuild :reason r} with r one of :absent :layout-changed :byte-order
:records-differ :entries-truncated :unreadable.
The caller reindexes on any :rebuild — which is always legal, because the index is
derived state and this is a cache of it.
Map `dir/index` into `store`, or say why it cannot be. Returns `{:index :mapped …}` or
`{:index :rebuild :reason r}` with `r` one of `:absent` `:layout-changed` `:byte-order`
`:records-differ` `:entries-truncated` `:unreadable`.
The caller reindexes on any `:rebuild` — which is always legal, because the index is
derived state and this is a cache of it.(save! dir store stamp-fn)Write a mapped snapshot of store (a columnar IndexStore) under dir/index, stamped
with (stamp-fn). Returns {:index :saved …}, or {:index :skipped :reason r}.
The stamp arrives as a thunk rather than as a record store: what an image is valid
against is a fingerprint of the records, and which store computed it is none of this
namespace's business — record-store/slot-fingerprint is what the disk KB passes.
A thunk, so load! can decline before paying for it.
Compacts the trie first — the CSR arrays compact! produces are the on-disk layout,
so there is no serialization step, only a write. An empty index saves nothing and
drops any meta that survives, since a stale one describing a wiped KB is the one thing
worse than none.
Write a mapped snapshot of `store` (a columnar `IndexStore`) under `dir/index`, stamped
with `(stamp-fn)`. Returns `{:index :saved …}`, or `{:index :skipped :reason r}`.
The stamp arrives as a thunk rather than as a record store: what an image is valid
against is a *fingerprint of the records*, and which store computed it is none of this
namespace's business — `record-store/slot-fingerprint` is what the disk KB passes.
A thunk, so `load!` can decline before paying for it.
Compacts the trie first — the CSR arrays `compact!` produces *are* the on-disk layout,
so there is no serialization step, only a write. An empty index saves nothing and
drops any meta that survives, since a stale one describing a wiped KB is the one thing
worse than none.(snapshot-root dir)cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |