A key-interning KvBackend (vaelii.impl.kv) for the columnar index's non-trie
families — the secondary roots, the rule / exception indexes, and the inverted term
index.
Those families are flat structured-vector-key → handle-set maps, and the columnar
measurement (bench/…/densetrie.clj) found their boxed vector keys
([:term-index term], [:functor-root pred], …) to be ~150 MB — the majority of the
columnar index once the trie went native. This backend keeps the values as
IntPostings (Phase 1's tiered
int[]/Roaring set) but collapses the keys: the term is interned to an int through
the shared trie dictionary (vaelii.impl.tokens) — so a predicate/individual gets
the same id the trie edges use — and the whole key becomes one packed long
(family | pos | term-id) into a single primitive Long2ObjectOpenHashMap. No boxed
vectors, no HAMT nodes, one map.
It stays a full KvBackend so the existing composition (an embedded KvIndexStore
over it) is unchanged: only the recognized index families are int-routed; any other key
— the slot roster and the term roster (whose members are names, not handles), a
scalar, a counter, the contract test's synthetic keys — falls back to a plain in-memory
backend (in the columnar store the trie is native, so no [:trie …] key ever reaches
here).
Every handle family routes, the predicate-scoped argument roots included: their
(pred, pos) scope is interned to a dense id of its own (argfam-id) and rides the
pos field, which no other family uses. So the fallback holds only vocabulary-scaled
name sets, and the fact-scaled mass is one packed map — or, under a snapshot, one
mapped run. Single-writer, like every index;
kv-members / kv-intersect materialize a fresh Clojure set at the boundary — but
kv-intersect builds it at the size of the answer, narrowing through
postings/intersect-postings in whichever representation each posting is in, a mapped run
included. Proven set-equal to MemoryKvBackend on the index families by
dense_roots_oracle_test —
which, like every behavioural check, cannot see a family that falls back when it should
route, since the fallback answers identically; dense_routing_test reads the
representation and covers that.
Single-threaded, which is narrower than single-writer. The mapped-section fields
on DenseRoots are ^:unsynchronized-mutable, so installing or thawing a snapshot
publishes through no barrier and a second thread may read this backend mid-install —
mapped? true against a mkeys it has not seen, say. The atom- and lock-based
backends give an incidental reader beside the writer a consistent view; this one does
not. Same trade as vaelii.impl.columnar, whose docstring states it: these fields are
read on the hot lookup path, and a volatile read there buys a guarantee the engine's own
single writer never needs.
A key-interning `KvBackend` (`vaelii.impl.kv`) for the columnar index's non-trie families — the secondary roots, the rule / exception indexes, and the inverted term index. Those families are flat `structured-vector-key → handle-set` maps, and the columnar measurement (`bench/…/densetrie.clj`) found their **boxed vector keys** (`[:term-index term]`, `[:functor-root pred]`, …) to be ~150 MB — the majority of the columnar index once the trie went native. This backend keeps the *values* as `IntPostings` (Phase 1's tiered `int[]`/Roaring set) but collapses the keys: the term is interned to an `int` through the **shared trie dictionary** (`vaelii.impl.tokens`) — so a predicate/individual gets the same id the trie edges use — and the whole key becomes one packed `long` (`family | pos | term-id`) into a single primitive `Long2ObjectOpenHashMap`. No boxed vectors, no HAMT nodes, one map. It stays a full `KvBackend` so the existing composition (an embedded `KvIndexStore` over it) is unchanged: only the recognized index families are int-routed; any other key — the slot roster and the term roster (whose members are *names*, not handles), a scalar, a counter, the contract test's synthetic keys — falls back to a plain in-memory backend (in the columnar store the trie is native, so no `[:trie …]` key ever reaches here). **Every handle family routes**, the predicate-scoped argument roots included: their `(pred, pos)` scope is interned to a dense id of its own (`argfam-id`) and rides the `pos` field, which no other family uses. So the fallback holds only vocabulary-scaled name sets, and the fact-scaled mass is one packed map — or, under a snapshot, one mapped run. Single-writer, like every index; `kv-members` / `kv-intersect` materialize a fresh Clojure set at the boundary — but `kv-intersect` builds it at the size of the *answer*, narrowing through `postings/intersect-postings` in whichever representation each posting is in, a mapped run included. Proven set-equal to `MemoryKvBackend` on the index families by `dense_roots_oracle_test` — which, like every behavioural check, cannot see a family that falls back when it should route, since the fallback answers identically; `dense_routing_test` reads the representation and covers that. **Single-*threaded*, which is narrower than single-writer.** The mapped-section fields on `DenseRoots` are `^:unsynchronized-mutable`, so installing or thawing a snapshot publishes through no barrier and a second thread may read this backend mid-install — `mapped?` true against a `mkeys` it has not seen, say. The atom- and lock-based backends give an incidental reader beside the writer a consistent view; this one does not. Same trade as `vaelii.impl.columnar`, whose docstring states it: these fields are read on the hot lookup path, and a volatile read there buys a guarantee the engine's own single writer never needs.
(argfam-table b remap)The scope dictionary as {:preds int[] :positions int[]}, indexed by scope id, with
each predicate taken through remap into the durable dictionary's id space — the same
int[] the packed keys' term halves are remapped by.
A pair's predicate is interned into the term dictionary when the pair is
(argfam-id), so every id here has a term id to be written as.
The scope dictionary as `{:preds int[] :positions int[]}`, indexed by scope id, with
each predicate taken through `remap` into the durable dictionary's id space — the same
`int[]` the packed keys' term halves are remapped by.
A pair's predicate is interned into the term dictionary when the pair is
(`argfam-id`), so every id here has a term id to be written as.(dense-roots dict)A key-interning KvBackend sharing dict (the columnar trie's token dictionary) so a
term interned by the trie and by a root get the same id.
A key-interning `KvBackend` sharing `dict` (the columnar trie's token dictionary) so a term interned by the trie and by a root get the same id.
(fallback-entries b)The entries the routed families do not claim: the term roster and the slot roster,
whose members are names rather than handles. Both are vocabulary-scaled, which
is what lets a snapshot write them as one nippy blob and load them resident without
the blob tracking the fact count (disk/index_snapshot.clj, "The residency split").
The entries the routed families do **not** claim: the term roster and the slot roster, whose members are *names* rather than handles. Both are **vocabulary-scaled**, which is what lets a snapshot write them as one nippy blob and load them resident without the blob tracking the fact count (`disk/index_snapshot.clj`, "The residency split").
(load-argfam! b preds positions n)Rebuild the scope dictionary from a snapshot's table, ids implied by position — the
same first-writer-wins order vaelii.impl.tokens allocates in, so an id read out of a
packed key names the pair it named when the image was written.
Rebuild the scope dictionary from a snapshot's table, ids implied by position — the same first-writer-wins order `vaelii.impl.tokens` allocates in, so an id read out of a packed key names the pair it named when the image was written.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |