A key-interning KvBackend (vaelii.impl.kv) for the columnar index's non-trie
families — the context root, the count tries ending in the context, the opposed members
by context, the exception index, and the inverted term index.
Those families are flat structured-vector-key → handle-set maps, and the columnar
measurement (bench/…/densetrie.clj) found their boxed vector keys
([:term-index term], [:context-root ctx], …) to be ~150 MB — the majority of the
columnar index once the trie went native. This backend keeps the values as
IntPostings (Phase 1's tiered
int[]/Roaring set) but collapses the keys: the term is interned to an int through
the shared trie dictionary (vaelii.impl.tokens) — so a predicate/individual gets
the same id the trie edges use — and the whole key becomes one packed long
(family | pos | term-id) into a single primitive Long2ObjectOpenHashMap. No boxed
vectors, no HAMT nodes, one map.
It stays a full KvBackend so the existing composition (an embedded KvIndexStore
over it) is unchanged: only the recognized index families are int-routed; any other key
— a counter keyed by the vocabulary, a scalar, the contract test's synthetic keys —
falls back to a plain in-memory backend (in the columnar store the trie is native, so no
[:trie …] key ever reaches here).
Every handle family routes, the count tries' leaves included: an argument leaf's
(pred, pos, ctx) scope, and any other count trie leaf's context scope (ctx, 0, ctx),
is interned to a dense id of its own (argfam-id) and rides the pos field, which
the flat families do not use. The tries' child sets and every name set (the term,
slot and membership rosters, the keys a handle installs) route as packed keys too, their
members held as ids in the shared dictionary. So the fallback holds only the
vocabulary's counters, and the fact-scaled mass is one packed map — or, under a
snapshot, one mapped run. Single-writer, like every index;
kv-members / kv-intersect materialize a fresh Clojure set at the boundary — but
kv-intersect builds it at the size of the answer, narrowing through
postings/intersect-postings in whichever representation each posting is in, a mapped run
included. Proven set-equal to MemoryKvBackend on the index families by
dense_roots_oracle_test —
which, like every behavioural check, cannot see a family that falls back when it should
route, since the fallback answers identically; dense_routing_test reads the
representation and covers that.
Single-threaded, which is narrower than single-writer. The mapped-section fields
on DenseRoots are ^:unsynchronized-mutable, so installing or thawing a snapshot
publishes through no barrier and a second thread may read this backend mid-install —
mapped? true against a mkeys it has not seen, say. The atom- and lock-based
backends give an incidental reader beside the writer a consistent view; this one does
not. Same trade as vaelii.impl.columnar, whose docstring states it: these fields are
read on the hot lookup path, and a volatile read there buys a guarantee the engine's own
single writer never needs.
A key-interning `KvBackend` (`vaelii.impl.kv`) for the columnar index's non-trie families — the context root, the count tries ending in the context, the opposed members by context, the exception index, and the inverted term index. Those families are flat `structured-vector-key → handle-set` maps, and the columnar measurement (`bench/…/densetrie.clj`) found their **boxed vector keys** (`[:term-index term]`, `[:context-root ctx]`, …) to be ~150 MB — the majority of the columnar index once the trie went native. This backend keeps the *values* as `IntPostings` (Phase 1's tiered `int[]`/Roaring set) but collapses the keys: the term is interned to an `int` through the **shared trie dictionary** (`vaelii.impl.tokens`) — so a predicate/individual gets the same id the trie edges use — and the whole key becomes one packed `long` (`family | pos | term-id`) into a single primitive `Long2ObjectOpenHashMap`. No boxed vectors, no HAMT nodes, one map. It stays a full `KvBackend` so the existing composition (an embedded `KvIndexStore` over it) is unchanged: only the recognized index families are int-routed; any other key — a counter keyed by the vocabulary, a scalar, the contract test's synthetic keys — falls back to a plain in-memory backend (in the columnar store the trie is native, so no `[:trie …]` key ever reaches here). **Every handle family routes**, the count tries' leaves included: an argument leaf's `(pred, pos, ctx)` scope, and any other count trie leaf's context scope `(ctx, 0, ctx)`, is interned to a dense id of its own (`argfam-id`) and rides the `pos` field, which the flat families do not use. The tries' child sets and every name set (the term, slot and membership rosters, the keys a handle installs) route as packed keys too, their members held as ids in the shared dictionary. So the fallback holds only the vocabulary's counters, and the fact-scaled mass is one packed map — or, under a snapshot, one mapped run. Single-writer, like every index; `kv-members` / `kv-intersect` materialize a fresh Clojure set at the boundary — but `kv-intersect` builds it at the size of the *answer*, narrowing through `postings/intersect-postings` in whichever representation each posting is in, a mapped run included. Proven set-equal to `MemoryKvBackend` on the index families by `dense_roots_oracle_test` — which, like every behavioural check, cannot see a family that falls back when it should route, since the fallback answers identically; `dense_routing_test` reads the representation and covers that. **Single-*threaded*, which is narrower than single-writer.** The mapped-section fields on `DenseRoots` are `^:unsynchronized-mutable`, so installing or thawing a snapshot publishes through no barrier and a second thread may read this backend mid-install — `mapped?` true against a `mkeys` it has not seen, say. The atom- and lock-based backends give an incidental reader beside the writer a consistent view; this one does not. Same trade as `vaelii.impl.columnar`, whose docstring states it: these fields are read on the hot lookup path, and a volatile read there buys a guarantee the engine's own single writer never needs.
(argfam-table b remap)The scope dictionary as {:preds int[] :positions int[] :contexts int[]}, indexed by
scope id, with each predicate and context taken through remap into the durable
dictionary's id space — the same int[] the packed keys' term halves are remapped by.
A node's scope [pred pos] has no context and writes -1 there; a context scope
[ctx 0 ctx] writes its context as the predicate and 0 as the position.
A scope's names are interned into the term dictionary when the scope is
(argfam-id), so every id here has a term id to be written as.
The scope dictionary as `{:preds int[] :positions int[] :contexts int[]}`, indexed by
scope id, with each predicate and context taken through `remap` into the durable
dictionary's id space — the same `int[]` the packed keys' term halves are remapped by.
A node's scope `[pred pos]` has no context and writes -1 there; a context scope
`[ctx 0 ctx]` writes its context as the predicate and 0 as the position.
A scope's names are interned into the term dictionary when the scope is
(`argfam-id`), so every id here has a term id to be written as.(dense-roots dict)A key-interning KvBackend sharing dict (the columnar trie's token dictionary) so a
term interned by the trie and by a root get the same id.
A key-interning `KvBackend` sharing `dict` (the columnar trie's token dictionary) so a term interned by the trie and by a root get the same id.
(fallback-entries b)The entries the routed families do not claim: the counters keyed by a predicate,
functor, rule key or kind, and the one :mint-in node's children. All are
vocabulary-scaled, which is what lets a snapshot write them as one nippy blob and
load them resident without the blob tracking the fact count (disk/index_snapshot.clj,
"The residency split").
The entries the routed families do **not** claim: the counters keyed by a predicate, functor, rule key or kind, and the one `:mint-in` node's children. All are **vocabulary-scaled**, which is what lets a snapshot write them as one nippy blob and load them resident without the blob tracking the fact count (`disk/index_snapshot.clj`, "The residency split").
(load-argfam! b preds positions contexts n)Rebuild the scope dictionary from a snapshot's table, ids implied by position — the
same first-writer-wins order vaelii.impl.tokens allocates in, so an id read out of a
packed key names the scope it named when the image was written.
Rebuild the scope dictionary from a snapshot's table, ids implied by position — the same first-writer-wins order `vaelii.impl.tokens` allocates in, so an id read out of a packed key names the scope it named when the image was written.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |