A key-interning KvBackend (vaelii.impl.kv) for the columnar index's non-trie
families — the context root, the count tries ending in the context, the opposed members
by context, the exception index, and the inverted term index.
Those families are flat structured-vector-key → handle-set maps, and the columnar
measurement (bench/…/densetrie.clj) found their boxed vector keys
([:term-index term], [:context-root ctx], …) to be ~150 MB — the majority of the
columnar index once the trie went native. This backend keeps the values as
IntPostings (Phase 1's tiered
int[]/Roaring set) but collapses the keys: the term is interned to an int through
the shared trie dictionary (vaelii.impl.tokens) — so a predicate/individual gets
the same id the trie edges use — and the whole key becomes one packed long
(family | pos | term-id) into a single primitive Long2ObjectOpenHashMap. No boxed
vectors, no HAMT nodes, one map.
It stays a full KvBackend so the existing composition (an embedded KvIndexStore
over it) is unchanged: only the recognized index families are int-routed; any other key
— the slot roster and the term roster (whose members are names, not handles), a
scalar, a counter, the contract test's synthetic keys — falls back to a plain in-memory
backend (in the columnar store the trie is native, so no [:trie …] key ever reaches
here).
Every handle family routes, the count tries' leaves included: an argument leaf's
(pred, pos, ctx) scope, and any other count trie leaf's context scope (ctx, 0, ctx),
is interned to a dense id of its own (argfam-id) and rides the pos field, which
the flat families do not use. The tries' child sets route as packed keys too, their
members held as context ids. So the fallback holds only vocabulary-scaled name sets
and counters, and the fact-scaled mass is one packed map — or, under a snapshot, one
mapped run. Single-writer, like every index;
kv-members / kv-intersect materialize a fresh Clojure set at the boundary — but
kv-intersect builds it at the size of the answer, narrowing through
postings/intersect-postings in whichever representation each posting is in, a mapped run
included. Proven set-equal to MemoryKvBackend on the index families by
dense_roots_oracle_test —
which, like every behavioural check, cannot see a family that falls back when it should
route, since the fallback answers identically; dense_routing_test reads the
representation and covers that.
Single-threaded, which is narrower than single-writer. The mapped-section fields
on DenseRoots are ^:unsynchronized-mutable, so installing or thawing a snapshot
publishes through no barrier and a second thread may read this backend mid-install —
mapped? true against a mkeys it has not seen, say. The atom- and lock-based
backends give an incidental reader beside the writer a consistent view; this one does
not. Same trade as vaelii.impl.columnar, whose docstring states it: these fields are
read on the hot lookup path, and a volatile read there buys a guarantee the engine's own
single writer never needs.
A key-interning `KvBackend` (`vaelii.impl.kv`) for the columnar index's non-trie families — the context root, the count tries ending in the context, the opposed members by context, the exception index, and the inverted term index. Those families are flat `structured-vector-key → handle-set` maps, and the columnar measurement (`bench/…/densetrie.clj`) found their **boxed vector keys** (`[:term-index term]`, `[:context-root ctx]`, …) to be ~150 MB — the majority of the columnar index once the trie went native. This backend keeps the *values* as `IntPostings` (Phase 1's tiered `int[]`/Roaring set) but collapses the keys: the term is interned to an `int` through the **shared trie dictionary** (`vaelii.impl.tokens`) — so a predicate/individual gets the same id the trie edges use — and the whole key becomes one packed `long` (`family | pos | term-id`) into a single primitive `Long2ObjectOpenHashMap`. No boxed vectors, no HAMT nodes, one map. It stays a full `KvBackend` so the existing composition (an embedded `KvIndexStore` over it) is unchanged: only the recognized index families are int-routed; any other key — the slot roster and the term roster (whose members are *names*, not handles), a scalar, a counter, the contract test's synthetic keys — falls back to a plain in-memory backend (in the columnar store the trie is native, so no `[:trie …]` key ever reaches here). **Every handle family routes**, the count tries' leaves included: an argument leaf's `(pred, pos, ctx)` scope, and any other count trie leaf's context scope `(ctx, 0, ctx)`, is interned to a dense id of its own (`argfam-id`) and rides the `pos` field, which the flat families do not use. The tries' child sets route as packed keys too, their members held as context ids. So the fallback holds only vocabulary-scaled name sets and counters, and the fact-scaled mass is one packed map — or, under a snapshot, one mapped run. Single-writer, like every index; `kv-members` / `kv-intersect` materialize a fresh Clojure set at the boundary — but `kv-intersect` builds it at the size of the *answer*, narrowing through `postings/intersect-postings` in whichever representation each posting is in, a mapped run included. Proven set-equal to `MemoryKvBackend` on the index families by `dense_roots_oracle_test` — which, like every behavioural check, cannot see a family that falls back when it should route, since the fallback answers identically; `dense_routing_test` reads the representation and covers that. **Single-*threaded*, which is narrower than single-writer.** The mapped-section fields on `DenseRoots` are `^:unsynchronized-mutable`, so installing or thawing a snapshot publishes through no barrier and a second thread may read this backend mid-install — `mapped?` true against a `mkeys` it has not seen, say. The atom- and lock-based backends give an incidental reader beside the writer a consistent view; this one does not. Same trade as `vaelii.impl.columnar`, whose docstring states it: these fields are read on the hot lookup path, and a volatile read there buys a guarantee the engine's own single writer never needs.
(argfam-table b remap)The scope dictionary as {:preds int[] :positions int[] :contexts int[]}, indexed by
scope id, with each predicate and context taken through remap into the durable
dictionary's id space — the same int[] the packed keys' term halves are remapped by.
A node's scope [pred pos] has no context and writes -1 there; a context scope
[ctx 0 ctx] writes its context as the predicate and 0 as the position.
A scope's names are interned into the term dictionary when the scope is
(argfam-id), so every id here has a term id to be written as.
The scope dictionary as `{:preds int[] :positions int[] :contexts int[]}`, indexed by
scope id, with each predicate and context taken through `remap` into the durable
dictionary's id space — the same `int[]` the packed keys' term halves are remapped by.
A node's scope `[pred pos]` has no context and writes -1 there; a context scope
`[ctx 0 ctx]` writes its context as the predicate and 0 as the position.
A scope's names are interned into the term dictionary when the scope is
(`argfam-id`), so every id here has a term id to be written as.(dense-roots dict)A key-interning KvBackend sharing dict (the columnar trie's token dictionary) so a
term interned by the trie and by a root get the same id.
A key-interning `KvBackend` sharing `dict` (the columnar trie's token dictionary) so a term interned by the trie and by a root get the same id.
(fallback-entries b)The entries the routed families do not claim: the term roster and the two slot
rosters, whose members are names rather than handles, and the predicate-extent and
rule-index node counts, one per predicate or rule key. All are vocabulary-scaled, which
is what lets a snapshot write them as one nippy blob and load them resident without
the blob tracking the fact count (disk/index_snapshot.clj, "The residency split").
The entries the routed families do **not** claim: the term roster and the two slot rosters, whose members are *names* rather than handles, and the predicate-extent and rule-index node counts, one per predicate or rule key. All are **vocabulary-scaled**, which is what lets a snapshot write them as one nippy blob and load them resident without the blob tracking the fact count (`disk/index_snapshot.clj`, "The residency split").
(load-argfam! b preds positions contexts n)Rebuild the scope dictionary from a snapshot's table, ids implied by position — the
same first-writer-wins order vaelii.impl.tokens allocates in, so an id read out of a
packed key names the scope it named when the image was written.
Rebuild the scope dictionary from a snapshot's table, ids implied by position — the same first-writer-wins order `vaelii.impl.tokens` allocates in, so an id read out of a packed key names the scope it named when the image was written.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |