A key-interning KvBackend (vaelii.impl.kv) for the columnar index's non-trie
families — the secondary roots, the rule / exception indexes, and the inverted term
index.
Those families are flat structured-vector-key → handle-set maps, and the columnar
measurement (bench/…/densetrie.clj) found their boxed vector keys
([:term-index term], [:functor-root pred], …) to be ~150 MB — the majority of the
columnar index once the trie went native. This backend keeps the values as
IntPostings (Phase 1's tiered
int[]/Roaring set) but collapses the keys: the term is interned to an int through
the shared trie dictionary (vaelii.impl.tokens) — so a predicate/individual gets
the same id the trie edges use — and the whole key becomes one packed long
(family | pos | term-id) into a single primitive Long2ObjectOpenHashMap. No boxed
vectors, no HAMT nodes, one map.
It stays a full KvBackend so the existing composition (an embedded KvIndexStore
over it) is unchanged: only the recognized index families are int-routed; any other key
— the argument roots and slot roster (see route for why neither packs), the term
roster (term names, not handles), a scalar, a counter, the contract test's synthetic
keys — falls back to a plain in-memory backend (in the columnar store the trie is
native, so no [:trie …] key ever reaches here). Single-writer, like every index;
kv-members / kv-intersect materialize a fresh Clojure set at the boundary — but
kv-intersect builds it at the size of the answer, narrowing through
dense/intersect-postings in whichever representation each posting is in, a mapped run
included. Proven set-equal to MemoryKvBackend on the index families by
dense_roots_oracle_test —
which, like every behavioural check, cannot see a family that falls back when it should
route, since the fallback answers identically; dense_routing_test reads the
representation and covers that.
Single-threaded, which is narrower than single-writer. The mapped-section fields
on DenseRoots are ^:unsynchronized-mutable, so installing or thawing a snapshot
publishes through no barrier and a second thread may read this backend mid-install —
mapped? true against a mkeys it has not seen, say. The atom- and lock-based
backends give an incidental reader beside the writer a consistent view; this one does
not. Same trade as vaelii.impl.columnar, whose docstring states it: these fields are
read on the hot lookup path, and a volatile read there buys a guarantee the engine's own
single writer never needs.
A key-interning `KvBackend` (`vaelii.impl.kv`) for the columnar index's non-trie families — the secondary roots, the rule / exception indexes, and the inverted term index. Those families are flat `structured-vector-key → handle-set` maps, and the columnar measurement (`bench/…/densetrie.clj`) found their **boxed vector keys** (`[:term-index term]`, `[:functor-root pred]`, …) to be ~150 MB — the majority of the columnar index once the trie went native. This backend keeps the *values* as `IntPostings` (Phase 1's tiered `int[]`/Roaring set) but collapses the keys: the term is interned to an `int` through the **shared trie dictionary** (`vaelii.impl.tokens`) — so a predicate/individual gets the same id the trie edges use — and the whole key becomes one packed `long` (`family | pos | term-id`) into a single primitive `Long2ObjectOpenHashMap`. No boxed vectors, no HAMT nodes, one map. It stays a full `KvBackend` so the existing composition (an embedded `KvIndexStore` over it) is unchanged: only the recognized index families are int-routed; any other key — the argument roots and slot roster (see `route` for why neither packs), the term roster (term *names*, not handles), a scalar, a counter, the contract test's synthetic keys — falls back to a plain in-memory backend (in the columnar store the trie is native, so no `[:trie …]` key ever reaches here). Single-writer, like every index; `kv-members` / `kv-intersect` materialize a fresh Clojure set at the boundary — but `kv-intersect` builds it at the size of the *answer*, narrowing through `dense/intersect-postings` in whichever representation each posting is in, a mapped run included. Proven set-equal to `MemoryKvBackend` on the index families by `dense_roots_oracle_test` — which, like every behavioural check, cannot see a family that falls back when it should route, since the fallback answers identically; `dense_routing_test` reads the representation and covers that. **Single-*threaded*, which is narrower than single-writer.** The mapped-section fields on `DenseRoots` are `^:unsynchronized-mutable`, so installing or thawing a snapshot publishes through no barrier and a second thread may read this backend mid-install — `mapped?` true against a `mkeys` it has not seen, say. The atom- and lock-based backends give an incidental reader beside the writer a consistent view; this one does not. Same trade as `vaelii.impl.columnar`, whose docstring states it: these fields are read on the hot lookup path, and a volatile read there buys a guarantee the engine's own single writer never needs.
(dense-roots dict)A key-interning KvBackend sharing dict (the columnar trie's token dictionary) so a
term interned by the trie and by a root get the same id.
A key-interning `KvBackend` sharing `dict` (the columnar trie's token dictionary) so a term interned by the trie and by a root get the same id.
(fallback-entries b)The entries the routed families do not claim. That is the term roster and the
slot roster (names, not handles) — but also the predicate-scoped argument roots,
which are fact-scaled: a posting per [pred pos term] triple the stored facts
exhibit. A snapshot writes all of it as one nippy blob rather than a column, and
loads it resident, so the arg-root mass sits outside the mapped-run residency split
(disk/index_snapshot.clj, "The residency split").
The entries the routed families do **not** claim. That is the term roster and the slot roster (names, not handles) — but also the predicate-scoped argument roots, which are **fact-scaled**: a posting per `[pred pos term]` triple the stored facts exhibit. A snapshot writes all of it as one nippy blob rather than a column, and loads it resident, so the arg-root mass sits outside the mapped-run residency split (`disk/index_snapshot.clj`, "The residency split").
(install-mapped! b keys offsets handles n)Install mapped columns (a LongBuffer and two IntBuffers over a snapshot), replacing
whatever the routed families held.
Install mapped columns (a `LongBuffer` and two `IntBuffer`s over a snapshot), replacing whatever the routed families held.
(mapped? b)Are the routed families reading out of an mmap'd snapshot? True exactly while nothing has been written since one was installed, since a write thaws.
Are the routed families reading out of an mmap'd snapshot? True exactly while nothing has been written since one was installed, since a write thaws.
(snapshot-columns b remap)The routed families as {:keys :offsets :handles} heap arrays, keys sorted and their
term ids taken through remap (an int[] from this dictionary's ids to the durable
ones). The roster key holds no term, so it is passed through unmapped.
The routed families as `{:keys :offsets :handles}` heap arrays, keys sorted and their
term ids taken through `remap` (an `int[]` from this dictionary's ids to the durable
ones). The roster key holds no term, so it is passed through unmapped.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |