A key-interning KvBackend (vaelii.impl.kv) for the columnar index's non-trie
families — the secondary roots, the rule / exception indexes, and the inverted term
index.
Those families are flat structured-vector-key → handle-set maps, and the columnar
measurement (bench/…/densetrie.clj) found their boxed vector keys ([:argument-root pos term], [:term-index term], …) to be ~150 MB — the majority of the columnar index once the
trie went native. This backend keeps the values as IntPostings (Phase 1's tiered
int[]/Roaring set) but collapses the keys: the term is interned to an int through
the shared trie dictionary (vaelii.impl.tokens) — so a predicate/individual gets
the same id the trie edges use — and the whole key becomes one packed long
(family | pos | term-id) into a single primitive Long2ObjectOpenHashMap. No boxed
vectors, no HAMT nodes, one map.
It stays a full KvBackend so the existing composition (an embedded KvIndexStore
over it) is unchanged: only the recognized index families are int-routed; any other key
— the term roster (term names, not handles), a scalar, a counter, the contract test's
synthetic keys — falls back to a plain in-memory backend (in the columnar store the trie
is native, so no [:trie …] key ever reaches here). Single-writer, like every index;
kv-members / kv-intersect materialize a fresh Clojure set at the boundary — but
kv-intersect builds it at the size of the answer, narrowing through
dense/intersect-postings in whichever representation each posting is in, a mapped run
included. Proven set-equal to MemoryKvBackend on the index families by
dense_roots_oracle_test —
which, like every behavioural check, cannot see a family that falls back when it should
route, since the fallback answers identically; dense_routing_test reads the
representation and covers that.
A key-interning `KvBackend` (`vaelii.impl.kv`) for the columnar index's non-trie families — the secondary roots, the rule / exception indexes, and the inverted term index. Those families are flat `structured-vector-key → handle-set` maps, and the columnar measurement (`bench/…/densetrie.clj`) found their **boxed vector keys** (`[:argument-root pos term]`, `[:term-index term]`, …) to be ~150 MB — the majority of the columnar index once the trie went native. This backend keeps the *values* as `IntPostings` (Phase 1's tiered `int[]`/Roaring set) but collapses the keys: the term is interned to an `int` through the **shared trie dictionary** (`vaelii.impl.tokens`) — so a predicate/individual gets the same id the trie edges use — and the whole key becomes one packed `long` (`family | pos | term-id`) into a single primitive `Long2ObjectOpenHashMap`. No boxed vectors, no HAMT nodes, one map. It stays a full `KvBackend` so the existing composition (an embedded `KvIndexStore` over it) is unchanged: only the recognized index families are int-routed; any other key — the term roster (term *names*, not handles), a scalar, a counter, the contract test's synthetic keys — falls back to a plain in-memory backend (in the columnar store the trie is native, so no `[:trie …]` key ever reaches here). Single-writer, like every index; `kv-members` / `kv-intersect` materialize a fresh Clojure set at the boundary — but `kv-intersect` builds it at the size of the *answer*, narrowing through `dense/intersect-postings` in whichever representation each posting is in, a mapped run included. Proven set-equal to `MemoryKvBackend` on the index families by `dense_roots_oracle_test` — which, like every behavioural check, cannot see a family that falls back when it should route, since the fallback answers identically; `dense_routing_test` reads the representation and covers that.
(dense-roots dict)A key-interning KvBackend sharing dict (the columnar trie's token dictionary) so a
term interned by the trie and by a root get the same id.
A key-interning `KvBackend` sharing `dict` (the columnar trie's token dictionary) so a term interned by the trie and by a root get the same id.
(fallback-entries b)The entries the routed families did not claim — the term roster and anything a future family adds before this backend learns to route it. Vocabulary-scaled and irregularly shaped, so a snapshot writes them as one nippy blob rather than a column.
The entries the routed families did **not** claim — the term roster and anything a future family adds before this backend learns to route it. Vocabulary-scaled and irregularly shaped, so a snapshot writes them as one nippy blob rather than a column.
(load-fallback! b entries)(install-mapped! b keys offsets handles n)Install mapped columns (a LongBuffer and two IntBuffers over a snapshot), replacing
whatever the routed families held.
Install mapped columns (a `LongBuffer` and two `IntBuffer`s over a snapshot), replacing whatever the routed families held.
(mapped? b)Are the routed families reading out of an mmap'd snapshot? True exactly while nothing has been written since one was installed, since a write thaws.
Are the routed families reading out of an mmap'd snapshot? True exactly while nothing has been written since one was installed, since a write thaws.
(snapshot-columns b remap)The routed families as {:keys :offsets :handles} heap arrays, keys sorted and their
term ids taken through remap (an int[] from this dictionary's ids to the durable
ones). The roster key holds no term, so it is passed through unmapped.
The routed families as `{:keys :offsets :handles}` heap arrays, keys sorted and their
term ids taken through `remap` (an `int[]` from this dictionary's ids to the durable
ones). The roster key holds no term, so it is passed through unmapped.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |