vaelii.impl.dense-roots — com.vaelii/vaelii 0.14.0

A key-interning KvBackend (vaelii.impl.kv) for the columnar index's non-trie families — the secondary roots, the rule / exception indexes, and the inverted term index.

Those families are flat structured-vector-key → handle-set maps, and the columnar measurement (bench/…/densetrie.clj) found their boxed vector keys ([:term-index term], [:functor-root pred], …) to be ~150 MB — the majority of the columnar index once the trie went native. This backend keeps the values as IntPostings (Phase 1's tiered int[]/Roaring set) but collapses the keys: the term is interned to an int through the shared trie dictionary (vaelii.impl.tokens) — so a predicate/individual gets the same id the trie edges use — and the whole key becomes one packed long (family | pos | term-id) into a single primitive Long2ObjectOpenHashMap. No boxed vectors, no HAMT nodes, one map.

It stays a full KvBackend so the existing composition (an embedded KvIndexStore over it) is unchanged: only the recognized index families are int-routed; any other key — the slot roster and the term roster (whose members are names, not handles), a scalar, a counter, the contract test's synthetic keys — falls back to a plain in-memory backend (in the columnar store the trie is native, so no [:trie …] key ever reaches here).

Every handle family routes, the predicate-scoped argument roots included: their (pred, pos) scope is interned to a dense id of its own (argfam-id) and rides the pos field, which no other family uses. So the fallback holds only vocabulary-scaled name sets, and the fact-scaled mass is one packed map — or, under a snapshot, one mapped run. Single-writer, like every index; kv-members / kv-intersect materialize a fresh Clojure set at the boundary — but kv-intersect builds it at the size of the answer, narrowing through dense/intersect-postings in whichever representation each posting is in, a mapped run included. Proven set-equal to MemoryKvBackend on the index families by dense_roots_oracle_test — which, like every behavioural check, cannot see a family that falls back when it should route, since the fallback answers identically; dense_routing_test reads the representation and covers that.

Single-threaded, which is narrower than single-writer. The mapped-section fields on DenseRoots are ^:unsynchronized-mutable, so installing or thawing a snapshot publishes through no barrier and a second thread may read this backend mid-install — mapped? true against a mkeys it has not seen, say. The atom- and lock-based backends give an incidental reader beside the writer a consistent view; this one does not. Same trade as vaelii.impl.columnar, whose docstring states it: these fields are read on the hot lookup path, and a volatile read there buys a guarantee the engine's own single writer never needs.

argfam-table clj

(argfam-table b remap)

The scope dictionary as {:preds int[] :positions int[]}, indexed by scope id, with each predicate taken through remap into the durable dictionary's id space — the same int[] the packed keys' term halves are remapped by.

A pair's predicate is interned into the term dictionary when the pair is (argfam-id), so every id here has a term id to be written as.

dense-roots clj

(dense-roots dict)

A key-interning KvBackend sharing dict (the columnar trie's token dictionary) so a term interned by the trie and by a root get the same id.

fallback-entries clj

(fallback-entries b)

The entries the routed families do not claim: the term roster and the slot roster, whose members are names rather than handles. Both are vocabulary-scaled, which is what lets a snapshot write them as one nippy blob and load them resident without the blob tracking the fact count (disk/index_snapshot.clj, "The residency split").

load-argfam! clj

(load-argfam! b preds positions n)

Rebuild the scope dictionary from a snapshot's table, ids implied by position — the same first-writer-wins order vaelii.impl.tokens allocates in, so an id read out of a packed key names the pair it named when the image was written.

load-fallback! clj

(load-fallback! b entries)

PMappedRoots clj protocol

install-mapped! clj

(install-mapped! b keys offsets handles n)

Install mapped columns (a LongBuffer and two IntBuffers over a snapshot), replacing whatever the routed families held.

mapped? clj

(mapped? b)

Are the routed families reading out of an mmap'd snapshot? True exactly while nothing has been written since one was installed, since a write thaws.

sections clj

(sections b)

What this backend holds, by section — {:routed :keys :offsets :handles :argfam :fallback} — for a residency measurement (vaelii.bench.budget). The objects themselves, never a copy: snapshot-columns builds fresh heap arrays to write, and sizing those would size a temporary rather than what a running KB holds.

Which section carries the mass is the whole reading, so each says what it scales with:

• :routed — the mutable map the routed families use before a snapshot is installed and after a write thaws one. Fact-scaled: every handle family is in here.
• :keys / :offsets / :handles — the installed columns. The first two are vocabulary-scaled, :handles is the fact-scaled mass, and all three are buffers over the image, so they belong in a caller's mapped total rather than its heap one.
• :argfam — the (pred, pos) scope dictionary the packed argument keys cite (argfam-id). Vocabulary-scaled, bounded by distinct predicates × their arities.
• :fallback — the backend under everything the routed families do not claim: the term roster and the slot roster, whose members are names rather than handles (fallback-entries). Vocabulary-scaled, which is what lets a snapshot write it as one nippy blob.

The heap/mapped split is the caller's to make from the objects — a buffer says whether it is direct — so this reports the shape and judges nothing.

snapshot-columns clj

(snapshot-columns b remap)

The routed families as {:keys :offsets :handles} heap arrays, keys sorted and their term ids taken through remap (an int[] from this dictionary's ids to the durable ones). The roster key holds no term, so it is passed through unmapped.