A bidirectional token dictionary — path-token ↔ int — the first new durable
ground truth the dense (:memory-columnar) index rests on.
The columnar trie (vaelii.impl.columnar) labels its edges with int tokens rather
than boxed structured values, so it needs a stable token → int map and an int → token inverse to decode a node's child labels back for children. A token is
whatever a trie path level can be (sentex/path): a symbol (predicate / individual /
type / context), a number, a keyword (:false / :rule), nil (a rule's assumption
/ constraint slot), a [::subterm k] arity marker, or a whole literal list (a
:false body, a rule's antecedent vector). They arrive already canonical from
sentex/path (the sentence went through sentex/canon), so the dictionary interns
them as-is — re-canonicalizing would turn a marker vector into a list and break
sentex/subterm-mark?, exactly as the current KvIndexStore relies on raw path
tokens as keys.
Content-keyed, first-writer-wins. An id is allocated the first time a token is
seen and never changes; ids count up from 0 (array-friendly). The id value depends
on first-encounter order, but the index's correctness does not — an id is an opaque
edge label the inverse map decodes — so a rebuild from the records (reindex / recover)
that re-interns in a different order yields an equal index (identical lookups), which
is the order-independence that matters. A durable columnar index that persisted its
int edges would instead load this dictionary before reading them; that is the seam
Phase 2's durable variant uses.
Keyed by Clojure equality, not Java's — see Key below. That is not a refinement;
it is what makes this dictionary answer the same questions the flat map it replaces
answers.
Single-writer, like the index it serves: the maps are mutated in place under the one-writer contract, no atom.
A bidirectional **token dictionary** — `path-token ↔ int` — the first new durable ground truth the dense (`:memory-columnar`) index rests on. The columnar trie (`vaelii.impl.columnar`) labels its edges with `int` tokens rather than boxed structured values, so it needs a stable `token → int` map and an `int → token` inverse to decode a node's child labels back for `children`. A *token* is whatever a trie path level can be (`sentex/path`): a symbol (predicate / individual / type / context), a number, a keyword (`:false` / `:rule`), `nil` (a rule's assumption / constraint slot), a `[::subterm k]` arity marker, or a whole literal list (a `:false` body, a rule's antecedent vector). They arrive already canonical from `sentex/path` (the sentence went through `sentex/canon`), so the dictionary interns them **as-is** — re-canonicalizing would turn a marker vector into a list and break `sentex/subterm-mark?`, exactly as the current `KvIndexStore` relies on raw path tokens as keys. **Content-keyed, first-writer-wins.** An id is allocated the first time a token is seen and never changes; ids count up from 0 (array-friendly). The id *value* depends on first-encounter order, but the index's correctness does not — an id is an opaque edge label the inverse map decodes — so a rebuild from the records (`reindex` / `recover`) that re-interns in a different order yields an equal index (identical lookups), which is the order-independence that matters. A durable columnar index that persisted its `int` edges would instead load this dictionary before reading them; that is the seam Phase 2's durable variant uses. **Keyed by Clojure equality, not Java's** — see `Key` below. That is not a refinement; it is what makes this dictionary answer the same questions the flat map it replaces answers. **Single-writer**, like the index it serves: the maps are mutated in place under the one-writer contract, no atom.
(clear-tokens! d)Drop every mapping (the whole-dictionary wipe clear-index! needs).
Drop every mapping (the whole-dictionary wipe `clear-index!` needs).
(id-token d id)The token an id decodes to (nil is a real token).
The token an id decodes to (`nil` is a real token).
(intern-token! d tok)The id for tok, allocating a fresh one (first-writer-wins) if absent.
The id for `tok`, allocating a fresh one (first-writer-wins) if absent.
(token-count d)How many distinct tokens are interned.
How many distinct tokens are interned.
(token-id d tok)The existing id for tok, or -1 — no allocation (the absent-path fast exit).
The existing id for `tok`, or -1 — no allocation (the absent-path fast exit).
(token-dict)cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |