A durable token dictionary for a disk store: symbol/keyword ↔ int, append-only,
ids assigned in append order and never reused. It is what lets a record frame spell
its sentence as ids rather than names (vaelii.impl.disk.codec), which is 2.6× smaller
than the positional frame it replaces — the vocabulary is written once here instead of
once per frame in all 100M of them.
The ordering that makes it safe. A frame referencing an id the dictionary cannot decode is unreadable data, so the dictionary must never lag the records that cite it. Two rules give that, and neither costs a write:
emit! interns as it
encodes, which happens before the frame is written), so the token log leads the
record log in write order at all times;fsync fsyncs this log first, holding the sentexes kind lock, so nothing is
appended between the two fsyncs. Every record durable after a tick therefore has
its tokens durable too.fsyncing per new token would also give the ordering, and is what this did first —
but it makes a cold load fsync-bound (measured: ~217 records/s, since a new token is
not rare during one). Between ticks the two logs can still skew on a machine crash,
exactly as the record log and its idx can; open-record-store repairs it the same way
it repairs those, by tombstoning a record whose ids the dictionary does not hold.
Only symbols and keywords are interned. Those are bounded by the ontology.
Numbers and strings are not — a KB of measurements would mint a dictionary entry per
distinct value — so codec carries them beside the id stream as literals instead.
Ids are content-keyed and first-writer-wins, so they are stable for the life of the
store; the id value depends on first-encounter order, which nothing above this reads.
Tokens are never deleted (an id must keep decoding), so the log has no dead frames and
needs no compaction; only a whole-store clear-records! empties it.
Writes take the log's lock (there is one writer, and the append must not interleave); the reverse map is an atom holding a vector, so a decode — which runs on every fetch, under a different lock — reads it without one and still sees a safely published entry.
A **durable** token dictionary for a disk store: `symbol/keyword ↔ int`, append-only, ids assigned in append order and never reused. It is what lets a record frame spell its sentence as ids rather than names (`vaelii.impl.disk.codec`), which is 2.6× smaller than the positional frame it replaces — the vocabulary is written once here instead of once per frame in all 100M of them. **The ordering that makes it safe.** A frame referencing an id the dictionary cannot decode is unreadable data, so the dictionary must never lag the records that cite it. Two rules give that, and neither costs a write: - a token is appended **before** the record frame citing it (`emit!` interns as it encodes, which happens before the frame is written), so the token log leads the record log in *write* order at all times; - `fsync` fsyncs this log **first, holding the sentexes kind lock**, so nothing is appended between the two fsyncs. Every record durable after a tick therefore has its tokens durable too. fsyncing *per new token* would also give the ordering, and is what this did first — but it makes a cold load fsync-bound (measured: ~217 records/s, since a new token is not rare during one). Between ticks the two logs can still skew on a machine crash, exactly as the record log and its idx can; `open-record-store` repairs it the same way it repairs those, by tombstoning a record whose ids the dictionary does not hold. **Only symbols and keywords are interned.** Those are bounded by the ontology. Numbers and strings are not — a KB of measurements would mint a dictionary entry per distinct value — so `codec` carries them beside the id stream as literals instead. Ids are **content-keyed and first-writer-wins**, so they are stable for the life of the store; the id *value* depends on first-encounter order, which nothing above this reads. Tokens are never deleted (an id must keep decoding), so the log has no dead frames and needs no compaction; only a whole-store `clear-records!` empties it. Writes take the log's lock (there is one writer, and the append must not interleave); the *reverse* map is an atom holding a vector, so a decode — which runs on every fetch, under a different lock — reads it without one and still sees a safely published entry.
(clear! {:keys [fwd rev lock log]})Empty the dictionary and its log — only for a whole-store wipe, since it invalidates every id any surviving frame holds.
Empty the dictionary and its log — only for a whole-store wipe, since it invalidates every id any surviving frame holds.
(duplicate-count {:keys [fwd rev]})How many of the log's frames hold a token some earlier frame already holds — Java-unequal
but Clojure-equal, an integral pair such as 2 and (int 2).
Zero for a log this build wrote: intern! looks the token up under Key, so the pair
never gets a second frame. A log written before the forward map was keyed that way can
hold one, and the arithmetic is exact rather than a scan — the reverse vector is one
entry per frame and the forward map one per distinct token, so their difference is
the number of frames that collapse.
It is the difference between a dictionary that reloads whole and one that does not, and
so between an index snapshot that maps and one that is rebuilt on every open forever:
index-snapshot/load-dictionary! re-interns the log in id order, and a collapsing pair
shifts every id after it — which is what the mapped edges cite.
How many of the log's frames hold a token some earlier frame already holds — Java-unequal but Clojure-equal, an integral pair such as `2` and `(int 2)`. Zero for a log this build wrote: `intern!` looks the token up under `Key`, so the pair never gets a second frame. A log written before the forward map was keyed that way can hold one, and the arithmetic is exact rather than a scan — the reverse vector is one entry per frame and the forward map one per *distinct* token, so their difference is the number of frames that collapse. It is the difference between a dictionary that reloads whole and one that does not, and so between an index snapshot that maps and one that is rebuilt on every open forever: `index-snapshot/load-dictionary!` re-interns the log in id order, and a collapsing pair shifts every id after it — which is what the mapped edges cite.
(intern! {:keys [fwd rev lock log]} tok)The id for tok, allocating (and durably recording) a fresh one if it is new.
The id for `tok`, allocating (and durably recording) a fresh one if it is new.
(open-token-log dir)Open dir/tokens.log and rebuild the in-memory maps from it: frame i holds the
token with id i. A torn trailing frame is truncated exactly as a record log's is —
the token it held was never handed out, because the append is fsynced before the id
is returned.
Open `dir/tokens.log` and rebuild the in-memory maps from it: frame *i* holds the token with id *i*. A torn trailing frame is truncated exactly as a record log's is — the token it held was never handed out, because the append is fsynced before the id is returned.
(repair-duplicates! {:keys [fwd rev lock log]})Rewrite the log with each Clojure-equal token once, keeping first-encounter order, and return the new entry count. A no-op returning the current count when there is nothing to repair.
The ids move, which is the whole reason this is a separate call rather than
something open-token-log does for itself. Every id past the first duplicate shifts
down, so this is only ever legal for a caller whose citations are derived state it is
about to rebuild — the index snapshot's dictionary, whose ids are cited by the mapped
trie edges and nothing else, and only once that image is already condemned. The record
store's log is the opposite case: its ids are cited by every frame it holds, and there
is no pair to repair in it anyway (only symbols and keywords are interned there, and
those are Java-equal exactly when they are Clojure-equal).
Durable before it returns, because the caller's next act is to trust the new numbering.
Rewrite the log with each Clojure-equal token once, keeping first-encounter order, and return the new entry count. A no-op returning the current count when there is nothing to repair. **The ids move**, which is the whole reason this is a separate call rather than something `open-token-log` does for itself. Every id past the first duplicate shifts down, so this is only ever legal for a caller whose citations are derived state it is about to rebuild — the index snapshot's dictionary, whose ids are cited by the mapped trie edges and nothing else, and only once that image is already condemned. The record store's log is the opposite case: its ids are cited by every frame it holds, and there is no pair to repair in it anyway (only symbols and keywords are interned there, and those are Java-equal exactly when they are Clojure-equal). Durable before it returns, because the caller's next act is to trust the new numbering.
(token {:keys [rev]} id)The token id id decodes to. An id outside the dictionary means a frame referencing
a token the dictionary never recorded — unreadable data rather than a nil field, so it
throws instead of decoding to something plausible.
The token id `id` decodes to. An id outside the dictionary means a frame referencing a token the dictionary never recorded — unreadable data rather than a nil field, so it throws instead of decoding to something plausible.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |