A durable token dictionary for a disk store: symbol/keyword ↔ int, append-only,
ids assigned in append order and never reused. It is what lets a record frame spell
its sentence as ids rather than names (vaelii.impl.disk.codec), which is 2.6× smaller
than the positional frame it replaces — the vocabulary is written once here instead of
once per frame in all 100M of them.
The ordering that makes it safe. A frame referencing an id the dictionary cannot decode is unreadable data, so the dictionary must never lag the records that cite it. Two rules give that, and neither costs a write:
emit! interns as it
encodes, which happens before the frame is written), so the token log leads the
record log in write order at all times;fsync fsyncs this log first, holding the sentexes kind lock, so nothing is
appended between the two fsyncs. Every record durable after a tick therefore has
its tokens durable too.fsyncing per new token would also give the ordering, and is what this did first —
but it makes a cold load fsync-bound (measured: ~217 records/s, since a new token is
not rare during one). Between ticks the two logs can still skew on a machine crash,
exactly as the record log and its idx can; open-record-store repairs it the same way
it repairs those, by tombstoning a record whose ids the dictionary does not hold.
Only symbols and keywords are interned. Those are bounded by the ontology.
Numbers and strings are not — a KB of measurements would mint a dictionary entry per
distinct value — so codec carries them beside the id stream as literals instead.
Ids are content-keyed and first-writer-wins, so they are stable for the life of the
store; the id value depends on first-encounter order, which nothing above this reads.
Tokens are never deleted (an id must keep decoding), so the log has no dead frames and
needs no compaction; only a whole-store clear-records! empties it.
Writes take the log's lock (there is one writer, and the append must not interleave); the reverse map is an atom holding a vector, so a decode — which runs on every fetch, under a different lock — reads it without one and still sees a safely published entry.
A **durable** token dictionary for a disk store: `symbol/keyword ↔ int`, append-only, ids assigned in append order and never reused. It is what lets a record frame spell its sentence as ids rather than names (`vaelii.impl.disk.codec`), which is 2.6× smaller than the positional frame it replaces — the vocabulary is written once here instead of once per frame in all 100M of them. **The ordering that makes it safe.** A frame referencing an id the dictionary cannot decode is unreadable data, so the dictionary must never lag the records that cite it. Two rules give that, and neither costs a write: - a token is appended **before** the record frame citing it (`emit!` interns as it encodes, which happens before the frame is written), so the token log leads the record log in *write* order at all times; - `fsync` fsyncs this log **first, holding the sentexes kind lock**, so nothing is appended between the two fsyncs. Every record durable after a tick therefore has its tokens durable too. fsyncing *per new token* would also give the ordering, and is what this did first — but it makes a cold load fsync-bound (measured: ~217 records/s, since a new token is not rare during one). Between ticks the two logs can still skew on a machine crash, exactly as the record log and its idx can; `open-record-store` repairs it the same way it repairs those, by tombstoning a record whose ids the dictionary does not hold. **Only symbols and keywords are interned.** Those are bounded by the ontology. Numbers and strings are not — a KB of measurements would mint a dictionary entry per distinct value — so `codec` carries them beside the id stream as literals instead. Ids are **content-keyed and first-writer-wins**, so they are stable for the life of the store; the id *value* depends on first-encounter order, which nothing above this reads. Tokens are never deleted (an id must keep decoding), so the log has no dead frames and needs no compaction; only a whole-store `clear-records!` empties it. Writes take the log's lock (there is one writer, and the append must not interleave); the *reverse* map is an atom holding a vector, so a decode — which runs on every fetch, under a different lock — reads it without one and still sees a safely published entry.
(clear! {:keys [fwd rev lock log]})Empty the dictionary and its log — only for a whole-store wipe, since it invalidates every id any surviving frame holds.
Empty the dictionary and its log — only for a whole-store wipe, since it invalidates every id any surviving frame holds.
(close! {:keys [log lock]})(fsync {:keys [log lock]} fsync?)(intern! {:keys [fwd rev lock log]} tok)The id for tok, allocating (and durably recording) a fresh one if it is new.
The id for `tok`, allocating (and durably recording) a fresh one if it is new.
(open-token-log dir)Open dir/tokens.log and rebuild the in-memory maps from it: frame i holds the
token with id i. A torn trailing frame is truncated exactly as a record log's is —
the token it held was never handed out, because the append is fsynced before the id
is returned.
Open `dir/tokens.log` and rebuild the in-memory maps from it: frame *i* holds the token with id *i*. A torn trailing frame is truncated exactly as a record log's is — the token it held was never handed out, because the append is fsynced before the id is returned.
(token {:keys [rev]} id)The token id id decodes to. An id outside the dictionary means a frame referencing
a token the dictionary never recorded — unreadable data rather than a nil field, so it
throws instead of decoding to something plausible.
The token id `id` decodes to. An id outside the dictionary means a frame referencing a token the dictionary never recorded — unreadable data rather than a nil field, so it throws instead of decoding to something plausible.
(token-count {:keys [rev]})cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |