How a record is shaped on its way into a log frame, and back.
nippy freezes a Clojure record by writing its type tag and every field name into
the frame — so a store of 100M sentexes writes vaelii.impl.sentex.AtomicSentex and
:sentence :context :id :truth :strength 100M times. Measured on the real corpus,
that scaffolding is 56% of the store (87 of 155 B/record) and it says nothing a
frame needs to carry: the field layout is a property of the code, identical in every
frame.
So a frame holds the fields positionally — a plain vector, the shape known here —
which is 1.85× smaller and needs no dictionary, no id allocation, and no new durable
ground truth (lein bench-records). The codec is per kind, because each kind has
one known set of shapes: a sentex frame is tagged atomic/rule, a justification frame
is a bare vector (there is only one shape), and provenance is an open application map
that passes through untouched.
Reading is backward-compatible in both directions. decode dispatches on the
thawed frame: a vector is positional, anything else is returned as it thawed. So a
store written before this codec reads exactly as it did (its frames are records), and
a plain map handed to put-sentex — which the tests do, and which is not an AtomicSentex
— round-trips as the map it is.
Decoding also interns the symbols it rebuilds (sentex/intern-deep), so a record
paged off disk shares the one vocabulary object per name with every other record and
with the in-memory store, rather than minting its own copy per fetch. That matters
most for the records the hot cache retains.
Tokenized bodies are the second, opt-in step (vaelii.disk.tokens): the positional
frame still spells its sentence out in full, and the vocabulary — the same few hundred
thousand predicate and individual names — is written into every one of the frames. A
tokenized frame replaces the s-expression fields with a varint byte string of ids from
the durable dictionary (vaelii.impl.disk.tokens), 2.6× smaller again. It is a
fourth and fifth frame tag, not a format change: a store can hold plain and tokenized
frames side by side, so enabling it costs no rewrite and disabling it leaves what is
already written readable.
How a record is shaped on its way into a log frame, and back. nippy freezes a Clojure record by writing its **type tag and every field name** into the frame — so a store of 100M sentexes writes `vaelii.impl.sentex.AtomicSentex` and `:sentence :context :id :truth :strength` 100M times. Measured on the real corpus, that scaffolding is **56% of the store** (87 of 155 B/record) and it says nothing a frame needs to carry: the field layout is a property of the code, identical in every frame. So a frame holds the fields **positionally** — a plain vector, the shape known here — which is 1.85× smaller and needs no dictionary, no id allocation, and no new durable ground truth (`lein bench-records`). The codec is per *kind*, because each kind has one known set of shapes: a sentex frame is tagged `atomic`/`rule`, a justification frame is a bare vector (there is only one shape), and provenance is an open application map that passes through untouched. **Reading is backward-compatible in both directions.** `decode` dispatches on the thawed frame: a vector is positional, anything else is returned as it thawed. So a store written before this codec reads exactly as it did (its frames are records), and a plain map handed to `put-sentex` — which the tests do, and which is not an `AtomicSentex` — round-trips as the map it is. Decoding also **interns** the symbols it rebuilds (`sentex/intern-deep`), so a record paged off disk shares the one vocabulary object per name with every other record and with the in-memory store, rather than minting its own copy per fetch. That matters most for the records the hot cache retains. **Tokenized bodies** are the second, opt-in step (`vaelii.disk.tokens`): the positional frame still spells its sentence out in full, and the vocabulary — the same few hundred thousand predicate and individual names — is written into every one of the frames. A tokenized frame replaces the s-expression fields with a varint byte string of ids from the durable dictionary (`vaelii.impl.disk.tokens`), 2.6× smaller again. It is a *fourth and fifth frame tag*, not a format change: a store can hold plain and tokenized frames side by side, so enabling it costs no rewrite and disabling it leaves what is already written readable.
(by-kind dict tokenize?)kind-name -> {:enc :dec} for a store. dict is its durable token dictionary, which
a store that has ever written a tokenized frame needs whether or not it still writes
them — tokenize? governs only the encoder. Provenance is an open application map
with no fixed shape, so it is stored as it comes.
`kind-name -> {:enc :dec}` for a store. `dict` is its durable token dictionary, which
a store that has ever written a tokenized frame needs **whether or not it still writes
them** — `tokenize?` governs only the encoder. Provenance is an open application map
with no fixed shape, so it is stored as it comes.(decode-justification v)(decode-sentex v)The inverse of encode-sentex: a positional vector back to its record (symbols
interned), anything else as it thawed.
The inverse of `encode-sentex`: a positional vector back to its record (symbols interned), anything else as it thawed.
(encode-justification d)A Justification as a positional vector; anything else unchanged.
A `Justification` as a positional vector; anything else unchanged.
(encode-sentex sx)An AtomicSentex or RuleSentex as a positional vector; anything else unchanged.
An `AtomicSentex` or `RuleSentex` as a positional vector; anything else unchanged.
(tokenized-frame? v)Whether a thawed frame spells its body as dictionary ids — what a store must have a dictionary to read.
Whether a thawed frame spells its body as dictionary ids — what a store must have a dictionary to read.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |