Write a KB out as a portable export dump — a directory holding the record store's three streams, in a format that survives a backend change, an index-representation change, and a record class rename.
The :disk store directory looks like an archive and is not one: it holds frozen
records, so renaming a record class makes every frame in it thaw to a
{:nippy/unthawable …} placeholder — silently, because a placeholder is a perfectly
good map. Hence the one non-negotiable rule of this format:
A frame never carries a class name.
A frame is the record's field map — (into {} record), a plain map — so a rename
changes nothing a dump holds, and vaelii.impl.io.import's field-map already
accepts one.
What a dump holds is what the record store holds, because everything else the KB has
is derived from it: sentexes, justifications, and per-handle provenance. A premise
needs no stream of its own — a premise is a sentex whose :strength is non-nil, a
field on the record in both backends, so the mark rides along with it. The index is
a cache (reindex), the taxonomy and the TMS labels are recomputed (recover).
<dump>/
meta.edn the marker, the schema, and the counts
sentexes.nippy.stream one frame per sentex
justifications.nippy.stream one frame per justification
provenance.nippy.stream [handle provenance-map] per frame (omitted when none)
index/entries.nippy.stream [key value] per frame — only in :records+index
index/index.edn the layout version + the records fingerprint
The index is optional and always discardable. :variant :records+index writes it
as well, in the [structured-key value] projection every index backend shares
(p/index-entries), so an index written by one backend loads into another. It is a
cache: a reader replays it only when it can prove the entries were derived from
exactly the records beside them (vaelii.impl.io.fingerprint), and rebuilds otherwise.
That is what makes writing it safe — an index that does not match its records is worse
than no index, because every lookup then answers confidently and short.
Framing is the chunked layout vaelii.impl.io.frames writes and reads: a
run of [int32 length][compressed chunk], each chunk an independent compression
window over back-to-back nippy frames. Constant memory on both sides — the writer
holds one chunk, never the corpus. meta.edn states the framing rather than
implying it through a version number, because the engine's dumps number their own
format and a reader must never have to guess.
meta.edn is written last, which makes it double as the completion marker:
vaelii.impl.catalog/classify keys on it, so a half-written or cancelled export is
not offered as loadable. That is the one ordering constraint in the whole format.
Export from a KB nobody is writing: the walk fetches record by record, and the single-writer contract offers no snapshot to walk instead.
Write a KB out as a portable **export dump** — a directory holding the record
store's three streams, in a format that survives a backend change, an
index-representation change, and a record class rename.
The `:disk` store directory looks like an archive and is not one: it holds frozen
*records*, so renaming a record class makes every frame in it thaw to a
`{:nippy/unthawable …}` placeholder — silently, because a placeholder is a perfectly
good map. Hence the one non-negotiable rule of this format:
> **A frame never carries a class name.**
A frame is the record's **field map** — `(into {} record)`, a plain map — so a rename
changes nothing a dump holds, and `vaelii.impl.io.import`'s `field-map` already
accepts one.
What a dump holds is what the record store holds, because everything else the KB has
is derived from it: sentexes, justifications, and per-handle provenance. A premise
needs no stream of its own — a premise *is* a sentex whose `:strength` is non-nil, a
field on the record in both backends, so the mark rides along with it. The index is
a cache (`reindex`), the taxonomy and the TMS labels are recomputed (`recover`).
<dump>/
meta.edn the marker, the schema, and the counts
sentexes.nippy.stream one frame per sentex
justifications.nippy.stream one frame per justification
provenance.nippy.stream [handle provenance-map] per frame (omitted when none)
index/entries.nippy.stream [key value] per frame — only in :records+index
index/index.edn the layout version + the records fingerprint
**The index is optional and always discardable.** `:variant :records+index` writes it
as well, in the `[structured-key value]` projection every index backend shares
(`p/index-entries`), so an index written by one backend loads into another. It is a
*cache*: a reader replays it only when it can prove the entries were derived from
exactly the records beside them (`vaelii.impl.io.fingerprint`), and rebuilds otherwise.
That is what makes writing it safe — an index that does not match its records is worse
than no index, because every lookup then answers confidently and short.
**Framing** is the chunked layout `vaelii.impl.io.frames` writes and reads: a
run of `[int32 length][compressed chunk]`, each chunk an independent compression
window over back-to-back nippy frames. Constant memory on both sides — the writer
holds one chunk, never the corpus. `meta.edn` *states* the framing rather than
implying it through a version number, because the engine's dumps number their own
format and a reader must never have to guess.
**`meta.edn` is written last**, which makes it double as the completion marker:
`vaelii.impl.catalog/classify` keys on it, so a half-written or cancelled export is
not offered as loadable. That is the one ordering constraint in the whole format.
Export from a KB nobody is writing: the walk fetches record by record, and the
single-writer contract offers no snapshot to walk instead.What makes a dumped index and a dumped set of records provably the same KB.
The index is derived from the records, so an index dump is only valid against the records it was derived from. Nothing downstream can notice when it is not: a stale trie node simply reports fewer handles and a query returns fewer results — no exception, no warning, just a KB that quietly knows less. So the writer records a fingerprint over the records and the reader recomputes it while storing them; a mismatch discards the index and rebuilds.
A sum, not a chain. The digest is the sum (mod 2⁶⁴) of a per-record hash, so it is commutative — the reader accumulates it in whatever order the frames arrive, which is not the order the writer walked — and additive rather than xor, so two identical records cannot cancel each other out.
What is hashed is what the index is a function of: the handle, :sentence,
:context, :truth, and a rule's :antecedent / :consequent. sentex/path,
kv/root-keys, kv/sentex-terms and the rule index read exactly those, and the handle
because a posting is a set of handles — the same content at a different handle makes
every posting naming it wrong. Deliberately not hashed: :strength, :varmap,
:direction, :defeasible. None of them changes a single index entry, so a dump
differing in one of them still has a valid index, and hashing them would make the cache
go unused for a reason that is not a reason. (:strength also arrives after the
storing pass — a premise mark is applied once every record is down — so hashing it
would cost the second pass over the records this exists to avoid.)
The hash is FNV-1a over the fields' own hash values: cheap, deterministic across
runs, and strong enough for the question being asked, which is whether two sets of
records are the same — not whether an adversary can forge one. A dump is not a
trust boundary; if it were, this would be a real digest and signed.
Two granularities, one rule. accumulator (over record-hash) folds in what a
record says, and needs the record — the export rides it on the sentex walk it is
already making, and the import rides it on the frames it is already decoding.
slot-accumulator (over slot-hash) folds in where each record is:
(id, offset, length) off the idx, no frame decoded and no record fetched. The mapped index snapshot (vaelii.impl.disk.index-snapshot) is
checked with that one, because an image exists to make an open cost bytes read rather
than records read, and a content digest would put every record back on the open path.
It is the coarser hash in one direction and the finer one in the other: it cannot see
a record rewritten to the same bytes at the same offset (nothing can produce that),
and it does see a rewrite the index does not care about — a premise mark, or a
compaction moving every frame — so a snapshot is discarded and rebuilt after one.
Both directions are safe, because a discard is always legal for derived state.
What makes a dumped index and a dumped set of records provably the same KB. The index is derived from the records, so an index dump is only valid *against the records it was derived from*. Nothing downstream can notice when it is not: a stale trie node simply reports fewer handles and a query returns fewer results — no exception, no warning, just a KB that quietly knows less. So the writer records a fingerprint over the records and the reader recomputes it while storing them; a mismatch discards the index and rebuilds. **A sum, not a chain.** The digest is the sum (mod 2⁶⁴) of a per-record hash, so it is commutative — the reader accumulates it in whatever order the frames arrive, which is not the order the writer walked — and *additive* rather than xor, so two identical records cannot cancel each other out. **What is hashed is what the index is a function of**: the handle, `:sentence`, `:context`, `:truth`, and a rule's `:antecedent` / `:consequent`. `sentex/path`, `kv/root-keys`, `kv/sentex-terms` and the rule index read exactly those, and the handle because a posting *is* a set of handles — the same content at a different handle makes every posting naming it wrong. Deliberately **not** hashed: `:strength`, `:varmap`, `:direction`, `:defeasible`. None of them changes a single index entry, so a dump differing in one of them still has a valid index, and hashing them would make the cache go unused for a reason that is not a reason. (`:strength` also arrives after the storing pass — a premise mark is applied once every record is down — so hashing it would cost the second pass over the records this exists to avoid.) The hash is FNV-1a over the fields' own `hash` values: cheap, deterministic across runs, and strong enough for the question being asked, which is whether two sets of records are *the same* — not whether an adversary can forge one. A dump is not a trust boundary; if it were, this would be a real digest and signed. **Two granularities, one rule.** `accumulator` (over `record-hash`) folds in what a record *says*, and needs the record — the export rides it on the sentex walk it is already making, and the import rides it on the frames it is already decoding. `slot-accumulator` (over `slot-hash`) folds in where each record *is*: `(id, offset, length)` off the idx, no frame decoded and no record fetched. The mapped index snapshot (`vaelii.impl.disk.index-snapshot`) is checked with that one, because an image exists to make an open cost bytes read rather than records read, and a content digest would put every record back on the open path. It is the coarser hash in one direction and the finer one in the other: it cannot see a record rewritten to the same bytes at the same offset (nothing can produce that), and it *does* see a rewrite the index does not care about — a premise mark, or a compaction moving every frame — so a snapshot is discarded and rebuilt after one. Both directions are safe, because a discard is always legal for derived state.
The chunked nippy stream framing every serialization in the engine writes to a file or a sink — one home, so the export dump, the import reader and the snapshot sink cannot drift.
A chunked stream is a run of [int32 length][compressed chunk], each chunk an
independent compression window over back-to-back nippy frames. Constant memory on both
sides: the writer holds one chunk and never the corpus, and the reader thaws a chunk on
demand and drops it. A frame is whatever the caller freezes — a record's field map, an
index [key value] pair, a JTMS [handle label] pair — this namespace neither reads a
frame nor cares what one is.
Why it is its own namespace and not export's private helper. Three callers write
and read this format — vaelii.impl.io.export (the dump), vaelii.impl.io.import (the
reader) and vaelii.impl.io.snapshot (the derived-state image) — and the snapshot sink
is meant to be a thin adapter over this framing, not a second copy of it. A second
copy is exactly the drift a shared projection was created to avoid one layer up: two
writers that agree today and diverge on the next compression tweak. So the framing lives
once, here, and the container-specific concerns (a dump's meta.edn, an image's
manifest, which streams exist) stay with each caller.
The legacy single-window reader (read-window-seq) is kept for a v4/v5 foreign dump that
wrote one compression window over the whole file; the engine's own dumps have been chunked
since v6.
The chunked **nippy stream** framing every serialization in the engine writes to a file or a sink — one home, so the export dump, the import reader and the snapshot sink cannot drift. A chunked stream is a run of `[int32 length][compressed chunk]`, each chunk an independent compression window over back-to-back nippy frames. Constant memory on both sides: the writer holds one chunk and never the corpus, and the reader thaws a chunk on demand and drops it. A *frame* is whatever the caller freezes — a record's field map, an index `[key value]` pair, a JTMS `[handle label]` pair — this namespace neither reads a frame nor cares what one is. **Why it is its own namespace and not `export`'s private helper.** Three callers write and read this format — `vaelii.impl.io.export` (the dump), `vaelii.impl.io.import` (the reader) and `vaelii.impl.io.snapshot` (the derived-state image) — and the snapshot sink is meant to be *a thin adapter over this framing, not a second copy of it*. A second copy is exactly the drift a shared projection was created to avoid one layer up: two writers that agree today and diverge on the next compression tweak. So the framing lives once, here, and the container-specific concerns (a dump's `meta.edn`, an image's manifest, which streams exist) stay with each caller. The legacy single-window reader (`read-window-seq`) is kept for a v4/v5 foreign dump that wrote one compression window over the whole file; the engine's own dumps have been chunked since v6.
Synthesize a knowledge base of a chosen shape.
The two other kinds of KB are given: a shipped ontology (vaelii.impl.starter) is
fixed content, and an imported corpus (vaelii.impl.io.import, or a translated one)
is whatever the source says. Neither lets you ask what happens at ten times the
rules, and that is the question a scale or behaviour measurement is made of. So this
namespace generates a KB from a handful of numbers — how many types, individuals,
predicates, facts and rules, how the rules split forward/backward, how many of them are
defeasible — and each number is a knob the browser renders as a slider (knobs).
Two properties make a generated KB usable as a measurement rather than as noise:
java.util.Random in a fixed
order, so the same parameters give the same KB — a shape can be reproduced from the
numbers alone, and a run compared against a rerun.plan is pure — the whole KB as data, nothing asserted. load-into asserts it,
reporting progress through an optional :on-progress callback (which may throw to
cancel the load, the seam vaelii.impl.catalog cancels on).
Synthesize a knowledge base of a chosen **shape**. The two other kinds of KB are given: a shipped ontology (`vaelii.impl.starter`) is fixed content, and an imported corpus (`vaelii.impl.io.import`, or a translated one) is whatever the source says. Neither lets you ask *what happens at ten times the rules*, and that is the question a scale or behaviour measurement is made of. So this namespace generates a KB from a handful of numbers — how many types, individuals, predicates, facts and rules, how the rules split forward/backward, how many of them are defeasible — and each number is a knob the browser renders as a slider (`knobs`). Two properties make a generated KB usable as a measurement rather than as noise: * **Deterministic.** Everything is drawn from one seeded `java.util.Random` in a fixed order, so the same parameters give the same KB — a shape can be reproduced from the numbers alone, and a run compared against a rerun. * **Stratified.** Predicates are split into layers: facts populate layer 0, and a rule concluding a layer-k predicate draws its antecedents only from layers below k. The rule set is therefore acyclic, so forward chaining cascades base → derived → further-derived and terminates, instead of the runaway recursion a rule set wired at random produces. Individuals and predicates are Zipf-sampled, so the corpus has hot terms and a long tail like a real one rather than a uniform smear. `plan` is pure — the whole KB as data, nothing asserted. `load-into` asserts it, reporting progress through an optional `:on-progress` callback (which may throw to cancel the load, the seam `vaelii.impl.catalog` cancels on).
Import a vaelii export dump — a directory of record streams — into a KB,
landing in exactly the state the engine's own restart path (reindex / recover) already
knows how to produce.
A dump is a directory whose meta.edn is the marker and schema; every other file is
a nippy stream:
sentexes.nippy.stream one field-map frame per sentex
justifications.nippy.stream one per justification
provenance.nippy.stream [handle map] per frame — optional
A :records+index dump also carries the index, as a cache that is used only when
it can be proved to describe the records that were just stored (see the index below);
otherwise the index is rebuilt, and the summary says which happened and why.
Framing. A chunked stream is a run of [int32 length][compressed chunk], each
chunk a compression window over back-to-back nippy frames; a window stream is one
compression window over the lot. Our own dumps state which (:framing); a
foreign dump's is inferred from its own version line. Both are constant-memory lazy
seqs.
A frame of our own dialect is a plain field map whose :sentence is already
there — but a rule's set/*Rule wrappers and its variable names canonicalized into
the record (:direction / :defeasible / :varmap), so both are written back around
it before the constructor sees it. A frame that is not ours goes to a foreign
reader (vaelii.impl.foreign), which is resolved at runtime and may not be in the
build at all. The discrimination is on the frame, never on meta.edn's
:dialect: a declaration is not an authority over the bytes beside it, and keying off
the frame keeps a mixed dump readable.
Whatever the dialect, every sentence is re-canonicalized through this build's own
constructor (res/kb-sentex). A stored canonical form is never trusted, not even
our own: variable numbering, symmetric argument order and comparison folding belong to
the reading build, and a record indexed under a key this build's lookup never
reproduces would be silently unfindable.
Handles are preserved for a dump of ours: every record is stored at the handle the
dump gave it, so a handle means the same thing either side of an export. Safe because
the destination must be empty and because a store's counter clears any handle written
that way (p/next-id) — without which the next assert would mint handle 1 again and
overwrite the first imported record. Two things can still stop a handle landing as
given, and neither is silent: a frame with no :id, and a frame whose canonical form
is one already stored, which collapses onto that handle (a dedup this build is
right to perform — two engine forms can canonicalize to one stored record — and the
dump's numbering cannot survive it). Either makes the import :remapped, and then one
old->new map carries the dump's ids across: justification references, and the
(sentexHandle H) a meta-sentex embeds inside stored content, which
rewrite-embedded-handles! rewrites in the sentence. A meta-sentex whose embedded
handle cannot be resolved is dropped, and the drop reaches the map as well as the
store (forget-deleted): a dump id whose record is gone has to stop resolving, or the
references to it resolve to a handle nothing is stored at.
The index is replayed only when it can be proved to fit, and discarding it is
always safe — which is what makes a cache out of what would otherwise be a risk. An
index that does not match its records is worse than no index: every lookup then
answers confidently and short, and nothing in the engine is positioned to notice. So
all three of these must hold (index-decision):
kv/index-layout-version);vaelii.impl.io.fingerprint) — accumulated, not recomputed, since a second
pass over the records to validate a cache would cost more than the cache saves;Anything else rebuilds, at :info, with the reason named. A cache that silently
stops being used is a cache nobody maintains.
The store-facing replay is written against the engine's real seams: populate the record
store with the re-canonicalized records + justifications + premise marks, then either
install the dumped index (p/index-load) or rebuild it (reindex), then
core/recover — which rebuilds the JTMS and the taxonomy from the records, so a
replayed index shortcuts the index and nothing else. Out of scope: the :pg-memory
variant.
Import a vaelii **export dump** — a directory of record streams — into a KB,
landing in exactly the state the engine's own restart path (`reindex` / `recover`) already
knows how to produce.
A dump is a directory whose `meta.edn` is the marker and schema; every other file is
a nippy stream:
sentexes.nippy.stream one field-map frame per sentex
justifications.nippy.stream one per justification
provenance.nippy.stream [handle map] per frame — optional
A `:records+index` dump also carries the index, as a **cache** that is used only when
it can be proved to describe the records that were just stored (see *the index* below);
otherwise the index is rebuilt, and the summary says which happened and why.
**Framing.** A chunked stream is a run of `[int32 length][compressed chunk]`, each
chunk a compression window over back-to-back nippy frames; a window stream is one
compression window over the lot. Our own dumps **state** which (`:framing`); a
foreign dump's is inferred from its own version line. Both are constant-memory lazy
seqs.
**A frame of our own dialect is a plain field map** whose `:sentence` is already
there — but a rule's `set/*Rule` wrappers and its variable names canonicalized *into*
the record (`:direction` / `:defeasible` / `:varmap`), so both are written back around
it before the constructor sees it. A frame that is *not* ours goes to a foreign
reader (`vaelii.impl.foreign`), which is resolved at runtime and may not be in the
build at all. The discrimination is on the **frame**, never on `meta.edn`'s
`:dialect`: a declaration is not an authority over the bytes beside it, and keying off
the frame keeps a mixed dump readable.
Whatever the dialect, **every sentence is re-canonicalized** through this build's own
constructor (`res/kb-sentex`). A stored canonical form is never trusted, not even
our own: variable numbering, symmetric argument order and comparison folding belong to
the *reading* build, and a record indexed under a key this build's `lookup` never
reproduces would be silently unfindable.
**Handles are preserved** for a dump of ours: every record is stored at the handle the
dump gave it, so a handle means the same thing either side of an export. Safe because
the destination must be empty and because a store's counter clears any handle written
that way (`p/next-id`) — without which the next `assert` would mint handle 1 again and
overwrite the first imported record. Two things can still stop a handle landing as
given, and neither is silent: a frame with no `:id`, and a frame whose canonical form
is one already stored, which **collapses** onto that handle (a dedup this build is
right to perform — two engine forms can canonicalize to one stored record — and the
dump's numbering cannot survive it). Either makes the import `:remapped`, and then one
`old->new` map carries the dump's ids across: justification references, and the
`(sentexHandle H)` a meta-sentex embeds *inside stored content*, which
`rewrite-embedded-handles!` rewrites in the sentence. A meta-sentex whose embedded
handle cannot be resolved is **dropped**, and the drop reaches the map as well as the
store (`forget-deleted`): a dump id whose record is gone has to stop resolving, or the
references to it resolve to a handle nothing is stored at.
**The index is replayed only when it can be proved to fit**, and discarding it is
always safe — which is what makes a cache out of what would otherwise be a risk. An
index that does not match its records is *worse* than no index: every lookup then
answers confidently and short, and nothing in the engine is positioned to notice. So
all three of these must hold (`index-decision`):
* the entries are keyed in the layout this build reads (`kv/index-layout-version`);
* the fingerprint accumulated **while storing** equals the one written beside the
entries (`vaelii.impl.io.fingerprint`) — accumulated, not recomputed, since a second
pass over the records to validate a cache would cost more than the cache saves;
* the handles were preserved, so a posting names the record it named in the source.
Anything else rebuilds, at `:info`, with the reason named. A cache that silently
stops being used is a cache nobody maintains.
The store-facing replay is written against the engine's real seams: populate the record
store with the re-canonicalized records + justifications + premise marks, then either
install the dumped index (`p/index-load`) or rebuild it (`reindex`), then
`core/recover` — which rebuilds the JTMS and the taxonomy **from the records**, so a
replayed index shortcuts the index and nothing else. Out of scope: the `:pg-memory`
variant.A snapshot of the KB's derived state, and the two-op sink it is written through.
Derived state — the index, and (from 0.9.0's second thread) the taxonomy and the JTMS
labels — is rebuilt from the records on every open by reindex / recover, at a cost
that is O(records). A snapshot is a cache of that rebuild: written once, read back on
the next open, and installed instead of recomputed. It is never a source of truth. Its
only failure mode is "recompute", never "wrong answer", which is what keeps it clear of
order independence — a snapshot is stamped to the exact records it was derived from, and a
mismatch takes the slow path (reindex/recover), never a stale belief.
There is more than one place a snapshot might live — a directory of files, a Postgres
blob, memory for a test — and letting each invent its own serialization is the drift the
shared [key value] index projection was created to avoid one layer up. So the shape is
one sink:
SnapshotSink knows only how to write a named section (a constant-memory stream of
frames) and to commit a manifest; a SnapshotSource knows how to read a section back
and to read the manifest;[key value], and later the taxonomy's edges and the
JTMS's labels) and the validity check (decision) live here, above the sink, written
once;file-sink / file-source over vaelii.impl.io.frames, and
memory-medium for a test — live below it, each a small adapter.A section written to one sink loads from another, the same property p/index-entries /
p/index-load already give the index across backends.
It is written last, exactly as a dump's meta.edn is: a half-written image has no
manifest, so read-manifest returns nil and the image is never offered — the caller
rebuilds. Opening a file-sink deletes any existing manifest first, so the window in
which an old manifest could describe half-rewritten sections does not exist.
decision is lifted from vaelii.impl.disk.index-snapshot/decision: one reason per
mismatch class, and any non-nil reason discards the whole image and falls back to a
rebuild. The classes:
:absent — no manifest (missing, or a save that never committed);:layout-changed — the snapshot format version, or kv/index-layout-version, is not
this build's (an index in another layout reads as empty rather
than wrong, which is the undiagnosable failure this forecloses);:records-differ — the manifest's records stamp is not the store's now (a different
KB, or the same one after a write);:entries-truncated — a section is short or unreadable, caught while installing (a torn
nippy chunk reads as a clean EOF, so truncation shows only as a
frame count below what the manifest recorded).There is no :byte-order class here, though index-snapshot has one: that image writes
raw little-endian int runs, so an image from a machine of the other endianness must be
refused; these sections are nippy frames, which are endian-neutral, so a byte-order
mismatch cannot arise and the format version subsumes any encoding change.
The stamp is vaelii.impl.io.fingerprint's records digest, which ports: a :disk
store hashes its slots, a :memory store folds its records, a SQL store answers a count
— and all compare against the same manifest number. Which digest a caller passes is its
business; the sink only compares.
A **snapshot** of the KB's derived state, and the two-op *sink* it is written through.
Derived state — the index, and (from 0.9.0's second thread) the taxonomy and the JTMS
labels — is rebuilt from the records on every open by `reindex` / `recover`, at a cost
that is O(records). A snapshot is a *cache* of that rebuild: written once, read back on
the next open, and installed instead of recomputed. It is never a source of truth. Its
only failure mode is "recompute", never "wrong answer", which is what keeps it clear of
order independence — a snapshot is stamped to the exact records it was derived from, and a
mismatch takes the slow path (`reindex`/`recover`), never a stale belief.
## One image, many sinks
There is more than one place a snapshot might live — a directory of files, a Postgres
blob, memory for a test — and letting each invent its own serialization is the drift the
shared `[key value]` index projection was created to avoid one layer up. So the shape is
one **sink**:
* a snapshot is a set of **named sections** plus a **manifest**;
* a `SnapshotSink` knows only how to *write a named section* (a constant-memory stream of
frames) and to *commit a manifest*; a `SnapshotSource` knows how to *read a section back*
and to *read the manifest*;
* the **projections** (the index's `[key value]`, and later the taxonomy's edges and the
JTMS's labels) and the **validity check** (`decision`) live here, above the sink, written
once;
* the **targets** — `file-sink` / `file-source` over `vaelii.impl.io.frames`, and
`memory-medium` for a test — live below it, each a small adapter.
A section written to one sink loads from another, the same property `p/index-entries` /
`p/index-load` already give the index across backends.
## The manifest is the commit point
It is written **last**, exactly as a dump's `meta.edn` is: a half-written image has no
manifest, so `read-manifest` returns nil and the image is never offered — the caller
rebuilds. Opening a `file-sink` deletes any existing manifest first, so the window in
which an old manifest could describe half-rewritten sections does not exist.
## Validity is the whole design
`decision` is lifted from `vaelii.impl.disk.index-snapshot/decision`: one reason per
mismatch class, and any non-nil reason discards the *whole* image and falls back to a
rebuild. The classes:
* `:absent` — no manifest (missing, or a save that never committed);
* `:layout-changed` — the snapshot format version, or `kv/index-layout-version`, is not
this build's (an index in another layout reads as *empty* rather
than wrong, which is the undiagnosable failure this forecloses);
* `:records-differ` — the manifest's records stamp is not the store's now (a different
KB, or the same one after a write);
* `:entries-truncated` — a section is short or unreadable, caught while installing (a torn
nippy chunk reads as a clean EOF, so truncation shows only as a
frame count below what the manifest recorded).
There is no `:byte-order` class here, though `index-snapshot` has one: that image writes
raw little-endian `int` runs, so an image from a machine of the other endianness must be
refused; these sections are nippy frames, which are endian-neutral, so a byte-order
mismatch cannot arise and the format version subsumes any encoding change.
The stamp is `vaelii.impl.io.fingerprint`'s records digest, which **ports**: a `:disk`
store hashes its slots, a `:memory` store folds its records, a SQL store answers a count
— and all compare against the same manifest number. Which digest a caller passes is its
business; the sink only compares.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |