Write a KB out as a portable export dump — a directory holding the record store's three streams, in a format that survives a backend change, an index-representation change, and a record class rename.
The :disk store directory looks like an archive and is not one: it holds frozen
records, so renaming a record class makes every frame in it thaw to a
{:nippy/unthawable …} placeholder — silently, because a placeholder is a perfectly
good map. Hence the one non-negotiable rule of this format:
A frame never carries a class name.
A frame is the record's field map — (into {} record), a plain map — so a rename
changes nothing a dump holds, and vaelii.impl.io.import's field-map already
accepts one.
What a dump holds is what the record store holds, because everything else the KB has
is derived from it: sentexes, justifications, and per-handle provenance. A premise
needs no stream of its own — a premise is a sentex whose :strength is non-nil, a
field on the record in both backends, so the mark rides along with it. The index is
a cache (reindex), the taxonomy and the TMS labels are recomputed (recover).
<dump>/
meta.edn the marker, the schema, and the counts
sentexes.nippy.stream one frame per sentex
justifications.nippy.stream one frame per justification
provenance.nippy.stream [handle provenance-map] per frame (omitted when none)
index/entries.nippy.stream [key value] per frame — only in :records+index
index/index.edn the layout version + the records fingerprint
The index is optional and always discardable. :variant :records+index writes it
as well, in the [structured-key value] projection every index backend shares
(p/index-entries), so an index written by one backend loads into another. It is a
cache: a reader replays it only when it can prove the entries were derived from
exactly the records beside them (vaelii.impl.io.fingerprint), and rebuilds otherwise.
That is what makes writing it safe — an index that does not match its records is worse
than no index, because every lookup then answers confidently and short.
Framing is the chunked layout vaelii.impl.io.import/read-chunked-seq reads: a
run of [int32 length][compressed chunk], each chunk an independent compression
window over back-to-back nippy frames. Constant memory on both sides — the writer
holds one chunk, never the corpus. meta.edn states the framing rather than
implying it through a version number, because the engine's dumps number their own
format and a reader must never have to guess.
meta.edn is written last, which makes it double as the completion marker:
vaelii.impl.catalog/classify keys on it, so a half-written or cancelled export is
not offered as loadable. That is the one ordering constraint in the whole format.
Export from a KB nobody is writing: the walk fetches record by record, and pure's single-writer contract offers no snapshot to walk instead.
Write a KB out as a portable **export dump** — a directory holding the record
store's three streams, in a format that survives a backend change, an
index-representation change, and a record class rename.
The `:disk` store directory looks like an archive and is not one: it holds frozen
*records*, so renaming a record class makes every frame in it thaw to a
`{:nippy/unthawable …}` placeholder — silently, because a placeholder is a perfectly
good map. Hence the one non-negotiable rule of this format:
> **A frame never carries a class name.**
A frame is the record's **field map** — `(into {} record)`, a plain map — so a rename
changes nothing a dump holds, and `vaelii.impl.io.import`'s `field-map` already
accepts one.
What a dump holds is what the record store holds, because everything else the KB has
is derived from it: sentexes, justifications, and per-handle provenance. A premise
needs no stream of its own — a premise *is* a sentex whose `:strength` is non-nil, a
field on the record in both backends, so the mark rides along with it. The index is
a cache (`reindex`), the taxonomy and the TMS labels are recomputed (`recover`).
<dump>/
meta.edn the marker, the schema, and the counts
sentexes.nippy.stream one frame per sentex
justifications.nippy.stream one frame per justification
provenance.nippy.stream [handle provenance-map] per frame (omitted when none)
index/entries.nippy.stream [key value] per frame — only in :records+index
index/index.edn the layout version + the records fingerprint
**The index is optional and always discardable.** `:variant :records+index` writes it
as well, in the `[structured-key value]` projection every index backend shares
(`p/index-entries`), so an index written by one backend loads into another. It is a
*cache*: a reader replays it only when it can prove the entries were derived from
exactly the records beside them (`vaelii.impl.io.fingerprint`), and rebuilds otherwise.
That is what makes writing it safe — an index that does not match its records is worse
than no index, because every lookup then answers confidently and short.
**Framing** is the chunked layout `vaelii.impl.io.import/read-chunked-seq` reads: a
run of `[int32 length][compressed chunk]`, each chunk an independent compression
window over back-to-back nippy frames. Constant memory on both sides — the writer
holds one chunk, never the corpus. `meta.edn` *states* the framing rather than
implying it through a version number, because the engine's dumps number their own
format and a reader must never have to guess.
**`meta.edn` is written last**, which makes it double as the completion marker:
`vaelii.impl.catalog/classify` keys on it, so a half-written or cancelled export is
not offered as loadable. That is the one ordering constraint in the whole format.
Export from a KB nobody is writing: the walk fetches record by record, and pure's
single-writer contract offers no snapshot to walk instead.What makes a dumped index and a dumped set of records provably the same KB.
The index is derived from the records, so an index dump is only valid against the records it was derived from. Nothing downstream can notice when it is not: a stale trie node simply reports fewer handles and a query returns fewer results — no exception, no warning, just a KB that quietly knows less. So the writer records a fingerprint over the records and the reader recomputes it while storing them; a mismatch discards the index and rebuilds.
A sum, not a chain. The digest is the sum (mod 2⁶⁴) of a per-record hash, so it is commutative — the reader accumulates it in whatever order the frames arrive, which is not the order the writer walked — and additive rather than xor, so two identical records cannot cancel each other out.
What is hashed is what the index is a function of: the handle, :sentence,
:context, :truth, and a rule's :antecedent / :consequent. sentex/path,
kv/root-keys, kv/sentex-terms and the rule index read exactly those, and the handle
because a posting is a set of handles — the same content at a different handle makes
every posting naming it wrong. Deliberately not hashed: :strength, :varmap,
:direction, :defeasible. None of them changes a single index entry, so a dump
differing in one of them still has a valid index, and hashing them would make the cache
go unused for a reason that is not a reason. (:strength also arrives after the
storing pass — a premise mark is applied once every record is down — so hashing it
would cost the second pass over the records this exists to avoid.)
The hash is FNV-1a over the fields' own hash values: cheap, deterministic across
runs, and strong enough for the question being asked, which is whether two sets of
records are the same — not whether an adversary can forge one. A dump is not a
trust boundary; if it were, this would be a real digest and signed.
Two granularities, one rule. of-records hashes what a record says, and needs
the record — which is what a dump's reader has in hand anyway. of-slots hashes
where each record is: (id, offset, length) off the idx, no frame decoded and no
record fetched. The mapped index snapshot (vaelii.impl.disk.index-snapshot) is
checked with that one, because an image exists to make an open cost bytes read rather
than records read, and a content digest would put every record back on the open path.
It is the coarser hash in one direction and the finer one in the other: it cannot see
a record rewritten to the same bytes at the same offset (nothing can produce that),
and it does see a rewrite the index does not care about — a premise mark, or a
compaction moving every frame — so a snapshot is discarded and rebuilt after one.
Both directions are safe, because a discard is always legal for derived state.
What makes a dumped index and a dumped set of records provably the same KB. The index is derived from the records, so an index dump is only valid *against the records it was derived from*. Nothing downstream can notice when it is not: a stale trie node simply reports fewer handles and a query returns fewer results — no exception, no warning, just a KB that quietly knows less. So the writer records a fingerprint over the records and the reader recomputes it while storing them; a mismatch discards the index and rebuilds. **A sum, not a chain.** The digest is the sum (mod 2⁶⁴) of a per-record hash, so it is commutative — the reader accumulates it in whatever order the frames arrive, which is not the order the writer walked — and *additive* rather than xor, so two identical records cannot cancel each other out. **What is hashed is what the index is a function of**: the handle, `:sentence`, `:context`, `:truth`, and a rule's `:antecedent` / `:consequent`. `sentex/path`, `kv/root-keys`, `kv/sentex-terms` and the rule index read exactly those, and the handle because a posting *is* a set of handles — the same content at a different handle makes every posting naming it wrong. Deliberately **not** hashed: `:strength`, `:varmap`, `:direction`, `:defeasible`. None of them changes a single index entry, so a dump differing in one of them still has a valid index, and hashing them would make the cache go unused for a reason that is not a reason. (`:strength` also arrives after the storing pass — a premise mark is applied once every record is down — so hashing it would cost the second pass over the records this exists to avoid.) The hash is FNV-1a over the fields' own `hash` values: cheap, deterministic across runs, and strong enough for the question being asked, which is whether two sets of records are *the same* — not whether an adversary can forge one. A dump is not a trust boundary; if it were, this would be a real digest and signed. **Two granularities, one rule.** `of-records` hashes what a record *says*, and needs the record — which is what a dump's reader has in hand anyway. `of-slots` hashes where each record *is*: `(id, offset, length)` off the idx, no frame decoded and no record fetched. The mapped index snapshot (`vaelii.impl.disk.index-snapshot`) is checked with that one, because an image exists to make an open cost bytes read rather than records read, and a content digest would put every record back on the open path. It is the coarser hash in one direction and the finer one in the other: it cannot see a record rewritten to the same bytes at the same offset (nothing can produce that), and it *does* see a rewrite the index does not care about — a premise mark, or a compaction moving every frame — so a snapshot is discarded and rebuilt after one. Both directions are safe, because a discard is always legal for derived state.
Synthesize a knowledge base of a chosen shape.
The two other kinds of KB are given: a shipped ontology (vaelii.impl.starter) is
fixed content, and an imported corpus (vaelii.impl.io.import, or a translated one)
is whatever the source says. Neither lets you ask what happens at ten times the
rules, and that is the question a scale or behaviour measurement is made of. So this
namespace generates a KB from a handful of numbers — how many types, individuals,
predicates, facts and rules, how the rules split forward/backward, how many of them are
defeasible — and each number is a knob the browser renders as a slider (knobs).
Two properties make a generated KB usable as a measurement rather than as noise:
java.util.Random in a fixed
order, so the same parameters give the same KB — a shape can be reproduced from the
numbers alone, and a run compared against a rerun.plan is pure — the whole KB as data, nothing asserted. load-into asserts it,
reporting progress through an optional :on-progress callback (which may throw to
cancel the load, the seam vaelii.impl.catalog cancels on).
Synthesize a knowledge base of a chosen **shape**. The two other kinds of KB are given: a shipped ontology (`vaelii.impl.starter`) is fixed content, and an imported corpus (`vaelii.impl.io.import`, or a translated one) is whatever the source says. Neither lets you ask *what happens at ten times the rules*, and that is the question a scale or behaviour measurement is made of. So this namespace generates a KB from a handful of numbers — how many types, individuals, predicates, facts and rules, how the rules split forward/backward, how many of them are defeasible — and each number is a knob the browser renders as a slider (`knobs`). Two properties make a generated KB usable as a measurement rather than as noise: * **Deterministic.** Everything is drawn from one seeded `java.util.Random` in a fixed order, so the same parameters give the same KB — a shape can be reproduced from the numbers alone, and a run compared against a rerun. * **Stratified.** Predicates are split into layers: facts populate layer 0, and a rule concluding a layer-k predicate draws its antecedents only from layers below k. The rule set is therefore acyclic, so forward chaining cascades base → derived → further-derived and terminates, instead of the runaway recursion a rule set wired at random produces. Individuals and predicates are Zipf-sampled, so the corpus has hot terms and a long tail like a real one rather than a uniform smear. `plan` is pure — the whole KB as data, nothing asserted. `load-into` asserts it, reporting progress through an optional `:on-progress` callback (which may throw to cancel the load, the seam `vaelii.impl.catalog` cancels on).
Import a vaelii export dump — a directory of record streams — into a pure KB,
landing in exactly the state pure's own restart path (reindex / recover) already
knows how to produce.
A dump is a directory whose meta.edn is the marker and schema; every other file is
a nippy stream:
sentexes.nippy.stream one field-map frame per sentex
justifications.nippy.stream one per justification
provenance.nippy.stream [sentex-handle map] per frame — optional
A :records+index dump also carries the index, as a cache that is used only when
it can be proved to describe the records that were just stored (see the index below);
otherwise the index is rebuilt, and the summary says which happened and why.
Framing. A chunked stream is a run of [int32 length][compressed chunk], each
chunk a compression window over back-to-back nippy frames; a window stream is one
compression window over the lot. Our own dumps state which (:framing); a
foreign dump's is inferred from its own version line. Both are constant-memory lazy
seqs.
A frame of our own dialect is a plain field map whose :sentence is already
there — but a rule's set/*Rule wrappers and its variable names canonicalized into
the record (:direction / :defeasible / :varmap), so both are written back around
it before the constructor sees it. A frame that is not ours goes to a foreign
reader (vaelii.impl.foreign), which is resolved at runtime and may not be in the
build at all. The discrimination is on the frame, never on meta.edn's
:dialect: a declaration is not an authority over the bytes beside it, and keying off
the frame keeps a mixed dump readable.
Whatever the dialect, every sentence is re-canonicalized through this build's own
constructor (res/kb-sentex). A stored canonical form is never trusted, not even
our own: variable numbering, symmetric argument order and comparison folding belong to
the reading build, and a record indexed under a key this build's lookup never
reproduces would be silently unfindable.
Handles are preserved for a dump of ours: every record is stored at the handle the
dump gave it, so a handle means the same thing either side of an export. Safe because
the destination must be empty and because a store's counter clears any handle written
that way (p/next-id) — without which the next assert would mint handle 1 again and
overwrite the first imported record. Two things can still stop a handle landing as
given, and neither is silent: a frame with no :id, and a frame whose canonical form
is one already stored, which collapses onto that handle (a dedup this build is
right to perform — two engine forms can canonicalize to one pure record — and the
dump's numbering cannot survive it). Either makes the import :remapped, and then one
old->new map carries the dump's ids across: justification references, and the
(sentexHandle H) a meta-sentex embeds inside stored content, which
rewrite-embedded-handles! rewrites in the sentence.
The index is replayed only when it can be proved to fit, and discarding it is
always safe — which is what makes a cache out of what would otherwise be a risk. An
index that does not match its records is worse than no index: every lookup then
answers confidently and short, and nothing in the engine is positioned to notice. So
all three of these must hold (index-decision):
kv/index-layout-version);vaelii.impl.io.fingerprint) — accumulated, not recomputed, since a second
pass over the records to validate a cache would cost more than the cache saves;Anything else rebuilds, at :info, with the reason named. A cache that silently
stops being used is a cache nobody maintains.
The store-facing replay is written against pure's real seams: populate the record
store with the re-canonicalized records + justifications + premise marks, then either
install the dumped index (p/index-load) or rebuild it (reindex), then
core/recover — which rebuilds the JTMS and the taxonomy from the records, so a
replayed index shortcuts the index and nothing else. Out of scope: the :pg-memory
variant.
Import a vaelii **export dump** — a directory of record streams — into a pure KB,
landing in exactly the state pure's own restart path (`reindex` / `recover`) already
knows how to produce.
A dump is a directory whose `meta.edn` is the marker and schema; every other file is
a nippy stream:
sentexes.nippy.stream one field-map frame per sentex
justifications.nippy.stream one per justification
provenance.nippy.stream [sentex-handle map] per frame — optional
A `:records+index` dump also carries the index, as a **cache** that is used only when
it can be proved to describe the records that were just stored (see *the index* below);
otherwise the index is rebuilt, and the summary says which happened and why.
**Framing.** A chunked stream is a run of `[int32 length][compressed chunk]`, each
chunk a compression window over back-to-back nippy frames; a window stream is one
compression window over the lot. Our own dumps **state** which (`:framing`); a
foreign dump's is inferred from its own version line. Both are constant-memory lazy
seqs.
**A frame of our own dialect is a plain field map** whose `:sentence` is already
there — but a rule's `set/*Rule` wrappers and its variable names canonicalized *into*
the record (`:direction` / `:defeasible` / `:varmap`), so both are written back around
it before the constructor sees it. A frame that is *not* ours goes to a foreign
reader (`vaelii.impl.foreign`), which is resolved at runtime and may not be in the
build at all. The discrimination is on the **frame**, never on `meta.edn`'s
`:dialect`: a declaration is not an authority over the bytes beside it, and keying off
the frame keeps a mixed dump readable.
Whatever the dialect, **every sentence is re-canonicalized** through this build's own
constructor (`res/kb-sentex`). A stored canonical form is never trusted, not even
our own: variable numbering, symmetric argument order and comparison folding belong to
the *reading* build, and a record indexed under a key this build's `lookup` never
reproduces would be silently unfindable.
**Handles are preserved** for a dump of ours: every record is stored at the handle the
dump gave it, so a handle means the same thing either side of an export. Safe because
the destination must be empty and because a store's counter clears any handle written
that way (`p/next-id`) — without which the next `assert` would mint handle 1 again and
overwrite the first imported record. Two things can still stop a handle landing as
given, and neither is silent: a frame with no `:id`, and a frame whose canonical form
is one already stored, which **collapses** onto that handle (a dedup this build is
right to perform — two engine forms can canonicalize to one pure record — and the
dump's numbering cannot survive it). Either makes the import `:remapped`, and then one
`old->new` map carries the dump's ids across: justification references, and the
`(sentexHandle H)` a meta-sentex embeds *inside stored content*, which
`rewrite-embedded-handles!` rewrites in the sentence.
**The index is replayed only when it can be proved to fit**, and discarding it is
always safe — which is what makes a cache out of what would otherwise be a risk. An
index that does not match its records is *worse* than no index: every lookup then
answers confidently and short, and nothing in the engine is positioned to notice. So
all three of these must hold (`index-decision`):
* the entries are keyed in the layout this build reads (`kv/index-layout-version`);
* the fingerprint accumulated **while storing** equals the one written beside the
entries (`vaelii.impl.io.fingerprint`) — accumulated, not recomputed, since a second
pass over the records to validate a cache would cost more than the cache saves;
* the handles were preserved, so a posting names the record it named in the source.
Anything else rebuilds, at `:info`, with the reason named. A cache that silently
stops being used is a cache nobody maintains.
The store-facing replay is written against pure's real seams: populate the record
store with the re-canonicalized records + justifications + premise marks, then either
install the dumped index (`p/index-load`) or rebuild it (`reindex`), then
`core/recover` — which rebuilds the JTMS and the taxonomy **from the records**, so a
replayed index shortcuts the index and nothing else. Out of scope: the `:pg-memory`
variant.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |