Liking cljdoc? Tell your friends :D

vaelii.impl.io.digest

The content digest of a stored sentex: "sha256:" and the 64 lowercase hex characters of a SHA-256 over the sentex's context, sign and canonical sentence, encoded by encode!. A (sentexHandle n) inside the sentence digests as the identity of the record n names, so two stores holding one sentex compute one digest for it. A digest is a checksum and locates nothing: a delta carries one per reference, and a merge compares it with the digest of the sentex the reference resolves to. See docs/storage.md, "Content digests".

The **content digest** of a stored sentex: `"sha256:"` and the 64 lowercase hex
characters of a SHA-256 over the sentex's context, sign and canonical sentence, encoded
by `encode!`.  A `(sentexHandle n)` inside the sentence digests as the identity of the
record `n` names, so two stores holding one sentex compute one digest for it.  A digest
is a checksum and locates nothing: a delta carries one per reference, and a merge
compares it with the digest of the sentex the reference resolves to.  See
docs/storage.md, "Content digests".
raw docstring

vaelii.impl.io.export

Write a KB out as a portable export dump: a directory holding the record store's streams in a format that survives a backend change, an index-representation change and a record class rename. A frame never carries a class name; it is the record's field map. meta.edn is written last and is the completion marker. The index is optional and always discardable. See docs/storage.md, "Exporting and importing a dump".

Write a KB out as a portable export dump: a directory holding the record store's streams
in a format that survives a backend change, an index-representation change and a record
class rename. A frame never carries a class name; it is the record's field map. `meta.edn`
is written last and is the completion marker. The index is optional and always
discardable. See docs/storage.md, "Exporting and importing a dump".
raw docstring

vaelii.impl.io.fingerprint

What makes a dumped index and a dumped set of records provably the same KB.

The index is derived from the records, so an index dump is only valid against the records it was derived from. Nothing downstream can notice when it is not: a stale trie node simply reports fewer handles and a query returns fewer results — no exception, no warning, just a KB that quietly knows less. So the writer records a fingerprint over the records and the reader recomputes it while storing them; a mismatch discards the index and rebuilds.

A sum, not a chain. The digest is the sum (mod 2⁶⁴) of a per-record hash, so it is commutative — the reader accumulates it in whatever order the frames arrive, which is not the order the writer walked — and additive rather than xor, so two identical records cannot cancel each other out.

What is hashed is what the index is a function of: the handle, the sentence (sentex/sentence-of, which is a rule's implies form), :context, the sign (sentex/polarity), a rule's :antecedent / :consequent, and a choice or constraint rule's :effect, which the trie key spells in its two trailing slots (a :derive rule's is left out, so its hash is its implies form's). sentex/path, kv/root-keys, kv/sentex-terms and the rule index read exactly those, and the handle because a posting is a set of handles — the same content at a different handle makes every posting naming it wrong. Deliberately not hashed: :strength, :varmap, :engines, :defeasible. None of them changes a single index entry, so a dump differing in one of them still has a valid index, and hashing them would make the cache go unused for a reason that is not a reason. (:strength also arrives after the storing pass — a premise mark is applied once every record is down — so hashing it would cost the second pass over the records this exists to avoid.)

The hash is FNV-1a over the fields' own hash values: cheap, deterministic across runs, and strong enough for the question being asked, which is whether two sets of records are the same — not whether an adversary can forge one. A dump is not a trust boundary; if it were, this would be a real digest and signed.

Two granularities, one rule. accumulator (over record-hash) folds in what a record says, and needs the record — the export rides it on the sentex walk it is already making, and the import rides it on the frames it is already decoding. slot-accumulator (over slot-hash) folds in where each record is: (id, offset, length) off the idx, no frame decoded and no record fetched. The mapped index snapshot (vaelii.impl.disk.index-snapshot) is checked with that one, because an image exists to make an open cost bytes read rather than records read, and a content digest would put every record back on the open path. It is the coarser hash in one direction and the finer one in the other: it cannot see a record rewritten to the same bytes at the same offset (nothing can produce that), and it does see a rewrite the index does not care about — a premise mark, or a compaction moving every frame — so a snapshot is discarded and rebuilt after one. Both directions are safe, because a discard is always legal for derived state.

What makes a dumped index and a dumped set of records provably the same KB.

The index is derived from the records, so an index dump is only valid *against the
records it was derived from*.  Nothing downstream can notice when it is not: a stale
trie node simply reports fewer handles and a query returns fewer results — no
exception, no warning, just a KB that quietly knows less.  So the writer records a
fingerprint over the records and the reader recomputes it while storing them; a
mismatch discards the index and rebuilds.

**A sum, not a chain.**  The digest is the sum (mod 2⁶⁴) of a per-record hash, so it is
commutative — the reader accumulates it in whatever order the frames arrive, which is
not the order the writer walked — and *additive* rather than xor, so two identical
records cannot cancel each other out.

**What is hashed is what the index is a function of**: the handle, the sentence
(`sentex/sentence-of`, which is a rule's `implies` form), `:context`, the sign
(`sentex/polarity`), a rule's `:antecedent` / `:consequent`, and a choice or
constraint rule's `:effect`, which the trie key spells in its two trailing slots
(a `:derive` rule's is left out, so its hash is its implies form's).  `sentex/path`,
`kv/root-keys`, `kv/sentex-terms` and the rule index read exactly those, and the handle
because a posting *is* a set of handles — the same content at a different handle makes
every posting naming it wrong.  Deliberately **not** hashed: `:strength`, `:varmap`,
`:engines`, `:defeasible`.  None of them changes a single index entry, so a dump
differing in one of them still has a valid index, and hashing them would make the cache
go unused for a reason that is not a reason.  (`:strength` also arrives after the
storing pass — a premise mark is applied once every record is down — so hashing it
would cost the second pass over the records this exists to avoid.)

The hash is FNV-1a over the fields' own `hash` values: cheap, deterministic across
runs, and strong enough for the question being asked, which is whether two sets of
records are *the same* — not whether an adversary can forge one.  A dump is not a
trust boundary; if it were, this would be a real digest and signed.

**Two granularities, one rule.**  `accumulator` (over `record-hash`) folds in what a
record *says*, and needs the record — the export rides it on the sentex walk it is
already making, and the import rides it on the frames it is already decoding.
`slot-accumulator` (over `slot-hash`) folds in where each record *is*:
`(id, offset, length)` off the idx, no frame decoded and no record fetched.  The mapped index snapshot (`vaelii.impl.disk.index-snapshot`) is
checked with that one, because an image exists to make an open cost bytes read rather
than records read, and a content digest would put every record back on the open path.
It is the coarser hash in one direction and the finer one in the other: it cannot see
a record rewritten to the same bytes at the same offset (nothing can produce that),
and it *does* see a rewrite the index does not care about — a premise mark, or a
compaction moving every frame — so a snapshot is discarded and rebuilt after one.
Both directions are safe, because a discard is always legal for derived state.
raw docstring

vaelii.impl.io.frames

The chunked nippy stream framing every serialization in the engine writes to a file or a sink — one home, so the export dump, the import reader and the snapshot sink cannot drift.

A chunked stream is a run of [int32 length][compressed chunk], each chunk an independent compressed stream over back-to-back nippy frames. Constant memory on both sides: the writer holds one chunk and never the corpus, and the reader thaws a chunk on demand and drops it. A frame is whatever the caller freezes — a record's field map, an index [key value] pair, a JTMS [handle label] pair — this namespace neither reads a frame nor cares what one is.

Why it is its own namespace and not export's private helper. Three callers write and read this format — vaelii.impl.io.export (the dump), vaelii.impl.io.import (the reader) and vaelii.impl.io.snapshot (the index image) — and the snapshot sink is meant to be a thin adapter over this framing, not a second copy of it. A second copy is exactly the drift a shared projection was created to avoid one layer up: two writers that agree today and diverge on the next compression tweak. So the framing lives once, here, and the container-specific concerns (what a dump's meta.edn records, an image's manifest) stay with each caller. The dump's stream names are not one of those concerns: which streams exist and what each is called is the layout the writer and the reader must agree on — a second copy is the same drift one level down — so the names live here too, beside the framing they name (meta-file and its six siblings below).

The legacy single-stream reader (read-single-stream-seq) is kept for a v4/v5 foreign dump that wrote one compressed stream over the whole file; the engine's own dumps have been chunked since v6.

The chunked **nippy stream** framing every serialization in the engine writes to a
file or a sink — one home, so the export dump, the import reader and the snapshot sink
cannot drift.

A chunked stream is a run of `[int32 length][compressed chunk]`, each chunk an
independent compressed stream over back-to-back nippy frames.  Constant memory on both
sides: the writer holds one chunk and never the corpus, and the reader thaws a chunk on
demand and drops it.  A *frame* is whatever the caller freezes — a record's field map, an
index `[key value]` pair, a JTMS `[handle label]` pair — this namespace neither reads a
frame nor cares what one is.

**Why it is its own namespace and not `export`'s private helper.**  Three callers write
and read this format — `vaelii.impl.io.export` (the dump), `vaelii.impl.io.import` (the
reader) and `vaelii.impl.io.snapshot` (the index image) — and the snapshot sink
is meant to be *a thin adapter over this framing, not a second copy of it*.  A second
copy is exactly the drift a shared projection was created to avoid one layer up: two
writers that agree today and diverge on the next compression tweak.  So the framing lives
once, here, and the container-specific concerns (what a dump's `meta.edn` records, an
image's manifest) stay with each caller.  The dump's stream **names** are not one of
those concerns: which streams exist and what each is called is the layout the writer
and the reader must agree on — a second copy is the same drift one level down — so the
names live here too, beside the framing they name (`meta-file` and its six siblings
below).

The legacy single-stream reader (`read-single-stream-seq`) is kept for a v4/v5 foreign dump that
wrote one compressed stream over the whole file; the engine's own dumps have been chunked
since v6.
raw docstring

vaelii.impl.io.import

Import a vaelii export dump, a directory of record streams, into a KB, landing in the state the engine's own restart path (reindex / recover) produces. meta.edn is the marker and schema, and every other file is a nippy stream. Every sentence is re-canonicalized through res/kb-sentex; handles are preserved for a dump of ours; the dumped index is replayed only when it is proved to fit, and the summary says which happened and why. A frame that is not of our dialect goes to a foreign reader (vaelii.impl.foreign). Out of scope: the :pg-memory variant. See docs/storage.md, "Importing a dump".

Import a vaelii export dump, a directory of record streams, into a KB, landing in the state
the engine's own restart path (`reindex` / `recover`) produces. `meta.edn` is the marker
and schema, and every other file is a nippy stream. Every sentence is re-canonicalized
through `res/kb-sentex`; handles are preserved for a dump of ours; the dumped index is
replayed only when it is proved to fit, and the summary says which happened and why. A frame
that is not of our dialect goes to a foreign reader (`vaelii.impl.foreign`). Out of scope:
the `:pg-memory` variant. See docs/storage.md, "Importing a dump".
raw docstring

vaelii.impl.io.snapshot

A snapshot of the KB's derived state, and the two-op sink it is written through.

Derived state — the index, and (from 0.9.0's second thread) the taxonomy and the JTMS labels — is rebuilt from the records on every open by reindex / recover, at a cost that is O(records). A snapshot is a cache of that rebuild: written once, read back on the next open, and installed instead of recomputed. It is never a source of truth. Its only failure mode is "recompute", never "wrong answer", which is what keeps it clear of order independence — a snapshot records the fingerprint of the exact records it was derived from, and a mismatch takes the slow path (reindex/recover), never a stale belief.

One image, many sinks

There is more than one place a snapshot might live — a directory of files, a Postgres blob, memory for a test — and letting each invent its own serialization is the drift the shared [key value] index projection was created to avoid one layer up. So the shape is one sink:

  • a snapshot is a set of named sections plus a manifest;
  • a SnapshotSink knows only how to write a named section (a constant-memory stream of frames) and to commit a manifest; a SnapshotSource knows how to read a section back and to read the manifest;
  • the projections (the index's [key value], and later the taxonomy's edges and the JTMS's labels) and the validity check (decision) live here, above the sink, written once;
  • the targets — file-sink / file-source over vaelii.impl.io.frames, and memory-medium for a test — live below it, each a small adapter.

A section written to one sink loads from another, the same property p/index-entries / p/index-load already give the index across backends.

The protocol has out-of-tree implementations, so it is a published shape and not a private one. vaelii-postgres (vaelii.postgres.snapshot/pg-sink / pg-source) and vaelii-sqlite (vaelii.sqlite.snapshot/sqlite-sink / sqlite-source) each implement both protocols and drive them through save-index! / load-index!, and both suites cross-load a section against memory-medium to check the two targets agree. What is reached only from this repo's own tests is file-sink / file-source: the reference target, and the one that shows an implementer what a section and a manifest have to be.

The manifest is the commit point

It is written last, exactly as a dump's meta.edn is: a half-written image has no manifest, so read-manifest returns nil and the image is never offered — the caller rebuilds. Opening a file-sink deletes any existing manifest first, so the crash window in which an old manifest could describe half-rewritten sections does not exist.

Validity is the whole design

decision is lifted from vaelii.impl.disk.index-snapshot/decision: one reason per mismatch class, and any non-nil reason discards the whole image and falls back to a rebuild. The classes:

  • :absent — no manifest (missing, or a save that never committed);
  • :layout-changed — the snapshot format version, or kv/index-layout-version, is not this build's (an index in another layout reads as empty rather than wrong, which is the undiagnosable failure this forecloses);
  • :records-differ — the manifest's records fingerprint is not the store's now (a different KB, or the same one after a write);
  • :entries-truncated — a section is short or unreadable, caught while installing (a torn nippy chunk is indistinguishable from a clean EOF, so truncation shows only as a frame count below what the manifest recorded).

There is no :byte-order class here, though index-snapshot has one: that image writes raw little-endian int runs, so an image from a machine of the other endianness must be refused; these sections are nippy frames, which are endian-neutral, so a byte-order mismatch cannot arise and the format version subsumes any encoding change.

The records fingerprint is vaelii.impl.io.fingerprint's records digest, which ports: a :disk store hashes its slots, a :memory store folds its records, a SQL store answers a count — and all compare against the same manifest number. Which digest a caller passes is its business; the sink only compares.

A **snapshot** of the KB's derived state, and the two-op *sink* it is written through.

Derived state — the index, and (from 0.9.0's second thread) the taxonomy and the JTMS
labels — is rebuilt from the records on every open by `reindex` / `recover`, at a cost
that is O(records).  A snapshot is a *cache* of that rebuild: written once, read back on
the next open, and installed instead of recomputed.  It is never a source of truth.  Its
only failure mode is "recompute", never "wrong answer", which is what keeps it clear of
order independence — a snapshot records the fingerprint of the exact records it was
derived from, and a mismatch takes the slow path (`reindex`/`recover`), never a stale
belief.

## One image, many sinks

There is more than one place a snapshot might live — a directory of files, a Postgres
blob, memory for a test — and letting each invent its own serialization is the drift the
shared `[key value]` index projection was created to avoid one layer up.  So the shape is
one **sink**:

* a snapshot is a set of **named sections** plus a **manifest**;
* a `SnapshotSink` knows only how to *write a named section* (a constant-memory stream of
  frames) and to *commit a manifest*; a `SnapshotSource` knows how to *read a section back*
  and to *read the manifest*;
* the **projections** (the index's `[key value]`, and later the taxonomy's edges and the
  JTMS's labels) and the **validity check** (`decision`) live here, above the sink, written
  once;
* the **targets** — `file-sink` / `file-source` over `vaelii.impl.io.frames`, and
  `memory-medium` for a test — live below it, each a small adapter.

A section written to one sink loads from another, the same property `p/index-entries` /
`p/index-load` already give the index across backends.

**The protocol has out-of-tree implementations, so it is a published shape and not a private
one.**  `vaelii-postgres` (`vaelii.postgres.snapshot/pg-sink` / `pg-source`) and
`vaelii-sqlite` (`vaelii.sqlite.snapshot/sqlite-sink` / `sqlite-source`) each implement
both protocols and drive them through `save-index!` / `load-index!`, and both suites
cross-load a section against `memory-medium` to check the two targets agree.  What is
reached only from this repo's own tests is `file-sink` / `file-source`: the reference
target, and the one that shows an implementer what a section and a manifest have to be.

## The manifest is the commit point

It is written **last**, exactly as a dump's `meta.edn` is: a half-written image has no
manifest, so `read-manifest` returns nil and the image is never offered — the caller
rebuilds.  Opening a `file-sink` deletes any existing manifest first, so the crash window in
which an old manifest could describe half-rewritten sections does not exist.

## Validity is the whole design

`decision` is lifted from `vaelii.impl.disk.index-snapshot/decision`: one reason per
mismatch class, and any non-nil reason discards the *whole* image and falls back to a
rebuild.  The classes:

* `:absent`          — no manifest (missing, or a save that never committed);
* `:layout-changed`  — the snapshot format version, or `kv/index-layout-version`, is not
                       this build's (an index in another layout reads as *empty* rather
                       than wrong, which is the undiagnosable failure this forecloses);
* `:records-differ`  — the manifest's records fingerprint is not the store's now (a
                       different KB, or the same one after a write);
* `:entries-truncated` — a section is short or unreadable, caught while installing (a torn
                       nippy chunk is indistinguishable from a clean EOF, so truncation shows only as a
                       frame count below what the manifest recorded).

There is no `:byte-order` class here, though `index-snapshot` has one: that image writes
raw little-endian `int` runs, so an image from a machine of the other endianness must be
refused; these sections are nippy frames, which are endian-neutral, so a byte-order
mismatch cannot arise and the format version subsumes any encoding change.

The records fingerprint is `vaelii.impl.io.fingerprint`'s records digest, which
**ports**: a `:disk` store hashes its slots, a `:memory` store folds its records, a SQL
store answers a count — and all compare against the same manifest number.  Which digest
a caller passes is its business; the sink only compares.
raw docstring

vaelii.impl.io.text

The text KB format, read and written: one file per context, one s-expression per sentence, the format the shipped ontology is authored in (resources/kb/, vaelii.host.seed). Every form is read with clojure.edn; the file name is the context. peel-form takes the (set/monotonic S) and (set/inert S) wrappers off a whole form. Text holds premises and inert sentexes in the author's spelling, with a handle as the (ist Cx S) reference it resolves from; the export dump (io.export, io.import) holds every record at its own handle. See docs/kbs.md, "Text you exported yourself".

The text KB format, read and written: one file per context, one s-expression per
sentence, the format the shipped ontology is authored in (`resources/kb/`,
`vaelii.host.seed`). Every form is read with `clojure.edn`; the file name is the context.
`peel-form` takes the `(set/monotonic S)` and `(set/inert S)` wrappers off a whole form.
Text holds premises and inert sentexes in the author's spelling, with a handle as the
`(ist Cx S)` reference it resolves from; the export dump (`io.export`, `io.import`) holds
every record at its own handle.  See docs/kbs.md, "Text you exported yourself".
raw docstring

vaelii.impl.io.thaw

The class-name check on every nippy thaw the engine runs over a file.

A frozen nippy value can name a class in three of its type ids, and reading one resolves that name and builds an instance of it: a record frame (Class/forName, then the static create), a deftype frame (Class/forName, then the first public constructor over the fields that follow), and a Serializable frame (an ObjectInputStream over the bytes that follow). nippy 3.9.0 checks the third against *thaw-serializable-allowlist* and the first two against nothing at all.

Every file the engine reads is untrusted input — a store directory or a dump arrives from wherever an operator copied it — so all three are checked here, and the check is the tightest one there is:

A vaelii frame never carries a class name.

That is already the export format's stated rule (vaelii.impl.io.export), and the disk codec writes a record's fields positionally for a size reason (vaelii.impl.disk.codec), so nothing the engine writes states a class either. A frame that names one therefore came from somewhere else, and allowed-classes is empty: the name is refused (:disallowed-class) before the class is resolved.

The Serializable allowlist is bound here rather than inherited. nippy's default is a safe set, but it lives in a dynamic var an embedding application is invited to widen — nippy documents allow-and-record-any-serializable-class-unsafe for exactly that migration — and a host that widened it would widen the engine's file readers with it. Binding it per read makes the entry point this namespace's rather than the host's.

How the first two are checked, since nippy exposes no hook for them: the readers taoensso.nippy.io dispatches to are vars, and this namespace installs a checked reader in place of each. A refusal is thrown from there, which is before nippy's own try — so it travels rather than becoming the {:nippy/unthawable …} placeholder a failed resolution otherwise reads as, and no class is loaded on the way. The replacement calls straight through to the reader it replaced unless with-guard is in force, so a host application thawing its own records in this JVM is unaffected.

What holds that wrap to the release it was written against is expected-nippy-version: three internals are reached into here, and a bump that keeps their names while routing deserialization around them would narrow this entry point without reddening anything. So the version is checked at load and the namespace refuses to come up against another one.

The class-name check on every nippy thaw the engine runs over a file.

A frozen nippy value can **name a class** in three of its type ids, and reading one
resolves that name and builds an instance of it: a record frame (`Class/forName`, then
the static `create`), a deftype frame (`Class/forName`, then the first public
constructor over the fields that follow), and a `Serializable` frame (an
`ObjectInputStream` over the bytes that follow).  nippy 3.9.0 checks the third against
`*thaw-serializable-allowlist*` and the first two against nothing at all.

Every file the engine reads is **untrusted input** — a store directory or a dump
arrives from wherever an operator copied it — so all three are checked here, and the
check is the tightest one there is:

> **A vaelii frame never carries a class name.**

That is already the export format's stated rule (`vaelii.impl.io.export`), and the
disk codec writes a record's fields **positionally** for a size reason
(`vaelii.impl.disk.codec`), so nothing the engine writes states a class either.  A
frame that names one therefore came from somewhere else, and `allowed-classes` is
empty: the name is refused (`:disallowed-class`) before the class is resolved.

**The `Serializable` allowlist is bound here rather than inherited.**  nippy's default is
a safe set, but it lives in a dynamic var an embedding application is invited to
widen — nippy documents `allow-and-record-any-serializable-class-unsafe` for exactly
that migration — and a host that widened it would widen the engine's file readers with
it.  Binding it per read makes the entry point this namespace's rather than the host's.

**How the first two are checked**, since nippy exposes no hook for them: the readers
`taoensso.nippy.io` dispatches to are vars, and this namespace installs a checked
reader in place of each.  A refusal is thrown from there, which is *before* nippy's own
`try` — so it travels rather than becoming the `{:nippy/unthawable …}` placeholder a
failed resolution otherwise reads as, and no class is loaded on the way.  The
replacement calls straight through to the reader it replaced unless `with-guard` is in
force, so a host application thawing its own records in this JVM is unaffected.

**What holds that wrap to the release it was written against** is
`expected-nippy-version`: three internals are reached into here, and a bump that keeps
their names while routing deserialization around them would narrow this entry point without
reddening anything.  So the version is checked at load and the namespace refuses to
come up against another one.
raw docstring

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
←Move to previous article
→Move to next article
Ctrl+/Jump to the search field
× close