A snapshot of the KB's derived state, and the two-op sink it is written through.
Derived state — the index, and (from 0.9.0's second thread) the taxonomy and the JTMS
labels — is rebuilt from the records on every open by reindex / recover, at a cost
that is O(records). A snapshot is a cache of that rebuild: written once, read back on
the next open, and installed instead of recomputed. It is never a source of truth. Its
only failure mode is "recompute", never "wrong answer", which is what keeps it clear of
order independence — a snapshot is stamped to the exact records it was derived from, and a
mismatch takes the slow path (reindex/recover), never a stale belief.
There is more than one place a snapshot might live — a directory of files, a Postgres
blob, memory for a test — and letting each invent its own serialization is the drift the
shared [key value] index projection was created to avoid one layer up. So the shape is
one sink:
SnapshotSink knows only how to write a named section (a constant-memory stream of
frames) and to commit a manifest; a SnapshotSource knows how to read a section back
and to read the manifest;[key value], and later the taxonomy's edges and the
JTMS's labels) and the validity check (decision) live here, above the sink, written
once;file-sink / file-source over vaelii.impl.io.frames, and
memory-medium for a test — live below it, each a small adapter.A section written to one sink loads from another, the same property p/index-entries /
p/index-load already give the index across backends.
The seam has out-of-tree implementations, so it is a published shape and not a private
one. vaelii-postgres (vaelii.postgres.snapshot/pg-sink / pg-source) and
vaelii-sqlite (vaelii.sqlite.snapshot/sqlite-sink / sqlite-source) each implement
both protocols and drive them through save-index! / load-index!, and both suites
cross-load a section against memory-medium to check the two targets agree. What is
reached only from this repo's own tests is file-sink / file-source: the reference
target, and the one that shows an implementer what a section and a manifest have to be.
It is written last, exactly as a dump's meta.edn is: a half-written image has no
manifest, so read-manifest returns nil and the image is never offered — the caller
rebuilds. Opening a file-sink deletes any existing manifest first, so the window in
which an old manifest could describe half-rewritten sections does not exist.
decision is lifted from vaelii.impl.disk.index-snapshot/decision: one reason per
mismatch class, and any non-nil reason discards the whole image and falls back to a
rebuild. The classes:
:absent — no manifest (missing, or a save that never committed);:layout-changed — the snapshot format version, or kv/index-layout-version, is not
this build's (an index in another layout reads as empty rather
than wrong, which is the undiagnosable failure this forecloses);:records-differ — the manifest's records stamp is not the store's now (a different
KB, or the same one after a write);:entries-truncated — a section is short or unreadable, caught while installing (a torn
nippy chunk reads as a clean EOF, so truncation shows only as a
frame count below what the manifest recorded).There is no :byte-order class here, though index-snapshot has one: that image writes
raw little-endian int runs, so an image from a machine of the other endianness must be
refused; these sections are nippy frames, which are endian-neutral, so a byte-order
mismatch cannot arise and the format version subsumes any encoding change.
The stamp is vaelii.impl.io.fingerprint's records digest, which ports: a :disk
store hashes its slots, a :memory store folds its records, a SQL store answers a count
— and all compare against the same manifest number. Which digest a caller passes is its
business; the sink only compares.
A **snapshot** of the KB's derived state, and the two-op *sink* it is written through.
Derived state — the index, and (from 0.9.0's second thread) the taxonomy and the JTMS
labels — is rebuilt from the records on every open by `reindex` / `recover`, at a cost
that is O(records). A snapshot is a *cache* of that rebuild: written once, read back on
the next open, and installed instead of recomputed. It is never a source of truth. Its
only failure mode is "recompute", never "wrong answer", which is what keeps it clear of
order independence — a snapshot is stamped to the exact records it was derived from, and a
mismatch takes the slow path (`reindex`/`recover`), never a stale belief.
## One image, many sinks
There is more than one place a snapshot might live — a directory of files, a Postgres
blob, memory for a test — and letting each invent its own serialization is the drift the
shared `[key value]` index projection was created to avoid one layer up. So the shape is
one **sink**:
* a snapshot is a set of **named sections** plus a **manifest**;
* a `SnapshotSink` knows only how to *write a named section* (a constant-memory stream of
frames) and to *commit a manifest*; a `SnapshotSource` knows how to *read a section back*
and to *read the manifest*;
* the **projections** (the index's `[key value]`, and later the taxonomy's edges and the
JTMS's labels) and the **validity check** (`decision`) live here, above the sink, written
once;
* the **targets** — `file-sink` / `file-source` over `vaelii.impl.io.frames`, and
`memory-medium` for a test — live below it, each a small adapter.
A section written to one sink loads from another, the same property `p/index-entries` /
`p/index-load` already give the index across backends.
**The seam has out-of-tree implementations, so it is a published shape and not a private
one.** `vaelii-postgres` (`vaelii.postgres.snapshot/pg-sink` / `pg-source`) and
`vaelii-sqlite` (`vaelii.sqlite.snapshot/sqlite-sink` / `sqlite-source`) each implement
both protocols and drive them through `save-index!` / `load-index!`, and both suites
cross-load a section against `memory-medium` to check the two targets agree. What is
reached only from this repo's own tests is `file-sink` / `file-source`: the reference
target, and the one that shows an implementer what a section and a manifest have to be.
## The manifest is the commit point
It is written **last**, exactly as a dump's `meta.edn` is: a half-written image has no
manifest, so `read-manifest` returns nil and the image is never offered — the caller
rebuilds. Opening a `file-sink` deletes any existing manifest first, so the window in
which an old manifest could describe half-rewritten sections does not exist.
## Validity is the whole design
`decision` is lifted from `vaelii.impl.disk.index-snapshot/decision`: one reason per
mismatch class, and any non-nil reason discards the *whole* image and falls back to a
rebuild. The classes:
* `:absent` — no manifest (missing, or a save that never committed);
* `:layout-changed` — the snapshot format version, or `kv/index-layout-version`, is not
this build's (an index in another layout reads as *empty* rather
than wrong, which is the undiagnosable failure this forecloses);
* `:records-differ` — the manifest's records stamp is not the store's now (a different
KB, or the same one after a write);
* `:entries-truncated` — a section is short or unreadable, caught while installing (a torn
nippy chunk reads as a clean EOF, so truncation shows only as a
frame count below what the manifest recorded).
There is no `:byte-order` class here, though `index-snapshot` has one: that image writes
raw little-endian `int` runs, so an image from a machine of the other endianness must be
refused; these sections are nippy frames, which are endian-neutral, so a byte-order
mismatch cannot arise and the format version subsumes any encoding change.
The stamp is `vaelii.impl.io.fingerprint`'s records digest, which **ports**: a `:disk`
store hashes its slots, a `:memory` store folds its records, a SQL store answers a count
— and all compare against the same manifest number. Which digest a caller passes is its
business; the sink only compares.(decision m stamp)Why the image described by manifest m cannot be trusted against stamp (the store's
current records fingerprint), or nil when it can. The medium-agnostic reading of the
question vaelii.impl.disk.index-snapshot/decision answers for a mapped image — that
one draws its own truncation and platform classes from the files it maps and does not
come through here. The image is a cache, so any non-nil reason discards the whole of
it and the caller rebuilds. :entries-truncated is not decided
here — a nippy stream's truncation reads as a clean EOF, so it can only be caught while
the section is installed, against the count the manifest recorded.
Why the image described by manifest `m` cannot be trusted against `stamp` (the store's current records fingerprint), or nil when it can. The medium-agnostic reading of the question `vaelii.impl.disk.index-snapshot/decision` answers for a mapped image — that one draws its own truncation and platform classes from the files it maps and does not come through here. The image is a cache, so any non-nil reason discards the whole of it and the caller rebuilds. `:entries-truncated` is not decided here — a nippy stream's truncation reads as a clean EOF, so it can only be caught while the section is installed, against the count the manifest recorded.
(file-sink root)(file-sink root
{:keys [compression chunk-size]
:or {compression :none chunk-size 10000}})A sink writing sections and a manifest.edn under root. Opening it deletes any
existing manifest, so from here until commit! the image reads as absent — the window
in which a stale manifest could describe half-rewritten sections is closed by
construction. :compression defaults to :none (an image is a fast local cache, and
the CPU a codec costs is usually the wrong trade for it); :chunk-size to 10000.
Bare, though it deletes a file. The ! convention marks an operation the KB cannot
take back (docs/api.md), and a manifest is an artifact of a cache rather than stored
knowledge: dropping it costs a reindex, which is the fallback load-index! takes on
any decision reason anyway. Nothing a record holds is reachable through it.
And it belongs here rather than in save-index!. Invalidating before the first
section is a property of this target — a directory of files, where the manifest and
the sections are separate objects with a window between them. A sink whose commit is
one transactional write has no such window and needs no such step, and save-index!
sits above the seam and writes through whichever sink it is handed. Moving the step up
would mean a third protocol op every out-of-tree implementer had to grow, to say
something two of the three targets have nothing to say about.
A sink writing sections and a `manifest.edn` under `root`. Opening it deletes any existing manifest, so from here until `commit!` the image reads as absent — the window in which a stale manifest could describe half-rewritten sections is closed by construction. `:compression` defaults to `:none` (an image is a fast local cache, and the CPU a codec costs is usually the wrong trade for it); `:chunk-size` to 10000. **Bare, though it deletes a file.** The `!` convention marks an operation the KB cannot take back (`docs/api.md`), and a manifest is an artifact of a cache rather than stored knowledge: dropping it costs a `reindex`, which is the fallback `load-index!` takes on any `decision` reason anyway. Nothing a record holds is reachable through it. **And it belongs here rather than in `save-index!`.** Invalidating before the first section is a property of *this* target — a directory of files, where the manifest and the sections are separate objects with a window between them. A sink whose commit is one transactional write has no such window and needs no such step, and `save-index!` sits above the seam and writes through whichever sink it is handed. Moving the step up would mean a third protocol op every out-of-tree implementer had to grow, to say something two of the three targets have nothing to say about.
(file-source root)A read-only source over a directory file-sink wrote. Reads nothing until asked —
read-manifest for the validity check, read-section for the install.
A read-only source over a directory `file-sink` wrote. Reads nothing until asked — `read-manifest` for the validity check, `read-section` for the install.
The snapshot's own layout number, beside kv/index-layout-version (which says what the
section entries mean). Bump when a section's shape or the manifest's shape changes.
The snapshot's own layout number, beside `kv/index-layout-version` (which says what the section *entries* mean). Bump when a section's shape or the manifest's shape changes.
(index-frames index)The index's entries as [key value] frames — the portable projection every backend
shares (p/index-entries). This streams what the store holds rather than deriving
anything: a writer that called index-sentex would be reindex with extra steps,
writing an index it had just computed instead of the one the KB was answering from.
Each pair is normalized to a vector: the backends emit a mix of MapEntrys and plain
vectors (the map-backed stores yield entries, the tiered one yields vectors, the columnar
one yields both), and nippy gives MapEntry its own type id — so without this, two
byte-identical logical indexes froze to byte-different streams according to which backend
held them. Shared verbatim with the export dump (vaelii.impl.io.export), so a dump's
index and a standalone image are one projection rather than two.
The index's entries as `[key value]` frames — the portable projection every backend shares (`p/index-entries`). This streams what the store holds rather than deriving anything: a writer that called `index-sentex` would be `reindex` with extra steps, writing an index it had just computed instead of the one the KB was answering from. Each pair is normalized to a **vector**: the backends emit a mix of `MapEntry`s and plain vectors (the map-backed stores yield entries, the tiered one yields vectors, the columnar one yields both), and nippy gives `MapEntry` its own type id — so without this, two byte-identical logical indexes froze to byte-different streams according to which backend held them. Shared verbatim with the export dump (`vaelii.impl.io.export`), so a dump's index and a standalone image are one projection rather than two.
(index-manifest stamp entry-count)The manifest for an index image: the two format numbers, the records stamp it is valid against, and the section's frame count for the truncation check.
The manifest for an index image: the two format numbers, the records stamp it is valid against, and the section's frame count for the truncation check.
(index-mismatch meta stamp)The content-validity core an index cache shares wherever it is stored: the layout it is
keyed in, and the records it was derived from. Returns :layout-changed,
:records-differ, or nil. Each container adds its own classes around this — an image
(decision, below) adds :absent and a format-version check; a dump
(vaelii.impl.io.import/index-decision) adds :absent and :handles-remapped.
The mapped image is the one container that asks the same two questions inline
rather than through here (vaelii.impl.disk.index-snapshot/decision): it interleaves a
byte-order class between them, and orders the whole cond by what a reader needs told
first rather than by what is cheapest to ask.
The content-validity core an index cache shares wherever it is stored: the layout it is keyed in, and the records it was derived from. Returns `:layout-changed`, `:records-differ`, or nil. Each container adds its own classes around this — an image (`decision`, below) adds `:absent` and a format-version check; a dump (`vaelii.impl.io.import/index-decision`) adds `:absent` and `:handles-remapped`. The **mapped** image is the one container that asks the same two questions inline rather than through here (`vaelii.impl.disk.index-snapshot/decision`): it interleaves a byte-order class between them, and orders the whole cond by what a reader needs told first rather than by what is cheapest to ask.
The section name the index's [key value] projection is written under.
The section name the index's `[key value]` projection is written under.
(install-entries! index frames)Install a stream of [key value] frames into index in bounded batches, returning the
count installed. Checked at the end, not realized first — an index image is several
entries per record, so holding one to inspect it would cost more heap than the records
did. A frame that is not a [key value] pair throws; the caller's rebuild clears
whatever was installed.
Public because the two places an index arrives from a file — a snapshot section
(load-index!, below) and a dump's index/entries stream (vaelii.impl.io.import) —
are the same install and must stay one: same batch size, same refusal, same count to
check against the manifest that vouches for the stream.
Install a stream of `[key value]` frames into `index` in bounded batches, returning the count installed. Checked at the end, not realized first — an index image is several entries per record, so holding one to inspect it would cost more heap than the records did. A frame that is not a `[key value]` pair throws; the caller's rebuild clears whatever was installed. Public because the two places an index arrives from a file — a snapshot section (`load-index!`, below) and a dump's `index/entries` stream (`vaelii.impl.io.import`) — are the same install and must stay one: same batch size, same refusal, same count to check against the manifest that vouches for the stream.
(load-index! source index stamp)Validate the image behind source against stamp (the store's current records
fingerprint) and install its index section into index, or report why it was discarded.
Returns {:index :replayed :entries n} or {:index :rebuild :reason r} with r one of
:absent :layout-changed :records-differ :entries-truncated.
The caller reindexes on any :rebuild, which is always legal because the index is derived
state and this is a cache of it — and reindex clears the index first, so a partial install
left by a truncated section is wiped. index must be empty on entry, the same
contract p/index-load carries.
Validate the image behind `source` against `stamp` (the store's current records
fingerprint) and install its index section into `index`, or report why it was discarded.
Returns `{:index :replayed :entries n}` or `{:index :rebuild :reason r}` with `r` one of
`:absent :layout-changed :records-differ :entries-truncated`.
The caller reindexes on any `:rebuild`, which is always legal because the index is derived
state and this is a cache of it — and reindex clears the index first, so a partial install
left by a truncated section is wiped. `index` must be **empty** on entry, the same
contract `p/index-load` carries.(memory-medium)An in-heap snapshot medium, usable as both a SnapshotSink and a SnapshotSource.
Holds the whole image in RAM — the price of being a test double and a portability oracle,
not a scale target.
An in-heap snapshot medium, usable as both a `SnapshotSink` and a `SnapshotSource`. Holds the whole image in RAM — the price of being a test double and a portability oracle, not a scale target.
(save-index! sink index stamp)Write index's projection through sink as the index section, then commit a manifest
stamped with stamp (the records fingerprint). Returns the committed manifest.
The manifest is committed after the section, so a crash mid-write leaves an image
with no manifest, which load-index! reads as :absent and rebuilds.
Write `index`'s projection through `sink` as the index section, then commit a manifest stamped with `stamp` (the records fingerprint). Returns the committed manifest. The manifest is committed **after** the section, so a crash mid-write leaves an image with no manifest, which `load-index!` reads as `:absent` and rebuilds.
Where a snapshot's bytes go — a directory, a database, memory. Two ops: stream a named section, and commit the manifest that vouches for the lot.
Where a snapshot's bytes go — a directory, a database, memory. Two ops: stream a named section, and commit the manifest that vouches for the lot.
(commit! sink manifest)Write manifest as the completion marker — last, after every section, so an
image with no committed manifest is never offered.
Write `manifest` as the completion marker — **last**, after every section, so an image with no committed manifest is never offered.
(write-section! sink name frames)Write the lazy seq frames to the section named name, one chunk in memory at a
time. Returns the number of frames written.
Write the lazy seq `frames` to the section named `name`, one chunk in memory at a time. Returns the number of frames written.
Where a snapshot's bytes come from. Two ops mirroring the sink: read the manifest, and stream a named section back.
Where a snapshot's bytes come from. Two ops mirroring the sink: read the manifest, and stream a named section back.
(read-manifest source)The committed manifest map, or nil when the image is absent or was never committed.
The committed manifest map, or nil when the image is absent or was never committed.
(read-section source name)A constant-memory lazy seq of the frames in the section named name.
A constant-memory lazy seq of the frames in the section named `name`.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |