Write a KB out as a portable export dump — a directory holding the record store's three streams, in a format that survives a backend change, an index-representation change, and a record class rename.
The :disk store directory looks like an archive and is not one: it holds frozen
records, so renaming a record class makes every frame in it thaw to a
{:nippy/unthawable …} placeholder — silently, because a placeholder is a perfectly
good map. Hence the one non-negotiable rule of this format:
A frame never carries a class name.
A frame is the record's field map — (into {} record), a plain map — so a rename
changes nothing a dump holds, and vaelii.impl.io.import's field-map already
accepts one.
What a dump holds is what the record store holds, because everything else the KB has
is derived from it: sentexes, justifications, and per-handle provenance. A premise
needs no stream of its own — a premise is a sentex whose :strength is non-nil, a
field on the record in both backends, so the mark rides along with it. The index is
a cache (reindex), the taxonomy and the TMS labels are recomputed (recover).
<dump>/
meta.edn the marker, the schema, and the counts
sentexes.nippy.stream one frame per sentex
justifications.nippy.stream one frame per justification
provenance.nippy.stream [handle provenance-map] per frame (omitted when none)
index/entries.nippy.stream [key value] per frame — only in :records+index
index/index.edn the layout version + the records fingerprint
The index is optional and always discardable. :variant :records+index writes it
as well, in the [structured-key value] projection every index backend shares
(p/index-entries), so an index written by one backend loads into another. It is a
cache: a reader replays it only when it can prove the entries were derived from
exactly the records beside them (vaelii.impl.io.fingerprint), and rebuilds otherwise.
That is what makes writing it safe — an index that does not match its records is worse
than no index, because every lookup then answers confidently and short.
Framing is the chunked layout vaelii.impl.io.import/read-chunked-seq reads: a
run of [int32 length][compressed chunk], each chunk an independent compression
window over back-to-back nippy frames. Constant memory on both sides — the writer
holds one chunk, never the corpus. meta.edn states the framing rather than
implying it through a version number, because the engine's dumps number their own
format and a reader must never have to guess.
meta.edn is written last, which makes it double as the completion marker:
vaelii.impl.catalog/classify keys on it, so a half-written or cancelled export is
not offered as loadable. That is the one ordering constraint in the whole format.
Export from a KB nobody is writing: the walk fetches record by record, and pure's single-writer contract offers no snapshot to walk instead.
Write a KB out as a portable **export dump** — a directory holding the record
store's three streams, in a format that survives a backend change, an
index-representation change, and a record class rename.
The `:disk` store directory looks like an archive and is not one: it holds frozen
*records*, so renaming a record class makes every frame in it thaw to a
`{:nippy/unthawable …}` placeholder — silently, because a placeholder is a perfectly
good map. Hence the one non-negotiable rule of this format:
> **A frame never carries a class name.**
A frame is the record's **field map** — `(into {} record)`, a plain map — so a rename
changes nothing a dump holds, and `vaelii.impl.io.import`'s `field-map` already
accepts one.
What a dump holds is what the record store holds, because everything else the KB has
is derived from it: sentexes, justifications, and per-handle provenance. A premise
needs no stream of its own — a premise *is* a sentex whose `:strength` is non-nil, a
field on the record in both backends, so the mark rides along with it. The index is
a cache (`reindex`), the taxonomy and the TMS labels are recomputed (`recover`).
<dump>/
meta.edn the marker, the schema, and the counts
sentexes.nippy.stream one frame per sentex
justifications.nippy.stream one frame per justification
provenance.nippy.stream [handle provenance-map] per frame (omitted when none)
index/entries.nippy.stream [key value] per frame — only in :records+index
index/index.edn the layout version + the records fingerprint
**The index is optional and always discardable.** `:variant :records+index` writes it
as well, in the `[structured-key value]` projection every index backend shares
(`p/index-entries`), so an index written by one backend loads into another. It is a
*cache*: a reader replays it only when it can prove the entries were derived from
exactly the records beside them (`vaelii.impl.io.fingerprint`), and rebuilds otherwise.
That is what makes writing it safe — an index that does not match its records is worse
than no index, because every lookup then answers confidently and short.
**Framing** is the chunked layout `vaelii.impl.io.import/read-chunked-seq` reads: a
run of `[int32 length][compressed chunk]`, each chunk an independent compression
window over back-to-back nippy frames. Constant memory on both sides — the writer
holds one chunk, never the corpus. `meta.edn` *states* the framing rather than
implying it through a version number, because the engine's dumps number their own
format and a reader must never have to guess.
**`meta.edn` is written last**, which makes it double as the completion marker:
`vaelii.impl.catalog/classify` keys on it, so a half-written or cancelled export is
not offered as loadable. That is the one ordering constraint in the whole format.
Export from a KB nobody is writing: the walk fetches record by record, and pure's
single-writer contract offers no snapshot to walk instead.(export! kb dir)(export! kb
dir
{:keys [variant compression chunk-size on-progress provenance?]
:or {variant :records
compression :gzip
chunk-size 10000
on-progress no-progress
provenance? true}})Export kb's records to dir as a portable dump, and return a summary:
{:variant :records :sentexes n :justifications n :provenance n
:index-entries n :bytes n :elapsed-ms n :dir "…"}
opts: {:variant :records|:records+index :compression :gzip|:xz|:none :chunk-size n :provenance? bool :on-progress f} (defaults :records, :gzip, 10000 and true;
:xz trades write speed for
a materially smaller archive — compression is a stream wrapper around a whole chunk,
orthogonal to the nippy encoding of the frames inside it). :on-progress is called
with {:phase :done :total} — the shape a corpus reader's load-dir!,
io.import/import-dump and io.generate/load-into report, so
vaelii.impl.catalog draws a bar from it — at every chunk boundary, in phase order
:sentexes, :justifications, :provenance, :index-entries, :meta. A callback
that throws is how a caller cancels: the throw propagates out of the phase it
interrupted, leaving a directory with no meta.edn, which is not a loadable dump.
Handles are exported as they stand — a justification names its antecedents by
handle, and an exceptWhen meta-sentex names its rule by handle inside a stored
sentence, so a dump that renumbered would have to rewrite content.
:records+index additionally writes index/, whose entries are only ever a cache: the
fingerprint beside them is computed on the same walk that writes the sentex frames,
so a reader can tell whether they describe the records it just stored, and rebuild if
not. It is the expensive half of the dump and then some — measured at 6.2 entries per
record and 3.8× the records' bytes, for a load twice as fast —
so it is opt-in rather than the default.
dir must not exist or must be empty (:type :not-empty), and meta.edn is
written last. The provenance stream is omitted entirely when no handle carries
provenance; the reader already treats it as optional.
:provenance? false omits it even when handles do carry it, and it is not a
micro-optimization: provenance is an open per-handle map an application layers whatever
it likes into, so it has no size bound of its own and can dominate the records it
annotates. Measured on the converted engine KB — 10.2M sentexes — the provenance
stream is 317.9 MB of a 556 MB dump, 57%, larger than the records, and what it
holds there is the extraction pipeline's own bookkeeping rather than anything a reader
of the ontology wants. Belief never reads provenance and the importer treats the
stream as optional, so a dump without it is a complete KB, just an unannotated one;
that makes this the right knob for a dump meant to be downloaded, and the wrong one for
a backup.
Export `kb`'s records to `dir` as a portable dump, and return a summary:
{:variant :records :sentexes n :justifications n :provenance n
:index-entries n :bytes n :elapsed-ms n :dir "…"}
`opts`: `{:variant :records|:records+index :compression :gzip|:xz|:none :chunk-size n
:provenance? bool :on-progress f}` (defaults `:records`, `:gzip`, 10000 and true;
`:xz` trades write speed for
a materially smaller archive — compression is a stream wrapper around a whole chunk,
orthogonal to the nippy encoding of the frames inside it). `:on-progress` is called
with `{:phase :done :total}` — the shape a corpus reader's `load-dir!`,
`io.import/import-dump` and `io.generate/load-into` report, so
`vaelii.impl.catalog` draws a bar from it — at every chunk boundary, in phase order
`:sentexes`, `:justifications`, `:provenance`, `:index-entries`, `:meta`. A callback
that **throws** is how a caller cancels: the throw propagates out of the phase it
interrupted, leaving a directory with no `meta.edn`, which is not a loadable dump.
Handles are exported as they stand — a justification names its antecedents by
handle, and an `exceptWhen` meta-sentex names its rule by handle *inside a stored
sentence*, so a dump that renumbered would have to rewrite content.
`:records+index` additionally writes `index/`, whose entries are only ever a cache: the
fingerprint beside them is computed on the **same walk** that writes the sentex frames,
so a reader can tell whether they describe the records it just stored, and rebuild if
not. It is the expensive half of the dump and then some — measured at 6.2 entries per
record and 3.8× the records' bytes, for a load twice as fast —
so it is opt-in rather than the default.
`dir` must not exist or must be empty (`:type :not-empty`), and `meta.edn` is
written **last**. The provenance stream is omitted entirely when no handle carries
provenance; the reader already treats it as optional.
**`:provenance? false` omits it even when handles do carry it**, and it is not a
micro-optimization: provenance is an open per-handle map an application layers whatever
it likes into, so it has no size bound of its own and can dominate the records it
annotates. Measured on the converted engine KB — 10.2M sentexes — the provenance
stream is 317.9 MB of a 556 MB dump, **57%, larger than the records**, and what it
holds there is the extraction pipeline's own bookkeeping rather than anything a reader
of the ontology wants. Belief never reads provenance and the importer treats the
stream as optional, so a dump without it is a complete KB, just an unannotated one;
that makes this the right knob for a dump meant to be downloaded, and the wrong one for
a backup.The dump's own marker. The engine's dumps carry no :format and number their
format on a line of their own, so ours announces whose it is rather than colliding
with a foreign version space.
The dump's own marker. The engine's dumps carry no `:format` and number their format on a line of their own, so ours announces whose it is rather than colliding with a foreign version space.
Version 1: field-map frames, chunked framing, handles preserved.
Version 1: field-map frames, chunked framing, handles preserved.
What a dump can hold. :records is the whole KB — everything else is derived from
the records and rebuilt on the way in. :records+index adds the index as a cache the
reader may or may not use; it never adds knowledge, only speed.
What a dump can hold. `:records` is the whole KB — everything else is derived from the records and rebuilt on the way in. `:records+index` adds the index as a cache the reader may or may not use; it never adds knowledge, only speed.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |