vaelii.impl.io.import — com.vaelii/vaelii 0.1.0

Import a vaelii export dump — a directory of record streams — into a pure KB, landing in exactly the state pure's own restart path (reindex / recover) already knows how to produce.

A dump is a directory whose meta.edn is the marker and schema; every other file is a nippy stream:

sentexes.nippy.stream        one field-map frame per sentex
justifications.nippy.stream  one per justification
provenance.nippy.stream      [sentex-handle map] per frame — optional

A :records+index dump also carries the index, as a cache that is used only when it can be proved to describe the records that were just stored (see the index below); otherwise the index is rebuilt, and the summary says which happened and why.

Framing. A chunked stream is a run of [int32 length][compressed chunk], each chunk a compression window over back-to-back nippy frames; a window stream is one compression window over the lot. Our own dumps state which (:framing); a foreign dump's is inferred from its own version line. Both are constant-memory lazy seqs.

A frame of our own dialect is a plain field map whose :sentence is already there — but a rule's set/*Rule wrappers and its variable names canonicalized into the record (:direction / :defeasible / :varmap), so both are written back around it before the constructor sees it. A frame that is not ours goes to a foreign reader (vaelii.impl.foreign), which is resolved at runtime and may not be in the build at all. The discrimination is on the frame, never on meta.edn's :dialect: a declaration is not an authority over the bytes beside it, and keying off the frame keeps a mixed dump readable.

Whatever the dialect, every sentence is re-canonicalized through this build's own constructor (res/kb-sentex). A stored canonical form is never trusted, not even our own: variable numbering, symmetric argument order and comparison folding belong to the reading build, and a record indexed under a key this build's lookup never reproduces would be silently unfindable.

Handles are preserved for a dump of ours: every record is stored at the handle the dump gave it, so a handle means the same thing either side of an export. Safe because the destination must be empty and because a store's counter clears any handle written that way (p/next-id) — without which the next assert would mint handle 1 again and overwrite the first imported record. Two things can still stop a handle landing as given, and neither is silent: a frame with no :id, and a frame whose canonical form is one already stored, which collapses onto that handle (a dedup this build is right to perform — two engine forms can canonicalize to one pure record — and the dump's numbering cannot survive it). Either makes the import :remapped, and then one old->new map carries the dump's ids across: justification references, and the (sentexHandle H) a meta-sentex embeds inside stored content, which rewrite-embedded-handles! rewrites in the sentence.

The index is replayed only when it can be proved to fit, and discarding it is always safe — which is what makes a cache out of what would otherwise be a risk. An index that does not match its records is worse than no index: every lookup then answers confidently and short, and nothing in the engine is positioned to notice. So all three of these must hold (index-decision):

• the entries are keyed in the layout this build reads (kv/index-layout-version);
• the fingerprint accumulated while storing equals the one written beside the entries (vaelii.impl.io.fingerprint) — accumulated, not recomputed, since a second pass over the records to validate a cache would cost more than the cache saves;
• the handles were preserved, so a posting names the record it named in the source.

Anything else rebuilds, at :info, with the reason named. A cache that silently stops being used is a cache nobody maintains.

The store-facing replay is written against pure's real seams: populate the record store with the re-canonicalized records + justifications + premise marks, then either install the dumped index (p/index-load) or rebuild it (reindex), then core/recover — which rebuilds the JTMS and the taxonomy from the records, so a replayed index shortcuts the index and nothing else. Out of scope: the :pg-memory variant.

export-format clj

The marker vaelii.impl.io.export writes. Version numbers alone cannot tell a dump of ours from a foreign one — ours starts at 1, which sits inside numbering somebody else was already using — so the dialect is named rather than deduced.

import-dump clj

(import-dump kb dir)

(import-dump kb
             dir
             {:keys [belief? report-every on-progress]
              :or {belief? true report-every 500000 on-progress no-progress}})

Import a vaelii export dump from dir into the (empty) kb — one vaelii.impl.io.export wrote, or one in a foreign dialect this build still carries a reader for (vaelii.impl.foreign).

With {:belief? true} (the default) it lands in the state pure's own restart path produces: the record store populated from the re-canonicalized records + the justifications + premise marks, the index rebuilt (reindex), belief recovered (recover).

With {:belief? false} it stores + indexes every sentex but skips what rests on what, the premise marks, and recover — the whole corpus is browsable / findable / countable but not belief-queryable. This is the path for a corpus past what recover's per-node relabel and the in-RAM JTMS scale to (recover resettles a region per premise and per justification — millions of them would not finish).

Reads meta.edn first and dispatches on it: the version is gated against its own dialect's numbering, the destination must be empty, and only the :records / :records+index variants are read. :pg-memory and an unknown variant throw.

opts: {:belief? bool :report-every n :on-progress f}. :on-progress is called every report-every frames with {:phase :done :total} — the phases a dump has, in order: :sentexes, then (on the belief path) :justifications for one of ours or whatever phase a foreign reader reports, then either :index-entries (a replay) or :reindex (a rebuild). A callback that throws aborts the import where it stands, which is how a caller cancels one; the KB is left holding what had already landed, since an import is not a transaction.

The summary reports the :dialect read and the :handle-policy used — :preserved (every record is at the handle the dump gave it) or :remapped (with :collapsed, how many frames canonicalized onto a handle already stored) — and what became of the index: {:index :replayed :entries n} or {:index :rebuilt :reason r} (:absent / :layout-changed / :handles-remapped / :records-differ / :entries-truncated). A caller cannot see any of it for itself, and the first two are load-bearing for the third: an index entry is a posting of handles, so it can only be replayed over a :preserved import.

:naming is the count of what this import stored that assert would have refused — {:checked n :refused n :by-class {…}}, logged as a warning when it is not zero. Neither import path runs the naming check (both build records directly, which is what makes a corpus this size loadable at all), so the disagreement between the two doors is closed by reporting it: the operator who chose the bulk path learns the number while the records go past, rather than from a re-assertion that throws a year later.

read-meta clj

(read-meta dir)

Read a dump's meta.edn (the marker + schema) without loading any records.

supported-export-versions clj

Export-format versions this build reads.