vaelii.impl.io.import — com.vaelii/vaelii 0.8.0

Import a vaelii export dump — a directory of record streams — into a KB, landing in exactly the state the engine's own restart path (reindex / recover) already knows how to produce.

A dump is a directory whose meta.edn is the marker and schema; every other file is a nippy stream:

sentexes.nippy.stream        one field-map frame per sentex
justifications.nippy.stream  one per justification
provenance.nippy.stream      [handle map] per frame — optional

A :records+index dump also carries the index, as a cache that is used only when it can be proved to describe the records that were just stored (see the index below); otherwise the index is rebuilt, and the summary says which happened and why.

Framing. A chunked stream is a run of [int32 length][compressed chunk], each chunk a compression window over back-to-back nippy frames; a window stream is one compression window over the lot. Our own dumps state which (:framing); a foreign dump's is inferred from its own version line. Both are constant-memory lazy seqs.

A frame of our own dialect is a plain field map whose :sentence is already there — but a rule's set/*Rule wrappers and its variable names canonicalized into the record (:direction / :defeasible / :varmap), so both are written back around it before the constructor sees it. A frame that is not ours goes to a foreign reader (vaelii.impl.foreign), which is resolved at runtime and may not be in the build at all. The discrimination is on the frame, never on meta.edn's :dialect: a declaration is not an authority over the bytes beside it, and keying off the frame keeps a mixed dump readable.

Whatever the dialect, every sentence is re-canonicalized through this build's own constructor (res/kb-sentex). A stored canonical form is never trusted, not even our own: variable numbering, symmetric argument order and comparison folding belong to the reading build, and a record indexed under a key this build's lookup never reproduces would be silently unfindable.

Handles are preserved for a dump of ours: every record is stored at the handle the dump gave it, so a handle means the same thing either side of an export. Safe because the destination must be empty and because a store's counter clears any handle written that way (p/next-id) — without which the next assert would mint handle 1 again and overwrite the first imported record. Two things can still stop a handle landing as given, and neither is silent: a frame with no :id, and a frame whose canonical form is one already stored, which collapses onto that handle (a dedup this build is right to perform — two engine forms can canonicalize to one stored record — and the dump's numbering cannot survive it). Either makes the import :remapped, and then one old->new map carries the dump's ids across: justification references, and the (sentexHandle H) a meta-sentex embeds inside stored content, which rewrite-embedded-handles! rewrites in the sentence. A meta-sentex whose embedded handle cannot be resolved is dropped, and the drop reaches the map as well as the store (forget-deleted): a dump id whose record is gone has to stop resolving, or the references to it resolve to a handle nothing is stored at.

The index is replayed only when it can be proved to fit, and discarding it is always safe — which is what makes a cache out of what would otherwise be a risk. An index that does not match its records is worse than no index: every lookup then answers confidently and short, and nothing in the engine is positioned to notice. So all three of these must hold (index-decision):

• the entries are keyed in the layout this build reads (kv/index-layout-version);
• the fingerprint accumulated while storing equals the one written beside the entries (vaelii.impl.io.fingerprint) — accumulated, not recomputed, since a second pass over the records to validate a cache would cost more than the cache saves;
• the handles were preserved, so a posting names the record it named in the source.

Anything else rebuilds, at :info, with the reason named. A cache that silently stops being used is a cache nobody maintains.

The store-facing replay is written against the engine's real seams: populate the record store with the re-canonicalized records + justifications + premise marks, then either install the dumped index (p/index-load) or rebuild it (reindex), then core/recover — which rebuilds the JTMS and the taxonomy from the records, so a replayed index shortcuts the index and nothing else. Out of scope: the :pg-memory variant.

belief-modes clj

What :belief? may be, and what each one loads.

true and false are the two ends and :stored is the middle, which exists because storing what rests on what and settling it are separable work and only the second one fails to finish at corpus scale. A dump read :stored lands every justification, every premise mark and all the provenance, and leaves the network empty for a recover that is somebody's own job to schedule.

export-format clj

The marker vaelii.impl.io.export writes. Version numbers alone cannot tell a dump of ours from a foreign one — ours starts at 1, which sits inside numbering somebody else was already using — so the dialect is named rather than deduced.

import-dump clj

(import-dump kb dir)

(import-dump kb
             dir
             {:keys [belief? report-every on-progress]
              :or {belief? true report-every 500000 on-progress no-progress}
              :as opts})

Import a vaelii export dump from dir into the (empty) kb — one vaelii.impl.io.export wrote, or one in a foreign dialect this build still carries a reader for (vaelii.impl.foreign).

With {:belief? true} (the default) it lands in the state the engine's own restart path produces: the record store populated from the re-canonicalized records + the justifications + premise marks, the index rebuilt (reindex), belief recovered (recover).

With {:belief? false} it stores + indexes every sentex but skips what rests on what, the premise marks, and recover — the whole corpus is browsable / findable / countable but not belief-queryable. This is the path for a corpus past what recover's per-node relabel and the in-RAM JTMS scale to (recover resettles a region per premise and per justification — millions of them would not finish).

With {:belief? :stored} it does everything true does except the recover: every justification, premise mark and provenance entry is stored and the index is installed, and the network is left empty for a recover somebody schedules later. Two things make this its own mode rather than a variant of either end.

Storing what rests on what and settling it are separable work, and only the second fails to finish at corpus scale — so a corpus that cannot afford recover today should not have to discard its justifications forever to say so. And for a foreign dialect the records-only path is not a deferral at all: preserve? is (ours? meta), so no strength is carried onto a record and no premise is rostered, and a later recover over that store rebuilds belief from nothing. :stored is the only way a foreign corpus can be loaded now and believed later.

What the KB answers in between is exactly what {:belief? false} answers — stored-side reads work, every believed one is empty — and the browser says so (docs/web.md, Reading a KB that is not finished). The difference is not in what it answers, it is in what it can become.

Reads meta.edn first and dispatches on it: the version is gated against its own dialect's numbering, the destination must be empty, and only the :records / :records+index variants are read. :pg-memory and an unknown variant throw.

opts: {:belief? true|:stored|false :report-every n :on-progress f}. An unknown :belief? value is refused by name, since anything truthy would otherwise mean true and run the recover. :on-progress is called every report-every frames with {:phase :done :total} — the phases a dump has, in order: :sentexes, then (on the belief path) :justifications for one of ours or whatever phase a foreign reader reports, then either :index-entries (a replay) or :reindex (a rebuild). A callback that throws aborts the import where it stands, which is how a caller cancels one; the KB is left holding what had already landed, since an import is not a transaction.

Every refusal of the dump itself lands before the first write, for that reason — the version gate, the empty destination, the variant, and the reserved :out slot on a justification frame (assert-no-naf-justifications!), which is read out of the file rather than met in the middle of the justification phase. So a refused dump leaves the store exactly as it found it and the retry needs no clear!. What that does not cover is a failure the dump cannot be asked about in advance — a cancelled callback, a torn stream, a full disk — and there the sentence above still holds.

The summary reports the :dialect read and the :handle-policy used — :preserved (every record is at the handle the dump gave it) or :remapped (with :collapsed, how many frames canonicalized onto a handle already stored) — and what became of the index: {:index :replayed :entries n} or {:index :rebuilt :reason r} (:absent / :layout-changed / :handles-remapped / :records-differ / :entries-truncated). A caller cannot see any of it for itself, and the first two are load-bearing for the third: an index entry is a posting of handles, so it can only be replayed over a :preserved import.

:naming is the count of what this import stored that assert would have refused — {:checked n :refused n :by-class {…}}, logged as a warning when it is not zero. Neither import path runs the naming check (both build records directly, which is what makes a corpus this size loadable at all), so the disagreement between the two doors is closed by reporting it: the operator who chose the bulk path learns the number while the records go past, rather than from a re-assertion that throws a year later.

:refused is the same account for the door one over — {:checked n :skipped n :by-type {…}}, also logged as a warning. These are frames whose sentence this build will not construct: the structural checks live inside the sentex constructor, so there is no record to store when one fires, and the frame is skipped rather than taken as a reason to abandon the load. A rule an older build stored, or another engine's, can be one a since-widened check refuses; :sentexes and :frames differ by exactly these plus :collapsed.

A dropped meta-sentex is reported the same way, and so is what it takes with it. A remapped import drops a meta-sentex whose embedded (sentexHandle H) names a rule this dump does not carry (:dropped-meta-sentexes), which leaves :orphaned-ids dump ids with no record to resolve to; every reference to one is dropped, and :dropped-justifications-orphaned is how many justifications that cost — the ones whose every unresolved reference is such an id, so they would have resolved whole had the records stayed. A subset of :dropped-justifications, which also counts the references a dump makes to sentexes it never carried. Reported apart because they are different facts: the second is what the dump is like, the first is what this load did to it.

Skipping is not repair. A skipped rule is gone from the store, and every justification, provenance entry and meta-sentex naming it fails to resolve and is dropped in turn. What the number buys is that the operator learns which records those were from a summary, on a load that finished, instead of from a stack trace eight hours into one that did not.

read-meta clj

(read-meta dir)

Read a dump's meta.edn (the marker + schema) without loading any records.

supported-export-versions clj

Export-format versions this build reads.