vaelii.impl.io.import — com.vaelii/vaelii 0.24.0

Import a vaelii export dump — a directory of record streams — into a KB, landing in exactly the state the engine's own restart path (reindex / recover) already knows how to produce.

A dump is a directory whose meta.edn is the marker and schema; every other file is a nippy stream:

sentexes.nippy.stream        one field-map frame per sentex
justifications.nippy.stream  one per justification
provenance.nippy.stream      [handle map] per frame — optional

A :records+index dump also carries the index, as a cache that is used only when it can be proved to describe the records that were just stored (see the index below); otherwise the index is rebuilt, and the summary says which happened and why.

Framing. A chunked stream is a run of [int32 length][compressed chunk], each chunk a compression window over back-to-back nippy frames; a window stream is one compression window over the lot. Our own dumps state which (:framing); a foreign dump's is inferred from its own version line. Both are constant-memory lazy seqs.

A frame of our own dialect is a plain field map whose :sentence is already there — but a rule's set/*Rule wrappers and its variable names canonicalized into the record (:engines / :defeasible / :effect / :varmap), so both are written back around it before the constructor sees it. A frame that is not ours goes to a foreign reader (vaelii.impl.foreign), which is resolved at runtime and may not be in the build at all. The discrimination is on the frame, never on meta.edn's :dialect: a declaration is not an authority over the bytes beside it, and keying off the frame keeps a mixed dump readable.

Whatever the dialect, every sentence is re-canonicalized through this build's own constructor (res/kb-sentex). A stored canonical form is never trusted, not even our own: variable numbering, symmetric argument order and comparison folding belong to the reading build, and a record indexed under a key this build's lookup never reproduces would be silently unfindable.

Handles are preserved for a dump of ours: every record is stored at the handle the dump gave it, so a handle means the same thing either side of an export. Safe because the destination must be empty and because a store's counter clears any handle written that way (p/next-id) — without which the next assert would mint handle 1 again and overwrite the first imported record. Two things can still stop a handle landing as given, and neither is silent: a frame with no :id, and a frame whose canonical form is one already stored, which collapses onto that handle (a dedup this build is right to perform — two engine forms can canonicalize to one stored record — and the dump's numbering cannot survive it). Either makes the import :remapped, and then one old->new map carries the dump's ids across: justification references, and the (sentexHandle H) a meta-sentex embeds inside stored content, which rewrite-embedded-handles! rewrites in the sentence. A meta-sentex whose embedded handle cannot be resolved is dropped, and the drop reaches the map as well as the store (forget-deleted): a dump id whose record is gone has to stop resolving, or the references to it resolve to a handle nothing is stored at.

The index is replayed only when it can be proved to fit, and discarding it is always safe — which is what makes a cache out of what would otherwise be a risk. An index that does not match its records is worse than no index: every lookup then answers confidently and short, and nothing in the engine is positioned to notice. So all three of these must hold (index-decision):

• the entries are keyed in the layout this build reads (kv/index-layout-version);
• the fingerprint accumulated while storing equals the one written beside the entries (vaelii.impl.io.fingerprint) — accumulated, not recomputed, since a second pass over the records to validate a cache would cost more than the cache saves;
• the handles were preserved, so a posting names the record it named in the source.

Anything else rebuilds, at :info, with the reason named. A cache that silently stops being used is a cache nobody maintains.

The store-facing replay is written against the engine's real protocols: populate the record store with the re-canonicalized records + justifications + premise marks, then either install the dumped index (p/index-load) or rebuild it (reindex), then core/recover — which rebuilds the JTMS and the taxonomy from the records, so a replayed index shortcuts the index and nothing else. Out of scope: the :pg-memory variant.

belief-modes clj

What :belief? may be, and what each one loads.

true and false are the two ends and :stored is the middle, which exists because storing what rests on what and settling it are separable work and only the second one fails to finish at corpus scale. A dump read :stored lands every justification, every premise mark and all the provenance, and leaves the network empty for a recover that is somebody's own job to schedule.

export-format clj

The marker vaelii.impl.io.export writes. Version numbers alone cannot tell a dump of ours from a foreign one — ours starts at 1, which sits inside numbering somebody else was already using — so the dialect is named rather than deduced.

import-dump clj

(import-dump kb dir)

(import-dump kb
             dir
             {:keys [belief? report-every on-progress]
              :or {belief? true report-every 500000 on-progress no-progress}
              :as opts})

Import a vaelii export dump from dir into the (empty) kb — one vaelii.impl.io.export wrote, or one in a foreign dialect this build still carries a reader for (vaelii.impl.foreign).

With {:belief? true} (the default) it lands in the state the engine's own restart path produces: the record store populated from the re-canonicalized records + the justifications + premise marks, the index rebuilt (reindex), belief recovered (recover). A dump carrying a reasoning image (export!'s :belief?) is installed in place of the recover when the import kept every handle and the records it landed, the source identity and the belief policies all equal the image's stamp; the summary's :reasoning-image says which happened ({:reasoning :installed} or {:reasoning :recovered :reason r}).

With {:belief? false} it stores + indexes every sentex but skips what rests on what, the premise marks, and recover — the whole corpus is browsable / findable / countable but not belief-queryable. This is the path for a corpus past what recover's per-node relabel and the in-RAM JTMS scale to (recover resettles a region per premise and per justification — millions of them would not finish).

With {:belief? :stored} it does everything true does except the recover: every justification, premise mark and provenance entry is stored and the index is installed, and the network is left empty for a recover somebody schedules later. Two things make this its own mode rather than a variant of either end.

Storing what rests on what and settling it are separable work, and only the second fails to finish at corpus scale — so a corpus that cannot afford recover today should not have to discard its justifications forever to say so. And for a foreign dialect the records-only path is not a deferral at all: preserve? is (ours? meta), so no strength is carried onto a record and no premise is rostered, and a later recover over that store rebuilds belief from nothing. :stored is the only way a foreign corpus can be loaded now and believed later.

What the KB answers in between is exactly what {:belief? false} answers — stored-side reads work, every believed one is empty — and the browser says so (docs/web.md, Reading a KB that is not finished). The difference is not in what it answers, it is in what it can become.

Reads meta.edn first and dispatches on it: the version is gated against its own dialect's numbering, the destination must be empty, and only the :records / :records+index variants are read. :pg-memory and an unknown variant throw.

opts: {:belief? true|:stored|false :report-every n :on-progress f}. An unknown :belief? value is refused by name, since anything truthy would otherwise mean true and run the recover. :on-progress is called every report-every frames with {:phase :done :total} — the phases a dump has, in order: :sentexes, then (on the belief path) :justifications for one of ours or whatever phase a foreign reader reports, then either :index-entries (a replay) or :reindex (a rebuild). A callback that throws aborts the import where it stands, which is how a caller cancels one; the KB is left holding what had already landed, since an import is not a transaction.

A dump refused for its own content leaves the store as it found it, so the retry needs no clear!. Two mechanisms hold this. The refusals that meta.edn, the reader map and a pre-pass decide land before the first write: the version gate, the empty destination, the variant, a foreign reader with no :replay-belief!, and a non-empty :out on a justification frame (assert-no-naf-justifications!). The refusals a frame decides in the middle of the sentex stream (content-refusals: a handle named twice, a frame that is not ours with no reader for it, foreign manifests that do not read, a frame naming a class) are met after the frames before it are stored, and the import then empties the record store and the index before it rethrows (clearing-on-refusal). The destination held no sentex when the import began, so the wipe removes only what this import wrote. Any other failure — a cancelled callback, a torn stream, a full disk, a store that stops — leaves the KB holding what had already landed, as the paragraph above says.

The summary reports the :dialect read and the :handle-policy used — :preserved (every record is at the handle the dump gave it) or :remapped (with :collapsed, how many frames canonicalized onto a handle already stored) — and what became of the index: {:index :replayed :entries n} or {:index :rebuilt :reason r} (:absent / :layout-changed / :handles-remapped / :records-differ / :entries-truncated). A caller cannot see any of it for itself, and the first two are required for the third: an index entry is a posting of handles, so it can only be replayed over a :preserved import.

:naming is the count of what this import stored that assert would have refused — {:checked n :refused n :by-class {…}}, logged as a warning when it is not zero. Neither import path runs the naming check (both build records directly, which is what makes a corpus this size loadable at all), so the disagreement between the two entry points is closed by reporting it: the operator who chose the bulk path learns the number while the records go past, rather than from a re-assertion that throws a year later.

:refused is the same account for the entry point one over — {:checked n :skipped n :by-type {…}}, also logged as a warning. These are frames whose sentence this build will not construct: the structural checks live inside the sentex constructor, so there is no record to store when one fires, and the frame is skipped rather than taken as a reason to abandon the load. A rule an older build stored, or another engine's, can be one a since-widened check refuses. The frames assert refuses whose record would answer wrongly — an open literal, a rule whose antecedent functor is a variable, a sentex in a query context — are skipped and counted here too, under the :type assert raises; an (ist Ctx S) frame is stored as S in Ctx, as assert stores it (the note above as-asserted). A rule frame assert would expand is stored as one record per form, counted in :expanded as {:frames n :records n}. :sentexes and :frames differ by exactly these refusals, :collapsed, and the further forms :expanded counts.

A dropped meta-sentex is reported the same way, and so is what it takes with it. A remapped import drops a meta-sentex whose embedded (sentexHandle H) names a rule this dump does not carry (:dropped-meta-sentexes), which leaves :orphaned-ids dump ids with no record to resolve to; every reference to one is dropped, and :dropped-justifications-orphaned is how many justifications that cost — the ones whose every unresolved reference is such an id, so they would have resolved whole had the records stayed. A subset of :dropped-justifications, which also counts the references a dump makes to sentexes it never carried. Reported apart because they are different facts: the second is what the dump is like, the first is what this load did to it.

Skipping is not repair. A skipped rule is gone from the store, and every justification, provenance entry and meta-sentex naming it fails to resolve and is dropped in turn. What the number buys is that the operator learns which records those were from a summary, on a load that finished, instead of from a stack trace late into one that did not.

read-edn-manifest clj

(read-edn-manifest f)

The EDN manifest in f, read under the manifest byte bound — dfiles/read-edn-manifest, which sits in vaelii.impl.disk.files so that the store sentinels open-kb reads go through the same reader.

read-meta clj

(read-meta dir)

Read a dump's meta.edn (the marker + schema) without loading any records — under dfiles/manifest-bytes, since a dump directory is whatever an operator copied.

supported-export-versions clj

Export-format versions this build reads.