vaelii.impl.io.import — com.vaelii/vaelii 0.13.0

Import a vaelii export dump — a directory of record streams — into a KB, landing in exactly the state the engine's own restart path (reindex / recover) already knows how to produce.

A dump is a directory whose meta.edn is the marker and schema; every other file is a nippy stream:

sentexes.nippy.stream        one field-map frame per sentex
justifications.nippy.stream  one per justification
provenance.nippy.stream      [handle map] per frame — optional

A :records+index dump also carries the index, as a cache that is used only when it can be proved to describe the records that were just stored (see the index below); otherwise the index is rebuilt, and the summary says which happened and why.

Framing. A chunked stream is a run of [int32 length][compressed chunk], each chunk a compression window over back-to-back nippy frames; a window stream is one compression window over the lot. Our own dumps state which (:framing); a foreign dump's is inferred from its own version line. Both are constant-memory lazy seqs.

A frame of our own dialect is a plain field map whose :sentence is already there — but a rule's set/*Rule wrappers and its variable names canonicalized into the record (:direction / :defeasible / :varmap), so both are written back around it before the constructor sees it. A frame that is not ours goes to a foreign reader (vaelii.impl.foreign), which is resolved at runtime and may not be in the build at all. The discrimination is on the frame, never on meta.edn's :dialect: a declaration is not an authority over the bytes beside it, and keying off the frame keeps a mixed dump readable.

Whatever the dialect, every sentence is re-canonicalized through this build's own constructor (res/kb-sentex). A stored canonical form is never trusted, not even our own: variable numbering, symmetric argument order and comparison folding belong to the reading build, and a record indexed under a key this build's lookup never reproduces would be silently unfindable.

Handles are preserved for a dump of ours: every record is stored at the handle the dump gave it, so a handle means the same thing either side of an export. Safe because the destination must be empty and because a store's counter clears any handle written that way (p/next-id) — without which the next assert would mint handle 1 again and overwrite the first imported record. Two things can still stop a handle landing as given, and neither is silent: a frame with no :id, and a frame whose canonical form is one already stored, which collapses onto that handle (a dedup this build is right to perform — two engine forms can canonicalize to one stored record — and the dump's numbering cannot survive it). Either makes the import :remapped, and then one old->new map carries the dump's ids across: justification references, and the (sentexHandle H) a meta-sentex embeds inside stored content, which rewrite-embedded-handles! rewrites in the sentence. A meta-sentex whose embedded handle cannot be resolved is dropped, and the drop reaches the map as well as the store (forget-deleted): a dump id whose record is gone has to stop resolving, or the references to it resolve to a handle nothing is stored at.

The index is replayed only when it can be proved to fit, and discarding it is always safe — which is what makes a cache out of what would otherwise be a risk. An index that does not match its records is worse than no index: every lookup then answers confidently and short, and nothing in the engine is positioned to notice. So all three of these must hold (index-decision):

• the entries are keyed in the layout this build reads (kv/index-layout-version);
• the fingerprint accumulated while storing equals the one written beside the entries (vaelii.impl.io.fingerprint) — accumulated, not recomputed, since a second pass over the records to validate a cache would cost more than the cache saves;
• the handles were preserved, so a posting names the record it named in the source.

Anything else rebuilds, at :info, with the reason named. A cache that silently stops being used is a cache nobody maintains.

The store-facing replay is written against the engine's real seams: populate the record store with the re-canonicalized records + justifications + premise marks, then either install the dumped index (p/index-load) or rebuild it (reindex), then core/recover — which rebuilds the JTMS and the taxonomy from the records, so a replayed index shortcuts the index and nothing else. Out of scope: the :pg-memory variant.

belief-modes clj

What :belief? may be, and what each one loads.

true and false are the two ends and :stored is the middle, which exists because storing what rests on what and settling it are separable work and only the second one fails to finish at corpus scale. A dump read :stored lands every justification, every premise mark and all the provenance, and leaves the network empty for a recover that is somebody's own job to schedule.

export-format clj

The marker vaelii.impl.io.export writes. Version numbers alone cannot tell a dump of ours from a foreign one — ours starts at 1, which sits inside numbering somebody else was already using — so the dialect is named rather than deduced.

import-dump clj

(import-dump kb dir)

(import-dump kb
             dir
             {:keys [belief? report-every on-progress]
              :or {belief? true report-every 500000 on-progress no-progress}
              :as opts})

Import a vaelii export dump from dir into the (empty) kb — one vaelii.impl.io.export wrote, or one in a foreign dialect this build still carries a reader for (vaelii.impl.foreign).

With {:belief? true} (the default) it lands in the state the engine's own restart path produces: the record store populated from the re-canonicalized records + the justifications + premise marks, the index rebuilt (reindex), belief recovered (recover).

With {:belief? false} it stores + indexes every sentex but skips what rests on what, the premise marks, and recover — the whole corpus is browsable / findable / countable but not belief-queryable. This is the path for a corpus past what recover's per-node relabel and the in-RAM JTMS scale to (recover resettles a region per premise and per justification — millions of them would not finish).

With {:belief? :stored} it does everything true does except the recover: every justification, premise mark and provenance entry is stored and the index is installed, and the network is left empty for a recover somebody schedules later. Two things make this its own mode rather than a variant of either end.

Storing what rests on what and settling it are separable work, and only the second fails to finish at corpus scale — so a corpus that cannot afford recover today should not have to discard its justifications forever to say so. And for a foreign dialect the records-only path is not a deferral at all: preserve? is (ours? meta), so no strength is carried onto a record and no premise is rostered, and a later recover over that store rebuilds belief from nothing. :stored is the only way a foreign corpus can be loaded now and believed later.

What the KB answers in between is exactly what {:belief? false} answers — stored-side reads work, every believed one is empty — and the browser says so (docs/web.md, Reading a KB that is not finished). The difference is not in what it answers, it is in what it can become.

Reads meta.edn first and dispatches on it: the version is gated against its own dialect's numbering, the destination must be empty, and only the :records / :records+index variants are read. :pg-memory and an unknown variant throw.

opts: {:belief? true|:stored|false :report-every n :on-progress f}. An unknown :belief? value is refused by name, since anything truthy would otherwise mean true and run the recover. :on-progress is called every report-every frames with {:phase :done :total} — the phases a dump has, in order: :sentexes, then (on the belief path) :justifications for one of ours or whatever phase a foreign reader reports, then either :index-entries (a replay) or :reindex (a rebuild). A callback that throws aborts the import where it stands, which is how a caller cancels one; the KB is left holding what had already landed, since an import is not a transaction.

Every refusal of the dump itself lands before the first write, for that reason — the version gate, the empty destination, the variant, and the reserved :out slot on a justification frame (assert-no-naf-justifications!), which is read out of the file rather than met in the middle of the justification phase. So a refused dump leaves the store exactly as it found it and the retry needs no clear!. What that does not cover is a failure the dump cannot be asked about in advance — a cancelled callback, a torn stream, a full disk — and there the sentence above still holds.

The summary reports the :dialect read and the :handle-policy used — :preserved (every record is at the handle the dump gave it) or :remapped (with :collapsed, how many frames canonicalized onto a handle already stored) — and what became of the index: {:index :replayed :entries n} or {:index :rebuilt :reason r} (:absent / :layout-changed / :handles-remapped / :records-differ / :entries-truncated). A caller cannot see any of it for itself, and the first two are load-bearing for the third: an index entry is a posting of handles, so it can only be replayed over a :preserved import.

:naming is the count of what this import stored that assert would have refused — {:checked n :refused n :by-class {…}}, logged as a warning when it is not zero. Neither import path runs the naming check (both build records directly, which is what makes a corpus this size loadable at all), so the disagreement between the two doors is closed by reporting it: the operator who chose the bulk path learns the number while the records go past, rather than from a re-assertion that throws a year later.

:refused is the same account for the door one over — {:checked n :skipped n :by-type {…}}, also logged as a warning. These are frames whose sentence this build will not construct: the structural checks live inside the sentex constructor, so there is no record to store when one fires, and the frame is skipped rather than taken as a reason to abandon the load. A rule an older build stored, or another engine's, can be one a since-widened check refuses; :sentexes and :frames differ by exactly these plus :collapsed.

A dropped meta-sentex is reported the same way, and so is what it takes with it. A remapped import drops a meta-sentex whose embedded (sentexHandle H) names a rule this dump does not carry (:dropped-meta-sentexes), which leaves :orphaned-ids dump ids with no record to resolve to; every reference to one is dropped, and :dropped-justifications-orphaned is how many justifications that cost — the ones whose every unresolved reference is such an id, so they would have resolved whole had the records stayed. A subset of :dropped-justifications, which also counts the references a dump makes to sentexes it never carried. Reported apart because they are different facts: the second is what the dump is like, the first is what this load did to it.

Skipping is not repair. A skipped rule is gone from the store, and every justification, provenance entry and meta-sentex naming it fails to resolve and is dropped in turn. What the number buys is that the operator learns which records those were from a summary, on a load that finished, instead of from a stack trace eight hours into one that did not.

manifest-bytes clj

The most an EDN manifest may hold — a dump's meta.edn, a store's format.edn, a corpus's report.edn, an index's index.edn, a machine's catalog.edn.

Every one of them is a handful of keys: a marker, a version, some counts, a compression name. They are also the first thing read about a directory nobody has promised anything about — vaelii.impl.catalog probes every entry of the KB search path this way, and a load reads one before it opens a stream — so an unbounded read is a whole file pulled into a string on the strength of its name. A megabyte is orders of magnitude above the largest of them (a hand-written catalog.edn naming thousands of KBs) and still a bound.

read-edn-manifest clj

(read-edn-manifest f)

The EDN manifest in f, read under manifest-bytes — or a refusal (:manifest-too-large) naming the file and the bound.

The bound is on the read, not on the file's stated length. File.length answers 0 for a FIFO and for most of /proc, and a symlink to one of those is a slurp that never ends; reading a bounded number of bytes and refusing the one past the bound needs the file to say nothing true about itself. Bytes rather than characters, so the figure the refusal states is the figure that was read.

Content the EDN reader cannot parse is refused by name too (:malformed-manifest). A manifest cut mid-form — the shape a crashed writer and a half-copied directory both leave — otherwise raises a bare RuntimeException ("EOF while reading"), which is neither a :type a caller can discriminate on nor a fact about the file it names. Which of the two refusals means "not a KB" and which means "a broken one" is the caller's to decide, and vaelii.impl.catalog decides it differently from the loaders.

read-meta clj

(read-meta dir)

Read a dump's meta.edn (the marker + schema) without loading any records — under manifest-bytes, since a dump directory is whatever an operator copied.

supported-export-versions clj

Export-format versions this build reads.