Import a vaelii export dump, a directory of record streams, into a KB, landing in the state
the engine's own restart path (reindex / recover) produces. meta.edn is the marker
and schema, and every other file is a nippy stream. Every sentence is re-canonicalized
through res/kb-sentex; handles are preserved for a dump of ours; the dumped index is
replayed only when it is proved to fit, and the summary says which happened and why. A frame
that is not of our dialect goes to a foreign reader (vaelii.impl.foreign). Out of scope:
the :pg-memory variant. See docs/storage.md, "Importing a dump".
Import a vaelii export dump, a directory of record streams, into a KB, landing in the state the engine's own restart path (`reindex` / `recover`) produces. `meta.edn` is the marker and schema, and every other file is a nippy stream. Every sentence is re-canonicalized through `res/kb-sentex`; handles are preserved for a dump of ours; the dumped index is replayed only when it is proved to fit, and the summary says which happened and why. A frame that is not of our dialect goes to a foreign reader (`vaelii.impl.foreign`). Out of scope: the `:pg-memory` variant. See docs/storage.md, "Importing a dump".
A fn from the stream reader stream-reader-for chooses to the reader the import reads
every stream file through, or nil for the chosen reader itself. A reader takes [file compression] and answers the file's frames. The test fixtures bind it to answer the
frames of a dump they restore many times from a vector thawed once per JVM
(vaelii.test-util/restore!).
A fn from the stream reader `stream-reader-for` chooses to the reader the import reads every stream file through, or nil for the chosen reader itself. A reader takes `[file compression]` and answers the file's frames. The test fixtures bind it to answer the frames of a dump they restore many times from a vector thawed once per JVM (`vaelii.test-util/restore!`).
What :belief? may be, and what each one loads.
true and false are the two ends and :stored is the middle, which exists because
storing what rests on what and settling it are separable work and only the second one
fails to finish at corpus scale. A dump read :stored lands every justification, every
premise mark and all the provenance, and leaves the network empty for a recover that
is somebody's own job to schedule.
What `:belief?` may be, and what each one loads. `true` and `false` are the two ends and `:stored` is the middle, which exists because storing what rests on what and *settling* it are separable work and only the second one fails to finish at corpus scale. A dump read `:stored` lands every justification, every premise mark and all the provenance, and leaves the network empty for a `recover` that is somebody's own job to schedule.
(check-dump! meta)Refuse a dump meta describes that this build does not read: a format version outside
its dialect's numbering (:unsupported-version), and a variant other than :records /
:records+index (:unsupported-variant). Returns the variant.
Refuse a dump `meta` describes that this build does not read: a format version outside its dialect's numbering (`:unsupported-version`), and a variant other than `:records` / `:records+index` (`:unsupported-variant`). Returns the variant.
(check-import-opts! opts)Refuse an opts key import-dump does not read, an unknown :belief? value, and a non-nil
non-map opts, before meta.edn is looked at. A misspelt :belief? key or value would
otherwise take the default and run the recover the caller meant to skip.
Refuse an opts key `import-dump` does not read, an unknown `:belief?` value, and a non-nil non-map `opts`, before `meta.edn` is looked at. A misspelt `:belief?` key or value would otherwise take the default and run the recover the caller meant to skip.
(dump-reasoners kb meta dir {want :reasoners})The reasoners kb registers once the dump meta describes has landed: nil when meta
records no set or records kb's own, and the dump's set under :reasoners :stored when
it holds every name kb registered. Any other pair is refused (:reasoner-mismatch);
import-dump asks before the first write, and vaelii.core/import! registers the
answer after the recover.
The reasoners `kb` registers once the dump `meta` describes has landed: nil when `meta` records no set or records `kb`'s own, and the dump's set under `:reasoners :stored` when it holds every name `kb` registered. Any other pair is refused (`:reasoner-mismatch`); `import-dump` asks before the first write, and `vaelii.core/import!` registers the answer after the recover.
The marker vaelii.impl.io.export writes. Version numbers alone cannot tell a dump
of ours from a foreign one — ours starts at 1, which sits inside numbering somebody
else was already using — so the dialect is named rather than deduced.
The marker `vaelii.impl.io.export` writes. Version numbers alone cannot tell a dump of ours from a foreign one — ours starts at 1, which sits inside numbering somebody else was already using — so the dialect is named rather than deduced.
(import-dump kb dir)(import-dump kb
dir
{:keys [belief? report-every on-progress drop-principal?]
:or {belief? true report-every 500000 on-progress no-progress}
:as opts})Import a vaelii export dump from dir into the empty kb: one vaelii.impl.io.export
wrote, or a foreign dialect this build has a reader for (vaelii.impl.foreign). Reads
meta.edn first; the version is checked against its dialect's numbering and the
destination must be empty (:pg-memory and an unknown variant throw).
opts: {:belief? true|:stored|false :report-every n :on-progress f :drop-principal? bool}. :belief? true (the default) lands in the state the restart path produces,
installing a dump's reasoning image in place of recover when every handle was kept and
the records, source identity and belief policies equal its manifest; {:reasoning :installed} or {:reasoning :recovered :reason r} says which. false stores and indexes
every sentex and skips justifications, premise marks and recover. :stored does
everything true does except recover. An unknown :belief? value is refused by name.
:drop-principal? true lands provenance without :principal. :on-progress is called
every report-every frames with {:phase :done :total}; a callback that throws cancels
the import where it stands, and the KB keeps what landed.
A dump refused for its own content leaves the store as it found it (the refusals before the
first write, and content-refusals through clearing-on-refusal).
The summary reports :dialect, :handle-policy (:preserved, or :remapped with
:collapsed), :index ({:index :replayed :entries n} or {:index :rebuilt :reason r}),
:naming (what assert would have refused, {:checked n :refused n :by-class {…}}),
:refused ({:checked n :skipped n :by-type {…}}, frames the constructor will not build
or whose record would answer wrongly), :expanded, :dropped-meta-sentexes,
:orphaned-ids, :dropped-justifications and :dropped-justifications-orphaned.
Skipping is not repair: whatever names a skipped rule is dropped in turn. See
docs/storage.md, "Importing a dump".
Import a vaelii export dump from `dir` into the empty `kb`: one `vaelii.impl.io.export`
wrote, or a foreign dialect this build has a reader for (`vaelii.impl.foreign`). Reads
`meta.edn` first; the version is checked against its dialect's numbering and the
destination must be empty (`:pg-memory` and an unknown variant throw).
`opts`: `{:belief? true|:stored|false :report-every n :on-progress f :drop-principal?
bool}`. `:belief? true` (the default) lands in the state the restart path produces,
installing a dump's reasoning image in place of `recover` when every handle was kept and
the records, source identity and belief policies equal its manifest; `{:reasoning
:installed}` or `{:reasoning :recovered :reason r}` says which. `false` stores and indexes
every sentex and skips justifications, premise marks and `recover`. `:stored` does
everything `true` does except `recover`. An unknown `:belief?` value is refused by name.
`:drop-principal? true` lands provenance without `:principal`. `:on-progress` is called
every `report-every` frames with `{:phase :done :total}`; a callback that throws cancels
the import where it stands, and the KB keeps what landed.
A dump refused for its own content leaves the store as it found it (the refusals before the
first write, and `content-refusals` through `clearing-on-refusal`).
The summary reports `:dialect`, `:handle-policy` (`:preserved`, or `:remapped` with
`:collapsed`), `:index` (`{:index :replayed :entries n}` or `{:index :rebuilt :reason r}`),
`:naming` (what `assert` would have refused, `{:checked n :refused n :by-class {…}}`),
`:refused` (`{:checked n :skipped n :by-type {…}}`, frames the constructor will not build
or whose record would answer wrongly), `:expanded`, `:dropped-meta-sentexes`,
`:orphaned-ids`, `:dropped-justifications` and `:dropped-justifications-orphaned`.
Skipping is not repair: whatever names a skipped rule is dropped in turn. See
docs/storage.md, "Importing a dump".(read-edn-manifest f)The EDN manifest in f, read under the manifest byte bound — dfiles/read-edn-manifest,
which sits in vaelii.impl.disk.files so that the store sentinels open-kb reads go
through the same reader.
The EDN manifest in `f`, read under the manifest byte bound — `dfiles/read-edn-manifest`, which sits in `vaelii.impl.disk.files` so that the store sentinels `open-kb` reads go through the same reader.
(read-meta dir)Read a dump's meta.edn (the marker + schema) without loading any records — under
dfiles/manifest-bytes, since a dump directory is whatever an operator copied.
Read a dump's `meta.edn` (the marker + schema) without loading any records — under `dfiles/manifest-bytes`, since a dump directory is whatever an operator copied.
Export-format versions this build reads.
Export-format versions this build reads.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |