What knowledge bases this process can load, and the lifecycle of loading one.
Everything above the engine assumes it is holding the KB. A browser that lists the KBs available, loads one while you watch, and switches to it needs two things the engine does not have: a description of a KB that has not been loaded yet, and somewhere for a load that takes minutes to run while the pages keep answering. Those are the two halves here.
A source is a KB you could load, as data — a kind, a name, and wherever the content comes from. Six kinds:
| kind | content | loader |
|---|---|---|
:core | the CoreContext vocabulary head alone | vaelii.impl.core-context |
:starter | the shipped schema-only ontology | vaelii.impl.starter |
:generated | synthesized from numbers — types, rules, a fwd mix | vaelii.impl.io.generate |
:corpus | a translated sentence corpus (OpenCyc) | a foreign reader, :cyc-corpus |
:dump | a vaelii export dump | vaelii.impl.io.import |
:store | an on-disk KB already in vaelii's own format | opened in place |
The first three ship in this repo and are always offered. The last three are found:
each directory on the search path (VAELII_KB_PATH, else ./kbs and ~/.vaelii/kbs)
is probed, and what marks it — a corpus meta.edn, a dump meta.edn, a records/ +
index/ pair — decides its kind. A catalog.edn (VAELII_KB_CATALOG, else
~/.vaelii/catalog.edn) names sources outside the search path. Nothing about a
machine's paths is baked into the repo.
An entry is a source that has been loaded, or is loading: a KB, a status, and a
progress reading the loaders report into (:on-progress, reported by every loader —
the corpus reader, io.import/import-dump and io.generate/load-into). One load runs at a time, on its
own thread, and is cancelled by a flag the progress callback throws on — the loaders
have no other safe interruption point, and an import is not a transaction, so a
cancelled load leaves the KB holding what had already landed.
A KB is readable before it is finished. activate asks only that an entry hold a
KB, so the one arriving can be the one every page reads — a corpus is browsable from
its first thousand sentexes, and a store that opens in seconds is browsable while
belief is still being rebuilt behind it. What that costs a reader is completeness, not
correctness, and active-caveat is what says so.
And a KB can go back out. export-entry! writes a loaded one as an export dump on
the same thread discipline a load runs on, which closes the loop: a dump written under
the search path is a :dump source the moment its meta.edn lands, so exporting and
reloading needs nothing outside this namespace.
Unloading never deletes an on-disk KB. A memory-backed entry has its stores cleared (they would otherwise hold the corpus for the life of the JVM); a disk-backed one is closed — the file lock released, the directory left exactly as it was. The same directory can then be loaded again, or opened by another process.
What knowledge bases this process can load, and the lifecycle of loading one. Everything above the engine assumes it is holding *the* KB. A browser that lists the KBs available, loads one while you watch, and switches to it needs two things the engine does not have: a description of a KB that has not been loaded yet, and somewhere for a load that takes minutes to run while the pages keep answering. Those are the two halves here. **A source** is a KB you could load, as data — a kind, a name, and wherever the content comes from. Six kinds: | kind | content | loader | |--------------|-----------------------------------------------------|--------| | `:core` | the CoreContext vocabulary head alone | `vaelii.impl.core-context` | | `:starter` | the shipped schema-only ontology | `vaelii.impl.starter` | | `:generated` | synthesized from numbers — types, rules, a fwd mix | `vaelii.impl.io.generate` | | `:corpus` | a translated sentence corpus (OpenCyc) | a foreign reader, `:cyc-corpus` | | `:dump` | a vaelii export dump | `vaelii.impl.io.import` | | `:store` | an on-disk KB already in vaelii's own format | opened in place | The first three ship in this repo and are always offered. The last three are **found**: each directory on the search path (`VAELII_KB_PATH`, else `./kbs` and `~/.vaelii/kbs`) is probed, and what marks it — a corpus `meta.edn`, a dump `meta.edn`, a `records/` + `index/` pair — decides its kind. A `catalog.edn` (`VAELII_KB_CATALOG`, else `~/.vaelii/catalog.edn`) names sources outside the search path. Nothing about a machine's paths is baked into the repo. **An entry** is a source that has been loaded, or is loading: a KB, a status, and a progress reading the loaders report into (`:on-progress`, reported by every loader — the corpus reader, `io.import/import-dump` and `io.generate/load-into`). One load runs at a time, on its own thread, and is cancelled by a flag the progress callback throws on — the loaders have no other safe interruption point, and an import is not a transaction, so a cancelled load leaves the KB holding what had already landed. **A KB is readable before it is finished.** `activate` asks only that an entry hold a KB, so the one arriving can be the one every page reads — a corpus is browsable from its first thousand sentexes, and a store that opens in seconds is browsable while belief is still being rebuilt behind it. What that costs a reader is completeness, not correctness, and `active-caveat` is what says so. **And a KB can go back out.** `export-entry!` writes a loaded one as an export dump on the same thread discipline a load runs on, which closes the loop: a dump written under the search path is a `:dump` source the moment its `meta.edn` lands, so exporting and reloading needs nothing outside this namespace. **Unloading never deletes an on-disk KB.** A memory-backed entry has its stores cleared (they would otherwise hold the corpus for the life of the JVM); a disk-backed one is *closed* — the file lock released, the directory left exactly as it was. The same directory can then be loaded again, or opened by another process.
(activate key)Make entry key the one the browser reads. Anything holding a KB can be
activated, a load still running included; only an entry with no KB yet is refused,
and that is a statement about there being nothing there rather than about the load.
A half-loaded KB answers about what has landed, which is a prefix and not a wrong
answer — the ordinary open-world condition this engine is built on, where an absent
fact never means a false one. Reading beside the loader is sound for the same reason
a reader thread beside the writer is: one writer, and every store mutation lands
atomically (docs/storage.md, the single-writer contract). What a reader is owed is
being told, which is active-caveat's job and the browser's — not being refused.
Make entry `key` the one the browser reads. Anything **holding a KB** can be activated, a load still running included; only an entry with no KB yet is refused, and that is a statement about there being nothing there rather than about the load. A half-loaded KB answers about what has landed, which is a *prefix* and not a wrong answer — the ordinary open-world condition this engine is built on, where an absent fact never means a false one. Reading beside the loader is sound for the same reason a reader thread beside the writer is: one writer, and every store mutation lands atomically (docs/storage.md, the single-writer contract). What a reader is owed is being *told*, which is `active-caveat`'s job and the browser's — not being refused.
(active)The key of the active entry, or nil.
The key of the active entry, or nil.
(active-caveat)What is provisional about the KB the browser is reading, or nil when nothing is.
{:key :name :status :progress :belief?}.
Two independent reasons an answer can be less than the whole truth, and a reader is owed both:
recover builds together and
:belief? false skips together. That empties more than queries: with no JTMS every
believed answer is empty, and with no genl/genlContext closures there is no type
hierarchy either, so a fully stored KB renders as one with no types and no contexts
at all.The second outlives the first: a store opened without :recover? is :ready and stays
that way, and a :dump imported with :belief? false likewise. That is why it is
reported beside the status rather than folded into it — the dangerous case is the one
that looks finished. The TMS is what is probed, since the two are built as a pair.
An empty KB is not beliefless, it is empty; and an entry that is not an in-process KB
(an attached daemon) has nothing here to report, exactly as stats and footprint
have nothing to say about it.
What is provisional about the KB the browser is reading, or nil when nothing is.
`{:key :name :status :progress :belief?}`.
Two independent reasons an answer can be less than the whole truth, and a reader is
owed both:
- the load is **still running** (or stopped part-way), so what is stored is a prefix
of what was asked for;
- **belief and the taxonomy are not built**, which `recover` builds together and
`:belief? false` skips together. That empties more than queries: with no JTMS every
believed answer is empty, and with no genl/genlContext closures there is no type
hierarchy either, so a fully stored KB renders as one with no types and no contexts
at all.
The second outlives the first: a store opened without `:recover?` is `:ready` and stays
that way, and a `:dump` imported with `:belief? false` likewise. That is why it is
reported beside the status rather than folded into it — the dangerous case is the one
that looks finished. The TMS is what is probed, since the two are built as a pair.
An empty KB is not beliefless, it is empty; and an entry that is not an in-process KB
(an attached daemon) has nothing here to report, exactly as `stats` and `footprint`
have nothing to say about it.(active-entry)(active-kb)The KB the browser should be reading, or nil when nothing is loaded.
The KB the browser should be reading, or nil when nothing is loaded.
The sources that need no files beyond the ones in this repo. :options is what the
UI offers per source, as data: each entry is a form control, and load-source reads
the same keys back out of the params map.
The sources that need no files beyond the ones in this repo. `:options` is what the UI offers per source, as data: each entry is a form control, and `load-source` reads the same keys back out of the params map.
(cancel! key)Ask a running load to stop at its next progress report. ! because what it leaves
behind is a half-loaded KB — the loaders write as they go and none of them is a
transaction.
Ask a running load to stop at its next progress report. `!` because what it leaves behind is a half-loaded KB — the loaders write as they go and none of them is a transaction.
(cancel-export!)Ask a running export to stop at its next chunk boundary. ! because what it leaves
behind is a directory holding part of a dump.
Ask a running export to stop at its next chunk boundary. `!` because what it leaves behind is a directory holding part of a dump.
(classify d)The kind of KB in directory d, or nil. Reads the marker each writer leaves: a
corpus's meta.edn carries the context order it was written in, a dump's carries a
:format-version, and a vaelii :disk store is a records/ + index/ pair.
The kind of KB in directory `d`, or nil. Reads the marker each writer leaves: a corpus's `meta.edn` carries the context order it was written in, a dump's carries a `:format-version`, and a vaelii `:disk` store is a `records/` + `index/` pair.
(entries)Every entry, in load order, as views.
Every entry, in load order, as views.
(entry key)One entry, whole (:kb included) — for a caller that wants the KB itself.
One entry, whole (`:kb` included) — for a caller that wants the KB itself.
(export-entry! key dir opts)Write the KB in entry key out as a dump in dir, on its own thread, and return the
job. opts are vaelii.core/export!'s (:variant, :compression).
Three refusals, each about something an export cannot be correct in the face of:
export! walks it record by record with no snapshot to
walk instead, so a dump of a KB something is still writing is a dump of no single
state.! for what a cancelled one leaves behind: a directory holding part of a dump. It is
not a loadable dump — meta.edn is written last and is what classify keys on — but
it is bytes on disk that nothing here will clean up.
Write the KB in entry `key` out as a dump in `dir`, on its own thread, and return the job. `opts` are `vaelii.core/export!`'s (`:variant`, `:compression`). Three refusals, each about something an export cannot be correct in the face of: - **another export is running.** One at a time, as with loads. A *load* is not refused, and does not refuse this: a load fills some other KB, and blocking on it would be a rule about this process's busyness rather than about the dump. - **the entry is not an in-process KB.** A daemon serves its KB from another host, and that is where its dump would be written — so the export belongs to that daemon's own surface, not to a form here that would name a path on the wrong machine. - **the KB is still loading.** `export!` walks it record by record with no snapshot to walk instead, so a dump of a KB something is still writing is a dump of no single state. `!` for what a cancelled one leaves behind: a directory holding part of a dump. It is not a *loadable* dump — `meta.edn` is written last and is what `classify` keys on — but it is bytes on disk that nothing here will clean up.
(export-job)The running — or last — export, as something safe to render: the cancel flag and the thread dropped, elapsed time filled in. Nil until something has been exported.
One slot, not a list. An export is a job with a report, and the report worth keeping is the last one; a finished job stays visible until the next export replaces it, which is what lets the panel say where the dump went.
The running — or last — export, as something safe to render: the cancel flag and the thread dropped, elapsed time filled in. Nil until something has been exported. One slot, not a list. An export is a job with a report, and the report worth keeping is the last one; a finished job stays visible until the next export replaces it, which is what lets the panel say where the dump went.
(exporting?)Is an export running? One at a time, as with loads.
Is an export running? One at a time, as with loads.
(footprint key)An estimate of what the KB in entry key costs in RAM, by component:
{:sentexes :index :records :tms :total :estimated? true} (bytes), or nil for an entry
with no in-process KB.
Cheap by construction, because this runs on every render of a page that polls: the
sentex count is the trie's own root count (O(1)) and the belief check is one first
over the network's datums. Nothing is walked, and nothing is measured — see
resident-bytes-per-sentex for where the coefficients come from and how wrong they
can be.
Two adjustments make it a statement about this KB rather than about a generic one:
a :disk KB pages its records, so the record term is dropped (what stays resident is
the bounded hot-record LRU, which does not grow with the corpus), and a KB loaded
without belief — an import with :belief? false, a store opened without :recover? —
has no truth-maintenance network at all, so that term goes too.
Nil for an entry that is not an in-process KB, as stats is: an attached daemon holds
its corpus in another process, so nothing here is about this one's memory — and asking
it would be a round trip per render.
An estimate of what the KB in entry `key` costs in RAM, by component:
`{:sentexes :index :records :tms :total :estimated? true}` (bytes), or nil for an entry
with no in-process KB.
Cheap by construction, because this runs on every render of a page that polls: the
sentex count is the trie's own root count (O(1)) and the belief check is one `first`
over the network's datums. Nothing is walked, and nothing is measured — see
`resident-bytes-per-sentex` for where the coefficients come from and how wrong they
can be.
Two adjustments make it a statement about *this* KB rather than about a generic one:
a `:disk` KB pages its records, so the record term is dropped (what stays resident is
the bounded hot-record LRU, which does not grow with the corpus), and a KB loaded
without belief — an import with `:belief? false`, a store opened without `:recover?` —
has no truth-maintenance network at all, so that term goes too.
Nil for an entry that is not an in-process KB, as `stats` is: an attached daemon holds
its corpus in *another* process, so nothing here is about this one's memory — and asking
it would be a round trip per render.(heap)What this JVM's heap is doing right now — {:used :committed :max} in bytes, read
off the memory MX bean.
A measurement, not an estimate, but a coarse one: :used includes garbage that has not
been collected, so it drifts up between collections and drops without anything being
freed. Read it as the shape of the curve, not as a number to subtract KBs from.
What this JVM's heap is doing **right now** — `{:used :committed :max}` in bytes, read
off the memory MX bean.
A measurement, not an estimate, but a coarse one: `:used` includes garbage that has not
been collected, so it drifts up between collections and drops without anything being
freed. Read it as the shape of the curve, not as a number to subtract KBs from.(holder fallback)A deref-able that always yields the KB to read — the active entry's, or fallback
when nothing is loaded. This is what the browser is built against: app holds one of
these instead of a KB, so activating another entry re-points every page at once.
A deref-able that always yields the KB to read — the active entry's, or `fallback` when nothing is loaded. This is what the browser is built against: `app` holds one of these instead of a KB, so activating another entry re-points every page at once.
(human-bytes n)A byte count as something to read at a glance. Public because the browser shows byte counts of its own (a KB's estimated footprint, the heap) and there is no reason for two spellings of the same figure.
A byte count as something to read at a glance. Public because the browser shows byte counts of its own (a KB's estimated footprint, the heap) and there is no reason for two spellings of the same figure.
(in-process? key)Does entry key hold a KB in this JVM? A KB is a map with a record store in it; an
attached daemon is registered as an entry too, and its KB is somewhere else — which is
why stats and footprint have nothing to say about one, and why an export of one is
written on that daemon's host rather than here.
Does entry `key` hold a KB in *this* JVM? A KB is a map with a record store in it; an attached daemon is registered as an entry too, and its KB is somewhere else — which is why `stats` and `footprint` have nothing to say about one, and why an export of one is written on that daemon's host rather than here.
(load-source source-id)(load-source source-id params)Start loading the source with id source-id under params, on its own thread.
Returns the entry key, or throws when another load is already running or the id names
no source.
The entry is registered :loading before this returns, so the caller can render it
immediately; entries then reports its progress until it settles into :ready (the KB
is queryable, and activated when nothing else is) or :failed / :cancelled.
Start loading the source with id `source-id` under `params`, on its own thread. Returns the entry key, or throws when another load is already running or the id names no source. The entry is registered `:loading` before this returns, so the caller can render it immediately; `entries` then reports its progress until it settles into `:ready` (the KB is queryable, and activated when nothing else is) or `:failed` / `:cancelled`.
(loading?)Is a load running? One runs at a time: they are minutes long and memory-hungry, and two at once would make each other's timings meaningless.
Is a load running? One runs at a time: they are minutes long and memory-hungry, and two at once would make each other's timings meaningless.
(memory)The memory picture the browser shows: the JVM heap as measured, every loaded entry's estimated footprint, and their sum.
The memory picture the browser shows: the JVM heap as measured, every loaded entry's estimated footprint, and their sum.
(predicted-footprint {:keys [total]})What a source would cost in RAM if it were loaded, in bytes — its own count of what it holds put through the same coefficients — or nil for a source that does not know how big it is (the generator, whose size is whatever the sliders say).
A load builds belief and holds its records in RAM unless a directory is named, so this is the full three-component figure: the ceiling, not the floor.
What a source would cost in RAM if it were loaded, in bytes — its own count of what it holds put through the same coefficients — or nil for a source that does not know how big it is (the generator, whose size is whatever the sliders say). A load builds belief and holds its records in RAM unless a directory is named, so this is the full three-component figure: the ceiling, not the floor.
(register! key name kb)(register! key name kb {:keys [where source]})File an already-built KB as a :ready entry and make it active if nothing else is —
how the browser's own startup KB, and an attached daemon, get into the list beside the
ones the catalog loads.
opts: :where says what unloading it should release (nil for a KB this process does
not own — an attached daemon is nobody's to close); :source names the source it came
from, so a KB registered at startup shows as loaded rather than being offered again.
File an already-built KB as a `:ready` entry and make it active if nothing else is — how the browser's own startup KB, and an attached daemon, get into the list beside the ones the catalog loads. `opts`: `:where` says what unloading it should release (nil for a KB this process does not own — an attached daemon is nobody's to close); `:source` names the source it came from, so a KB registered at startup shows as *loaded* rather than being offered again.
(reset-registry!)Forget every entry, releasing each as unload! does, and stop a running export. For
a process shutting down and for tests; nothing in the browser calls it.
Forget every entry, releasing each as `unload!` does, and stop a running export. For a process shutting down and for tests; nothing in the browser calls it.
What one stored sentex costs in RAM, per resident component — the coefficients
footprint multiplies out. Measured on a real-assert load through bench-scale and
bench-memory; resident size is linear in
the sentex count (one record, one index path and one node per sentex, and trie prefix
sharing only reduces the per-sentex cost), which is what makes a single coefficient
the right shape for an estimate.
Shape-dependent, and that is the estimate's main error term: a fact of arity 2 with no compound arguments indexes at ~1,549 B while a richer one measured ~2,158 B. The leaner figure is used, so a corpus of fat sentences reads low.
What one stored sentex costs in RAM, per resident component — the coefficients `footprint` multiplies out. Measured on a real-assert load through `bench-scale` and `bench-memory`; resident size is linear in the sentex count (one record, one index path and one node per sentex, and trie prefix sharing only *reduces* the per-sentex cost), which is what makes a single coefficient the right shape for an estimate. Shape-dependent, and that is the estimate's main error term: a fact of arity 2 with no compound arguments indexes at ~1,549 B while a richer one measured ~2,158 B. The leaner figure is used, so a corpus of fat sentences reads low.
(search-path)The directories discovery walks: VAELII_KB_PATH (:-separated) when set, else the
vaelii.kb.path system property, else ./kbs and ~/.vaelii/kbs. A path entry that
is a KB directory counts as one source; otherwise its children are probed, one level
down. (The property mirrors vaelii.disk.dir, and is what a test sets — a JVM cannot
change its own environment.)
The directories discovery walks: `VAELII_KB_PATH` (`:`-separated) when set, else the `vaelii.kb.path` system property, else `./kbs` and `~/.vaelii/kbs`. A path entry that *is* a KB directory counts as one source; otherwise its children are probed, one level down. (The property mirrors `vaelii.disk.dir`, and is what a test sets — a JVM cannot change its own environment.)
(sources)Every KB this process can load, built-ins first: the shipped ontologies and the generator, then whatever the catalog file names, then whatever the search path holds. Recomputed per call — dropping a corpus into a search-path directory makes it appear on the next page load, with no restart.
Every KB this process can load, built-ins first: the shipped ontologies and the generator, then whatever the catalog file names, then whatever the search path holds. Recomputed per call — dropping a corpus into a search-path directory makes it appear on the next page load, with no restart.
(stats kb)The headline counts for a loaded KB. term-count is one set-size read and the
context sizes are one each, so this is cheap even on a corpus of millions — deliberately
so, since it runs every time the page lists the entries.
Nil for anything that is not an in-process KB (an attached daemon is registered as an entry too, and its counts are the daemon's to report).
The headline counts for a loaded KB. `term-count` is one set-size read and the context sizes are one each, so this is cheap even on a corpus of millions — deliberately so, since it runs every time the page lists the entries. Nil for anything that is not an in-process KB (an attached daemon is registered as an entry too, and its counts are the daemon's to report).
(unload! key)Take an entry down: cancel it if it is still loading, drop it from the registry, and release what it held.
A memory-backed KB is cleared — its stores are keyed by space number and would
otherwise hold the corpus for the life of the JVM. A disk-backed one is closed, not
cleared: the file lock is released and the directory is left exactly as it was, so
unloading an on-disk KB never destroys it. The ! is for the memory case, which does.
Take an entry down: cancel it if it is still loading, drop it from the registry, and release what it held. **A memory-backed KB is cleared** — its stores are keyed by space number and would otherwise hold the corpus for the life of the JVM. **A disk-backed one is closed, not cleared**: the file lock is released and the directory is left exactly as it was, so unloading an on-disk KB never destroys it. The `!` is for the memory case, which does.
(write-blocked? kb)Is kb one a loader is still writing?
Reading beside a loader is sound; writing beside one is not. A store mutation lands atomically, so a reader sees a consistent prefix — but two interleaved writers are not serializable at all, and this process's one writer is already spoken for while a load runs (docs/storage.md, the single-writer contract). So activating a KB mid-load buys a reader everything except the right to change it.
Asked of the KB itself rather than of the active entry, and by identity, because
those are not the same question: loading a second KB in the background is no reason to
stop writing to the one on screen, and a caller holding a KB the catalog never heard of
(a test, an embedding, an --attach) is nobody's loader's business.
Is `kb` one a loader is still **writing**? Reading beside a loader is sound; writing beside one is not. A store mutation lands atomically, so a reader sees a consistent prefix — but two interleaved writers are not serializable at all, and this process's one writer is already spoken for while a load runs (docs/storage.md, the single-writer contract). So activating a KB mid-load buys a reader everything except the right to change it. Asked of the **KB itself** rather than of the active entry, and by identity, because those are not the same question: loading a second KB in the background is no reason to stop writing to the one on screen, and a caller holding a KB the catalog never heard of (a test, an embedding, an `--attach`) is nobody's loader's business.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |