Storage protocols so the record store and index store each have swappable implementations (in-memory by default, on-disk for durability, or an alternate KV store later). The rest of the system programs against these protocols and never against a concrete backend.
Declarations only, no code. The fallbacks that go with the optional capabilities
— count-sentexes, sentex-sink, hinting and the rest — are next door in
vaelii.impl.capabilities, because IndexStore below is large enough that
re-evaluating the form (as cloverage does, form by form, to instrument a namespace)
overflows the JVM's 64 KB per-method bytecode limit. This namespace is therefore
loaded but not instrumented (scripts/coverage.sh); a protocol carries nothing to
cover, so the split costs the measurement nothing and keeps those fallbacks in it.
vaelii.impl.jtms-protocol is split from vaelii.impl.jtms for the same reason.
Storage protocols so the record store and index store each have swappable implementations (in-memory by default, on-disk for durability, or an alternate KV store later). The rest of the system programs against these protocols and never against a concrete backend. **Declarations only, no code.** The fallbacks that go with the optional capabilities — `count-sentexes`, `sentex-sink`, `hinting` and the rest — are next door in `vaelii.impl.capabilities`, because `IndexStore` below is large enough that re-evaluating the form (as cloverage does, form by form, to instrument a namespace) overflows the JVM's 64 KB per-method bytecode limit. This namespace is therefore loaded but not instrumented (scripts/coverage.sh); a protocol carries nothing to cover, so the split costs the measurement nothing and keeps those fallbacks in it. `vaelii.impl.jtms-protocol` is split from `vaelii.impl.jtms` for the same reason.
Optional, and beside BulkLoading rather than in it: the two per-handle writes
that follow a record rather than being part of it — the premise mark and the provenance
map. Separate because a store may bulk-load records without being able to bulk-update
what is already there, and a partly-implemented protocol is worse than two.
Both exist because the import path writes them in a loop over handles it already
holds. The premise marks are decided only once the whole sentex stream is read (a
dump id that collapses onto a stored handle keeps the strongest strength), and the
provenance stream is read after the records; so neither can ride the record write, and
both are n writes on a store where a write is a round trip. On a corpus import over a
server that is the largest remaining block of round trips by a wide margin.
Neither op changes what the per-handle version does — same guard, same end state. A handle with no sentex is still not marked.
**Optional**, and beside `BulkLoading` rather than in it: the two per-handle writes that follow a record rather than being part of it — the premise mark and the provenance map. Separate because a store may bulk-load records without being able to bulk-update what is already there, and a partly-implemented protocol is worse than two. Both exist because the import path writes them in a **loop over handles it already holds**. The premise marks are decided only once the whole sentex stream is read (a dump id that collapses onto a stored handle keeps the strongest strength), and the provenance stream is read after the records; so neither can ride the record write, and both are `n` writes on a store where a write is a round trip. On a corpus import over a server that is the largest remaining block of round trips by a wide margin. Neither op changes what the per-handle version does — same guard, same end state. A handle with no sentex is still not marked.
(mark-premise-batch store id->strength)Mark every handle in id->strength a premise at its strength, in as few writes as
this store can manage. Returns nil.
Mark every handle in `id->strength` a premise at its strength, in as few writes as this store can manage. Returns nil.
(put-provenance-batch store entries)Persist every [id prov] pair in entries, overwriting, in as few writes as this
store can manage. Returns nil.
Persist every `[id prov]` pair in `entries`, overwriting, in as few writes as this store can manage. Returns nil.
Optional, and beside RecordStore for the same reason as Prefetching and
Tallying: a store with an ingest path faster than a record at a time implements it,
and one whose put-sentex is already a map assoc does not — in which case a bulk
loader runs loop-sink's put-per-record loop and pays nothing for the capability.
The seam is a sink and not a batched put, and the difference is the handle. A
batched put returning handles makes its caller wait for a batch to land before it can do
anything with any of it, and the import path's whole shape is that it indexes each
record from the copy already in hand rather than reading it back (import's
records-only pass says why: on a durable store the read-back re-pages every record). So
the handle is decided caller-side and the sink is told, which is also what lets a dump
preserve its own numbering through one.
opts is {:premises? bool} for a sentex sink — whether a record carrying a
:strength is rostered as a premise by the write. It is an option because the engine's
two import paths differ on it: the records-only pass marks inline, and the belief pass
aggregates the marks afterwards over dump ids that collapsed onto one handle, where the
record's own strength is the first frame's and the mark is the strongest. A justification
sink has no such question.
**Optional**, and beside `RecordStore` for the same reason as `Prefetching` and
`Tallying`: a store with an ingest path faster than a record at a time implements it,
and one whose `put-sentex` is already a map assoc does not — in which case a bulk
loader runs `loop-sink`'s put-per-record loop and pays nothing for the capability.
The seam is a **sink and not a batched put**, and the difference is the handle. A
batched put returning handles makes its caller wait for a batch to land before it can do
anything with any of it, and the import path's whole shape is that it indexes each
record from the copy already in hand rather than reading it back (`import`'s
records-only pass says why: on a durable store the read-back re-pages every record). So
the handle is decided caller-side and the sink is *told*, which is also what lets a dump
preserve its own numbering through one.
`opts` is `{:premises? bool}` for a sentex sink — whether a record carrying a
`:strength` is rostered as a premise by the write. It is an option because the engine's
two import paths differ on it: the records-only pass marks inline, and the belief pass
aggregates the marks afterwards over dump ids that collapsed onto one handle, where the
record's own strength is the first frame's and the mark is the strongest. A justification
sink has no such question.(open-justification-sink store opts)A RecordSink writing justifications in bulk.
A `RecordSink` writing justifications in bulk.
(open-sentex-sink store opts)A RecordSink writing sentexes in bulk.
A `RecordSink` writing sentexes in bulk.
The index store — the count-aware trie, the secondary root indexes, the
rule predicate index, the exception re-check index, and the inverted term index.
Every entry is derived from the records, so it can be thrown away and rebuilt
(vaelii.impl.reindex); it needs no durability of its own.
The index store — the count-aware trie, the secondary root indexes, the rule predicate index, the exception re-check index, and the inverted term index. Every entry is derived from the records, so it can be thrown away and rebuilt (`vaelii.impl.reindex`); it needs no durability of its own.
(sentexes-with-term store term)Handles of sentexes the term index keys by term.
Handles of sentexes the term index keys by `term`.
(rules-by-consequent store pred)Handles of rules concluding pred.
Handles of rules concluding pred.
(rules-with-exception-on store pred)Handles of rules whose exception mentions pred.
Handles of rules whose exception mentions pred.
(unindex-exception! store handle preds)Deregister a rule handle from its exception predicates.
Deregister a rule handle from its exception predicates.
(sentexes-with-terms store terms)Handles of sentexes keyed by all terms — one intersection.
Handles of sentexes keyed by all `terms` — one intersection.
(index-sentex store sentex handle)Insert a ground sentex handle into the trie.
Insert a ground sentex handle into the trie.
(count-with-arg store pos term)How many fact sentexes have term at pos.
How many fact sentexes have `term` at `pos`.
(count-in-context store context)How many sentexes are in context.
How many sentexes are in `context`.
(unindex-sentex! store sentex handle)Remove a sentex handle from the trie.
Remove a sentex handle from the trie.
(children store prefix)Child tokens registered under an interior prefix.
Child tokens registered under an interior prefix.
(sentexes-with-functor store pred)Handles of fact sentexes whose functor is pred (any arity, either polarity).
Handles of fact sentexes whose functor is `pred` (any arity, either polarity).
(exception-rules store)Handles of every rule carrying an exception.
Handles of every rule carrying an exception.
(index-entries store)Every index entry as a lazy [key value] seq — the portable projection.
Every index entry as a lazy `[key value]` seq — the portable projection.
(lookup store pattern)Handles whose path matches a full pattern.
Handles whose path matches a full pattern.
(sentexes-in-context store context)Handles of sentexes asserted in context.
Handles of sentexes asserted in `context`.
(terms store)Every symbol term the index is keyed by — the KB's vocabulary, unordered.
Every symbol term the index is keyed by — the KB's vocabulary, unordered.
(clear-index! store)Remove every index entry (wipe the store — reindex rebuilds).
Remove every index entry (wipe the store — `reindex` rebuilds).
(count-at store prefix)Sentex count under a path prefix.
Sentex count under a path prefix.
(rules-by-antecedent store pred)Handles of rules with an antecedent on pred.
Handles of rules with an antecedent on pred.
(sentexes-with-arg store pos term)Handles of fact sentexes with term at 1-based argument pos.
Handles of fact sentexes with `term` at 1-based argument `pos`.
(term-count store)How many distinct symbol terms the index is keyed by (the roster's own count, no walk).
How many distinct symbol terms the index is keyed by (the roster's own count, no walk).
(count-with-functor store pred)How many fact sentexes have functor pred.
How many fact sentexes have functor `pred`.
(unindex-rule! store handle ante-preds conseq-pred)Deregister a rule handle.
Deregister a rule handle.
(index-rule store handle ante-preds conseq-pred)Register a rule handle by its predicates.
Register a rule handle by its predicates.
(leaf-at store path)Handles stored exactly at path's leaf — an exact read, never a wildcard match.
Handles stored exactly at `path`'s leaf — an exact read, never a wildcard match.
(index-load store entries)Install [key value] entries into an empty index, in this store's own representation.
Install `[key value]` entries into an empty index, in this store's own representation.
(index-exception store handle preds)Register a rule handle under each predicate its exception mentions.
Register a rule handle under each predicate its exception mentions.
(exception-rule? store handle)Is handle in the exception/watched-rule roster? (O(1) membership — the firing-path gate.)
Is `handle` in the exception/watched-rule roster? (O(1) membership — the firing-path gate.)
(count-children store prefix)How many child tokens sit under an interior prefix.
How many child tokens sit under an interior prefix.
(sentexes-with-args store pred pos-terms)Handles with functor pred AND each [pos term] — one set intersection.
Handles with functor `pred` AND each `[pos term]` — one set intersection.
Optional, and beside RecordStore rather than an op in it: a store that keeps a
cache in front of a fetch expensive enough to be worth avoiding implements this, and one
whose fetch is already a page touch does not — in which case a caller runs exactly the
code it ran without this protocol in the world.
prefetch-sentexes! is a hint, never an answer. It returns nothing, and every
record still comes back through get-sentex, so it cannot change what a query matches
on any backend however wrong its guess about what is worth fetching. That is the whole
reason the capability is shaped this way rather than as a batched read returning
records: a batched read has to be proven equal to the per-handle loop on every
implementation, and a cache warmed ahead of that loop is equal to it by construction.
The store decides whether to act, and it is the only party that can: it is handed the handles a caller is about to walk and it knows which of them its own cache already holds, so "is one query cheaper than these fetches" is answered from the actual set rather than from a setting somebody tuned.
**Optional**, and beside `RecordStore` rather than an op in it: a store that keeps a cache in front of a fetch expensive enough to be worth avoiding implements this, and one whose fetch is already a page touch does not — in which case a caller runs exactly the code it ran without this protocol in the world. `prefetch-sentexes!` is a **hint, never an answer.** It returns nothing, and every record still comes back through `get-sentex`, so it cannot change what a query matches on any backend however wrong its guess about what is worth fetching. That is the whole reason the capability is shaped this way rather than as a batched read returning records: a batched read has to be *proven* equal to the per-handle loop on every implementation, and a cache warmed ahead of that loop is equal to it by construction. **The store decides whether to act**, and it is the only party that can: it is handed the handles a caller is about to walk and it knows which of them its own cache already holds, so "is one query cheaper than these fetches" is answered from the actual set rather than from a setting somebody tuned.
(prefetch-justifications! store ids)The same for justifications. A separate op rather than a kind argument, mirroring
get-sentex / get-justification: the two are different rows in every store that
has rows, and a caller always knows which it is about to walk.
The same for justifications. A separate op rather than a kind argument, mirroring `get-sentex` / `get-justification`: the two are different rows in every store that has rows, and a caller always knows which it is about to walk.
(prefetch-sentexes! store ids)Warm this store's cache for the sentexes at ids if that is cheaper than the fetches
it saves. Returns nil. Never required — a caller may skip it entirely and read the
same records at the same handles.
Warm this store's cache for the sentexes at `ids` if that is cheaper than the fetches it saves. Returns nil. Never required — a caller may skip it entirely and read the same records at the same handles.
An open bulk write. A caller opens one with capabilities/sentex-sink /
justification-sink, writes a stream of records to it, and closes it
(java.io.Closeable, so with-open).
Do not read a record back before that close. A sink may hold everything it was
given until then — a COPY stream lands when the copy ends — so a fetch of a handle
just written may find nothing there. This is a restriction on the caller and not a
promise about visibility: an implementation is free to make a write readable at once,
and the loop the default sink runs does.
The handle is the caller's, decided before the write and returned rather than
computed: a record carrying an :id lands at it, and one without gets next-id. That
is what makes a sink usable where a batched put is not — the import path indexes each
record from the copy already in hand and needs the handle now, and a COPY returns
nothing per row.
An open bulk write. A caller opens one with `capabilities/sentex-sink` / `justification-sink`, writes a stream of records to it, and **closes it** (`java.io.Closeable`, so `with-open`). **Do not read a record back before that close.** A sink may hold everything it was given until then — a `COPY` stream lands when the copy ends — so a fetch of a handle just written may find nothing there. This is a restriction on the caller and not a promise about visibility: an implementation is free to make a write readable at once, and the loop the default sink runs does. The **handle is the caller's**, decided before the write and returned rather than computed: a record carrying an `:id` lands at it, and one without gets `next-id`. That is what makes a sink usable where a batched put is not — the import path indexes each record from the copy already in hand and needs the handle *now*, and a `COPY` returns nothing per row.
(write-record! sink rec)Write rec to the sink and return its handle — (:id rec) when it carries one, a
freshly minted one when it does not. Not readable until the sink is closed.
Write `rec` to the sink and return its handle — `(:id rec)` when it carries one, a freshly minted one when it does not. Not readable until the sink is closed.
The record store — canonical sentexes and justifications, keyed by integer handle. The durable ground truth: everything else the KB holds is derived from it.
The three fetches are counted (vaelii.impl.profile's :fetches), because a
record read is not an index read and no index tally can stand in for one: a probe that
narrows to a single lookup and then pages a record per candidate handle costs almost
nothing by :reads and everything by this. Every implementation tallies its own kind
on the protocol method, so the number counts what a caller asked for and not what a
backend does internally.
The record store — canonical sentexes and justifications, keyed by integer handle. The durable ground truth: everything else the KB holds is derived from it. **The three fetches are counted** (`vaelii.impl.profile`'s `:fetches`), because a record read is not an index read and no index tally can stand in for one: a probe that narrows to a single `lookup` and then pages a record per candidate handle costs almost nothing by `:reads` and everything by this. Every implementation tallies its own kind on the protocol method, so the number counts what a *caller* asked for and not what a backend does internally.
(clear-records! store)Remove every stored record (wipe the whole store).
Remove every stored record (wipe the whole store).
(get-sentex store id)Fetch a sentex by handle, or nil.
Fetch a sentex by handle, or nil.
(sentex-ids store)Every live sentex handle, as a set.
Every live sentex handle, as a set.
(mark-premise store id strength)Record a handle as an asserted premise at strength.
Record a handle as an asserted premise at `strength`.
(unmark-premise! store id)Record that a handle is no longer a premise.
Record that a handle is no longer a premise.
(put-sentex store sentex)Persist a sentex; return its handle.
Persist a sentex; return its handle.
(put-justification store d)Persist a justification; return its handle.
Persist a justification; return its handle.
(delete-sentex! store id)Remove a sentex record.
Remove a sentex record.
(get-provenance store id)The provenance map for handle id, or nil.
The provenance map for handle `id`, or nil.
(delete-provenance! store id)Remove the provenance for handle id.
Remove the provenance for handle `id`.
(next-id store)Allocate the next monotonic handle — one above every
handle the store holds, including one that arrived as
an explicit :id on a put rather than from here. A
handle is an identity, so no store may issue one twice.
Allocate the next monotonic handle — one above every handle the store holds, including one that arrived as an explicit `:id` on a put rather than from here. A handle is an identity, so no store may issue one twice.
(delete-justification! store id)Remove a justification record.
Remove a justification record.
(premise-ids store)Every handle currently marked a premise, as a set.
Every handle currently marked a premise, as a set.
(put-provenance store id prov)Persist the provenance map for handle id (overwrites).
Persist the provenance map for handle `id` (overwrites).
(get-justification store id)Fetch a justification by handle, or nil.
Fetch a justification by handle, or nil.
(justification-ids store)Every live justification handle, as a set.
Every live justification handle, as a set.
(premise-strength store id)The recorded assumption strength of a premise handle.
The recorded assumption strength of a premise handle.
Optional, and beside RecordStore for the same reason as Prefetching: the two
questions a caller asks an enumeration that do not need the enumeration.
(count (sentex-ids store)) is how the engine asks how many records is this, and
(first (sentex-ids store)) is how it asks is this store empty — open-kb asks both
before the KB has answered anything, and import asks the first again to report what it
loaded. On a store whose enumeration is a read of its own state those cost nothing. On
one whose enumeration is a query they cost the whole table: every handle over the
wire and a roster built out of it, to answer with one number.
A store implements this when it can answer without enumerating — a SELECT count(*), a
LIMIT 1 — and callers go through the helpers below, which fall back to the enumeration
and so read identically on a store that does not.
There is no premise tally and no sentex-by-sentex sampling here: only the questions the engine asks are on the seam, so an implementer knows every one of them is worth a statement.
**Optional**, and beside `RecordStore` for the same reason as `Prefetching`: the two questions a caller asks an enumeration that do not need the enumeration. `(count (sentex-ids store))` is how the engine asks *how many records is this*, and `(first (sentex-ids store))` is how it asks *is this store empty* — `open-kb` asks both before the KB has answered anything, and `import` asks the first again to report what it loaded. On a store whose enumeration is a read of its own state those cost nothing. On one whose enumeration is a **query** they cost the whole table: every handle over the wire and a roster built out of it, to answer with one number. A store implements this when it can answer without enumerating — a `SELECT count(*)`, a `LIMIT 1` — and callers go through the helpers below, which fall back to the enumeration and so read identically on a store that does not. There is no premise tally and no sentex-by-sentex sampling here: only the questions the engine asks are on the seam, so an implementer knows every one of them is worth a statement.
(a-justification-id store)Some live justification handle, or nil.
Some live justification handle, or nil.
(a-premise-id store)Some handle marked a premise, or nil.
Some handle marked a premise, or nil.
(a-sentex-id store)Some live sentex handle, or nil when the store holds none. Which one is the store's choice — every caller either tests it for nil or reads the record to prove the store is readable at all, and none of them depends on which handle came back.
Some live sentex handle, or nil when the store holds none. *Which* one is the store's choice — every caller either tests it for nil or reads the record to prove the store is readable at all, and none of them depends on which handle came back.
(justification-tally store)How many live justifications, without building the roster.
How many live justifications, without building the roster.
(sentex-tally store)How many live sentexes, without building the roster.
How many live sentexes, without building the roster.
The index-rule consequent slot's catch-all for a rule whose consequent functor is a
variable — a rule with concrete antecedents concluding (?p …), which
rules/check-indexable-functors allows (the antecedent binds ?p, so a forward firing
is ground, and the range check guarantees it). Filed under this one bucket rather than
the canonical ?var0 no goal can spell, and unioned into every concrete-goal answer by
resolution/concluding-rule-handles. Written via rules/consequent-index-pred; a
keyword can never collide with a predicate, which is always a symbol.
The `index-rule` consequent slot's catch-all for a rule whose consequent functor is a **variable** — a rule with concrete antecedents concluding `(?p …)`, which `rules/check-indexable-functors` allows (the antecedent binds `?p`, so a forward firing is ground, and the range check guarantees it). Filed under this one bucket rather than the canonical `?var0` no goal can spell, and unioned into every concrete-goal answer by `resolution/concluding-rule-handles`. Written via `rules/consequent-index-pred`; a keyword can never collide with a predicate, which is always a symbol.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |