What a KB is asked, and what each answer costs the index — four tallies behind one switch.
The index has six families and several access paths into them (docs/indexing.md),
and which of them earns its keep is a question about a workload, not about the code:
a KB whose every pattern leads with a ground first argument pays for three secondary
root families it never reads, and a KB that asks (?type Muffet) a thousand times a
second lives or dies on the argument-slot roster. Nothing in the engine answers that
from the outside, so this is the instrument that does.
Five tallies, one per question:
:goals — every retrieval decision the matchers took, keyed by the literal's
shape and by the access path that shape chose. This is the distribution an index
policy would have to serve. Two matchers decide: res/candidate-handles (the trie
against the roots) and res/matches-hierarchical (the set-algebra path). It counts
retrievals rather than questions — a matcher that fans over a predicate's spec
closure records one entry per sub-predicate — so a total here is index traffic, not
a count of what a caller asked.:reads — every IndexStore read, by family. This is the one that answers
whether a family earns its keep: a KB that never reads the argument roots is a KB
paying three write taxes for an access path nothing takes. The trie counts as two
families here, :trie-lookup and :trie-counts, because retrieval and the cost
model read the same structure for unrelated reasons and a run can be dominated by
either.:fan — every trie walk KvIndexStore.lookup performed, keyed by the path's
first token, with the node probes it cost. A walk that narrows visits one node per
level; a walk that fans out visits the whole child set at the level it got stuck, and
this is where that shows up as a number rather than as an anecdote.:writes — what one index-sentex wrote, per family, keyed by functor. Every
family is a tax on every assert, so a policy that adds one is priced here.:retracts — the same for unindex-sentex!, and a separate tally rather than a
sign on the one above, because the two do not have the same shape. An assert's cost
is a constant per family; a retraction's is not, since a trie node is deleted only
when the last sentex under it goes and how often that happens is a property of how
much prefix the corpus shares. :dead is that number — trie nodes this retraction
emptied — and it is the one quantity here a corpus can move without changing what it
holds. Merged into :writes, the per-assert constant a gate is set from would stop
being readable.A goal's key is [functor truth adornment path], where the adornment is one character
per argument, in position order:
b a ground atom the roots key (a symbol: an individual, a type, a context)
B a ground compound (keyed whole by the argument roots)
n a ground token the roots do NOT key (a number, a string)
f an open atom (a variable)
F an open compound (a compound holding a variable)
So (parentOf ?x Tom) is [parentOf :true "fb" :arg-roots] and
(mass ?o (QuantityFn ?n Kilogram)) is [mass :true "fF" :structural]. functor is
:open when the functor is itself a variable, which is the shape that puts every
argument behind it. Arity is the adornment's length, so the key carries it too.
The alphabet is what the index distinguishes rather than what a reader would: b and
B are one family's keys and n is no key at all, which is why a ground number after a
variable keeps the trie while a ground symbol there does not.
The switch and the store are one atom: nil when off, so every seam is a deref and a
nil? check, which is what the observer seam costs the reference chainer
(vaelii.impl.observe). Anything heavier on a retrieval path would show up in lein perf as a constant, and a ratio cannot see a constant.
On, it is one swap! per event over a persistent map. That is not free and is not
meant to be: a profiling run measures shape, and every quantity here is a count, so a
run under the instrument answers the same as a run without it, more slowly.
:fan is the one tally that is not index-independent. It is KvIndexStore's,
which covers every backend the KvBackend adapters reach — the flat map, the dense
one, the on-disk WAL, an overlay. vaelii.impl.columnar walks its own native trie
and counts no node probes, so a columnar run reports no fan at all rather than a
fabricated one, and profile_test pins that silence. Every other tally holds on
both: that store keeps the goal, read, write and retract tallies itself, because it
writes and walks the index rather than going through KvIndexStore to do it.p/lookup callers (find-sentex-handle, the level-0 raw
read) appear in :fan and :reads and not in :goals.snapshot is plain data and stop returns the last one. Nothing here formats: what a
reading means is the caller's business, and vaelii.bench.profile is the caller that
has an opinion.
What a KB is **asked**, and what each answer costs the index — four tallies behind
one switch.
The index has six families and several access paths into them (`docs/indexing.md`),
and which of them earns its keep is a question about a *workload*, not about the code:
a KB whose every pattern leads with a ground first argument pays for three secondary
root families it never reads, and a KB that asks `(?type Muffet)` a thousand times a
second lives or dies on the argument-slot roster. Nothing in the engine answers that
from the outside, so this is the instrument that does.
Five tallies, one per question:
* **`:goals`** — every retrieval decision the matchers took, keyed by the literal's
*shape* and by the access path that shape chose. This is the distribution an index
policy would have to serve. Two matchers decide: `res/candidate-handles` (the trie
against the roots) and `res/matches-hierarchical` (the set-algebra path). It counts
**retrievals rather than questions** — a matcher that fans over a predicate's spec
closure records one entry per sub-predicate — so a total here is index traffic, not
a count of what a caller asked.
* **`:reads`** — every `IndexStore` read, by family. This is the one that answers
whether a family earns its keep: a KB that never reads the argument roots is a KB
paying three write taxes for an access path nothing takes. The trie counts as two
families here, `:trie-lookup` and `:trie-counts`, because retrieval and the cost
model read the same structure for unrelated reasons and a run can be dominated by
either.
* **`:fan`** — every trie walk `KvIndexStore.lookup` performed, keyed by the path's
first token, with the node probes it cost. A walk that narrows visits one node per
level; a walk that fans out visits the whole child set at the level it got stuck, and
this is where that shows up as a number rather than as an anecdote.
* **`:writes`** — what one `index-sentex` wrote, per family, keyed by functor. Every
family is a tax on every assert, so a policy that *adds* one is priced here.
* **`:retracts`** — the same for `unindex-sentex!`, and a separate tally rather than a
sign on the one above, because the two do not have the same shape. An assert's cost
is a constant per family; a retraction's is not, since a trie node is deleted only
when the last sentex under it goes and how often that happens is a property of how
much prefix the corpus shares. `:dead` is that number — trie nodes this retraction
emptied — and it is the one quantity here a corpus can move without changing what it
holds. Merged into `:writes`, the per-assert constant a gate is set from would stop
being readable.
## The shape key
A goal's key is `[functor truth adornment path]`, where the adornment is one character
per argument, in position order:
b a ground atom the roots key (a symbol: an individual, a type, a context)
B a ground compound (keyed whole by the argument roots)
n a ground token the roots do NOT key (a number, a string)
f an open atom (a variable)
F an open compound (a compound holding a variable)
So `(parentOf ?x Tom)` is `[parentOf :true "fb" :arg-roots]` and
`(mass ?o (QuantityFn ?n Kilogram))` is `[mass :true "fF" :structural]`. `functor` is
`:open` when the functor is itself a variable, which is the shape that puts every
argument behind it. Arity is the adornment's length, so the key carries it too.
The alphabet is what the *index* distinguishes rather than what a reader would: `b` and
`B` are one family's keys and `n` is no key at all, which is why a ground number after a
variable keeps the trie while a ground symbol there does not.
## Off by default, and free when off
The switch and the store are one atom: nil when off, so every seam is a deref and a
`nil?` check, which is what the observer seam costs the reference chainer
(`vaelii.impl.observe`). Anything heavier on a retrieval path would show up in `lein
perf` as a constant, and a ratio cannot see a constant.
On, it is one `swap!` per event over a persistent map. That is not free and is not
meant to be: a profiling run measures *shape*, and every quantity here is a count, so a
run under the instrument answers the same as a run without it, more slowly.
## What it does not see
* **`:fan` is the one tally that is not index-independent.** It is `KvIndexStore`'s,
which covers every backend the `KvBackend` adapters reach — the flat map, the dense
one, the on-disk WAL, an overlay. `vaelii.impl.columnar` walks its own native trie
and counts no node probes, so a columnar run reports **no fan at all** rather than a
fabricated one, and `profile_test` pins that silence. Every other tally holds on
both: that store keeps the goal, read, write and retract tallies itself, because it
writes and walks the index rather than going through `KvIndexStore` to do it.
* A retrieval that reaches the index without going through either matcher has no
shape here: the direct `p/lookup` callers (`find-sentex-handle`, the level-0 raw
read) appear in `:fan` and `:reads` and not in `:goals`.
## Reading it
`snapshot` is plain data and `stop` returns the last one. Nothing here formats: what a
reading *means* is the caller's business, and `vaelii.bench.profile` is the caller that
has an opinion.(profiling?)Is the instrument collecting?
Is the instrument collecting?
(record-fan pattern visits widest handles)Tally one trie walk: visits node probes, a frontier that reached widest, and
handles handles at the terminus. Keyed by the path's first token, which is the
functor for a positive fact and :false for a negative.
Tally one trie walk: `visits` node probes, a frontier that reached `widest`, and `handles` handles at the terminus. Keyed by the path's first token, which is the functor for a positive fact and `:false` for a negative.
(record-goal pat path)Tally one candidate-handles decision: the pattern sentex it was asked for and the
access path it chose. A deref and a nil? check when the instrument is off.
Tally one `candidate-handles` decision: the pattern sentex it was asked for and the access path it chose. A deref and a `nil?` check when the instrument is off.
(record-index-retract sentex counts)Tally one unindex-sentex!, the same families as record-index-write plus :dead,
the trie nodes this retraction emptied and deleted.
:dead is why this is a tally of its own. Every other quantity here is decided by
the sentex — its arity, its terms, its indexable arguments — so it reads the same
whenever the retraction happens. How many nodes die is decided by what else is
still stored under the same prefix, so the same sentex retracted from a dense corpus
and from a sparse one costs differently, and a family's retraction cost cannot be
quoted as a constant the way its assert cost can.
Tally one `unindex-sentex!`, the same families as `record-index-write` plus `:dead`, the trie nodes this retraction emptied and deleted. `:dead` is why this is a tally of its own. Every other quantity here is decided by the sentex — its arity, its terms, its indexable arguments — so it reads the same whenever the retraction happens. How many nodes die is decided by what *else* is still stored under the same prefix, so the same sentex retracted from a dense corpus and from a sparse one costs differently, and a family's retraction cost cannot be quoted as a constant the way its assert cost can.
(record-index-write sentex counts)Tally one index-sentex: how many trie levels it touched and how many keys each of
the flat families took. Call sites guard on profiling? so the counts map is built
only while collecting.
Tally one `index-sentex`: how many trie levels it touched and how many keys each of the flat families took. Call sites guard on `profiling?` so the counts map is built only while collecting.
(record-literal sentence path)Tally one retrieval decision taken over a bare sentence rather than a pattern sentex — the set-algebra matcher's, which is handed the literal and admits only positive ones.
Tally one retrieval decision taken over a bare sentence rather than a pattern sentex — the set-algebra matcher's, which is handed the literal and admits only positive ones.
(record-read family)Tally one IndexStore read against the family that answered it. This is the tally
that says whether a family earns its keep, so the names are the families and not the
method names: several methods read the argument roots, and what a policy would drop is
the family.
Tally one `IndexStore` read against the family that answered it. This is the tally that says whether a family earns its keep, so the names are the families and not the method names: several methods read the argument roots, and what a policy would drop is the family.
(shape-of body truth path)One literal's tally key, [functor truth adornment path]. Public because the oracle
asks for a shape directly rather than inferring it from a count.
One literal's tally key, `[functor truth adornment path]`. Public because the oracle asks for a shape directly rather than inferring it from a count.
(snapshot)The tallies so far as plain data, or nil when the instrument is off:
{:elapsed-ms how long this run has been collecting :goals {[functor truth adornment path] count} :reads {family count} :fan {first-token {:calls :visits :widest :handles :decades {n count}}} :writes {functor {:asserts :levels :terms :roots :roster :slots}} :retracts {functor {:retracts :levels :terms :roots :roster :slots :dead}}}
The tallies so far as plain data, or nil when the instrument is off:
{:elapsed-ms how long this run has been collecting
:goals {[functor truth adornment path] count}
:reads {family count}
:fan {first-token {:calls :visits :widest :handles :decades {n count}}}
:writes {functor {:asserts :levels :terms :roots :roster :slots}}
:retracts {functor {:retracts :levels :terms :roots :roster :slots :dead}}}(start)Begin collecting, dropping whatever a previous run left. Bare, not !: a tally is
derived from a workload nobody stored, so nothing it holds is knowledge and re-running
the workload recomputes it.
Begin collecting, dropping whatever a previous run left. Bare, not `!`: a tally is derived from a workload nobody stored, so nothing it holds is knowledge and re-running the workload recomputes it.
(stop)Stop collecting and return the final snapshot (nil when it was not running).
Stop collecting and return the final snapshot (nil when it was not running).
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |