An outside judge over what the knowledge base concluded: every claim glossed into one English line, handed to a model, and answered agree / disagree / unsure.
vaelii.impl.llm.text reads English into the KB, where the danger is a model
writing something false into the store and the defence is a reviewer between the two.
Here the KB is the one making claims and the model is the one being asked, so nothing
a model says can reach the store: this namespace calls no writer, and a verdict is a
line in a report. A disagreement is a finding for a person to read, never a
retraction, never a defeat class, and never a test failure that edits the KB to make
the number go up.
That is the whole of why the judge is worth having. The engine can check that a conclusion follows and that a sentence is well formed; nothing in it can check that the knowledge is true, and a KB full of well-formed nonsense passes every gate this repo has. An outside reader is the only instrument for that, and a model is a reader who will do it for two hundred claims without getting bored.
A model handed (genl penguin bird) is judging our notation. vaelii.impl.gloss
composes the English from the KB's own comments — the vocabulary documents itself,
and the first clause of each comment is already a template — so what the judge sees is
the knowledge base's sentence rather than a paraphrase somebody wrote for the prompt.
A sentence the KB documents nothing about glosses to :named, which is barely more
than the s-expression, so it is left out and counted as skipped: an unanswerable
question dressed up as a low score measures the prompt and not the KB.
Fido is awake is not judgeable on its own — nobody knows Fido. Given that Fido is a dog, Fido is awake is: it is the everyday question of whether that is a reasonable thing to say about a dog you have just been told about. So a derived claim carries the facts its justification rests on, glossed the same way.
It does not carry the rule, and that omission is the design. Show the rule and the question becomes does this follow, which is validity — the one thing the engine already guarantees and the one thing an outside judge is not needed for.
Most of this KB is defaults, and a default is not a universal. A judge forced to
answer yes or no about an animal is awake will pick one and the disagreement rate
will measure the coin. unsure is where a claim that depends on particulars nobody
supplied belongs, and the counts are reported apart so a reader can see how much of
the answer was a shrug.
What the rate is not is an accuracy: a careful judge marking a default false is telling you the default has exceptions, which the KB already knows and stores as a default for exactly that reason. So each disagreement carries the claim's strength, and the disagreements — not the rate — are the output.
An outside judge over what the knowledge base concluded: every claim glossed into one English line, handed to a model, and answered *agree / disagree / unsure*. ## This is the other direction, and the trust runs the other way `vaelii.impl.llm.text` reads English **into** the KB, where the danger is a model writing something false into the store and the defence is a reviewer between the two. Here the KB is the one making claims and the model is the one being asked, so nothing a model says can reach the store: this namespace calls no writer, and a verdict is a line in a report. A disagreement is a **finding for a person to read**, never a retraction, never a defeat class, and never a test failure that edits the KB to make the number go up. That is the whole of why the judge is worth having. The engine can check that a conclusion follows and that a sentence is well formed; nothing in it can check that the knowledge is *true*, and a KB full of well-formed nonsense passes every gate this repo has. An outside reader is the only instrument for that, and a model is a reader who will do it for two hundred claims without getting bored. ## The claim is glossed, and a claim the KB cannot gloss is not sent A model handed `(genl penguin bird)` is judging our notation. `vaelii.impl.gloss` composes the English from the KB's **own** comments — the vocabulary documents itself, and the first clause of each comment is already a template — so what the judge sees is the knowledge base's sentence rather than a paraphrase somebody wrote for the prompt. A sentence the KB documents nothing about glosses to `:named`, which is barely more than the s-expression, so it is left out and counted as skipped: an unanswerable question dressed up as a low score measures the prompt and not the KB. ## A derived claim is shown its situation, and never its rule *Fido is awake* is not judgeable on its own — nobody knows Fido. *Given that Fido is a dog, Fido is awake* is: it is the everyday question of whether that is a reasonable thing to say about a dog you have just been told about. So a derived claim carries the facts its justification rests on, glossed the same way. It does **not** carry the rule, and that omission is the design. Show the rule and the question becomes *does this follow*, which is validity — the one thing the engine already guarantees and the one thing an outside judge is not needed for. ## Three verdicts, because two would make the number meaningless Most of this KB is defaults, and a default is not a universal. A judge forced to answer yes or no about *an animal is awake* will pick one and the disagreement rate will measure the coin. `unsure` is where a claim that depends on particulars nobody supplied belongs, and the counts are reported apart so a reader can see how much of the answer was a shrug. What the rate is **not** is an accuracy: a careful judge marking a default false is telling you the default has exceptions, which the KB already knows and stores as a default for exactly that reason. So each disagreement carries the claim's strength, and the disagreements — not the rate — are the output.
(agreement {:keys [judged skipped]})The arithmetic over a judged set.
Two rates, because they answer different questions. :rate is agreement over every
claim that was put, which is the one that moves when the KB changes. :decided is
agreement over the ones the judge committed on, which is the one to quote when a model
shrugs at half of them. Both are nil rather than zero over an empty denominator — a
rate over nothing is not a bad score, it is no score.
The arithmetic over a judged set. Two rates, because they answer different questions. `:rate` is agreement over every claim that was put, which is the one that moves when the KB changes. `:decided` is agreement over the ones the judge committed on, which is the one to quote when a model shrugs at half of them. Both are `nil` rather than zero over an empty denominator — a rate over nothing is not a bad score, it is no score.
(claims kb handles)handles -> the claim maps a judging turn is built from, sorted by content.
Sorted by the glossed text, not by handle: two KBs holding the same knowledge in a different order must produce the same prompt, or a run is not comparable with the one before it. Each map carries what a reader of the report needs — the sentence, its context, whether anybody asserted it, the strength it is held at, and the situation a derived one rests on.
:source is the gloss's, and judgeable? reads it.
`handles` -> the claim maps a judging turn is built from, sorted by content. Sorted by the **glossed text**, not by handle: two KBs holding the same knowledge in a different order must produce the same prompt, or a run is not comparable with the one before it. Each map carries what a reader of the report needs — the sentence, its context, whether anybody asserted it, the strength it is held at, and the situation a derived one rests on. `:source` is the gloss's, and `judgeable?` reads it.
How many claims go in one turn. Small enough that a model keeps the numbering straight over the whole answer, large enough that two hundred claims are ten calls and not two hundred model loads.
How many claims go in one turn. Small enough that a model keeps the numbering straight over the whole answer, large enough that two hundred claims are ten calls and not two hundred model loads.
The window a judging turn asks for. A batch of twenty one-line claims is a small prompt, and asking for a window the host is not already serving costs a model reload — which on a shared host is not a private act.
The window a judging turn asks for. A batch of twenty one-line claims is a small prompt, and asking for a window the host is not already serving costs a model reload — which on a shared host is not a private act.
(judge claims)(judge claims {:keys [batch-size] :as opts})Judge every claim, in batches. Returns
{:judged [claim + :verdict + :note …] ; sorted as `claims` sorted them
:skipped [claim …] ; no English to judge
:batches n :usage {…} :model "…"}
opts: :provider (default: the offline stub), :model, :num-ctx, :max-tokens,
:batch-size.
Renumbering per batch is deliberate: each turn's item numbers start at zero, so a
model that answers item: 3 in the fourth batch has answered the fourth claim of that
batch and not the fourth of the run. Nothing here writes to a KB — the claims arrived
as data and leave as data.
Judge every claim, in batches. Returns
{:judged [claim + :verdict + :note …] ; sorted as `claims` sorted them
:skipped [claim …] ; no English to judge
:batches n :usage {…} :model "…"}
`opts`: `:provider` (default: the offline stub), `:model`, `:num-ctx`, `:max-tokens`,
`:batch-size`.
Renumbering per batch is deliberate: each turn's item numbers start at zero, so a
model that answers `item: 3` in the fourth batch has answered the fourth claim of that
batch and not the fourth of the run. Nothing here writes to a KB — the claims arrived
as data and leave as data.(judge-batch claims {:keys [provider] :as opts})One turn over one batch of claims -> the claims with :verdict and :note on them.
A claim the answer skipped comes back :unanswered, which is a different thing from
every verdict and is counted as one.
One turn over one batch of claims -> the claims with `:verdict` and `:note` on them. A claim the answer skipped comes back `:unanswered`, which is a different thing from every verdict and is counted as one.
(judgeable? claim)Is there an English sentence here to judge? A gloss that is entirely :named is the
formal sentence with its symbols spaced out, and asking about it measures the prompt.
Is there an English sentence here to judge? A gloss that is entirely `:named` is the formal sentence with its symbols spaced out, and asking about it measures the prompt.
(line {:keys [index text givens]})One claim as the prompt shows it: its number, the situation it rests on when it has one, and the claim itself.
Each given keeps the capital it was composed with — lower-casing an opening Fido to
make the join read better would misspell the vocabulary, which is the one thing a gloss
may not do.
One claim as the prompt shows it: its number, the situation it rests on when it has one, and the claim itself. Each given keeps the capital it was composed with — lower-casing an opening `Fido` to make the join read better would misspell the vocabulary, which is the one thing a gloss may not do.
How many supporting facts a derived claim shows. A conclusion resting on more than this has a situation nobody reads at a glance, and the line stops being one sentence.
How many supporting facts a derived claim shows. A conclusion resting on more than this has a situation nobody reads at a glance, and the line stops being one sentence.
The JSON schema decoding is constrained to. item is the number of the claim being
answered, and it is what makes a skipped claim visible: an answer with fifteen verdicts
to twenty claims has five gaps, and gaps are reported rather than read as agreement.
The JSON schema decoding is constrained to. `item` is the number of the claim being answered, and it is what makes a skipped claim visible: an answer with fifteen verdicts to twenty claims has five gaps, and gaps are reported rather than read as agreement.
(parse-verdicts text n)A model's answer -> {item -> {:verdict :note}}.
Tolerant in the three ways this answer goes wrong and strict in the one that matters:
a fence is stripped, a word outside the enum reads as :unsure rather than throwing,
and a repeated item keeps the first answer — but an item number that names no claim
is dropped, because a verdict that cannot be attached to a claim is not evidence about
anything.
A model's answer -> `{item -> {:verdict :note}}`.
Tolerant in the three ways this answer goes wrong and strict in the one that matters:
a fence is stripped, a word outside the enum reads as `:unsure` rather than throwing,
and a repeated item keeps the **first** answer — but an item number that names no claim
is dropped, because a verdict that cannot be attached to a claim is not evidence about
anything.(report result)The whole result as text: the counts, then every disagreement with the claim's strength and the judge's note.
The disagreements are the report and the rate is the index into them, so they are
printed in full rather than summarized. A :default beside one is not an excuse for
it — it is the fact a reader needs to decide whether the judge found a bad claim or a
claim that is simply defeasible, and those want different work.
The whole result as text: the counts, then every disagreement with the claim's strength and the judge's note. The disagreements are the report and the rate is the index into them, so they are printed in full rather than summarized. A `:default` beside one is not an excuse for it — it is the fact a reader needs to decide whether the judge found a bad claim or a claim that is simply defeasible, and those want different work.
The instruction half, which never varies — so a provider that caches a system turn caches this one across every batch of a run.
The instruction half, which never varies — so a provider that caches a system turn caches this one across every batch of a run.
(user-turn claims)The volatile half: the numbered claims and nothing else. No vocabulary card and no formal sentences — a judge shown the s-expression would answer about the s-expression.
The volatile half: the numbered claims and nothing else. No vocabulary card and no formal sentences — a judge shown the s-expression would answer about the s-expression.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |