vaelii.impl.llm.score — com.vaelii/vaelii 0.1.0

Scoring a set of candidate entries against a hand-written one.

vaelii.impl.llm.text produces candidates and vaelii.core/check-edit says whether each is admissible. Neither says whether it is right, and nothing in the engine can: the whole reason the reading direction needs a reviewer is that every check passes on a well-formed translation of a claim the text did not make. So the only honest measure is against knowledge somebody wrote by hand, and this is the arithmetic for that.

The gold set is read out of a KB, never transcribed

A second copy of the fables' sentexes in a scoring fixture would drift from the ones the suite actually loads, and a score against a stale gold set is worse than no score. So the gold is a set of handles in a loaded KB, and a candidate matches when vaelii.core/handle-of finds it under one of them.

That also means the comparison uses the engine's own canonical form rather than a reimplementation of it: a rule whose variables are named differently, whose antecedents arrive in another order, or whose symmetric arguments are the other way round is the same sentence to handle-of and is therefore the same sentence here (docs/canonicalization.md). Nothing about matching is this namespace's opinion.

Two scores, because the constants are unrecoverable

A fable introduces its characters by kind — a lion, a mouse — so the names in the formal version (LionA, MouseA) are the modeller's, and no reader of the text could produce them. A strict score therefore reads zero on stories whose structure was recovered perfectly, which measures the naming convention rather than the reading.

So score reports both, and the pair is the finding:

• strict — the candidate matched a gold handle as written;
• aligned — the same comparison after renaming the candidate's introduced individuals onto the gold's, one-for-one, by the types each is asserted to have (alignment). A renaming is a bijection or it is not applied, so alignment can never merge two characters into one to score better.

Nothing here writes: handle-of is find-only, and the alignment is arithmetic over sentences.

alignment clj

(alignment gold-sentences candidate-sentences)

A renaming of the candidates' introduced individuals onto the gold's, or {}.

Matched by overlap in the one-place claims made of each (typed-as): a candidate individual said to be a lion aligns with the gold individual said to be a lion. Overlap rather than equality, because a candidate that also invented a claim about its character — (has_black_and_white_feathers Lion1) beside (lion Lion1) — is a character the reading did recover, and scoring it as unrecovered would count the spurious claim twice.

A bijection: pairs are taken greedily in descending overlap, each side used once, so no renaming can collapse two characters into one to make more sentences match. Ties break on the two names, so the answer is a function of the two sentence sets and of nothing else. A candidate that used a name the gold uses is left alone, whatever the gold says about it — that candidate got the name right, and renaming it onto a different gold character would turn a wrong claim into a matching one.

Deliberately narrow. Aligning on anything richer — the relations a character stands in, say — would start fitting the candidate set to the gold, and a score that repairs its own input measures the repair.

derived-handles clj

(derived-handles kb context)

The sentexes context holds that nobody asserted — what forward chaining put there. A candidate landing on one of these is neither right nor wrong but not news: it is the same no-op re-assert vaelii.impl.llm.session/new-assertions counts as :known on the page path, and it is scored the same way — out of precision's denominator rather than against it.

gold-handles clj

(gold-handles kb context)

The hand-written version of one document, as handles: every premise stored in context.

Two filters, both load-bearing. Its own context, not the cone above it — a story context sees the upper ontology through genlContext, and scoring a reader of one fable against the whole shipped schema would measure a recall it was never asked for. And premises only: a forward-chained conclusion like (repaidKindness MouseA LionA) is the engine's contribution rather than the modeller's, so asking a reader to produce it would score the chaining twice. exceptWhen meta-sentexes stay, because a rule's exception is something the modeller wrote.

derived-handles is the other half — what the KB stores and nobody wrote.

line clj

(line name {:keys [strict aligned]})

One scored document as a table row — name, then the aligned pass's numbers with the strict ones beside them, because the gap between the two is the finding.

rename clj

(rename m sentence)

sentence with every symbol m maps replaced, at any depth. Goes through vaelii.impl.sentex/canon so the result is the shape the store keys on: a lazy seq, a PersistentList and a vector are all = and freeze to different bytes, so a renamed sentence built out of lazy seqs would miss a handle-of match that is really there.

score clj

(score kb context sentences)

Score sentences — a candidate set, all filed in context — against the hand-written sentexes in that context.

{:strict  {:matched 3 :candidates 11 :derivable 1 :gold 6 :precision .3 :recall .5 :f1 .38}
 :aligned {:matched 5 …}
 :renaming {Lion1 LionA, Mouse1 MouseA}
 :missing [(repaidKindness …) …]        gold nobody produced, as readable sentences
 :spurious [(hunter Hunter1) …]         produced, matching nothing stored}

:missing and :spurious are the aligned pass's, since those are the confusions worth a reader's time: the strict pass's misses are dominated by the naming of characters, which is a fact about the convention and not about the reading. Both are sorted by their printed form, so a report is a function of the score and not of arrival order.

A duplicate candidate matching an already-matched handle counts once: recall counts gold covered, so writing the same sentence twice cannot buy a second match.

table clj

(table scored)

Several scored documents — [[name score] …] — as a markdown table, with a total row. The total is computed from the summed counts rather than by averaging the rates, so a short document cannot weigh as much as a long one.