vaelii.impl.llm.text — com.vaelii/vaelii 0.1.0

The document-scoped prompt: text in, candidates out — never knowledge in.

vaelii.impl.gloss composes English out of the KB and says why that direction is the dangerous one: nothing in the engine can check that a sentence means what a text said. This namespace is the other direction, and it has to answer that argument rather than ignore it, because every check the engine has passes on a correct-looking translation of the wrong claim — naming, well-formedness, argument types and disjointness all read the sentence. So what is built here is a candidate generator with a reviewer between it and the store, and every decision below follows from that:

• a candidate is a [sentence context opts] entry, which is the shape vaelii.core/edit already takes and the browser's editor already parses — so a proposal lands as a reviewable diff and there is no second write path;
• a candidate carries the span of the text it came from, so an accepted sentence is auditable back to the sentence that produced it;
• a candidate is :default, never :monotonic unless the reader says so — a translated guess asserted as known-true would defeat hand-written defaults;
• what the pipeline could not translate is part of the answer (coverage), because a reader who is shown only the two-thirds that worked reads it as a reader that understood the document.

Where the boundary is. Nothing here is in vaelii.core, and nothing here writes. Like web / serve / llm, this is an application over the engine: the engine's contribution is the critic (check-edit), the vocabulary (the taxonomy and the declarations), the provenance side map, and the equality partition — all of them public reads that existed already. See docs/reading.md.

Resolution is the problem; parsing is not

A pipeline that coins has_black_and_white_feathers for every sentence produces fragmentation rather than knowledge, and no naming check refuses it (docs/naming.md says so in as many words). So the document's own words are resolved against the vocabulary the KB already has before a model is asked anything:

• spellings turns each run of the document's words into the symbols a KB term could be spelled as — prepared for winter into preparedForWinter and prepared_for_winter, Fido into Fido — and known asks the KB which of them it has. Generating and asking runs the opposite way from inverting the KB's vocabulary into the words each term is written with, which is the one read here that would grow with the KB;
• resolve-in walks the document longest-run-first and non-overlapping, so a compound predicate is not shredded into the words its name is made of;
• every resolved term is the equality partition's representative, so a word spelled at a retired name resolves to the term that name was merged into.

What resolves becomes the vocabulary card (document-inventory), which is the prevention half of the fragmentation guard; the detection half is vaelii.impl.llm.inventory/coined, unchanged and shared with the other three paths.

candidate-entry clj

(candidate-entry {:keys [sentence segment confidence strength]}
                 context
                 source
                 segs)

One read candidate -> the [sentence context opts] entry vaelii.core/edit takes.

opts carries the provenance — {:source :segment :span :confidence} — which is the reason to prefer this over a script that pastes facts in: the per-handle provenance map is open and unread by belief (docs/storage.md), so an accepted sentence stays auditable back to the characters it came from. A candidate whose segment names no sentence gets no span and says so with :segment nil, rather than being given a plausible one.

:strength is omitted unless the candidate claimed :monotonic, so a candidate is :default by construction: a translated guess that defeated a hand-written default would be the worst outcome this pipeline has available.

confidence-ranks clj

The three tiers a candidate can claim, as the number provenance records.

A rank, not a probability: it orders a review queue and nothing else reads it. A number rather than the keyword because ordering is the whole job, and belief never reads provenance at all — so this cannot become a defeat class even by accident. There are two strength classes and there will not be a third (docs/nmtms.md); a parser's confidence lives here instead.

coverage clj

(coverage segs candidates untranslated)

Which of the document's sentences the reader translated, and which it did not:

{:segments 5 :covered 4
 :uncovered [{:index 1 :span [42 61] :text "It was a fine morning."
              :reason "scene-setting, no claim"}]}

Computed, not reported. A segment is uncovered when no candidate names it, whatever the model listed in untranslated — a model that quietly drops a third of a document and says nothing would otherwise read as one that understood it. A stated reason is attached where there is one and the segment really did produce nothing; where the model claimed a sentence was untranslatable and produced a candidate from it, the candidate wins and the claim is dropped.

Taken over the candidates as read, not over the ones that survived. A sentence the reader turned into something the KB already stores, or into something the critic refused, was still read — reporting it as untranslated would blame the reader for the winnowing, and coverage is a claim about the document, not about the batch.

This is part of the default answer rather than something a caller asks for, which is the whole point of it.

declared-in clj

(declared-in kb context)

Every predicate declared in context's cone, as [[predicate arity] …] — its unaryPredicate / binaryPredicate / ternaryPredicate memberships, read at that context so the answer is what a sentex filed there would be allowed to use.

Scoped where vaelii.impl.llm.inventory/declared-arities is not, and that is the difference that matters here: a candidate is filed in one context, and the vocabulary of the microtheory it lands in is precisely the vocabulary it may reuse. What this puts on the card is names and shapes, never content — a predicate's declaration says it exists and takes two arguments, and says nothing about what is true of anything.

Ordered nearest microtheory first, because that is where a cap has to cut. The cone of a leaf context runs from the story it is about up to the vocabulary head, and a predicate the story's own theory declares is worth more to a reader of that story than one CoreContext declares — so the cone is sorted by how much each context sees, which is largest at the leaf and smallest at the head. Alphabetical within one context, so the order is a function of the taxonomy and never of arrival.

The cone is walked term by term because a read at a ground context is exact-context: sentexes-matching at LionMouseContext answers about that context alone, and what the context sees is context-up, a cached closure lookup. So this is one narrow read per context in the cone.

document-inventory clj

(document-inventory kb resolved context)

(document-inventory kb
                    resolved
                    context
                    {:keys [max-relations max-types max-genls]
                     :or {max-relations 60 max-types 60 max-genls 4}})

The vocabulary a document should be written in, in vaelii.impl.llm.inventory/inventory's shape so its renderer takes it unchanged: {:types :relations :structural :dropped}.

Three relevance tiers, in the order a token cap should cut them from the bottom:

1. what the document's own words resolved to — the vocabulary the text demonstrably wants, and the reason resolution runs before the model is asked anything;
2. what an argIsa licenses for a resolved type, so a document about a mouse is offered the relations a mouse can stand in;
3. everything else context declares, nearest microtheory first (declared-in) — the vocabulary of the theory the candidates land in. Last because it is the least targeted, and included at all because a document rarely spells a predicate the way the KB does: repay the kindness does not resolve repaidKindness, and a reader who cannot see that name coins a synonym for it.

The type block is the resolved types plus one step up the taxonomy, so mouse puts animal on the card and the block reads as a hierarchy.

Cost is the vocabulary's, not the document's and not the KB's — the same shape vaelii.impl.llm.inventory/inventory has, and the same reason: the taxonomy's node set, the arity declarations, one comment read per rendered term, and one narrow argIsa query per resolved type and per rendered predicate. Nothing here walks facts. The card tracks the document (tiers 1 and 2 are seeded by what resolved); the reads track how much vocabulary the KB has.

opts: :max-relations (60), :max-types (60), :max-genls (4). Smaller than the page path's caps by design: a document resolves many terms where a page has one, and the window belongs to the text.

entry-provenance clj

(entry-provenance entry)

The provenance map on an entry built by candidate-entry, or nil.

known clj

(known kb spellings)

The generated spellings the KB actually has a term for, as {spelling representative}.

One roster read, not a probe apiece. vaelii.impl.llm.inventory/unknown-terms asks the index's O(1) roots first because it is asked about a proposal, where nearly every functor is already known; a document is the opposite case — a hundred words generate hundreds of spellings and almost none of them name anything — so the roots would be probed hundreds of times to answer no, where the roster answers all of them at once.

A term is kept only if its spelling gives it a role a candidate could use: a context is dropped (candidates are filed in the caller's context, never one the text names) and so is the structural frame. Every value is the equality partition's representative, so a word spelled at a retired name resolves to the term that name was merged into.

numbered clj

(numbered segs)

The document as the prompt shows it: one numbered line per segment. Numbering is what the segment field on every candidate refers back to, so it is the whole of how a span survives the round trip through a model.

output-schema clj

The JSON schema decoding is constrained to — the contract on this path, for the page path's measured reason: on generation a schema is what makes a local model answer in s-expressions at all rather than in an essay.

Two fields do work no other path needs. segment is the index of the sentence the candidate came from, and it is how the span reaches provenance — the model is the only thing that knows which sentence it was translating, and asking for the offsets instead would be asking it to count characters. untranslated is where a sentence it could not formalize goes, with a reason; the coverage report does not trust it (a segment that produced no candidate is uncovered whether the model said so or not), but a stated reason is worth more to a reviewer than an inferred silence.

resolutions clj

(resolutions kb segs)

Every term the document's own words already name, in document order.

This is the whole of what the pipeline knows about the text before a model is asked anything, and it is what the vocabulary card is built from — the prevention half of the fragmentation guard. Two passes over the document and one KB read: every spelling the document could contain is generated first, known answers all of them together, and the walk is then arithmetic.

resolve-in clj

(resolve-in known segs)

The resolutions in segs against an already-computed known map, as [{:surface "prepared for winter" :term preparedForWinter :segment 0 :span [12 31]} …].

Longest run first and non-overlapping: prepared for winter wins over winter, so a compound predicate is not shredded into the words its name is made of. Within one length the spelling order of spellings decides, which puts the predicate reading ahead of the type reading — the same precedence the naming invariants give a bare lowercase word.

Separate from resolutions so the walk is testable without a KB: everything here is arithmetic over the segments and the map.

review-queue clj

(review-queue entries flagged)

The candidates in the order a reviewer should read them: flagged first, then least confident first, then document order.

flagged is the set of entry indices something has a reservation about — a coined functor (vaelii.impl.llm.inventory/coined) or a shape vaelii.impl.llm.correct would rewrite. Both are failures the check chain admits by design and only a person can settle, so they sort to the top whatever the candidate claims about itself; confidence is the parser's own account of how much of the claim was reading and how much was guessing, and it breaks the tie — which is all a confidence is for here.

segments clj

(segments doc)

A document cut into sentences: [{:index i :text "…" :span [start end]} …].

The span is what everything downstream is for — (subs doc start end) is the segment's text, exactly, so an accepted candidate points back at characters rather than at a paraphrase. A test holds that identity.

Cut at a run of ./!/? followed by whitespace or the end of the document, which is crude and deliberately so: an abbreviation splits a segment in two, which costs a narrower span and no correctness, while a cleverer splitter would need a model and put one on the reading side of the pipeline where there is nothing to check it. A semicolon or a colon does not cut — a segment may carry several claims, and which claim came from which clause is not something the span can honestly say.

spellings clj

(spellings words)

The symbols a run of words could be a KB term spelled as.

The KB reads a term's role off its spelling, so a phrase has one candidate per convention: prepared for winter could be the predicate preparedForWinter or the type prepared_for_winter, and one word could be lion, Lion or its singular. Generating the spellings and asking the KB which exist is the whole of resolution — the alternative, inverting the KB's vocabulary into the words each term is written with, is the one read here that would grow with the knowledge base.

Nothing is filtered by plausibility: allThroughThe is generated and the KB says no, at the cost of one set lookup.

system-prompt clj

The instruction half — static, because everything about the KB and the document rides in the user turn.

What it spends its tokens on is the four failures this direction has that the other three do not: inventing vocabulary the card already holds (the shared failure, and the one the card is there to prevent), stating a general claim as a fact with a loose variable, introducing a character twice under two names, and quietly translating what the text did not say. The last is the one no check downstream can catch, so it is the one stated most plainly: the reader would rather have a sentence reported as untranslatable than a fluent sentence that is not in the text.

user-turn clj

(user-turn kb segs resolved context instruction)

(user-turn kb segs resolved context instruction opts)

The volatile half: the numbered document, the vocabulary its own words resolved to, the context candidates are filed in, and the reader's instruction if they gave one.

The document comes first and the vocabulary after it, the opposite of the page path. The card is here to be consulted while translating rather than to be summarized, and a reader's instruction — when there is one — goes last so it is the newest thing in the window.