com.blockether.vis.internal.ctx-engine — com.blockether/vis 0.1.38

Pure helpers for the model-facing context snapshot.

The mutable session state lives outside this namespace; these functions only advance the turn/iteration cursor, project form envelopes for persisted result messages, and compute utilization metadata.

advance-iter clj

(advance-iter ctx _form-results-vec)

Advance the cursor so the next iter starts at "iter" (current+1), "next_form" 1. Prior tool outputs are carried by append-only message history, not by mutable context state. CTX is STRING-KEYED end to end — it is bound into Python as the context dict and nippy-persisted, and both surfaces are strings-only.

block->envelope clj

(block->envelope block position cursor)

(block->envelope block _position cursor head-tag-resolver)

Project one loop-side block {:code :result :error :stdout} plus its 1-based position and the engine cursor into the form envelope shape (one block = one form, 1:1):

{:scope :tag :src :duration-ms :result :error :stdout}

:src carries the block's source text. :tag is derived from the source via classify-form-tag. :result is included only when the block has one (engine convention: drop on default/nil). :error is included only when the block errored. :stdout (what the block PRINTED) is the single display surface — channels paint it, and the model reads it back. :duration-ms is derived from the loop's block envelope so persisted TUI replays keep the same footer timing as live progress bubbles.

Every result is walked through realize-value so any IDeref (Var, atom) is derefed and lazy seqs land as data, never as #:vis{:ref :expr} placeholders left over from persistence flattening unrealized seqs.

blocks->forms clj

(blocks->forms blocks cursor)

(blocks->forms blocks {:keys [turn iter]} head-tag-resolver)

Map a loop-side blocks vec into a vec of engine envelopes. :cursor is {:turn :iter} of THIS iter; each block gets a 1-based form position by its index in the vec.

3-arity passes head-tag-resolver (see classify-form-tag) through to every block->envelope call so extension-declared mutation tools (struct_patch, db/commit!, any symbol with inline :tag on its vis/symbol entry) classify correctly without the engine hard-coding their symbol set.

classify-form-tag clj

(classify-form-tag src)

(classify-form-tag src head-tag-resolver)

Classify a form-source string as :observation or :mutation.

1-arity: pure, engine-only. Returns :mutation when the head is a member of core-mutation-heads; everything else is :observation. Use this from contexts that have no access to the extension registry (tests, pure tools).

2-arity: takes head-tag-resolver, an optional fn (fn [^Symbol head]) -> :mutation | :observation | nil. The resolver wins when it returns a non-nil tag; on nil the engine falls back to core-mutation-heads. The integration layer in loop.clj builds the resolver from extension/op-tag so every extension-declared tool (struct_patch, db/commit!, anything new an extension ships) classifies correctly without the engine hard- coding its symbol.

compact-src clj

(compact-src src)

One-line, length-capped form source for the you ran: scope index rendered in the cross-turn <conversation-so-far> resume block.

compress-scopes clj

(compress-scopes scopes universe)

Compress a seq/set of concrete tN/iN scopes into a compact, model-legible token vector, RELATIVE to universe (every tN/iN currently on the wire):

• a turn whose EVERY universe iteration is present → "tN/*"
• a run of consecutive such FULL turns → "tA-tB/*"
• full turns covering the WHOLE universe → ["t*"]
• a partially-present turn stays explicit, its iteration numbers themselves run-compressed: "tN/iM" (singleton) or "tN/iA-iC" (run). Tokens are ordered by (turn, iter). Scopes/universe entries that don't parse are ignored. Pure — same inputs → same output.

DEFAULT_PROMPT_BUDGET_TOKENS clj

Soft per-call operating budget surfaced to the model as session_utilization.auto_compress_above. It keeps routine work below a 200k-token handled-context allowance while the provider-reported input limit remains the hard ceiling. Compared against provider-reported input tokens (no local tokenizer) — see utilization.

empty-ctx clj

(empty-ctx)

(empty-ctx session-id)

A minimal CTX scaffold with all model-facing keys filled by empty / default values. Useful as the starting point for scenario replays.

Includes the engine-ephemeral key "engine_warnings" so the rest of the system can swap! it without nil-puncturing. Stripped at persistence boundaries via strip-ephemeral.

CTX is STRING-KEYED end to end: it is bound into Python as the context dict (strings-only boundary) and nippy-persisted (strings-only DB), so there is no keyword->string projection anywhere between.

enter-turn clj

(enter-turn ctx turn-pos)

Idempotent turn-start sync. Sets :session/turn to turn-pos, resets :session/scope to {:turn turn-pos :iter 1 :next-form 1}, clears :engine/blockers, then runs gc-pass. Safe to call repeatedly with the same turn-pos (no-op semantically); safe to call when ctx was loaded fresh from DB at turn-pos > 1.

THIS is what the integration layer (vis loop) calls at the start of every turn. Single chokepoint for engine turn-state advance — no advance-turn alias, no auto-incrementing variant. The integration layer always knows the target turn-pos (DB tracks session_turn_soul.position), so the engine takes it as an explicit arg.

expand-through clj

(expand-through summaries universe)

Resolve every fold SELECTOR on each summary against universe (the caller's own live iteration scopes) into a concrete "scopes" set, so ONE intent expands consistently wherever summaries are read (apply-summaries: the trailer; resume / prior-turn: that turn's forms; folds-view: the ledger). Selector keys (all optional, their results UNIONED): "scopes" explicit ids — a tN/iN is kept verbatim; a bare tN EXPANDS to every iteration of that turn present in universe. "through" tN/iN cursor → every universe scope AT OR BEFORE it (start→cursor). "from"/"to" inclusive window — either bound optional (open start / end). "since" tN/iN cursor → every universe scope AT OR AFTER it (cursor→newest). A range cursor may also be a bare tN — a WHOLE-TURN boundary: through/to tN cover all of turn N (its last iteration), from/since tN its first. The range keys are dropped after resolution and the union lands in "scopes".

EXPLICIT whole-turn intent is recorded as "turns" (a set of turn numbers): a bare tN token, or a RANGE selector whose resolved window covers every universe iteration of that turn. An ENUMERATED tN/iN list never yields whole-turn intent — even when it happens to name every iteration — so a fine-grained fold can never silently erase a turn's Q/A recap downstream (previous-turn-context keys Q/A removal off "turns", not scope cover).

Intents with none of these keys pass through untouched. Pure — same inputs → same output.

finalize-turn clj

(finalize-turn ctx _form-scope _args)

Finalize a turn: the loop ships :answer to the channel and the engine returns ctx unchanged so the turn can settle.

fold-key clj

(fold-key k)

Parse fold_session's KEY — the first of the verb's two arguments (a key and an optional gist), and the only shape the verb accepts — into one selector intent plus the label its ack card shows. Tokens are comma- or space-separated and .. and - are the same range separator: "t2" a whole turn "t2/i5" one iteration (a tN/iM/fK form id folds its iteration) "t2/i1-i56" an inclusive window; the right side may drop the shared turn "t1/i3-t4" a window across turns (a bare tN bound covers all of turn N) "-t2/i56" open start: every settled step at or before the cursor "t2/i5-" open end: every settled step at or after it Several keys union ("t1, t2/i1-i3"); at most ONE range per call, because one intent carries one window. The parsed intent is what expand-through resolves; its string keys are a RECORDED shape (they persist in the ctx nippy blob), never something a caller passes in. Returns {:intent … :label …}, {:error <model-facing refusal>} for an unparseable token, a second range or an argument that is not a key string, or nil when the key named nothing at all. Pure.

fold-key-grammar clj

The fold KEY grammar in ONE line, said verbatim by every refusal so the caller reads the whole vocabulary at the moment it typed something else.

folds-view clj

(folds-view summaries universe weights util)

(folds-view summaries universe weights util turn-weights)

Model-facing LIVE BUDGET derived from the recorded fold_session intents.

The GIST of each fold lives ONCE — in its transcript breadcrumb, rendered in place where the collapsed content was, carrying its file:line anchors (see pretty-scopes). So this view deliberately holds NO gists; echoing them here would duplicate the breadcrumb that is already on the wire. It emits only the volatile budget signal:

"now" VOLATILE but TINY — shaped "context <U>% · saved <C>/<T> (<P>%, ~<toks> tok) · live <scopes>": saved how much of the wire (<T> = every tN/iN still on the wire, so folded scopes that already scrolled off the trailer never inflate it) is folded away, priced BOTH in scopes (<C>/<T>) AND — when weights are stamped — in reclaimed context (~<toks> tok, summed from engine_iter_weights with the SAME estimator the fold_session card and the over-budget nudge use, PLUS the Q/A recaps of explicitly whole-turn-folded turns priced from turn-weights / engine_turn_weights); and live the compressed scopes STILL on the wire (accounting only; current-turn scopes are not foldable). No gists. context is the LIVE per-call saturation (util's saturation), so the same delta carries how full the window is now. No gists, no position (the # tN/iN step tag the model already sees carries that) — so re-emitting it every iteration costs a handful of tokens.

Returns {"now" …} when a universe (the live tN/iN scopes this send) is stamped: selectors resolved (expand-through), covered folds dropped (supersede-summaries), budget computed against the live wire. Without one (resume / fresh seed, before the first live send) returns {} — the breadcrumbs still carry every gist until the next send re-stamps the universe. The token clause is best-effort: no weights (or a scope not yet weighed) simply omits ~<toks> tok, leaving the scope counts — never breaks the line. turn-weights (optional fifth arg, {turn-number → ~tokens}) prices the removed Q/A recap of every whole-turn-folded turn into the same clause. Pure.

form-head-name clj

(form-head-name src)

Return the head call NAME (a string) of src — a Python source string — or nil when src is not a name(...) call form. Leading comments and blank lines are skipped. Reading the head name (rather than scanning the raw source) avoids false positives — a "struct_patch(x)" inside a string can't match. Used by classify-form-tag.

gc-pass clj

(gc-pass ctx)

Passthrough. Tasks/facts/archive are gone — there is nothing to GC. Kept so the turn-lifecycle chokepoint (enter-turn) and any external callers stay valid.

malformed-scope? clj

(malformed-scope? s)

True if s is a string but does not parse as ::cs/scope-form.

model-facing-keys clj

EXACT set of session_* keys the model is meant to see. Security access is included as an environment-derived value; engine bookkeeping remains hidden.

model-form-envelope clj

(model-form-envelope r)

Project one executed-form envelope onto the MODEL contract the trailer stores: :scope :src :result :error (+ engine forensic fields), WITHOUT the channel sink slice, the host failure/metadata chains, or :tag. The mutation/observation tag stays load-bearing on LIVE op envelopes and on the persisted rows — but NOTHING reads it from the trailer (the observation-prune that once did was removed), and the model can see what a form does from :src. The full envelopes stay on the progress chunks and the persisted session_turn_iteration.forms rows — channels and the persisted forms rows keep total fidelity; the trailer is what rides every prompt.

Empty payloads are DROPPED, not rendered: "result": None / [] / {} and empty :error maps say nothing the form's absence of an error/result doesn't already say — the :src stays, the dead field goes.

over-budget-hint clj

(over-budget-hint util current-turn since-turn)

Stateful compaction guidance for session_utilization.

Pressure starts at 75% of auto_compress_above, escalates at 90%, and becomes mandatory above the operating ceiling. Once armed it remains visible while pressure remains; an ignored warning must never silently expire. The hint names fold_session, the safe settled boundary, and the evidence to preserve. Pure.

parse-scope-form clj

(parse-scope-form s)

Parse a tN/iM (or legacy tN/iM/fK) scope into {:turn :iter :form} or nil if malformed. :form is nil for an iteration-level scope. Pure value-or-nil.

pretty-scopes clj

(pretty-scopes scopes universe)

Render a seq of concrete tN/iN scopes as ONE beautiful, model-legible anchor string using the SAME grammar the fold ledger speaks (compress-scopes + join-scopes): same-turn iter-runs merge (t3/i3-i4,i7-i8), whole turns collapse to tN/* / tA-tB/* / t*. universe (the live wire scopes, or nil) drives the whole-turn collapse; nil/empty keeps every turn explicit. nil for an empty/unparseable seq. Pure. Shared by the ledger AND the transcript fold breadcrumb so a collapsed region reads identically wherever it surfaces — one anchor vocabulary, not two.

scope-compare clj

(scope-compare a b)

Total order on form-scope strings by (turn, iter, form). Returns int. Compares parsed segments; malformed scopes sort before all valid ones to make their presence obvious in render.

scope-key clj

(scope-key scope)

Ordered key for a scope so ranges can compare scopes: "t1/i2" or "t1/i2/f3" → [1 2] (the form index is dropped — ranges cover WHOLE iterations). nil when the scope can't be parsed, so callers skip it rather than mis-order it. Lives here (below apply-summaries in loop) so BOTH the wire (apply-summaries) and the render-time ledger (folds-view) resolve fold selectors through the SAME function.

session-view clj

(session-view ctx)

(session-view ctx _warnings)

THE single projection from engine-internal ctx to the model-facing session_* view.

Both consumers derive from this, so the rendered <context> block and the Python session dict are the same map by construction:

• ctx-renderer/render-ctx serializes this view
• ctx-loop/session-snapshot binds this view as read-only session

Keeps ONLY model-facing-keys (so engine bookkeeping never leaks) and projects "engine_utilization" → "session_utilization". The second arity takes legacy warnings for call-site compatibility but ignores them. Pure; STRING keys in and out.

strip-ephemeral clj

(strip-ephemeral ctx)

Remove transient context keys before Nippy-snapshotting: engine bookkeeping must never become durable session state.

supersede-summaries clj

(supersede-summaries summaries)

Collapse 'summary of summary': drop any summary whose scope set is fully COVERED by another's — a proper subset, or an equal set recorded earlier — so re-folding a region with a broader/newer gist REPLACES the finer breadcrumb instead of stacking a second line. Coverage is never lost: every scope of a dropped summary is present in the one that supersedes it (the superset wins; for equal sets the later/newer wins), and the dropped summary's explicit whole-turn intent ("turns") is MERGED into a surviving coverer so a fold-of-fold can never resurrect an already-folded turn's Q/A recap. Order-stable. Expects scopes already resolved (run AFTER expand-through). Pure.

turn-key clj

(turn-key scope)

Turn number of a bare WHOLE-TURN scope "tN" (no /iN); nil otherwise, so expand-through only whole-turn-expands an id that is JUST a turn — a plain "t1/i2" stays a single iteration.

utilization clj

(utilization request-tokens window-tokens turn-tokens fold-cap)

Pure: the "session_utilization" map the model reads to see how much of the context window the LAST request consumed. Keys are spelled out so they can't be misread: last_request_tokens input size of the most recent model call model_input_limit HARD per-call ceiling (provider rejects above) saturation last-request / model-input-limit, as a rounded percentage — how FULL the per-call window is headroom_tokens tokens still free before the ceiling (model_input_limit - last request); the actionable 'can I keep going or must I fold?' auto_compress_above soft guardrail threshold for request size turn_total_tokens cumulative input this turn (billing, NOT a per-call limit — may exceed the limit safely) hint throttled compaction nudge, present ONLY when the handled context has grown past auto_compress_above (a bigger task) — the actionable partner to the passive ceiling numbers; added by session-view from over-budget-hint, self-silences after 3 turns Returns nil until a request has actually been measured (req <= 0), so the first iter of a turn shows nothing rather than a bogus 0%.