com.blockether.vis.internal.env-python — com.blockether/vis 0.1.38

Embedded-GraalPy sandbox machinery — the agent's action substrate. The agent writes Python; this ns embeds a GraalPy org.graalvm.polyglot.Context, marshals values across the Clojure↔Python boundary, wires the Clojure tool fns into the Python globals as ProxyExecutables (so grep({"query": "x"}) in Python runs the Clojure grep), and runs the model's code block as ONE whole-block coroutine.

Public surface used by the loop:

create-python-context / set-python-binding! / bind-and-bump! / bind-and-bump-with-doc! / count-top-level-forms / validate-non-empty-block! / validate-no-banned-defs! / persist-session-defs! / restore-session-defs! / SYSTEM_VAR_NAMES / system-var-sym? / lru-atom / current-turn-position / fresh-lru-atom / run-python-block / map-polyglot-error / bind-ctx! / ctx->python-str

The :python-context slot holds the GraalPy Context; the Python top scope is context.getBindings("python"). GraalPy ships in the default deps (runs on GraalVM CE 25.1.3 → Truffle gets the Graal JIT; see .graalvm-version).

*auto-repair-brackets?* clj

When true, a bracket-balance syntax hint ALSO appends repair-bracket-balance's single-candidate suggested fix. OFF by default: the walker only DIAGNOSES; the auto-fix stays gated behind this flag until proven safe in the wild.

*current-turn-position* clj

*lru-atom* clj

->clj clj

(->clj v)

Polyglot Value (a Python value) -> Clojure data. STRINGS-ONLY boundary: dicts -> maps with VERBATIM STRING keys (exactly what Python held — no keywordizing, no regex key-shape sniffing), lists/tuples -> vectors, host objects (Java values that crossed the boundary, e.g. UUIDs) -> their underlying Java value via asHostObject, callables/opaque objects -> the raw Value. A non-string Python dict key (int, tuple, ...) stringifies via its Clojure conversion so the map stays string-keyed and total.

->py clj

(->py x)

Clojure value -> something GraalPy accepts as a Python value. STRINGS-ONLY boundary: map keys must be strings and no keyword/symbol may appear at any depth — a violation throws boundary-violation! naming the key path. Primitives and Strings pass through (the Context auto-converts Java boxed types); collections become polyglot proxies, which GraalPy shows as ForeignDict/ForeignList — subscript, len, iteration, .keys(), dict(_), {**_} and even isinstance(_, dict) behave, but json.dumps refuses them (it dispatches on the exact type), so a value the guest must SERIALIZE is rebuilt by __vis_pyify__ or handed over as a JSON string; leaves convert via leaf->py (UUID/Temporal/Date -> ISO strings).

AUTO_IMPORTED_PYTHON_NAMES clj

Python names installed into builtins for every python_execution context. This is the model-facing inventory; keep it synchronized with auto-imports-python and its real-context regression test.

BANNED_DEF_HEADS clj

Python constructs refused pre-eval — belt-and-suspenders against the obvious sandbox-escape footguns on top of the Context restrictions.

bind-and-bump! clj

(bind-and-bump! env sym val)

Set sym -> val in the env's Python sandbox.

bind-and-bump-with-doc! clj

(bind-and-bump-with-doc! env sym doc val)

Like bind-and-bump! but also records doc in the side __vis_docs__ dict so a future live-vars view can surface name + doc (Python has no var metadata channel for doc text).

bind-ctx! clj

(bind-ctx! python-context data)

Bind the standing context in the sandbox as an ordered polyglot dict (->py → ProxyHashMap, which GraalPy treats as a NATIVE dict: session["k"], .get, .items(), comprehensions and {**ctx} all work) built from the SAME projection the renderer prints — so the live dict and the wire's session["a"]["b"] = … structural deltas agree. Bound under session only — decoupled from r, no legacy context alias. No JSON round-trip.

boundary-view clj

(boundary-view x)

(boundary-view x path)

What a plain-data Clojure value LOOKS LIKE after the GraalPy round trip — the mechanical composition of ->py then ->clj without a Python context. STRINGS-ONLY: string map keys pass VERBATIM, sets/seqs -> vectors, UUID/Temporal/Date leaves -> ISO strings. A keyword/symbol anywhere (key or value, any depth) throws boundary-violation! exactly like the real boundary — fix the producer fixture, never catch it. Idempotent.

Every tool result the model sees in production (serialized structurally by ctx-renderer/render-form-value) has already crossed this boundary, so assertions about what the model reads MUST be written against THIS shape. Tests feed (boundary-view raw-result) to pin that contract without booting GraalPy.

boundary-violation! clj

(boundary-violation! kind x path)

Throw on a keyword/symbol trying to cross the Clojure->Python boundary. The boundary is STRINGS-ONLY: every map that crosses (tool results, ctx, verb payloads) is built with string keys and string enum values at the SOURCE — there is no silent keyword->string conversion, so a keyword here means a producer bug, not data. path is the key path down from the value handed to ->py, so the offending producer field is nameable.

The message ALWAYS says WHERE: an empty path means the offending key sits on the value handed to the boundary itself, and a bare :result/:success? there is vis' own internal envelope — the producer must hand Python the PAYLOAD (:result), never the envelope that wraps it. Without that clause a report reads non-string-key :result with no location at all.

collect-garbage! clj

(collect-garbage! environment)

Best-effort GC between turns. Two steps, because GraalPy reclaims native-extension (numpy/pandas/PIL) memory in TWO stages:

1. guest gc.collect() runs the cycle detector, marking dead native cycles so their Java mirrors become weakly reachable, and
2. a JVM System.gc() then lets the Java tracing GC collect those mirrors and drain the reference queue that actually frees the native RSS (see graalpython IMPLEMENTATION_DETAILS: the guest collect ALONE does not free non-cyclic native objects whose only managed ref was just dropped). Runs while the interpreter is idle between turns, the cheapest time for a pause. Never throws; a closed/cancelled context is ignored.

BOUNDED, and that is the whole point. .eval first acquires this context's Python GIL. In-flight guest work is NOT what holds it up — GraalPy releases the GIL around foreign calls (PythonContext.releaseGilAroundForeignCall; PythonLanguage.shouldGilBeLockedDuringForeignCalls defaults to false), which is why a sibling thread keeps ticking through a whole sh.wait. What blocks here is a LEAKED GIL: a guest thread Thread.interrupt-ed at a GIL boundary and then abandoned dies inside PythonContext.ensureGilAfterFailure, which takes the lock UNINTERRUPTIBLY, and a ReentrantLock whose owner is dead is never released by anyone. Waiting for that is unbounded AND uninterruptible (a cancelled token does not unpark PythonContext.acquireGil), and it sits between the engine unwinding and gateway.state/run-turn! appending the terminal event. One such context wedged a finished turn forever: no turn.completed / turn.cancelled on the wire, the session pinned to a turn nobody was running, the queued backlog never drained, and every channel showed a live panel that Esc could not close. GC is best effort, so give it a budget and walk away; the abandoned daemon thread completes the collect whenever the GIL frees. rt/guest-safepoint! is what keeps a cancel from leaking one in the first place.

count-top-level-forms clj

(count-top-level-forms python-context code)

Number of top-level Python statements in code, parsed inside the session's own GraalPy Context. Comment-/whitespace-only blocks return 0. The source is passed directly to a cached helper — no shared scratch global, auxiliary Context, or cross-thread race.

create-python-context clj

(create-python-context custom-bindings)

(create-python-context custom-bindings roots-fn)

(create-python-context custom-bindings roots-fn network-opts)

(create-python-context custom-bindings roots-fn network-opts stdin)

(create-python-context custom-bindings roots-fn network-opts stdin stderr)

(create-python-context custom-bindings
                       roots-fn
                       network-opts
                       stdin
                       stderr
                       gate-fn)

Create one persistent, deny-by-default GraalPy Context for a Vis session.

custom-bindings maps symbols to tool/verb functions or values. roots-fn optionally grants filesystem access confined to the current workspace roots. Every session Context rides the process-wide shared-engine; parsing and extension shims live inside that same Context. Rendering is pure JVM code, so normal sessions allocate no auxiliary GraalPy contexts. Extensions deliberately SHARE this one session Context (installed as guest callables, not separate contexts) — so a session holds exactly one Context; the only extra is a transient fork-context! child, created solely for sub_loop parallelism. The 4-arity stdin (optional InputStream) is wired to the guest sys.stdin — used by vis-agent python to forward the caller's real stdin; agent sandboxes leave it nil. The 5-arity stderr (optional OutputStream) does the same for guest sys.stderr; nil keeps the JVM's System/err. The 6-arity gate-fn is the :fs/access gate the confined filesystem asks before every path operation (see extension/fs-access-gate); nil leaves only root confinement.

ctx->python-str clj

(ctx->python-str data)

Render plain boundary data as a deterministic, executable Python literal.

This is deliberately a pure JVM serializer: rendering never enters GraalPy, never waits behind a session's GIL, and needs no process-global printer Context or lock. It mirrors the Clojure->Python boundary (string-only map keys, list-like collections, ISO strings for Date/UUID/Temporal) and keeps insertion order plus the historical 100-column layout.

eval-block clj

(eval-block python-context code)

Evaluate a whole Python code block in python-context. Returns {:source code :result <clj>} on success; throws the PolyglotException on failure (caller maps it to the engine error shape). Globals (defs/imports/ state) persist across calls in the same context.

fork-context! clj

(fork-context! custom-bindings)

(fork-context! custom-bindings roots-fn)

(fork-context! custom-bindings roots-fn network-opts)

(fork-context! custom-bindings roots-fn network-opts gate-fn)

Fork a CHILD agent Context for a sub_loop — same deny-by-default sandbox as the main context, built ON the shared Engine so it is SAFE to create even while the parent's eval is running (GraalVM-verified: no Truffle deadlock). custom-bindings wires the child's tool/verb fns, which close over the CHILD's env (its own ctx-atom). Returns the same {:python-context :sandbox-ns :initial-ns-keys} shape as create-python-context. The caller owns the child Context's lifecycle (close it when the sub_loop ends). roots-fn (optional) confines the child's Python filesystem to the current filesystem roots, same as the parent; gate-fn carries the parent's :fs/access gate into the child.

fresh-lru-atom clj

(fresh-lru-atom)

graal-resource-cache-redirected clj

map-polyglot-error clj

(map-polyglot-error python-context e code)

Map a GraalPy PolyglotException into the engine's op-error shape. :phase is :python/syntax for parse errors, else :python/runtime; :line/:column come from the Python source location when present. A host (Clojure-tool) exception is unwrapped so its real message surfaces. Recurring syntax-failure classes get an actionable hint prepended: a NON-ASCII char in code position (em-dash, x, curly quote - CPython's invalid character, precise wherever it lands), a PROSE-leading reply (see prose-leading-syntax-hint, first-line only), and - via parse-diagnose - an unbalanced double-quote or an unbalanced (), [], {} bracket pinpointed to its line/col.

normalize-dict-key clj

(normalize-dict-key s)

Model-input hygiene at the ONE inbound conversion: a dict key spelled ":target" is still a STRING (the model drifting into colon spelling while reading keyword-heavy source), so strip the single leading colon when an identifier char follows and the call just works — no lecture, no failure. Data keys are untouched: line numbers start with a digit, paths with a letter or /, neither with :. Produces strings, never keywords.

partial-stdout clj

(partial-stdout ctx)

Whatever the block currently running in ctx has PRINTED so far, or nil when it printed nothing.

The eval watchdog kills a block from OUTSIDE the guest, so the flat {:stdout} | {:error} outcome run-python-block builds is never reached and everything printed before the wall — every progress line of a long fetch loop — used to be dropped with the frame. The capture buffer outlives the interrupt, so the timeout envelope drains it from here.

persist-session-defs! clj

(persist-session-defs! python-context session-id)

Write this session's own defs beside the session, for a LATER process.

Globals persist naturally across turns because the interpreter does — but the interpreter dies with the PROCESS. Restart the gateway and every helper the session refined is gone while the transcript still shows it, so the next call is a NameError against code the model can still read. __vis_defs_snapshot__ renders the module aliases, scalar constants and function sources that re-create them; this stores that text at paths/sandbox-defs-file.

Best effort and never in a block's way: any failure is dropped. Returns the file when it wrote one, nil otherwise.

polyglot-noise-silenced clj

PROCESS_SURFACE clj

THE sentences about this sandbox's process surface — written ONCE, here, and said verbatim by every surface that has to say them:

• the sandbox-shims prompt block (prompt/sandbox-shims-prompt-block);
• the POSIX refusal (vis-shims/posix.py), when subprocess / os.system / os.popen is called;
• a live handle that cannot be driven (__VisShell__.__vis_op__ in vis-python/async_runtime.py);
• the corpus entry named shell (env-python/create-python-context), when the shell tools are off and apropos("shell") / doc("shell") would otherwise answer with silence.

Every copy of one fact drifts, and the copy the model reads at the moment it is blocked is the one that must be right. Composed, never concatenated ad hoc: ban is the rule and is all the PROMPT says (invocation grammar belongs to the shell symbol's own docs, not to a supplemental block); ban + use is what a call site says, because there the model is already writing the call; off is the toggle state, and it names BOTH doors so silence is never read as "subprocess might still work"; off + extension is what DISCOVERY says, because a model that reads off alone concludes nothing in this product can start a process. The toggle closes the MODEL's door only: an installed Python extension's vis.shell is wired unconditionally in python-extensions/build-context and no toggle gates it.

Reaches Python as the __vis_process_surface__ global (see install-process-surface!), so no .py file carries a copy.

python-binding-names clj

(python-binding-names sym)

Canonical Python global plus intentional compatibility aliases for sym. Used by provider/native discovery to deduplicate the same capability.

remove-python-binding! clj

(remove-python-binding! python-context sym)

Remove sym from the Python sandbox globals ENTIRELY — the member key disappears, so apropos/dir no longer list it and calling it raises a plain NameError. This is how a deactivated tool must vanish: putMember nil only parks a None under the name, which apropos still lists and which calls as 'NoneType is not callable'.

restore-session-defs! clj

(restore-session-defs! python-context session-id)

Re-create the helper definitions an EARLIER process persisted for session-id.

Runs once, on a FRESH context, before the session's first block. The restored source is registered as a real block, so defs("name") and inspect.getsource read a restored helper back exactly like a local one, and it goes through the SAME rewrite (__vis_normalize_module__) so it RUNS like one: a stray await, a plain def whose body awaits, a tool call in its body. Only defs and their imports/constants are stored, so nothing re-executes a previous block's side effects, and a statement that BINDS a name which is a bound tool in THIS process is dropped: a snapshot written before the tool existed used to overwrite it for the whole process.

Returns the number of session-defined functions live afterwards, or nil when there was nothing to restore.

run-python-block clj

(run-python-block python-context code & [_opts])

Evaluate one Python code block in python-context as ONE WHOLE-BLOCK coroutine, returning the FLAT sum-typed outcome:

{:stdout <printed>} ; SUCCESS — python_execution (what it print()ed) {:result <value>} ; SUCCESS — a native tool value (nothing printed) {:error <op-error>} ; FAILURE — the raised error IS the result

__vis_run_async__ AST-wraps the block in an async def, AUTO-SETTLES every bare tool-call STATEMENT at every depth (so grep(x) without await runs even inside try: or a def body), drives it as a single coroutine, and maps any raised exception against the WHOLE source. The program runs exactly as the model wrote it — Python's own halt-on-exception decides what ran. Assigning a bound tool name is allowed: __vis_run_async__ keeps that binding BLOCK-LOCAL, so the shadow works here and the callable survives for the next block.

seed-cli-runtime! clj

(seed-cli-runtime! python-context {:keys [argv env]})

Seed a standalone vis-agent python CLI context with script argv (bound to sys.argv) and, when non-empty, an env map merged into os.environ. This is what gives the CLI real-python semantics: unlike the deny-by- default AGENT sandbox (env scrubbed for isolation — the human never sees their shell here), the human-run CLI forwards trailing script args and, by default, the caller's environment.

Same shape as any host→guest seeding here: values cross via putMember, then a guest eval assigns them (a JSON hop keeps ProxyHashMaps off the boundary and reuses the auto-imported json). Best-effort: a bad value never aborts startup.

set-python-binding! clj

(set-python-binding! python-context sym val)

Bind sym -> val in the Python sandbox globals. Clojure fns are wired as callables; everything else is marshalled.

ASYNC-BY-DEFAULT: a tool fn bound here is also DEFERRED (wrapped by __vis_deferred__, same as build-agent-context's defer step) so await tool(...) / gather(tool(...)) work. This matters because extension and foundation tools are (re)installed via this fn AFTER the context's own defer pass — without deferring here they'd stay raw/synchronous and the await the prompt teaches would fail. The compaction verbs (fold_session/session_drop/__vis_par__) are bound via create-python-context, not here, so they stay direct. No-op when the async preamble isn't installed (the printer/parser helper contexts never bind tools).

set-python-binding-doc! clj

(set-python-binding-doc! python-context sym doc)

Record doc text for sym in the sandbox __vis_docs__ dict that in-sandbox doc(name) / apropos(pat) read — and that a deferred tool carries as its __doc__, so help(tool) answers the same contract.

set-python-binding-signature! clj

(set-python-binding-signature! python-context sym signature)

Record signature — the Python parameter list from extension/symbol-signature — for sym in the sandbox __vis_sigs__ dict. A deferred tool hangs it off __wrapped__, so inspect.signature(tool) reports the declared parameters instead of the async trampoline's (*a, **k).

shared-engine clj

sym->py-name clj

(sym->py-name sym)

Clojure tool/binding symbol -> a Python-LEGAL global name. Purely mechanical: / and - fold to _ (alias fold + kebab->snake); a trailing ! (mutation marker) is dropped; a trailing ? (predicate) becomes an is_ prefix. So git/status -> git_status, git/commit! -> git_commit, search/web -> search_web, file-exists -> file_exists. FULL SNAKE: this is how the agent reaches the tools — git_status() calls git/status.

A tiny compatibility alias layer may additionally expose selected historical short names (currently find_files/find for grep), but the snake name remains canonical.

system-var-sym? clj

(system-var-sym? sym)

SYSTEM_VAR_NAMES clj

Engine-owned symbols hidden from user live-var listings.

validate-no-banned-defs! clj

(validate-no-banned-defs! python-context code)

Throws :vis/banned-def-head when code references a banned construct (BANNED_DEF_HEADS). Parse failures are silent — the eval that follows surfaces a clean syntax error with line/column.

validate-non-empty-block! clj

(validate-non-empty-block! python-context code)

Throws :vis/empty-block when code parses to zero top-level statements (comment-only blocks). Iterations that produce no evidence are rejected at the model boundary.