Process boundaries for model Python and trusted Python extensions.
Each session's sandbox owns a confined runtime worker. Its trusted extension namespaces run in a separate worker, preserving their host APIs and native library support without sharing interpreter memory or host-call authority with model code. Registration outside a session uses the shared trusted worker.
Confinement, imports, native libraries and interpreter state are process-wide. A per-session extension worker keeps that state separate from other sessions as well as from the sandbox. Only the host chooses a worker's role.
The wire is ONE line of JSON per message over a unix socket, both ways. The
parent asks (install-runtime, install-tool, exec, run, run-block,
eval, confine, network, stdin, interrupt, stack-diagnostics,
dump-stacks, close); the child asks
back with host, because the registry that knows what a name may call, the
persistence handle and the caller's dynamic binding frame all live in the
parent (python-host/dispatch). stdout is NOT the wire: Python that prints,
or a native library writing to fd 1, would corrupt it, so a child's own stdio
goes to a log file instead.
A message carrying op is a request, one without is its reply, so each side
numbers its own requests and no id can collide. Work has no timeout: a block
or extension tool may legitimately run for minutes. CONTROL is different:
an interrupt that cannot reach the child is bounded, because cancellation
must be able to retire that process instead of parking its caller forever.
A child that DIES is what the pump reports — every call waiting on it fails
at once with the child's log to read.
An interrupt reaches BOTH ends of a host call. The child's async exception
lands only when the guest next runs Python, and a guest parked in host
(a shell handle's wait, a long tool) runs none until the parent answers —
so interrupt! also fails every host call the worker has in flight, and the
tool thread serving it is interrupted. Measured before that: the unwind
watch expired, the worker was killed, and the next block silently started a
FRESH interpreter that had the runtime but none of the session's tools. A
retired key now refuses to restart until the session is rebuilt.
Process boundaries for model Python and trusted Python extensions. Each session's sandbox owns a confined runtime worker. Its trusted extension namespaces run in a separate worker, preserving their host APIs and native library support without sharing interpreter memory or host-call authority with model code. Registration outside a session uses the shared trusted worker. Confinement, imports, native libraries and interpreter state are process-wide. A per-session extension worker keeps that state separate from other sessions as well as from the sandbox. Only the host chooses a worker's role. The wire is ONE line of JSON per message over a unix socket, both ways. The parent asks (`install-runtime`, `install-tool`, `exec`, `run`, `run-block`, `eval`, `confine`, `network`, `stdin`, `interrupt`, `stack-diagnostics`, `dump-stacks`, `close`); the child asks back with `host`, because the registry that knows what a name may call, the persistence handle and the caller's dynamic binding frame all live in the parent (`python-host/dispatch`). stdout is NOT the wire: Python that prints, or a native library writing to fd 1, would corrupt it, so a child's own stdio goes to a log file instead. A message carrying `op` is a request, one without is its reply, so each side numbers its own requests and no id can collide. Work has no timeout: a block or extension tool may legitimately run for minutes. CONTROL is different: an interrupt that cannot reach the child is bounded, because cancellation must be able to retire that process instead of parking its caller forever. A child that DIES is what the pump reports — every call waiting on it fails at once with the child's log to read. An interrupt reaches BOTH ends of a host call. The child's async exception lands only when the guest next runs Python, and a guest parked in `host` (a shell handle's `wait`, a long tool) runs none until the parent answers — so `interrupt!` also fails every host call the worker has in flight, and the tool thread serving it is interrupted. Measured before that: the unwind watch expired, the worker was killed, and the next block silently started a FRESH interpreter that had the runtime but none of the session's tools. A retired key now refuses to restart until the session is rebuilt.
True while a stop only puts a session to sleep, such as an idle sandbox. Such a
stop ends the worker process but keeps the processes that it started. They stay
in [[kept-children]] until stop!. Every other stop ends the whole tree.
True while a stop only puts a session to sleep, such as an idle sandbox. Such a stop ends the worker process but keeps the processes that it started. They stay in [[kept-children]] until [[stop!]]. Every other stop ends the whole tree.
(configure! k policy-fn)Register the live session policy used whenever worker k starts or restarts.
Must happen before the first interpreter request; a missing policy fails closed.
A session being (re)built under a key that was retired is that key's new life.
Register the live session policy used whenever worker `k` starts or restarts. Must happen before the first interpreter request; a missing policy fails closed. A session being (re)built under a key that was retired is that key's new life.
(confine! k session read write refusal)Confine k's interpreter to read/write, or lift it with two empty lists.
The policy is that PROCESS's, which is why one worker per session is the whole
point: what used to be every session in the gateway is now this session.
Confine `k`'s interpreter to `read`/`write`, or lift it with two empty lists. The policy is that PROCESS's, which is why one worker per session is the whole point: what used to be every session in the gateway is now this session.
(extension-worker-key session)(extension-worker-key session packages)Host-owned worker identity for a session's trusted extensions, not its sandbox. The key is never accepted from Python or reconstructed from a wire payload. A declared project's package directory is immutable startup data, not a later sys.path override; nil keeps the shared-package default.
Host-owned worker identity for a session's trusted extensions, not its sandbox. The key is never accepted from Python or reconstructed from a wire payload. A declared project's package directory is immutable startup data, not a later sys.path override; nil keeps the shared-package default.
(forget-policy! k)Forget k after ordinary session disposal; stopping a wedged worker alone keeps
the policy so no late call can restart outside the boundary.
Forget `k` after ordinary session disposal; stopping a wedged worker alone keeps the policy so no late call can restart outside the boundary.
(guest-source-dir)Stage Vis-owned Python guest modules in the current home, restoring missing files.
The first call in a process compares every module with the shipped source. Each worker start and snapshot calls this again, so a later call checks only that the modules exist and publishes again when one is missing.
Stage Vis-owned Python guest modules in the current home, restoring missing files. The first call in a process compares every module with the shipped source. Each worker start and snapshot calls this again, so a later call checks only that the modules exist and publishes again when one is missing.
(interrupt! k session)Interrupt the current worker for session without starting a replacement.
Fail its host calls too, so a guest parked in one can unwind. The entire
exchange is bounded, including writes; on timeout the caller must retire
the unresponsive worker. Answer whether the child acknowledged the interrupt.
Interrupt the current worker for `session` without starting a replacement. Fail its host calls too, so a guest parked in one can unwind. The entire exchange is bounded, including writes; on timeout the caller must retire the unresponsive worker. Answer whether the child acknowledged the interrupt.
(keep-spares! warmup packages)Keep one warm trusted worker ready for each package directory in packages (nil
names the shared packages). warmup is Python that runs in a spare before its first
context; nil stops every spare. A gateway calls this at start and
retain-spares! after each extension load; tests and one-shot commands keep none.
Keep one warm trusted worker ready for each package directory in `packages` (nil names the shared packages). `warmup` is Python that runs in a spare before its first context; nil stops every spare. A gateway calls this at start and [[retain-spares!]] after each extension load; tests and one-shot commands keep none.
(pending-replies k)Snapshot the replies still owed by k, without starting a worker. These settle
only when the guest replies or exits, even if its host caller was cancelled.
Snapshot the replies still owed by `k`, without starting a worker. These settle only when the guest replies or exits, even if its host caller was cancelled.
(prestart-worker! k warmup)Start the worker for k before anything asks for it, and run warmup Python in a
throwaway namespace of it. Its first context then skips those imports and compiles.
The start and the warm-up hold the lock of k, and the worker is published only
after its warm-up. A caller that asks for k meanwhile waits, because a warm-up
that runs next to an extension load breaks that load. An existing worker, also a
dead or retired one, is left as it is.
Start the worker for `k` before anything asks for it, and run `warmup` Python in a throwaway namespace of it. Its first context then skips those imports and compiles. The start and the warm-up hold the lock of `k`, and the worker is published only after its warm-up. A caller that asks for `k` meanwhile waits, because a warm-up that runs next to an extension load breaks that load. An existing worker, also a dead or retired one, is left as it is.
(retain-spares! packages)Keep spares only for packages, the package directories of the loaded extensions
(nil names the shared packages), while spares are on. Stop each other spare: an
updated extension must not leave a warm worker for its old directory.
Keep spares only for `packages`, the package directories of the loaded extensions (nil names the shared packages), while spares are on. Stop each other spare: an updated extension must not leave a warm worker for its old directory.
(retire-worker! k reason)Record bounded local RPC/JVM evidence and request Python stacks, then stop k
and refuse to restart until rebuilt or disposed. Ordinary disposal does not dump stacks.
Record bounded local RPC/JVM evidence and request Python stacks, then stop `k` and refuse to restart until rebuilt or disposed. Ordinary disposal does not dump stacks.
(retired? k)True when k was retired and not yet rebuilt or disposed.
True when `k` was retired and not yet rebuilt or disposed.
The worker for Python that belongs to no single session: extension files loading at startup, whose REGISTRATION is the gateway's and not a session's.
The worker for Python that belongs to no single session: extension files loading at startup, whose REGISTRATION is the gateway's and not a session's.
(start-worker! k)Start the worker for k before its first request, so that independent
workers can start in parallel. Never replaces a dead or retired worker.
Start the worker for `k` before its first request, so that independent workers can start in parallel. Never replaces a dead or retired worker.
(stop!)Stop every worker and every spare, in parallel, and end every process that a worker started, also those that sleeping workers kept. Idempotent. A worker that outlives this process exits on EOF alone and leaves its children running, so a JVM shutdown hook calls this for every host.
Stop every worker and every spare, in parallel, and end every process that a worker started, also those that sleeping workers kept. Idempotent. A worker that outlives this process exits on EOF alone and leaves its children running, so a JVM shutdown hook calls this for every host.
(stop-worker! k)Stop the worker for k, if there is one. Idempotent. Closing the socket
releases every pending parent call; a child that does not leave promptly is
force-killed so retired sessions cannot accumulate processes.
Stop the worker for `k`, if there is one. Idempotent. Closing the socket releases every pending parent call; a child that does not leave promptly is force-killed so retired sessions cannot accumulate processes.
(worker-live? k)True when k already owns a live worker; never starts one.
True when `k` already owns a live worker; never starts one.
(worker-lost? k)True when k was retired, or its started worker exited; never starts one.
A key that never started a worker is not lost.
True when `k` was retired, or its started worker exited; never starts one. A key that never started a worker is not lost.
(worker-pid k)PID of the live worker for k, or nil.
PID of the live worker for `k`, or nil.
(worker-pids)PIDs of every live session or shared Python worker this process owns.
PIDs of every live session or shared Python worker this process owns.
(worker-ready? k session)Can the existing worker of k enter session before a new turn starts? See
workers-ready?.
Can the existing worker of `k` enter `session` before a new turn starts? See [[workers-ready?]].
(workers-ready? probes)Can each existing worker in probes, pairs of a worker key and a namespace,
enter its namespace before a new turn starts?
Process liveness alone does not prove the interpreter can acquire its GIL. The requests run at the same time under one deadline, so two workers wait no longer than one. Bound the entire exchange, including a blocked socket write. Only call between turns: a running block may legitimately occupy the interpreter for minutes. Never starts or replaces a worker; the engine owns recovery after a false result. Caller cancellation propagates instead of declaring a healthy worker broken.
Can each existing worker in `probes`, pairs of a worker key and a namespace, enter its namespace before a new turn starts? Process liveness alone does not prove the interpreter can acquire its GIL. The requests run at the same time under one deadline, so two workers wait no longer than one. Bound the entire exchange, including a blocked socket write. Only call between turns: a running block may legitimately occupy the interpreter for minutes. Never starts or replaces a worker; the engine owns recovery after a false result. Caller cancellation propagates instead of declaring a healthy worker broken.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |