Gateway-lifecycle SANDBOX CAPABILITY: ONE shared loopback egress proxy and ONE ephemeral MITM CA for the WHOLE daemon, keyed PER SESSION.
Why shared, not per-session (turn 32): the gateway is multi-tenant — many clients/sessions hit one daemon. A per-session proxy+CA thrashes listeners and mints a fresh CA per session; worse, every child would carry a different trust root. Instead there is ONE listener and ONE CA (the "same certs" property — every child trusts the same root), plus a REGISTRY mapping a per-session TOKEN → that session's live policy fn.
How a connection is attributed to a session: the jailed child's proxy env carries
its unguessable token in the proxy URL userinfo (http://<token>@127.0.0.1:<port>);
curl/git/requests/… send it back as Proxy-Authorization: Basic base64(<token>:);
the proxy hands the token to resolve-policy, which looks up the registry.
FAIL-CLOSED: a request whose token is missing or not registered to a LIVE session
is DENIED (a :deny-all? sentinel policy) — the shared door never serves a policy
it cannot attribute. The token is a random UUID, so one session cannot reach
another's (broader) policy by guessing.
Lazy: the proxy listener and the CA keygen happen only on first ensure-proxy! /
ensure-ca! — a gateway that never jails a shell child opens neither.
Gateway-lifecycle SANDBOX CAPABILITY: ONE shared loopback egress proxy and ONE ephemeral MITM CA for the WHOLE daemon, keyed PER SESSION. Why shared, not per-session (turn 32): the gateway is multi-tenant — many clients/sessions hit one daemon. A per-session proxy+CA thrashes listeners and mints a fresh CA per session; worse, every child would carry a different trust root. Instead there is ONE listener and ONE CA (the "same certs" property — every child trusts the same root), plus a REGISTRY mapping a per-session TOKEN → that session's live policy fn. How a connection is attributed to a session: the jailed child's proxy env carries its unguessable token in the proxy URL userinfo (`http://<token>@127.0.0.1:<port>`); curl/git/requests/… send it back as `Proxy-Authorization: Basic base64(<token>:)`; the proxy hands the token to `resolve-policy`, which looks up the registry. FAIL-CLOSED: a request whose token is missing or not registered to a LIVE session is DENIED (a `:deny-all?` sentinel policy) — the shared door never serves a policy it cannot attribute. The token is a random UUID, so one session cannot reach another's (broader) policy by guessing. Lazy: the proxy listener and the CA keygen happen only on first `ensure-proxy!` / `ensure-ca!` — a gateway that never jails a shell child opens neither.
Cross-process gateway event bus.
The gateway's live event log + SSE fan-out (gateway.state) is a
PROCESS-LOCAL in-memory registry: append-event! only reaches
subscribers inside the SAME JVM. That is why a turn streaming in the
TUI process is invisible to another process watching the SAME
conversation — each process has its own registry, and the only thing
they share is the persisted DB (which lands whole turns, not the live
token stream). So two watchers never stream together.
This bus closes that gap with the simplest transport that needs no
schema change and no always-on daemon: a shared append-only journal
under ~/.vis/gateway/events/<sid>.ndjson. Every LOCALLY-produced
gateway event is publish!ed there (one JSON line, tagged with this
process's producer id). A background tailer in each process follows
those files and re-delivers FOREIGN events (producer != self) into the
local registry via a delivery fn wired by gateway.state — so every
process's subscribers see the same stream, live.
Ordering/seq: exactly ONE turn runs per session at a time, so at any
moment a single producer owns the stream and its monotonic :seq is
authoritative for every watcher. The producer truncates the journal at
each turn.started, bounding a file to one turn's worth of deltas;
consumers detect the truncation (offset past EOF) and rewind.
Degrades safely: any IO failure is swallowed and the process falls back to today's in-process-only behavior.
Cross-process gateway event bus. The gateway's live event log + SSE fan-out (`gateway.state`) is a PROCESS-LOCAL in-memory registry: `append-event!` only reaches subscribers inside the SAME JVM. That is why a turn streaming in the TUI process is invisible to another process watching the SAME conversation — each process has its own registry, and the only thing they share is the persisted DB (which lands whole turns, not the live token stream). So two watchers never stream together. This bus closes that gap with the simplest transport that needs no schema change and no always-on daemon: a shared append-only journal under `~/.vis/gateway/events/<sid>.ndjson`. Every LOCALLY-produced gateway event is `publish!`ed there (one JSON line, tagged with this process's `producer` id). A background tailer in each process follows those files and re-delivers FOREIGN events (producer != self) into the local registry via a delivery fn wired by `gateway.state` — so every process's subscribers see the same stream, live. Ordering/seq: exactly ONE turn runs per session at a time, so at any moment a single producer owns the stream and its monotonic `:seq` is authoritative for every watcher. The producer truncates the journal at each `turn.started`, bounding a file to one turn's worth of deltas; consumers detect the truncation (offset past EOF) and rewind. Degrades safely: any IO failure is swallowed and the process falls back to today's in-process-only behavior.
HTTP/SSE client for the long-lived gateway daemon.
Interactive channels call this facade instead of gateway.state directly. It
discover-or-starts the one daemon for the current DB, then speaks the same
HTTP/SSE API every other client uses. This is the thin-client half of the
gateway-daemon plan: token refresh, turn execution, and live streaming happen
in ONE process.
HTTP/SSE client for the long-lived gateway daemon. Interactive channels call this facade instead of `gateway.state` directly. It discover-or-starts the one daemon for the current DB, then speaks the same HTTP/SSE API every other client uses. This is the thin-client half of the gateway-daemon plan: token refresh, turn execution, and live streaming happen in ONE process.
Gateway daemon discovery + registry (build order step 1).
One long-lived gateway per DB owns execution; every TUI/web/whatever is a thin client of it. This namespace answers the boot-time question: "is a gateway already running for my DB — attach; else spawn one, DETACHED (nobody's child), then hand back where to connect."
Registry: one EDN file per DB at ~/.vis/gateway/registry/<sha256(db)>.edn
holding {:pid :port :host :secret :db :created-at}. Freshness = the recorded
:pid is still alive AND a caller-supplied probe confirms the port+secret
are really OUR daemon (guards OS pid reuse — see D4/Q3 in TODO-gateway-daemon).
Design decisions (locked, see TODO-gateway-daemon.md):
--db share one daemon).register-self! from serve-main!), so a
spawner never needs the child pid — it just polls for a fresh registry.:memory never registers/spawns (headless one-shot stays in-process).Effects (spawn, probe, pid-liveness) are injectable so the orchestration
in discover-or-start! is unit-testable without a real process.
Gateway daemon discovery + registry (build order step 1).
One long-lived gateway per DB owns execution; every TUI/web/whatever is a
thin client of it. This namespace answers the boot-time question: "is a
gateway already running for my DB — attach; else spawn one, DETACHED (nobody's
child), then hand back where to connect."
Registry: one EDN file per DB at `~/.vis/gateway/registry/<sha256(db)>.edn`
holding `{:pid :port :host :secret :db :created-at}`. Freshness = the recorded
`:pid` is still alive AND a caller-supplied `probe` confirms the port+secret
are really OUR daemon (guards OS pid reuse — see D4/Q3 in TODO-gateway-daemon).
Design decisions (locked, see TODO-gateway-daemon.md):
- Q2 registry key = the DB path (two dirs sharing `--db` share one daemon).
- Q3 race = port-bind winner is the daemon; the loser attaches. The daemon
SELF-REGISTERS on startup (via [[register-self!]] from `serve-main!`), so a
spawner never needs the child pid — it just polls for a fresh registry.
- Q5 `:memory` never registers/spawns (headless one-shot stays in-process).
Effects (`spawn`, `probe`, pid-liveness) are injectable so the orchestration
in [[discover-or-start!]] is unit-testable without a real process.Android push (Firebase Cloud Messaging HTTP v1).
Apple's APNs lives in gateway.push; this is its Android twin and the two
are dispatched on the registered device's :platform. Credentials are a
Google service-account JSON — from the macOS keychain (service vis-fcm,
account service_account), from the environment, or from a file under
~/.vis/fcm/. Key material is never returned, logged or sent over the wire.
Android push (Firebase Cloud Messaging HTTP v1). Apple's APNs lives in `gateway.push`; this is its Android twin and the two are dispatched on the registered device's `:platform`. Credentials are a Google service-account JSON — from the macOS keychain (service `vis-fcm`, account `service_account`), from the environment, or from a file under `~/.vis/fcm/`. Key material is never returned, logged or sent over the wire.
Gateway half of the typed human-input pause.
com.blockether.vis.internal.human-input BLOCKS the extension thread and
publishes :human-input/request / :human-input/close on every channel the
request names — [:tui :app] by default. The TUI draws its dialog from the
:tui channel; this namespace serves the :app channel and is the reason a
companion-app operator ever learns a run is waiting on them:
human_input.request SESSION event, so it rides
SSE live, sits in the replay ring for a client that connects later, and
reaches the push tap (which alerts the phone);human_input.close, so every client drops the form
the moment ANY surface answers — TUI, app or a timeout;pending / submit! / cancel! back the REST routes so the app
can answer a request it finds already open.A request raised outside a gateway session carries no :session-id. It is
still published (the TUI shows it) and silently skipped here — a session
event with no session has nowhere to go.
Gateway half of the typed human-input pause.
`com.blockether.vis.internal.human-input` BLOCKS the extension thread and
publishes `:human-input/request` / `:human-input/close` on every channel the
request names — `[:tui :app]` by default. The TUI draws its dialog from the
`:tui` channel; this namespace serves the `:app` channel and is the reason a
companion-app operator ever learns a run is waiting on them:
- the request becomes a `human_input.request` SESSION event, so it rides
SSE live, sits in the replay ring for a client that connects later, and
reaches the push tap (which alerts the phone);
- the close becomes `human_input.close`, so every client drops the form
the moment ANY surface answers — TUI, app or a timeout;
- [[pending]] / [[submit!]] / [[cancel!]] back the REST routes so the app
can answer a request it finds already open.
A request raised outside a gateway session carries no `:session-id`. It is
still published (the TUI shows it) and silently skipped here — a session
event with no session has nowhere to go.Gateway pairing helpers for remote clients.
The QR payload is deliberately tiny and URL-shaped so native apps can scan it without an HTTP round trip:
vis://gateway?url=http%3A%2F%2F100.64.0.10%3A7890&token=...
Tailscale fits naturally: if a 100.64.0.0/10 interface is present we prefer it over LAN addresses, otherwise we fall back to site-local IPv4 addresses.
Gateway pairing helpers for remote clients. The QR payload is deliberately tiny and URL-shaped so native apps can scan it without an HTTP round trip: vis://gateway?url=http%3A%2F%2F100.64.0.10%3A7890&token=... Tailscale fits naturally: if a 100.64.0.0/10 interface is present we prefer it over LAN addresses, otherwise we fall back to site-local IPv4 addresses.
THE single source of truth for gateway <-> client version compatibility.
Vis ships three INDEPENDENTLY updatable halves that share one HTTP/SSE
wire: the gateway daemon, the TUI/CLI client, and the Vis Companion app.
Any of them can lag. A daemon started by yesterday's binary keeps running
across brew upgrade; a phone holds a cached web build for weeks; a
Tailscale peer runs last month's release. Without an explicit contract a
breaking wire change surfaces as a mystery 404, a missing field, or an
event type nobody renders.
So every peer publishes two numbers next to its human release version:
protocol the wire protocol IT speaks
min-* the OLDEST counterpart it still speaks to
Compatibility is then a pure comparison (verdict) — never feature
sniffing, never guessing from a release string. Both halves advertise, both
halves judge, so whichever side is newer can explain the mismatch even when
the other side is too old to know the concept exists.
Bump protocol-version on any BREAKING wire change (a removed field, a
renamed event type, a changed status shape). Raise min-client-protocol
or min-gateway-protocol only when the old shape genuinely stops
working — additive changes keep the number and are negotiated through
/v1/capabilities features instead.
Wire shape (snake_case strings, per the gateway wire contract):
{"protocol": 1, "min_client": 1, "min_gateway": 1, "version": "0.1.5"}
THE single source of truth for gateway <-> client version compatibility.
Vis ships three INDEPENDENTLY updatable halves that share one HTTP/SSE
wire: the gateway daemon, the TUI/CLI client, and the Vis Companion app.
Any of them can lag. A daemon started by yesterday's binary keeps running
across `brew upgrade`; a phone holds a cached web build for weeks; a
Tailscale peer runs last month's release. Without an explicit contract a
breaking wire change surfaces as a mystery 404, a missing field, or an
event type nobody renders.
So every peer publishes two numbers next to its human release version:
`protocol` the wire protocol IT speaks
`min-*` the OLDEST counterpart it still speaks to
Compatibility is then a pure comparison ([[verdict]]) — never feature
sniffing, never guessing from a release string. Both halves advertise, both
halves judge, so whichever side is newer can explain the mismatch even when
the other side is too old to know the concept exists.
Bump [[protocol-version]] on any BREAKING wire change (a removed field, a
renamed event type, a changed status shape). Raise [[min-client-protocol]]
or [[min-gateway-protocol]] only when the old shape genuinely stops
working — additive changes keep the number and are negotiated through
`/v1/capabilities` features instead.
Wire shape (snake_case strings, per the gateway wire contract):
{"protocol": 1, "min_client": 1, "min_gateway": 1, "version": "0.1.5"}Native push notifications (Apple Push Notification service).
ONE job: when a turn finishes on this gateway, wake the phone that asked to be woken. Everything here is server-side; the app only ever hands us a device token.
Three moving parts:
Credentials. A token-based APNs auth key (.p8, ES256) plus its key
id, the Apple team id and the app's bundle id (the APNs topic).
Resolved from VIS_APNS_KEY_PATH / VIS_APNS_KEY_ID / VIS_APNS_TEAM_ID
/ VIS_APNS_TOPIC, else auto-discovered from ~/.vis/apns/AuthKey_<kid>.p8
(the key id is read off the filename) with the team/topic still from env
or ~/.vis/apns/apns.edn. No credentials = push silently OFF; the gateway
keeps working exactly as before.
A device registry at ~/.vis/devices.edn — device token -> platform,
APNs environment, client label/version, timestamps. Registration is
idempotent on the token. Tokens are SECRETS: nothing here logs more than
a masked prefix.
A sender — a signed ES256 JWT (cached, refreshed well inside Apple's
one-hour window) over HTTP/2 to api.push.apple.com. A device that
registered with the wrong environment is retried once against the other
host, and an APNs BadDeviceToken/Unregistered verdict evicts the
device so a stale token cannot accumulate.
The wire surface lives in gateway.server (/v1/devices); this namespace
knows nothing about Ring.
Native push notifications (Apple Push Notification service). ONE job: when a turn finishes on this gateway, wake the phone that asked to be woken. Everything here is server-side; the app only ever hands us a device token. Three moving parts: 1. **Credentials.** A token-based APNs auth key (`.p8`, ES256) plus its key id, the Apple team id and the app's bundle id (the APNs *topic*). Resolved from `VIS_APNS_KEY_PATH` / `VIS_APNS_KEY_ID` / `VIS_APNS_TEAM_ID` / `VIS_APNS_TOPIC`, else auto-discovered from `~/.vis/apns/AuthKey_<kid>.p8` (the key id is read off the filename) with the team/topic still from env or `~/.vis/apns/apns.edn`. No credentials = push silently OFF; the gateway keeps working exactly as before. 2. **A device registry** at `~/.vis/devices.edn` — device token -> platform, APNs environment, client label/version, timestamps. Registration is idempotent on the token. Tokens are SECRETS: nothing here logs more than a masked prefix. 3. **A sender** — a signed ES256 JWT (cached, refreshed well inside Apple's one-hour window) over HTTP/2 to `api.push.apple.com`. A device that registered with the wrong environment is retried once against the other host, and an APNs `BadDeviceToken`/`Unregistered` verdict evicts the device so a stale token cannot accumulate. The wire surface lives in `gateway.server` (`/v1/devices`); this namespace knows nothing about Ring.
Relayed push — how a gateway wakes a phone WITHOUT holding Apple's or Google's signing key.
APNs binds a topic to the Apple team that owns it: a key minted by anyone
else, aimed at someone else's bundle id, is refused forever (403 InvalidProviderToken / TopicDisallowed). So a self-hosted gateway can
never push to a companion built and signed by somebody else — unless the
signing key stays on infrastructure the app's publisher runs, and the
gateway is given a capability instead of a credential.
That capability is a GRANT. The device asks the relay for one and hands
it to this gateway during "notify this device"; the gateway POSTs
{grant, title, body} and the relay signs and forwards. Consequences worth
the indirection:
.p8, no service-account JSON, nothing revocable
only by breaking push for everyone else;The relay itself lives in apps/vis-companion-relay (a Cloudflare Worker),
and every gateway names the publisher's one by DEFAULT (DEFAULT-URL): a
machine nobody configured is already able to push. A device may name another
and is believed — which relay can sign for a build is a property of the
BUILD, so the app mints its grant at the relay serving the app it is and
posts {grant, relay_url} to /v1/devices. VIS_PUSH_RELAY_URL, or
~/.vis/relay.edn {:url "https://push.example.com"}, replaces the default
on one machine; the direct gateway.push / gateway.fcm credentials still
work exactly as before.
Relayed push — how a gateway wakes a phone WITHOUT holding Apple's or
Google's signing key.
APNs binds a topic to the Apple team that owns it: a key minted by anyone
else, aimed at someone else's bundle id, is refused forever (`403
InvalidProviderToken` / `TopicDisallowed`). So a self-hosted gateway can
never push to a companion built and signed by somebody else — unless the
signing key stays on infrastructure the app's publisher runs, and the
gateway is given a capability instead of a credential.
That capability is a GRANT. The *device* asks the relay for one and hands
it to this gateway during "notify this device"; the gateway POSTs
`{grant, title, body}` and the relay signs and forwards. Consequences worth
the indirection:
* this gateway holds no `.p8`, no service-account JSON, nothing revocable
only by breaking push for everyone else;
* this gateway never learns the raw APNs/FCM device token, so a gateway you
do not trust cannot fingerprint the device it notifies;
* a grant expires by itself. Its expiry travels inside it, sealed, so the
relay keeps no list of anybody and an abandoned gateway simply goes mute.
The relay itself lives in `apps/vis-companion-relay` (a Cloudflare Worker),
and every gateway names the publisher's one by DEFAULT (`DEFAULT-URL`): a
machine nobody configured is already able to push. A device may name another
and is believed — which relay can sign for a build is a property of the
BUILD, so the app mints its grant at the relay serving the app it is and
posts `{grant, relay_url}` to `/v1/devices`. `VIS_PUSH_RELAY_URL`, or
`~/.vis/relay.edn` `{:url "https://push.example.com"}`, replaces the default
on one machine; the direct `gateway.push` / `gateway.fcm` credentials still
work exactly as before.Gateway HTTP/SSE server.
Clojure-native stack: reitit-ring routes -> Ring middleware -> the
Ring Jetty adapter on JDK virtual threads (:virtual-threads? true).
SSE is a Ring StreamableResponseBody whose virtual thread is the
connection's SINGLE socket writer: replay rides first, then it drains a
bounded per-connection event queue that state/fan-out! enqueues onto,
emitting a heartbeat comment on idle to keep the pipe warm and detect
dead clients.
This is internal plumbing, not a channel: it registers no channel
descriptor and owns no renderer - it ships canonical IR and the
client renders (§4.1). Any host process (the vis-agent gateway start daemon, a
TUI run, an embedded caller) can start it alongside whatever else it
is doing via start!.
Gateway HTTP/SSE server. Clojure-native stack: reitit-ring routes -> Ring middleware -> the Ring Jetty adapter on JDK virtual threads (`:virtual-threads? true`). SSE is a Ring `StreamableResponseBody` whose virtual thread is the connection's SINGLE socket writer: replay rides first, then it drains a bounded per-connection event queue that `state/fan-out!` enqueues onto, emitting a heartbeat comment on idle to keep the pipe warm and detect dead clients. This is internal plumbing, not a channel: it registers no channel descriptor and owns no renderer - it ships canonical IR and the client renders (§4.1). Any host process (the `vis-agent gateway start` daemon, a TUI run, an embedded caller) can start it alongside whatever else it is doing via `start!`.
Gateway session manager.
One process-global registry over the live session fleet: per-session
ordered event log (monotonic :seq, ring-buffered), SSE subscriber
fan-out, async turn submission with idempotency keys, cancellation,
and turn/cost metrics.
The engine is reached ONLY through the same internal surfaces the
TUI channel uses: loop/create!-send!-close! for the
lifecycle, :hooks {:on-chunk ...} phased chunks for the live
stream, ctx-loop/session-snapshot for the context. No engine state
lives here - this namespace owns wire bookkeeping (events, turn
records, subscribers), nothing else.
Gateway session manager.
One process-global registry over the live session fleet: per-session
ordered event log (monotonic `:seq`, ring-buffered), SSE subscriber
fan-out, async turn submission with idempotency keys, cancellation,
and turn/cost metrics.
The engine is reached ONLY through the same internal surfaces the
TUI channel uses: `loop/create!`-`send!`-`close!` for the
lifecycle, `:hooks {:on-chunk ...}` phased chunks for the live
stream, `ctx-loop/session-snapshot` for the context. No engine state
lives here - this namespace owns wire bookkeeping (events, turn
records, subscribers), nothing else.Wire encoding for the HTTP gateway.
One dumb, deterministic boundary: engine EDN -> JSON. Keyword/symbol
keys become snake_case strings (namespace dropped), keyword VALUES keep
their full ns/name (a badge role like :tool-color/search must survive
the hop — dropping the namespace made the remote TUI see :search while
the hop), non-JSON leaves fall back to str. The walker makes zero semantic
or rendering decisions. Canonical message content is already a string-keyed
vector of typed block maps before it reaches this boundary.
canonical is the SAME shape on the Clojure side: by definition what
parse-json ∘ json-str yields — snake_case STRING keys — serve it from
a facade and in-process readers see exactly what a remote client sees.
Wire encoding for the HTTP gateway. One dumb, deterministic boundary: engine EDN -> JSON. Keyword/symbol keys become snake_case strings (namespace dropped), keyword VALUES keep their full `ns/name` (a badge role like `:tool-color/search` must survive the hop — dropping the namespace made the remote TUI see `:search` while the hop), non-JSON leaves fall back to `str`. The walker makes zero semantic or rendering decisions. Canonical message content is already a string-keyed vector of typed block maps before it reaches this boundary. `canonical` is the SAME shape on the Clojure side: by definition what `parse-json` ∘ `json-str` yields — snake_case STRING keys — serve it from a facade and in-process readers see exactly what a remote client sees.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |