Cross-process gateway event bus.
The gateway's live event log + SSE fan-out (gateway.state) is a
PROCESS-LOCAL in-memory registry: append-event! only reaches
subscribers inside the SAME JVM. That is why a turn streaming in the
TUI process is invisible to another process watching the SAME
conversation — each process has its own registry, and the only thing
they share is the persisted DB (which lands whole turns, not the live
token stream). So two watchers never stream together.
This bus closes that gap with the simplest transport that needs no
schema change and no always-on daemon: a shared append-only journal
under ~/.vis/gateway/events/<sid>.ndjson. Every LOCALLY-produced
gateway event is publish!ed there (one JSON line, tagged with this
process's producer id). A background tailer in each process follows
those files and re-delivers FOREIGN events (producer != self) into the
local registry via a delivery fn wired by gateway.state — so every
process's subscribers see the same stream, live.
Ordering/seq: exactly ONE turn runs per session at a time, so at any
moment a single producer owns the stream and its monotonic :seq is
authoritative for every watcher. The producer truncates the journal at
each turn.started, bounding a file to one turn's worth of deltas;
consumers detect the truncation (offset past EOF) and rewind.
Degrades safely: any IO failure is swallowed and the process falls back to today's in-process-only behavior.
Cross-process gateway event bus. The gateway's live event log + SSE fan-out (`gateway.state`) is a PROCESS-LOCAL in-memory registry: `append-event!` only reaches subscribers inside the SAME JVM. That is why a turn streaming in the TUI process is invisible to another process watching the SAME conversation — each process has its own registry, and the only thing they share is the persisted DB (which lands whole turns, not the live token stream). So two watchers never stream together. This bus closes that gap with the simplest transport that needs no schema change and no always-on daemon: a shared append-only journal under `~/.vis/gateway/events/<sid>.ndjson`. Every LOCALLY-produced gateway event is `publish!`ed there (one JSON line, tagged with this process's `producer` id). A background tailer in each process follows those files and re-delivers FOREIGN events (producer != self) into the local registry via a delivery fn wired by `gateway.state` — so every process's subscribers see the same stream, live. Ordering/seq: exactly ONE turn runs per session at a time, so at any moment a single producer owns the stream and its monotonic `:seq` is authoritative for every watcher. The producer truncates the journal at each `turn.started`, bounding a file to one turn's worth of deltas; consumers detect the truncation (offset past EOF) and rewind. Degrades safely: any IO failure is swallowed and the process falls back to today's in-process-only behavior.
HTTP/SSE client for the long-lived gateway daemon.
Interactive channels call this facade instead of gateway.state directly. It
discover-or-starts the one daemon for the current DB, then speaks the same
HTTP/SSE API every other client uses. This is the thin-client half of the
gateway-daemon plan: token refresh, turn execution, and live streaming happen
in ONE process.
WHICH daemon is a policy of this namespace: normally the one this machine
manages for the current DB, or — through connect-remote! (the --gateway
flag / VIS_GATEWAY_URL) — a gateway on another machine, attached to over HTTP
and never spawned, restarted or stopped from here.
HTTP/SSE client for the long-lived gateway daemon. Interactive channels call this facade instead of `gateway.state` directly. It discover-or-starts the one daemon for the current DB, then speaks the same HTTP/SSE API every other client uses. This is the thin-client half of the gateway-daemon plan: token refresh, turn execution, and live streaming happen in ONE process. WHICH daemon is a policy of this namespace: normally the one this machine manages for the current DB, or — through `connect-remote!` (the `--gateway` flag / `VIS_GATEWAY_URL`) — a gateway on another machine, attached to over HTTP and never spawned, restarted or stopped from here.
Gateway daemon discovery + registry (build order step 1).
One long-lived gateway per DB owns execution; every TUI/web/whatever is a thin client of it. This namespace answers the boot-time question: "is a gateway already running for my DB — attach; else spawn one, DETACHED (nobody's child), then hand back where to connect."
Registry: one EDN file per DB at ~/.vis/gateway/registry/<sha256(db)>.edn
holding {:pid :port :host :secret :db :created-at}. Freshness = the recorded
:pid is still alive AND a caller-supplied probe confirms the port+secret
are really OUR daemon (guards OS pid reuse — see D4/Q3 in TODO-gateway-daemon).
Design decisions (locked, see TODO-gateway-daemon.md):
--db share one daemon).register-self! from serve-main!), so a
spawner never needs the child pid — it just polls for a fresh registry.:memory never registers/spawns (headless one-shot stays in-process).Effects (spawn, probe, pid-liveness) are injectable so the orchestration
in discover-or-start! is unit-testable without a real process.
Gateway daemon discovery + registry (build order step 1).
One long-lived gateway per DB owns execution; every TUI/web/whatever is a
thin client of it. This namespace answers the boot-time question: "is a
gateway already running for my DB — attach; else spawn one, DETACHED (nobody's
child), then hand back where to connect."
Registry: one EDN file per DB at `~/.vis/gateway/registry/<sha256(db)>.edn`
holding `{:pid :port :host :secret :db :created-at}`. Freshness = the recorded
`:pid` is still alive AND a caller-supplied `probe` confirms the port+secret
are really OUR daemon (guards OS pid reuse — see D4/Q3 in TODO-gateway-daemon).
Design decisions (locked, see TODO-gateway-daemon.md):
- Q2 registry key = the DB path (two dirs sharing `--db` share one daemon).
- Q3 race = port-bind winner is the daemon; the loser attaches. The daemon
SELF-REGISTERS on startup (via [[register-self!]] from `serve-main!`), so a
spawner never needs the child pid — it just polls for a fresh registry.
- Q5 `:memory` never registers/spawns (headless one-shot stays in-process).
Effects (`spawn`, `probe`, pid-liveness) are injectable so the orchestration
in [[discover-or-start!]] is unit-testable without a real process.Android push (Firebase Cloud Messaging HTTP v1).
Apple's APNs lives in gateway.push; this is its Android twin and the two
are dispatched on the registered device's :platform. Credentials are a
Google service-account JSON — from the macOS keychain (service vis-fcm,
account service_account), from the environment, or from a file under
~/.vis/fcm/. Key material is never returned, logged or sent over the wire.
Android push (Firebase Cloud Messaging HTTP v1). Apple's APNs lives in `gateway.push`; this is its Android twin and the two are dispatched on the registered device's `:platform`. Credentials are a Google service-account JSON — from the macOS keychain (service `vis-fcm`, account `service_account`), from the environment, or from a file under `~/.vis/fcm/`. Key material is never returned, logged or sent over the wire.
The ONE reader of a push credential out of the macOS login keychain.
Both push transports keep their signing secret there rather than in a file on
disk: APNs its ES256 .p8 (service vis-apns), FCM its Google service-account
JSON (service vis-fcm). The secret never sits in a world-readable file.
Each answer is held in memory for CACHE_TTL_MS before security is asked
again. It used to be read on demand and never cached, so locking the keychain
revoked access immediately - but push/config asks for five secrets, and it
runs inside capabilities, a request every connected client makes: a JFR
profile of a live gateway caught the daemon forking security at a steady
rate, ~21 ms of wall and ~10 ms of CPU a fork, to re-read five values that
change only when a human edits the keychain. Locking the keychain now revokes
access within the TTL rather than instantly; an unlocked-again keychain is
likewise noticed within it.
nil anywhere but macOS, and nil while vis.push.home is set: a redirected push
home means a test fixture, and the developer's real keychain must never leak
into it.
The ONE reader of a push credential out of the macOS login keychain. Both push transports keep their signing secret there rather than in a file on disk: APNs its ES256 `.p8` (service `vis-apns`), FCM its Google service-account JSON (service `vis-fcm`). The secret never sits in a world-readable file. Each answer is held in memory for `CACHE_TTL_MS` before `security` is asked again. It used to be read on demand and never cached, so locking the keychain revoked access immediately - but `push/config` asks for five secrets, and it runs inside `capabilities`, a request every connected client makes: a JFR profile of a live gateway caught the daemon forking `security` at a steady rate, ~21 ms of wall and ~10 ms of CPU a fork, to re-read five values that change only when a human edits the keychain. Locking the keychain now revokes access within the TTL rather than instantly; an unlocked-again keychain is likewise noticed within it. nil anywhere but macOS, and nil while `vis.push.home` is set: a redirected push home means a test fixture, and the developer's real keychain must never leak into it.
Gateway pairing helpers for remote clients.
The QR payload is deliberately tiny and URL-shaped so native apps can scan it without an HTTP round trip:
vis://gateway?url=http%3A%2F%2F100.64.0.10%3A7890&token=...
Tailscale fits naturally: if a 100.64.0.0/10 interface is present we prefer it over LAN addresses, otherwise we fall back to site-local IPv4 addresses.
Gateway pairing helpers for remote clients. The QR payload is deliberately tiny and URL-shaped so native apps can scan it without an HTTP round trip: vis://gateway?url=http%3A%2F%2F100.64.0.10%3A7890&token=... Tailscale fits naturally: if a 100.64.0.0/10 interface is present we prefer it over LAN addresses, otherwise we fall back to site-local IPv4 addresses.
Native push notifications (Apple Push Notification service).
ONE job: when a turn finishes on this gateway, wake the phone that asked to be woken. Everything here is server-side; the app only ever hands us a device token.
Three moving parts:
Credentials. A token-based APNs auth key (.p8, ES256) plus its key
id, the Apple team id and the app's bundle id (the APNs topic).
Resolved from VIS_APNS_KEY_PATH / VIS_APNS_KEY_ID / VIS_APNS_TEAM_ID
/ VIS_APNS_TOPIC, else auto-discovered from ~/.vis/apns/AuthKey_<kid>.p8
(the key id is read off the filename) with the team/topic still from env
or ~/.vis/apns/apns.edn. No credentials = push silently OFF; the gateway
keeps working exactly as before.
A device registry at ~/.vis/devices.edn — device token -> platform,
APNs environment, client label/version, timestamps. Registration is
idempotent on the token. Tokens are SECRETS: nothing here logs more than
a masked prefix.
A sender — a signed ES256 JWT (cached, refreshed well inside Apple's
one-hour window) over HTTP/2 to api.push.apple.com. A device that
registered with the wrong environment is retried once against the other
host, and an APNs BadDeviceToken/Unregistered verdict evicts the
device so a stale token cannot accumulate.
The wire surface lives in gateway.server (/v1/devices); this namespace
knows nothing about Ring.
Native push notifications (Apple Push Notification service). ONE job: when a turn finishes on this gateway, wake the phone that asked to be woken. Everything here is server-side; the app only ever hands us a device token. Three moving parts: 1. **Credentials.** A token-based APNs auth key (`.p8`, ES256) plus its key id, the Apple team id and the app's bundle id (the APNs *topic*). Resolved from `VIS_APNS_KEY_PATH` / `VIS_APNS_KEY_ID` / `VIS_APNS_TEAM_ID` / `VIS_APNS_TOPIC`, else auto-discovered from `~/.vis/apns/AuthKey_<kid>.p8` (the key id is read off the filename) with the team/topic still from env or `~/.vis/apns/apns.edn`. No credentials = push silently OFF; the gateway keeps working exactly as before. 2. **A device registry** at `~/.vis/devices.edn` — device token -> platform, APNs environment, client label/version, timestamps. Registration is idempotent on the token. Tokens are SECRETS: nothing here logs more than a masked prefix. 3. **A sender** — a signed ES256 JWT (cached, refreshed well inside Apple's one-hour window) over HTTP/2 to `api.push.apple.com`. A device that registered with the wrong environment is retried once against the other host, and an APNs `BadDeviceToken`/`Unregistered` verdict evicts the device so a stale token cannot accumulate. The wire surface lives in `gateway.server` (`/v1/devices`); this namespace knows nothing about Ring.
Relayed push — how a gateway wakes a phone WITHOUT holding Apple's or Google's signing key.
APNs binds a topic to the Apple team that owns it: a key minted by anyone
else, aimed at someone else's bundle id, is refused forever (403 InvalidProviderToken / TopicDisallowed). So a self-hosted gateway can
never push to a companion built and signed by somebody else — unless the
signing key stays on infrastructure the app's publisher runs, and the
gateway is given a capability instead of a credential.
That capability is a GRANT. The device asks the relay for one and hands
it to this gateway during "notify this device"; the gateway POSTs
{grant, title, body} and the relay signs and forwards. Consequences worth
the indirection:
.p8, no service-account JSON, nothing revocable
only by breaking push for everyone else;The relay itself lives in apps/vis-companion-relay (a Cloudflare Worker),
and every gateway names the publisher's one by DEFAULT (DEFAULT-URL): a
machine nobody configured is already able to push. A device may name another
and is believed — which relay can sign for a build is a property of the
BUILD, so the app mints its grant at the relay serving the app it is and
posts {grant, relay_url} to /v1/devices. VIS_PUSH_RELAY_URL, or
~/.vis/relay.edn {:url "https://push.example.com"}, replaces the default
on one machine; the direct gateway.push / gateway.fcm credentials still
work exactly as before.
Relayed push — how a gateway wakes a phone WITHOUT holding Apple's or
Google's signing key.
APNs binds a topic to the Apple team that owns it: a key minted by anyone
else, aimed at someone else's bundle id, is refused forever (`403
InvalidProviderToken` / `TopicDisallowed`). So a self-hosted gateway can
never push to a companion built and signed by somebody else — unless the
signing key stays on infrastructure the app's publisher runs, and the
gateway is given a capability instead of a credential.
That capability is a GRANT. The *device* asks the relay for one and hands
it to this gateway during "notify this device"; the gateway POSTs
`{grant, title, body}` and the relay signs and forwards. Consequences worth
the indirection:
* this gateway holds no `.p8`, no service-account JSON, nothing revocable
only by breaking push for everyone else;
* this gateway never learns the raw APNs/FCM device token, so a gateway you
do not trust cannot fingerprint the device it notifies;
* a grant expires by itself. Its expiry travels inside it, sealed, so the
relay keeps no list of anybody and an abandoned gateway simply goes mute.
The relay itself lives in `apps/vis-companion-relay` (a Cloudflare Worker),
and every gateway names the publisher's one by DEFAULT (`DEFAULT-URL`): a
machine nobody configured is already able to push. A device may name another
and is believed — which relay can sign for a build is a property of the
BUILD, so the app mints its grant at the relay serving the app it is and
posts `{grant, relay_url}` to `/v1/devices`. `VIS_PUSH_RELAY_URL`, or
`~/.vis/relay.edn` `{:url "https://push.example.com"}`, replaces the default
on one machine; the direct `gateway.push` / `gateway.fcm` credentials still
work exactly as before.Canonical in-memory registry of gateway-managed stateful resources. Every long-lived thing vis spawns registers here so the agent, footer, and shutdown all share one live view. Resources are deliberately not persisted: their processes belong to the gateway and die when the gateway dies.
Canonical in-memory registry of gateway-managed stateful resources. Every long-lived thing vis spawns registers here so the agent, footer, and shutdown all share one live view. Resources are deliberately not persisted: their processes belong to the gateway and die when the gateway dies.
Runtime identity, daemon staleness and compatibility diagnostics.
Canonical protocol numbers, headers, handshake parsing and the pure compatibility
verdict live in com.blockether.vis.contract.gateway. This namespace contributes
the release/build identity of the running process and the concrete client/server
adapters that combine it with that contract.
Runtime identity, daemon staleness and compatibility diagnostics. Canonical protocol numbers, headers, handshake parsing and the pure compatibility verdict live in `com.blockether.vis.contract.gateway`. This namespace contributes the release/build identity of the running process and the concrete client/server adapters that combine it with that contract.
Gateway HTTP/SSE server.
Clojure-native stack: reitit-ring routes -> Ring middleware -> a Jetty 12
CORE handler (ring.adapter.jetty9 — no servlet layer) on JDK virtual
threads (:virtual-threads? true).
SSE is a Ring StreamableResponseBody whose virtual thread is the
connection's SINGLE socket writer: replay rides first, then it drains a
bounded per-connection event queue that state/fan-out! enqueues onto,
emitting a heartbeat comment on idle to keep the pipe warm and detect
dead clients.
This is internal plumbing, not a channel: it registers no channel
descriptor and owns no renderer - it ships canonical IR and the
client renders (§4.1). Any host process (the vis-agent gateway start daemon, a
TUI run, an embedded caller) can start it alongside whatever else it
is doing via start!.
Gateway HTTP/SSE server. Clojure-native stack: reitit-ring routes -> Ring middleware -> a Jetty 12 CORE handler (`ring.adapter.jetty9` — no servlet layer) on JDK virtual threads (`:virtual-threads? true`). SSE is a Ring `StreamableResponseBody` whose virtual thread is the connection's SINGLE socket writer: replay rides first, then it drains a bounded per-connection event queue that `state/fan-out!` enqueues onto, emitting a heartbeat comment on idle to keep the pipe warm and detect dead clients. This is internal plumbing, not a channel: it registers no channel descriptor and owns no renderer - it ships canonical IR and the client renders (§4.1). Any host process (the `vis-agent gateway start` daemon, a TUI run, an embedded caller) can start it alongside whatever else it is doing via `start!`.
Concrete SSE framing for the gateway's Ring server transport.
Concrete SSE framing for the gateway's Ring server transport.
Gateway session manager.
One process-global registry over the live session fleet: per-session
ordered event log (monotonic :seq, ring-buffered), SSE subscriber
fan-out, async turn submission with idempotency keys, cancellation,
and turn/cost metrics.
The engine is reached ONLY through the same internal surfaces the
TUI channel uses: loop/create!-send!-close! for the
lifecycle, :hooks {:on-chunk ...} phased chunks for the live
stream, ctx-loop/session-snapshot for the context. No engine state
lives here - this namespace owns wire bookkeeping (events, turn
records, subscribers), nothing else.
Gateway session manager.
One process-global registry over the live session fleet: per-session
ordered event log (monotonic `:seq`, ring-buffered), SSE subscriber
fan-out, async turn submission with idempotency keys, cancellation,
and turn/cost metrics.
The engine is reached ONLY through the same internal surfaces the
TUI channel uses: `loop/create!`-`send!`-`close!` for the
lifecycle, `:hooks {:on-chunk ...}` phased chunks for the live
stream, `ctx-loop/session-snapshot` for the context. No engine state
lives here - this namespace owns wire bookkeeping (events, turn
records, subscribers), nothing else.Gateway-free, serial NDJSON transport for the Python SDK. Uses the same SDK handlers and protocol as HTTP; never starts Jetty or performs discovery. The owning process must select its own database before entering this loop.
Gateway-free, serial NDJSON transport for the Python SDK. Uses the same SDK handlers and protocol as HTTP; never starts Jetty or performs discovery. The owning process must select its own database before entering this loop.
Gateway transport for the shared View lifecycle.
The engine publishes :view/open, :view/patch and :view/close envelopes on
every selected channel. This namespace projects the :app copy into the matching
view.open, view.patch and view.close session events, preserving :kind so
clients choose capability policy without guessing from event names.
An :input View blocks while a :live View streams coalesced patches on
[[live-flush-ms]]. Operator intent for either kind enters through one action!
and one kind-independent REST resource; the engine owns the closed action
vocabulary and capability checks.
Gateway transport for the shared View lifecycle. The engine publishes `:view/open`, `:view/patch` and `:view/close` envelopes on every selected channel. This namespace projects the `:app` copy into the matching `view.open`, `view.patch` and `view.close` session events, preserving `:kind` so clients choose capability policy without guessing from event names. An `:input` View blocks while a `:live` View streams coalesced patches on [[live-flush-ms]]. Operator intent for either kind enters through one [[action!]] and one kind-independent REST resource; the engine owns the closed action vocabulary and capability checks.
Gateway-local Web Push (RFC 8291 and RFC 8292).
Each gateway owns one generated VAPID P-256 key pair in its own ~/.vis
home. The browser gets this gateway's public key, registers its subscription
here, and this gateway encrypts and sends notifications directly to the
browser's push service. There is no publisher URL or shared web relay.
Java interop is kept at the cryptographic and file boundaries. The rest of the namespace passes ordinary Clojure maps and byte arrays between small helpers so the protocol steps remain visible and testable.
Gateway-local Web Push (RFC 8291 and RFC 8292). Each gateway owns one generated VAPID P-256 key pair in its own `~/.vis` home. The browser gets this gateway's public key, registers its subscription here, and this gateway encrypts and sends notifications directly to the browser's push service. There is no publisher URL or shared web relay. Java interop is kept at the cryptographic and file boundaries. The rest of the namespace passes ordinary Clojure maps and byte arrays between small helpers so the protocol steps remain visible and testable.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |