:wagoe/http {:virtual-threads? true} ; each request on its own virtual thread
:wagoe/rpc {:virtual-threads? true} ; same, for a service's RPC listener
:wagoe/jobs {:workers {:virtual-threads? true}} ; each worker loop on one
Wagoe’s architecture (FC/IS + hexagonal ports) is meant to let you scale vertically, horizontally, or a mix — mostly by configuration rather than rewrites. This page is an honest map of how far that holds today: which knobs exist, which components are already safe to run as many replicas, and which still hold in-process state that you must account for.
| Axis | Meaning |
|---|---|
Vertical | One process, more resources. Bigger heap, larger connection/thread pools, more CPU. Pure configuration in Wagoe. |
Horizontal | Many processes (replicas) behind a load balancer, sharing backing services (Postgres, Redis). Requires that no request-handling state lives in a single process. |
Functional decomposition | Slice modules into separate deployables (microservices-style) that scale independently. A cross-module call that was in-process becomes a network call. Highest leverage, highest effort. |
All sizing knobs live in resources/conf/{dev,test,prod,acc}/config.edn plus
environment variables. Change the value, restart the process. No code.
| Knob | Where | Notes |
|---|---|---|
DB connection pool |
| HikariCP. prod default |
HTTP server |
| Jetty. |
JVM heap / GC |
| Default |
Cache (Redis) pool |
| Jedis pool. prod default |
|
Watch the multiplication: with N replicas each holding a pool of |
Off by default. Turn them on per listener and for job workers:
:wagoe/http {:virtual-threads? true} ; each request on its own virtual thread
:wagoe/rpc {:virtual-threads? true} ; same, for a service's RPC listener
:wagoe/jobs {:workers {:virtual-threads? true}} ; each worker loop on one
A request that blocks on the database or an HTTP call then holds no platform
thread. Jetty caps concurrent tasks at 200, its VirtualThreadPool default;
the connector’s acceptor and selectors hold a few of those.
Context still applies. Tenant schema, MDC and the tracing and error contexts are set and read on the thread that serves the request or runs the job, and each request gets a fresh one.
Size the connection pool for 200. Virtual threads add no connections; a
request beyond maximum-pool-size waits connection-timeout-ms, then fails.
Pinning is why it is opt-in. On JDK 21 a virtual thread blocked inside
synchronized holds its carrier. The PostgreSQL and MySQL drivers, HikariCP,
Logback and Clojure 1.12 avoid it; SQLite and H2 do not. Check with JFR’s
jdk.VirtualThreadPinned event before turning it on in production.
Cross-module calls go through protocols defined in each module’s ports.clj
(enforced by bb check:ports). Core logic depends on a protocol, never on a
concrete adapter. That seam is the scaling lever: swap an in-process adapter for
a distributed one — Redis, a queue, a remote service — without touching the
functional core.
Several libraries already ship both adapters, and they select one the same way.
:memory and :redis mean the same thing in each; :db exists where a module
has a database adapter, which today is jobs alone. The pre-1.0 spellings —
:in-memory, :redis-streams, :database — still work and warn (#524):
:wagoe/cache {:provider :redis ...} ; | :memory (the default)
:wagoe/realtime {:provider :redis ...} ; | :memory (the default)
:wagoe/events {:provider :redis ...} ; | :memory — no default, throws otherwise
:wagoe/jobs {:provider :redis ...} ; | :memory (the default) | :db
This is the template every other seam follows: the protocol is the contract, the distributed adapter is "just configuration" once it exists.
With one condition that is easy to leave implicit. Swapping adapters is only
safe if both answer the protocol the same way, and a protocol does not enforce
that — a test across both adapters does. libs/cache had thirteen divergences
between its two adapters, each invisible to per-adapter suites that only ever
asked one of them, until a contract sweep enumerated the surface (#392). Any
new seam here inherits that risk, so a second adapter comes with a sweep that
runs every method against both.
| Component | N replicas | Detail |
|---|---|---|
Cache | ✅ | Redis adapter ( |
Jobs | ✅ | Redis queue ( |
Auth / sessions | ✅ | DB-backed, pure core ( |
Multi-tenancy | ✅ | schema-per-tenant ( |
Email / external | ✅ | Async via the jobs queue / stateless IO adapters. |
Event bus | ✅ | Redis Streams adapter ( |
Readiness checks | ✅ |
|
Rate limiting | ✅ | The config-driven |
Graceful shutdown | ✅ | Integrant |
Realtime / WebSocket | ✅ * | Replica-safe via |
| Shape | When |
|---|---|
Single fat node | Vertical only. One process, large heap and pools. Everything works, including realtime and the in-memory adapters. Rate limiting is the one exception: the |
N stateless web replicas | The main horizontal mode. N copies of the uberjar behind a load balancer, sharing Postgres + Redis. Cache, jobs, auth, tenancy all scale. Caveats: activate |
Web / worker split | Dedicated job-worker processes separate from web. |
Module as its own service |
|
The same image runs all four; the container argument selects the mode. Compose files, manifests and what has actually been brought up are in Deployment Topologies.
Use the Redis cache and jobs adapters, never :memory, for more than one replica.
Register all job handlers on every instance. A dequeued job with no local handler is re-enqueued so another instance can run it, and is dead-lettered only after :max-requeue-age-ms (default 5 min) of going continuously unhandled — so a job-type that no worker registers costs five minutes of re-enqueue churn before it fails. (:max-requeues, default 10000, is a runaway backstop, not the budget: give-up is age-based so a slow handler-owning worker cannot lose a job to attempt exhaustion.)
Enable rate limiting (:wagoe/http :rate-limit :enabled? true) with an active Redis cache for a global limit across replicas. In :prod this is not optional — the boot fails without a reachable cache. :wagoe/cache is :inactive in the shipped prod profile; move it to :active first.
Behind a load balancer or reverse proxy, set :wagoe/http :trusted-proxies to its CIDRs. Without it every client shares the proxy’s address, so they share one rate-limit bucket and the audit log records the proxy. See Deployment Patterns.
Read secrets with #secret, from the environment or from mounted files (#secret [:file "db-password"]), not literals in config.edn. A missing one fails boot. See config.
Keep replicas × maximum-pool-size under Postgres max_connections.
Confirm your load balancer points health probes at /health/ready (503-aware), not /health/live.
For WebSocket: use :provider :redis on :wagoe/realtime to scale across replicas (ADR-035). Sticky sessions / single-node are only required with the default :memory provider.
Deploy Redis, and a load balancer for N replicas. The root docker-compose.yml is a single-instance dev stack (db, app, docs, dev-tools — no Redis); the replica and per-service references are deploy/compose/multi-instance.yml, deploy/compose/per-service.yml and deploy/k8s/wagoe.yaml.
The third axis: run a module (or a few) as its own process, scaled and deployed
independently of the rest. This is where the ports.clj seam pays off most — and
where the most net-new infrastructure is needed. It is not free "by config"
today, but the architecture is positioned for it.
| Asset | How it helps |
|---|---|
Per-module activation | Modules are gated by |
The protocol seam | Consumers depend on the protocol (e.g. |
Wire format ready | Muuntaja (JSON / EDN / Transit) is already in the HTTP stack ( |
Remote-adapter template |
|
Clean data boundaries |
|
An acyclic dependency graph |
|
Context plumbing |
|
✅ Generic remote-port adapter — shipped in #384.
wagoe.platform.shell.rpc.client/remote-adapter returns a value implementing
any protocol by calling a service over HTTP, built from the protocol’s own
:sigs so every module’s port works without a bespoke client. The counterpart
…rpc.server/rpc-handler serves a protocol from the process that owns it.
correlation-id, tenant and auth ride the same headers the interceptor
pipeline already uses. See The remote-port adapter.
Network resilience — timeouts, retries and a shared circuit breaker are in the remote-port adapter (#384, #390). Service discovery is not: URLs are configuration, hardcoded or env-supplied. The libs/external adapters share them through wagoe.platform.resilience: FTPS retries transient failures, SMTP, IMAP and Twilio can retry a call that never connected (:retries), and passing a :cache enables the breaker.
✅ Break the allowlisted dependency cycles — done. allowed-cycle-edges in check_deps.clj is #{}; admin↔user and platform↔{user,tenant,admin,workflow,search} were dissolved rather than allowlisted, so no module pair is blocked from separation by a cycle.
✅ Async option — shipped in #387 as libs/events: IEventPublisher / IEventSubscriber / IEventHistory, with a Redis Streams adapter (at-least-once, consumer groups) and an in-memory one. A publisher does not know who is listening and is unaffected if a consumer is down — the complement to the synchronous remote-port adapter.
Data ownership decision — schema-per-tenant assumes co-located modules in one Postgres. Across services either share the DB (pragmatic) or give each service its own; there are no distributed transactions, so split writes become eventual-consistency.
✅ Service launch mode — shipped in #385: java -jar wagoe.jar service <module>… boots a named subset plus the platform, and starts that module’s RPC endpoint when one is configured. See Running a module as a service.
Running configuration — compose files, Kubernetes manifests, the environment variables and what has actually been brought up — is in Deployment Topologies.
service boots only the modules named and the platform they need:
java -jar wagoe.jar service payments # one module
java -jar wagoe.jar service user tenant # several in one process
In the test profile service user runs 20 of the application’s 34 components.
/health answers, /web/login answers, /api/v1/tenants is a 404 — the tenant
module is not there to serve it.
A service that serves HTTP needs the user module in the same process: it is what
signs a request in. Without it every route that is not :public true answers
401 — the platform fails closed rather than open. Include user in the slice, or keep the slice
behind RPC only.
Which keys belong to which module is declared, not guessed:
;; config.edn, under :active — merged over
;; wagoe.system-config/default-service-catalogue, replacing an entry of the same name
;; rather than merging into it
:wagoe/services
{:payments {:keys [:wagoe/payment-provider]
:rpc {:protocol 'my.app.payments.ports/IPaymentProvider
:component :wagoe/payment-provider}}}
The keys must be the ones the config actually emits. Naming a component that
does not exist does not fail — the real component is then claimed by nobody,
counts as platform, and runs inside every service. wagoe.service-launch-test
asserts every emitted key is either claimed by a module or listed as platform,
which is the only thing that catches it.
A key no entry claims counts as platform and runs everywhere. That is the deliberate failure direction: a catalogue that has fallen behind boots a service larger than it needs to be, rather than one missing a component it depended on.
With :wagoe/rpc in config, a service also starts the remote-port endpoint on
its own port, so the rest of the deployment can call it. Without it the boot
says so explicitly — a module running alone that nothing can reach is a process
doing no work, and it otherwise reports itself healthy.
The mechanics are in wagoe.platform.core.system-selection, and the one thing
worth knowing about them: refs to unselected modules are removed before the
dependency closure is taken. :wagoe/http-handler refers to every module, so
following its refs first pulls the whole system back in — and the result still
boots and still passes a health check, which is what makes the mistake worth
naming.
A caller depends on a protocol, never on a concrete record. So a module can move into its own process without its callers changing — provided something implements that protocol by making a network call. That is this:
;; In the process that owns payments — serve the protocol on its own listener
(jetty/run-jetty (rpc-server/rpc-app pay-ports/IPaymentProvider provider
{:service-key service-key})
{:port 3001 :join? false})
;; In the process that consumes it — a value that satisfies IPaymentProvider
(def payments
(rpc-client/remote-adapter pay-ports/IPaymentProvider "http://payments:3001"
{:timeout-ms 5000
:service-key service-key
:cache cache ; enables the breaker
:context {:correlation-id id :tenant-id t}}))
(ports/create-checkout-session payments {...}) ; unchanged at the call site
rpc-app is a standalone Ring handler, not a route map for a module’s :api
or :web slot, and that is deliberate. The router rewrites both: :api paths
gain the version prefix and :web paths gain /web, so a client on the
default :path gets a 404 either way. :web is worse than merely wrong — a
POST there is CSRF-validated when CSRF is enabled, so the call is rejected 403
by a check meant for browser forms, for which a service-to-service caller has
no token.
Underneath the mechanics: this endpoint invokes port methods, and the public
listener is not where it belongs. A sliced-out service serves it on a listener
reachable only from inside the deployment — which is what the service launch
mode (#385) starts. A service that serves it elsewhere tells the client with
:path.
The adapter is built from the protocol’s :sigs, so adding a method to a port
carries it across the hop with no client change. Each adapter is its own object
implementing that protocol’s interface — not a shared type extended in place,
which would make every adapter already built satisfy each newly adapted
protocol, so a payments adapter would answer satisfies? for ICache and send
cache calls to the payments URL.
What it does, and what it deliberately does not:
Context |
|
Wire format |
|
Transport errors | Returned as data, in the |
Thrown exceptions | Raised again on the near side, keeping the |
HTTP status | A typed error keeps the status it has in-process, read from the same
|
Malformed requests | Answered, not thrown. The endpoint is reachable by anything that can post to
it, so a body carrying no operation — or an operation that is not a name, or
the wrong number of arguments for one — comes back as Argument count is checked against the protocol’s own |
Retries | Only failures where the call may not have completed, and only
Which failures count as never-executed is decided conservatively. A
The HTTP client’s own retry handler is disabled. Apache HttpClient resends
low-level I/O failures before this code sees an outcome, so An answer is never retried, whatever |
Exposure | The server resolves an operation against the protocol’s own |
Authentication | A service key in This is not a substitute for keeping the endpoint off a public listener. It is the part that can be enforced in code. |
Service discovery | Not included — the URL is configuration. Hardcoded or env-supplied for now, as the MVP in this document assumes. |
Circuit breaker | Opt-in: pass a The failure count is an atomic increment, not a read-modify-write: many callers hitting one outage at the same moment is what a shared breaker is for, and it is exactly when a read-modify-write loses increments — each reads the same value and writes back the same successor, so a burst advances the counter by one and the breaker never trips. A probe that fails reopens the window from that moment and releases its lease, so the outage is not forgotten on the original window’s schedule and the next window can still be probed. An invalid It protects the traffic after the burst, not the burst. Calls issued at the same instant all pass the check before any of them has failed. Trips on consecutive A refused call returns If the cache itself is unreachable the breaker fails open — the worst case is the behaviour of no breaker, which is better than a cache outage taking every remote call with it. |
| Module | Effort | Why |
|---|---|---|
payments | Easy | Depends on |
core, observability | Easy | Leaf / infra; no sibling deps. (Usually shared libs, not standalone services.) |
user, tenant, external | With work | Depend on platform + the in-process service assumption in middleware. The remote adapter exists and the cycles are gone; what remains is deciding data ownership and pointing consumers at a remote adapter instead of the local component — a wiring change in the application’s config, which is why |
admin, search, workflow | Entangled |
|
|
Recommended path: prove the remote-port adapter by extracting payments as a
standalone service (it depends on |
The architecture delivers the promise; these are the concrete pieces that make "scale by configuration" fully true. These came out of the scaling spike (#198):
✅ Realtime Redis pub/sub adapter — shipped in #213 (ADR-035). WebSocket is now replica-safe via :provider :redis on :wagoe/realtime.
✅ Graceful connection draining — shipped in #218. Configurable shutdown grace (:wagoe/http :drain-timeout-ms) lets rollouts finish in-flight requests; Jetty GracefulHandler + setStopTimeout wired in wiring.clj.
✅ Default rate-limit wiring — shipped in #219. Config-driven http-rate-limit-protection is in the default pipeline; enable via :wagoe/http :rate-limit and it uses the Redis cache for a cross-replica limit (per-process fallback documented).
✅ Jobs hardening — shipped in #220. Missing-handler jobs are re-enqueued (bounded) instead of silently dead-lettered, an empty-registry worker warns at startup, and scheduled-job promotion is an atomic claim (ZREM / swap-vals!) so a due job runs exactly once across workers.
✅ Opt-in virtual threads — for HTTP and RPC listeners and job workers; see Virtual threads.
✅ Deploy topology reference — done (#386). Compose + k8s with N replicas, Redis, a load balancer, the web/worker split and a module-as-a-service, in Deployment Topologies. No instance-id variable: nothing in the codebase reads one, so adding it would have been decoration.
For functional decomposition (the bigger bet):
✅ Generic remote-port adapter + RPC envelope — done (#384), with the circuit breaker in #390. clj-http client implementing a module protocol over transit, context propagation, typed errors, retry bounded to calls that never executed.
✅ Service launch mode — done (#385). wagoe.main boots a named module subset as an independent service, with its RPC endpoint on its own port.
✅ Break allowlisted dependency cycles — done (#279, #282, #283). admin↔user and platform↔{user,tenant,admin,workflow,search} are gone; check_deps.clj allowlists no cycle edges, so the gate now fails on any cycle at all.
✅ Event-bus port + adapter — done (#387). libs/events, Redis Streams and in-memory. Kafka would be another adapter behind the same ports, not a change to them.
What is left, in order:
Service discovery — remote-port URLs are configuration today. Fine for a fixed deployment, insufficient once services move.
Data ownership across services — schema-per-tenant assumes co-located modules in one Postgres (see the entry above).
Activate :wagoe/events in the prod profile — it is configured in test only, so the event bus is currently a test-profile capability in practice.
Can you improve this documentation? These fine people already did:
Thijs Creemers & thijscreemersEdit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |