ADR-0023: Long-running transactions as branch-and-merge sagas

Status: Accepted (2026-08-25); implemented but for the thin-client and SQL polish of the delivery sketch's last phase — the :db.saga/* vocabulary, its lifecycle transitions and the rules the writer holds them to, the peer API, corium_sys.sagas, and corium saga; id-block leasing in the allocator, the branch overlay and its step pipeline, and the tier-2 read surfaces; the merge, with its conflict scan, guards, resolutions, and atomic commit-and-flip; and the way a saga ends when nobody ends it — compensation composed in the transactor and applied atomically with the flip, the expiry sweep, the liveness invariant, and branch retention. Design: docs/design/long-running-transactions.md. Builds on ADR-0016 (transaction metadata), follows the plan/apply pattern of ADR-0020, and assigns its expiry sweep to the operator service of ADR-0019.

Context

A Corium transaction is one atomic batch, serialized and durable in milliseconds. Atomicity, isolation, and brevity are welded together, and real workloads need the first two without the third: multi-step business processes that run for days, human-reviewed repairs and backfills, incrementally prepared imports published in one motion. Such work must be durable across crashes, must commit as a whole or leave canonical state untouched, and must not hold anything resembling a lock in a system whose readers are coordination-free by invariant.

The visibility requirement is the tension that shaped the design. A saga must not be invisible to the wider system — outside readers may legitimately want to know work is in flight, and some want to read its partial progress — yet readers must not be required to join the saga or even know sagas exist, and a reader who did observe partial progress needs a way to adapt when the saga rolls back.

Existing mechanisms each fail alone: Db::with_transaction is speculative but ephemeral and process-local; db fork is durable and writable but heavyweight (log-prefix copy), lives in the user catalog, and has no merge; transaction metadata labels work but cannot make many transactions atomic.

Three alternative models were rejected in design iteration (recorded in full in the design doc): classic interleaved-commit sagas with compensations make partial state canonical for every reader and reduce atomicity to best-effort compensation; long-lived write intents are locks held across days and violate the coordination-free invariant; tentative datoms in the parent log with status-flipped visibility make as-of views status-dependent — visibility would no longer be a pure fold of the log prefix — and put a saga filter on every hot read path.

Decision

A saga is a database branch plus a registry entry in the parent database.

• Branch. Opening a saga creates a lightweight overlay branch — the parent's published state as of opening basis t₀ plus the branch's own log — hosted in the parent's transactor process. O(1) creation, segments shared by content address, parent's encryption keys, not in the user catalog. Steps are ordinary fully-validated durable transactions against the branch, serialized in a per-branch pipeline: pre-leased id blocks mean a step never enters the parent's writer queue — only open, extend, abort, and merge do. The branch keeps its own timeline (t from t₀, its own :db/txInstant monotonicity); branch transaction entities never merge, so the tx partition needs no grants. Schema changes on a branch are refused.
• Registry. Engine-installed vocabulary (:db.saga/*: id, status, basis, owner, expiry, id grants, advisory footprint, checked reservations, outcome refs) in every database; enum values are namespaced idents in the :db convention (:db.saga.status/open, abbreviated :open etc. in prose). Open, extend, commit, abort, and expire are ordinary parent transactions on the saga entity.
• Reservations bind the saga, never other writers. Beyond the advisory footprint, a saga may reserve the exact pre-existing entities (and/or whole attributes) it operates on. The branch pipeline enforces the declaration at step time: writes to unreserved pre-existing entities are refused, and refs from novelty to pre-existing entities must target reserved ones (reverse-ref/VAET visibility is why), so branch-created entities attach to the parent graph only through the reserved set. Readers get a reliable effect boundary ("X is outside the set" means untouched, as of the registry basis read; :db.saga/sealed fixes the set at open), and the merge's write–write, dangling-ref, and retraction-miss scans confine to the reserved set. Parent writers are never constrained — races on reserved entities are still arbitrated by the merge scan, now with an early warning tier-1 tooling can watch for.
• Entity-id grants. The parent's writer leases the branch disjoint per-partition sequence blocks, recorded in the registry; branch allocations survive the merge verbatim, so ids resolved by the saga's application — or seen by outside readers of the branch — never change identity at commit. Abandoned blocks are simply holes in a 42-bit space.
• Merge. Commit squashes the branch's net novelty into one parent transaction, validated inside the single-writer path against the parent's current state and schema: write–write conflict scan over (t₀, now], uniqueness, dangling refs, retraction misses (cardinality-many assertions deliberately union), plus explicit guards (CAS-shaped preconditions, guard queries) for read dependencies — supplied with the commit request or declared durably as :db.saga/guard step metadata along the way, with commit evaluating the union. The registry flip to :committed rides in the same transaction — atomicity is structural, retries are idempotent. Effects are replayed, not inputs: tx functions are never silently re-evaluated against state the owner didn't observe; drift fails loudly with an EDN conflict report, and a retry may carry per-conflict resolutions fenced to that report — accept-parent for any conflict class, override only for cardinality-one write–write, where it has an exact observed expansion; uniqueness, dangling-ref, and retraction-miss conflicts are never override-able, because each override would write outside what the owner observed or the saga touched. No splicing of branch transactions into parent history — the parent log records what happened on the parent's timeline; step-level history stays queryable in the branch, retained post-commit under a per-database retention policy (per-saga override at open).
• Visibility by tiers. Unaware readers see canonical facts only and pay nothing. Registry-aware readers discover in-flight sagas — and advisory footprints — with plain Datalog and watch status transitions in ordinary tx-reports. Branch readers get the branch's full Db value (Datalog, Pull, SQL, time views) without locks, registration, or effect on the saga; their adaptation contract is: everything read from a branch is provisional under that saga id until the registry says :committed, and the merge transaction's saga id maps what they saw onto what landed.
• Expiry is mandatory. Every saga carries an owner-extendable deadline; an operator-service sweep (in-transactor fallback) expires overdue sagas and reclaims branches after a grace window, so abandoned sagas cannot pin t₀-era segments forever. :expired is distinct from :aborted. The retention window runs from the transition that finished the saga — a per-database policy, with a per-saga :db.saga/retain-for override declared at open — and it covers all three terminal states: a committed saga's branch is the step-grain annex, an aborted or expired one's is what the orchestrator reads to find what still needs unwinding outside the database.
• A saga is live only where its branch lives. Registry datoms travel with backup, restore, fork, and replication; branches do not (v1 backups exclude them, forks never copy them). Any database that finds :open registry entries with no branch — a restored parent, a fork taken mid-saga — expires them on first open, whether or not the periodic sweep is enabled. So that absence means that and not "not stepped yet", a branch's durable existence begins with its registry entry: the transaction opening a saga writes the branch's metadata root, and the overlay itself is still built on demand — but only ever from a root already written, since building one on demand would forge the invariant's own signal. An abort or expiry reaching an inherited entry before the pass does skips the compensation for the same reason.
• Compensation is explicit, and split by boundary. Abort's default stays total — one status transaction, branch discarded. A saga may register a compensating transaction (static EDN tx data, or a :db/fn invoked with the parent's current value plus the branch value): fresh tx data validated like any transaction and applied atomically with the abort/expiry flip, saga id on the transaction entity — a deliberately authored, user-facing failure record, never a partial landing of branch novelty. Branch facts are inputs to it; its granted ids are refused as outputs; a filtered (subset) merge was considered and rejected. Registration is durable registry data, so expiry applies it for a crashed owner; liveness outranks — a compensation failing at expiry is recorded (:db.saga/on-abort-error) and the saga expires without it, and a branchless expiry (restore, fork) skips it so diverged timelines never double-apply a failure record. The transactor, not the client, composes that transaction, because the function form is invoked with two database values and no client holds the second.
• External effects stay a layer above, with a durable ledger. The branch makes the database side atomic; workflows with outside side effects use the registry as durable orchestrator state and the retained branch as the record of what needs compensating — classic saga orchestration over an atomic core, not instead of one. Reverse progress gets the forward treatment: an external-compensation ledger (:db.saga/compensations component entities — key, status, detail, completed-at, error) written by the orchestrator as ordinary parent transactions and never executed by the engine, surviving branch deletion, seedable atomically at abort by the compensating transaction. No new saga status: :aborted/:expired stay terminal, and "fully compensated" is derived from the ledger.

V1 excludes: read-set tracking (serializability beyond write-write is opt-in via guards), nested and cross-database sagas, base refresh and rebase-commit modes, abort-time subset merges and any engine execution of external compensations, and any pgwire BEGIN mapping.

Consequences

• Parent readers can never observe state that later rolls back — abort needs no compensating retractions in the database (a registered compensation records; it never undoes, because nothing landed) and no reader ever adapts to a retraction it didn't opt into. The cost is that partial progress is only visible to those who ask, which is the requirement, but means tier-0 writers can race an in-flight saga; the conflict scan at merge, not any reservation, is what protects the saga, so long sagas over hot entities will see conflict reports and must resolve or re-do work.
• The transactor grows real machinery: branch bookkeeping, per-branch pipelines, id-block leasing in the allocator, and the merge scan — O(branch novelty + parent novelty since t₀) inside the writer path. Merge of a large saga is an observable pause for that database, like index activation in ADR-0020; the conflict scan bounds it, splicing was rejected partly to keep it one append. Steps never queue in the parent's writer, but a chatty saga still shares the transactor process's CPU, I/O, and cache for its whole lifetime.
• A parent schema migration that invalidates branch novelty strands an open saga through no fault of its own: merge validates against current schema and fails loudly, and with no base refresh in v1 the recourse is salvage-and-redo. This is the sharpest edge among the v1 limits; migration planning should report open sagas as advisory impact.
• An open branch pins parent segments at t₀ and holds id grants; GC pressure and grant consumption are bounded by mandatory expiry rather than by trusting owners. The bound is only as good as the sweep, so a node with sweeping disabled owes the duty to an operator; and because the window is measured from the finishing transition rather than from the deadline, a saga that expires late still gets the full grace period its owner would have had.
• Tier-2 reads cost at most three merge layers — published parent root ≤ t₀, the frozen parent gap (index-basis, t₀], the branch tail — never N + 1: no view unions branches, and branch index publication (the existing indexing job) collapses long-lived branches back to ordinary two-layer reads. Tier-0/1 reads are unchanged at any N.
• Entity-level reservations are registry datoms in the parent, so they price themselves: fine for the tens-of-entities workflow sagas they serve, wrong for bulk work, which reserves attributes instead. The explicit-id loophole also closes: while grants are live, the parent's writer refuses transactions naming ids inside granted blocks — allocator integrity, not a lock.
• Squashing trades parent-log granularity for timeline honesty: parent history shows one labeled commit; auditors needing step grain must consult the retained branch, and retention policy becomes an operational knob with storage cost.
• Effects-replay semantics mean a merge can commit a value computed from stale reads if the owner declared no guard — the same explicit-optimism trade ADR-0015 and ADR-0020 already chose; the remedy is guards, and the failure mode is a loud conflict report, never a quiet recompute.
• An aborted saga is no longer necessarily markless: a registered compensation writes deliberate, saga-labeled facts into canonical state in the abort transaction itself. The atomicity promise is unchanged — saga novelty never half-lands — but auditors should know an abort or expiry transaction can carry user-facing data, and that expiry runs compensations with no owner present, authorized as the recorded owner against then-current policy (drift fails it into :db.saga/on-abort-error rather than blocking expiry).
• The compensation ledger gives orchestrators standard reverse-progress bookkeeping in the parent — resumable from data, surviving branch deletion, seedable atomically at abort — at the price of more reserved vocabulary and one more SQL surface (corium_sys.saga_compensations); the engine still never executes an external compensation, and retention may end early for a fully-resolved ledger but is never extended by a pending one.
• Compensating functions made the :db/fn runtime multi-database: a token now names which database in scope it stands for, and the expander takes a set rather than one value. That is a small widening of a deliberately narrow sandbox surface — the databases remain read-only, a token naming one out of scope is an error rather than a fall back to the primary, and no existing function sees any difference — but it is the first place a :db/fn reads outside the value its transaction is being prepared against, and future sandbox work has to keep that in view.
• Every surface grows a saga face: bootstrap vocabulary, peer API, protocol, CLI, console, a corium_sys.sagas SQL relation, authz applied to branch views (ADR-0021 unchanged), operator-service sweep job. The registry-first delivery order keeps each phase shippable and the data plane free of operator-service dependence.
• Fork remains what it was — an independent database with an independent life; sagas do not replace it, and with_transaction remains the zero-cost speculative tool for single-process what-ifs.