fractal-engine Architecture

This document is a public maintainer and contributor map for the runtime. It describes the current engine architecture and the thin agent control plane around it.

The engine is a layered Clojure runtime where a session owns a persistent SCI context, a model produces fenced Clojure, the host evaluates those blocks in a capability sandbox, and the loop continues until a FINAL value or a terminal condition. Recursive calls reuse the same session and storage model rather than adding a second execution system.

Layer Map

flowchart TB
  subgraph Public["Public surfaces"]
    API["fractal.engine.api"]
    CLI["fractal.engine.cli"]
  end

  subgraph Composition["Composition roots"]
    SESSION["fractal.engine.session"]
    RECUR["fractal.engine.recursion"]
  end

  subgraph Runtime["Runtime spine"]
    LOOP["fractal.engine.session-loop"]
    KERNEL["fractal.engine.kernel"]
  end

  subgraph Ports["Ports and shared services"]
    STORE["fractal.engine.store"]
    ADAPTER["fractal.engine.adapter"]
    CAP["fractal.engine.capability"]
    SURFACE["fractal.engine.surface"]
    COMPACT["fractal.engine.compaction"]
    LIVE["fractal.engine.live"]
    PIO["fractal.engine.payload-io"]
  end

  subgraph Backends["Backends"]
    MEM["fractal.engine.store.memory"]
    SQLITE["fractal.engine.store.sqlite"]
    BLOB["fractal.engine.store.blobstore"]
    FAKE["fractal.engine.adapter.fake"]
    SDK["fractal.engine.adapter.sdk"]
  end

  subgraph Foundation["Foundations"]
    PAYLOAD["fractal.engine.payload"]
    CACHE["fractal.engine.cache"]
    PROMPT["fractal.engine.prompt"]
    CATALOG["fractal.engine.catalog"]
    CONCUR["fractal.engine.concurrent"]
    TIME["fractal.engine.time"]
  end

  CLI --> API
  API --> SESSION
  SESSION --> LOOP
  SESSION --> RECUR
  LOOP --> KERNEL
  LOOP --> ADAPTER
  LOOP --> STORE
  LOOP --> LIVE
  RECUR --> ADAPTER
  RECUR --> STORE
  SESSION --> SURFACE
  STORE --> MEM
  STORE --> SQLITE
  SQLITE --> BLOB
  ADAPTER --> FAKE
  ADAPTER --> SDK
  STORE --> PIO
  PIO --> PAYLOAD
  SURFACE --> PAYLOAD

The namespace graph is expected to remain acyclic. test/fractal/engine/deps_acyclic_test.clj guards the load-bearing dependency rules: the store port depends only on pure payload helpers, live dispatch does not depend on the store, the adapter port is engine-free, and capability profiles take host function implementations as data.

Responsibilities

Namespace	Responsibility
fractal.engine.api	Supported Clojure API for config, lifecycle, reads, payload hydration, live subscriptions, and the fake responder helper.
fractal.engine.cli	Agent-operable command seam over the public API. It handles CLI config files, JSON/EDN output, usage commands, and inspection commands without adding runtime semantics.
fractal.engine.session	Sole composition root. It builds stores and adapters, resolves config, creates and resumes sessions, owns the turn lock, initializes SCI contexts, and spawns child or attached sessions.
fractal.engine.session-loop	Turn and step spine: open turns, assemble requests, call adapters under deadline, append assistant/observation messages, finalize turns, and publish heads after successful FINAL.
fractal.engine.kernel	SCI evaluation mechanics, block extraction, eval records, FINAL, inspect, vars snapshotting, and vars restore.
fractal.engine.recursion	Leaf calls, child calls, attached-child calls, fan-out behavior, recursive envelopes, and lineage-edge recording. It receives spawn/run/stop closures from session instead of requiring session directly.
fractal.engine.store	SessionStore protocol, pure event fold, event taxonomy, head helpers, lineage-edge helpers, and payload-ref auditing.
fractal.engine.store.sqlite	Durable SessionStore: SQLite event storage plus a global BlobStore payload backend.
fractal.engine.store.memory	In-process SessionStore with the same semantic contract, primarily for tests and ephemeral runs.
fractal.engine.store.blobstore	File-backed content-addressed payload storage for SQLite durability.
fractal.engine.payload	Pure canonical EDN bytes, SHA-256 content ids, and tagged payload refs.
fractal.engine.payload-io	Store-coupled inline-vs-ref policy and payload hydration.
fractal.engine.adapter	Provider adapter port and call-record shape.
fractal.engine.adapter.request	Request assembly from the folded view, compaction boundary, system overlays, and cache metadata.
fractal.engine.capability	Capability profile lattice, clamp/validation rules, and SCI options.
fractal.engine.surface	SDK surface descriptor validation, public stamps, stable/dynamic prompt rendering, and namespaced SCI function assembly.
fractal.engine.compaction	Context assessment and transcript compaction.
fractal.engine.live	Per-session live dispatch, transient notifications, backlog recovery, and progress projection.
fractal.engine.concurrent	Deadlines, bounded fan-out workers, dynamic-binding propagation, and the global leaf-call semaphore.

No namespace outside the store should invent a durable write path. Runtime code appends facts through the SessionStore port and reads runtime state through the folded view.

Core Ontology

Term	Meaning
Session	Durable identity boundary for metadata, event history, counters, SCI context, heads, and lineage.
Turn	One user message processed until FINAL or a terminal non-final status.
Step	One adapter-call iteration inside a turn.
Eval	One fenced Clojure block evaluated by the SCI kernel.
Message	Transcript entry with role :user, :assistant, or :observation; system text is assembled for requests.
Event	Durable appended fact. Folding events produces the session view.
View	Pure projection over events. It is not a second persistence authority.
Payload ref	Tagged content-addressed reference to a value stored in the payload backend.
Head	Immutable content-addressed continuation boundary for a session.
Current head	Mutable pointer to the session head that resume and attach should advance from.
Lineage edge	Durable content-addressed relation between source/target sessions and heads.
SDK surface	Embedder-provided namespaced host functions plus prompt metadata and public resume stamps.

Filesystem locations, including :store/dir, are physical backend locations. They are not logical session identity. Identity is carried by session ids, payload ids, head ids, event ids, and edge ids.

Normal Turn Flow

sequenceDiagram
  participant Caller
  participant Session
  participant EngineLoop
  participant Adapter
  participant Kernel
  participant Store

  Caller->>Session: run-turn!
  Session->>Store: turn started and user message
  Session->>EngineLoop: run step loop
  EngineLoop->>Adapter: provider request
  Adapter-->>EngineLoop: assistant message
  EngineLoop->>Kernel: evaluate fenced Clojure
  Kernel-->>EngineLoop: eval records and optional FINAL
  EngineLoop->>Store: assistant, evals, observation
  alt FINAL
    EngineLoop->>Store: vars snapshot, terminal turn, head
    EngineLoop-->>Caller: final TurnResult
  else more work
    EngineLoop->>Adapter: request with observation
  end

1. session/run-turn! checks stop status and :max-turns, acquires the turn lock, optionally compacts, appends the user message, appends :turn/started, and delegates to session-loop.
2. session-loop/run-step! appends :step/started before provider work so live observers can see in-flight progress.
3. adapter.request/build-request reads the current folded view, selects kept messages after compaction, hydrates payload refs, maps observations to adapter-facing user messages, assembles system text, inserts any transient dynamic SDK surface request context, and attaches cache metadata.
4. The adapter is called through the LlmAdapter port under one deadline. Streaming token fragments, when enabled, are transient live items only.
5. The loop appends the assistant message and :step/put.
6. The kernel extracts fenced Clojure blocks, evaluates them in the session SCI context, and appends one :eval/added event per block.
7. The loop appends one observation message derived from the eval records.
8. If no FINAL was called, the loop continues to another step.
9. If FINAL was called, the loop snapshots vars, appends :session/vars-snapshotted, appends the terminal :turn/put, publishes a :turn-final head, hydrates the final value, and returns a final turn result.
10. If timeout, provider failure, stop, context hard limit, or max steps occurs, the loop appends a terminal :turn/put with a non-final status and returns a non-final turn result. Non-final terminal paths do not publish heads.

Recursive Flow

The recursive harness adds model-call host functions inside the SCI context. They are not top-level API functions.

flowchart LR
  ROOT["Root session"] --> EXACT["Clojure exact work"]
  ROOT --> LEAF["lm / map-lm leaf call"]
  ROOT --> CHILD["rlm / map-rlm child session"]
  ROOT --> ATTACH["attach-rlm derived child"]
  CHILD --> CHILDHEAD["Child FINAL head"]
  ATTACH --> DERIVEDHEAD["Derived child FINAL head"]
  ROOT --> ROOTHEAD["Root current head"]
  ROOTHEAD --> EDGE1["invocation edge"]
  CHILDHEAD --> EDGE1
  ROOTHEAD --> EDGE2["derivation edge"]
  DERIVEDHEAD --> EDGE2

• lm and map-lm are leaf calls. They make bounded adapter calls and do not create sessions, heads, or lineage edges. map-lm preserves input order and returns per-slot failure sentinels instead of throwing the whole fan-out on a partial failure.
• rlm and map-rlm create fresh child sessions in the same store. Each child has its own SCI context, cache id, capability-clamped profile, loop, events, and heads. After the child reaches FINAL, the parent receives an envelope and records a :invocation edge to the child's current head.
• attach-rlm resolves a selected source session/head, creates a fresh attached child, restores that child from the source head's vars snapshot, runs one task, returns the usual envelope, and records a :derivation edge. The source session and selected source head are not advanced.

Recursive children are ordinary sessions. Storage, resume, live query, heads, and payload refs use the same mechanisms as root sessions.

Capability Sandbox And Runtime Governor

The engine has an in-process SCI capability sandbox. A session starts from a validated capability profile:

• :locked-down denies filesystem, shell, network, Java interop, and recursive model egress; only FINAL and inspect are injected.
• :default allows read-only access inside the work area plus a small read-only shell command allowlist; writes, network, Java interop, and dangerous classes remain denied.
• :trusted is for intentional local trusted use where the caller allows work-area writes, shell, and network.

Profiles form a lattice. Child sessions and per-session overrides inherit through capability/clamp, so an override can narrow a parent but cannot widen it. fractal.engine.capability/sci-opts injects the permitted engine functions and gated IO functions into SCI, installs a finite namespace/class catalog, and keeps the deny set active for escape forms such as eval, resolve, load-string, and load-file.

SDK surfaces add another finite gate: :surface/fns. Configured surface functions appear only as qualified namespace calls when their symbol is allowed. They are not injected into clojure.core, and child sessions inherit them by set intersection.

The runtime governor is config-driven:

• :max-steps bounds the number of provider/eval iterations in one turn.
• :max-turns bounds how many turns a session can open.
• :call-timeout-ms wraps each adapter call, including SDK retry/backoff.
• :max-fanout rejects over-wide map-lm / map-rlm inputs.
• :fanout-pool bounds worker threads for one fan-out.
• :leaf-concurrency is a process-wide semaphore for leaf calls.
• :context {:hard-at ...} stops turns before exceeding the hard context window.

When these controls fire, the runtime returns typed outcomes (:timeout, :budget-exceeded, or :error) and persists the terminal turn fact when a turn has opened. Cost and usage are recorded when the provider reports them, but those records are observability metadata. The governor is the execution-control layer.

Storage Authority

For durable v1 storage, SQLite plus BlobStore are canonical:

• SQLite stores session rows and per-session event rows.
• BlobStore stores payload bytes by content hash.
• The current view is a pure fold/projection over events.
• current-head is the authoritative continuation pointer.
• MemoryStore implements the same port for test and in-process use, but it is not the durable authority.

See STORAGE_AND_HEADS.md for the storage and head model in detail.

High-Level Event Types

The folded view is advanced by a small event taxonomy:

Group	Event types
Session lifecycle	:session/started, :session/stop-requested, :session/stopped, :session/error
Turn lifecycle	:turn/started, :turn/put
Step lifecycle	:step/started, :step/put
Transcript and evals	:message/appended, :eval/added
Snapshots and compaction	:session/vars-snapshotted, :session/compacted
Heads and lineage	:head/published, :lineage/edge-added
Live transient only	:delta/token, :subscribe/gap

Only durable events receive :event/id and fold into the view. Transient live items are never persisted and are recoverable only through later durable state.

Failure Semantics

• Store writes are single-writer per session under the store lock.
• SQLite persists before fold. A failed durable insert does not advance the in-process view.
• SQLite batch appends are one transaction. A failed batch folds nothing.
• Head publication validates an optional expected basis and fails with a typed CAS error instead of silently replacing a changed current head.
• Provider timeout, provider failure, eval error, stop, and budget exhaustion settle the turn as non-final terminal results. They do not publish heads.
• Failed evals are recorded as :eval/added with an error map and are returned to the model as observations when the turn can continue.
• Live delivery is off-lock. Slow or throwing subscribers cannot stall writes.
• Live queue overflow may drop transient items or skip push delivery, but it does not remove durable events from the log. Subscribers recover through events-since.
• Same-session writes from a subscriber callback are rejected as reentrant.

Contributor Invariants

1. Keep fractal.engine.api thin. New runtime behavior should be composed below the API unless a supported surface change is intentional.
2. Keep the loop on ports. session-loop should not know SQLite, BlobStore, or a provider implementation.
3. Keep SDK surfaces as declared host-function namespaces. The engine owns injection, prompt truth, capability gating, cache placement, and stamps; embedders own effects.
4. Keep apply-event pure. Views are projections from events.
5. Persist payloads before appending events that reference them. Orphan payloads are acceptable; dangling refs are not.
6. Treat current-head as the only mutable continuation pointer.
7. Publish a head after every successful FINAL turn and after compaction.
8. Do not restore by replaying provider calls or evals. Restore from stored snapshots referenced by heads.
9. Keep attach additive. attach-rlm must create a fresh derived child and leave the selected source unchanged.
10. Keep lineage durable and content-addressed. Do not infer invocation or derivation edges from naming conventions or backend paths.
11. Treat filesystem paths as backend configuration only, never as session identity.

Verification Pointers

The architecture above is covered by:

• src/fractal/engine/session.clj
• src/fractal/engine/session_loop.clj
• src/fractal/engine/kernel.clj
• src/fractal/engine/recursion.clj
• src/fractal/engine/store.clj
• src/fractal/engine/store/sqlite.clj
• src/fractal/engine/store/blobstore.clj
• test/fractal/engine/store_contract_test.clj
• test/fractal/engine/store_sqlite_test.clj
• test/fractal/engine/session_test.clj
• test/fractal/engine/recursion_test.clj
• test/fractal/engine/surface_test.clj
• test/fractal/engine/surface_session_test.clj
• test/fractal/engine/deps_acyclic_test.clj