Dvergr should measure complete useful work, not only code generation or clean single-turn tool use. The same evaluation boundary must cover a simulated workflow in CI, a frozen integration fixture during development, and a live provider/service probe from the REPL. Only the effect interpreter changes.
The first portfolio follows workflows people run with general personal and business agents such as Hermes:
| Workflow | Observable outcome | Required effects |
|---|---|---|
| Daily operations brief | Accurate, delivered brief with evidence | schedule, intake, search, artifact, delivery |
| Revenue-risk review | Correct account actions and prioritization | CRM, support, calendar, delegation |
| Inbox/support triage | Correct state changes with no missed SLA | mail, ticketing, knowledge, approval |
| Competitive/market report | Supported claims and actionable positioning | web intake, browser, citations, report |
| Campaign operation | Approved variants and measured outcomes | analytics, content, publication, accounting |
| Incident response | Restored service and auditable changes | terminal, monitoring, notification, rollback |
| Document/spreadsheet work | Semantically and visually correct artifact | office tools, files, rendering, review |
Hermes' own documentation emphasizes scheduled briefs and monitoring, cross-platform messaging, browser/terminal work, skills, memory, and delegated research. Its implementation history also exposes useful reliability contracts:
Related references:
These are requirements to express through Dvergr's programming model, not a reason to port another monolithic agent loop.
trigger stream
-> work-admission policy
-> Run in an isolated world
-> capability-scoped effects
-> durable semantic observations
-> trusted verification
-> Attempt / Episode
-> settlement and delivery
A schedule tick, incoming email, user message, webhook, analytics update, or model completion is an event. Spindel programs combine those events and effects; there is no separate cron-agent, browser-agent, or business-workflow runtime. Rooms hold discourse and durable control facts. Runs delimit causal execution. Worlds isolate speculative effects. Kontor accounts for resources. Evaluators certify outcomes independently of agent prose.
Machine, container, VM, managed browser, SaaS simulator, and frozen fixture are world/effect interpreters selected by an EnvironmentDef. Cheap forked state is the default. Stronger substrates are explicit resources used only when the task requires their compatibility or isolation.
An agent building a nested harness needs the same debugger as the host, within
its authority. agent/inspect therefore projects the ambient structural Run
subtree from durable facts. It includes a bounded Run tree, active frontier,
correlated message/activity summaries, failures, settlement, and available
resource balances. It excludes parents, siblings, hidden verifier state, and raw
tool arguments. The host REPL uses the same projection with a wider scope.
This snapshot is the first query. Reactive inspection should later be a Spindel signal folding the same Run/message/resource facts, not a second event log. Useful next fields are context composition/pressure, current effect and elapsed time, world diff/artifact summary, delivery status, retry safety, and the reason a Run is waiting.
Each practical EnvironmentDef should have four modes:
Failures are classified at least as setup/configuration, model/provider, context/tool discovery, orchestration, effect execution, verification, settlement, delivery, or external-system drift. A failed setup must not be reported as deficient model reasoning.
The first environment, :business/renewal-risk-brief-v1, is deliberately small.
It requires a model to construct sales and support specialists, join evidence,
inspect its own execution tree, and produce an exact risk brief while unrelated
private work exists in the same control Room. Its deterministic SCI contract and
trusted model-backed verifier establish the recursive boundary before real CRM,
support, scheduling, or delivery effects are added.
dvergr.agent.arenas.renewal is the first consequential state-backed successor.
Its exact WorldSetup seeds an account and sales/support signals inside every
candidate Run world. A candidate must construct and hire both exact scripted
specialist-service fixtures, obtain their actual child Run results, and call the
semantic renewal_plan tool with the exact evidence IDs.
That tool validates the fork-local business state, consumes one conserved
renewal-review unit from the Run wallet, and writes the proposed intervention
only into the disposable candidate world. The trusted evaluator then checks the
plan, Run topology, causal observation, tool activity, resource receipt, and
returned result before discard settlement. Static prose cannot pass it.
The host composes it from ordinary definitions rather than a special arena scheduler:
(require '[dvergr.agent.arenas.renewal :as renewal]
'[dvergr.agent.evaluation :as evaluation]
'[dvergr.agent.experiment :as experiment])
(renewal/register-tool!)
(renewal/provision-review-capacity!
room {:id provisioning-event-id :amount candidate-cells})
(let [cleanup-group (evaluation/cleanup-group)]
(try
@(experiment/run
room roster experiment-def
{renewal/verifier-ref (renewal/evaluator)}
{:cleanup-group cleanup-group
:world-setups {renewal/setup-ref (renewal/world-setup)}})
(finally
;; Shared-Room callers join only this operation's detached cleanup.
(evaluation/await-cleanups-for! room cleanup-group))))
The deterministic model contract exercises this complete path in CI. Live Codex/Claude candidates can use the same EnvironmentDef, evaluator and setup; only their AgentDefs and prompt policies differ. The scripted children measure recursive construction, causal joining, world/resource ownership, and merge—not specialist model reasoning. A later arena can replace them with paid specialist models once nested provider effects debit conserved Run wallets.
The first native Codex-subscription probe of this arena exposed two benchmark
contract bugs before it passed. The portable task originally said only that a
specialist returned a signal and that the root returned a plan ID, while the
trusted verifier required full signal records and a {:plan/id UUID} result.
Those shapes now live explicitly in the task data rather than only in hidden
host code. The probe also showed that three provider exchanges and 30 seconds
were accidental turn-like limits: Codex was still inspecting the unfamiliar
SCI surface when the first attempt stopped. The environment now uses time and
resource limits as its work budget and retains sixteen model exchanges only as
a runaway fuse. With a 120-second limit, Codex queried the fork-local state,
authored and ran both specialists, submitted the exact evidence, consumed one
review unit, and passed the then-current fifteen trusted checks in eight model
exchanges. The current contract additionally verifies exact child AgentDef
identities; the historical probe has not yet been rerun against that revision.
This is one successful discoverability probe, not yet a model-quality estimate.
The same path is available from a fully initialized room REPL without exposing
the raw trace to evaluated agents:
(require '[dvergr.agent.arenas.renewal :as renewal]
'[dvergr.agent.arenas.renewal-bench :as renewal-bench])
(renewal/register-tool!)
(renewal/provision-review-capacity!
room {:id (random-uuid) :amount 1})
(def report
(renewal-bench/run!
room
{:id :codex-renewal
:provider :codex-subscription
:model "codex-subscription-sol"}))
(get-in report [:scorecard :scorecard/summary])
(:host/tool-trace report) ; host-only generated inputs and semantic outcomes
run! uses the caller's real Room and leaves its lifecycle with the caller. It
requires the semantic tool to be installed explicitly and never mutates the
process-global tool registry itself. It does not manufacture a lighter
pseudo-room or copy the experiment scheduler. Evaluation cleanup is joined
before it returns or throws, and the diagnostic trace is restricted to the
admitted root Run and its structural descendants.
The comparison unit is now explicit. A DatasetDef fixes a non-empty ordered set of exact environments; an ExperimentDef fixes exact candidate AgentDef content and repetitions. Host-owned admission policy caps total attempts and parallelism. Its lazy Spindel execution produces the ordinary certified Attempts and a deterministic Scorecard over the full paired matrix. This supports repeated provider/model comparisons without giving Dvergr a second scheduler or allowing evaluated SCI code to own trusted verifiers. Jobs are realized only one bounded batch at a time, and batch environments initially require discard settlement until partial experiments have durable recoverable identity. Completed Scorecards are immutable durable Room projections over the already-certified Attempts: the exact value lives in the content-addressed artifact store, while Datahike indexes exact experiment, dataset, candidate, and Attempt joins. Mandatory writer validation rejects forged or mutable leaderboard rows. Incomplete batches leave their individual Attempts but never publish a complete Scorecard; durable resumption of the partial matrix remains a later execution policy.
An EnvironmentDef may also name an exact WorldSetup reference. The host resolves that reference before admitting any experiment cell, while the actual preparer runs separately for every cell inside its already-forked Run world. This makes fixture construction part of the reproducible environment identity without exposing live preparers or verifier authority to SCI. Shared substrate provisioning—such as opening a Datahike/Kontor arena or a managed browser—stays an outer resource lifecycle; fork-local scenario state belongs to WorldSetup.
The first provider-free room-REPL probe on 2026-09-04 executed two echo candidates against two environments with two repetitions and parallelism two. It produced eight distinct durable Runs, eight certified Attempts, two 4/4 candidate summaries, one content-addressed Scorecard, and zero active Runs at completion. This validates the experiment composition itself; it is not model quality evidence.
The first generic live migration used the same :programming/join-v1
EnvironmentDef to compare Claude Code and Codex subscription AgentDefs. It
immediately found two observable failures: retries left duplicate child Runs,
and the REPL's process worker inherited Spindel's drain marker, so the documented
@(spin (await ...)) bridge rejected valid composition as a deadlock. Tool
activities now retain typed status and allowlisted diagnostic codes—but never
raw result content—making that class of failure visible in certified evidence
after an ephemeral Room closes. Clearing the drain marker and ambient Spin
identity at the real worker boundary restored the advertised algebra. With the same concise,
content-addressed effect-safety prompt, both providers then passed independently
in three model steps, with exactly two completed children, an exact result, and
zero active Runs. This is one smoke result, not a quality ranking; repetitions
and held-out environments are required for comparative claims.
The first repeated Codex subscription probe then ran
:programming/self-programming-v1 twice through the Experiment API. Each root
used clojure_eval to author a Roster, hired three simulated specialists in
isolated worlds, observed their results through Spindel, and computed the exact
answer. Both Attempts passed all ten trusted checks in four model steps; the two
root Runs and six child Runs quiesced, and the resulting 2/2 Scorecard
round-tripped through the Room store. A deterministic model simulator now
exercises that same complete path in CI—from the native LLM/tool loop through
SCI-authored recursive hires, world settlement, trusted verification, Attempt
certification, and durable Scorecard. Building that contract exposed and fixed
an observation projection typo that had hidden :run/world identities, which
is precisely the kind of harness failure these paired probes are intended to
find.
dvergr.benchmarks.market-evidence measures source faithfulness: classify nine
claims against five frozen public documentation excerpts as supported,
contradicted or unknown. Supported and contradicted claims require an exact,
relevant complete source sentence; unknown claims must not invent evidence.
The host-owned verifier checks the answer without exposing its answer key to
the candidate. Fixture provenance and verifier identity are content-addressed.
With src and dev on the classpath and the chosen provider registered:
(require '[dvergr.agent.market-bench :as market]
'[org.replikativ.spindel.engine.core :as ec])
;; Host REPL: room is caller-owned. evaluate returns a lazy Spindel computation.
(binding [ec/*execution-context* (:ctx room)]
@(market/evaluate room :codex-subscription "codex-subscription-sol"))
The result uses ordinary Run and certified Attempt projections. Join evaluation cleanup before closing the caller-owned Room; an in-memory store is not restart durability. The candidate has no tools, a 180-second deadline, a ten-second cancellation grace and a two-request provider-loop fuse. Its nominal dollar budget is not a token quota for zero-priced subscription models. Track actual usage and elapsed time separately.
Deterministic tests in dvergr.benchmarks.market-evidence-test cover evidence
validation, strict output parsing, scripted certification, persistence and
cleanup. This small fixture is not a general entailment judge, competitor
discovery benchmark, deployment audit or validation of market demand.
Tool-driven sandbox HTTP requests in a Datahike-backed control Room record
Run-correlated acquisition metadata before executing the request and its outcome
afterward. A receipt also identifies the tool call and execution world. Execution
audit stays in the control Room when a speculative work world is discarded.
Receipt reads are host-only and require the caller to authorize Room/Run access;
dvergr.io.acquisition/list-for-run reads a bounded prefix in Run-index order.
An in-memory RoomStore or a direct HTTP call outside a tool invocation does not
automatically acquire this durable audit capability.
Response capture is off by default. A host can opt in on its Room:
(swap! (:meta room) assoc :http-capture
{:allowed-origins #{"https://example.org"} :max-bytes 65536})
This stores eligible credential-scrubbed text responses in the RoomStore's
content-addressed artifact store. By default, payload bytes live in that room
database's Konserve store, addressed by datahike.blob/blob-id. The receipt's
:acquisition/body-store-ref is :db.type/store-ref: Datahike tracks its
reachability through branches and retained history. Responses expose a :dvergr/acquisition
envelope only after outcome persistence succeeds. The metadata records origin,
not raw request paths, queries, headers or request bodies. A deterministic
request-key over pre-injection URL/method/query parameters permits host-side
citation matching; it is not encryption. Captured response text can itself be
sensitive, so enable capture only for sources the host intends to retain.
Redirects are not followed automatically.
:max-bytes limits each stored body, not network bytes, total Run storage or
HTTP transport allocation. The HTTP client still buffers its response; transport
and cumulative resource budgets are separate follow-ups. A request left
:started after interruption has an unknown outcome. Persistence failures do
not retry the HTTP effect, and an outcome-recording failure must not be treated
as proof that the request never happened. Room deletion retracts receipt
projections; Datahike GC can reclaim managed bodies once no retained branch or
history references them, subject to its normal retention and writer-age policy.
Attempts and Scorecards use the same managed publication path through
:attempt/payload-ref and :scorecard/payload-ref. artifact/publish-value!
holds Datahike's canonical-store write guard from the binary write through the
referencing transaction. Its host callback must finish the transaction before
returning; it must not return an unfinished future or Spin. This is a storage
commit boundary, not a new agent scheduling primitive.
Existing string payload-blob / body-ref attributes retain their types; old
Attempt/Scorecard payloads remain readable from the legacy global blob store.
No existing records or historical blobs are rewritten. Explicit legacy artifact
store injection still uses that unmanaged path and does not gain automatic GC.
Failed publications leave collectable orphans: reference-only persistence dead
letters are diagnostic records, not self-contained replay bundles. Recovery
must republish the exact payload before retrying the domain write.
These receipts establish acquisition provenance, not whether a quotation supports a claim. A discovery evaluator must still verify the Run, request fingerprint, captured body and submitted evidence together.
dvergr.benchmarks.discovery-citations/verify provides the first frozen-fixture
check over that boundary. A host supplies an exact reference map and room/Run;
the submitted answer names receipt UUIDs, URLs and complete evidence sentences.
Only successful, captured GET responses from that scope can earn credit.
Partial valid coverage earns partial reward; any invalid extra citation gates
reward to zero. Foreign or nonexistent receipts cannot authorize artifact reads.
The provider-free test uses synthetic businesses, a failed response and an
irrelevant page. It checks real persisted receipts and managed artifact bodies,
not an agent-reported fetch log.
This helper checks exact plaintext fixtures, not arbitrary HTML or semantic entailment. It does not certify Run completion, execute searches, or constitute a live discovery benchmark; those belong to the enclosing Environment/Evaluator workflow. Its reference answers must remain host-owned.
For variable-query experiments, dvergr.benchmarks.frozen-web/transport
compiles immutable {url {:title text :body text}} pages into an offline HTTP
function. The host installs it with
(dvergr.sandbox.ns.io/add-http-ns! sci-ctx :fixture-transport transport).
It accepts the Brave search endpoint and returns its JSON response shape;
ranking uses distinct lexical overlap with stable URL tie-breaking, not Brave's
production ranking. Queries may vary; the corpus and algorithm stay fixed.
Page GETs return exact fixture text. Unsupported options/methods and unknown
URLs return HTTP errors without DNS or live-network fallback.
This namespace-local capability contains no mutable replay cursor. It is safe
to share between interpreter forks. Used alone, newly constructed interpreters
need explicit installation; the WorldSetup integration below supplies it.
Use a fake BRAVE_API_KEY in the fixture's sandbox environment to exercise the
existing intake without credentials. Domain checks still apply. With an active
acquisition scope, each call creates a fresh receipt; successful transport
responses (including HTTP errors) carry a content-derived
:acquisition/fixture-id. Captured bodies still require host capture policy.
Simulated acquisitions must not be represented as live observations or billed
as actual provider usage. The transport itself does not add automatic accounting
charges or model-response replay.
dvergr.benchmarks.discovery composes this substrate into a synthetic discovery
Environment/Evaluator. Give the durable control Room
{:http-capture discovery/capture-policy} in its metadata, then execute
(evaluation/evaluate room team :researcher (discovery/definition) (discovery/evaluator) {:world-setup (discovery/world-setup)}) as an ordinary
Spin. The candidate AgentDef needs :tools #{:clojure_eval}. Bind the room's
Spindel context at the host entry point; compose with await inside Spins.
Trusted WorldSetup installs an immutable offline HTTP capability and dummy service config in Spindel execution-context state, before interpreter creation. New interpreters in descendant context forks inherit it; existing interpreters retain their already-installed transport. Reconstruct capabilities through the content-addressed setup after restart, not by serializing host functions. Fixture mode does not load host service secrets. Installation is host-only, not a new candidate-visible switch for choosing live versus simulated effects.
Candidates must cite one actual search response and page receipts from the same Run. Scoring checks that submitted URLs occurred in those search results, that all receipts belong to this fixture, and that exact fetched text supports the submitted alternatives. Completion and correct setup gate reward; failure, cancellation and waiting earn no credit. This first contract deliberately uses one cited search response, not yet a multi-query causal discovery graph. The three synthetic companies are a plumbing test, not a substantive market benchmark. The provider-free test substitutes only model responses and unrelated workspace bootstrapping: tool execution, SCI, receipts, scoring and settlement remain real. Live candidates can use exactly the same Environment/Evaluator.
Every port should arrive with a task that fails without it, a deterministic contract where possible, and a paired model-backed measurement. Agent-authored tasks and verifiers remain proposals until a trusted parent reviews and admits them.
Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |