The foerster ↔ raster block contract

Status: draft 0, agreed in outline between the inference and raster sides (2026-09-28; the inference side was spindel's, and is now foerster, on spindel's worlds). Changes go through this file.

Division of labour

foerster owns everything about which variables exist and how they are explored: addresses and traces, the random streams, proposals, acceptance, particles and forked worlds, and the statistical meaning of every estimate.

Raster owns fixed-shape numerical code: evaluating a block, its log density and derivatives, compilation, caches, buffers and device sessions.

This bridge adapts one to the other. It must not add a compiler cache, an argument binder or a device-session convention of its own. It consumes Raster's public compilation API (raster.compiler.pipeline/compile-aot) and reverse-mode API. The caller supplies the existing argument binding. Typed wrappers may adapt the returned value/gradient representation; differentiation, optimization and executable generation remain Raster's responsibility.

A block

A block is a fixed-shape numerical fragment of a model: a group of latent variables together with the density factors they touch. foerster treats a block as one choice site whose value is the block's latents.

A block is a description plus capabilities. The description is data:

{:block/id        :logreg
 :block/latents   [{:name :beta  :shape [3] :dtype :f64
                    :support :real}                 ; or :positive, [:interval a b], :simplex …
                   {:name :sigma :shape []  :dtype :f64
                    :support :positive :transform :log}]
 :block/layout    {:order [:beta :sigma]            ; θ = concat, row-major
                   :coordinates :unconstrained}     ; θ lives in R^n
 :block/inputs    [{:name :X :shape [:n 3]} {:name :y :shape [:n]}]  ; not differentiated
 :block/target    :complete-conditional             ; see below
 :block/capabilities #{:log-density :value+grad}}

Capabilities are functions over primitive arrays. Required:

capability	signature	meaning
:log-density	(f ^doubles θ inputs) → double	log target at θ, unconstrained coordinates, change-of-variable terms included
:value+grad	(f ^doubles θ inputs) → [double ^doubles g]	the same value and ∂/∂θ; inputs get no gradient

Optional, each with an explicit domain and convention in the description: :jvp, :vjp, :hvp, :constrain (θ → latent values, for the trace), :unconstrain, :inverse, :log-abs-det-jacobian, :eval (forward simulation), :batch (a leading particle dimension), and for drawing a block:

capability	signature	meaning
:sample	(f inputs) → ^doubles θ	a draw of θ, not necessarily from the target (whose normalizer is unknown)
:sample-log-density	(f ^doubles θ inputs) → double	the log density of what :sample draws; required whenever a block is drawn from — importance sampling, SMC, single-site MH proposals — since the draw is weighed by log target(θ) − log sample-density(θ)

A block without :sample cannot be drawn from and starts at an :init.

A capability a block does not declare is absent: foerster never falls back to finite differences behind a caller's back. An undeclared derivative rejects the program.

The target

:complete-conditional means the log density includes every factor the block's latents affect: their priors and all downstream observations and latents that depend on them, with every variable outside the block held fixed as an input. A site's own prior alone is not enough for HMC.

foerster keeps HMC exact even when a block's target is incomplete. The leapfrog map is volume preserving and reversible for any position-only force field, and foerster computes the acceptance ratio on the full trace log joint, by replay. An incomplete target therefore costs efficiency, never correctness. foerster reports it: when a replay changes the log probability of a site outside the block, the step records a diagnostic.

Parameter layout

θ is a flat double[] in the declared order, in unconstrained coordinates. Every constrained latent names its transform. The block's log density includes the log-absolute-Jacobian of each transform. The trace records both θ and the constrained values: sites downstream see the constrained values.

Randomness

Every draw a seeded run makes depends only on what is drawn, never on when:

stream(world, key) = generator seeded by hash(seed(world), key)

• Run seed: a session's root seed, drawn in program order from the seeded process generator.
• Stable site identity: the site's structural address. World seeds derive per fork from (parent seed, address, fork index).
• Draw counter: the position within that stream.
• Replay versus fresh proposal: a kept or constrained site draws nothing. A proposal draws in the proposal's world, which has a fresh seed.

Raster randomness inside a block (e.g. par/splitmix64) takes its seed from the stream of the site that runs the block. Bit-identical floating point across devices is not part of the contract; reproducibility is per device and backend.

Residuals and lifetime

Draft 0 is synchronous: capabilities are pure functions and retain nothing between calls. When residuals arrive (a tape for vjp, a device buffer), they are owned by the call that made them and released when the world that holds them (a spindel world) is released. That world is a savepoint anchor, a trace or a particle. Asynchronous completion goes through one shared completion service that resumes the waiting spin on its executor.

Verification

Fast and deterministic, in every test run:

• The Gaussian block against its analytic posterior, from a pure-Clojure reference block and from raster.
• Finite differences against :value+grad at random points, with a stated tolerance.
• For derivative capabilities, the adjoint identity ⟨Jv, w⟩ = ⟨v, Jᵀw⟩.
• Logistic regression (non-conjugate) against a long reference run.

Statistical, as a separate seeded experiment with explicit thresholds:

• Simulation-based calibration. Rank statistics are tested against uniformity at a stated significance level and sample size. Finite samples never produce perfectly uniform ranks.

Milestones

1. Pure-Clojure reference block + HMC-within-Gibbs (foerster.block, foerster.hmc); the Gaussian oracle. Done. The logistic-regression oracle (non-conjugate, against grid quadrature): done.
2. Raster block from deftm log densities (foerster-raster.block), with gradients with respect to θ only. Done.
3. Batched particles (:batch) through raster dimensions; worlds fork only where structure diverges.
4. Residuals, asynchronous completion, device execution.
5. Implicit solves; estimator-based inference (ADEV, differentiable SMC) against enumerable and analytic oracles.

Vector-parameter JVM-AOT adapter

org.replikativ.foerster-raster.gradient/compile-gradient accepts a density Var whose first argument is a flat (Array double) parameter vector, plus a source descriptor {:arguments [theta :- (Array double) ...] :dimension d}. It returns an invoked AOT callable with the same arguments and a double[] result containing value followed by d natural-coordinate derivatives. Inputs after theta are not differentiated. Compiler options pass to Raster without suppressing its guards.

An optional :scalar-source {:args [...] :body form} is an equivalent density with d Double arguments followed by the same data arguments. The adapter checks that signature, chooses scalar AD only within the qualified 18-argument boundary, and preserves the vector-parameter caller interface. The caller is responsible for mathematical equivalence; independent numerical tests must establish it. This explicit source alternative is a temporary representation workaround, not automatic compiler scalarization. No application source tree is rewritten.

aot-block accepts the description, density Var, the same {:args binder} convention as raster-block, and the compiled callable. It checks the returned gradient length and exposes the required block capabilities. The declared coordinate convention controls Foerster's transformations and Jacobians; the adapter adds none itself. The adapter owns no cache and captures no observations. Callers own structural reuse and bound generated definitions/worker lifetime.

This API is qualified on Raster 0.2.1674. A newer Raster qualification or removal of the explicit scalar source requires separate evidence.

raster-block also accepts :compile? in its binding map. Its default true retains automatic compilation through Raster's runtime AD API, including that API's documented fallback. False explicitly selects evaluated runtime AD. A consumer that labels a backend :runtime must request false; use the explicit AOT adapter when successful compilation is a requirement rather than a preference.