pretrained-rstr

⚠️ Experimental: pretrained-rstr is under active development. APIs may change before 1.0. Feedback welcome!

Run pretrained HuggingFace models — text embeddings, speech-to-text, and decoder LLMs — natively on the JVM, on the raster typed-dispatch compiler. No Python, no ONNX runtime: weights load from safetensors, quantize to int8/int4 streams, and run on raster's CPU int8-MAC kernels and GPU-resident Level Zero/OpenCL programs.

(require '[pretrained.embed :as emb]
         '[pretrained.asr :as asr]
         '[pretrained.lm :as lm])

;; every modality is the same shape: (load-X :key [dir] [opts]) then a task verb.
;; a registry key auto-downloads weights from HF on first use; an explicit local dir
;; skips the download (bring your own weights for a known model).

;; embeddings
(def e (emb/load-embedder :qwen3-embedding-0.6b))
(emb/embed-texts e ["Datahike is a durable Datalog database."])
;; => {:data float[n*1024] :n 1 :dim 1024}   (L2-normalized rows)

;; speech-to-text — any audio format (wav pure-JVM; mp3/ogg-opus/m4a via ffmpeg)
(def m (asr/load-asr :moonshine-streaming-medium))
(asr/transcribe m "voice-note.oga")
;; => "And so my fellow Americans, ask not what your country can do for you, ..."
(asr/transcribe m "talk.wav" {:timestamps? true})
;; => {:text "..." :words [{:word "And" :start 0.0 :end 0.02} ...]}

;; decoder LLMs
(def g (lm/load-lm :gemma-3-270m-it))                ;; or (lm/load-lm :gemma-3-270m-it {:gpu? true})
(lm/generate-text g "The capital of France is" 20)
;; => " Paris. ..."

Models

Registry key	Task	Size	Quality (validated)
:qwen3-embedding-0.6b (+-gpu)	embeddings, last-token	0.6B Q8	cos 0.999 vs torch f32
:embeddinggemma-300m	embeddings, bidirectional + mean pool	300M Q8 (GPU)	cos 0.99 vs torch; 768d matryoshka
:all-minilm-l6-v2, :bge-small-en-v1.5	embeddings (BERT tier)	23–33M f32	parity with sentence-transformers
:moonshine-streaming-medium	English ASR, true streaming	245M	WER 1.62% == HF torch (LibriSpeech-100); word timestamps
:qwen3-asr-0.6b / -1.7b	multilingual ASR (52 languages)	0.6/1.7B	transcript char-identical to torch gold
:gemma-3-270m-it / :gemma-3-1b-it	decoder LLM	≤1B Q4/Q8	token-exact GPU decode vs oracle; CPU ≈ llama.cpp speed
:qwen3-0.6b / :qwen3-1.7b, :smollm2-135m-instruct / :smollm2-360m-instruct	decoder LLMs	≤1.7B Q4/Q8	same descriptor-driven engine (shared attention/norm stack)

Embeddings feed directly into proximum (emb/rows → HNSW vector index) and umap-rstr (emb/flat-doubles → 2-D layouts). BERT-family sentence encoders (MiniLM/bge, mean-pool) run self-contained in pretrained.arch.bert — no extra dependency; :engine :encoder registry entries route there automatically.

How it works

A model architecture is a descriptor — a role→tensor-name map plus ~10 flags (norm type/gain, rope variants, GQA, qk-norm, sliding windows, sandwich norms, MoE routing) — interpreted by one generic engine over raster's compilable deftm blocks. Adding a standard decoder-LM is a descriptor, not engine code.

• pretrained.embed / pretrained.asr / pretrained.lm — task-level APIs, all the same shape: a curated registry + HF auto-download + load-X/task-verb (CPU or {:gpu? true})
• pretrained.decoder — the descriptor-driven decode engine (load-hf, decode-step, generate-cached); GPU-resident decode/prefill in pretrained.decoder-gpu
• pretrained.loader — the low-level generic loader: architecture registry, tokenizer auto-detection, from-pretrained (dispatch any HF dir on config.json model_type — the advanced, unvalidated path behind the curated pretrained.lm registry)
• pretrained.arch.* — decoder descriptors as pure data: gemma3, llama, qwen3, qwen3-moe, embedding-gemma, plus the self-contained BERT encoder bert; pretrained.asr.* — moonshine, qwen3-asr
• pretrained.tokenizer.{sp,bpe,wordpiece} — HF tokenizer.json tokenizers
• pretrained.hub — sha-pinned downloads with resume + sha256 into ~/.cache/raster/models (HF_TOKEN honored; every load-* also accepts a local dir)
• pretrained.safetensors / pretrained.audio — format frontends (bf16/f16 fast paths; WAV pure-JVM, other audio via ffmpeg)

Quantized execution: linear weights repack into int8/int4 streams executed by raster's spin-pool int8-MAC kernel (CPU) or GPU dp4a kernels. Q8_0 is measured lossless for embeddings; decode uses Q4. Quantized streams are disk-cached next to the weights — the first load quantizes once (~30s for 0.6B), warm loads take ~5s.

Durable attention-state cache

Decoder inference can checkpoint immutable, prefix-hashed attention-state chunks without blocking the generation thread. Datahike indexes logical identity and placement policy; Konserve/Boring stores contiguous FP32 payloads; restore mmaps one chunk at a time and uploads its slices directly through Raster. The final prompt token remains pending, so restoring a prefix resumes at exactly the same continuation boundary as uninterrupted inference.

(require '[pretrained.continuation.benchmark :as bench]
         '[pretrained.continuation.manager :as cache]
         '[pretrained.model-identity :as identity])

(def fingerprint
  (identity/compatibility-fingerprint
   model {:execution-variant {:backend :ze
                              :linear-weights :q4k
                              :attention-state :float32}}))

(def manager
  (cache/open-manager
   {:store {:backend :memory :id (random-uuid)}
    :schema-flexibility :write :keep-history? false :value-caps :default}
   "/var/tmp/pretrained-kv"
   {:chunk-size 256}))

;; dstate is a bound pretrained.decoder-gpu state; prompt-ids is a token vector.
(bench/benchmark-gpu-prefix! manager dstate fingerprint prompt-ids
                             {:warmups 1 :iterations 5})

The benchmark separates prefill, checkpoint submission/drain, first measured restore, and process/page-cache-warm restore; it does not label warm pages as cold SSD. Chunk identities use Hasch and are published as Datahike :db.type/store-ref values, while the prefix hash separately identifies the causal token chain.

For cluster durability, pass an already connected authoritative Konserve store (for example Konserve-S3) as :chunk-backend-store. The manager owns its local filestore but the caller retains ownership of the backend connection:

(cache/open-manager datahike-config "/var/tmp/pretrained-kv"
                    {:chunk-size 256
                     :chunk-backend-store s3-store})

Chunk checkpoints write the local mmap-compatible frontend first. Their :captured future can therefore complete while the remote copy is in flight; :published completes only after every Konserve write-behind receipt succeeds and the Datahike transaction commits. Datahike never advertises a store-ref that only exists in one worker's cache.

pretrained.continuation.placement records declarative per-worker demands and observed replicas. pretrained.continuation.replica/open-executor connects that control plane to a bounded background copy worker: it moves bytes off-band, verifies the immutable Hasch identity and catalog metadata, then transitions the target replica through copying to ready or failed. Transaction listeners only offer work and never perform I/O. The built-in promoter uses Konserve's tiered store to explicitly synchronize one content key from its authoritative backend into the worker's local filestore frontend. It waits for that write and verifies the frontend before publishing ready; restore then mmaps the concrete frontend, not the tiered wrapper. The current tier sync decodes once during promotion. Remote or raw-copy transports can implement the narrow ReplicaPromoter effect without replacing Konserve's storage API, and Yggdrasil can version the catalog through its Datahike adapter.

The distributed paged inference design separates durable transfer chunks from GPU allocation pages and specifies the path to continuous batching, asynchronous restore/checkpoint streams, and explainable admission, prefetch, and eviction policy.

With MinIO listening on localhost:9000, the optional cluster alias runs a self-cleaning, model-free end-to-end check. It writes through a worker-local filestore to S3, commits catalog and placement facts through a Kabel writer, promotes the chunks to a second worker, mmaps them, and restarts that worker:

;; clojure -M:distributed-demo
(require '[pretrained.distributed-continuation-demo :as distributed])
(distributed/run-minio-smoke!)

open-authority!, open-worker!, checkpoint-gpu-prefix!, and restore-gpu-prefix! expose the same stages for a bound model. The source phase returns a small serializable manifest for the destination phase. Concurrent workers should run in separate JVMs; the smoke turns them over sequentially because distributed-scope intentionally keeps one in-process route per remote peer.

(require '[konserve.tiered :as tiered]
         '[pretrained.continuation.placement :as placement]
         '[pretrained.continuation.replica :as replica])

(def worker-b-tiered
  (tiered/connect-tiered-store
   worker-b-store shared-store
   :write-policy :frontend-only
   :read-policy :frontend-first
   :opts {:sync? true}))

(def executor
  (replica/open-executor
   connection "worker-b" :ssd
   (replica/konserve-tiered-promoter worker-b-tiered)))

(placement/request!
 connection {:model-fingerprint fingerprint :prefix-hash prefix
             :node "worker-b" :tier :ssd :priority 10})

Resident paged attention

pretrained.continuation.page-pool maps the same durable chunks into stable worker-local Raster allocations. A logical page covers every attention-state slab and layer, so full prefix pages can be shared safely while a partial tail uses copy-on-write. Durable chunk size and device page size are independent: a 256-token Konserve object can scatter into sixteen 16-token GPU pages.

pretrained.continuation.paged-attention binds those pools to Raster's semantic AttentionProblem and verified KernelGraph. Fixed-capacity runners update packed query positions and dense page tables between submissions, allowing unrelated continuations to form one batch without changing graph pointers.

pretrained.continuation.paged-append reserves one physical slot per batch lane and binds projected resident FP32 K/V rows directly to Raster 0.2.355's routed FP16 assignment graph. The same reservation batch is reused across layers and becomes visible only after every layer write succeeds. Attention may lease its prospective post-append route and queue behind assignment on the same in-order Raster session without publishing unfinished state.

pretrained.continuation.scheduler plans bounded continuous batches with decode lanes first and chunked repair/prefill work in the remaining capacity. It also chooses between measured resident, restore, recompute, and explicitly enabled approximate-repair paths. pretrained.continuation.residency admits routes under GPU pressure with explainable cost-aware eviction; dirty, pinned, and leased routes are never victims.

(require '[pretrained.attention-state :as attention-state]
         '[pretrained.continuation.manager :as cache]
         '[pretrained.continuation.page-pool :as pages]
         '[pretrained.continuation.paged-attention :as paged-attn])

(def pool
  (pages/open-pool!
   (:sess dstate)
   (attention-state/layout (:model dstate))
   {:page-size 16 :physical-pages 1024 :dtype :half}))

(cache/restore-paged-prefix!
 manager pool :request-a fingerprint prompt-ids
 {:admit? true
  :policy {:durable? true :reuse-probability 0.6
           :recompute-ms 18.0 :reload-ms 5.0 :last-access 42}})

(def runner
  (paged-attn/open-runner!
   pool {:layer 0 :batch-size 8 :total-query-tokens 8
         :q-heads 4 :kv-heads 1 :qk-head-dim 64 :value-head-dim 64
         :pages-per-sequence 128}))

For model execution, :query-view and :output-view may be FP16 Raster ResidentBufferViews owned by adjacent projection and output graphs. The runner then uploads only the small route descriptors and returns the resident output view after completion; query and attention tensors never cross the host.

The current routed attention and append leaves are deliberately simple FP16 correctness references. Both run from caller-owned resident views and expose asynchronous events; no projected K/V or attention tensor crosses the host. Page restoration, append transactions, prospective attention routes, and graph execution are integrated, but the existing contiguous decoder does not yet select this path automatically. The remaining hot-path work is to split the generated layer at its projection/attention boundaries and execute scheduler batches through these runners, followed by optimized backend lowerings.

Validation methodology

Every port is validated against its reference implementation before it ships: layer-by-layer activation comparison vs HF transformers golds (typical agreement ~1e-6 relative in f32), then end-to-end anchors — token-exact decode, character-exact transcripts, cos ≥ 0.999 embeddings vs torch f32. The anchors are repeatable tests:

clojure -M:test        # fast, model-free unit tests
# with local weights (see test/pretrained/anchors_test.clj):
clojure -A:dev:test:valhalla -M -e "(require 'clojure.test 'pretrained.anchors-test) \
  (clojure.test/run-tests 'pretrained.anchors-test)"

Requirements

• JDK 21+ (Panama FFI); the raster Valhalla toolchain for full performance (see raster's README for the JVM flags)
• OpenBLAS for the f32 GEMM paths
• ffmpeg (optional, for non-WAV audio)
• GPU (optional): Level Zero for Intel, or a compatible OpenCL ICD for Intel/NVIDIA/AMD

License

Copyright © 2026 Christian Weilbach

pretrained-rstr is MIT licensed. Model weights carry their own licenses (all registry models are Apache-2.0/MIT) — you are responsible for complying with each model's terms.