pretrained-rstr

⚠️ Experimental: pretrained-rstr is under active development. APIs may change before 1.0. Feedback welcome!

Run pretrained HuggingFace models — text embeddings, speech-to-text, and decoder LLMs — natively on the JVM, on the raster typed-dispatch compiler. No Python, no ONNX runtime: weights load from safetensors, quantize to int8/int4 streams, and run on raster's CPU int8-MAC kernels and GPU-resident Level Zero/OpenCL programs.

(require '[pretrained.embed :as emb]
         '[pretrained.asr :as asr]
         '[pretrained.lm :as lm])

;; every modality is the same shape: (load-X :key [dir] [opts]) then a task verb.
;; a registry key auto-downloads weights from HF on first use; an explicit local dir
;; skips the download (bring your own weights for a known model).

;; embeddings
(def e (emb/load-embedder :qwen3-embedding-0.6b))
(emb/embed-texts e ["Datahike is a durable Datalog database."])
;; => {:data float[n*1024] :n 1 :dim 1024}   (L2-normalized rows)

;; speech-to-text — any audio format (wav pure-JVM; mp3/ogg-opus/m4a via ffmpeg)
(def m (asr/load-asr :moonshine-streaming-medium))
(asr/transcribe m "voice-note.oga")
;; => "And so my fellow Americans, ask not what your country can do for you, ..."
(asr/transcribe m "talk.wav" {:timestamps? true})
;; => {:text "..." :words [{:word "And" :start 0.0 :end 0.02} ...]}

;; decoder LLMs
(def g (lm/load-lm :gemma-3-270m-it))                ;; or (lm/load-lm :gemma-3-270m-it {:gpu? true})
(lm/generate-text g "The capital of France is" 20)
;; => " Paris. ..."

Models

Registry key	Task	Size	Quality (validated)
:qwen3-embedding-0.6b (+-gpu)	embeddings, last-token	0.6B Q8	cos 0.999 vs torch f32
:embeddinggemma-300m	embeddings, bidirectional + mean pool	300M Q8 (GPU)	cos 0.99 vs torch; 768d matryoshka
:all-minilm-l6-v2, :bge-small-en-v1.5	embeddings (BERT tier)	23–33M f32	parity with sentence-transformers
:moonshine-streaming-medium	English ASR, true streaming	245M	WER 1.62% == HF torch (LibriSpeech-100); word timestamps
:qwen3-asr-0.6b / -1.7b	multilingual ASR (52 languages)	0.6/1.7B	transcript char-identical to torch gold
:gemma-3-270m-it / :gemma-3-1b-it	decoder LLM	≤1B Q4/Q8	token-exact GPU decode vs oracle; CPU ≈ llama.cpp speed
:qwen3-0.6b / :qwen3-1.7b, :smollm2-135m-instruct / :smollm2-360m-instruct	decoder LLMs	≤1.7B Q4/Q8	same descriptor-driven engine (shared attention/norm stack)

Embeddings feed directly into proximum (emb/rows → HNSW vector index) and umap-rstr (emb/flat-doubles → 2-D layouts). BERT-family sentence encoders (MiniLM/bge, mean-pool) run self-contained in pretrained.arch.bert — no extra dependency; :engine :encoder registry entries route there automatically.

How it works

A model architecture is a descriptor — a role→tensor-name map plus ~10 flags (norm type/gain, rope variants, GQA, qk-norm, sliding windows, sandwich norms, MoE routing) — interpreted by one generic engine over raster's compilable deftm blocks. Adding a standard decoder-LM is a descriptor, not engine code.

• pretrained.embed / pretrained.asr / pretrained.lm — task-level APIs, all the same shape: a curated registry + HF auto-download + load-X/task-verb (CPU or {:gpu? true})
• pretrained.decoder — the descriptor-driven decode engine (load-hf, decode-step, generate-cached); GPU-resident decode/prefill in pretrained.decoder-gpu
• pretrained.loader — the low-level generic loader: architecture registry, tokenizer auto-detection, from-pretrained (dispatch any HF dir on config.json model_type — the advanced, unvalidated path behind the curated pretrained.lm registry)
• pretrained.arch.* — decoder descriptors as pure data: gemma3, llama, qwen3, qwen3-moe, embedding-gemma, plus the self-contained BERT encoder bert; pretrained.asr.* — moonshine, qwen3-asr
• pretrained.tokenizer.{sp,bpe,wordpiece} — HF tokenizer.json tokenizers
• pretrained.hub — sha-pinned downloads with resume + sha256 into ~/.cache/raster/models (HF_TOKEN honored; every load-* also accepts a local dir)
• pretrained.safetensors / pretrained.audio — format frontends (bf16/f16 fast paths; WAV pure-JVM, other audio via ffmpeg)

Quantized execution: linear weights repack into int8/int4 streams executed by raster's spin-pool int8-MAC kernel (CPU) or GPU dp4a kernels. Q8_0 is measured lossless for embeddings; decode uses Q4. Quantized streams are disk-cached next to the weights — the first load quantizes once (~30s for 0.6B), warm loads take ~5s.

Durable attention-state cache

Decoder inference can checkpoint immutable, prefix-hashed attention-state chunks without blocking the generation thread. Datahike indexes logical identity and placement policy; Konserve/Boring stores contiguous FP32 payloads; restore mmaps one chunk at a time and uploads its slices directly through Raster. The final prompt token remains pending, so restoring a prefix resumes at exactly the same continuation boundary as uninterrupted inference.

(require '[pretrained.continuation.benchmark :as bench]
         '[pretrained.continuation.manager :as cache]
         '[pretrained.model-identity :as identity])

(def fingerprint
  (identity/compatibility-fingerprint
   model {:execution-variant {:backend :ze
                              :linear-weights :q4k
                              :attention-state :float32}}))

(def manager
  (cache/open-manager
   {:store {:backend :memory :id (random-uuid)}
    :schema-flexibility :write :keep-history? false :value-caps :default}
   "/var/tmp/pretrained-kv"
   {:chunk-size 256}))

;; dstate is a bound pretrained.decoder-gpu state; prompt-ids is a token vector.
(bench/benchmark-gpu-prefix! manager dstate fingerprint prompt-ids
                             {:warmups 1 :iterations 5})

The benchmark separates prefill, checkpoint submission/drain, first measured restore, and process/page-cache-warm restore; it does not label warm pages as cold SSD. pretrained.continuation.placement adds declarative per-worker demands and observed replicas. Its Datahike listener only emits lightweight change notifications: a Spindel reaction or bounded worker performs the actual copy, while Yggdrasil can version the catalog through its Datahike adapter.

Validation methodology

Every port is validated against its reference implementation before it ships: layer-by-layer activation comparison vs HF transformers golds (typical agreement ~1e-6 relative in f32), then end-to-end anchors — token-exact decode, character-exact transcripts, cos ≥ 0.999 embeddings vs torch f32. The anchors are repeatable tests:

clojure -M:test        # fast, model-free unit tests
# with local weights (see test/pretrained/anchors_test.clj):
clojure -A:dev:test:valhalla -M -e "(require 'clojure.test 'pretrained.anchors-test) \
  (clojure.test/run-tests 'pretrained.anchors-test)"

Requirements

• JDK 21+ (Panama FFI); the raster Valhalla toolchain for full performance (see raster's README for the JVM flags)
• OpenBLAS for the f32 GEMM paths
• ffmpeg (optional, for non-WAV audio)
• GPU (optional): Level Zero for Intel, or a compatible OpenCL ICD for Intel/NVIDIA/AMD

License

Copyright © 2026 Christian Weilbach

pretrained-rstr is MIT licensed. Model weights carry their own licenses (all registry models are Apache-2.0/MIT) — you are responsible for complying with each model's terms.