Liking cljdoc? Tell your friends :D

com.blockether.vis.internal.speech.assets

Where a speech asset comes from, what it is licensed under, and how it lands on disk.

resources/vis-models/manifest.edn is the only answer to "may we ship this": every model and data directory Vis installs has an entry there with an SPDX id, an attribution line and :is-commercial-ok true. An artifact that is not in the manifest cannot be downloaded, because nothing else carries a URL.

Each entry lists its sources in preference order and every source delivers the SAME bytes, so one :sha256 verifies them all:

:hf Hugging Face. SKIPPED unless a token is configured, tried FIRST when one is. A token is never required — it only changes WHERE the same bytes come from. :pack the shared Vis assets-pack release, the default, so a first run needs no account anywhere. :upstream the project that published the artifact, last, as a fallback.

Sources are tried in that order and the first that installs cleanly wins, so an asset host being down is a slower install rather than a dead feature.

Where a speech asset comes from, what it is licensed under, and how it lands on
disk.

`resources/vis-models/manifest.edn` is the only answer to "may we ship
this": every model and data directory Vis installs has an entry there with an
SPDX id, an attribution line and `:is-commercial-ok true`. An artifact that is
not in the manifest cannot be downloaded, because nothing else carries a URL.

Each entry lists its sources in preference order and every source delivers the
SAME bytes, so one `:sha256` verifies them all:

  :hf        Hugging Face. SKIPPED unless a token is configured, tried FIRST
             when one is. A token is never required — it only changes WHERE
             the same bytes come from.
   :pack      the shared Vis assets-pack release, the default, so a first run
             needs no account anywhere.
  :upstream  the project that published the artifact, last, as a fallback.

Sources are tried in that order and the first that installs cleanly wins, so
an asset host being down is a slower install rather than a dead feature.
raw docstring

com.blockether.vis.internal.speech.attribution

THIRD_PARTY_MODELS.md renders the speech and decision asset manifests, so the credits a reader checks and the pinned downloads stay in step.

markdown is the whole file. assets-test fails when the copy in the tree drifts from it, and vis-agent speech models licenses --markdown reprints it.

`THIRD_PARTY_MODELS.md` renders the speech and decision asset manifests, so the
credits a reader checks and the pinned downloads stay in step.

`markdown` is the whole file. `assets-test` fails when the copy in the tree
drifts from it, and `vis-agent speech models licenses --markdown`
reprints it.
raw docstring

com.blockether.vis.internal.speech.cli

Commands for the gateway-owned speech subsystem. This namespace never loads a model or Sherpa: every runtime operation crosses the canonical gateway client.

Commands for the gateway-owned speech subsystem. This namespace never loads a
model or Sherpa: every runtime operation crosses the canonical gateway client.
raw docstring

com.blockether.vis.internal.speech.engine

The built-in local Parakeet transcription engine used by the gateway.

The built-in local Parakeet transcription engine used by the gateway.
raw docstring

com.blockether.vis.internal.speech.files

Streaming download, checksum and archive helpers for the gateway's speech assets.

Streaming download, checksum and archive helpers for the gateway's speech assets.
raw docstring

com.blockether.vis.internal.speech.qwen

Qwen3-TTS-12Hz-1.7B-CustomVoice text to speech through ONNX Runtime.

:qwen3-tts-local Nine built-in speakers. Each speaker can read ten languages, and the language of the text selects the language prompt.

The bundle is the cpu_int4 ONNX export of speed-brain-ai. Its graphs are:

text_embed text token ids to projected embeddings codec_embed codec token ids to talker embeddings talker_cache the talker with a KV cache: the first codebook of a frame code_predictor_loop the other 15 codebooks of a frame, sampled in the graph residual_embed the 16 codes of a frame to the next talker input tok_decoder frames to 24 kHz audio, 1920 samples for each frame

A voice imported from a recording uses the Base model, the cpu_int4 export of onnx-community. It has its own codec_embed and residual_embed, and these graphs:

speaker_encoder 24 kHz audio to the speaker vector of the codec prompt tok_encoder one second of 24 kHz audio to 13 frames of codes

bin/export-qwen-base makes its other graphs. Its text_embed and talker_cache are the onnx-community graphs with int8 activations, which are several times faster on a CPU. Its code predictor has a KV cache, so each codebook is one step of one row:

code_predictor_step rows and a KV cache to the logits of one codebook code_predictor_embed a code of one codebook to its input row

It uses the tokenizer and the decoder of the speaker bundle.

The prompt and the sampling follow the reference generation of Qwen3-TTS (Qwen3TTSModel.generate in mlx-audio): the whole text goes into the prefill, and each later step feeds the embedded frame plus the pad embedding. The KV caches stay in native memory, as in com.blockether.vis.internal.speech.vibevoice.

Qwen3-TTS-12Hz-1.7B-CustomVoice text to speech through ONNX Runtime.

  :qwen3-tts-local  Nine built-in speakers. Each speaker can read ten languages,
                    and the language of the text selects the language prompt.

The bundle is the `cpu_int4` ONNX export of speed-brain-ai. Its graphs are:

  text_embed           text token ids to projected embeddings
  codec_embed          codec token ids to talker embeddings
  talker_cache         the talker with a KV cache: the first codebook of a frame
  code_predictor_loop  the other 15 codebooks of a frame, sampled in the graph
  residual_embed       the 16 codes of a frame to the next talker input
  tok_decoder          frames to 24 kHz audio, 1920 samples for each frame

A voice imported from a recording uses the Base model, the `cpu_int4` export of
onnx-community. It has its own codec_embed and residual_embed, and these graphs:

  speaker_encoder  24 kHz audio to the speaker vector of the codec prompt
  tok_encoder      one second of 24 kHz audio to 13 frames of codes

`bin/export-qwen-base` makes its other graphs. Its text_embed and talker_cache
are the onnx-community graphs with int8 activations, which are several times faster
on a CPU. Its code predictor has a KV cache, so each codebook is one step of one row:

  code_predictor_step   rows and a KV cache to the logits of one codebook
  code_predictor_embed  a code of one codebook to its input row

It uses the tokenizer and the decoder of the speaker bundle.

The prompt and the sampling follow the reference generation of Qwen3-TTS
(`Qwen3TTSModel.generate` in mlx-audio): the whole text goes into the prefill,
and each later step feeds the embedded frame plus the pad embedding. The KV
caches stay in native memory, as in [[com.blockether.vis.internal.speech.vibevoice]].
raw docstring

com.blockether.vis.internal.speech.sherpa

sherpa-onnx JNI for speech, sharing Java decisions' ONNX Runtime on the macOS ARM64 and Linux release platforms. The upstream per-platform jar still bundles an older runtime; Vis extracts its JNI only and stages it beside ONNX Runtime Java 1.30.0. Unsupported platforms retain upstream's pair.

On a plain JVM the platform jar is downloaded on first speech use; native images embed only the host JNI. Both loaders use one versioned directory. A user-supplied sherpa_onnx.native.path remains authoritative.

The JitPack jar is length-checked and installed atomically; it is fetched from the original publisher, not mirrored by Vis. Its JNI answers the pinned version, as sherpa-test asserts.

;; JNI and not java.lang.foreign, because the choice is upstream's: the library ;; Vis ships, libsherpa-onnx-jni, exports 133 Java_* entry points and not one ;; SherpaOnnx* C symbol, so a Panama downcall has nothing to bind to. sherpa's C ;; API is a separate artifact — per-platform tarballs under no Maven coordinate — ;; whose 156 functions over 86 structs would leave us owning their layouts. Vis ;; does use FFM where it owns the boundary (internal/foundation/pty); here the ;; image registers the API jar's types for JNI instead ;; (reachability-metadata.json, pinned by sherpa-test).

sherpa-onnx JNI for speech, sharing Java decisions' ONNX Runtime on the
macOS ARM64 and Linux release platforms. The upstream per-platform jar still
bundles an older runtime; Vis extracts its JNI only and stages it beside
ONNX Runtime Java 1.30.0. Unsupported platforms retain upstream's pair.

On a plain JVM the platform jar is downloaded on first speech use; native
images embed only the host JNI. Both loaders use one versioned directory.
A user-supplied `sherpa_onnx.native.path` remains authoritative.

The JitPack jar is length-checked and installed atomically; it is fetched
from the original publisher, not mirrored by Vis. Its JNI answers the
pinned `version`, as `sherpa-test` asserts.

;; JNI and not `java.lang.foreign`, because the choice is upstream's: the library
;; Vis ships, `libsherpa-onnx-jni`, exports 133 `Java_*` entry points and not one
;; `SherpaOnnx*` C symbol, so a Panama downcall has nothing to bind to. sherpa's C
;; API is a separate artifact — per-platform tarballs under no Maven coordinate —
;; whose 156 functions over 86 structs would leave us owning their layouts. Vis
;; does use FFM where it owns the boundary (`internal/foundation/pty`); here the
;; image registers the API jar's types for JNI instead
;; (`reachability-metadata.json`, pinned by `sherpa-test`).
raw docstring

com.blockether.vis.internal.speech.synthesis

The gateway's built-in local Piper, pocket-tts, VibeVoice and Qwen3-TTS speaking engines.

The gateway's built-in local Piper, pocket-tts, VibeVoice and Qwen3-TTS speaking engines.
raw docstring

com.blockether.vis.internal.speech.transcode

Whatever somebody recorded, as the 16-bit PCM WAV every local model reads.

A recorder never asks what the model wants. An iPhone memo is .m4a (AAC in an MP4 box), an Android one .amr, .aac or .m4a, a browser records .ogg or .webm, a shared clip arrives as .mp3, and sherpa-onnx reads exactly ONE container: RIFF/WAVE, 16-bit PCM. So every path that turns audio into text meets the SAME question — voice INPUT in the TUI, a recording ATTACHED to a message, a clip IMPORTED as a voice — and it is answered once, here.

Vis ships no decoder of its own: AAC, Opus and AMR are a codec suite rather than a namespace, and a wrong decoder is silence that transcribes into confident words. ffmpeg is the one converter a machine reliably has, so a non-WAV recording goes through it into a TEMP mono 16 kHz file that is deleted the moment the work is done.

Whatever somebody recorded, as the 16-bit PCM WAV every local model reads.

A recorder never asks what the model wants. An iPhone memo is `.m4a` (AAC in an
MP4 box), an Android one `.amr`, `.aac` or `.m4a`, a browser records `.ogg` or
`.webm`, a shared clip arrives as `.mp3`, and sherpa-onnx reads exactly ONE
container: RIFF/WAVE, 16-bit PCM. So every path that turns audio into text meets
the SAME question — voice INPUT in the TUI, a recording ATTACHED to a message,
a clip IMPORTED as a voice — and it is answered once, here.

Vis ships no decoder of its own: AAC, Opus and AMR are a codec suite rather than
a namespace, and a wrong decoder is silence that transcribes into confident
words. `ffmpeg` is the one converter a machine reliably has, so a non-WAV
recording goes through it
into a TEMP mono 16 kHz file that is deleted the moment the work is done.
raw docstring

com.blockether.vis.internal.speech.tts

Local speech synthesis through sherpa-onnx.

Two families, one shape: make the assets present, build one OfflineTts, generate. They differ only in what a VOICE is.

:piper a voice IS a model — one VITS network per speaker, phonemized through eSpeak NG tables. Vis uses a system copy when present or the verified tables carried by a downloaded publisher model archive. :pocket-tts a voice is a reference CLIP the model clones, so the catalogue is a WAV per voice: the clips of pocket-voice-clips and the ones somebody imported through voices.clj. Vis exports those weights itself, so they arrive with every other model.

Which assets exist, where they come from and what they are licensed under is assets.clj's question; this namespace only asks for them by id.

Local speech synthesis through sherpa-onnx.

Two families, one shape: make the assets present, build one `OfflineTts`,
generate. They differ only in what a VOICE is.

  :piper       a voice IS a model — one VITS network per speaker, phonemized
               through eSpeak NG tables. Vis uses a system copy when present or the
               verified tables carried by a downloaded publisher model archive.
  :pocket-tts  a voice is a reference CLIP the model clones, so the catalogue
               is a WAV per voice: the clips of `pocket-voice-clips` and the
                ones somebody imported through `voices.clj`. Vis exports those
                weights itself, so they arrive with every other model.

Which assets exist, where they come from and what they are licensed under is
`assets.clj`'s question; this namespace only asks for them by id.
raw docstring

com.blockether.vis.internal.speech.vibevoice

Microsoft VibeVoice speech through ONNX Runtime, without sherpa-onnx.

Two engines share one runtime here:

:vibevoice-local VibeVoice-Realtime-0.5B text to speech. A voice is a cached prompt state in voices.bin, so the catalogue is fixed by the bundle. :vibevoice-asr-local VibeVoice-ASR-Streaming-1.5B transcription, in chunks of 22 frames with 4 frames of lookahead.

The graphs, the step loops and the noise generator follow the reference runners of the vis-podcaster repository (scripts/vibevoice_onnx.py and scripts/vibevoice_asr_onnx.py). A change to a loop here changes the audio, so change the reference runner with it.

The KV caches stay in native memory: each step feeds the tensors of the previous OrtSession$Result back in, and that result closes only after its replacement exists.

Microsoft VibeVoice speech through ONNX Runtime, without sherpa-onnx.

Two engines share one runtime here:

  :vibevoice-local      VibeVoice-Realtime-0.5B text to speech. A voice is a
                        cached prompt state in `voices.bin`, so the catalogue is
                        fixed by the bundle.
  :vibevoice-asr-local  VibeVoice-ASR-Streaming-1.5B transcription, in chunks of
                        22 frames with 4 frames of lookahead.

The graphs, the step loops and the noise generator follow the reference runners
of the vis-podcaster repository (`scripts/vibevoice_onnx.py` and
`scripts/vibevoice_asr_onnx.py`). A change to a loop here changes the audio, so
change the reference runner with it.

The KV caches stay in native memory: each step feeds the tensors of the previous
`OrtSession$Result` back in, and that result closes only after its replacement
exists.
raw docstring

com.blockether.vis.internal.speech.voices

Reference clips somebody brought: the voices Vis did not ship.

pocket-tts clones a RECORDING instead of selecting a baked speaker, so a voice in that engine IS a WAV file - which means a voice is something a person can make and hand to Vis, and no licence anywhere can stop them. This namespace is where such a clip lives: checked and normalized once on the way in, listed beside the clips the bundle carries, and readable by id from every surface (CLI, gateway, app).

Imported clips deliberately do NOT live under the models directory. An asset is something Vis can fetch again; a recording is the user's own, and clearing a model cache must never take somebody's voice with it.

Reference clips somebody brought: the voices Vis did not ship.

pocket-tts clones a RECORDING instead of selecting a baked speaker, so a
voice in that engine IS a WAV file - which means a voice is something a
person can make and hand to Vis, and no licence anywhere can stop them. This
namespace is where such a clip lives: checked and normalized once on the way
in, listed beside the clips the bundle carries, and readable by id from every
surface (CLI, gateway, app).

Imported clips deliberately do NOT live under the models directory. An asset
is something Vis can fetch again; a recording is the user's own, and clearing
a model cache must never take somebody's voice with it.
raw docstring

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
←Move to previous article
→Move to next article
Ctrl+/Jump to the search field
× close