Direct Java sherpa-onnx integration for Parakeet TDT ASR.
Direct Java sherpa-onnx integration for Parakeet TDT ASR.
Where a speech asset comes from, what it is licensed under, and how it lands on disk.
resources/vis-models/manifest.edn is the only answer to "may we ship
this": every model and data directory Vis installs has an entry there with an
SPDX id, an attribution line and :is-commercial-ok true. An artifact that is
not in the manifest cannot be downloaded, because nothing else carries a URL.
Each entry lists its sources in preference order and every source delivers the
SAME bytes, so one :sha256 verifies them all:
:hf Hugging Face. SKIPPED unless a token is configured, tried FIRST when one is. A token is never required — it only changes WHERE the same bytes come from. :pack the Vis VOICE_ASSETS_PACK release, the default, so a first run needs no account anywhere. :upstream the project that published the artifact, last, as a fallback.
Sources are tried in that order and the first that installs cleanly wins, so an asset host being down is a slower install rather than a dead feature.
Where a speech asset comes from, what it is licensed under, and how it lands on
disk.
`resources/vis-models/manifest.edn` is the only answer to "may we ship
this": every model and data directory Vis installs has an entry there with an
SPDX id, an attribution line and `:is-commercial-ok true`. An artifact that is
not in the manifest cannot be downloaded, because nothing else carries a URL.
Each entry lists its sources in preference order and every source delivers the
SAME bytes, so one `:sha256` verifies them all:
:hf Hugging Face. SKIPPED unless a token is configured, tried FIRST
when one is. A token is never required — it only changes WHERE
the same bytes come from.
:pack the Vis VOICE_ASSETS_PACK release, the default, so a first run
needs no account anywhere.
:upstream the project that published the artifact, last, as a fallback.
Sources are tried in that order and the first that installs cleanly wins, so
an asset host being down is a slower install rather than a dead feature.THIRD_PARTY_MODELS.md is nobody's prose: it is
resources/vis-models/manifest.edn rendered, so the credits a reader checks
and the sources the installer obeys can never say different things.
markdown is the whole file. assets-test fails when the copy in the tree
drifts from it, and vis-agent speech models licenses --markdown
reprints it.
`THIRD_PARTY_MODELS.md` is nobody's prose: it is `resources/vis-models/manifest.edn` rendered, so the credits a reader checks and the sources the installer obeys can never say different things. `markdown` is the whole file. `assets-test` fails when the copy in the tree drifts from it, and `vis-agent speech models licenses --markdown` reprints it.
Commands for the gateway-owned speech subsystem. This namespace never loads a model or Sherpa: every runtime operation crosses the canonical gateway client.
Commands for the gateway-owned speech subsystem. This namespace never loads a model or Sherpa: every runtime operation crosses the canonical gateway client.
The gateway's fixed local speech engines and their shared asynchronous job lifecycle.
The gateway's fixed local speech engines and their shared asynchronous job lifecycle.
The built-in local Parakeet transcription engine used by the gateway.
The built-in local Parakeet transcription engine used by the gateway.
Streaming download, checksum and archive helpers for the gateway's speech assets.
Streaming download, checksum and archive helpers for the gateway's speech assets.
sherpa-onnx's native libraries, for THIS platform and no other.
sherpa publishes one native jar per platform, each 8-13 MB and each carrying
BOTH libsherpa-onnx-jni and the exact libonnxruntime it was linked
against, side by side under sherpa-onnx/native/<platform>/. Depending on
all five in deps.edn makes every machine download 51 MB to use one of them,
so core depends on the 187 KB API jar alone and the pair arrives
here, one of three ways:
sherpa_onnx.native.path names a directory holding
both libraries. This is sherpa's own first loading method, so it is also
the seam for a self-built native (an espeak-free one, say) and nothing is
downloaded or checked out from under the user.build.clj puts the BUILD HOST's native jar on the image
classpath and -H:IncludeResources bakes that one directory in, and any
JVM run that puts a sherpa-onnx-native-lib-* jar on the classpath.deps.edn pins, unpacked into ~/.vis, and handed to
sherpa through sherpa_onnx.native.path.No digest is pinned for that download: JitPack rebuilds a tag when its cache
evicts, so a pinned digest would eventually break every user rather than
catch anything. Integrity comes from the transfer being length-checked
(files/download!), from the install being atomic, and from the loaded
library having to answer version — which sherpa-native-test asserts.
;; JNI and not java.lang.foreign, because the choice is upstream's: the library
;; Vis ships, libsherpa-onnx-jni, exports 133 Java_* entry points and not one
;; SherpaOnnx* C symbol, so a Panama downcall has nothing to bind to. sherpa's C
;; API is a separate artifact — per-platform tarballs under no Maven coordinate —
;; whose 156 functions over 86 structs would leave us owning their layouts. Vis
;; does use FFM where it owns the boundary (internal/foundation/pty); here the
;; image registers the API jar's types for JNI instead
;; (reachability-metadata.json, pinned by sherpa-test).
sherpa-onnx's native libraries, for THIS platform and no other. sherpa publishes one native jar per platform, each 8-13 MB and each carrying BOTH `libsherpa-onnx-jni` and the exact `libonnxruntime` it was linked against, side by side under `sherpa-onnx/native/<platform>/`. Depending on all five in `deps.edn` makes every machine download 51 MB to use one of them, so core depends on the 187 KB API jar alone and the pair arrives here, one of three ways: - **Already loadable** — `sherpa_onnx.native.path` names a directory holding both libraries. This is sherpa's own first loading method, so it is also the seam for a self-built native (an espeak-free one, say) and nothing is downloaded or checked out from under the user. - **Embedded** — the libraries are classpath resources. That is the native image, where `build.clj` puts the BUILD HOST's native jar on the image classpath and `-H:IncludeResources` bakes that one directory in, and any JVM run that puts a `sherpa-onnx-native-lib-*` jar on the classpath. - **Downloaded** — the host platform's jar is fetched once from the same JitPack coordinate `deps.edn` pins, unpacked into `~/.vis`, and handed to sherpa through `sherpa_onnx.native.path`. No digest is pinned for that download: JitPack rebuilds a tag when its cache evicts, so a pinned digest would eventually break every user rather than catch anything. Integrity comes from the transfer being length-checked (`files/download!`), from the install being atomic, and from the loaded library having to answer `version` — which `sherpa-native-test` asserts. ;; JNI and not `java.lang.foreign`, because the choice is upstream's: the library ;; Vis ships, `libsherpa-onnx-jni`, exports 133 `Java_*` entry points and not one ;; `SherpaOnnx*` C symbol, so a Panama downcall has nothing to bind to. sherpa's C ;; API is a separate artifact — per-platform tarballs under no Maven coordinate — ;; whose 156 functions over 86 structs would leave us owning their layouts. Vis ;; does use FFM where it owns the boundary (`internal/foundation/pty`); here the ;; image registers the API jar's types for JNI instead ;; (`reachability-metadata.json`, pinned by `sherpa-test`).
The gateway's built-in local Piper and pocket-tts speaking engines.
The gateway's built-in local Piper and pocket-tts speaking engines.
Whatever somebody recorded, as the 16-bit PCM WAV every local model reads.
A recorder never asks what the model wants. An iPhone memo is .m4a (AAC in an
MP4 box), an Android one .amr, .aac or .m4a, a browser records .ogg or
.webm, a shared clip arrives as .mp3, and sherpa-onnx reads exactly ONE
container: RIFF/WAVE, 16-bit PCM. So every path that turns audio into text meets
the SAME question — voice INPUT in the TUI, a recording ATTACHED to a message,
a clip IMPORTED as a voice — and it is answered once, here.
Vis ships no decoder of its own: AAC, Opus and AMR are a codec suite rather than
a namespace, and a wrong decoder is silence that transcribes into confident
words. ffmpeg is the one converter a machine reliably has, so a non-WAV
recording goes through it
into a TEMP mono 16 kHz file that is deleted the moment the work is done.
Whatever somebody recorded, as the 16-bit PCM WAV every local model reads. A recorder never asks what the model wants. An iPhone memo is `.m4a` (AAC in an MP4 box), an Android one `.amr`, `.aac` or `.m4a`, a browser records `.ogg` or `.webm`, a shared clip arrives as `.mp3`, and sherpa-onnx reads exactly ONE container: RIFF/WAVE, 16-bit PCM. So every path that turns audio into text meets the SAME question — voice INPUT in the TUI, a recording ATTACHED to a message, a clip IMPORTED as a voice — and it is answered once, here. Vis ships no decoder of its own: AAC, Opus and AMR are a codec suite rather than a namespace, and a wrong decoder is silence that transcribes into confident words. `ffmpeg` is the one converter a machine reliably has, so a non-WAV recording goes through it into a TEMP mono 16 kHz file that is deleted the moment the work is done.
Local speech synthesis through sherpa-onnx.
Two families, one shape: make the assets present, build one OfflineTts,
generate. They differ only in what a VOICE is.
:piper a voice IS a model — one VITS network per speaker, phonemized
through eSpeak NG tables. Vis uses a system copy when present or the
verified tables carried by a downloaded publisher model archive.
:pocket-tts a voice is a reference CLIP the model clones, so the catalogue
is a WAV per voice: the clips the bundle ships and the ones
somebody imported through voices.clj. Vis exports those
weights itself, so they arrive with every other model.
Which assets exist, where they come from and what they are licensed under is
assets.clj's question; this namespace only asks for them by id.
Local speech synthesis through sherpa-onnx.
Two families, one shape: make the assets present, build one `OfflineTts`,
generate. They differ only in what a VOICE is.
:piper a voice IS a model — one VITS network per speaker, phonemized
through eSpeak NG tables. Vis uses a system copy when present or the
verified tables carried by a downloaded publisher model archive.
:pocket-tts a voice is a reference CLIP the model clones, so the catalogue
is a WAV per voice: the clips the bundle ships and the ones
somebody imported through `voices.clj`. Vis exports those
weights itself, so they arrive with every other model.
Which assets exist, where they come from and what they are licensed under is
`assets.clj`'s question; this namespace only asks for them by id.Reference clips somebody brought: the voices Vis did not ship.
pocket-tts clones a RECORDING instead of selecting a baked speaker, so a voice in that engine IS a WAV file - which means a voice is something a person can make and hand to Vis, and no licence anywhere can stop them. This namespace is where such a clip lives: checked and normalized once on the way in, listed beside the clips the bundle carries, and readable by id from every surface (CLI, gateway, app).
Imported clips deliberately do NOT live under the models directory. An asset is something Vis can fetch again; a recording is the user's own, and clearing a model cache must never take somebody's voice with it.
Reference clips somebody brought: the voices Vis did not ship. pocket-tts clones a RECORDING instead of selecting a baked speaker, so a voice in that engine IS a WAV file - which means a voice is something a person can make and hand to Vis, and no licence anywhere can stop them. This namespace is where such a clip lives: checked and normalized once on the way in, listed beside the clips the bundle carries, and readable by id from every surface (CLI, gateway, app). Imported clips deliberately do NOT live under the models directory. An asset is something Vis can fetch again; a recording is the user's own, and clearing a model cache must never take somebody's voice with it.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |