Microsoft VibeVoice speech through ONNX Runtime, without sherpa-onnx.
Two engines share one runtime here:
:vibevoice-local VibeVoice-Realtime-0.5B text to speech. A voice is a
cached prompt state in voices.bin, so the catalogue is
fixed by the bundle.
:vibevoice-asr-local VibeVoice-ASR-Streaming-1.5B transcription, in chunks of
22 frames with 4 frames of lookahead.
The graphs, the step loops and the noise generator follow the reference runners
of the vis-podcaster repository (scripts/vibevoice_onnx.py and
scripts/vibevoice_asr_onnx.py). A change to a loop here changes the audio, so
change the reference runner with it.
The KV caches stay in native memory: each step feeds the tensors of the previous
OrtSession$Result back in, and that result closes only after its replacement
exists.
Microsoft VibeVoice speech through ONNX Runtime, without sherpa-onnx.
Two engines share one runtime here:
:vibevoice-local VibeVoice-Realtime-0.5B text to speech. A voice is a
cached prompt state in `voices.bin`, so the catalogue is
fixed by the bundle.
:vibevoice-asr-local VibeVoice-ASR-Streaming-1.5B transcription, in chunks of
22 frames with 4 frames of lookahead.
The graphs, the step loops and the noise generator follow the reference runners
of the vis-podcaster repository (`scripts/vibevoice_onnx.py` and
`scripts/vibevoice_asr_onnx.py`). A change to a loop here changes the audio, so
change the reference runner with it.
The KV caches stay in native memory: each step feeds the tensors of the previous
`OrtSession$Result` back in, and that result closes only after its replacement
exists.Classifier-free guidance of each diffusion step, as upstream ships it.
Classifier-free guidance of each diffusion step, as upstream ships it.
(chunk-words chunks)Timed words from chunk texts. A chunk may end inside a word, so the chunks join first; each character keeps a time inside its chunk, and a word spans the times of its first and last character.
Timed words from chunk texts. A chunk may end inside a word, so the chunks join first; each character keeps a time inside its chunk, and a word spans the times of its first and last character.
Noise seed of a request without one, so the same text in the same voice gives the same audio.
Noise seed of a request without one, so the same text in the same voice gives the same audio.
(gaussian state n)n standard normal values. Each pair uses two uniforms; an odd n drops the
last sine value, as the reference does.
`n` standard normal values. Each pair uses two uniforms; an odd `n` drops the last sine value, as the reference does.
(level samples)Scale samples to a peak of 0.7 (-3 dBFS), as the Piper samples are, so no
voice plays louder than the one before it. Silence stays silence.
Scale `samples` to a peak of 0.7 (-3 dBFS), as the Piper samples are, so no voice plays louder than the one before it. Silence stays silence.
The most text tokens one ASR chunk may emit.
The most text tokens one ASR chunk may emit.
The longest TTS language-model context. Generation stops here even without an end-of-speech signal.
The longest TTS language-model context. Generation stops here even without an end-of-speech signal.
(noise-state seed)A fresh noise generator for seed.
A fresh noise generator for `seed`.
(prepare-voice-sample! voice-id)Make the sample that voice-sample called preparable and return it. Installs
the small pack, or speaks the sentence with the installed model and keeps the
file. Never downloads the model.
Make the sample that [[voice-sample]] called preparable and return it. Installs the small pack, or speaks the sentence with the installed model and keeps the file. Never downloads the model.
(read-wav file)A 16-bit PCM WAV as {:samples <mono floats> :sample-rate n}. Channels are
averaged. Validate the file first: this reader trusts its chunk sizes.
A 16-bit PCM WAV as `{:samples <mono floats> :sample-rate n}`. Channels are
averaged. Validate the file first: this reader trusts its chunk sizes.(release!)Close every loaded session. The next request loads its bundle again.
Close every loaded session. The next request loads its bundle again.
(resample samples from to)samples at from Hz as to Hz, with a Hann-windowed sinc of 16 zero
crossings on each side. The cutoff follows the lower of the two rates.
`samples` at `from` Hz as `to` Hz, with a Hann-windowed sinc of 16 zero crossings on each side. The cutoff follows the lower of the two rates.
(sample-text voice-id)The sentence that voice-id reads for its sample, in the voice's language.
The sentence that `voice-id` reads for its sample, in the voice's language.
The Piper sample sentence in each language of the catalogue, by base language.
The Piper sample sentence in each language of the catalogue, by base language.
(speak dir text voice-id)(speak dir
text
voice-id
{:keys [seed report] :or {seed default-seed report (fn [_])}})Speak text in voice-id with the bundle in dir. Returns the mono samples
and their rate.
Speak `text` in `voice-id` with the bundle in `dir`. Returns the mono samples and their rate.
(speak-sample! voice-id out)Speak the shared sample sentence in voice-id into out with the installed
model, levelled. The file appears whole or not at all.
bin/make-vibevoice-samples builds the pack with it.
Speak the shared sample sentence in `voice-id` into `out` with the installed model, levelled. The file appears whole or not at all. `bin/make-vibevoice-samples` builds the pack with it.
(synthesize! {:keys [text voice-id on-progress seed]})Speak text and return the WAV that was written, plus the facts a player
needs. The model is installed first when it is missing, reported as :preparing.
Speak `text` and return the WAV that was written, plus the facts a player needs. The model is installed first when it is missing, reported as `:preparing`.
(transcribe {:keys [audio-path on-progress]})Transcribe one recording: the words, and the timed lines they fall into. The times come from the chunk that heard each word, so a line is accurate to about three seconds.
Transcribe one recording: the words, and the timed lines they fall into. The times come from the chunk that heard each word, so a line is accurate to about three seconds.
(transcribe-samples dir samples)(transcribe-samples dir
samples
{:keys [seed report context]
:or {seed default-seed report (fn [_])}})The text of each audio chunk of 24 kHz samples, as {:start :end :text} in
seconds, with the bundle in dir.
The text of each audio chunk of 24 kHz `samples`, as `{:start :end :text}` in
seconds, with the bundle in `dir`.(voice-sample voice-id)The sample of voice-id, without making one: {:audio-path …} on disk,
{:is-preparable true} when the pack or the installed model can make it, or nil.
The sample of `voice-id`, without making one: `{:audio-path …}` on disk,
`{:is-preparable true}` when the pack or the installed model can make it, or nil.(voices)The voices the bundle speaks in, from the manifest, so a surface can list them before the download.
The voices the bundle speaks in, from the manifest, so a surface can list them before the download.
(warm)Load the transcription model when it is installed. True when it is loaded now.
Load the transcription model when it is installed. True when it is loaded now.
(write-wav! out samples sample-rate)Write mono float samples in -1..1 as a 16-bit PCM WAV.
Write mono float `samples` in -1..1 as a 16-bit PCM WAV.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |