Qwen3-TTS-12Hz-1.7B-CustomVoice text to speech through ONNX Runtime.
:qwen3-tts-local Nine built-in speakers. Each speaker can read ten languages, and the language of the text selects the language prompt.
The bundle is the cpu_int4 ONNX export of speed-brain-ai. Its graphs are:
text_embed text token ids to projected embeddings codec_embed codec token ids to talker embeddings talker_cache the talker with a KV cache: the first codebook of a frame code_predictor_loop the other 15 codebooks of a frame, sampled in the graph residual_embed the 16 codes of a frame to the next talker input tok_decoder frames to 24 kHz audio, 1920 samples for each frame
A voice imported from a recording uses the Base model, the cpu_int4 export of
onnx-community. It has its own codec_embed and residual_embed, and these graphs:
speaker_encoder 24 kHz audio to the speaker vector of the codec prompt tok_encoder one second of 24 kHz audio to 13 frames of codes
bin/export-qwen-base makes its other graphs. Its text_embed and talker_cache
are the onnx-community graphs with int8 activations, which are several times faster
on a CPU. Its code predictor has a KV cache, so each codebook is one step of one row:
code_predictor_step rows and a KV cache to the logits of one codebook code_predictor_embed a code of one codebook to its input row
It uses the tokenizer and the decoder of the speaker bundle.
The prompt and the sampling follow the reference generation of Qwen3-TTS
(Qwen3TTSModel.generate in mlx-audio): the whole text goes into the prefill,
and each later step feeds the embedded frame plus the pad embedding. The KV
caches stay in native memory, as in com.blockether.vis.internal.speech.vibevoice.
Qwen3-TTS-12Hz-1.7B-CustomVoice text to speech through ONNX Runtime.
:qwen3-tts-local Nine built-in speakers. Each speaker can read ten languages,
and the language of the text selects the language prompt.
The bundle is the `cpu_int4` ONNX export of speed-brain-ai. Its graphs are:
text_embed text token ids to projected embeddings
codec_embed codec token ids to talker embeddings
talker_cache the talker with a KV cache: the first codebook of a frame
code_predictor_loop the other 15 codebooks of a frame, sampled in the graph
residual_embed the 16 codes of a frame to the next talker input
tok_decoder frames to 24 kHz audio, 1920 samples for each frame
A voice imported from a recording uses the Base model, the `cpu_int4` export of
onnx-community. It has its own codec_embed and residual_embed, and these graphs:
speaker_encoder 24 kHz audio to the speaker vector of the codec prompt
tok_encoder one second of 24 kHz audio to 13 frames of codes
`bin/export-qwen-base` makes its other graphs. Its text_embed and talker_cache
are the onnx-community graphs with int8 activations, which are several times faster
on a CPU. Its code predictor has a KV cache, so each codebook is one step of one row:
code_predictor_step rows and a KV cache to the logits of one codebook
code_predictor_embed a code of one codebook to its input row
It uses the tokenizer and the decoder of the speaker bundle.
The prompt and the sampling follow the reference generation of Qwen3-TTS
(`Qwen3TTSModel.generate` in mlx-audio): the whole text goes into the prefill,
and each later step feeds the embedded frame plus the pad embedding. The KV
caches stay in native memory, as in [[com.blockether.vis.internal.speech.vibevoice]].The longest text of one generation. A longer text is split at sentence ends, as the quality of one prefill drops on long text.
The longest text of one generation. A longer text is split at sentence ends, as the quality of one prefill drops on long text.
Silence between two chunks of one text.
Silence between two chunks of one text.
(chunks text)text in pieces of at most chunk-chars, split at sentence ends and line
breaks. A sentence that is longer stays whole.
`text` in pieces of at most [[chunk-chars]], split at sentence ends and line breaks. A sentence that is longer stays whole.
The Base model, which speaks in an imported voice.
The Base model, which speaks in an imported voice.
Codec tokens of the talker head, with the control tokens above 2047.
Codec tokens of the talker head, with the control tokens above 2047.
Frames before a decoder window that the decoder hears again: the attention window of the codec, so a window boundary does not change the audio.
Frames before a decoder window that the decoder hears again: the attention window of the codec, so a window boundary does not change the audio.
(decode-frames bundle frames)(decode-frames bundle frames context)The 24 kHz samples of frames. The last decode-context frames of context,
the frames of a recording, are decoded before them and then dropped, so the
speech continues the recording as in the reference.
The 24 kHz samples of `frames`. The last [[decode-context]] frames of `context`, the frames of a recording, are decoded before them and then dropped, so the speech continues the recording as in the reference.
Frames that one decoder call returns as audio.
Frames that one decoder call returns as audio.
Noise seed of a request without one, so the same text in the same voice gives the same audio.
Noise seed of a request without one, so the same text in the same voice gives the same audio.
(encode-reference {:keys [env audio-encoder]} samples)The frames of 24 kHz samples, one for each started 80 ms. The encoder reads one
second at a time, and encoder-stride frames of each window are kept.
The frames of 24 kHz `samples`, one for each started 80 ms. The encoder reads one second at a time, and [[encoder-stride]] frames of each window are kept.
Frames kept from each encoder window. A second is 12.5 frames, so a stride of 12 frames keeps every window on the 80 ms grid.
Frames kept from each encoder window. A second is 12.5 frames, so a stride of 12 frames keeps every window on the 80 ms grid.
Samples that one tok_encoder call reads: one second.
Samples that one tok_encoder call reads: one second.
The Vis export of the Base model: its code predictor, talker and text_embed.
The Vis export of the Base model: its code predictor, talker and text_embed.
(frame-limit piece)The most frames for the text piece. One character takes about one frame, and a
Chinese character about three. A clone that never ends its speech stops here,
not at max-frames.
The most frames for the text `piece`. One character takes about one frame, and a Chinese character about three. A clone that never ends its speech stops here, not at [[max-frames]].
Audio samples of one 80 ms frame at 24 kHz.
Audio samples of one 80 ms frame at 24 kHz.
(generate-frames {:keys [env] :as bundle}
{:keys [embeds rows pad limit] :or {limit max-frames}}
noise
report)Every frame of one text chunk as 16-code arrays, until the end-of-speech token
or limit frames, max-frames by default. report gets the frame count now
and then.
Every frame of one text chunk as 16-code arrays, until the end-of-speech token or `limit` frames, [[max-frames]] by default. `report` gets the frame count now and then.
The most frames of one text chunk, about 160 seconds. Generation stops here even without an end-of-speech token.
The most frames of one text chunk, about 160 seconds. Generation stops here even without an end-of-speech token.
(prefill-embeddings bundle ids codec)(prefill-embeddings {:keys [config text-embed] :as bundle}
ids
codec
{:keys [text-ids steps]})The talker input of the prefill for the text token ids and the flat codec
prompt rows: the role, the codec prompt with the speaker, the whole text, then
the codec start. A clone ref adds the transcript of the recording before the
text and its frames after the codec start, as the in-context prompt of the
reference does. Returns the flat embeddings and the pad embedding that each
later step adds.
The talker input of the prefill for the text token `ids` and the flat `codec` prompt rows: the role, the codec prompt with the speaker, the whole text, then the codec start. A clone `ref` adds the transcript of the recording before the text and its frames after the codec start, as the in-context prompt of the reference does. Returns the flat embeddings and the pad embedding that each later step adds.
(prepare-voice-sample! voice-id)Speak the shared sample sentence in voice-id with the installed model and keep
the file. Never downloads the model.
Speak the shared sample sentence in `voice-id` with the installed model and keep the file. Never downloads the model.
(reference {:keys [env tokenizer residual-embed] :as bundle}
{:keys [clip clip-text]})The prompt of the imported voice for the Base bundle: the speaker vector of
its recording, and with a transcript also the text ids, frames and frame
embeddings for the in-context prompt.
The prompt of the imported `voice` for the Base `bundle`: the speaker vector of its recording, and with a transcript also the text ids, frames and frame embeddings for the in-context prompt.
Applied to the first codebook only, as the reference does.
Applied to the first codebook only, as the reference does.
Codebooks that the code predictor adds to each frame.
Codebooks that the code predictor adds to each frame.
(sample-first-code logits offset history eos noise)The first codebook of the next frame from the last row of the talker logits
at offset: control tokens except end of speech are blocked, recent tokens are
penalised, then temperature, top-k and one draw from noise.
The first codebook of the next frame from the last row of the talker `logits` at `offset`: control tokens except end of speech are blocked, recent tokens are penalised, then temperature, top-k and one draw from `noise`.
(speak dir text voice)(speak dir
text
voice
{:keys [seed report] :or {seed default-seed report (fn [_])}})Speak text in voice with the speaker bundle in dir. An imported voice uses
the installed Base model. Returns the mono samples and their rate.
Speak `text` in `voice` with the speaker bundle in `dir`. An imported voice uses the installed Base model. Returns the mono samples and their rate.
(synthesize! {:keys [text voice-id on-progress seed]})Speak text and return the WAV that was written, plus the facts a player
needs. The model is installed first when it is missing, reported as :preparing.
Speak `text` and return the WAV that was written, plus the facts a player needs. The model is installed first when it is missing, reported as `:preparing`.
The sampling temperature of the reference generation.
The sampling temperature of the reference generation.
(text-language text)The language of text as a Qwen3-TTS language name, or nil when the text is in
none of its ten languages. The script decides first; for Latin script, the most
frequent common words decide.
The language of `text` as a Qwen3-TTS language name, or nil when the text is in none of its ten languages. The script decides first; for Latin script, the most frequent common words decide.
(voice-sample voice-id)The sample of voice-id, without making one: {:audio-path …} on disk,
{:is-preparable true} when the installed model can speak it, or nil.
The sample of `voice-id`, without making one: `{:audio-path …}` on disk,
`{:is-preparable true}` when the installed model can speak it, or nil.(voices)The speakers of the bundle, from the manifest, so a surface can list them before the download. Then the voices imported from a recording, which the Base model speaks.
The speakers of the bundle, from the manifest, so a surface can list them before the download. Then the voices imported from a recording, which the Base model speaks.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |