Open-source text to speech: five model families for voice agents

Open-source text-to-speech models at a glance

Piper and Kokoro are compact starting points for local speech with selected voices. CosyVoice 3 is a candidate when incremental text input, audio output and pronunciation controls are requirements. Chatterbox separates English expression from multilingual cloning, while Qwen3-TTS adds voice design with a material serving caveat.

Self-hosting means running the trained model parameters, or weights, on infrastructure you control. A downloadable checkpoint of those weights does not establish commercial rights to its code, weights or voices. Those are separate parts of the decision.

The shortlist is organized by deployment fit. Parameter counts and variant labels do not establish memory requirements, concurrent-call capacity or production cost.

Four different meanings of streaming

A voice agent receives text from a language model and sends audio to a caller. Streaming can describe different parts of that path:

1. Complete utterance: generation finishes before the application receives the waveform.
2. Sentence or segment output: the application receives one completed segment at a time. Playback can overlap with generation of later segments.
3. Incremental audio output: audio chunks arrive before synthesis of the current utterance finishes, even if the model requires all its text first.
4. Text-in/audio-out streaming: the model accepts more text while returning audio. This is also called bi-streaming.

An HTTP server can divide a completed audio file into response chunks. That changes transport, not when synthesis produces the first playable sound. Similarly, a Python generator can yield whole sentences without exposing an incremental decoder.

Cancellation is another contract. Once a caller interrupts, the runtime needs to stop generation, discard queued audio and prevent stale speech from playing. Streaming alone does not provide that behavior.

Piper: local speech with per-voice licensing

Piper is a local speech engine built around voice models in ONNX, a portable model format. Its separate voice artifacts make it a useful baseline for an offline application, appliance or deployment with a fixed voice inventory. It is less directly aligned with a requirement to clone a speaker from a reference clip.

The current continuation is Piper’s GPL runtime. Its runtime is GPL-3.0-or-later. The MIT license of archived rhasspy/piper does not describe this current runtime. Voice rights need separate treatment: the Piper voice collection license tag does not replace the provenance and terms attached to an individual voice. A release inventory therefore needs the exact engine revision and each selected voice artifact.

The default path finishes a sentence before returning its audio. A long sentence can delay the first sound even when subsequent playback keeps up. Piper also has a split-model streaming path that separates an encoder and decoder so the decoder can return chunks. That work sits under the changelog's unreleased changes, so it should not be treated as the stable default interface.

Deployment includes the engine, voice files and eSpeak NG for converting written text into pronunciation symbols. A small model download is useful for distribution, but it says little about loaded memory or sessions per host. Piper fits teams that want local speech and can own the exact voice, licensing and serving configuration.

Kokoro-82M: compact synthesis with preset voices

Kokoro-82M v1.0 has 82 million parameters, with Apache-2.0 licenses for the checkpoint and inference library. It provides preset voices rather than a reference-audio cloning workflow. That is a useful starting point when an agent needs a consistent voice without reproducing a particular person's identity.

The voice collection spans American and British English, Japanese, Mandarin, Spanish, French, Hindi, Italian and Brazilian Portuguese. Coverage is uneven. French has one listed voice, and non-English output depends on the available training audio and text-to-pronunciation processing. A language name in the catalog does not establish equal performance across languages.

The Python pipeline yields text, pronunciation symbols called phonemes, and audio for each input split. It supports segment-wise playback, but it does not establish incremental audio generation within each segment. The service still needs a text-buffering policy, ordered playback and cancellation behavior.

Kokoro's voice and input limits matter for agents: very short inputs can be weaker, while long inputs can rush. Bundling short inputs may help synthesis but adds waiting; splitting long inputs introduces boundaries where delivery can change. A short acknowledgment such as “Yes” and a long address confirmation are therefore different workloads.

The standard stack includes Python, PyTorch, Misaki pronunciation processing and eSpeak NG fallbacks. Kokoro is a compact preset-voice candidate, with language coverage and segmentation as important selection criteria.

CosyVoice 3: text-in and audio-out streaming

Fun-CosyVoice 3.0 uses a 0.5-billion-parameter checkpoint under Apache-2.0, alongside an Apache-2.0 code repository. Its combination of bi-streaming and pronunciation control makes it a candidate for agents that must start speaking while more text arrives.

It supports nine languages, Chinese dialect and accent controls, and zero-shot voice cloning across languages. Zero-shot cloning uses reference audio without a separate voice-training run. Instructions can control delivery, including speed, volume and emotion. Pronunciation inpainting allows targeted corrections through Chinese Pinyin and English CMU phonemes, a phonetic symbol set.

These controls address different failures. Text normalization determines whether 12.50 becomes the intended spoken amount. A phoneme correction determines how a name sounds. Voice conditioning determines who appears to be speaking. One control does not substitute for the others.

The inference examples expose a stream option on synthesis calls. The deployment path still determines chunk delivery and interaction with the playback queue.

The operating surface is broader than a single ONNX voice file: model downloads, text frontend assets, Python dependencies and optional serving runtimes. NVIDIA Docker and vLLM paths have their own version requirements. CosyVoice belongs on a streaming-focused shortlist when the team can own those dependencies and reference-voice permissions. Its feature set does not establish a universal hardware minimum or a proven call-level latency.

Chatterbox: English and multilingual voice control

The Chatterbox model family separates English and multilingual voice-selection problems. Turbo and Multilingual V3 have MIT-licensed code and model checkpoints, but their size, language scope and controls differ.

Chatterbox Turbo is a 350-million-parameter English model. It accepts reference audio for voice cloning and supports expressive tags such as [laugh]. Its agent-oriented design makes it relevant when English delivery and voice identity matter. Expressive tags also need an application policy: a generated laugh may suit a character and be inappropriate during an account dispute.

Multilingual V3 is a 0.5-billion-parameter model covering 23 languages. It supports reference-audio voice prompting and cloning, with controls for expression and style. It is the relevant branch when the same voice identity needs to work across languages. Cloning still requires permission to use the reference speaker's voice; the MIT license does not supply that permission.

The standard examples call model.generate(...), receive a waveform and save it. They do not establish an incremental output interface. An agent-oriented description should therefore remain separate from a streaming-service requirement. A wrapper that segments text can create a segment-based pipeline, with the same buffering tradeoffs described above.

The standard GPU examples load the model with device="cuda", but model size alone does not specify video memory or concurrent capacity. Chatterbox also watermarks generated audio. The engineering scope includes model serving, reference-audio handling, language-specific pronunciation and interruption behavior. This family fits a voice-control shortlist; a live conversational service needs additional serving evidence.

Qwen3-TTS: voice design with a serving caveat

The Qwen3-TTS project offers Apache-2.0 code. Its released 12Hz variants are labeled 0.6B and 1.7B, with Apache-2.0 checkpoint licenses. Its three modes should be treated as separate choices: CustomVoice checkpoints select built-in voices, VoiceDesign checkpoints create a voice from a description, and Base uses reference audio for cloning.

The family covers ten languages. CustomVoice provides nine built-in voices, while the 1.7B VoiceDesign variant accepts natural-language descriptions. Those controls are useful when a product needs a designed identity or several distinct voices. Multiple available voices do not imply one request can render an entire cast; our multi-speaker TTS guide explains that distinction.

The serving boundary is material. Qwen3-TTS describes a streaming architecture, but the high-level Python methods return completed waveform arrays. In the generation wrapper, setting non_streaming_mode=False simulates streamed text input; it does not enable true streamed input or streamed generation through that interface. The project's vLLM-Omni instructions also describe offline inference, with online serving to follow.

That leaves a useful model family for full-utterance synthesis and voice-control work, with additional serving work before committing to an interactive stream. It does not establish that streaming is impossible through another implementation.

The standard examples use CUDA and reduced-precision model loading. Model and tokenizer versions, attention implementation and the selected serving stack all affect the deployment footprint. Treat Qwen3-TTS as a conditional live-agent candidate rather than translating its architecture-level latency into an end-to-end promise.

Two releases outside the commercial shortlist

F5-TTS illustrates the code-versus-weights distinction. Its code is MIT licensed, but the official pretrained checkpoints are CC-BY-NC-4.0. Those checkpoints do not provide a commercial deployment permission simply because the inference code is permissive. Separately licensed weights would be a different decision.

Microsoft VibeVoice-Realtime illustrates the intended-use distinction. Its MIT metadata and streaming interface do not remove the research-use warning. Microsoft does not recommend commercial or real-world use without further testing and development. Supported use is single-speaker English, and output includes an audible AI disclosure and a watermark. It belongs outside this production shortlist.

For every candidate, the shipped artifact set includes more than a repository: weights, tokenizer, pronunciation frontend, voice files and any reference recordings. Licensing and voice-consent records belong alongside the pinned release configuration.

Self-hosting cost follows capacity and ownership

There is no useful universal “cost per minute” for these models. A deployment's total includes inference hardware, idle reserve, redundancy, storage and egress, plus engineering and operating time. Dividing only a GPU's hourly rate by generated audio minutes leaves out much of the service.

For capacity, the relevant unit is a workload on a pinned hardware and software configuration. It needs to capture:

• First playable audio: from the text becoming available through synthesis, encoding and actual playback.
• Sustained generation: real-time factor, meaning generation time divided by audio duration, alongside gaps between chunks.
• Load and failure: queueing, peak memory, cold starts, concurrent sessions, worker restarts and fallback behavior.
• Conversation correctness: names, digits, dates, abbreviations, interruptions and speech already queued for playback.

A fast isolated request does not establish capacity during a burst. A process that fits in memory does not establish that it meets the conversation's response-time budget. Self-hosted TTS is a defensible choice when model control or local processing justifies owning that service. For an API-based speech-output layer, our hosted TTS comparison addresses a separate buying decision.

Where Dasha fits around the speech model

A TTS model supplies speech output. A production voice agent also needs speech recognition, conversation logic, tool execution, telephony, interruption handling and operational visibility. Self-hosting the model leaves those layers to the application and its runtime.

We help technical teams build and run production voice AI agents through a managed runtime, REST APIs and a web application, with telephony, integrations, testing and monitoring. Dasha is an adjacent runtime and operations option, not an open-source TTS model or a row in this model shortlist. Model selection and runtime compatibility remain separate decisions.

If owning the model is the requirement, the shortlist should remain focused on its license, serving contract and deployment footprint. If the requirement is operating a complete conversational product without assembling every runtime layer, start a Dasha evaluation.