Speech-to-speech models can make a voice agent more responsive and expressive. The category label also hides an important architecture choice. One service may process live audio directly, while another speech-to-speech API orchestrates speech recognition, reasoning, and synthesis behind a single endpoint. That difference changes what your application can control, inspect, and approve. The right choice depends on the conversation, the actions it can take, and the evidence your team needs in production.
What is a speech-to-speech model?
A speech-to-speech model accepts spoken audio, interprets it, and generates spoken audio in response. In conversational AI, it lets a user talk with a model without typing or reading. Direct models process audio inside one tightly integrated model or service. Cascaded systems deliver the same user experience through explicit speech-to-text (STT), large language model (LLM), and text-to-speech (TTS) stages.
The phrase describes the input and output. It does not fully describe the model's internals. Moshi's speech-to-speech research uses audio tokens and parallel user and system audio streams, while also predicting time-aligned text tokens before audio. NVIDIA NemotronLabs VoiceChat 11B accepts audio and returns a user transcript, agent text, and agent audio. An application can therefore have an audio-in, audio-out interface even when text remains part of the model's reasoning or output path.
Three terms keep the architecture clear:
| Term | Application-visible path | What your team controls |
|---|---|---|
| Direct or native audio model | Audio → model → audio | Session instructions, tools, audio transport, and the controls exposed by one model provider |
| Cascaded speech-to-speech system | Audio → STT → LLM and tools → TTS → audio | Each component, the intermediate text, policy checks, and handoffs between stages |
| Hybrid voice system | Direct audio for some turns, explicit text or structured logic for others | Routing rules, shared state, and the boundary where a turn moves into a controlled path |
Speech-to-speech also has narrower meanings in voice conversion and translation. Voice conversion changes how speech sounds while preserving its content. Speech-to-speech translation changes the spoken language. A conversational speech model must also decide what to say, maintain state, and often call tools.
At Dasha, we treat the model as one part of the complete voice AI stack. The production system still needs media transport, turn detection, conversation state, tool execution, cancellation, transfers, testing, and diagnostics.
How direct, cascaded, and hybrid speech systems work
Direct audio models
A direct model encodes incoming sound into learned representations or audio tokens. The model uses those representations to understand speech, reason, decide when to respond, and generate outgoing audio. A full-duplex model can listen while it speaks, which supports interruptions, backchannels such as “mm-hmm,” and overlapping speech without forcing every exchange into a rigid turn.
This design can preserve information that a plain transcript drops, including tempo, emphasis, emotion, and non-speech sound. It also removes explicit network handoffs between separate STT, LLM, and TTS services. The tradeoff is a larger opaque boundary. Your application may receive a transcript, yet that transcript may be an auxiliary reconstruction rather than the exact representation the model used.
Cascaded speech-to-speech systems
A cascade exposes the stages:
- Voice activity detection and endpointing decide when speech is present and when an utterance is ready.
- Streaming STT produces partial and final text.
- An LLM or deterministic policy processes the text, maintains state, and calls tools.
- TTS turns approved response text into audio.
- The runtime plays, cancels, or replaces that audio as the conversation changes.
This path is often shown as a serial chain. A production implementation overlaps safe work. STT emits partial text while the user is speaking. The LLM can begin after a stable input arrives. TTS can start from a complete phrase while the rest of the answer is still being generated. The end-to-end result depends on the critical path, rather than the sum of four isolated benchmark numbers.
Explicit stages make provider choice, text policy checks, pronunciation control, and component diagnosis easier. They also create more interfaces to operate. An STT error can flow into the tool call, an LLM can generate text that TTS pronounces poorly, and queued audio can survive an interruption unless the runtime cancels it correctly.
Hybrid systems
A hybrid design uses direct audio where conversational flow matters most and a structured path where exactness or approval matters more. A free-form explanation might remain in the direct session. A payment change, address update, or booking can move into an explicit state machine that confirms each field before a tool runs.
Hybrid systems add their own risk. The direct and structured paths must agree on what the user heard, which values are provisional, and which actions completed. Use one authoritative conversation state and record every transition between paths.
Direct audio vs cascaded pipelines
Direct audio is a good fit for fluid conversation. A cascade is a good fit for visible control. Hybrid design is useful when one workflow needs both.
| Decision factor | Direct audio model | Cascaded pipeline | Hybrid system |
|---|---|---|---|
| Conversational timing | Can model overlap and respond without a hard text boundary | Depends heavily on endpointing and cancellation policy | Can reserve direct audio for turns that benefit from overlap |
| Prosody and emotion | Can reason over signals that transcription omits | Requires separate audio analysis or loses those signals | Keeps audio cues where useful and structured text where required |
| Exact entities | Must be tested at the transcript, spoken response, and tool-argument levels | Intermediate text is visible and can be confirmed before use | Can route identifiers and sensitive fields through confirmation |
| Tool safety | Structured tool calls still need validation outside the model | Policy and schema checks can sit between reasoning and execution | Applies hard controls to consequential actions only |
| Observability | Usually stronger at the session and event level than at each speech component | Each STT, LLM, tool, and TTS boundary can be traced | Requires evidence that joins both paths |
| Provider choice | Model, voice, turn-taking, and tool semantics are often tied to one provider | Components can be replaced independently | Preserves some choice with added routing work |
| Operating burden | Fewer components for the team to connect | More queues, retries, versions, and failure modes to manage | Most state and regression paths to test |
| Best fit | Open-ended assistants, companions, games, and voice-first interfaces | Transactional support, approval-heavy workflows, and reusable text agents | Products mixing natural conversation with sensitive actions |
Direct does not automatically mean faster
Direct models can lower the model-path latency floor and begin audio without waiting for a complete text answer. Caller-perceived delay still includes transport, turn detection, provider queues, tool calls, and playback buffering. A poorly tuned direct session can feel slower than a well-streamed cascade.
Measure from the last audible user speech to the first audible agent audio at the user's endpoint. Track the first substantive response separately so a quick filler cannot hide a slow answer. Our voice AI latency guide gives the full measurement and instrumentation plan.
Transcripts do not solve observability by themselves
A transcript can omit tone, overlap, and audio the user never heard. It can also differ from the representation that drove a direct model's answer. Preserve the incoming and outgoing audio, transcript events, tool arguments and results, conversation state, and playback timeline. That evidence should answer what the user said, what the model inferred, what the user heard, and what changed in the backend.
Early action creates a state-commit problem
Streaming lets a system prepare work before the user finishes. It should not turn an unstable value into an irreversible action. A caller may say, “Book Boston on Friday, actually, make that Chicago on Monday.” The runtime needs a clear distinction between provisional and confirmed fields.
A 2026 full-duplex voice benchmark tested tool use with real human disfluencies and found that mid-utterance self-corrections remained difficult across all evaluated systems. The practical control is model-independent: delay consequential writes until required fields are stable, validate tool arguments outside the model, and use idempotency and confirmation for sensitive actions.
Current speech-to-speech models and APIs
At Dasha, we operate one layer above a foundation model. Our current Dasha capability inventory covers telephony, conversation orchestration, tools, testing, monitoring, and call inspection through a managed production runtime. The model and API landscape below shows the different architecture choices a team may evaluate before deciding how much of the surrounding runtime it wants to own.
| Option | Architecture | Useful fit | Main evaluation question |
|---|---|---|---|
| OpenAI GPT-Realtime-2 | Proprietary direct audio model with text, audio, and image input, audio output, and function calling | Live voice agents already aligned with OpenAI's model and tool ecosystem | Do its tool, transcript, voice, and provider controls meet the workflow's exactness and portability needs? |
| Google Gemini Live API | Proprietary native audio and vision session API with tools and transcripts | Voice-plus-vision interfaces and multilingual live interaction | Does its preview status, session design, and model behavior meet the release and support requirements? |
| Amazon Nova 2 Sonic | Proprietary bidirectional speech-to-speech model with function calling | Teams building inside AWS that want one live speech model | How do session limits, tool handling, language behavior, and AWS dependency affect the product? |
| NVIDIA NemotronLabs VoiceChat 11B | Open full-duplex model released publicly in August 2026 under OpenMDW 1.1, with checkpoint and container workflows for self-deployment | Teams evaluating an open speech model on their own NVIDIA GPU infrastructure | Do its license, GPU requirements, reasoning quality, and operating burden fit production plans? |
| Kyutai Moshi | Open research speech-text model with parallel audio streams | Research, self-hosted experimentation, and full-duplex interaction work | Can the team operate it and close the gaps in tools, safety, languages, and workflow quality? |
| Hugging Face speech-to-speech reference pipeline | Open modular VAD, STT, LLM, and TTS cascade | Local or mixed-hosting deployments that need replaceable components | Is full ownership of model serving, timing, scaling, and diagnostics worth the control? |
These options should form a pilot shortlist, rather than a universal ranking. Direct-model leaderboards can help narrow it, but they do not compare the full operating burden of a cascade or managed runtime. They also reflect one test set, transport, prompt, region, and model version.
Which speech-to-speech architecture should you choose?
Start from the cost of a wrong turn.
Choose a direct audio model when the product depends on expressive speech, fast conversational flow, overlapping talk, or voice-plus-vision context, and the provider exposes enough tools, logs, and controls for the workflow.
Choose a cascaded pipeline when exact transcripts, provider choice, reusable text-agent logic, pre-speech policy checks, custom pronunciation, or component-level traces are primary requirements.
Choose a hybrid system when the conversation can be fluid until it reaches a sensitive action. Define the routing boundary before implementation. If teams invent that boundary case by case, state and test coverage will fragment.
Choose a managed runtime when the product needs custom business logic and production evidence, while owning media buffers, provider queues, interruption cleanup, telephony, and trace assembly would consume engineering time better spent on the product. Choose a modular framework when those low-level controls are part of the product's differentiation.
How to evaluate a speech-to-speech model
1. Set hard gates before scoring
Wrong sensitive actions, unconfirmed writes, missed mandatory handoffs, and data-policy violations should fail a candidate. A lower latency score cannot offset them.
2. Build the test set from the workflow
Use real target-channel audio or representative recordings. Include accents, narrowband phone audio, background noise, long pauses, backchannels, interruptions, self-corrections, code-switching, names, addresses, dates, amounts, and alphanumeric identifiers. Add tool timeouts, rejected calls, duplicate events, and failed transfers.
3. Score the complete system
The Artificial Analysis methodology separates reasoning, conversational dynamics, agentic task performance, and human preference. A production evaluation also needs traceability, recovery, tail latency, and operating cost.
Use this 100-point starting scorecard and change the weights before testing:
| Dimension | Weight | Pass evidence |
|---|---|---|
| Verified task completion | 25 | Correct backend outcome, including every required step |
| Entity and tool correctness | 20 | Final confirmed values match tool arguments and results |
| Speech quality and conversational dynamics | 15 | Clear audio, appropriate pacing, pause handling, interruption handling, and no false barge-ins |
| Caller-perceived latency | 15 | p50, p95, and p99 meet the budget at target concurrency |
| Debugging and replay evidence | 10 | A failed turn can be reconstructed across audio, state, model, tool, and playback events |
| Reliability and recovery | 10 | Timeouts, no-audio turns, retries, transfers, and cancellations follow the defined policy |
| Total cost and switching burden | 5 | Cost per verified outcome includes providers, telephony, failures, and engineering operations |
4. Compare fixed versions under the same conditions
Use the same audio, prompt, tools, region, channel, load, and success definition. Pin the model version when the provider supports it. Report failed and silent turns instead of removing them from latency results.
5. Gate every change with regression tests
Model, prompt, VAD, voice, tool, and network changes can alter the conversation. Build the suite from production failures and run it before a limited rollout. Our voice agent testing guide covers turn-taking, entities, tools, failure recovery, safety, load, and release evidence.
Where Dasha fits
A model can generate speech. A production agent must also handle calls, state, tools, failures, versions, and the evidence required to improve it. Our current product documentation covers the managed voice runtime, REST APIs, and web application we provide for that wider job. Teams can configure agents, models and voices, connect customer webhooks or Model Context Protocol tools, run inbound and outbound conversations, and inspect completed calls. Your team retains responsibility for business rules, backend systems, carrier setup, compliance, and production acceptance.
We are a strong fit when you need a production voice product without operating every runtime seam. A low-level media framework or self-hosted model stack is a better fit when direct ownership of audio buffers, model servers, and transport internals is a product requirement.
Start building with Dasha using one real call path, one business tool, and one measurable outcome.