The voice AI stack: a practical seven-layer model

What is a voice AI stack?

A voice AI stack is the complete system that receives live audio, understands a speaker, decides and acts, returns speech, and records enough evidence to operate the conversation safely.

The familiar diagram shows speech-to-text (STT), a large language model (LLM), and text-to-speech (TTS). Those are three important components. They are only part of the production system. A phone or browser call also needs media transport, turn-taking, conversation state, tool execution, interruption handling, transfers, testing, monitoring, and deployment controls.

For technical teams that want to own their product rather than every infrastructure seam, we recommend starting with our voice AI backend. We manage the voice and conversation runtime and connect it to telephony configured through your Twilio account or SIP provider. Your team owns its carrier setup, business rules, downstream systems, compliance, and production acceptance. Customer-defined tools execute through your webhooks or connected Model Context Protocol (MCP) servers, while our runtime orchestrates the request and processes the conversation data and tool results needed for the interaction. A media framework or fully custom stack is a better fit when direct ownership of each audio primitive is part of your differentiation.

A practical seven-layer model for a voice AI stack

We use the following practical decomposition to reason about ownership, latency, and failure. It is a design tool, not a published industry standard or a representation of Dasha's internal architecture. Treating the entire stack as one black-box request makes diagnosis far harder.

1. Channel and media

The latency clock starts before STT. A telephone call may cross a carrier, a SIP trunk, a media gateway, and a transcoding boundary before your application sees audio. SIP establishes and manages sessions, while the Real-time Transport Protocol (RTP) and its control protocol carry timed media and report reception quality. The SIP standard and RTP/RTCP standard support keeping signaling, media transport, and application behavior as separate concerns.

A browser call has its own network path through Interactive Connectivity Establishment (ICE), Traversal Using Relays around NAT (TURN), and encrypted real-time transport. The WebRTC transport standard reflects how much machinery sits below a browser microphone. TURN is a fallback for difficult network paths, not a relay that every call necessarily uses.

Test the real target channel. Clean local audio does not reveal packet loss, mobile jitter, narrowband codecs, echo, or routing failures. We recommend keeping the incoming and outgoing tracks separate when your channel allows it. That makes transcription, interruption analysis, and debugging clearer.

2. Audio processing and turn-taking

Turn-taking decides when the system may act. A VAD detects speech. An endpointer decides that an utterance is complete. Barge-in logic decides whether new speech should stop the agent.

These decisions are linked. Aggressive endpointing shortens pauses and increases the risk of cutting off a caller who is thinking or reading a number. Conservative endpointing preserves the utterance and adds dead air. A turn-taking study across ten languages found that human speakers generally minimize silence and overlap, with meaningful timing variation across languages. The study did not test voice agents or establish a responsiveness threshold. It supports testing endpointing by language and workflow instead of choosing a universal millisecond target.

Do not give one configuration to every workflow. A restaurant order, an insurance intake, and a technical troubleshooting call have different pause patterns and costs of interruption.

3. Speech understanding

Streaming STT can emit partial hypotheses before it commits a final turn. Partial text helps the runtime prepare and may support disposable, read-only work. We recommend waiting for confirmed input before consequential or irreversible actions. Any speculative result should be discarded when the transcript changes. Otherwise a corrected account number can arrive after a tool call has already used the wrong value.

Word error rate is a broad model measure. We recommend adding entity-level tests because names, dates, amounts, addresses, confirmation codes, and negation often determine whether the task succeeds. Build the speech test set from real channel audio, including accents, crosstalk, background sound, code-switching, and low-volume speakers.

An audio-to-audio model can combine this layer with reasoning and speech generation. For transactional workflows, we recommend keeping an explicit representation of confirmed entities and completed actions in the application state.

4. Conversation runtime

The runtime owns the life of the call. It tracks what the caller has said, which fields are confirmed, what the agent is allowed to do, whether a tool is in flight, and what should happen after an interruption or timeout.

As an illustrative design model, we often sketch a turn state machine like this:

listening → committing input → reasoning or tool wait → speaking → interrupted or listening

This is a practical model, not a standard or a map of Dasha's internal runtime. Actual states depend on the chosen architecture. We recommend recording the time and cause of each transition. When a caller interrupts, the runtime should stop outbound audio through the controls that architecture exposes, preserve valid state, and accept the next utterance. In a chained pipeline, this usually includes canceling queued TTS and marking unplayed words correctly. Otherwise the model may treat unheard words as delivered conversation history.

5. Knowledge, tools, and handoff

Knowledge retrieval supplies approved context. Tools read or change real systems. Keep those concerns separate so a retrieved sentence can never acquire the authority of a tool instruction.

For every tool, we recommend defining:

• an input schema and authentication boundary
• timeout, retry, and cancellation behavior
• an idempotency strategy for writes
• the user confirmation required before a sensitive action
• a result the runtime can validate
• a fallback or human destination

This is also a security boundary. NIST's Generative AI Profile covers risks that include confabulation, data privacy, information security, and unsafe or unauthorized behavior. OWASP's prompt injection guidance directly addresses instructions that change a model's intended behavior. Limit the tools and data exposed to each workflow, validate arguments outside the model, require human approval for sensitive writes where appropriate, and log every attempted action.

A transfer is part of the workflow, not an exception after it. Pass the reason, verified identity, collected fields, actions attempted, and conversation context. Measure whether the destination answered and received that context.

6. Speech output

TTS quality includes pronunciation, prosody, stability, and time to first audio. Evaluate it on the phrases your product will actually say. Product names, alphanumeric codes, street addresses, currencies, and dates expose issues that a polished sample sentence will miss.

In a chained STT-LLM-TTS pipeline, stream complete semantic chunks instead of waiting for the whole answer. Keep chunks short enough to cancel quickly. Normalize text before synthesis and use a pronunciation dictionary for domain terms when the selected TTS supports it. We recommend linking the generated text, synthesized audio, and playback timeline in your diagnostic evidence. Audio-native speech-to-speech systems may not expose a separate text or TTS boundary, so their streaming and interruption controls need a different test plan.

7. Operations and governance

Production ownership begins when the call ends. Your team needs to answer which version ran, what the caller heard, which tools executed, where time was spent, why a transfer occurred, and whether the intended backend state changed.

OpenTelemetry's trace model provides spans, timestamps, attributes, events, and links for an end-to-end request path. It does not prescribe whether a voice system should create one trace for a call, a turn, or a wider workflow. We recommend correlating each turn's channel, transcript, model, retrieval, tool, output, and transfer events inside an end-to-end call trace, or an equivalent structure your team can query.

Build regression suites from real failures. The evaluation guidance supports task-specific tests, log-derived cases, automated measures, human judgment, and continuous evaluation as a system changes. For voice, we recommend release gates that cover normal cases, accents and noise, interruptions, ambiguous entities, tool timeouts, duplicate events, unsafe requests, and failed transfers. The exact cases and pass thresholds should come from your workflow. Our voice agent testing guide gives a fuller test plan.

Choose an architecture by ownership, not by component count

There is no standard four-pattern taxonomy. We use the non-exhaustive categories below to separate component design from operational ownership. A voice-agent architecture guide describes the two technical poles: direct speech-to-speech and a chained STT-LLM-TTS pipeline. The managed, modular, and hybrid labels below form our practical framework for choosing who operates each layer.

We follow the managed-runtime approach. Through our REST APIs and web application, a team can configure agents, available LLMs and voices, add web voice, connect knowledge, orchestrate customer webhook or MCP tools, route human transfers, and start inbound or outbound conversations. Production phone numbers connect through the customer's Twilio account or manually configured SIP credentials. Carrier availability, registration, routing, caller ID, consent, and regulatory requirements remain customer acceptance work.

For a completed call, Call Inspector can show the transcript, LLM interactions, tool executions, timeline, latency breakdowns, and the call recording when recording was configured. Availability of individual fields depends on the call and its configuration. This is diagnostic product evidence, not a standards-format distributed trace.

This fit is strongest for technical teams building a voice product that needs a production runtime and operational evidence. A modular framework remains a reasonable choice when your engineers need direct control of buffers, transports, and model adapters. A packaged, fixed-workflow product may suit a nontechnical operator who wants minimal implementation work.

Build a latency budget around the user's turn

Latency is an end-to-end property. Provider benchmark numbers cannot tell you how quickly your complete system reacts to a real caller.

As a starting instrumentation plan, we recommend recording these timestamps when the chosen architecture exposes them:

1. User speech starts.
2. User speech ends.
3. Input is committed.
4. The LLM or policy engine returns its first usable output.
5. A tool starts and ends, if used.
6. TTS returns its first audio.
7. Playback starts.
8. The user interrupts or the response completes.

Then calculate distributions and tails. The Google SRE monitoring guidance explains why averages can conceal slow outliers and why successful and failed request latency should be separated. For a voice stack, we also recommend segmenting by channel, geography, language, workflow, provider, tool, and release version.

Reduce delay by overlapping safe work. Stream transcripts and audio. Open connections before they enter the critical path. Cache stable context. Keep prompts and spoken answers concise. Start a read-only lookup early only when a changing partial transcript cannot cause the wrong request. Never trade a confirmed transactional input for a faster number on a dashboard.

Model cost per successful outcome

A per-minute rate omits much of the economics. The full cost includes telephony, STT, model input and output, retrieval, tool infrastructure, TTS, the runtime, recordings and logs, failed attempts, human transfers, support, and engineering operations.

Use this decision framework:

cost per successful outcome = total channel + model + platform + human + operating cost / verified successful outcomes

This is a planning measure, not an accounting standard. Label every cost term, observation window, retry, transferred minute, and verified outcome. Define the denominator in backend terms. A booking exists, a payment arrangement is recorded, a ticket contains the required fields, or a qualified call reaches the correct team. A friendly goodbye is not task completion.

Long calls can also grow model input as conversation history accumulates. Track model cost by turn and workflow rather than assuming it rises linearly with call minutes. Include retry traffic, duplicate tool calls, transferred minutes, and retained media in the model.

A production selection checklist

Before choosing providers or a managed runtime, write one target workflow and answer these questions:

• Channel: Which PSTN, SIP, WebRTC, codec, DTMF, and transfer paths are required?
• Conversation: What counts as end-of-turn, a valid interruption, silence, and recovery?
• Accuracy: Which entities must be captured exactly, and how are they confirmed?
• Actions: Which reads and writes are allowed, and where do validation and idempotency live?
• Failure: What happens when speech, a model, a tool, TTS, or a transfer times out?
• Evidence: Can correlated evidence reconstruct what happened across every layer?
• Release: Can you version, test, limit traffic, observe, and roll back a change?
• Data: Who can access transcripts, recordings, prompts, and tool payloads, and for how long?
• Scale: What happens to connection setup, queues, quotas, and downstream tools at peak concurrency?
• Economics: What is the cost per verified outcome at expected and peak traffic?

Run the final design through a production readiness checklist before exposing real customers. No feature list or benchmark makes a system production-ready on its own. Define acceptance criteria for the specific workflow, channel, risk, and traffic profile. The best stack is the one your team can observe, change, and recover, while preserving a responsive conversation.

To evaluate that model without building the runtime from scratch, build with Dasha using one real call path, one business tool, and one measurable outcome.