Voice AI infrastructure: what a production system must own

Voice AI infrastructure in one minute

Voice AI infrastructure is the complete system that carries a live conversation from a phone or browser through speech processing, agent reasoning, business tools, and speech output, while preserving security, state, reliability, and diagnostic evidence.

A production design has three interacting planes:

The central architecture decision is where to draw ownership and failure boundaries. Fast models cannot compensate for a media path that drops audio. Accurate speech recognition cannot repair a duplicated payment. A detailed transcript cannot explain a failure unless it connects to the exact agent version, provider events, tool calls, and final business state.

Our voice AI stack guide owns the detailed component map. Production infrastructure adds the contracts that make those components one operable system.

Start with a session contract

Every call should enter the runtime with a small, authoritative session envelope. This is the contract that the media, agent, tools, and telemetry layers share. It should include:

• a unique conversation and trace identifier;
• the authenticated tenant, environment, region, and agent version;
• the channel, codec, and media capabilities;
• recording, retention, consent, and transfer policy;
• tool permissions and data-access scope;
• deadlines, retry limits, and fallback policy; and
• the terminal outcome and reason when the session ends.

The model can read selected fields from this envelope. It should not choose its own tenant, authorization scope, or retention policy. Those values come from trusted control-plane configuration and authenticated ingress.

This contract creates a stable spine through an otherwise probabilistic system. It also gives each layer enough context to reject an invalid request instead of guessing.

1. Put a clear boundary at the telephony edge

The telephony edge connects a carrier, private branch exchange, browser, or mobile client to the real-time runtime. For phone calls, Session Initiation Protocol (SIP) usually handles call setup, modification, and termination, while Real-time Transport Protocol (RTP) carries audio. The SIP specification and RTP specification separate those signaling and media concerns for a reason: either can fail while the other appears healthy.

A production edge should own:

• inbound authentication and routing;
• normalized call lifecycle events;
• phone-number, trunk, and tenant mapping;
• codec negotiation and bounded transcoding;
• region selection and admission control;
• hangup, transfer, and timeout behavior; and
• correlation between carrier identifiers and the internal conversation ID.

Treat browser voice as another channel, rather than a clean substitute for phone testing. Web Real-Time Communication (WebRTC) uses a different connection setup, media path, device stack, and network environment. A design that works on office Wi-Fi may still fail on narrowband phone audio, mobile jitter, carrier timeouts, or a live transfer.

Carrier failover also needs an honest contract. New calls can often be sent through another route after a provider or region is marked unhealthy. Moving an established call without losing media or state is a much harder problem. For many systems, reliable failover means draining one route and protecting new admissions while existing calls finish or follow a defined transfer or termination policy.

2. Stream media as a full-duplex system

Full duplex means both sides can speak and hear at the same time. The media layer must continuously ingest caller audio, feed speech processing, emit agent audio, and stop that output when the caller interrupts.

The main design concerns are:

• Jitter and packet loss: A jitter buffer smooths irregular packet arrival, but too much buffering adds delay.
• Codec conversion: Repeated decoding, resampling, and re-encoding can add latency and reduce recognition quality.
• Backpressure: A slow consumer must not create an unbounded queue of audio or generated speech.
• Cancellation: An interruption must flush audio waiting in every downstream buffer, not only stop the text generator.
• Clocking: Media timestamps and system clocks must be good enough to reconstruct the order of speech, model, tool, and playback events.

WebRTC's standard statistics model exposes transport signals such as packet loss, jitter, round-trip time, and audio level. The WebRTC statistics specification is a useful reference even if the production channel is telephony, because it shows the level of media evidence an operator needs.

Latency should be measured across the complete caller-perceived turn, then attributed to media, endpointing, recognition, reasoning, tools, synthesis, and playback. Our voice AI latency guide covers that measurement method in depth.

3. Treat turn detection as state control

Voice activity detection (VAD) estimates whether an audio segment contains speech. Endpointing decides that the user has yielded the turn. Barge-in is the policy that lets new user speech stop the agent. These are related controls with different jobs.

When a caller interrupts, the system has to reconcile four states:

1. what text or audio the model generated;
2. what audio entered the playback queue;
3. what the runtime sent toward the caller; and
4. what conversation history should influence the next turn.

If the runtime records the complete generated answer as spoken, the agent may later assume the caller heard words that were cancelled. If it cancels every in-flight task, it may abandon a business action that already reached the system of record.

Use a turn controller with explicit events for speech start, provisional endpoint, committed endpoint, playback start, interruption, and playback stop. Tune the policy by channel and use case. A caller spelling an account number needs different endpoint behavior from a caller answering yes or no.

4. Choose how speech becomes meaning

Automatic speech recognition (ASR) turns audio into text. A streaming recognizer emits partial hypotheses before producing a stable result. The runtime must distinguish tentative text from committed input, especially for names, dates, addresses, and identifiers that can change late in the utterance.

Speech recognition infrastructure should expose:

• partial and final transcript events with timestamps;
• confidence or stability signals where available;
• channel, language, and acoustic configuration;
• recognition errors and stream restarts; and
• critical entities in a form that downstream validation can check.

The broader system can then use one of three speech architectures:

A direct audio model still needs trusted application state, tool authorization, interruption handling, and outcome evidence. A cascaded pipeline exposes more intermediate events, while also creating more places for timeouts and mismatched state. Our speech-to-speech model guide owns the full architecture comparison and evaluation method.

5. Make the agent runtime the consistency layer

The model proposes language and actions. The runtime owns execution.

That runtime should coordinate:

• the session event loop and conversation state;
• model requests, streaming output, and cancellation;
• prompt and policy assembly from approved versions;
• tool scheduling, timeouts, and results;
• output filtering and speech commit;
• handoff and terminal outcomes; and
• correlated events for the operational record.

Keep the live execution path separate from the control plane that creates agents, changes configuration, schedules work, and reviews completed sessions. A control-plane slowdown should not stall live audio. A compromised live session should not gain permission to rewrite its own deployment policy.

Context sent to a model is also different from durable state. Conversation history helps the next response. It is not the authoritative record of whether a refund, reservation, or account update succeeded. Our AI agent runtime guide explains state, durable execution, sandboxing, and control-plane separation in more detail.

6. Put tools behind a strict trust boundary

Tools connect the agent to customer data and business systems. They turn a conversational error into a possible operational or security incident, so the runtime should treat every tool call as untrusted input.

Require:

• typed names, arguments, and result schemas;
• server-derived tenant and user scope;
• least-privilege credentials for each action;
• idempotency keys for repeatable writes;
• deadlines and bounded retries;
• explicit confirmation for consequential actions; and
• a reconciliation path for timeouts and partial success.

Idempotency means the same requested operation can be retried without creating a second effect. It matters because a timeout only proves that the caller did not receive a response. The downstream system may have completed the operation.

Separate the kinds of state the infrastructure holds:

Never announce success from a planned tool call. Speak the result only after the system of record confirms it, or describe the action as pending when the workflow is genuinely asynchronous.

7. Make speech output cancelable and truthful

Text-to-speech (TTS) turns response text into audio. In a live system it also needs streaming, pronunciation control, fast cancellation, and a clear failure policy.

The output path should:

• begin synthesis from stable response chunks;
• avoid splitting numbers, names, and tool results at unsafe boundaries;
• flush queued audio on barge-in;
• record which chunks were generated and sent;
• wait for confirmed business state before speaking success; and
• switch to a fallback voice or message only at a tested boundary.

Caching can help with fixed greetings and disclosures. It is risky for personalized or policy-sensitive content unless cache keys include every value that changes the spoken result. A cross-tenant audio cache is a data-isolation defect.

Build observability around one conversation timeline

Voice-agent observability is the ability to reconstruct what the user experienced, what the system decided, what it changed, and why. Logs alone are insufficient when each provider uses different identifiers and clocks.

Use one conversation ID to connect:

• call signaling and media-quality events;
• caller speech, endpoints, interruptions, and playback;
• transcript revisions and recognized entities;
• model version, request, first output, finish reason, and usage;
• tool input, authorization scope, retries, result, and downstream record;
• agent, prompt, voice, and policy versions;
• transfer or hangup reason; and
• final product outcome.

Distributed tracing represents work as related spans and events. The OpenTelemetry trace model provides a useful implementation vocabulary, but the requirement is the correlated evidence, rather than a specific telemetry vendor.

Keep four signal types distinct:

• Metrics show rates, latency distributions, queue depth, capacity, and error budgets.
• Traces explain one conversation across services and providers.
• Logs record detailed service and policy events.
• Artifacts include audio, transcripts, model exchanges, and tool payloads under stricter access and retention rules.

Alert on user-visible symptoms and exhausted capacity, rather than every provider error. A retried model request that still produces a timely correct turn is diagnostic evidence. Silent audio, an incorrect action, or a growing admission queue is an operational incident.

Isolate failures before adding failover

Failover without isolation can spread an outage. Automatic retries can overload a struggling dependency, while a provider switch can introduce a new voice, transcript format, tool schema, or conversation-state mismatch.

Use separate capacity pools, bounded queues, deadlines, circuit breakers, and concurrency limits around each external dependency. Circuit breakers temporarily stop calls to a dependency that is failing. Bounded queues prevent stale work from consuming memory and delaying healthy sessions. The Amazon Builders' Library guide to timeouts and backoff explains why retries need limits, backoff, and jitter.

Define recovery per failure domain:

Fallbacks should be tested as real product variants. A backup recognizer can tokenize entities differently. A backup model may lack the same tool behavior. A new voice can change pacing enough to expose endpointing problems. Route new sessions first, then expand only when the fallback passes the same release gates as the primary path.

Enforce tenant and data isolation at every boundary

Multitenancy means one system serves multiple customer organizations while keeping their identities, configuration, capacity, and data separate. A tenant identifier in a database row is only one part of the design.

Production isolation should cover:

• Identity: Bind the tenant to the authenticated API key, signed client token, phone route, or trusted server context.
• Authorization: Check every agent, call, recording, transcript, tool, knowledge source, and configuration object against that tenant.
• Secrets: Separate provider and tool credentials, with independent rotation and audit history.
• Data: Partition operational records, retrieval indexes, object storage, caches, exports, and deletion jobs.
• Capacity: Apply per-tenant quotas so one traffic spike cannot consume the shared service.
• Telemetry: Prevent tenant data from entering shared labels, dashboards, or support views without access checks.
• Policy: Apply recording, retention, regional, and redaction rules before data reaches an incompatible store or provider.

Security boundaries should follow resources and actions, rather than trust everything inside one network. The US National Institute of Standards and Technology (NIST) zero trust architecture describes this resource-centered model: access is evaluated for the specific subject, asset, and request.

Keep provider egress explicit. Know which audio, text, prompts, tool results, and metadata leave the runtime, under which tenant policy, and with which credentials. Redact or omit data before transmission when the provider does not need it.

Scale by workload shape

Concurrent calls are only one capacity measure. A useful model includes:

• new call attempts per second;
• simultaneous media sessions;
• audio bandwidth and transcoding load;
• turns, model tokens, and synthesized audio per second;
• tool and database requests per turn;
• transfer and webhook bursts; and
• recording, transcription, and post-call work after hangup.

Separate admission, live execution, and post-call processing so each can scale and fail independently. Protect active conversations before accepting more work. Autoscale on leading indicators such as admission queue delay, reserved session slots, active streams, and provider quotas, alongside CPU and memory.

Regional pools reduce network distance and create useful failure boundaries. Session state should stay region-affine during a call. The control plane can direct new sessions to healthy capacity, while asynchronous records replicate according to the product's consistency and residency rules.

Load tests need real workload proportions. A thousand silent connected calls do not exercise recognition, model streaming, speech synthesis, tools, or interruption cancellation like a thousand active conversations.

Choose an ownership model deliberately

Architecture labels hide operating work. Compare options by the responsibilities your team retains.

A managed platform fits teams that want to own the customer workflow and business systems while delegating the live runtime. A framework fits teams whose product requires low-level pipeline control and who can operate the surrounding services. A model-first design fits deliberate provider alignment. A custom stack fits hard requirements for infrastructure or media ownership that justify the ongoing engineering cost.

Our voice agent framework comparison evaluates the managed, framework, and SDK paths against their production responsibilities.

Use a failure-first evaluation checklist

A greeting and one successful tool call prove the happy path. A useful evaluation forces the system to preserve state and evidence when dependencies fail.

Media and turn-taking

• Run the intended phone and browser paths separately.
• Include packet loss, jitter, narrowband audio, background speech, and a live transfer.
• Interrupt during generation, synthesis, playback, and a tool call.
• Confirm that cancelled speech does not enter the next turn as heard context.

Speech and agent behavior

• Test corrections, spelling, long pauses, backchannels, dates, names, and identifiers.
• Compare direct, cascaded, or hybrid paths on the same task and channel.
• Gate task success and critical-entity accuracy separately from voice quality.
• Record the exact model, prompt, voice, and turn-policy versions.

Tools and state

• Exercise a read, a consequential write, and a human handoff.
• Inject timeouts, duplicate delivery, malformed results, and partial success.
• Prove idempotency against the real system of record.
• Confirm that spoken outcomes match committed business state.

Reliability and scale

• Fail each carrier, speech, model, tool, storage, and region dependency in isolation.
• Check admission control, bounded queues, circuit breakers, and recovery messages.
• Load the actual turn and tool mix, rather than idle sessions.
• Demonstrate how the team disables a bad version and drains unhealthy capacity.

Security and operations

• Attempt cross-tenant access through every object and integration type.
• Trace provider egress, secrets, recording policy, retention, and deletion.
• Reconstruct one failed call from ingress to final business outcome.
• Assign an owner and runbook to every remaining production boundary.

The winner is the architecture that meets the product's outcome, safety, and recovery gates with an operating burden the team can sustain.

Where Dasha fits

We built Dasha for technical teams that want a managed production runtime and operational surface while keeping control of their agent behavior, tools, and business systems.

The current platform provides a web application and REST APIs for building and running voice agents. Its deployment documentation covers phone deployment and SIP configuration, while the WebSocket documentation covers real-time web voice and chat. Agents can connect to external systems through tools and functions, and completed calls can be examined in Call Inspector.

Dasha is a strong fit when:

• phone or web voice is part of a serious product;
• your team wants a managed live runtime instead of assembling every media and provider seam;
• business tools, product policy, and customer data stay under your application's control; and
• testing, call inspection, activity history, and production operation matter beyond the first demo.

A framework-first or custom stack is a better fit when source-level control of codecs, media buffers, transport behavior, or every runtime component is itself a product requirement. A direct model API may be enough when the workflow is narrow and your team already owns the channel, state, tools, observability, and incident response around it. Dasha is also unnecessary for a short-lived prototype with no production operating requirement.

Use the Dasha getting-started path to run one real call path with one business tool, then inject an interruption and a tool timeout. That small failure-first pilot will show whether the ownership boundary fits your product.