Voice AI Latency: Set a Real Budget and Hit It

What counts as good voice AI latency?

Voice AI latency is the elapsed time from the end of a caller's speech to the first audible sample of agent audio at the caller's endpoint. This raw caller-perceived turn gap includes a filler or backchannel if that is the first audio. It is also called voice-to-voice response latency or time to first audio (TTFA). Track time to first substantive response audio separately so a quick filler cannot hide a slow answer. Voice AI delay, lag, and voice agent latency usually refer to the raw interval, but every reported result should state its measurement boundary and how fillers are treated.

There is no source-backed universal threshold for "good" voice AI latency. Good latency is short and stable enough for the specific turn, channel, language, caller population, and task. A fast "yes" should arrive sooner than a response that depends on an account lookup. Set a service-level objective (SLO) from production baselines and user tests, then tighten it using completion, repair, interruption, and abandonment data from your own calls.

Human turn timing needs context. A peer-reviewed overview of human communication reports conversational gaps of roughly 200 ms, even though producing a simple one-word utterance takes at least about 600 ms. People achieve this timing by predicting turn endings and planning before the other person finishes. Turn-taking research shows that timing is a coordination problem, rather than a processor-speed test.

There is no culture-free pass line. A study of question responses across ten languages found a shared preference for little silence or overlap, alongside average gap differences within 250 ms of the cross-language mean. It also found that some kinds of responses, including answers that resist a question's premise, are systematically slower. Cross-language findings support a use-case and turn-class budget instead of one universal threshold.

We built Dasha around that end-to-end problem. Our managed runtime handles telephony, speech, tools, and streaming while coordinating model calls as one production path, with call inspection for latency diagnosis.

Voice Benchmark disclosure: Dasha built and maintains Voice Benchmark, provides the Dasha-based testing agent, and is itself tested alongside other providers under the same automated protocol.

The live Voice Benchmark results use two distinct summaries. Current latency is the average response time across turns in a provider's most recent successful automated call test. Median latency is the median of recent test-level averages. The ranking uses the current result as its primary key, not the recent median. Both change as hourly tests run, so the latest-call average and recent-history median should never be treated as the same statistic. The published methodology defines response latency as tested-agent speech start minus testing-agent speech end.

The public benchmark covers inbound, US English phone calls from US-based testing infrastructure. Network conditions, provider load, dialogue variation, and time of day can affect a run. The public data does not expose raw per-turn timestamps or fully specify every entrant's model, voice, prompt, endpointing, phone route, concurrency, or warm state. It also does not fully specify the history window and formula behind every summary metric. Only providers with a recent successful test qualify for ranking. Treat the results as a live comparison under this standardized public scope, rather than a universal ranking of every configuration or workload.

Define the boundary before comparing numbers

"Latency" can describe several incompatible measurements. Label the boundary every time you report a result.

Caller-perceived turn latency is the main experience metric:

first audible agent sample at caller - last audible caller sample

Runtime turn latency removes the public voice legs and measures between the media ingress and egress boundaries of your system. It is useful for diagnosing the platform, but it is smaller than what the caller experiences.

Component latency measures one operation, such as LLM time to first token (TTFT) or text-to-speech time to first audio (TTFA). It helps locate a bottleneck. It cannot prove that the complete voice agent is fast.

A text-to-speech provider may report a low TTFA from request receipt to its first returned audio chunk. The caller still waits for turn detection, transcription, model generation, orchestration, network transport, and playout buffering. Calling the component number "voice AI latency" erases most of the turn.

Telephony delay is another common category error. ITU-T G.114 addresses one-way network transmission time, not the silence between a caller finishing and an AI agent beginning its reply. ITU-T G.114 is relevant to the transport legs, but its transmission thresholds are not end-to-end voice-agent response targets.

Where delay enters the voice pipeline

A cascaded voice agent usually moves through speech-to-text (STT), reasoning and tools, then text-to-speech (TTS). Real pipelines overlap some of this work. The diagnostic model still needs every boundary.

Streaming changes the arithmetic. Interim transcription happens while the caller is speaking. TTS can start after a stable phrase while the LLM continues generating. Tool preparation can sometimes begin from a stable intent before the final transcript arrives.

Because stages overlap, the end-to-end result is the critical path, not the sum of every component duration. The same rule applies to percentiles. Adding the p95 for STT, LLM, and TTS does not produce end-to-end p95 because each stage's slowest five percent may occur on different turns. Measure the complete turn distribution directly.

How to measure voice AI latency correctly

1. Record from the user's side

Use a two-channel recording or a test caller that captures sent and received audio on one clock. This produces the cleanest start and stop markers. Server events alone cannot reveal the caller's uplink, downlink, or playout delay.

Mark the end of audible speech, excluding trailing room noise. For the raw caller-perceived gap, mark the first audible agent sample and state whether your detector counts a click or breath as speech. Mark the first audio that advances the answer as a separate substantive-response event. A filler or backchannel can satisfy raw TTFA, but it cannot satisfy the substantive-response metric.

2. Add trace events at every system boundary

Give every call and turn a trace ID. Capture these events with monotonic durations inside each service:

• final caller speech sample received
• endpoint committed
• stable transcript available
• LLM request sent and first token received
• each tool request and response
• first text committed to speech
• TTS request sent and first audio chunk received
• first audio chunk sent to the media layer
• playback stopped after an interruption

Dasha's Call Inspector brings transcripts, LLM activity, tool executions, timeline events, and STT, LLM, and TTS latency breakdowns together for completed calls. Recordings are available when recording is enabled. That evidence makes a slow turn diagnosable without reconstructing it from unrelated provider dashboards.

3. Keep one canonical end-to-end metric

Distributed traces explain the turn, while caller-side audio decides the experience. Use synchronized clocks if you compare absolute timestamps across machines. Prefer a monotonic clock for local duration measurements so a wall-clock correction cannot create a negative or inflated span.

For browser calls, WebRTC exposes round-trip, jitter-buffer, packet-loss, and playout statistics. The WebRTC statistics specification is a W3C Candidate Recommendation Draft that defines those fields. They remain transport diagnostics and do not replace an audio-to-audio turn measurement.

4. Segment turns before calculating percentiles

At minimum, split results by:

• tool-free, retrieval, write-action, and transfer turns
• first turn versus later turns
• phone carrier, browser, and other supported channels
• caller region and runtime region
• language and speech condition
• model, voice, prompt, and agent version
• normal load, peak load, and cold start

A fast greeting can hide a slow account lookup. A global p50 can hide one region's poor p95. Preserve a representative rollup, then keep the slices needed to locate regressions.

5. Report the distribution and failures

For each slice, report p50, p95, and p99, plus the sample count. Pair them with timeout rate, technical failure rate, and turns that never produced audio. Do not silently remove retries, canceled model calls, or failed tool turns.

Track at least two other timing metrics:

• Interruption cutoff latency: caller speech start to the agent audio stopping.
• False endpoint rate: turns where the agent started speaking before the caller had finished.

A low response number paired with frequent false endpoints describes an impatient agent, not a responsive one.

6. Join latency to outcome quality

A release gate should protect task success, critical-entity accuracy, tool correctness, repair rate, and policy compliance alongside latency. Our voice agent evaluation guide shows how to keep these dimensions separate so a faster average cannot hide a broken workflow.

Consider this illustrative trace:

The caller-perceived latency is 755 ms. Endpointing consumes 240 ms, the transcript stabilizes 35 ms later, and media egress adds the final 90 ms. Optimizing the 35 ms transcript-finalization span would have little effect. Reducing endpoint delay without raising false endpoints, or reducing reasoning TTFT, offers more value.

How to reduce voice agent latency

1. Fix turn detection first

A voice activity detector (VAD) answers whether speech is present. It does not know whether the speaker has finished a thought. A fixed silence threshold will cut off reflective callers when it is too short and make every turn sluggish when it is too long.

Combine acoustic speech detection with lexical, semantic, or prosodic end-of-turn signals. Tune the policy by language, noise condition, and turn type. Test hesitations, self-corrections, spelled identifiers, dates, and phrases such as "let me think". Gate the change on both response latency and false endpoints.

2. Stream the full path

Use streaming STT so the agent has stable partial text before the utterance ends. Stream LLM output, then send a stable clause or phrase to TTS without waiting for the whole answer. Start the next stage when its input is safe to commit, rather than when the preceding stage has completely finished.

Streaming creates cancellation work. If the caller interrupts, stop playback, cancel or discard outstanding generation, and prevent stale audio from reaching the next turn. Speculative work only helps when the system can retract it safely.

3. Remove avoidable hops and buffering

Co-locate the runtime and latency-sensitive dependencies where provider regions allow it. Reuse secure connections. Avoid a chain of gateways that decode, re-encode, and buffer the same audio. Inspect jitter and playout configuration before shrinking buffers, since an undersized buffer trades delay for dropouts.

The best region is the one that shortens the whole path for the caller population. Placing the application beside the LLM can still be a poor choice if it sends every phone call and speech request across an ocean.

4. Shorten the reasoning path

Route simple turns to a model that meets their capability requirements. Keep the active prompt and conversation state concise. Remove duplicate instructions and irrelevant retrieved context. Ask the model for speech-sized answers so TTS receives a stable first phrase early.

For tools, measure connection setup, queueing, server work, retries, and payload parsing separately. Cache safe reference data with an explicit freshness policy. Never precompute a personalized fact or a write-action confirmation before the underlying tool has succeeded.

5. Engineer the tail

Warm the services that otherwise cold-start. Put limits on queues and retries. Apply backpressure before the system accepts more concurrent work than it can serve within the SLO. Set timeouts by dependency and define a caller-safe recovery path.

Load tests should reproduce call concurrency, turn rate, audio duration, tool mix, model mix, and provider limits. A single developer call says little about p95 production latency.

6. Treat progress cues as conversation design

A short, truthful phrase such as "I'll look that up" can confirm that a long tool turn has begun. It does not reduce the time to the answer, so report substantive-response latency separately. Repeating filler on every turn increases handle time and sounds evasive.

Pre-synthesized audio is appropriate for deterministic content such as a greeting or a fixed transition. Dynamic amounts, dates, account status, and success confirmations should wait for current data.

Fast components can still produce a slow or brittle agent

Several optimizations improve one number while damaging the call:

• A shorter silence threshold lowers endpoint delay and raises cutoffs for callers who pause mid-sentence.
• Acting on unstable transcripts starts work earlier and can send the agent down the wrong branch.
• A smaller model may lower TTFT and lose critical-entity or tool accuracy.
• Tiny media buffers reduce playout delay and increase glitches under jitter.
• Immediate filler audio improves perceived responsiveness and can conceal a slow dependency.
• A component benchmark isolates one service and says little about the carrier-to-caller path.

Native speech-to-speech models can remove explicit STT and TTS boundaries. Cascaded pipelines make transcripts, provider choice, tool control, and component diagnosis easier. Either architecture can be fast or slow once telephony, tools, safety checks, load, and interruption handling enter the path. Choose against the operational requirements, then benchmark the finished agent.

A production latency release gate

A usable gate states the metric, slice, percentile, sample, and quality guardrail.

Version the test set and compare a candidate with the current production baseline under the same conditions. Release only when the latency distribution improves or stays within its allowed margin and every hard quality gate passes.

Why does a voice agent feel slow when its APIs are fast?

The missing time is usually before, between, or after the API calls. End-of-turn waiting, transcript finalization, serial orchestration, tool calls, cross-region travel, queueing, audio encoding, and caller playout can exceed the measured STT, LLM, or TTS time. A caller-side recording aligned with a component trace reveals the actual critical path.

Can voice AI have zero latency?

No. Audio must travel, the system must decide that a turn is complete, and it must produce playable speech. A system can begin speculative work while the caller is talking, but incorrect prediction creates interruptions and wrong responses. The useful goal is a tight, stable latency distribution that preserves task accuracy and turn-taking quality.

Build for the whole turn

Latency belongs to the complete conversational runtime. Set an audio-to-audio SLO, expose the critical path, test the tail under load, and keep speed tied to task and turn-taking quality. We give technical teams the managed runtime, APIs, telephony, testing, and call-level evidence to do that without operating a patchwork voice stack. Start building on Dasha.