A voice agent can feel responsive on most turns and still fail users often enough to matter. P99 latency exposes that slow tail. Measuring it well requires a precise event pair, a representative population, aggregatable telemetry, and the traces behind the slowest turns.
What p99 latency means
P99 latency is the 99th-percentile cutoff of a set of durations. About 99% of the observations are at or below that value, and about 1% are above it. If a voice-agent cohort has a p99 of X ms, about 1% of the measured durations were above X ms during the stated measurement window.
P99 describes the slow tail of a distribution. The maximum remains a separate measurement, and p99 is never a universal performance target. Its meaning depends on what was timed, which observations qualified, and the conditions under which they were collected.
One common calculation is the nearest-rank method:
rank = ceil(0.99 × N) p99 = sorted_durations[rank - 1]
With 10,000 eligible turns, the rank is 9,900. About 100 turns sit above that observation. With fewer than 100 observations, nearest-rank p99 collapses to the maximum. That is one reason a low-volume p99 jumps when a single slow turn enters or leaves the window.
Percentile libraries can use different interpolation rules, especially on small samples. The National Institute of Standards and Technology (NIST) documents several methods that produce slightly different estimates. Pick one implementation, record it with the metric, and avoid comparing results calculated by incompatible methods.
| Statistic | What it answers | What it can hide |
|---|---|---|
| Mean | What was the arithmetic average? | The shape of a skewed distribution |
| p50 | How long did a typical turn take? | Every turn in the slower half |
| p95 | Where did the slowest 5% begin? | The most severe 5% |
| p99 | Where did the slowest 1% begin? | The ordering and severity inside that 1% |
| Maximum | What was the single slowest observation? | Whether the event was common or isolated |
P50, p95, and p99 work as a set. P50 describes the ordinary path. P95 shows whether slower behavior has become noticeable across a broader slice. P99 exposes rare paths, saturation, retries, and other tail events that typical-turn metrics miss.
Decide what duration p99 describes
A percentile has no operational value until the event pair is explicit. Voice systems contain several plausible clocks, and they answer different questions.
The primary user-visible measurement is:
first audible playback at the user - last user speech sample
This is end-to-end turn latency. It includes endpointing, transcript commitment, model and tool work, speech synthesis, network delivery, buffering, and playback start. The last speech sample can require a two-channel recording or an offline audio detector because the runtime learns that speech ended only after it has waited for evidence.
A second useful measurement starts later:
first text-to-speech (TTS) audio byte at the runtime - endpoint decision
This decision-to-first-byte duration helps diagnose the runtime after the system has committed the user's turn. It excludes the time spent deciding that the user finished and stops before client delivery and playback. It will normally be smaller than the end-to-end user-visible value.
Keep the names distinct. Time to first audio byte does not establish when a user heard audio. Language-model time to first token (TTFT) covers one component and does not include endpointing, tools, TTS, transport, or playback.
Before calculating p99, write a metric contract:
| Contract field | Example definition |
|---|---|
| Event pair | Last speech sample to first audible playback |
| Observation unit | One completed user-to-agent turn |
| Eligible population | Production phone turns expected to produce agent speech |
| Failure treatment | No-audio and timeout turns count as objective misses and remain visible separately |
| Cohort | Channel, caller region, runtime region, turn class, model and agent version |
| Window | Fixed release-test run or named rolling production window |
| Sample volume | Eligible turns, successful audible turns, and failures |
| Conditions | Concurrency, warm or cold state, carrier route, provider configuration, and tool path |
Do not merge materially different paths for convenience. Phone and browser audio, warm and cold turns, tool-free and tool-using turns, and distant regions can have different distributions. A fleet-wide rollup is useful only beside the cohorts that explain it.
Our broader voice AI latency guide covers latency budgets and whole-pipeline optimization. The rest of this guide stays on percentile instrumentation, aggregation, and tail diagnosis.
Instrument one correlated trace per turn
Create a root span or equivalent record for every user-to-agent turn. Attach call_id, turn_id, channel, region, turn class, agent version, model, provider, route, and warm-state attributes. Keep unique IDs in traces or logs rather than metric labels, where they would create unbounded cardinality.
Record these events when the architecture exposes them:
| Event | What it establishes |
|---|---|
user_speech_end | Last measurable speech sample in user-side or ingress audio |
endpoint_committed | Runtime decides that the user turn is complete |
input_committed | Final transcript or semantic input becomes safe for the response path |
llm_first_text_token | First text token arrives from the large language model (LLM) |
tool_started and tool_finished | External action, retrieval, retry, and queue timing |
tts_input_committed | First speakable text enters synthesis |
tts_first_audio_byte | First generated audio reaches the runtime |
client_first_audio_received | First response audio reaches the client or media edge |
playback_started | First response audio becomes audible |
playback_completed | Optional full-response and buffering analysis |
Native speech-to-speech systems may lack explicit transcript and TTS stages. They still need the outer user_speech_end, turn-decision, first-output-audio, client-receipt, and playback events. Use architecture-specific internal events between those stable boundaries.
OpenTelemetry spans provide start and end timestamps, attributes, and timestamped events. Propagate one trace context across speech, orchestration, tools, model, synthesis, and media services so that the evidence remains correlated to the same turn.
For completed Dasha calls, our Call Inspector brings transcripts, audio when recording is enabled, model activity, tool executions, timeline events, and latency breakdowns into one inspection view. Your user-side measurement remains the authority for caller-perceived playback.
Handle clocks and trace sampling deliberately
Use a monotonic clock for durations inside one process. Wall clocks can step during synchronization and create negative or inflated spans. Cross-machine event comparison requires synchronized clocks, while a caller-side recording or client instrument remains the authority for the final audible boundary.
Record the end-to-end duration into a histogram for every eligible turn that reaches the stop event. Maintain total-eligible, no-audio, and timeout counters outside the histogram so failures remain in the objective denominator. Sampled traces support diagnosis, while a biased trace sample cannot serve as the denominator for a fleet percentile. Head sampling decides before the system knows whether a trace will be slow. Tail sampling can retain traces based on overall latency or errors after spans arrive, which makes it useful for investigating the tail. Keep an unbiased sample of normal turns for comparison and monitor the sampler itself.
Calculate p99 without breaking the math
There are two sound calculation paths.
Calculate an exact sample percentile from turn rows
For a release test or a bounded analysis window:
- Select the eligible cohort once.
- Calculate one canonical end-to-end duration per turn from its correlated event pair.
- Preserve timeout and no-audio outcomes as failures. Do not let a missing stop event silently improve the result.
- Sort the successful durations and apply the chosen quantile method.
- Report p50, p95, p99, the maximum, eligible count, audible-success count, and failure count together.
Raw turn rows let you retrieve the actual slow traces. They also avoid estimation error from histogram buckets. Retain them according to your privacy and data-retention controls.
Calculate an aggregatable percentile from histograms
Histograms count observations in latency buckets. They can be combined across compatible instances before calculating a fleet quantile. Summaries that export a precomputed quantile cannot be combined this way.
A classic Prometheus query for p99 by runtime region looks like this:
histogram_quantile( 0.99, sum by (le, runtime_region) ( rate(voice_turn_audible_latency_seconds_bucket{ channel="phone", turn_class="no_tool" }[30m]) ) )
The important order is aggregation first, quantile second. Averaging instance p99 values gives every instance equal weight regardless of traffic and discards the underlying distribution. Averaging hourly p99 values creates the same problem across time. Aggregate compatible bucket counts over the intended population and window, then calculate the quantile once.
Bucket design determines accuracy. A classic histogram estimates the quantile within the bucket containing it, so a wide bucket can make the p99 estimate too coarse for a narrow objective. Place useful boundaries around the latency range and objective you need to distinguish. Keep bucket layouts compatible across instances. Prometheus explains the tradeoffs among classic histograms, native histograms, and non-aggregatable summaries.
A direct objective calculation can be better than alerting on an interpolated p99. Put a bucket boundary at the approved latency threshold, divide the count at or below that boundary by the eligible-turn count, and alert on the resulting compliance ratio. This answers the service-level-objective question without estimating where the percentile falls inside a bucket.
Never add component p99 values
Suppose endpointing, model TTFT, TTS first-byte, and playback each have a p99. Their slowest 1% can occur on different turns. Adding those four values constructs a synthetic turn that may never have existed.
Keep correlated per-turn traces, calculate the actual end-to-end duration for each turn, and then calculate p99 across those durations. When stages overlap, the user-visible latency follows the critical path. Summing overlapping spans double-counts time.
Component percentiles remain useful diagnostics. Compare them by cohort and link them back to the end-to-end turn. Treat them as clues about where tail time accumulates, never as substitutes for the complete-turn distribution.
Interpret p50, p95, and p99 together
The gaps between percentiles often say more than one value alone.
| Observed pattern | Likely interpretation | Next comparison |
|---|---|---|
| p50, p95, and p99 rise together | A broad path slowed | Deployment, model, network, or load change across all turns |
| p50 stays stable while p95 and p99 rise | An intermittent slow path affects a meaningful slice | Queue time, tool class, route, retry, cold state, or region |
| p50 and p95 stay stable while p99 rises | A rarer condition entered the slowest 1% | Individual tail traces and low-frequency cohort tags |
| p99 rises as concurrency rises | Saturation or queueing is a candidate cause | Queue depth, utilization, provider limits, and admission rate |
| End-to-end p99 rises while component p99 values look stable | Missing boundary or cross-service delay | Endpointing, orchestration gaps, network transit, and playback buffer |
| p99 falls while no-audio or timeout rate rises | Slow failures may have disappeared from the duration sample | Eligibility, failure counters, cancellation, and timeout policy |
These patterns form investigation hypotheses. They do not prove a root cause. For example, concurrency and p99 may rise together because both follow traffic mix. A cohort comparison and trace evidence are still required.
Sample volume belongs beside every chart. A p99 based on 200 turns is determined by observations near the very top of a small sample. Extend the window when the cohort remains operationally comparable, show uncertainty where the decision warrants it, and use p95 as an additional, more stable tail view. Never hide a low count behind a smooth line.
Turn a latency requirement into a service-level objective
A service-level indicator (SLI) defines the measurement. A service-level objective (SLO) defines the desired range for that indicator. Google's site reliability engineering (SRE) guidance recommends stating the population, measurement conditions, and window so the objective drives a clear action.
An illustrative voice-agent objective can use this structure:
Over a rolling 28-day window, at least 99% of eligible no-tool U.S. phone turns produce first audible agent audio within the team's approved threshold after the last user speech sample. No-audio and timed-out turns count as misses.
The 28-day window and the threshold are planning choices in this example. Your team should choose them from user research, production baselines, turn classes, risk, and the cost of meeting the objective. There is no universal good p99 for every voice agent.
A complete objective should specify:
- the exact start and stop events;
- the eligible denominator and exclusions;
- the treatment of timeout, cancellation, transfer, and no-audio turns;
- channel, region, turn class, and other protected cohorts;
- the measurement window and minimum evidence required for a decision;
- the latency threshold and allowed miss rate;
- the response when the objective is at risk; and
- quality guardrails such as false endpoint, repair, interruption, and task-success rates.
A faster endpointing policy can reduce latency while increasing premature turn commits. A lower p99 does not justify more interruptions or failed tasks. Keep latency and quality as separate release gates. Our voice agent testing guide shows how to exercise the full deployed channel before release.
Diagnose the slowest voice-agent turns
A p99 chart tells you that a tail exists. The correlated turns explain it.
1. Retrieve the actual tail population
For the same cohort and window used by the metric, sort raw end-to-end durations and select the slowest 1%, including every turn tied at the cutoff. If raw durations are unavailable, use histogram exemplars or a latency-aware trace index to retrieve slow traces. Confirm that sampling did not drop the condition you are trying to inspect.
Start with the user-visible metric. Selecting traces by LLM TTFT alone misses slow endpointing, tools, transport, and playback.
2. Compare slow turns with matched normal turns
Match on channel, caller and runtime region, carrier route, turn class, agent version, model, provider, and load band. Then compare each event delta and span with p50 or p95 peers from that same cohort.
A global fast baseline can send the investigation in the wrong direction. A tool turn should be compared with the same tool class. A cold first turn should be compared with other cold first turns.
3. Mark the critical path
Lay the trace on one timeline. Identify the sequence that determined playback_started, including idle gaps between named spans. A long span off the critical path may have no user-visible effect. A short serial wait between services can dominate the final delay.
For every slow turn, ask:
- When did the user actually stop speaking?
- When did the runtime commit the turn?
- Which dependency blocked first speakable text?
- Did any request queue, retry, or time out?
- When did audio reach the client?
- How long did buffering delay playback?
4. Group evidence into testable hypotheses
| Hypothesis | Evidence to look for | Controlled comparison |
|---|---|---|
| Queueing or saturation | Queue wait, rising concurrency, constrained central processing unit (CPU), graphics processing unit (GPU), connection pool, or provider limit | Same workload below and near the suspected limit |
| Resource contention or cold state | Initialization span, cache miss, runtime pause, uneven host or worker | Warm versus cold and affected worker versus peers |
| Retry amplification | Repeated dependency attempts, backoff, duplicate work, late cancellation | Retry-free success path and injected transient failure |
| Network distance or loss | Region-specific round-trip time (RTT), retransmission, jitter, one route or carrier | Same turn path from another supported region or route |
| Audio buffering | Small server delay followed by large client-receipt-to-playback gap | Same audio through controlled buffer and network conditions |
| Endpointing policy | Large speech-end-to-commit gap, noise or hesitation tags, false endpoints after tuning | Fixed audio corpus across endpoint policies |
| Tool or retrieval path | Tail concentrated in one tool class, queue, or downstream service | Tool-free path and isolated dependency timing |
| Provider, model, or region choice | Tail clusters on one configuration combination | One controlled configuration change at a time |
These are common tail hypotheses, not confirmed Dasha causes. Preserve the trace, configuration, audio conditions, and downstream outcome before changing the system.
5. Reproduce under representative load
Replay the smallest scenario that retains the delay. Match concurrency, turn rate, audio duration, tool mix, network path, and provider configuration. A single local call can validate instrumentation, but it cannot reproduce a saturation-driven p99.
Change one factor at a time. If reducing retries lowers p99, verify that error and task-success rates remain acceptable. If shortening an endpoint threshold lowers p99, verify that false endpoint and repair rates do not rise.
6. Recalculate the full distribution
After a fix, run the same eligible cohort, window, and quantile method. Compare p50, p95, p99, maximum, failure rate, and quality guardrails. A changed definition or traffic mix invalidates the before-and-after claim.
Turn a reproducible production tail case into a regression scenario. That keeps the specific route, pause pattern, tool failure, or load state from returning unnoticed.
Use p99 as a diagnostic system, not a vanity number
P99 becomes useful when every value leads back to an event pair, a population, and a set of slow turns. Histograms provide an aggregatable fleet view. Traces expose the critical path. Caller-side audio closes the gap between first byte and first audible playback.
Dasha gives technical teams a managed runtime for production voice AI. If you want to evaluate whole-turn latency on a real phone and tool path, start building with Dasha.
