A voice agent can say the right words and still make the wrong change. It can also complete the task while giving the caller a slow, confusing experience. Voice agent evaluation turns call evidence into scores and release gates. A sound framework keeps outcomes, speech quality, timing, safety, and reliability visible so one good average cannot hide a serious regression.
What voice agent evaluation means
Voice agent evaluation is the process of scoring complete spoken interactions against explicit requirements. It answers three questions:
- Did the agent produce the correct and safe outcome?
- Did the spoken interaction work for the caller?
- Is the result consistent enough for the intended use?
Testing, evaluation, observability, and business measurement have separate jobs:
- Testing creates evidence by exercising scenarios. Our voice agent testing guide owns scenario execution, failure injection, and regression-suite workflow.
- Evaluation maps evidence to metrics, applies rubrics, compares versions, and produces a release decision.
- Observability exposes traces, logs, events, audio, and metrics so a team can understand system behavior. It supplies evidence for evaluation, while the scorecard supplies the judgment.
- Business measurement connects a deployed agent to commercial outcomes. Our voice AI business metrics guide owns that layer.
Offline evaluation scores a fixed, versioned dataset outside production, usually to compare a candidate with an approved baseline before release. Online evaluation scores completed production interactions under real caller and channel conditions. Both can reuse metric definitions, but they answer different questions: offline evaluation supports controlled comparison, while online evaluation measures performance on actual traffic. Neither term is a synonym for monitoring.
Keep six evaluation dimensions separate
A single quality score can improve while a candidate starts making unauthorized changes, loses accuracy for a caller group, or develops a long latency tail. Keep these dimensions visible through the release decision.
| Dimension | Question | Primary evidence | Decision metrics |
|---|---|---|---|
| Task outcome | Did the workflow reach the correct final state? | System-of-record state before and after the call | Task success, critical-field accuracy |
| Tools and workflow | Did the agent take the permitted path? | Tool calls, arguments, results, and side effects | Required-action rate, forbidden-action rate, duplicate side effects |
| Policy and safety | Did the agent remain within its authority? | Policy assertions, transcript, trace, and final state | Critical violation rate, required disclosure rate, correct escalation rate |
| Speech and conversation | Could both sides hear, take turns, and recover? | Audio tracks and event timestamps | Critical-entity accuracy, response latency, interruption cutoff, repair rate |
| Spoken response | Was the response clear and suitable for listening? | Audio scored against an anchored rubric | Speech fidelity, intelligibility, conciseness, conversation progression |
| Reliability / operability | Does the system behave consistently across valid operating conditions? | Call status, dependency events, transfer result, and repeated trials | Technical failure rate, transfer completion, latency tail, repeatable success |
ServiceNow's end-to-end EVA research shows why the separation matters. Its evaluated systems occupied different accuracy and experience regions, and its cascade systems showed a capability and latency tradeoff. The same research found a strong association between critical-entity transcription and task completion. A component score alone cannot represent the end-to-end result. (EVA findings)
Map each evidence source to a score
A metric is trustworthy only when its evidence can prove the claim. Start with the decision and work backward to the source.
| Evidence | What it can prove | Best scoring method | What it cannot prove alone |
|---|---|---|---|
| Downstream system state | Correct final state, exact fields, duplicate or partial writes | Deterministic assertion | Whether the caller heard an accurate confirmation |
| Tool execution record | Required order, authorization, arguments, results, retries, side effects | Deterministic assertion | Speech quality or caller comprehension |
| Transcript | Required disclosures, factual content, intent progression, conversational repair | Rules or a calibrated text judge | Pronunciation, crosstalk, clipping, or actual timing |
| Recorded audio | Intelligibility, pronunciation, speech fidelity, noise, overlap | Human review or a calibrated audio judge | Whether the database update was correct |
| Event timestamps | Response latency, silence, talk-over, interruption cutoff | Deterministic calculation | Whether the response was useful |
| Call and dependency events | Technical completion, transfer result, provider or tool failure | Deterministic classification | Whether a completed call achieved its task |
Consider an appointment change. The system-of-record state proves that exactly one appointment moved to the requested time. The tool record proves that identity verification preceded the update. The transcript proves that the agent disclosed the new time. The recording and event timeline prove that the confirmation was intelligible and timely. Passing one source does not compensate for failing another.
This mapping also separates decision metrics from diagnostics. Task success can block a release. Word error rate can explain why task success fell. Keep both, but do not give them equal authority.
Choose metrics that support a release decision
Use a small set of decision metrics with exact denominators and measurement rules.
| Metric | Calculation | Measurement rule |
|---|---|---|
| Task success rate | Correct final states / eligible calls | Read the downstream state. Do not accept the agent's statement that it succeeded. |
| Critical-entity accuracy | Correct critical values / evaluated critical values | Score dates, amounts, names, IDs, addresses, and confirmation codes separately. |
| Tool correctness | Calls with every required action and no forbidden action / eligible tool calls | Include authorization order, arguments, result handling, and duplicate protection. |
| Policy violation rate | Calls with a defined violation / eligible calls | Retain severity. A prohibited action remains visible even when the average is high. |
| Turn response latency | End of caller speech to first audible agent audio | Report by channel and scenario slice. |
| Interruption cutoff latency | Start of caller interruption to stopped agent playback | Use event timing and audio. A transcript cannot establish cutoff speed. |
| Conversation repair rate | Calls with a repeat, correction, or misunderstanding loop / evaluated calls | Tag the cause, such as transcription, turn detection, policy, or tool result. |
| Transfer completion rate | Transfers connected with required context / attempted transfers | Confirm the destination and the transferred context. |
| Technical failure rate | Calls ending because of a media, telephony, model, runtime, webhook, or tool error / eligible calls | Keep dependency categories separate for diagnosis. |
For latency, p50 is the median, so half of observations are at or below it. p95 is the value at or below which 95% of observations fall. p99 applies the same definition to 99%. Report all three because a stable median can coexist with a poor tail.
Word error rate (WER) is the number of substitutions, deletions, and insertions divided by the number of words in the reference transcript. It is useful for speech-recognition diagnosis. One wrong confirmation-code character can fail a task, while several filler-word errors may have no effect. Pair WER with critical-entity accuracy.
Mean opinion score (MOS) also needs a label. Human MOS, predicted MOS, and a custom naturalness rubric are different measures. The ITU-T P.800 standard defines subjective listening-test methods, including test conditions and rating procedures. Record the method with the result.
Design the evaluation set and sample before scoring
The evaluation set should represent both workload frequency and consequence. Include common intents, rare high-impact workflows, corrections, ambiguity, critical entities, supported languages, intended channels, realistic audio conditions, and the dependency outcomes that change expected behavior. Keep the failure catalog compact here. The testing guide covers how to execute those conditions.
Each scenario needs a versioned contract:
| Field | Purpose |
|---|---|
| Starting state | Makes the expected result reproducible |
| Caller goal and allowed variations | Defines what counts as the same scenario |
| Expected final state | Supplies deterministic ground truth |
| Required and forbidden actions | Encodes workflow and safety rules |
| Channel and caller-condition tags | Enables slice analysis |
| Subjective rubric | Defines how speech and conversation quality will be judged |
| Severity | Determines whether a failure can override aggregate scores |
Set the sample size before looking at the candidate result. Report every rate with its numerator, denominator, and confidence interval. Segment by language, channel, intent, caller condition, and dependency path. A global average can hide a failing slice. Where production mix is known, show both an unweighted slice table and a traffic-weighted rollup.
Repeat non-deterministic scenarios. For N scenarios run k times each, pass@k is the fraction of scenarios where at least one of the k trials passes. It measures attainable or ceiling performance. pass^k averages p_i^k across the N scenarios, where p_i is scenario i's observed pass rate. It estimates the probability that all k independent trials pass for a scenario, which measures consistency. A large gap between pass@k and pass^k means occasional success is masking unreliable behavior. (EVA aggregate metrics)
Combine deterministic checks, calibrated judges, and human review
Use the least subjective evaluator that can answer the question.
- Deterministic checks fit final state, exact fields, tool order, schema validity, duplicate side effects, transfer destination, and timing events.
- Model judges fit criteria that require interpretation, such as relevance, conciseness, recovery quality, and conversation progression.
- Human review fits judge calibration, high-severity cases, audio experience, new failure clusters, and disputed scores.
An unanchored one-to-five score is hard to reproduce. Define observable anchors. For a spoken-interaction rubric, 1 could mean the call cannot progress or contains a materially wrong spoken fact, 3 could mean the task completes with one material repair or confusing delay, and 5 could mean the call progresses clearly with correct entities and no avoidable repair. Give the judge the scenario goal, expected facts, and only the evidence needed for that criterion.
Calibrate a model judge against a human-labeled sample before it affects a gate. Measure false passes and false failures by slice. Freeze the judge model, prompt, rubric, and sampling settings during a candidate comparison. Blind version identity and randomize order for pairwise judgments. A transcript judge cannot score pronunciation, audio glitches, crosstalk, or interruption timing, so audio criteria require audio evidence.
Use statistical margins and severity rules for release gates
A non-inferiority margin is the largest pre-approved regression from the baseline that the candidate may show on a protected metric. For a success rate, a margin of -5 percentage points means the candidate cannot be more than five points worse. For latency, where lower is better, a +100 ms margin allows at most a 100 millisecond increase.
Set the margin from workflow risk before seeing results. Then require the confidence interval for the candidate-versus-baseline difference to remain inside that margin. A point estimate alone can make a small sample look decisive.
Severity rules sit above averages and composite scores. An unauthorized action, privacy breach, unsafe partial state, or corrupt side effect can block a candidate even when every aggregate metric improves. If a composite score is useful for reporting, calculate it only after hard gates pass and show every component beside it.
Worked release scorecard
Assume the spoken-interaction rubric uses the 1, 3, and 5 anchors above and treats scores of 4 or 5 as passing. Rate intervals below are 95% Wilson intervals. Rate-difference intervals use the Newcombe method based on those bounds, and latency intervals use paired bootstrap estimates.
| Criterion and evidence | Slice | Baseline | Candidate | Margin or hard limit | Severity override | Result |
|---|---|---|---|---|---|---|
| Correct final state | All eligible calls | 184/200, 92.0% (87.4% to 95.0%) | 188/200, 94.0% (89.8% to 96.5%) | -5 pp; difference CI -3.2 to +7.3 pp | No | Pass |
| Spoken-interaction score >=4 | Phone calls | 160/180, 88.9% (83.5% to 92.7%) | 165/180, 91.7% (86.7% to 94.9%) | -5 pp; difference CI -3.5 to +9.1 pp | No | Pass |
| Correct final state | Noisy phone, critical entities | 84/100, 84.0% (75.6% to 89.9%) | 94/100, 94.0% (87.5% to 97.2%) | -5 pp; difference CI +1.2 to +19.0 pp | No | Pass |
| Tool-turn p95 latency | Tool-using calls, 1,140 turns per version | 780 ms, bootstrap CI 740 to 820 ms | 820 ms, bootstrap CI 780 to 860 ms | +100 ms; difference CI +10 to +70 ms | No | Pass |
| Unauthorized side effect | All eligible calls | 0/200, 0% (0% to 1.9%) | 1/200, 0.5% (0.1% to 2.8%) | Hard limit: 0 | Yes, critical | Fail |
| Final release gate | Required scope | Core and slice gates pass | One critical override triggered | All hard gates must pass | Applied | Do not release |
The table preserves the denominator, uncertainty, slice, allowed regression, and reason for the final decision. The candidate's higher average cannot erase the unauthorized side effect.
Apply the scorecard to evidence from Dasha
For completed calls, our Call Inspector provides transcripts, model activity, tool executions, timeline events, and latency breakdowns. Recordings are available when recording is enabled. These sources cover much of the evidence map, while the expected downstream state, rubric, judge calibration, severity policy, and release decision remain owned by your evaluation design.
That boundary is intentional. We help technical teams avoid assembling the core voice stack from separate providers and provide traceable call evidence. We do not decide what success or acceptable risk means for a specific workflow.
Start building on Dasha with a versioned evaluation set and a release gate your team can explain.