Call center quality monitoring changes when the agent is software. A polite conversation can conceal an incorrect booking, an unauthorized action, or a failed handoff. Technical teams need a way to reconstruct what happened on real calls and turn those findings into fixes. The starting point is a reviewable call record, followed by a monitoring routine that connects conversations to actions and outcomes.
What call-quality monitoring means for an AI voice agent
Call center quality monitoring is the ongoing review of customer calls against defined service standards. For an AI voice agent, those standards cover the conversation, the actions it took through tools (application functions), and the outcome in the connected business system.
An effective production routine gathers call evidence, selects review cohorts, identifies failures, and assigns corrective work. It keeps serious policy violations separate from routine conversation-quality issues.
At Dasha, we help technical teams build and run production voice AI agents through a managed runtime, application programming interfaces (APIs), and a web application. Our call history and completed-call inspection give teams the evidence for this routine: transcripts, configured recordings, model interactions, tool executions, and timeline events. Your team defines the monitoring policy and decides what each finding requires.
Monitoring, testing, evaluation, and observability have different jobs
These activities share evidence, but answer different questions:
- Testing exercises planned scenarios, including failures, before and after changes.
- Evaluation applies defined criteria to judge behavior and support release decisions.
- Observability exposes runtime events and system health, such as errors and delays.
- Quality monitoring reviews actual production traffic to find where callers, actions, and outcomes diverge from expectations.
A call ending normally tells you about execution status. It does not establish that the caller's request was resolved. Likewise, a healthy service can still produce incorrect answers.
The National Institute of Standards and Technology recommends monitoring deployed AI behavior and comparing production performance with pre-deployment measurements in its AI risk-management playbook. Production monitoring supplies evidence that controlled scenarios alone cannot provide.
1. Make each call reviewable
Start with a stable call identifier that links the conversation to its technical events and business records. A reviewer should be able to move from a complaint to the relevant turn, tool execution, and resulting state without matching timestamps by hand.
Build a review record around three kinds of evidence:
| Evidence | What it can establish |
|---|---|
| Conversation | The caller's request, corrections, information given, and commitments made by the agent |
| Execution | Which tools ran, their arguments and responses, errors, retries, and event timing |
| Business outcome | Whether the intended booking, update, cancellation, or handoff actually reached the required state |
Keep the agent configuration used for the call alongside that record. Include relevant prompt, tool, and knowledge versions in your own deployment records so a later edit does not erase the context needed to investigate an older call.
For state-changing actions, the system of record, the database or service that owns the business record, is the strongest evidence of completion. A tool returning a 200 OK response can still return a business-level rejection. A request accepted for later processing may remain pending after the call ends.
The τ-bench customer-service benchmark evaluates simulated agents by comparing the final database state with the expected outcome. That provides a useful principle for live monitoring: separate what the agent said from what the system actually did. Final state also needs to sit alongside policy evidence, because an outcome can be achieved through an unauthorized action.
Represent missing evidence explicitly. Use outcomes such as resolved, unresolved, handed off, and unknown, with definitions appropriate to the workflow. An unavailable downstream result belongs in the unknown category until reconciliation completes. It should never become a pass merely because the transcript sounds conclusive.
2. Use representative and targeted review cohorts
A review cohort is a group of calls selected for a specific purpose. Keep a representative view of ordinary traffic and a targeted view of likely or high-impact failures.
Use four complementary cohorts:
- Routine traffic. Select calls throughout the reporting period, including apparently successful calls. This catches silent failures that error logs miss.
- Failure signals. Include tool errors, repeated clarification, caller complaints, abrupt endings, and mismatches between spoken commitments and recorded outcomes.
- High-risk workflows. Give actions involving sensitive information, permissions, or consequential account changes their own review policy.
- Changed or thinly represented segments. Include new configurations and relevant slices such as workflow, language, telephony route, or tenant (a customer organization in a shared product).
Include failed and interrupted calls in the monitoring population. When a recording or transcript is unavailable, retain that fact as a coverage gap. Removing difficult-to-review calls can remove the very failures the program needs to find.
Preserve the selection reason for every reviewed call. A complaint-heavy cohort is useful for diagnosis, but its failure rate does not describe all traffic. Keep targeted findings separate from representative trend reporting, and show the number of eligible and reviewed calls for each segment.
There is no universal review percentage. The appropriate coverage depends on call volume, workflow risk, reviewer capacity, and the conclusion you need to support. A low-volume, consequential action deserves a different policy from a routine information request.
For multitenant products, keep tenant-specific results visible within authorized access boundaries. A platform-wide average can hide a broken integration affecting one customer's calls.
3. Review outcomes, authority, recovery, and voice behavior
Keep the ongoing review criteria short enough to apply consistently. Four dimensions cover the monitoring job without turning every call into a large scoring exercise.
| Review dimension | Evidence to review |
|---|---|
| Task outcome | Correct final state, required information delivered, and alignment with the caller's latest request |
| Permitted behavior | Authorization, required confirmations, applicable policy, and forbidden actions |
| Recovery and escalation | Handling of corrections, ambiguity, unavailable tools, and the actual result of a handoff |
| Voice interaction | Audible clarity, interruption handling, repeated turns, silence, and delays at the relevant moment |
Write findings around observable behavior. A vague label such as poor understanding is less useful than a record showing that the caller corrected a date and the booking tool received the earlier date.
Separate critical violations from ordinary quality signals. An unauthorized account change needs its own incident path. A repetitive explanation may need a conversation-design fix. Averaging the two into a single score obscures what the team must do next.
For voice problems, use audio when available and align it with the event timeline. A transcript cannot establish whether speech overlapped, whether a name was pronounced clearly, or how a pause sounded to the caller. Timing data can locate a delay, while audio helps explain its effect on the interaction.
Short calls are not automatically good calls. Longer calls may reflect legitimate complexity. Treat duration and silence as triage signals, with task outcome and surrounding events providing the interpretation.
Our evaluation metrics guide covers detailed rubrics, grading, calibration, and release decisions. Production review needs enough criteria to classify findings and route them to that evaluation process.
4. Combine deterministic rules, automated triage, and human review
Assign each monitoring decision to the evidence and judgment it requires.
Deterministic rules apply explicit conditions to structured records: a required field is missing, a tool returned an error, a confirmation event is absent, or a downstream operation remains incomplete. Each rule needs a defined scope. A missing field is meaningful only when the workflow requires it.
Automated triage helps identify calls for attention. An analyzer can flag repeated questions, suspected policy departures, or possible unresolved requests. Treat these labels as review signals with supporting evidence, rather than final verdicts. Require a relevant transcript span or event reference so reviewers can investigate a flag.
Human review handles ambiguous intent, disputed outcomes, policy interpretation, and the caller's experience. Route consequential findings to someone with authority over the relevant workflow, rather than leaving them in a general analytics queue.
Maintain a human-reviewed reference set containing ordinary calls, known failures, and difficult borderline cases. Compare automated labels with those decisions by workflow. Include unflagged calls in human review so missed failures remain visible. Revisit disagreements when traffic, policy, or agent configuration changes.
Keep evidence collection, automated analysis, and human adjudication as separate coverage measures. An analyzer processing every available transcript does not mean every call received a correct quality judgment. Calls with missing transcripts, unsupported analysis, or unresolved labels remain gaps.
5. Turn findings into owned corrective work
A monitoring loop is complete when a finding reaches an owner, a corrective action, and a follow-up outcome. A dashboard alone leaves that work unfinished.
Use a consistent sequence:
- Classify the finding. Distinguish policy incidents, incorrect outcomes, integration failures, conversation defects, and voice-path problems.
- Preserve evidence. Attach the call identifier, relevant turns and events, configuration context, expected behavior, and observed result.
- Assign an owner. Route integration defects to the integration owner, unclear rules to the policy owner, and runtime failures to the team responsible for the affected component.
- Contain consequential failures. Use your incident process to restrict the affected action, route that workflow to a human, or restore an earlier configuration where appropriate.
- Add a reproducible regression case. Create a scenario that protects against a repeat of the failure, using the relevant user turns, starting state, and expected outcome.
- Follow the same failure category after the change. Compare equivalent workflow cohorts and preserve configuration boundaries in the reporting.
Avoid assigning every failure to the prompt. Incorrect tool arguments may originate in speech recognition, an outdated tool definition, or application logic. A delay may come from a dependency. The call evidence should narrow the cause before the team chooses a remedy.
The detailed scenario and regression workflow belongs in our voice agent testing guide. Monitoring contributes real failure cases; evaluation determines whether the proposed change is ready for release.
Report the health of the monitoring process alongside call findings. Useful operational measures include evidence availability, unknown outcomes, overdue reviews, confirmed critical findings, and the age of unresolved corrective work. Give urgent incidents an immediate escalation route instead of waiting for the next reporting cycle.
How to use Dasha's call evidence
We provide the managed runtime and call-inspection surfaces that support this workflow. The team's review policy, automated analysis, links to business outcomes, and corrective-work queue remain its own implementation or integrations.
Use our call history tools to search past calls and filter by date, agent, or status. The POST /api/v1/callresults/search endpoint provides programmatic access to call results, including transcripts and inspection links. Recording availability depends on configuration.
For completed-call analysis, open the Call Inspector. It brings together conversation text, available audio, model prompts and responses, tool execution evidence, and timeline events. Use those views to connect a suspicious turn with the action and delay around it. Completed-call inspection serves retrospective review; it does not provide in-call intervention.
Our activity logs supply additional operational context for tool executions, webhook deliveries (notifications sent to connected services), and API behavior. Together, these records help engineers investigate a finding. They do not establish that a customer request succeeded in an external system, so your monitoring pipeline still needs the relevant business outcome.
This is a builder workflow for operating AI voice agents. Human-agent coaching, workforce management, and broader contact-center quality programs require their own processes and tools.
Protect call evidence throughout the review workflow
Recordings, transcripts, model inputs, and tool arguments can contain sensitive information. Extending monitoring into an analytics store or external classifier adds another data-processing path.
Define the permitted recording and review purposes, applicable notice or consent requirements, access roles, retention schedule, and redaction policy for your deployment. Requirements vary with jurisdiction, industry, contracts, and the information captured. A recording setting alone does not establish compliance.
Restrict access to the calls a reviewer needs, including tenant boundaries. Redact sensitive values before exporting evidence where appropriate, and preserve useful event references without copying unnecessary customer details into tickets.
Apply retention rules to derived copies as well as original recordings: transcripts, analysis outputs, exported reports, and evidence attachments. The UK Information Commissioner's Office's storage limitation guidance ties retention to the processing purpose. It explains that the UK General Data Protection Regulation does not prescribe one duration for every data type. Set your deployment's schedule around its actual obligations and review purpose.
Build a monitoring loop around reviewable calls
The useful output of call center quality monitoring is an explainable finding that leads to corrective work. Keep conversation, execution, and business-outcome evidence connected. Preserve ordinary traffic alongside targeted review. Give serious failures an owner and bring reproducible cases into regression coverage.
For technical teams building production voice agents, our managed runtime and call-inspection tools provide the starting evidence. Start building with Dasha and make reviewability part of the first workflow you deploy.
