Voice activity detection: From audio frames to natural turns

What is voice activity detection?

In brief: Voice activity detection (VAD) estimates whether short audio frames contain speech. A production voice agent wraps that classifier with pre-roll, hysteresis, hangover, echo control, and often semantic end-of-turn logic. Evaluate the complete turn system, since frame accuracy alone cannot predict clipping, interruption, or response delay.

Voice activity detection is the process of identifying speech in an audio stream. A VAD model or algorithm examines a short frame of audio and returns either a speech/non-speech decision or a speech probability. Logic around the detector combines successive frames into speech segments and emits events such as speech_started and speech_stopped.

On Dasha, VAD sits inside our managed turn-taking path with selectable VAD and direct end-of-turn modes, so an agent does not need a separate detector service.

Those events support several jobs:

• Send speech segments to automatic speech recognition (ASR) instead of processing uninterrupted silence.
• Mark speech regions in recordings for transcription, diarization, or analytics.
• Decide when a real-time agent should start listening, wait, respond, or stop its own audio.
• Suppress silence packets in communications systems to reduce bandwidth and compute.

VAD answers whether speech is present. It does not identify the speaker, transcribe words, remove noise, detect a wake phrase, or prove that a person has finished a thought. Keeping those boundaries clear prevents one detector from becoming responsible for decisions it cannot make.

How VAD turns audio frames into events

A production detector is a streaming state machine, rather than a single threshold applied once.

1. Condition the input. Separate the caller and agent channels where possible. Decode, resample, and normalize audio consistently. Echo cancellation and noise suppression may run before VAD, but aggressive processing can also remove quiet speech.
2. Slice the stream into frames. Frames are short enough for timely decisions and long enough to contain useful acoustic evidence. The exact size is model-specific. The popular py-webrtcvad wrapper, for example, accepts 10, 20, or 30 millisecond frames of 16-bit mono pulse-code modulation (PCM) audio at supported sample rates.
3. Score each frame. Simple detectors use energy and spectral rules. Statistical and neural models learn a richer boundary between speech and sound such as music, traffic, typing, or line noise.
4. Stabilize the scores. Separate start and stop thresholds, minimum-duration rules, and rolling windows prevent a single uncertain frame from opening or closing a segment.
5. Protect the boundaries. A pre-roll buffer restores audio captured just before speech was declared. A trailing buffer, often called hangover, keeps short dips from clipping the end of a word.
6. Emit events. The turn manager, recognizer, recorder, and agent runtime consume stable speech-start and speech-stop events.

The scoring step can use several families of detectors. An energy-based detector compares loudness, signal-to-noise ratio, and related hand-built features against rules or an adaptive noise floor. Statistical detectors model the likelihood of speech and non-speech under different feature distributions. Neural detectors learn that boundary from labeled audio and can retain temporal context across frames. Neural models tend to handle varied noise better when their training data represents the target environment, while rule-based methods remain attractive for small, constrained devices. None of these acoustic methods understands whether the spoken thought is complete.

The core state logic can remain small even when the model is sophisticated:

pre_roll = RingBuffer(max_ms=pre_roll_ms)

for frame in audio_stream:
    score = vad.probability(frame)

    if state == "quiet":
        pre_roll.append(frame)
        speech_ms = speech_ms + frame_ms if score >= start_threshold else 0
        if speech_ms >= min_start_ms:
            emit("speech_started")
            send_to_recognizer(pre_roll.contents())
            state = "speaking"
            silence_ms = 0

    else:
        send_to_recognizer(frame)
        silence_ms = silence_ms + frame_ms if score < stop_threshold else 0
        if silence_ms >= stop_hangover_ms:
            emit("speech_stopped")
            state = "quiet"
            speech_ms = 0
            pre_roll.clear()

This example omits packet reordering, timestamps, maximum segment length, and recovery logic. It shows the important point: the model score is only one input. Buffering and state transitions determine what the rest of the product experiences.

Why VAD and end-of-turn detection are different

A pause can mean several things. The caller may have finished, taken a breath, searched for a number, or paused after saying "let me think." Acoustic VAD sees the same absence of speech in every case.

Endpointing groups frame-level activity into an utterance boundary. It usually considers the duration of silence and recognizer state. Semantic end-of-turn detection also considers words, syntax, conversational context, and sometimes prosody, the rhythm, stress, and intonation of speech. An incomplete clause can earn more wait time, while a complete answer can close sooner.

This distinction matters because turn timing is part of the conversation. A study of question-and-response sequences across ten languages found that turn transitions clustered around minimal gaps and minimal overlap, with modal response offsets between 0 and 200 milliseconds. It also found meaningful variation within and across languages. That is evidence for measuring timing as behavior, rather than choosing one universal timeout. Cross-language turn-taking research provides the underlying results.

Acoustic VAD is still valuable in a semantic system. It can detect speech onset quickly, gate compute, and provide the candidate endpoint that a semantic model accepts or delays. It also supports barge-in, where caller speech interrupts agent playback.

Practical VAD options for voice AI

The right choice depends on whether you need a raw classifier, a component for a custom stack, or turn-taking inside a managed voice runtime.

WebRTC VAD and Silero are useful reference points because their interfaces expose different operating models. WebRTC returns a voiced/unvoiced decision. Silero exposes a neural model plus helpers for timestamps and streaming use. A benchmark score still does not settle the choice. Audio format, available runtimes, license, concurrency, language and channel mix, and failure cost all matter.

Teams assembling these components themselves should treat VAD as one ownership boundary in the broader voice AI stack. The code above is the easy part. Deployment, buffering, call-state coordination, traces, and safe configuration changes create most of the ongoing work.

Tune VAD around failure costs

There is no universal VAD threshold. A false negative can erase a digit from an account number. A false positive can make an agent stop talking for a cough. Changing sensitivity moves risk between those outcomes.

Tune one control family at a time and observe the downstream behavior:

Start and stop thresholds can differ. This hysteresis makes it possible to demand strong evidence before opening a turn while tolerating softer frames after speech has begun. Pre-roll fixes late detection at the audio boundary without making the classifier more sensitive. Hangover protects word endings but is paid directly as endpoint delay. These controls solve different failure modes, so a single "aggressiveness" slider can hide useful detail.

Full-duplex agents need channel-aware VAD

Turn-taking gets harder while the agent is speaking. The input can contain the caller, the agent's acoustic echo, line echo, and ambient sound at the same time. Running VAD on a mixed recording cannot tell which source should trigger barge-in.

A safer production path is:

1. Keep inbound caller audio separate from outbound agent audio.
2. Run echo cancellation or use the telephony channel separation already available.
3. Apply VAD to the cleaned inbound stream even during text-to-speech playback.
4. Require enough caller evidence to declare an interruption.
5. Stop playback quickly, preserve the caller's pre-roll, and update conversation state once.
6. Record event timestamps for caller onset, barge-in decision, playback stop, transcript finality, turn decision, and next audible agent audio.

Those timestamps make a vague complaint such as "the agent talks over me" debuggable. They reveal whether the failure came from VAD, echo control, the turn manager, delayed playback cancellation, or stale state.

Test conversation behavior, not frame accuracy alone

Frame-level precision and recall are useful for comparing classifiers against labeled audio. They do not reveal whether callers lose words or wait through awkward silence. Add event and conversation metrics:

• Speech-start delay: actual speech onset to speech_started, reported as the 50th, 95th, and 99th percentiles (p50, p95, and p99).
• Leading audio loss: milliseconds or phonemes omitted before the retained segment.
• Endpoint delay: actual utterance end to the final turn decision.
• Mid-utterance stop rate: completed turns that were split during an internal pause.
• False activation rate: turns opened without target speech, per call or per audio hour.
• False barge-in rate: agent playback cancelled without a genuine caller interruption.
• Repair rate: calls where a person repeats, corrects, or restarts after a timing failure.

Create a test matrix across the audio your product will receive: narrowband and wideband codecs, packet loss, mobile connections, speaker distance, quiet and loud voices, accents, languages, overlapping speech, backchannels, filled pauses, numbers, spelled strings, music, and background talkers.

The open MUSAN corpus contains speech, music, and a range of technical and ambient noises, including tones and indistinct crowd noise. It is useful for repeatable stress mixtures. It does not reproduce carrier jitter, automatic gain control, acoustic echo, or the conversational choices real callers make, so production-like calls still belong in the test set.

Review labeled failures by category, then connect each category to a configuration or architecture change. Our voice agent testing guide shows how to extend this into regression gates for complete calls.

Configure turn-taking in Dasha

For agents running on Dasha, we keep the VAD and turn decision inside our managed runtime. The default asap_v2 mode is tuned for faster turn detection. The flux option runs recognition and end-of-turn detection through the Deepgram Flux pipeline, which models turn completion directly instead of relying on silence duration alone.

Set the mode under config.features.turnTaking:

{
  "config": {
    "features": {
      "turnTaking": {
        "version": "v1",
        "vad": "flux"
      }
    }
  }
}

The JSON example selects flux; the documentation screenshot shows the default asap_v2 configuration.

Choose asap_v2 when acoustic responsiveness fits the call. Evaluate flux when incomplete thoughts and natural pauses produce premature replies. The Dasha turn-taking controls apply to the main customer call, so transfer legs should be evaluated separately.

In either mode, judge the outcome with real call audio and event traces. A semantic turn model can still misread intent, and a fast acoustic detector can still outperform it on short, constrained responses.

Frequently asked questions

Does VAD detect what someone said?

No. VAD detects speech presence. Automatic speech recognition converts speech into words. A pipeline may use VAD to decide which audio reaches the recognizer.

Does VAD remove background noise?

No. Noise suppression changes the signal, while VAD classifies it. Running noise suppression before VAD can reduce false activations, though excessive suppression may also damage quiet speech.

Can VAD detect the end of a sentence?

Acoustic VAD detects a period without speech. It cannot know whether that pause ends a sentence. An endpointing rule or semantic turn model makes the broader turn-completion decision.

What VAD threshold should I use?

Use the threshold that minimizes the failures that matter in your own audio and call flow. Tune it against labeled production-like calls, keep start and stop behavior separate, and report tail latency and event errors alongside frame metrics.

If you want production turn-taking without owning the classifier, buffering, telephony, and failure handling separately, start building with Dasha.