Real-Time Voice Agent in Production: Cutting Conversational Dead Air with Turn Detection, Two-Pass EOU, and Barge-in

In text chat, 100-millisecond differences are usually imperceptible. Voice is different. Humans directly interpret “how quickly the other party starts responding after I finish speaking” as part of whether the system feels natural.

For a typical cascaded voice agent, the user-perceived first-response gap can be approximated as:

T_gap ≈ T_endpoint + T_stt_finalize + T_llm_first_token + T_tts_first_audio + T_playback_buffer

Many teams only optimize T_llm_first_token, leaving behind 500ms, 800ms, or even longer silence confirmation windows. The result: the model is fast, but the product still feels sluggish.

Real production optimization should first answer three questions:

1. When can we confidently say the user has finished their turn?
2. Can we start downstream computation before final confirmation?
3. When the user speaks again while the system is talking, how do we stop the entire pipeline consistently within tens to hundreds of milliseconds?

These correspond to Turn Detection, Two-Pass EOU, and Barge-in.

Core Principle 1: Turn Detection Is Not a Simple Silence Timer

VAD vs. STT Endpointing vs. Semantic Turn Detector

The simplest approach is Voice Activity Detection: after detecting a continuous stretch of silence, declare the user has finished. It’s cheap and fast, but has a fundamental flaw: a pause is not the same as being done.

The user might be thinking, catching their breath, or saying “I’d like to check… um… last month’s bill.” If the system jumps in at the first pause, the experience is worse than waiting an extra 200ms.

Production systems typically use one of three approaches:

Approach	Characteristics	Use Case
VAD-only	Low cost, fast response, but prone to misjudging pauses	Ultra-low latency, language coverage first
STT Endpointing	Relies on ASR’s end-of-utterance signal	Standard conversational flows
Semantic / Audio Turn Detector	Combines acoustic and even semantic cues to judge completion	Recommended for most voice agents

LiveKit’s current docs explicitly distinguish these modes and recommend Turn Detector for most voice agents. Its default endpointing window is 0.5s–3.0s in standard configs, shrinking to 0.3s–2.5s by default with the audio turn detector. These numbers aren’t optimal for every system, but they illustrate a key principle: end-of-turn detection needs both a lower bound and a safety upper bound.

Don’t Just Tune a Single silence_ms in Production

A more robust configuration than a fixed silence threshold is to split the parameters:

turn_detection:
  mode: semantic_audio
  min_endpoint_delay_ms: 300
  max_endpoint_delay_ms: 2500
  vad_min_silence_ms: 300
  force_commit_timeout_ms: 3000

The specific numbers aren’t the point—the four boundaries are:

• Too short → false end-of-turn;
• Too long → increased first-response dead air;
• Must allow the model or STT to emit a high-confidence end signal early;
• Must have a maximum wait time so abnormal sessions don’t hang indefinitely.

Core Principle 2: Two-Pass EOU Trades “Cancellable Speculation” for Latency

Why Two-Stage End-of-Utterance Is Needed

If the system insists on waiting for “final confirmation that the user is done” before calling the LLM, it inherently loses the entire confirmation window.

Two-Pass End of Utterance works like this:

1. A short silence in the first stage triggers an early EOU;
2. The current transcript is immediately sent to the LLM to start inference;
3. ASR state continues to be maintained;
4. If the user resumes speaking, cancel the old LLM/TTS task and restart with the new transcript;
5. Only after the final EOU does the current turn get formally committed.

NVIDIA Riva’s current docs already provide Two-Pass EOU. In their example, the first stage’s stop_history_eou can be set to a short window—their docs suggest around 240ms based on their testing. The final EOU still confirms on a longer window. This mechanism is essentially speculative execution in CPU design: do work that might be useful, but accept that it may be discarded.

Don’t Just Look at First-Response Time—Watch Restart Costs

Early triggering inevitably adds cost. If a user’s utterance contains several natural pauses, you might see:

240ms silence -> LLM request A
user resumes  -> cancel A
240ms silence -> LLM request B
user resumes  -> cancel B
final EOU     -> LLM request C

So Two-Pass EOU evaluation metrics should at least include:

• early_eou_trigger_rate
• llm_restart_rate
• wasted_output_tokens
• cancel_latency_ms
• first_agent_audio_after_final_speech_ms

NVIDIA ACE’s example docs also explicitly warn that early EOU can cause extra LLM calls and compute costs. Production environments can’t just show “first response latency dropped” while hiding the GPU cost amplification.

Core Principle 3: Barge-in Is Full-Pipeline Cancellation, Not a Player Pause

What the System Really Needs to Cancel When the User Speaks Again

Many demos implement Barge-in as just one thing: stop the speaker. That’s not enough. Once the user starts interrupting, you need to handle all of this in sync:

Speech start detected
  |--> stop client playback buffer
  |--> cancel TTS streaming
  |--> cancel / suppress current LLM generation
  |--> mark assistant turn as interrupted
  |--> truncate conversation history to actually heard audio
  |--> open new user turn

If you only stop the player, the LLM and TTS continue consuming resources in the background. Worse, the system may write the “second half of the response the user never heard” into conversation history, causing the next turn’s model to assume that content was already communicated.

LiveKit’s interruption mechanism stops Agent Speech after a user interrupt and automatically truncates history to what the user actually heard. Deepgram’s Twilio example emphasizes that after the server stops generation, the client still needs to clear Twilio’s buffered-but-unplayed audio—otherwise you get the “user already interrupted, but old audio keeps playing” tail.

Barge-in Needs a Generation ID

It’s recommended to assign a monotonically increasing generation_id to each turn’s generation:

interface VoiceTurn {
  turnId: string;
  generationId: number;
  state: "listening" | "thinking" | "speaking" | "interrupted" | "done";
}

function onAudioChunk(chunk: AudioChunk, generationId: number) {
  if (generationId !== session.activeGenerationId) return;
  playback.enqueue(chunk);
}

This way, even if cancellation signals and network packets arrive out of order, old TTS audio can’t “resurrect” in a new turn.

Engineering in Practice: Model the Voice Session as a State Machine

The most error-prone implementation treats ASR, LLM, and TTS as three independent WebSocket callback systems. A production version should explicitly maintain session state:

LISTENING
  | early EOU
  v
SPECULATIVE_THINKING
  | final EOU
  +--------------------+
  |                    |
  | user resumes       v
  +----> CANCELLED <-- THINKING
                          |
                          v
                       SPEAKING
                          | user barge-in
          +---------------+-------------+
          |                             |
          v                             v
     INTERRUPTED                     IDLE

The state machine should at minimum handle:

• Which transcript is provisional vs. final;
• Which generation is currently allowed to output audio;
• What events trigger cancellation;
• Which buffers must be cleared after cancellation;
• Where in the conversation history the final commit lands (character/timestamp);
• Whether stale events are allowed to take effect after a network reconnect.

Recommended Production Latency Metrics

Don’t just record a single voice_latency. Break it down by stage at minimum:

Metric	Meaning
speech_end_to_turn_commit_ms	User’s last valid speech to turn commit
turn_commit_to_llm_first_token_ms	Turn commit to LLM first token
llm_first_token_to_tts_first_audio_ms	First token to first playable audio
speech_end_to_agent_audio_ms	Total dead air from user’s end of speech to hearing the agent
barge_in_detect_ms	User’s re-entry to system recognizing the interrupt
barge_in_silence_ms	Interrupt signal to old audio actually stopping
false_end_rate	Rate of falsely judging the user as done
restart_rate	Rate of speculative inference being cancelled and restarted
wasted_tokens_per_turn	LLM output tokens wasted due to cancellation

The one to watch most closely is the P95/P99 of speech_end_to_agent_audio_ms. Averages are easily masked by a flood of short utterances; what actually makes users feel “this bot is unnatural” is usually the occasional long pause.

A Practical Tuning Order

Phase 1: Establish Real Segmented Metrics First

Timestamp ASR, LLM, TTS, and playback buffering before changing any parameters. Otherwise you’re optimizing by subjective listening alone.

Phase 2: Compress Endpointing, but Monitor False Ends in Parallel

Gradually shorten the minimum endpoint delay and observe in buckets by language, accent, and scenario:

• Short commands;
• Long sentences;
• Hesitations and pauses;
• 8kHz telephony audio;
• Noisy environments;
• Chinese mixed with English, numbers, and proper nouns.

Any “latency improvement” must be checked against false_end_rate.

Phase 3: Only Then Enable Two-Pass EOU

Only allow early EOU to trigger the LLM once your cancellation mechanism is reliable. Otherwise a bad early decision produces duplicate responses, stale audio streams, and corrupted conversation history.

Phase 4: Optimize Barge-in Last

The goal of Barge-in isn’t maximum sensitivity. It’s:

• Real interruptions stop quickly;
• Coughs, keyboard clicks, and TV noise don’t trigger frequently;
• Background generation is cancelled in sync;
• The user’s next-turn context stays consistent.

Deepgram’s current docs also note that pure client-side energy-based VAD is easily triggered by ambient noise; they recommend model-level speech/turn detection where applicable to reduce false interrupts.

When This Applies

This approach is especially relevant for:

• Phone support agents: Users are extremely sensitive to long silences and frequently interrupt.
• Sales and outbound agents: Natural turn-taking and interruption handling directly impact conversation completion rates.
• In-car / device voice assistants: High noise means fixed silence thresholds misjudge easily.
• Real-time translation and meeting assistants: Need to start processing early without prematurely committing to wrong segmentations.
• Game NPCs / companion voice apps: Higher demands on pacing, interrupting, and response timing.

For offline transcription or batch summarization, there’s no need to take on the complexity of speculative turn control.

Common Pitfalls

Pitfall 1: Crank VAD parameters to the minimum for lowest latency. Too-small thresholds turn normal thinking pauses into sentence boundaries. The net result is faster first response but more interruptions and restarts.

Pitfall 2: Only measure LLM TTFT. TTFT matters, but the voice experience also includes endpointing and TTS first audio. A system with 150ms TTFT can still feel sluggish due to an 800ms silence confirmation.

Pitfall 3: Two-Pass EOU is a free optimization. Speculative computation can be cancelled. You must account for invalid requests and wasted tokens in capacity planning.

Pitfall 4: Barge-in equals mute. Real Barge-in must coordinate playback buffers, TTS, LLM, and conversation state.

Pitfall 5: One endpointing config for all languages. Language rhythm, pause habits, and STT capabilities differ. At minimum, bucket configs and validation by language and audio channel.

Pre-Launch Checklist

• Timestamps recorded for speech-end, turn-commit, LLM-first-token, TTS-first-audio, and playback-start.
• P50/P95/P99 established for speech_end_to_agent_audio_ms.
• Tested with long pauses, hesitations, noise, heavy accents, and low sample-rate audio.
• False-end and premature-response rates tracked.
• Two-Pass EOU speculative inference reliably cancels when the user resumes speaking.
• After cancellation, old generation tokens and audio never re-enter the output stream.
• Player buffer clears in sync after Barge-in.
• Conversation history only retains agent content the user actually heard.
• Extra token/GPU cost from speculative restarts is monitored.
• Endpointing parameters support independent config per language, audio channel, or business scenario.
• A kill switch is ready to disable early EOU and fall back to final-only mode.

References

1. LiveKit, Turns overview — https://docs.livekit.io/agents/logic/turns/
2. LiveKit, Turn detector — https://docs.livekit.io/agents/logic/turns/turn-detector/
3. LiveKit, Adaptive interruption handling — https://docs.livekit.io/agents/logic/turns/adaptive-interruption-handling/
4. NVIDIA Riva, ASR Overview — https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-overview.html
5. NVIDIA Riva, Building and Deploying ASR Pipelines — https://docs.nvidia.com/deeplearning/riva/user-guide/docs/public/asr/asr-pipeline-configuration.html
6. Deepgram, Build a Flux-enabled Voice Agent — https://developers.deepgram.com/docs/flux/agent
7. Deepgram, Audio Preprocessing & Barge-In — https://developers.deepgram.com/voice-agent/optimize/audio-preprocessing-barge-in
8. Deepgram, Flux Quickstart — https://developers.deepgram.com/docs/flux/quickstart