Is a more sensitive VAD threshold always better?

No. A threshold too low treats noise as speech; too high clips soft or leading sounds. Production configuration must simultaneously monitor front-end clipping, false triggers, endpoint wait time, and transcription quality.

Why shouldn't Partial Transcripts be written directly to final business records?

Incremental text can be corrected by subsequent audio, and completion events for different utterance segments may not arrive in order. Use stable session, utterance, and item identifiers to reconcile before committing final results.

Is converting everything to 16 kHz mono suitable for all models?

No. 16 kHz mono is the typical input convention for models like Whisper, but different real-time models may support other formats. The system should convert based on the model adapter's declared capabilities, not hardcode a sample rate as a global truth.

LLM Real-Time Voice Input in Production: Low-Latency Recognition with VAD Segmentation, Partial/Final Reconciliation, and Audio Contracts

Background: Real-Time Voice Failures Often Occur Before the Model

Real-time voice applications are often oversimplified as “send microphone audio to the model, then pass the text to the LLM.” In production, failures usually don’t originate solely from the recognition model: the browser sends 48 kHz stereo, but the server parses it as 16 kHz mono; network reconnections cause duplicate audio segments; VAD ends a sentence too early; the frontend treats an unstable Partial Transcript as the final text; completion events for different utterance segments arrive out of order, resulting in reversed captions, duplicate content, or erroneous tool invocations.

Therefore, the voice input pipeline should be treated as a production system with a clear protocol and state machine. The core goal isn’t to pursue the absolute lowest latency, but to establish a measurable balance among these four objectives:

Correct Boundaries: Don’t miss sentence starts, soft sounds, or trailing sounds, and don’t excessively ingest prolonged noise.
Consistent State: Incremental text can be modified; final text must be submitted exactly once per unique utterance segment.
Deterministic Format: Sample rate, channel count, encoding, byte order, and time base are all traceable.
Explainable Latency: Be able to distinguish the time spent in each stage: capture, network, VAD, first token recognition, and final confirmation.

Core Principle 1: Establish the Audio Contract Before Discussing Model Performance

The Audio Contract Must Be Versioned

Each audio session should record at least the following fields:

{
  "audio_contract_version": "audio-v3",
  "codec": "pcm_s16le",
  "sample_rate_hz": 16000,
  "channels": 1,
  "frame_duration_ms": 20,
  "sequence_origin": 0,
  "clock": "monotonic",
  "vad_profile": "call-center-noisy-v2",
  "transcription_profile": "realtime-balanced-v4"
}

These fields are not just logs; they are replay conditions. Without them, you cannot determine during offline replay whether a quality change came from the model, the resampler, VAD parameters, or a client-side codec upgrade.

The official Whisper implementation resamples audio to 16 kHz mono PCM and processes it in standard 30-second chunks; the audio frame step is 10 ms, and the timestamp token granularity is 20 ms. This demonstrates that format conversion and time base are integral parts of the model input, not trivial peripheral processing. However, 16 kHz should not be generalized as a fixed requirement for all models; the specific model adapter should declare its capabilities.

Recommended: Centralize Format Conversion

If the client, edge gateway, and recognition service all perform resampling, it creates difficult-to-trace duplicate processing. A more robust approach is:

Client: Responsible only for capture, lightweight packetization, and incrementing sequence numbers.
Audio Gateway: Handles decoding, channel normalization, resampling, and amplitude checks.
Model Adapter: Only receives frames that already conform to the declared format.
Raw Audio Retention: Determined by privacy policy and fault sampling strategy.

The audio gateway should attach session_id, stream_epoch, sequence_no, capture_ts, and duration_ms to every frame. Increment stream_epoch upon reconnection to prevent deduplication errors from old and new connections using the same sequence numbers.

Core Principle 2: VAD is a Tunable Latency-Completeness Decision Engine

OpenAI’s Realtime documentation distinguishes two types of turn detection: Server VAD segments based on silence, and Semantic VAD determines if the user has finished speaking based on semantic completeness. Common Server VAD parameters include activation threshold, pre-speech padding, and silence duration. Semantic VAD uses an eagerness setting to control whether to cut segments faster or allow the user more pause time.

This means VAD is not a simple “voice / no voice” switch but continuously answers three questions:

Is this signal speech?
Where does the speech start, and how much leading audio should be prepended?
Is the current pause an intra-sentence hesitation or a genuine end of utterance?

Don’t Just Tune a Single Silence Duration

Production VAD should use profiles based on the scenario:

Scenario	Characteristics	VAD Strategy Suggestion
Near-field Microphone	Low noise	Can end more aggressively
Telephone Narrowband Audio	Compression artifacts, double-talk	Handle compression and double-talk
Conference Room Far-field	Multiple speakers, echo	Increase pre-speech padding
Vehicle or Outdoor	Continuous noise	Higher noise rejection requirements
Long-Thinking Q&A	Long pauses	Allow longer intra-sentence pauses

NVIDIA Riva offers streaming intermediate results, neural VAD, End-of-Utterance, and Two-Pass EOU. The Two-Pass approach is valuable: it first produces an early transcription for downstream preprocessing, then waits for a more reliable final endpoint confirmation. This reveals a key design principle for real-time systems—early results and final results should be two distinct states, not the same text field being continuously overwritten.

Production Metrics for VAD

Beyond recognition error rates, also track:

Metric	Meaning
`speech_start_detection_ms`	Latency from true speech onset to detection
`front_end_clipping_ms`	Length of speech onset that was clipped
`end_of_utterance_wait_ms`	Wait time from true speech end to final submission
`false_turn_rate`	Proportion of noise segments submitted as speech
`split_utterance_rate`	Proportion of one sentence incorrectly split into multiple segments
`merged_utterance_rate`	Proportion of multiple sentences incorrectly merged
`empty_final_rate`	Proportion of final events that are empty

Relying only on average endpoint latency masks long P95/P99 waits and clipping in specific accents or noisy scenarios.

Core Principle 3: Partial and Final Must Be Reconciled via a State Machine

Real-time transcription typically returns both incremental and final text. OpenAI’s Realtime Transcription documentation clearly distinguishes delta from completed events and notes that completion events for different utterance segments are not guaranteed to arrive in order, requiring matching via item_id.

The correct data model is not a mutable string but a state machine for each utterance segment:

CAPTURING -> SPEECH_STARTED -> PARTIAL_ACTIVE -> AUDIO_COMMITTED
    -> FINAL_RECEIVED -> DOWNSTREAM_COMMITTED

Exception branches:
    -> CANCELLED -> TIMED_OUT -> SUPERSEDED -> REPLAY_REQUIRED

Partial is for Display and Pre-computation Only

Partial Transcripts can be used for:

Real-time captions;
Early language identification, intent candidate generation, and entity pre-warming;
Pre-fetching potentially needed tools or knowledge;
Estimating subsequent text token budgets.

However, they should not directly trigger irreversible actions, such as submitting orders, sending messages, modifying databases, or confirming identities. Subsequent audio can correct previous words; numbers, negations, and proper nouns are particularly volatile.

Final Should Not Be “Executed on Arrival” Either

Final events should first pass through the following gates:

Is session_id + stream_epoch + item_id unique?
Has this same completion event been processed before?
Is the audio sequence number range continuous, with no missing frames?
Does the Final correspond to the current valid session, not an old stream from before a reconnection?
Is it necessary to wait for earlier items to complete to restore logical order?
Does it meet the business-side minimum confidence or key entity review rules?

A simplified processing example:

type SegmentState = {
  itemId: string;
  epoch: number;
  partial: string;
  final?: string;
  status: "partial" | "final" | "committed" | "cancelled";
};

function onTranscriptEvent(event: TranscriptEvent) {
  const key = `${event.sessionId}:${event.epoch}:${event.itemId}`;
  const state = segmentStore.getOrCreate(key);

  if (event.type === "delta" && state.status === "partial") {
    state.partial += event.delta;
    publishEphemeralCaption(key, state.partial);
    return;
  }

  if (event.type === "completed") {
    if (state.status === "committed" || state.status === "cancelled") return;
    state.final = event.transcript;
    state.status = "final";
    finalReorderBuffer.offer(event.logicalOrder, state);
  }
}

Here, publishEphemeralCaption and the final business submission must use different channels. The former allows overwriting; the latter requires idempotency.

Engineering Implementation: Decompose Real-Time Voice into Six Replaceable Modules

1. Capture Adapter

Handles audio input from browsers, mobile devices, SIP, or hardware. Outputs frame sequence numbers, capture timestamps, and format declarations. Does not guess the model’s format.

2. Audio Normalizer

Performs decoding, channel conversion, resampling, amplitude normalization, and anomalous frame checks. Should expose metrics for resampling time, clipping rate, silence ratio, and input level distribution.

3. Turn Detector

Encapsulates Server VAD, Semantic VAD, or local neural VAD. Configuration must be released via vad_profile versions; direct online parameter overrides without audit are not allowed.

4. Streaming Transcriber

Responsible for sending ordered audio chunks to a vendor or self-hosted ASR and receiving Partial, Final, timestamps, confidence scores, and speaker labels. For optional fields the model doesn’t support, explicitly degrade rather than returning fabricated default values.

5. Transcript Reconciler

Handles out-of-order recovery, Partial replacement, Final idempotency, reconnection epoch isolation, and timestamp normalization. This is the most underestimated module in the voice pipeline.

6. Downstream Gate

Before delivering the final text to the LLM or business system, enforce key entity rules, session state checks, and classification of reversible vs. irreversible actions. The speech recognition result is just input evidence and should not bypass business authorization.

Timestamps and Replay: Don’t Just Save a Text String

When using sliding windows or buffered segments for long audio, timestamp drift, duplication, and hallucination are common. WhisperX uses VAD Cut & Merge with forced phoneme alignment to generate more accurate word-level timestamps, and its research notes that long-audio buffering or sliding window approaches are prone to drift, duplication, and timestamp inaccuracies.

A production replay package should include:

{
  "trace_id": "voice-01J...",
  "audio_contract_version": "audio-v3",
  "vad_profile": "call-center-noisy-v2",
  "model_profile": "realtime-balanced-v4",
  "audio_sha256": "...",
  "frame_ranges": [[0, 181], [182, 346]],
  "turn_events": [
    {"type": "speech_started", "at_ms": 420},
    {"type": "speech_stopped", "at_ms": 3180}
  ],
  "partial_events": [],
  "final_events": [],
  "client_clock_offset_ms": 0,
  "server_received_at": "2026-07-12T23:01:57Z"
}

For privacy-sensitive businesses, only store authorized sampled audio, irreversible audio fingerprints, and full event metadata. Without real audio, at least retain frame ranges, format fingerprints, event timelines, and final text to facilitate protocol error localization.

Evaluation Methodology: Measure Words, Boundaries, and Interaction Latency Simultaneously

Evaluating real-time voice with only WER or CER is insufficient. Establish a four-layer metric system:

Recognition Layer

WER/CER;
Accuracy for key entities: numbers, dates, amounts, emails, product names;
Proportion of empty, truncated, and duplicated transcriptions.

Boundary Layer

Sentence start clipping and end delay;
Incorrect sentence splits and merges;
Noise false triggers;
Recovery in interruption and double-talk scenarios.

State Layer

Proportion of characters modified from Partial to Final;
Final out-of-order rate;
Duplicate completion event rate;
Old epoch contamination rate after reconnection;
Proportion of timestamp rollbacks or overlaps.

Experience Layer

First Partial latency;
Final confirmation latency;
Latency from user finishing speech to LLM starting response;
Number of caption jitters;
Proportion of user repetitions caused by the system interrupting.

OpenAI’s current real-time transcription documentation also recommends testing with real microphones, telephone audio, accents, background noise, code-switching, domain-specific vocabulary, and long sessions, tracking empty text, truncation, and latency separately, rather than just WER.

Applicable Scenarios

This architecture is suitable for:

Real-time customer service and phone bots;
Meeting captions, minutes, and speaker attribution;
In-vehicle, wearable devices, and voice assistants;
Voice-driven Agents or tool calls;
Voice input in medical, insurance, and financial sectors requiring key entity review;
Multi-vendor ASR switching and self-hosted model canary deployments.

For offline podcast transcription without real-time interaction requirements, the Partial state machine can be simplified, but audio contracts, segmentation, and timestamp replay should still be retained.

Common Misconceptions

Misconception 1: Faster VAD Means Better Experience

Cutting segments too quickly truncates intra-sentence pauses and increases text fragmentation. Endpoint wait time should be tuned per scenario, not minimized blindly.

Misconception 2: Partial is Just a Prefix of Final

Incremental results may rewrite previous text; you cannot assume only character appending. Both UI and storage must support replacement.

Misconception 3: Completion Events Are Naturally Ordered

Network conditions, parallel recognition, and server scheduling can cause completion events for different utterance segments to arrive out of order. Reconciliation must be based on stable identifiers and logical sequence numbers.

Misconception 4: Converting All Audio to 16 kHz is Sufficient

Sample rate is just one part of the contract. Channel count, encoding, byte order, frame size, reconnection handling, and time base all affect the result.

Misconception 5: Real Noise Can Be Replaced with White Noise

Telephone compression, echo, double-talk, far-field reverberation, keyboard sounds, and mobile network packet loss have different structures. Synthetic white noise only covers limited scenarios.

Go-Live Checklist

More FAQ

Should VAD be on the client or server?

Both are possible, but only one component should be responsible for final utterance submission. Client-side VAD is suitable for reducing bandwidth and interaction latency; server-side VAD is easier for unified governance. A common approach is for the client to provide suggestive events, while the server makes the final decision based on a unified Profile.

When can Partial trigger the LLM early?

Only for reversible pre-computations, such as intent candidate generation, retrieval pre-fetching, or model warm-up. Any irreversible tool calls must wait for Final and business gates.

How to determine if a VAD adjustment is truly an improvement?

Compare endpoint wait time, front-end clipping, incorrect sentence splits, noise false triggers, Final quality, and user interruption rate simultaneously. Reducing latency at the cost of increased clipping is not an improvement.

References

OpenAI, Voice activity detection (VAD): https://developers.openai.com/api/docs/guides/realtime-vad
OpenAI, Realtime transcription: https://developers.openai.com/api/docs/guides/realtime-transcription
OpenAI Whisper, audio.py: https://github.com/openai/whisper/blob/main/whisper/audio.py
Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision: https://arxiv.org/abs/2212.04356
Bain et al., WhisperX: Time-Accurate Speech Transcription of Long-Form Audio: https://arxiv.org/abs/2303.00747
NVIDIA Riva, ASR Overview: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-overview.html