Should a real-time voice agent use speech-to-speech or an ASR-LLM-TTS pipeline?

For low latency, natural interruptions, and real-time tool calls, prefer speech-to-speech. For scenarios requiring strong audit trails, replaceable transcription, or strict approval before responses, a chained pipeline is easier to govern.

Should VAD parameters be as aggressive as possible?

No. Shorter silence thresholds reduce wait time but increase the risk of truncation, false interruptions, and submitting half-sentences. In production, test parameters per scenario based on noise, accent, speaking speed, and business tolerance.

Does a voice agent need user confirmation for tool calls?

For operations with side effects like orders, payments, identity changes, privacy actions, cancellations, deletions, or outbound calls, explicit confirmation should be required at the voice or text layer, with transcription, tool parameters, and confirmation events logged.

LLM Realtime Voice Agent in Production: Building Stable Conversations with WebRTC, VAD, and Barge-In Recovery

The Core Challenge: Voice Agents Must Handle Real Conversations, Not Just Speak

Text-based agents typically struggle with context, tool parameters, and output formatting. Real-time voice agents introduce a far more chaotic variable: users don’t press Enter to submit a request; they speak continuously amidst noise, pauses, corrections, interruptions, hesitations, and emotion.

A demo voice assistant can get by with just three steps: record audio, send it to the model, play the response. But in production scenarios like customer service, sales, education, meetings, in-car systems, or internal operations, specific problems emerge: the model responds before the user finishes; old responses keep playing when the user interrupts; noise is mistaken for a new query; tool calls start but the user changes their mind; the transcription and audio response don’t match; and there’s no auditable evidence chain after a call ends.

Therefore, the focus for productionizing a real-time voice agent isn’t just choosing a voice model. It’s about building a controllable pipeline around transport, endpoint detection, barge-in recovery, transcription buffering, tool calls, human handoff, and audit logging.

Architecture First: Speech-to-Speech vs. Chained Pipeline

OpenAI’s Voice Agents documentation categorizes voice agents into two types: a live audio speech-to-speech session where the model directly handles real-time audio input and output, and an explicit chained voice pipeline of speech-to-text, text agent, text-to-speech. The former is ideal for natural, low-latency conversations that support barge-in; the latter suits workflows requiring visible intermediate text, approval, policy checks, and swappable modules.

Use this boundary for production selection:

Scenario Type	Recommended Architecture	Reason
Coaching, shopping guides, Q&A, lightweight customer service, voice search, low-risk assistants	speech-to-speech	Low latency, natural interruptions, unified session events
Finance, healthcare, government customer service, complaint handling, ticket changes, identity verification	Chained pipeline	Visible intermediate text, policy checks, approval, audit
Mixed scenarios	speech-to-speech for general Q&A, text-based confirmation for high-risk actions	Balances experience and safety

WebRTC, WebSocket, and SIP: Don’t Mix Up the Transport Layer

Real-time voice agents have at least three common access methods:

WebRTC: Ideal for direct audio capture and playback in browsers and mobile apps. In OpenAI’s Realtime WebRTC example, the browser creates an RTCPeerConnection, captures the microphone track, establishes a data channel, and completes SDP exchange via an application server. Benefits: low latency, natively designed for real-time media, good browser support.
WebSocket: Better suited for scenarios where the server already has the raw audio stream, such as telephony systems, meeting recordings, edge gateways, or media workers. Google’s Gemini Live API also uses stateful WebSocket connections supporting audio, image, and text input.
SIP: More appropriate for traditional telephone networks, typically part of contact centers, voice gateways, or cloud communication platform integrations.

A common engineering mistake is treating WebRTC, WebSocket, and SIP as just “different connection types.” They determine authentication boundaries, log collection methods, reconnection strategies, media quality metrics, network issue diagnosis, and security models.

VAD: The Pacemaker of Your Voice Agent

VAD (Voice Activity Detection) determines when a user starts speaking and when they finish a turn. OpenAI’s Realtime VAD documentation explains that enabling VAD triggers events like input_audio_buffer.speech_started and input_audio_buffer.speech_stopped, allowing the application to manage session state or process transcription in segments.

VAD is not a simple on/off switch; it’s the pacemaker of your voice agent. It directly impacts three things:

Time to First Response: If the system waits too long to decide the user has finished, the assistant feels slow.
Truncation Risk: If the system cuts off too early, the user’s remaining words are lost, and the model may respond to a half-sentence.
Interruption Experience: If the system can’t quickly detect when the user interjects, the old response continues playing, creating a “the machine isn’t listening” experience.

OpenAI’s documentation distinguishes between server_vad and semantic_vad. server_vad segments audio based on silence, with configurable threshold, prefix_padding_ms, and silence_duration_ms. semantic_vad determines the end of a turn based on whether the user’s utterance is semantically complete, helping to reduce premature interruptions.

A production recommendation is to group VAD parameters by scenario rather than using a single global configuration. For example:

{
  "quiet_office": {
    "type": "server_vad",
    "threshold": 0.45,
    "prefix_padding_ms": 300,
    "silence_duration_ms": 450
  },
  "noisy_call_center": {
    "type": "server_vad",
    "threshold": 0.65,
    "prefix_padding_ms": 400,
    "silence_duration_ms": 700
  },
  "slow_speaker_or_elderly_user": {
    "type": "semantic_vad",
    "eagerness": "low"
  }
}

This is just an example; do not copy it directly to production. Real parameters must be stress-tested with actual recording samples, noise, devices, languages, accents, and business tolerance.

Barge-In Recovery: It’s More Than Just “Stop Playing”

Real-time voice products often highlight barge-in, the ability for a user to interrupt the model while it’s speaking. Google’s Gemini Live API also lists Barge-in as a key capability.

But from an engineering perspective, barge-in involves at least four state actions:

Detect the user starting to speak: Relies on VAD or frontend audio energy detection.
Stop or fade out the current TTS/model audio output: Otherwise, the user hears the old answer continuing.
Cancel or freeze the current response: Otherwise, the old response might continue generating text, calling tools, or writing state.
Restore context: The system needs to know what content has been played to the user, what was generated but not played, which tool calls have executed, and which must be undone or compensated.

A more robust approach is to decompose each voice interaction turn into event logs:

type VoiceTurnEvent =
  | { type: "user_speech_started"; ts: number }
  | { type: "user_speech_stopped"; ts: number; audioSegmentId: string }
  | { type: "transcript_delta"; ts: number; text: string; final: boolean }
  | { type: "assistant_audio_started"; ts: number; responseId: string }
  | { type: "assistant_audio_played"; ts: number; responseId: string; offsetMs: number }
  | { type: "user_barge_in"; ts: number; interruptedResponseId: string }
  | { type: "response_cancelled"; ts: number; responseId: string }
  | { type: "tool_call_requested"; ts: number; tool: string; argsHash: string }
  | { type: "tool_call_confirmed"; ts: number; confirmationText: string }
  | { type: "handoff_to_human"; ts: number; reason: string };

With event logs, barge-in recovery no longer depends on “what’s in current memory.” You can reconstruct: when the user interrupted, where the assistant was in its response, what content the user heard, and which actions are not yet complete.

Transcription Buffering: Don’t Just Save the Final Text

Many teams only save the final transcription text, which is insufficient for debugging voice agents. Real-time voice systems need to preserve three types of material:

Audio Segment Index: You don’t necessarily need to store raw audio long-term, but you should be able to locate the segment, its duration, sample rate, device info, and noise metrics.
Incremental Transcription Deltas: Save the differences between partial and final transcripts. This is crucial for diagnosing whether the model acted prematurely on partial text.
Playback Progress: Record the actual offset of assistant audio that was played. The model may generate 30 seconds of audio, but that doesn’t mean the user heard 30 seconds.

If the business involves dispute handling, quality assurance, or compliance audits, the transcription buffer must be linked with user confirmation events, tool call parameters, and human handoff records. Otherwise, you can only see “what the system answered,” not “what the user actually heard and confirmed.”

Tool Calls: Voice Scenarios Demand a Confirmation Layer

Real-time voice agents can easily translate natural language directly into tool calls, such as looking up orders, changing addresses, canceling services, creating tickets, sending SMS, or scheduling appointments. The problem is that voice input is more prone to mishearing, misspeaking, and ambiguous context references than text input.

Therefore, tool calls should be governed in three categories:

Risk Level	Example	Strategy
Low Risk	Weather queries, knowledge lookups, FAQ retrieval	Execute directly
Medium Risk	Account queries, quote generation, draft creation	Execute but log parameters and source transcription
High Risk	Payments, deletions, cancellations, refunds, outbound calls, identity changes	Require secondary confirmation; repeat key parameters

When confirming, don’t just ask “Are you sure?” Repeat the key parameters, e.g., “I will cancel your appointment on July 8th at 3 PM. Is that correct?”

A deployable tool strategy can be written as configuration:

tools:
  lookup_order:
    risk: low
    require_confirmation: false
    audit: transcript_and_args
  update_shipping_address:
    risk: high
    require_confirmation: true
    confirmation_fields:
      - order_id
      - new_address
    audit: transcript_args_confirmation_audio_offset
  refund_payment:
    risk: critical
    require_confirmation: true
    require_human_review: true
    audit: full_event_log

Human Handoff: Don’t Wait for Complete Model Failure

Human handoff for voice agents needs to be more proactive. Text-based bots can let users slowly read error messages; in voice scenarios, user patience is shorter and emotions change faster.

Set several hard trigger conditions:

Two consecutive VAD truncations
Two consecutive low-confidence ASR results
User explicitly mentions complaints, refunds, emergencies, risks, legal, medical, or identity issues
Tool call parameters cannot be confirmed
The model asks the user to repeat the same information twice
User interruption frequency is too high

Human handoff shouldn’t just be “transfer to an agent.” A better practice is to package the context for the human agent: the last N turns of transcription, user-confirmed parameters, incomplete tool calls, a model-generated summary, risk tags, and audio segment indices. This allows the agent to pick up directly without making the user repeat themselves.

Applicable Scenarios

Real-time voice agents are suitable for scenarios demanding low latency and natural interaction, such as voice customer service, pre-sales guidance, spoken language practice, in-car assistants, meeting assistants, game NPCs, smart hardware, real-time translation, and internal operations assistants.

However, they are not suitable for covering all complex business from the start. For high-value transactions, strict identity verification, medical advice, legal judgments, financial commitments, and irreversible operations, retain text-based confirmation, human approval, or fallback to traditional processes.

Common Pitfalls

Only stress-testing model latency, not end-to-end conversation latency: User perception is the sum of microphone capture, network transmission, VAD wait time, model inference, TTS generation, and playback buffering.
Using default VAD parameters: Thresholds vary drastically with different noise, devices, accents, speaking speeds, and business scenarios.
Only recording the final text: Real-time systems must save event streams, transcription deltas, playback progress, interruption points, and tool parameters to debug production issues effectively.
Treating barge-in as a UI feature: It’s fundamentally a session state machine capability requiring cancellation of old responses, stopping playback, freezing tool calls, and restoring context.
Not confirming voice-initiated tool calls: Mishearing and reference errors in voice amplify the risk of side effects, especially for orders, payments, deletions, refunds, and outbound calls.

Pre-Launch Checklist

Before going live, at least check the following items:

Layer	Check Item
Transport	Is the access method clear? Are only temporary credentials used on the browser side? Are security identifiers saved on the server? Could reconnection cause duplicate responses or tool calls?
Audio	Are different devices, noise levels, accents, and speaking speeds covered? Are sample rate, encoding, packet loss, first audio latency, and playback interruptions recorded?
VAD	Are thresholds configured per scenario? Is there a test set for false truncation, long silences, noise triggers, and user interruptions?
Session	Can old responses be cancelled? Is the audio offset the user has heard recorded? Is there a distinction between “generation complete” and “playback complete”?
Tools	Are tools classified by risk? Is secondary confirmation required? Are confirmation text, parameter hashes, and tool results logged? Is there a compensation or rollback mechanism on failure?
Human Handoff	Are there clear trigger conditions? Are transcriptions, risk tags, incomplete actions, and context summaries passed to the human agent?
Observability	Are metrics tracked for VAD false positive rate, barge-in success rate, first audio latency, transcription correction rate, tool confirmation failure rate, human handoff rate, user repetition rate, and call abandonment rate?

FAQ

Does a real-time voice agent have to use WebRTC?

Not necessarily. WebRTC is usually more suitable for direct audio capture and playback in browsers and mobile apps. For scenarios where the server already has the audio stream, such as telephony systems or media workers, WebSocket or SIP may be more natural. The key is to design the transport method together with authentication, logging, reconnection, and observability.

Can semantic VAD completely solve the problem of premature responses?

No. Semantic VAD can reduce the probability of early segmentation, but real-world pauses, hesitations, accents, noise, and overlapping speech will still cause misjudgments. It should be used in conjunction with a business state machine, barge-in recovery, and human handoff thresholds.

Does a voice agent need to save raw audio?

Not necessarily. For scenarios with high privacy requirements, you can save only segment indices, transcriptions, event logs, and necessary short-term caches. However, if quality assurance, dispute handling, or compliance review is involved, you must define the scope of audio retention, retention period, access permissions, and anonymization policies.