Background: The Challenge of Voice Agents Goes Beyond “Just Speaking”
The typical pipeline for a text-based LLM application is “user input text → model generates text → frontend displays the result.” A Voice Agent’s pipeline is far more complex: when a user speaks, the system must receive audio in real-time, detect human speech, transcribe the speech to text, determine when the user has finished, send the segment to the LLM for inference, convert the response back to speech, and play it out.
If each step waits for the previous one to complete entirely, the user will perceive a noticeable lag. If triggered too early, the system might interrupt the user or treat a partial utterance as a complete intent. The core of a good speech LLM experience isn’t the capability of a single model, but the coordination between streaming ASR, VAD, context segmentation, LLM inference, and TTS playback.
The OpenAI Realtime documentation categorizes real-time audio scenarios into different paths like Voice Agents, real-time translation, real-time transcription, and speech generation. It points out that real-time conversations are best suited for low-latency live audio, while file-based or bounded audio is better for request-based APIs. For production systems, this distinction is critical: the goal for real-time conversations is to optimize the user’s perceived latency, not just the accuracy of a single offline transcription.
Core Principle: Cascaded Streaming Architecture Remains Production Mainstream
End-to-end speech-to-speech models can directly receive audio and output audio, but enterprise-grade Voice Agents still frequently employ a cascaded streaming architecture:
Audio Input → VAD → Streaming ASR → Segment Buffer → LLM → Streaming TTS → Audio OutputThe advantage of this architecture is controllability. ASR can be independently optimized for recognition accuracy, the LLM can reuse the text agent’s tool calling, permission control, and business state, and TTS can independently select voice, speed, and playback strategy. The downside is a longer pipeline, and without streaming and parallel processing, end-to-end latency can quickly accumulate.
The Whisper paper demonstrates that large-scale weak supervision can improve the generalization of multilingual, multi-task speech recognition. However, a production Voice Agent cannot just focus on the ASR model itself; it must also handle engineering challenges like noise, accents, interruptions, pauses, long sentences, repetitions, and unstable transcriptions. For the user, the final experience depends on how natural the entire “listen, think, speak” pipeline feels.
Engineering Implementation: Decomposing the Speech Pipeline into Five Control Points
1. Audio Ingress: Don’t Feed All Audio into the Model
The audio ingress layer must first address sampling, encoding, noise reduction, reconnection, and client buffering. WebRTC is typically more suitable for browsers and mobile devices. When the server receives media streams, telephony streams, or worker audio, WebSocket or other server-side streaming paths can be used.
The ingress layer should also attach stable session IDs, user IDs, device information, sample rates, language preferences, and business scenarios. When troubleshooting latency, noise, and misrecognition later, these fields are far more valuable than the raw transcription text alone.
2. VAD: Deciding When to Listen and When to Submit
VAD (Voice Activity Detection) on the surface determines if human speech is present, but in a Voice Agent, it simultaneously influences four things:
| Dimension | Description |
|---|---|
| Start Transcription | Initiate the ASR stream upon detecting speech |
| Continue Waiting | Whether to pause or continue waiting during silence |
| Submit Segment | Determine if the current segment is ready for submission |
| Trigger Response | Decide when the LLM should start inference |
If VAD is too sensitive, background noise will be treated as user input. If VAD is too sluggish, the system will be slow to respond after the user finishes speaking. A more common problem is that VAD segments solely based on silence duration, breaking a complete intent into multiple fragments or prematurely triggering a response during a brief user pause.
Production systems typically need to combine three types of signals: voice activity, silence duration, and semantic completeness. In other words, a 300-millisecond pause by the user doesn’t necessarily mean they are done. But if the current segment already forms a complete intent and subsequent silence exceeds a threshold, the next step can be triggered.
3. Streaming ASR: Distinguishing Between Partial and Committed Transcriptions
Streaming ASR continuously outputs partial transcripts, but these intermediate results can be rolled back, rewritten, or completed. If the system writes every partial result directly into the long-term context, the LLM will later see a large amount of repetition and erroneous segments.
A more robust approach is to distinguish between two types of text:
{
"transcript_buffer": {
"partial": "Temporary text the user is speaking, can be updated or overwritten",
"committed": "Confirmed semantic segment, can enter the dialogue context",
"confidence": "Transcription confidence or quality signal",
"segment_id": "Segment number for tracking and replay"
}
}Partial transcriptions are used for real-time captions, intent prediction, and preparation. Committed transcriptions are the ones that enter the formal context. For high-accuracy scenarios like customer service, insurance, finance, and healthcare, secondary confirmation can be performed on key entities such as amounts, dates, names, ID numbers, addresses, and product names.
4. Segmented Context: Don’t Just Stuff Raw Speech Transcriptions into the Window
Speech input naturally contains filler words, repetitions, self-corrections, and meaningless pauses. For example, a user might say: “I want to check, um, not last month, this month’s bill.” If the system adds all the raw transcription to the context, the LLM might misinterpret the user’s intent.
It is recommended to divide the segmented context into four layers:
| Layer | Content | Purpose |
|---|---|---|
| Raw Transcript | Full original transcription | Auditing, replay, troubleshooting |
| Committed Segment | Confirmed semantic segment | LLM dialogue context |
| Dialogue State | Current task state | Track business process progress |
| Rolling Summary | Historical intents and confirmed facts | Long conversation compression, cross-turn memory |
The benefit is that the LLM sees “inferable context,” not an accumulation of noise.
5. TTS and Interruption: Don’t Wait for the Full Response to Generate Before Playing
Voice responses must also be streamed. If the TTS waits for the LLM to generate the complete answer before synthesizing, the user will perceive significant latency. A better approach is to have the LLM output text in short sentences, semantic chunks, or at punctuation boundaries, while the TTS synthesizes and the player plays the audio as it’s received.
User interruption must also be supported. Once the user starts speaking, the system should pause or stop the current playback and decide whether to discard the unplayed content. Without interruption control, the Voice Agent will sound like an answering machine that keeps talking, resulting in a poor experience.
Common Misconceptions
Misconception 1: Only Optimizing ASR Accuracy
ASR accuracy is important, but the perceived experience of a Voice Agent also depends on first-word transcription latency, submission latency, LLM first-token latency, TTS first-audio latency, and playback buffering. Improving the accuracy of a single point does not necessarily improve the overall experience. The end-to-end latency breakdown is as follows:
| Latency Metric | Description |
|---|---|
| ASR First Segment Latency | Audio input to first partial transcript |
| ASR Submission Latency | Audio input to committed segment confirmation |
| LLM First Token Latency | Context sent to LLM to first token generated |
| TTS First Audio Latency | LLM output to first audio frame synthesized |
| Playback Start Time | First audio frame to user hears sound |
Misconception 2: Treating Silence as the End of a Semantic Unit
Users often pause, think, and rephrase while speaking. Using only a silence threshold to determine the end of an utterance leads to interruptions or incorrect segmentation. Production systems should combine semantic completeness, business state, and interruption strategies.
Misconception 3: Treating Partial Transcripts as Final Facts
Intermediate results from streaming transcription can change. Writing partial results directly into the context causes repetition, misrecognition, and state pollution. Only committed segments should be written into the formal context.
Misconception 4: Ignoring End-to-End Observability
Monitoring only model latency is insufficient. A Voice Agent needs to record audio ingress latency, VAD decision time, ASR partial latency, ASR commit latency, LLM first token, TTS first audio, playback start time, and user interruption events.
Pre-Launch Checklist
It is recommended to check the following items before launch:
- Whether real-time conversations, request-based transcriptions, and file transcriptions are treated as three distinct paths.
- Whether sensitivity, silence threshold, and maximum wait time are configured for VAD.
- Whether partial transcripts and committed transcripts are distinguished.
- Whether segment_id is saved for troubleshooting transcription and context issues.
- Whether key entities are confirmed or undergo secondary validation.
- Whether raw transcriptions, confirmed segments, business state, and summary state are managed in separate layers.
- Whether user interruption of current TTS playback is supported.
- Whether end-to-end latency breakdown is recorded, not just model response time.
- Whether testing is done with real-world noise, accents, weak networks, long sentences, and multi-turn interruptions.
- Whether audio tokens, session duration, and concurrency limits are set to prevent cost overruns.
Applicable Scenarios
This approach is suitable for phone customer service, intelligent outbound calls, meeting assistants, in-car assistants, voice search, real-time translation, insurance claims consultation, financial customer service, medical triage, and enterprise internal voice assistants.
If the business is only offline meeting transcription, a complex Voice Agent architecture is unnecessary; request-based transcription and post-processing summarization might be simpler. If the business requires real-time Q&A, interruption, tool calling, and state tracking, a streaming voice pipeline should be the primary consideration.