A voice agent does not respond when the model finishes. It responds when the user hears the first playable audio chunk.
That distinction changes the architecture. End-of-turn detection, transcript finalization, context retrieval, model startup, text-to-speech, transport, and client buffering all consume the same silence budget. Optimizing only model latency can leave the conversation feeling unchanged.
The useful latency number
Time to first audio = endpointing + transcript finalization + first useful model branch + TTS startup + transport and playout.
The free calculator maps those stages, compares a parallel fast-and-slow design with a sequential slow-model path, and shows what percentage of the delay happens before inference begins.
1. Start the clock at the end of the user's intent
The first hard problem is deciding when the user has finished a turn. A silence threshold is simple, but it can wait too long or cut off a speaker who pauses. A semantic endpointer can use the partial transcript and conversational state, but it adds its own inference and calibration work. Many production systems combine acoustic and semantic signals.
For a latency budget, record the time between the final meaningful user audio and the moment the speech recognizer is allowed to finalize the turn. Keep this number separate from transcription. Otherwise a slow endpointer can disappear inside an apparently slow speech model.
2. Separate streaming transcription from finalization
Streaming speech recognition may produce partial words while the user is still talking. The response model usually needs a stable enough representation to act. That finalization step can include decoder work, punctuation, language detection, diarization, or a network round trip.
Measure the timestamp when the transcript becomes usable by the response path. Do not use the time when the first partial token appeared unless the architecture actually starts safe work from partial transcripts.
3. Race a fast interaction model against a slow reasoning branch
A sequential architecture waits for retrieval, tools, and a larger reasoning model before it begins speech. That keeps the logic simple, but it places every deep operation on the conversational critical path.
A parallel architecture splits the work:
Fast interaction branch: produces a safe, context-aware opening, handles acknowledgements, manages turn-taking, or asks a clarifying question.
Slow reasoning branch: prepares context, calls tools, searches, or runs deeper reasoning.
Coordinator: accepts the first useful text, begins speech, and incorporates or hands off to the deeper result when it arrives.
The first-audio calculation uses the faster of two values: the fast model's time to useful text, or context preparation plus the slow model's time to useful text. TTS and playout begin after that winning branch.
This is not free speed. The system must prevent the fast branch from making claims it cannot support, cancel stale work after an interruption, and make the handoff sound like one coherent speaker. If those controls are missing, lower latency can produce faster mistakes.
4. Work through one latency budget
Consider an example pipeline with these planning values:
Stage | Time | What it represents |
|---|---|---|
End-of-turn detection | 280 ms | Releasing the user's completed turn |
Transcript finalization | 220 ms | Stable text available to both branches |
Context preparation | 120 ms | Memory, retrieval, policy, or tool context |
Fast model to first text | 180 ms | Safe opening response |
Slow model to first text | 650 ms | Deeper response after context is ready |
TTS startup | 160 ms | First synthesized audio chunk |
Transport and playout | 60 ms | Delivery and client audio buffer |
The transcript is ready after 500 ms. The fast branch produces useful text 180 ms later. Add 160 ms for TTS and 60 ms for playout, and the first audio lands at 900 ms.
The sequential slow-model path waits 120 ms for context and 650 ms for the slow model. With the same speech stages, it reaches first audio at 1,490 ms. Parallel execution saves 590 ms in this estimate, even though the deep branch still needs 1,270 ms to become ready.
The less obvious result is that 500 of the 900 milliseconds happen before either response model produces text. A model swap alone cannot remove that part of the delay.
5. Diagnose the largest stage, not the most fashionable one
Once every stage is visible, choose the optimization that attacks the actual bottleneck.
Endpointing dominates: test adaptive silence windows, semantic endpointing, or safe speculative work on partial turns.
Transcript finalization dominates: inspect streaming cadence, region placement, model size, and what downstream work really requires a final transcript.
Context preparation dominates: prefetch stable context, reduce retrieval fan-out, or separate must-have context from enrichment that can arrive later.
The fast branch dominates: shorten its prompt, constrain its job, route it to a genuinely low-latency model, and measure time to useful text rather than total completion.
TTS dominates: stream smaller chunks, inspect voice/model choice, place synthesis near the response service, and verify when the client can actually begin playback.
Playout dominates: inspect network routes, WebRTC or websocket behavior, jitter buffering, audio decoding, and device-specific startup.
6. Measure one percentile from one trace population
A useful budget is internally consistent. Do not add median endpointing to tail-latency transcription and a best-case model startup. Pick a percentile and calculate every stage from the same environment, region, device class, and traffic population.
For each turn, log at least:
Last meaningful user-audio timestamp
Endpoint release timestamp
Usable transcript timestamp
Fast and slow branch start timestamps
First useful text from each branch
First synthesized audio chunk
First audible client playout
Interruption, cancellation, and branch-handoff events
Calculate both end-to-end latency and stage-level distributions. A healthy median can hide a tail where endpointer errors, cold models, tool calls, or client buffers create multi-second pauses.
7. Validate the experience, not just the stopwatch
Low time to first audio is useful only when the first words move the conversation forward. A fast filler phrase can reduce the metric while making the product feel evasive. A fast branch that later contradicts the slow branch damages trust.
Pair latency traces with conversation tests:
Can users interrupt naturally?
Does the system cancel stale speech and stale tool work?
Can the fast branch acknowledge uncertainty without inventing an answer?
Does the slow result arrive at a natural handoff point?
Are latency and quality stable across accents, noisy audio, long pauses, and weaker networks?
The architecture decision is not “fast model or smart model.” It is which work must block the first audible response, which work can run concurrently, and how the system stays coherent when the branches finish at different times.
Map the silence in your own stack
Enter measured stage timings, compare parallel and sequential paths, and share the exact budget with your team.
Get the next architecture teardown
ResearchAudio turns new AI systems into practical constraints and adoption decisions for engineers and builders. Join 52,000+ readers for free →
