Voice AI Latency: How to Get a Voice Agent Under 800ms (Full Budget Breakdown)

What is a good latency for a voice AI agent?

Under 800ms voice-to-voice, measured from the moment the caller stops speaking to the first syllable of the reply. That target is broad consensus across independent 2026 write-ups, not a single vendor's claim, and it comes from human turn-taking: people leave roughly 200–300ms of gap between speakers, and callers start reading agents as broken somewhere past a second.

The number that decides whether callers complain is the p95, not the median. A pipeline with a 400ms median and a 1,800ms tail feels worse to a caller than a consistent 700ms, because the tail is when they conclude nothing is coming and start talking, and now you have an interruption problem on top of a latency problem. DestiLabs' 2026 benchmark across production deployments reported roughly 680ms p50 against roughly 1,180ms p95, the shape of a real fleet: the median is fine and the tail is not.

So write the budget down per stage, instrument each stage separately, and alert on the p95 of each. Once the budget exists, arguments about which component to optimise stop being a matter of seniority. Before then, every team optimises the LLM, because the LLM is the part they find interesting.

If you're choosing components rather than tuning existing ones, the two decisions with the largest latency consequence are covered in choosing an STT provider for voice agents and choosing a TTS provider for voice agents.

Where does the time actually go in a voice agent?

Into end-of-turn detection, far more than anyone expects. On stock settings, before a single model runs, you can already be 550ms into your budget, and it gets reported to you as "the LLM is slow," because that's the stage everyone instruments first.

The arithmetic, from LiveKit's turn-detector documentation. The session waits between min_delay and max_delay after speech stops before committing a turn, defaulting to 0.5 and 3.0 seconds. In VAD mode that min_delay "effectively behaves like max(VAD silence, min_delay)." Silero VAD's default min_silence_duration is 0.55 seconds. So on defaults your effective endpointing floor is 550ms, the larger of the two, and it's entirely dead air.

Enable the audio turn detector and LiveKit's documented defaults change to min_delay 0.3 seconds and max_delay 2.5 seconds, because the model gives a confident end-of-turn signal so the session can commit sooner. The docs also require the VAD's min_silence_duration to be at least 0.25 seconds, and raise a ValueError at session start if it's lower. Set Silero to 0.25 and your floor becomes max(0.25, 0.3) = 300ms.

That's a 250ms saving from two configuration values and a model that costs nothing to run locally. It's the largest single latency win in a LiveKit pipeline, it requires no provider change, and it's available to anyone who reads the endpointing defaults table. The mechanics of why a semantic model can commit sooner than a timer are in turn detection and barge-in for voice agents.

How do you actually measure voice agent latency?

Five timestamps per turn, one structured log line, and the whole distribution rather than an average. Emit fewer than five and you'll see that a turn was slow but not which stage caused it, which is where most teams are when they start reading posts like this one.

The five: user speech ends, turn committed, STT final arrives, LLM's first token arrives, first TTS byte arrives. Four deltas fall out, and the sum of the four plus your measured transport is your voice-to-voice number. Log them with a call ID and a turn index so a support complaint traces to a specific turn rather than a vague afternoon.

Measure at the edge where you can. Server-side timestamps flatter you by omitting the transport legs at both ends, and the phone path behaves differently from the web path: different codecs, jitter and packet loss. If your only measurements are from a browser on your office wifi, your numbers are optimistic by an amount you haven't quantified.

Then never report an average. Report p50, p95 and p99 per stage. An average hides exactly the failures that cost you calls, and a p99 four times your p50 is a capacity or rate-limit problem wearing a latency costume.

import json, time
from dataclasses import dataclass, field, asdict

def now_ms() -> float:
    return time.perf_counter() * 1000.0

@dataclass
class TurnTrace:
    call_id: str
    turn: int
    speech_end_ms: float = 0.0
    turn_committed_ms: float = 0.0
    stt_final_ms: float = 0.0
    llm_first_token_ms: float = 0.0
    tts_first_byte_ms: float = 0.0
    transport_ms: float = 0.0   # measured separately, not guessed

    def deltas(self):
        return {
            "endpointing_ms": self.turn_committed_ms - self.speech_end_ms,
            "stt_final_ms":   self.stt_final_ms - self.turn_committed_ms,
            "llm_ttft_ms":    self.llm_first_token_ms - self.stt_final_ms,
            "tts_ttfa_ms":    self.tts_first_byte_ms - self.llm_first_token_ms,
        }

    def emit(self):
        d = self.deltas()
        d["voice_to_voice_ms"] = sum(d.values()) + self.transport_ms
        print(json.dumps({
            "evt": "turn_latency",
            "call_id": self.call_id,
            "turn": self.turn,
            **{k: round(v, 1) for k, v in d.items()},
        }))

# Does NOT handle: interrupted turns (the deltas go negative -- drop them),
# tool-call round trips, or clock skew between worker and edge.

$ $ tail -f agent.log | jq -c 'select(.evt=="turn_latency")'
{"turn":1,"endpointing_ms":552,"stt_final_ms":94,"llm_ttft_ms":188,"tts_ttfa_ms":58,"voice_to_voice_ms":952}
{"turn":2,"endpointing_ms":551,"stt_final_ms":101,"llm_ttft_ms":211,"tts_ttfa_ms":61,"voice_to_voice_ms":984}
{"turn":3,"endpointing_ms":550,"stt_final_ms":88,"llm_ttft_ms":1740,"tts_ttfa_ms":57,"voice_to_voice_ms":2495}
# note turn 3: the tail is the LLM, but the FLOOR is endpointing.
# 550ms on every single turn, before any model runs.

What are the three biggest voice agent latency mistakes?

Non-streaming TTS, synchronous tool calls before the first token, and over-conservative turn detection misdiagnosed as a slow model. In that order of frequency, and reverse order of how often they're correctly identified.

Non-streaming TTS is the most common and easiest to fix. If your code awaits a completed synthesis before publishing audio, you've turned time-to-first-audio into time-to-full-audio and added the entire utterance to the perceived gap. Both Rime models expose HTTP and WebSocket streaming; so does every provider worth using. Iterate the stream, and start synthesising on the first clause rather than the finished paragraph.

Synchronous tool calls are the sneakiest, because they're correct-looking code. If the model needs a database lookup before it can say anything, the caller hears silence for the duration of your query plus the model round trip. The fix is conversational, not technical: acknowledge first, then look up. "Let me check that for you" buys 400ms of perceived time for free and is what a human would say anyway.

The third is the one this post exists for. A turn-detection threshold tuned defensively adds 300–500ms of dead air to every turn, and because it's perfectly consistent it never looks like a spike, so it gets blamed on whatever component changed last. Measure endpointing as its own stage or you'll spend a sprint optimising the wrong thing.

Which component choices actually move the latency number?

TTS time-to-first-audio and the turn-detection architecture. Model quality barely matters for latency once you're streaming, and STT differences are small enough to sit inside the noise of your transport.

On TTS, the only number that matters is time to first audio, and Rime is the one vendor publishing it properly: Mist v3 at 37ms p50 and 56ms p90, Coda at 96ms p50 and 98ms p90, both stated on rime.ai/pricing as measured at one concurrency. That last clause does enormous work and Rime deserves credit for including it. Your production concurrency isn't one, and nobody's published TTFA survives contact with a busy socket pool unchanged.

On STT, Deepgram's Flux fuses transcription with turn detection in a single model, and Deepgram reports roughly 30% fewer false interruptions and 200–600ms lower agent response latency against a traditional pipeline. Those are Deepgram's own reported figures, not an independent measurement, so read them as a vendor's directional claim rather than a benchmark result. The architecture is real, and a genuine alternative to running a separate end-of-turn model.

Realtime speech-to-speech models cut latency by removing pipeline stages, and cost materially more. LiveKit lists OpenAI GPT Realtime at $0.0676 per minute against GPT-5 mini at roughly $0.0011 per minute on its published token assumptions, roughly 60x on the model line alone. That's a real trade, not a free win, and it should be made on your latency budget rather than enthusiasm.

How do you cut latency without spending more money?

Fix the endpointing floor, stream everything, acknowledge before you look things up, and keep your sockets warm. All four are free, and together worth more than any provider swap you could make.

The endpointing fix above is worth 250ms. Streaming end to end is worth 300–800ms depending on how long your agent's utterances are. Acknowledgement before tool calls buys 400ms of perceived time. Connection reuse, keeping the STT socket open and the TTS client pooled across turns rather than reconnecting, removes a TLS handshake from the critical path on every turn after the first.

Colocation is the fifth free win and the most commonly skipped. If your agent workers run in one region and your callers arrive through a SIP entry point in another, you're paying that round trip twice per turn. LiveKit advertises media transport worldwide in under 250ms, a good number for a global mesh and a bad one to spend twice on every turn because your worker pool lives in the wrong place.

Only after all five would I spend money. And when you do, spend it on TTS rather than the LLM: at 600 characters a minute the difference between a $15-per-million-character voice and a $50-per-million-character voice is about 2.1¢ a minute, which dwarfs anything the model line contributes. Full working in the TTS provider comparison.

How do you stop voice agent latency from regressing?

Alert on the p95 of each stage against its budget, and put a latency assertion in CI. Latency in a voice pipeline doesn't regress in one dramatic commit. It drifts, one dependency bump and one prompt change at a time, until somebody notices the demo feels off.

Per-stage budget alarms beat a single voice-to-voice alarm because they tell you where to look. "Voice-to-voice p95 exceeded 900ms" starts an investigation. "llm_ttft p95 exceeded 400ms" starts a fix. Set the thresholds from the budget table in this post, adjusted to whatever your own measurements say once you have a fortnight of data.

In CI, run a scripted turn against a staging agent N times and fail the build if the p95 exceeds your budget. It's a slow test and belongs on a nightly rather than every pull request, but it's the only mechanism I know that catches a provider's silent regional degradation before your callers do.

One caveat: your staging numbers will be better than production because staging has no concurrency. Rime publishes its TTFA figures at one concurrent stream and says so plainly. Assume every published latency number you've read, including the ones in this post, degrades under load, and size your budget with headroom rather than to the millimetre.

A closing honesty note, because latency is the argument most often used to justify building. Under about 20,000 minutes a month, a managed platform will get you under 800ms perfectly well and you should let it. The reasoning is in the voice AI build vs buy break-even, and latency is rarely the constraint that justifies owning a pipeline. If it genuinely is your product, and you want the budget instrumented and alerted from day one rather than reconstructed after a bad demo, that's what voice AI development is for. To see what each stage costs as well as what it takes, put your stack into the voice AI cost calculator.