Why Your Voice AI Agent Interrupts People (Turn Detection and Barge-In, Properly Explained)

Why does my AI voice agent interrupt me?

Because voice activity detection measures acoustic energy, not meaning. It can't tell "my account number is... uh... four seven..." from "that's all, thanks." Those two utterances can be acoustically identical at the pause and semantically opposite, so no value of any silence threshold distinguishes them. This is an architecture gap, not a parameter you haven't found yet.

You've probably discovered the trade already. Raise the silence threshold and the agent stops interrupting, but it feels sluggish and callers say "hello?" into dead air. Lower it and responsiveness returns along with the interruptions. You're moving along a single axis with a bad option at both ends, the signature of a missing component rather than a mistuned one.

The missing component is a model that reads the transcript or the audio and predicts whether the utterance is complete. LiveKit ships one; Deepgram fuses one into its Flux STT model. Either way you end up with two systems doing two different jobs: VAD detects speech presence and triggers interruption, and the end-of-turn model decides when to commit the turn.

There's a second, quieter failure underneath this. Even with perfect detection, if your TTS client can't cancel an in-flight synthesis, the agent keeps talking over the caller regardless of what your VAD concluded. Half the reader's problem lives in the TTS layer, which is why this post covers both and why choosing a TTS provider for voice agents is partly a barge-in decision.

“VAD measures energy, not meaning. That single sentence is the whole diagnosis, and it's why raising a threshold trades one bad experience for another instead of fixing anything.”— Neeraj Sharma, Axionry

What are the two failure modes, and why are they opposites?

False endpointing, where the agent commits your turn while you're still thinking, and missed barge-in, where the agent fails to stop when you start talking. They feel similar to a caller ("it talks over me") but have completely different causes and completely different fixes.

False endpointing is a commitment-path problem. The system decided you were finished and it was wrong. The fix is semantic: a model that understands the utterance is incomplete. Raising your silence threshold also fixes it, at the cost of making every correctly-detected turn slower, which is the trade you should refuse.

Missed barge-in is an interruption-path problem. The system knew you were speaking and failed to act, almost always because the in-flight TTS synthesis was never cancelled, or because audio already buffered downstream continued to play. No amount of end-of-turn model quality touches this.

The asymmetry that matters for tuning: interrupting a caller is much worse than waiting slightly too long. An interrupted caller feels dismissed; a caller who waits an extra 300ms feels like they're on a slightly slow phone line. So when you choose a point on the axis, bias toward patience, then claw the latency back from the free wins in the full voice AI latency budget.

What is semantic end-of-turn detection?

A small model that predicts whether the caller has finished, using meaning rather than silence. LiveKit ships two generations of this, and the newer one is a genuine architectural change most published writing hasn't caught up with yet.

The older approach, now marked deprecated in LiveKit's docs and slated for removal in version 2.0 of the Agents SDK, is a text model: an open-weights fine-tune of Qwen2.5-0.5B-Instruct, 396MB on disk, running on CPU in under 500MB of RAM with roughly 50–160ms of per-turn latency. It reads the transcript from your STT and predicts completion. It's still the option to reach for if you need fully open weights and can't use hosted inference.

The current approach is an audio turn detector that encodes the caller's audio directly, capturing not just what was said but how: intonation, pitch, rhythm. Because it doesn't depend on a transcript, it works with realtime speech-to-speech models without bolting on a separate STT. It ships in two versions: v1, the full model served on LiveKit Inference, and v1-mini, a lightweight version that runs locally on CPU free of charge in any context.

The audio model matters because prosody carries turn-completion information a transcript throws away. "From Dublin to New York." is a complete sentence on paper and an obviously unfinished thought when you hear the rising intonation. That's exactly the case the text model gets wrong, and LiveKit publishes the example.

# utterance: "Hi. I'd like to book a flight from Dublin to New York. This Friday."
transcript chunks arriving from STT:
  2.53s  "Hi. I'd like to book a flight"
  4.85s  "from Dublin to New York."
  6.14s  "This Friday."
TEXT end-of-turn model  -> commits at 2.76s, 4.88s, 6.40s
  two false endpoints. the agent interrupts twice.
AUDIO end-of-turn model -> commits at 6.41s
  one commit, at the true end of turn.

How do you configure turn detection in LiveKit?

Pass a TurnDetector into the session's turn-handling options and set two delays. The audio turn detector is built into the Agents SDK from livekit-agents 1.6.1 for Python and 1.4.7 for Node, so there's no separate plugin to install and no model-download step, a real simplification over the deprecated text detector, which needed both.

The two parameters that matter are min_delay and max_delay. min_delay is the minimum time to wait since the last detected speech before declaring the turn complete; max_delay is the ceiling before the agent commits regardless. Defaults are 0.5 and 3.0 seconds, and they drop to 0.3 and 2.5 seconds when the audio turn detector is active, because the model provides a confident signal so the session can commit sooner.

One constraint people trip over: the audio turn detector requires VAD, and the VAD's min_silence_duration must be at least 0.25 seconds or the session raises a ValueError at start. Silero's default of 0.55 seconds already satisfies it, and is also why your effective floor is 550ms until you lower it, since min_delay in VAD mode behaves like max(VAD silence, min_delay).

The third knob is unlikely_threshold, which sets how confident the model must be before considering the turn complete. Lower makes the detector eager, higher makes it patient. It accepts a scalar or a dict keyed by language code, the mechanism behind the use-case tuning in the next section. There's also a dynamic endpointing mode that adapts the delay within your min/max range using an exponential moving average of observed pause statistics.

from livekit.agents import (
    AgentSession, TurnHandlingOptions, EndpointingOptions, inference,
)
from livekit.plugins import silero

# --- Support call: people finish their sentences. Bias toward speed. ---
support = AgentSession(
    vad=silero.VAD.load(min_silence_duration=0.25),  # docs: must be >= 0.25
    turn_handling=TurnHandlingOptions(
        turn_detection=inference.TurnDetector(),      # audio model, v1 / v1-mini
        endpointing=EndpointingOptions(
            mode="dynamic",       # adapts within the min/max band via EMA
            min_delay=0.3,        # audio-detector default
            max_delay=2.5,        # audio-detector default
        ),
    ),
    # ... stt, llm, tts
)

# --- Technical interview: people pause to think. Bias toward patience. ---
interview = AgentSession(
    vad=silero.VAD.load(min_silence_duration=0.35),
    turn_handling=TurnHandlingOptions(
        turn_detection=inference.TurnDetector(
            unlikely_threshold=0.6,   # higher == more patient
        ),
        endpointing=EndpointingOptions(
            mode="fixed",
            min_delay=0.6,
            max_delay=4.0,   # a candidate thinking for 4s is normal, not idle
        ),
    ),
)

# What this does NOT handle: DTMF during a turn, multi-party rooms,
# or callers on a hands-free speaker where your own audio re-enters VAD.

Why doesn't barge-in work even when detection is right?

Because stopping the agent requires cancelling an in-flight TTS synthesis, and a surprising number of TTS integrations can't. You can have a flawless VAD and a state-of-the-art end-of-turn model and still ship an agent that talks over people, because detection and cancellation are different subsystems and only one is in the blog posts.

Three things have to happen on an interruption, in order and fast. Stop local playout. Cancel the in-flight synthesis request so the provider stops generating and you stop paying for characters nobody will hear. Truncate the conversation context. Skip the first and the caller hears buffered audio. Skip the second and you keep receiving audio for an utterance nobody wants.

The third is the one almost nobody writes about, and it produces the bug where the agent repeats itself after being interrupted. If you append the full generated response to the conversation history, the model believes it already said all of it. The caller heard five words; the model thinks it said forty. Next turn, the model refers back to information the caller never received, and the conversation quietly desynchronises.

Correct truncation requires knowing which words were actually played, which requires word-level timestamps from your TTS provider. Rime publishes this cleanly: Coda has word-level timestamps, Mist v3 doesn't. That makes timestamp support a barge-in correctness feature rather than a nice-to-have, and the reason I wouldn't pick a TTS purely on time-to-first-audio.

# --- WRONG: the "agent repeats itself" bug ---------------------------
async def on_interrupt_wrong(session, generated_text: str):
    await session.playout.stop()
    await session.tts.cancel()
    # BUG: records the whole generated response as if the caller heard it.
    session.chat_ctx.append(role="assistant", text=generated_text)
    # Next turn the model says "as I mentioned, your balance is..."
    # The caller never heard the balance. They heard five words.


# --- RIGHT: truncate to what was actually played ---------------------
async def on_interrupt_right(session, generated_text: str, word_marks):
    await session.playout.stop()
    played_ms = session.playout.elapsed_ms()      # audio actually delivered
    await session.tts.cancel()                    # stop synthesis + billing

    # word_marks: [(word, start_ms), ...] from the TTS provider.
    # Requires word-level timestamps -- Rime Coda has them, Mist v3 does not.
    spoken = [w for w, start_ms in word_marks if start_ms <= played_ms]
    spoken_text = " ".join(spoken)

    session.chat_ctx.append(role="assistant", text=spoken_text, interrupted=True)
    # The model now knows it was cut off mid-sentence, and what it got through.

How do you tune turn detection for your use case?

By matching your endpointing band to how long your callers actually pause. That varies enormously by context, which is why a threshold tuned on support-call data is wrong for an interview. This is the table nobody publishes, because it requires having run more than one kind of voice agent.

I built AccioMatrix, an AI assessment and interview platform, and interview audio is a genuinely adversarial case for turn detection. A candidate asked a system-design question pauses three or four seconds mid-answer and isn't finished; they're thinking, and interrupting them is worse than almost any other context because it destroys the impression that they're in a real interview. A support-call profile applied to that workload talks over every thoughtful candidate.

The inverse is also true. An appointment-booking agent tuned with interview patience feels broken, because callers give short, complete answers, "Tuesday", "the afternoon one", then sit through a long delay for a response that should have been instant.

This is also where my most-cited production result comes from. Integrating Retell in 48 hours and then working the turn-timing and false-positive problem drove false positives from 50% to 15%, a 70% reduction Retell published as an official customer case study. A meaningful share of that was turn timing, not model quality: an agent that responds too fast reads as interrupting, one that responds too slow gets talked over, and both get logged as detection failures.

How do you test turn detection before it embarrasses you?

Build a regression corpus of hard utterances and measure false-endpoint rate and missed-barge-in rate as two separate metrics. They move in opposite directions, so a single combined score hides the trade you're making and lets a change look neutral when it has shifted you along the axis.

Four categories belong in every corpus. Filled pauses: "my account number is, uh, four seven...". Trailing conjunctions: "I called yesterday and...". Complete short turns: "that's all, thanks". And list enumerations, which are the nastiest: "I need Tuesday, Wednesday... and Friday", where a natural pause sits between items and the utterance is unfinished.

Twelve to fifteen recorded examples across those four categories is enough to catch regressions, and you can produce it in an afternoon from real calls with the consent and PII handling you'd apply to any recording. LiveKit also open-sources an evaluation harness, eot-bench, which simulates the live turn-taking decisions a production agent makes, plus English and multilingual evaluation datasets on Hugging Face, worth reading before you write your own scoring logic.

Then wire the two rates into CI as separate assertions. If a dependency bump moves your false-endpoint rate from 8% to 14%, you want to learn that from a red build rather than from a customer describing your agent as rude. This is the same discipline as the latency regression test, and teams skip both for the same reason: neither fails until it matters.

One honest closing note. If you're prototyping, or running under about 20,000 minutes a month, a managed platform has already solved all of this for you and solved it well; I shipped on Retell precisely because it had. Owning turn detection is worth it when the tuning surface is the product, as it is for assessments, or when volume makes the margin matter, and the voice AI cost calculator will tell you which side of that line you're on in about a minute. If it says build, and you'd rather have it built and instrumented properly than debugged in production, that's what voice AI development is for.