Turn Detection and Barge-In Optimization

Start by separating latency from listening

Most failed voice-agent demos are described as “latency problems,” but the real issue is often that the agent does not know whose turn it is. If it answers while the user is still thinking, endpointing is too aggressive. If the user says “wait” and the agent keeps reading a long message, barge-in is broken. If the TV in the background cancels every response, VAD and echo handling are too permissive.

Use this when the agent already works

This workflow is for teams that already have a phone agent, web voice assistant, AI tutor, or companion prototype running. You do not need to rebuild the stack first; you need event traces, representative audio, and a repeatable way to replay failures. If your business logic is still unstable, fix that before chasing a 50ms VAD improvement.

Measure turns as events, not vibes

Log the exact time for user speech start, user speech stop, STT partial, STT final, LLM first token, TTS first audio, playback start, playback cancel, and tool-call start. A single timeline makes the problem obvious. The contrarian lesson is that faster settings can make the product feel worse if they produce more false starts and clipped words.

Know what VAD cannot decide

Voice activity detection only says that speech-like audio is present. It does not know whether the user is interrupting, backchanneling, coughing, or speaking to someone else in the room. Use VAD as the low-latency gate, then combine it with STT endpointing, model-based turn detection, or explicit business rules.

Tune the path with replayable samples

Build a small test corpus before tuning thresholds. Record 20-30 short interactions that include headset audio, speakerphone audio, car noise, background voices, long pauses, soft speech, and mid-response interruption. Label each failure as a false start, missed barge-in, dead air, cut-off word, or noise-triggered interruption.

Fix echo before semantic detection

If the agent’s own voice leaks back into the microphone, higher-level turn detection will be polluted. Browser clients should use WebRTC echo cancellation where possible; phone and embedded-device flows need their own loopback tests. Only after echo is under control should you tune VAD threshold, minimum speech duration, silence timeout, and endpoint delay.

Set a policy for interruption

Not every sound should cancel the agent. A clear “stop” or “wait” should cancel playback immediately. A soft “mm-hmm” during a long explanation may be a backchannel. A tool call that changes an order, books an appointment, or writes to a database may need to finish even if playback is cancelled. Separate cancellation of audio, reasoning, and irreversible actions.

Test provider defaults with your stack

Managed platforms such as Retell AI and Vapi can get you to a live phone test quickly, but interruption behavior depends on the selected STT, TTS, model, telephony route, and prompt. Self-hosted stacks with LiveKit, Pipecat, and Silero VAD give more control, but you inherit the replay system, metrics, and device testing.

Use two test modes, not one

Run scripted tests first, because they make regressions obvious: the same interruption phrase at the same timestamp should produce the same cancel behavior. Then run unscripted tests with people who naturally hesitate, backchannel, mumble, or speak over the agent. Scripted tests catch code regressions; unscripted tests catch product assumptions. You need both before trusting a voice interface.

Choose tools by the control surface

Choose LiveKit or Pipecat when you need to inspect and modify the media path, VAD, endpointing, and provider wiring. Choose OpenAI Realtime when a native speech-to-speech experience matters more than component-level replacement. Choose Retell AI or Vapi when the first milestone is a working phone agent with analytics and operational controls.

Treat cost as a minutes problem

Retell lists AI voice agents at $0.07-$0.31 per minute. Deepgram lists Flux and Nova STT from $0.0077 per minute, and Cartesia’s current plans bundle model credits with voice-agent minutes and concurrency. Those numbers are starting points, not a production budget. Run at least 100 representative calls and include failed retries, long silences, transfers, and human handoff.

Use the embedded build video as a test harness

The embedded Vapi voice-agent build is useful as a full-stack reference, but do not stop at “it answers.” After reproducing a basic agent, add a specific barge-in script: interrupt during the greeting, interrupt during a tool lookup, interrupt during a long confirmation, and speak from a noisy room. Those four tests catch most demo-to-production regressions.

Pick one primary metric per failure type

Use missed barge-in rate for interruptions, false interruption rate for noise, P95 dead air for endpointing, and clipped-word count for over-aggressive VAD. If you compress all of these into “latency,” the team will optimize the wrong setting. A product can have fast average response and still feel rude if it talks over users twice per call.

Launch with a failure matrix

The launch checklist should be concrete enough that another engineer can rerun it. Keep sample audio, event traces, current thresholds, provider settings, and the expected result for every test case. Track missed barge-in rate, false interruption rate, P95 dead air, clipped first words, and repeat-request rate.

Keep monitoring after launch

Turn detection drifts as devices, users, languages, and scripts change. A new TTS voice can be louder and trigger echo. A new caller segment can pause longer before answering. A new compliance script can make interruptions more common. Review failed calls weekly and promote fixed failures into regression tests.

Release checklist

Before launch, confirm echo cancellation on every supported client, push-to-talk fallback where echo cannot be controlled, interruption cancellation for TTS playback, protected execution for irreversible tool calls, and a replay set for noisy audio. Keep the current VAD and endpointing settings in version control. When someone changes a threshold later, you should be able to see exactly what changed and which calls got better or worse.