NV
Nate Vogel
19 hours ago
LiveKit barge-in looked fine until users talked over the TTS buffer
Hit this again last night shipping a LiveKit + ElevenLabs demo.
Goal was barge-in under 300ms. On paper streaming TTS looks fine. In practice the client was still draining ~800ms of audio after Whisper fired interrupt, so the agent kept talking over the user for a full sentence.
Fixed it by flushing the player buffer on interrupt and dropping in-flight TTS chunks. Without both, you get that awkward "assistant talks over you" vibe that kills the demo.
Anyone else measuring barge-in the hard way, or am I just bad at audio?
5 comments
Join the discussion
Log in to comment.
same pain. we were measuring player silence and calling it a win until a user talked mid-sentence and the agent finished the whole paragraph.
added a single timeline: mic open → silero VAD → Whisper partial → interrupt → audio stop. once those five stamps lived in one grafana panel the "300ms" story fell apart. ours was ~1.1s end-to-end on a good day.
dropping in-flight ElevenLabs chunks was the missing piece for us too. flush alone just made the next packet refill the buffer.
yeah this is why my "voice" prototype stayed text for three weeks. i had the same 700–900ms hangover with a different TTS stack — interrupt event fired, UI showed listening, audio kept playing.
flushing the client buffer alone wasn't enough either. had to cancel the server-side stream or it'd just refill the queue. cost me like $18 in ElevenLabs credits learning that.
are you measuring end-to-end (mic → interrupt → silence) or just the player side?
yeah the server cancel is the part demos skip. i flushed the player, celebrated, then watched the websocket dump another 3 chunks into the queue.
on mac m2 + local whisper it looked snappy in the lab. put a real user on airpods and suddenly you're fighting bluetooth buffer too. not fun.
Curious what you count as "barge-in under 300ms." Mic open → VAD → Whisper partial → interrupt → audio stop?
I've seen people publish the last hop only and call it done. If you're not logging those timestamps in one place the number is mostly vibes.
wait so are people actually shipping voice agents without that dual cancel?
i stuck a waveform screenshot in Linear every time barge-in failed during our Twilio + ElevenLabs week. half the tickets were "UI said listening, TTS still talking." same bug, different stack.
are you gating demos on that end-to-end number now, or still shipping on player-side silence?