Voice Agents That Handle Interrupts - Chintan Agrawal and Daniel Wirjo, AWS

Hey everyone, I'm Chintan, and I have my colleague Daniel with me. We're solutions architects on the AWS APJ startup team, so today we're going to talk about something we've been working on for a while, which is turn-taking in voice agents.

What I mean by that is: how does the system actually know that you're finished speaking and it's time for the agent to respond? These are all audio engineering problems; they're not LLM problems, because you can have the perfect model, perfect track, but the experience still might feel broken if the turn-taking is off.

So we're going to walk you through the concepts and three different approaches to solving this, and Daniel is going to, you know, make it more concrete by running all the three demo for you.

So to kick things off, we'll start with the 200 millisecond constraint, which is basically the physics of why voice is hard. Then we'll look into the pipeline architecture: what are the components, where turn-taking actually lives. We'll then look at three levels of solving this problem, going from simple silence detection all the way up to running your own turn detection model.

And then we'll also talk about the latency, the budget, and the production issues that we've seen.

So, but, before any of that, I just want to show you the problem statement more clearly, because you have the same user, you have the same sentence on both sides, but the user outcome is very different. Because on the left, when the user says, "I want to fly," and mid-way they try to correct themselves, the agent just does not notice, because it keeps going on for almost 2 seconds while the person is sitting there trying to get a word in.

On theright side, the same exact interaction is happening, but the agent is able to capture the interruption in under 200 milliseconds and immediately backs up so the user can now speak. Both the scenarios, the LLM was identical, it was the same model, it was the same prompt, but the difference in the user experience is so different, and the difference is purely because of the audio pipeline.

And how fast does it notice? Like, someone else is talking and it knows when to shut up. Like, that's the problem we're trying to solve today.

So we'll cover this 200 millisecond constraint, because 200 milliseconds is how fast humans switch turns with each other in a conversation, and the implications are pretty brutal, because at 800 milliseconds things start to feel off, while at 1.5 seconds your user would just hang up on you because you have, like, your agent.

Your chat agents might get 5 seconds to respond and nobody will care, but with voice agents you don't get that luxury. We've seen recently, like, Salesforce was able to build a pipeline and they published their results in March 26, but even their best measured response time was 755 milliseconds, so that's, like, almost 4x slower than how humans naturally take turns while talking.

So the question here becomes: how do we close this gap? And where we cannot close it, like, at least how do we get the turn-takingright so the user experience does not feel as bad as the raw numbers might suggest here.

So if you're working with a budget of 755 milliseconds and every millisecond counts, you also need to understand what's actually in the pipeline, like, where does the time go, and where does this turn-taking fit. So the pipeline itself, you probably might know the main components: the STT, the LLM, the TTS.

You have the audio in, text out, text in, audio out. But the piece that's often missing, like, from people while people are thinking about this is this voice activity detection, commonly called as VAD. It's a tiny component that sitsright at the front, and it is the one that controls turn-taking.

Its job is to basically detect whether the user has stopped talking or not. And the two other things that you said alongside the main pipeline are also very critical for turn-taking. So you have the smart turn detection, which watches the VAD signal plus the audio features, and then it has to make that decision that whether should it respond now or should it wait because the user might still be talking.

And the interruption handler is triggered when someone barges in, so what it'll do is it'll propagate a flush of downstream, like, the TTS would stop and the LLM generation need to be canceled so that the pipeline is ready for the new input within about, you know, 15 milliseconds.

So, if you remember, I was talking about, like, how there are different levels to, you know, building this turn-taking in voice agents. So we'll start with level 1. Level 1 is your Celero.v80, so this is a simple component which you fully own, and, like, a lot of production systems, even today, we see are running agents just by using this.

So coming to Celero.v80, it's a small 300k parameter model. It takes in, like, a short-term Fourier transform, it has a, and it takes raw audio and it'll convert it into spectral features. It has four convolutional layers, to pick up the pattern, and LSTM that gives it memory across the frame, so it's just not looking at one chunk in isolation.

And then it has a sigmoid that gives a probability of speech, and it's, like, a very small 2 GB model. And the one parameter that minimums silence milliseconds is basically the entire user experience of level 1. If you keep it very low, the agent is going to be snappy, like, it'll cut people off while they're still, you know, thinking.

But if you keep it very high, the agent might become super patient and it'll never interrupt, but you might get to, get, like, dead air where people will wonder if the agent is still connected or not. So there's, like, no universallyright answer, it depends on the domain.

If you're working on it from a perspective of a sales agent, you probably want it to be, like, 200 milliseconds. If you're in a domain where you need some to give some time to your user to respond, it could be, like, 1000 to 1200 milliseconds, it depends entirely on what you're building.

So VAD works well, like, that's why a lot of people are using it, and it's generally good at what it does, that is, to return fast response to the basic question of whether is someone talkingright now. But there are things, like, it was never designed to handle, because there'll be situations where just knowing that is there silence will not be enough, because you also need to answer these questions of how long to wait, like, the silence duration.

If someone is pausing for 300 milliseconds or for 200 milliseconds, the VAD is seeing the same exact thing. It has no idea, like, whether the person is catching up their breath or whether they've completely finished their thought. So, like, they cannot distinguish that.

And the second scenario is, like, what if the, you know, the user talks over the agent, like, it barges in. It does not know how to handle it. And the same silence can have different intent, and it can be, like, completely different things.

It could be, like, a completed sentence, it could be an incompleted thought, it could be, like, a thinking pause, or maybe it could be something as basic as, like, a backchannel acknowledgment. The situation of the silence is the same, but the intent, again, was very different in all the scenarios, but VAD considers all four equally, and it literally cannot tell them apart.

So those are the limitations of level 1, and with level 2, basically what you get is that you let the STT service tell you, like, when the turn is over. So Kartisheer Inc. does turn detectionright inside their STT websocket, so when they stream your audio in, the server would handle both the transcription and the turn detection together, and it will emit events when it thinks, like, the turn is done.

Similarly, Deepgram does the same thing with their, what they call as their, I think, Eddie endpointing, and their p50 latency for Kartisheer Inc. is about 300, and it's about 250 for Deepgram Nova 3. Both of these are also working really well, like, we see a lot of customers use that, because they are using, like, full audio signals plus some linguistic context to make these decisions, which is, like, much more information than your VAD will ever have.

However, the trade-off is with the transparency, because when it is working, it works great, but when it misfires or when it cuts someone off at the wrong moment, you, like, have no way to figure out, because there's no log that will say it was done because of this reason, because of the, what was, what it saw.

And that decision was made inside someone else's server, and you just got to live with that, basically.

So that brings us to level 3. So in level 3, you keep Celero.v80 running locally for the basic signal that is there as speech, and you add SmartTurn on top of it, which is a small model that runs during silence.

So SmartTurn v3.2 is the latest model, like, and let me share some numbers with you, and it has about 58.9% recall and 68.4% precision. So what this means is that about 6 out of 10 times when someone finishes a sentence, SmartTurn will be able to catch it quickly, but the other 4 times it might not be confident enough, but that's also fine because you still have your VAD timer which is running underneath as a safety net.

If the SmartTurn doesn't fire, your stop second will kick in, like, at the configured latency anyway, so it could be, like, 300 milliseconds. So in that scenario, you are never stuck waiting, you just, like, get the fast response when the model is confident, and you get a slightly slower but a safer response when it's not.

The other number on this list is from, like, a Meta's published paper earlier this year, around March, where they had reported a higher recall at 87.7%, but they haven't released the code, so you really cannot deploy it.

SmartTurn is BSG2 licensed, and it's an 8 MB small model, you can install it today as well. So yeah, like, that works out better for a lot of our customers deciding when to start talking. But, like, what about the other direction?

What happens when the user interrupts and while you are already, you know, the agent is talking? So on the left side, you can see what happens mechanically, like, user opens there, you mouth, the VAD will pick it up in 32 milliseconds, and then, like, 15 milliseconds the entire pipeline is flushed.

TTS stops, LLM cancels, everything is clean. Like, the user never hears the agent, you know, that part is taken care of by Pipecat. But the important thing is, like, should it have stopped there on? Because think about a normal conversation when someone says, yeah, like, they send an acknowledgment while you're speaking, you don't stop,right?

You know, they're just agreeing, but if you, they say, like, okay, wait, no, like, that was wrong, you have to stop immediately. Same thing here, like, the red is the, it's a correction stop on everything. The amber could be just some sort of a filler background noise.

You want to go ahead and keep going, maybe, in a bit quieter. Blue is, like, a signal for a background noise, so you can just ignore it and finish your sentence. So at level 3, because you own this classification, you can start distinguishing between, like, real interruption and someone just, you know, going, hmm.

So today, most of these systems stop every time, but this is a piece that's improving, like, we are seeing a lot of incremental improvement in this.

So by now, that you've seen all three levels, and if you look at these three Python files, the pipeline is almost identical in all of them. The only thing that differs is how you answer this question: when is the user done talking?

In file 1, you pass in Celero.v80 analyzer, and that says the user had been silent for 300 milliseconds, so you can assume they're done. In file 2, you swap it with the KartisheerTurnsSTTService class, and now the server is going to tell you when the turn is over.

And in file 3, you keep your Celero.v80 analyzer, but you also add, like, a local SmartTurn analyzer, version 3, which is sort of a small model that runs during your silence, and it's going to look at prosody,

and it's going to look at prosody and intonation to decide whether that pause meant I'm done or I'm still thinking. So you have the code-wise the same pipeline, but the configuration changes, which, you know, leads to completely different behavior.

So to summarize your three levels, your level 1 was Celero.v80, so you basically own the silence detection completely. Level 2, you let the STT provider handle it for you. In most cases, it'll speak smarter, but then you cannot see what happened inside it.

Level 3 is your VAD plus SmartTurn, so where you basically own everything, like, you have full portability.

And now coming to the latency piece, because we've been throwing around these numbers, 755 milliseconds, 1.3 seconds, but let me show you where the time actually goes in. So this is a breakdown from Quintla, who is a co-founder of Data4Day, and is a person who created Pipecat, and these are some of the numbers based on their production measurement.

So you have the numbers for machine encoding, which is about 40 milliseconds, then you have network and jitter buffer of 52. So these are physics, like, you can't really change them much. Then you have your transcription plus endpointing, that's about 300 milliseconds.

Again, that's your STT doing its thing.

LLM time to first byte, and this is generally the dominant bottleneck, because in a typical API setup, you're looking at 500 to 650 milliseconds, depending on what model you're calling and from which cloud provider, which region. And then you have your network out and playback, TTS, that's, it's going to take about 120 to 190, 85 milliseconds.

So you're totally looking at roughly 1100 to 1300 milliseconds in a standard setup calling cloud APIs. Now, the Quintla's team had demonstrated about 500 milliseconds total voice-to-voice by co-locating all models in the same GPU cluster, because they were able to eliminate the network hops between the service.

It's sort of an achievable flow, because if you're willing to invest in infrastructure, but for most of the developers today calling APIs, you are somewhere in the range of 800 to 1300 milliseconds. And the key insight to note here is that the STT and LLM together eat about two-thirds of the latency budget.

So these are really the only two levers that we have to move this latency metric ladle in a meaningful manner.

So coming to, like, which LLMs are actually fast enough for this, we benchmarked current models specifically for voice, and the target, we kept it as under 700 milliseconds time to first open, because anything slower will push our total response past where the user is going to start noticing.

So for now, like, recording in June 2026, what we've observed, Nemotron 3 Ultra gives us a 529 p50 latency, GPT-4.1 at about 536, but also, like, what matters more in voice, then p95.

So which LLMs are actually fast enough for this? We benchmarked current models, especially for voice, and

so coming to the, so coming to the LLMs models, which are actually fast enough for this, we benchmarked current models for voice, so these benchmarks are from June 26, and the target that we kept was under 700 milliseconds for time to first open, because anything slower would push our total response time past the limit, like, where users are going to start noticing.

So for now, the Nemotron 3 Ultra at 529 p50, we observed GPT-4.1 at 536, again p50. But what matters more in voice than maybe anywhere else is that the p95 tail, because GPT-4.1 was great at p50, but it spikes to 1.7 at p95.

And it was even worse for Claude Sonnet that it hit over 4 seconds, and because in a conversation you cannot average this out, because one slow response and your entire flow is gone. And there's one more dimension that people often miss, which is the multi-turn drift, because after 15 or 20 turns, sometimes models start ignoring parts of the system prompt, they might get too verbose, they go off-spirit.

Because in voice, that's vital, you can't just dump a wall of text on someone. If the instruction falling degrades across turns, then you have to do some sort of a context pruning or session resets.

So a few production issues that we've seen that cause interruptions significantly increase your escalation rate, because when agents cut people off incorrectly, then users are more likely to request for a human-in-the-loop support.

And 755 milliseconds, that's still, I think, the best measured voice-to-voice for a cascaded pipeline, like the ones which we propose. We are nowhere near the human speed of interaction, which means that the turn-taking also needs to do a lot of heavy lifting to make the experience feel better than the raw numbers it would suggest.

And at 58.9% recall on SmartTurn, that's actually the best deployable turn detection you can get today, because although it still leaves, like, 4 out of 10 turns to be filled by a VAD timer, but this space is going to improve a lot over the next few years, and we'll have better models.

And on infrastructure, running all of this in production is also hard, because five systems that all scale differently and can also fail differently. You can explore Pipecat cloud for managed hosting, where your focus is on agent logic, or the AWS Guidance Reference Architecture for enterprise deployments with all the enterprise guardrails.

So yeah, like,

that's it from me. Everything we've discussed, I've mentioned is pinned on the screen, the repo benchmark. Okay, over to you, Daniel.

Thanks, Chintan. Hi, everyone. My name is Daniel Wirjo. I'm also a Solutions Architect with the AWS startups team. So I think, Chintan, walk you through a great overview of some of the key challenges when building voice agents. So what I might do in this session is just to walk you through a hands-on demo, just to make the concepts a bit more concrete, and hopefully you can kind of incorporate some of the learnings into your own voice agent pipelines as well.

So what we'll be going through is just this Pipecat turn detection demo, which is just a simple repository that we've created. We'll talk through three examples, so one with silence detection, another with built-in turn detection in the speech-to-text model, and then finally we'll go through the open SmartTurn model as well.

So this is really useful when you either want full control of your voice pipeline, just from a compliance perspective, or if you want to also fine-tune and customize your own turn detection model based on your own data. So yeah, let's get started.

So let's just go through the source code of the first example. As you can see here, using Celero. is pretty easy in Pipecat. All you have to do is import the relevant libraries, and you can configure the parameters as well.

I haven't actually configured any of the parameters, so these are just the default, but you can change the parameters, like the confidence to trigger, the silence duration before the turn ends, and the minimum audio volume as well. And in terms of the voice pipeline, it's pretty standard.

So we've used an STT model, the LLM, and also a TTS model here. We've just defined it as the Kartisheer model for STT, same with the Sonic model for TTS. And as you can see, we've used the voice that sounds like a British lady.

And for the LLM and the voice agent's brain, we've just used the Anthropic Claude Haiku model. So we'll see what that looks like, and I'll spin up this exampleright now.

So if you haven't used Pipecat before, one of the nice things is that it can spin up a local prototyping environment, really useful for playing around with your voice agent pipeline. So let me just see if this now works.

Okay, so as you can see, it will come up with a local development environment. So what I might do is just connect to the voice agent, but before I do that, I might just mute just in case

I'm interrupting the voice agent. It should come up with, like, an opening greeting, and then I'll test it with some

interactions to see how the turn detection is working.

Hey there, I'm your travel assistant, and I'm excited to help you plan an amazing trip. So where are you thinking of going? Anywhere in the world you've got your eye on?

As you can see, the assistant's gone ahead to do the opening greeting, and on theright here in the terminal, I've just got some real-time logs just so we can see what's happening. So what I might do now is just, I'm going to pretend that I'm thinking about my travel destination and see how the voice agent behaves.

So I'm thinking of going to.

I'm all ears. Where are you thinking of heading?

As you can see, I haven't really finished my sentence yet, but the voice agent is already responding. So you can see how this works in the debug logs, and as you can see, there's an end-of-turn complete here. And the reason for that is due to the stop seconds parameter, which is set to 300 milliseconds, it looks like here, so that's why it's gone ahead and ended the turn.

What I might do is, I might just exit this local server and try the second example. So let's just clear that, and I'll refresh that. Okay, so I'm going to try the second example here.

And in this example, we'll use a speech-to-text model which has turn detection, so we'll see how the voice agent behaves. Okay,

I'm going to try the same example, so let me just mute myself here.

Hey there, I'm your travel assistant, and I'm excited to help you plan an amazing trip. So where are you thinking of heading? Are you dreaming of a beach getaway, a city adventure, mountains, or maybe somewhere completely different?

Allright, let's see how the model behaves this time. Hi, I'm thinking of going to.

As you can see, the model is pretty smart. It now can detect the turn, so let's see if we can find out what's going on in the debug logs. So let's see if it emits some of those turn-taking events here.

So as you can see, the Kartisheer Inc. model does emit some turn events here.

And as you can see, it hasn't really completed the turn, it hasn't sort of

responded back with any response. But maybe I can unmute myself and then resume and get to the destination. So, Sydney?

Oh, Sydney is fantastic. You're going to have such a great time there. Are you thinking about when you'd like to go, and how long are you planning to stay? That'll help me figure out the best flights and hotels for you.

So as you can see, once the model detects that it's a full complete sentence, it's able to then respond. Now, in practice, you would also incorporate some silence detection there, because as you can see, I was quite silent, I was muted for a while, and you'd probably want the voice agent to respond in practice.

Having, like, voice activity detection or silence detection is also useful for interrupting the voice models, or what you call barge-in as well. So definitely this is just a quick demo, but in practice, you would probably combine both the turn detection model and the silence detection.

Just for the final demo, I might quickly show the SmartTurn model as well and see what that looks like. It will behave very similar to this, it's just that the model will be running on either your local machine or on a GPU environment as well, if you want to deploy that into the cloud.

So let me just maybe clear this, just to avoid confusion.

And then I'll refresh this as well, and I'll start now the SmartTurn example. I'll refresh this to see if we can... Okay, there you go, it's starting it, and I'm just going to do the same example. So I'll mute myself again.

Hey there, I'm your travel assistant, and I'm excited to help you plan an amazing trip. So where are you thinking about going? Are you dreaming of a beach getaway, exploring a new city, or maybe something else entirely?

I'm thinking of going to.

I'm all ears. Where are you thinking of going?

Now, let's take a look at the both the silence detection, which is incorporated into this example, and also the turn detection model. Hopefully there are some debug logs that we can see. As you can see, it's not detecting that I've completed the turn.

The end-of-turn state is incomplete, but it did complete it due to the silence that

I've incurred there. So there was some silence, and therefore it sort of kept going with the next turn. What I might do is see if I can do a complete sentence and see if it can be detected through the local SmartTurn model.

I'm actually thinking of going to Sydney.

Oh, Sydney's fantastic. I'd love to help you plan that trip. So tell me, what time of year are you thinking about going, and how long are you planning to stay? Also, are you traveling solo, with a partner, or with family?

Okay, fantastic. So the voice agent was able to respond as soon as I completed the sentence, but let's see if that was actually intentional, or was that something that can be detected through the SmartTurn model. So let's have a look here to see if we can find the SmartTurn debug logs.

As you can see, this line here, sorry about the legibility of the highlight, but effectively this here says that the end-of-turn state is complete, and the SmartTurn model is actually classifying that there is a high probability there that the sentence is complete, and therefore it's okay to go to the next turn.

So that's, in a nutshell, how turn detection works. Hope that was helpful.