Why ChatGPT Keeps Interrupting You — Dr. Tom Shapland, LiveKit

I'm going to jumpright into it. I'm going to be talking about Voice AI's interruption problem, then after that I'm going to talk about how we currently handle interruptions and turn-taking in Voice AI, what we can learn from the study of human conversation on how to handle—how humans handle turn-taking, and then about some of the really neat and interesting new approaches out there for handling turn-taking and interruptions in Voice AI agents.

Um, Voice—excuse me, interruptions are the biggest problem in Voice AI agentsright now. When you're talking to ChatGPT Advanced Voice Mode and it interrupts you, it's annoying, but when you're talking—when a patient is talking to a Voice AI dental assistant and it interrupts the patient, the patient hangs up and the dentist stops paying the Voice AI developer.

This is our collective problem. This is all of our problem that we all have to solve. Um, and the problem is, like, turn-taking is just hard. Let me first, like, define what turn-taking is. Turn-taking is this unspoken system we have for who controls the floor on, um, between speakers during a conversation.

And fundamentally, it's hard because turn-taking happens really fast in human conversation, and there's no one size that fits all. So this is this really cool study that I pulled this data from where they're looking at how long it took a listener to start responding after the speaker finished speaking, um, across different cultures.

And we can see that the Danes take a relatively long amount of time to start speaking after the other speaker finishes speaking, but the Japanese do it almost instantaneously. So there's differences across cultures. That's part of what makes it hard.

There's also differences across individuals. I'm one of those people that takes a long time to respond. Even before I got into Voice AI, people would sometimes comment, being like, "Are you going to respond?" I'm like, "Yeah, yeah, thinking about it."

Um, and also, like, even though I'm one individual, there's a lot of variability in how quickly I respond. Like, if you make me angry, I'm probably going to respond kind of quicker. Um, so it's just a hard problem.

And in the next slide I want to talk about for people who are not very familiar with how Voice AI agent pipelines work, I'm going to provide, like, a simplified overview of how we handle turn-taking and interruptions in Voice AI agents currently.

So the user starts speaking, that's the speech input, and that audio—those audio chunks—are passed to a speech-to-text model. Um, and that speech-to-text model transcribes the audio into a transcription. The next step is something called a VAD that determines whether or not the user has finished speaking.

Um, I'm going to go more into that in a moment. The next step, if the user has finished speaking, the transcript is passed to an LLM, and the LLM outputs its chat completion. That chat completion is streamed out, and that stream is passed to a text-to-speech model where it's converted into audio.

And that audio, which is the audio of now of the Voice AI agent, is passed back to the user. Um, let's dig more into the voice activity detection system. It's a system with primarily two parts. So it's a machine learning model, a neural network that is detecting whether or not somebody is speaking.

So it's like speech or not speech. It's pretty simple. Um, machine—it's not—I shouldn't call it simple, but it's a really neat model, but it's ultimately just looking at speech or not speech. And then the next thing—next part of it—is a silence algorithm.

And the silence algorithm is saying, "Okay, if the person hasn't spoken for more than half a second, they're done speaking and it's time for the agent to start speaking." So that's the—in most production Voice AI systems, we're using something like that sort of VAD.

Um, and that's changing. We're building all sorts of new, interesting things that I'll cover later in the presentation. Um, the next part of my presentation, I want to dig into what we can learn from linguistics and academic research about how human—how turn-taking works in human conversations.

And one of the lines I read in a paper I read was that I really liked was that turn-taking in human conversation is a psycholinguistic puzzle, and that we respond in 200 milliseconds, but the process of, like, finding the words and generating speech and articulating speech takes 600 milliseconds.

So how can we possibly be speaking so quickly? How can the listener start returning an answer so quickly when it takes much longer to actually generate the speech? And the answer is that there has to be prediction going on.

The listener is predicting when the end of turn is going to occur, and then they start to generate speech before that end of turn. And what are the primary inputs in creating that prediction? The primary inputs on creating that prediction are the semantic, and this one's the most important one.

Like, what the content is of what the person is saying. Other inputs into this, you know, prediction algorithm in our head of when we're trying to predict when the speaker is going to finish speaking is the syntax, the structure of the sentence, the prosody, like the expressiveness, the tone, and then also visual cues.

Um, like most things in how the human mind works, we actually don't really know. Like, you know, it's complicated. Um, and but I would say one of the generally accepted models is what I'm going to walk through now of how turn-taking works in the human minds.

It's broken up into three stages. The first stage is semantic prediction. So what the listener is doing—and you'll notice this as you speak to other people at the conference, if you, like, are paying attention to your thought process—is you're constantly inferring the intended message of the person that's speaking to you.

So before they finish speaking, you're kind of figuring out, "Wait, what are they trying to say?" And then you're using what your prediction of what they're trying to say to, um, to then use that information to predict when the end of utterance will occur.

And you're not doing this just once. You're doing this multiple times, again and again,right? You're constantly updating this prediction as the speaker keeps going. So that's the first stage. Um, then the next stage, once it seems like you have a general idea, your prediction is coming true of, like, when you think the end of utterance will occur.

And as you start getting closer to that, you start refining that endpoint prediction, um, based on both the semantics and the syntax. And then as you start getting really—as the speaker starts getting really close to the end of turn, the listener finalizes the prediction by using prosody, by using information around, like, the tone and other acoustic features.

Um, so it's three steps: a semantic prediction, a refinement, and a finalization. And one of the things I want to point out is that the human mind is full duplex. We're both processing input and we're starting to generate output at the same time.

And I think that's really nicely described in this figure from this paper. Um, and so on the x-axis, what we have here is time, where the 0 millisecond is the end of the speaker's turn. And what these different blocks represent is what of mental processes that are happening inside the mind of the listener.

And you can see well before the end of the turn, there's this whole comprehension track that's going on where the listener is inferring the intended message of the speaker and making predictions about when they're going to start finish speaking.

And at the same time, there's also this production or generation track where the listener is starting to produce what they're going to say.

And we're going to talk more about full duplex models in computer in silico rather than human minds in a little bit in my presentation. Um, oh, Jordan Deersley just texted me.

He texted "Boo." Like, how does he know to boo me? He's not even here. Um,

okay, it's all good. We'll keep rolling. Uh, so, um, so let's go back to, uh, um, like, let's contrast this really interesting complex process that's going on in the human mind compared to current Voice AI systems. You'll see it's just so much more simple,right?

It's just speech or not speech. It's looking backwards. It's not making a prediction. Um, it's done in serial. Nothing's happening in parallel. Um, so it's much more simple. And that's part of the problem,right, of why these interruptions are happening.

Um, so there's, uh, I'm going to talk through three types of models and the approaches people are using in these three types of models. Um, so the prevailing model for building Voice AI agents is the cascading model system of models where you have, um, what we talked about earlier: speech-to-text, VAD, LLM, TTS.

And, uh, what we're doing in those is we're augmenting these new—the new approaches to better handling interruptions is we're augmenting the VAD with models that look at the semantics, syntax, or prosody. Um, and, uh, I want to jump into an example of it.

Um, I really have too much content for, uh, for my allotted time that I'm looking at down here, um, but maybe I can just take some time from Jordan. Um, so, uh, let me—let me, um, give an example for one of these semantic-type models that is used to augment VAD.

I'm going to talk about our model at LiveKit. It's a text-based semantic model. So what we're doing is we're taking the last four turns of the conversation as input. So that means that it's the, uh, the Voice AI agent's turn, then the user's turn, then the Voice AI agent's turn, and then the user's current turn.

Those are the inputs into a transformer model. And what we're—the token that we're predicting, um, because this is an LLM, we're predicting the end-of-utterance token. And if that end-of-utterance token, based on the content,right, based on the context and the semantics of that input, um, if the end-of-utterance token is saying that the end of turn hasn't happened yet, then we don't—we extend the silence algorithm part of the VAD and say, "Don't trigger the end of turn.

Wait longer." Um, so they work in concert. But that's generally the idea of how it works. I'm going to walk through a quick demo of how this works in action.

I needed to build a demo, but I wasn't.

Oh. So in this first part of this demo, what Shane is going to do is, uh, talk to a voice agent that is just using the traditional VAD that I was discussing earlier, that just looks at speech or not speech.

And in the second half of the demo, he's using our semantic end-of-utterance model.

100% sure what it was that I should.

Please continue what happened.

I had to build a demo for a LiveKit

turn detection, a turn detection demo where one of the agents, like.

Got it. What challenges did you face while building that demo?

It just kept on interrupting me. Being interrupted by the agent constantly. That was the worst part, for sure.

I understand. How did you overcome that challenge during the demo?

Well, hopefully we're going to overcome the challenge by using the new turn detection model that LiveKit's offering. So let's try that one now instead.

Hey, can you interview me about a time when I had to build a demo?

Absolutely. Can you share your experience with building a demo?

Yeah, definitely. So

I needed to build a demo, but I wasn't—I wasn't 100% sure what it was that I should build. And then I—I was thinking, like, probably the best way to show that would be, uh, would be side by side.

Yeah, thank you. I'll let Shane know that you all applauded his demo. He'll appreciate that. Yeah, you can really see it's a night-and-day difference when you augment the VAD with models that look at the semantics and syntax and prosody.

Um, which is a good segue to my next slide. So there's another type of—another approach people are taking to augmenting the VAD. And what they're doing is not just taking the semantic input, the text-based input, but they're also looking at the audio signal as well and trying to infer things from the acoustic features of the dialogue.

So they're taking the basic idea is the input is audio tokens and the output is the probability that the user has finished speaking. Um, Quinn in the daily team have built their also open-weight smart turn model that, uh, is this neat combination of a model that is both transformer and looking at acoustic characteristics.

Um, and then one of the new things that has just emerged has been, um, Assembly AI dropped their speech-to-text, new streaming speech-to-text service earlier this week. And their model is really neat in that it emits—it takes audio in and it emits out both a transcript and a likelihood that the speaker has finished speaking.

Um, so it's one model that's kind of doing both these two things at the same time. Um, and it's also looking at the acoustic features and the semantic features. One of the things I want—QTI also has one that they recently released that's pretty neat.

Um, one of the things I want to note about this, though, is if you're using your speech-to-text built-in end-of-utterance model, it's only seeing half the context. It's only seeing what the user is saying. It's not also seeing what the agent is saying.

So it doesn't quite have the full picture on the context. Um, but it works remarkably well. These—all these approaches work remarkably well and are a major step forward from these more traditional VADs. And, like, definitely something that if you're building a Voice AI agent after this conference, you should go, like, implement it.

They're pretty easy to implement on the different platforms too. Um, okay, so we often talk about in, uh, speech models or in Voice AI that, um, speech models are, uh, are going to save us. Um, speech-to-speech models, audio in and audio out, are going to save us.

Um, but actually, if you look at how these models work, the, like, OpenAI's real-time API, they're still using a VAD on the internals. Um, so they're still just looking at speech or not speech. Or you can opt to turn on their semantic—they call it semantic VAD, which is kind of a, uh, paradox.

It's not the best term, but turning on a semantic model that augments the VAD. Um, so to answer the title of my talk of why ChatGPT Advanced Voice Mode keeps interrupting you, it's because it thinks you're done speaking based on how long it's been since you last said a word, um, or based on what you've, uh, what you've said previously.

Um, and it's just not quite cutting it, um, when those interruptions happen. Um, uh, and it's a problem that is not totally solved. I want to also bring that up too, that, like, this is an ongoing problem, um, with all the different approaches.

Nothing has perfected it yet. Um, our end of LiveKit doesn't—although we power the transport, the audio layer transport for, uh, Advanced Voice Mode, um, OpenAI is not using our end-of-utterance model. So the next topic I want to cover is the, um, I'm running out of time here, but, uh, is full duplex models.

These are really neat. So a full duplex model is more like a human mind in that it's processing input and generating speech at the same time. Um, and as far as I know, there's not really any commercial applications of these.

Um, but they're—they're fundamentally, they're intuitive talkers. They're trained on the raw audio data. And the analogy I like to use is that it's like computer vision. In the early days of computer vision, we were handwriting algorithms to try to recognize a stop sign based on the color and the number of sides on it, et cetera.

Um, and it just didn't work very well. But when we started giving the raw image data to the neural network and let the neural network figure it out, all of a sudden it just started working. And I think it's a—actually, that, uh, what we learned from computer vision, that really helped us emerge from the AI winter.

That was a major kind of, uh, seeding process for where we are now with AI. Um, and it's a similar analogy with full duplex models in that we're handing them the raw audio data and we're just letting them figure out how turn-taking works rather than trying to handwrite all the rules.

Um, but the downside of these models is they're really optimized for, like, being really good at turn-taking, and they're kind of dumb LLMs. They're small models. They're not trained on a lot of data. They can't do instruction following very well.

Um, and just to give you a sense of, like, more specifics of how these models work, let's talk about the Moshi model. Um, what really made it more concrete for me of how this model works is this idea that it is always listening to input and it's always generating output.

And even when it's not its turn to speak, it's emitting natural silence. So it's just basically emitting silence that you can't hear, but it's still always emitting silence. Um, so it's always kind of doing both, just like a human is.

Um, SyncLLM, which is Meta AI's, uh, full duplex, like, experimental mode that you can access inside the app, um, is a similar full duplex model. Or it's also a full duplex model. Something neat that I want to bring up about SyncLLM is they're actually, in the internals of that model, they're forecasting what the user is saying about five tokens ahead or 200 milliseconds ahead, which is more closely like what humans are doing, except we're, uh, forecasting on a much longer timeframe.

And then lastly, my predictions for the future of how we'll solve this problem, uh, is I think full duplex models are neat, but I don't think they're going to solve the problem. Like, I think we just, for real production commercial use cases of Voice AI, we need more control.

Um, and we need more control over how it says things like brand names. Um, and instead, what I think is going to happen is we're going to get smarter and smarter VAD augmentations and faster and faster models in the cascade pipeline.

And we're just going to have more budget to work with to do a good job with this sort of thing. Um, and the reason I think that's true is, like, computers don't do math the same way humans do.

They don't have the same conceptual way of thinking about it. And, uh, LLMs think differently than us. And similarly, I wouldn't expect Voice AI to use the same mechanisms as the human mind to generate speech and to talk.

Um, thank you all for your attention. This was fun. I really appreciate it.

We do have some time, so I don't know if you want to take Q&A.

I would love to.

We could do that. Um, and I could start with the first question. So the demo you showed, there wasn't any response at the end,right?

I cut off the demo. It's actually a two-minute demo.

Right.

And I only have 18 minutes to speak, so I truncated it on both sides.

Fair. Because I was like, okay, maybe you just turned everything off and it was an impressive demo. No interruptions.

No speaking. Yeah.

Do you have the end of the demo? Do you want to show it or?

Um.

No worries if not.

It's more of the same idea. Like, what you could see is that Shane was, like, you know, taking his time talking and really pausing and thinking. It wasn't interrupting him. And then when it eventually would find his end of turn based on the context.

Cool. Can we find it on your Twitter or?

Yeah, it's on our LiveKit Twitter.

Awesome. Yeah. So we can look that up on the LiveKit Twitter. Awesome. Yeah, we can take some questions. I saw you had one.

Hi. How important are visual cues for turn detection in the human context? And are there any, um, is there any development to kind of replicate that in the, uh, Voice AI context as well?

Yeah, it's a really neat question of, like, how important are visual cues and are people working on integrating that into the turn-taking, um, intelligence for, uh, avatars and real-time experiences? So visual cues are actually, despite the fact that we are visual animals, um, very, very much so, like, visual is the most visceral, like, you know, input for us.

Visual cues are actually pretty low down the stack of, like, uh, predictors for when it will be end of turn. It really is semantics. That's, like, one of the main messages that I want to convey to people from this talk is it's the content of what people are saying is the main thing we're using to predict when they're going to finish speaking.

Um, and then these visual cues are, are, and these other ones are ancillary to it. Um, and I'm sure somebody is working on building something really cool where it's, like, multimodal and looking at visual cues to look at the, to infer the end of turn.

Um, I just haven't seen it yet and can't keep up with all the AI stuff on the internet. Yes.

What is the average cost for.

What is the average cost for usually a voice-generated call? And then how is, what is the effect when you try to keep regenerating the response?

So the question is, what's the average cost for a Voice AI call? Um, and what is the cost when you keep trying to regenerate the response? Um, so I would, I would first say that the, I think your reference to what is the cost to keep trying to regenerate the response is, um, what I, the way I want to answer that is actually the thing that's most expensive in the pipeline tends to be the text-to-speech.

So there's all these optimizations you can do in the cascade. And if you end up hitting the LLM multiple times within a turn, it's not, it's not all that costly and those sorts of things. Um, there's some really neat calculators online.

Because I'm personally not a Voice AI agent builder and those unit economics don't, uh, don't directly affect me. I don't have the numbers off the top of my head, but there's some really nice calculators. It's going to depend on how long the conversation is and that sort of thing.

Yeah, thanks for the demo. That was great. Uh, the question is about your new model that you just showed and that blew us away. So, uh, one is like, why is ChatGPT not using that model to improve their stuff?

And two, is it available for us to use now if I were to build a, uh, voice bot on LiveKit? And, uh, three, maybe during your development of that, one demo is great. Uh, do you also do some kind of benchmarking with users to see if, you know, this is like 50% better or something like that?

Yeah, so the first question is about why isn't OpenAI using our end-of-utterance model? I don't know why they're not. I think that's maybe above my pay grade of this company I just joined four weeks ago. Um, and the second question is like, can you, is our end-of-utterance model available, um, for use?

And so it's really easy on our website to, uh, or it's really easy to follow our quick start on our website and build a Voice AI agent that you can talk to. Um, and it's just one more line in our, in the pipeline that you, you build.

You just turn on, you have one more line in there and you get to use our end-of-utterance model. It's open weight. It's, you don't have to pay for it. It's just baked in. Um, and our docs show you pretty, I think by default it's in there.

Um, and the third question, uh, remind me of the third question. I'm sorry.

Did you do a benchmark?

Ah, yes, benchmarking. Um, so we have benchmarks where we have our test data set and, you know, the numbers of course look great. Um, but I think, uh, I was on a long call with our machine learning team this morning where we spent a lot of time just talking about, like, how do we get a good data set for benchmarking?

And it's just, it's just really, it's a tough problem. Um, and I feel like the industry as a whole doesn't have a good benchmark around turn-taking. Um, and that it's something that I'm sure will eventually emerge.

Uh, okay. We'll do one last question, I think. So there was, yes, you in the back. Not great to be sitting in the back if you want to ask questions, but.

Thank you, Tom, for the demo and the presentation. I got a question related to the back channel. So how did you tackle the back channel challenge in the turn-taking detection problem? So first, uh, for a natural conversational AI, the back channel is the one that cannot be, uh, ignored.

And sometimes it causes trouble for the voice agent to detect whether it is the, it is the endpointing. And second, for a typical back channel like yeah, mm, uh, yes, uh, those words can occur like a back channel or it can be started as the, uh, the agent who should be responding in the, in the following period.

So how the back channel is handled should be kind of important in this field.

So, um, my question was, or the question was about not when the, not the case where the AI is interrupting the human, but the case where the human is accidentally interrupting the AI. Um, and I didn't really cover that in my talk.

I was mostly focusing on the AI interrupting the human. Um, we don't have, our approach is simple. Like we're just using like the Solero or the normal VAD approach of like if the person is speaking for more than X milliseconds, assume it's not a back channel and that they're actually trying to interrupt the Voice AI.

But one of the things we want to build is another machine learning model that can like recognize the difference between whether or not it's a back channel or someone trying to, uh, interrupt the Voice AI. One quick note on those full duplex models, uh, the Meta AI one can natively back channel because it's like learned from the raw audio data.

So when you're talking to it, it'll go, mm-hmm, uh-huh, which is just so neat. Um, and, uh, yeah, it's just a, it's a tough problem, the back channeling thing.

Awesome. Um, yeah, if you have more questions, you can find Tom. Uh, please give another warm applause for Tom.

Thank you all.