Your Voice Agent Doesn't Need a Frontier Model - Joel Allou & Ornella Bahidika, Microsoft

Hi, I'm Ornella, and that's Joel, and we built this: a live AI voice tutor. It's run on a small model on purpose, and I want to tell you more why that's not a compromise. Quick gut check:

that silence on a voice call, that's the difference between a tutor and a broken app. When a voice agent pauses for even a second, your brain says it's dead. So when they interfere, a leader of every instant stay creates for the smartest, biggest model.

In voice, that instant is actually a backward. Because our budget was never IQ. It's milliseconds. The AI model needs to start talking in about 950 milliseconds. A frontier model that thinks for a full second has already lost the room, no matter how good the answer is.

So we made the model small and took the hardest jobs away from it. It doesn't decide when what happened in the lesson. It doesn't track what the student knows. It doesn't plan what's next. We have a system in place to do that, and it hands the model a summary every turn.

What's left for the model is one thing it's really good at: talking. And that's the theory. Joel, go ahead and show them what it actually feel like.

Yeah, if maybe I can add some color to what Ornella was mentioning. So if you think about the models of today, especially the frontier model, let's take Claude 4.7, which is from Anthropic. The model is really good at reasoning.

It can give it a problem, in this case a lesson, and it can reason through it. It can reason through what the student is asking, and it can come up with the answer. But that is actually precisely the problem, because the reasoning can take a couple of seconds, and those seconds are really valuable when you are building voice applications.

So what we are doing is saying, hey, let's extract all of the thinking away from the model so that the model focuses on only what matters, which is speaking in our case. So all of the thinking is extracted into a state machine.

So for ACE, we have thought about all the scenarios that are needed for a lesson. We have built a state machine that is able to coordinate each step to the next, and we've also added an intelligent layer on top to derive some of the mastery that a student might need for the lesson to be complete.

So everything, when it comes to what happens next, when it comes to what needs to be displayed, when it comes to how to actually answer a question, is all done outside of the model. And we simply feed that output to the model to speak out.

And so let's go ahead and look at an example and see how that works in real time. So the first video here is without the implementation we've done. So it's a simple Opus 4.7. We ask a very simple question, and as you can see, the model is thinking, it's reasoning, and it takes a couple of seconds to return the answer back to the user.

Hey, what if.

In this video, we've added everything we just talked about on Haiku 4.5, which is a much smaller model. Same question, but now you'll see that the answer comes in about 900 milliseconds. And so that's the beauty of building around the model.

So by removing all of the thinking, all of the logic, all of the reasoning from the model, and actually putting it within the code, we actually save a lot of time and allows us to use smaller models, which are cost-effective and actually better at real-time voice applications.

And as you can see, this feels almost instant. And again, that's because all of the smart parts have already happened prior to the model actually speaking. But I have to be honest, because this isn't necessarily free. It has a cost,right?

A small model like the Haiku 4.5, if it doesn't have any scaffolding, tends to drift on long structure and really needs strict rules in order to be able to stay organized. So the scaffolding piece is the price. But the good thing is, you pay it once and in code,right?

Not on every single turn. So here's the rule: pick the fastest model that your latency budget allows, and then spend the rest of your time actually building the scaffolding. So in our case,right, maybe you build a state machine, you build a reasoning process, you think about scenarios, what happens if this happens, how should your model handle it.

Everything that comes with the logic, everything that comes with the harnessing, you do that outside of the model, and then allowing the model to focus on that one thing that it's really good at. And so that's true for voice applications like ACE.

That's true for real-time applications where latency is of priority. And that's really true for anything that is high volume,right? In those cases, the model is the smallest part of the system. So this is Joel and Ornella, and we are building ACE again, and if you have any questions, let us know.

Thank you.

Thank you.