Don't Let the LLM Drive - Ornella Bahidika & Joel Allou, Microsoft

Hi, I'm Ornella—that's Joel—and we built Ace, a live AI voice tutor that runs a full lesson, start to finish, reliably. The trick is: the LLM is not in charge. If you've shipped a multi-step agent, you know this moment: it's near the demo, then a real user gets in and halfway through the agent decides it's done, or skips a state, or even loops.

The demo never shows you that. And the first fix everyone reaches for is, "Prompt is harder, add more rules." But reliability was never a prompting problem; it's a control problem. Think of it this way: the model is the talent, and the harness is the director.

The model is brilliant at delivering a line, but it's really terrible at remembering if it's on step 3 of 6. So we stop asking it to. A lesson is a small state machine with intro, teach, check, grade, advance, and wrap.

Each step hands the model a narrow contract: do this one thing, return it. The harness validates what comes back, advances the state, and decides what's next. The model never decides where we are; that's the design. Joel is going to show you the harness thing.

Yeah, so when we think about the frontier models of today—Ace, for example, Opus 4.7 Cloud from Anthropic—you'll see that oftentimes people leverage the model for essentially everything: for the thinking, for the processing,right, and for everything in between. While that can be good, it's not always effective in situations like ours, where we're building a live AI tutor that is speaking back and forth with students.

Right? For something like this, we have a need to actually build something that is reliable, something that is cost-effective, and something that is fast. Right? So this is where the idea of leveraging the concept of harness engineering has come in, where instead of having a model that is really intelligent, sort of, "go through everything for us," we will build all of these steps that are needed and provide only the input required for the model to execute a specific scenario.

So when we were building Ace, we actually thought very deeply about state machines. Right? What is the stepright now, and what are the possible steps that could come after? And within each of the steps, what are concrete things that we can provide to the model so that it is confined to that specific action, it is confined to that specific step at that particular moment, and only execute what needs to be done?

So by doing this, instead of having a very heavy model like a 4.7, we're actually able to rely on something like a Haiku 4.5, which is a much smaller model, doesn't have as much reasoning capabilities, but because of the harnessing around it, it's still able to perform at the level in which we expect, saving money, saving time, and saving latency.

So let's go ahead and play this recording, which will show us logs about a particular lesson. So, as you can see in this video, especially on theright side, we see logs on all of the different harnessing that are happening.

Right? So, for example, we see that there's harnessing for a section, which provides input to the model about exactly what to speak about, what to do. We have harnessing about drawing on the whiteboard. We have harnessing that deals with clearing the queue.

We have steps to how to end the lesson, and everything in between. Right? So everything that would allow us to actually build the lesson in a way that is reliable, even if there's a new scenario that comes in, we try to incorporate that in our state machine.

We try to incorporate that within the lesson. Right? So again, the model all it worries about is giving it input, it knows which action to take, and it provides the output of that action. Right? And so the model never really has to think.

It proposes, but ultimately it is the harness that decides. And so for Ace specifically, there are 3 things that we wanted to think about. Like, when is the lesson done is one. Right? Did the student actually get itright?

Like, did they actually learn in the way they were supposed to? And what comes next? Right? And so everything that comes within those 3 categories—all of the different questions, all of the different actions that the models need to take—we have engineered that outside of the model.

Right? So again, it's an input, the model receives it, and gives us an output. Right? And so this is very, very important, and we have found this to be very remarkable. Right? So again, this is applicable to really everything.

Right? It's applicable to something like Ace that is a voice model. It's applicable to coding agents, to apps runbooks. It's applicable to onboarding flows. Right? The same rule applies. Right? We want to find a way to not let the model think, but building abstractions around it.

So a good way to remember on whether you should use this abstraction is essentially to think about the reliability of your agent. Right? If it's somewhat of a coinflip, then you want to take the control flow out of the model.

You want the model to not make as many decisions as it should, and instead build those decisions around the model and simply feed an easy input so that the model can easily produce an output. Right? So don't let the model talk.

Right? Or, actually, let it talk, but don't let it drive. So we're Joel and Ornella. This is Ace, and if you have any questions, please let us know. Thank you.

Thank you.