Real AI Agents Need Planning, Not Just Prompting - Yuval Belfer

The year is 2022, OpenAI just released InstructGPT, which is the first model who can take instructions and actually follow them, really do what you want it to do. Hey, they've even called their model "InstructGPT." So how come, three and a half years later, in 2025, when they release GPT-4.1, they still have trouble to follow instruction?

And it's not just them,right? Every language model has some problems with following instructions. And why is that? So if you remember the good old days of InstructGPT, this was one of the examples that blew everybody's mind,right? Explain the moon landing to a 6-year-old, they got something nice, and everybody went bananas.

And then, with time, more and more developers started to use language models, and now the instruction looks more something like this, or this, or this. Right, and that's what everybody's doing: just taking all the information, all the context, all the constraints, all the requirements, just put everything in one prompt and hope for the better.

And really, even for simple tasks such as instruction following, LLMs alone are no longer enough. And this is where AI agents come to play. And agents don't just use prompting; they need planning. I'm Yuval, I work at AI21 Labs, where we developed our own language models and also an agentic framework, and I want to talk to you about planning.

So let's maybe start first with: what is an agent? There's a philosophical debate everybody's wondering, and the truth is, as an engineer myself, I don't care. Whatever works, just make it work. Call it an agent, call it a workflow, call it whatever.

Some people call LLMs a router, an agent, where we have some sort of a routing model that just takes any query and directs it into a specialized LLM. Is this an AI agent? Who cares? We also have function calling,right, where essentially we're providing the LLM a list of external tools that it can use and let it interact with other APIs, with the world, Google Search, where this is something that now a lot of people talk about MCP, where it standardized this and everything MCP-related is an agent.

Again, it doesn't matter as long as it works, and there's a lot of information about MCPright now, so we will not get into it. And maybe the most popular agent of it all is ReAct, Reasoning in Action, where this is a framework that can be done with any language model in the framework of thought, then act upon that thought, and observe, and so on and so on, every time the model wants to do some sort of a step toward the solution.

But this is something that happens each step at a time, and there is no look ahead to the entire plan. Right, this is kind of like part of a plan: I know the situationright now, I know what the next step I want to go to, and so on and so on.

And what is planning that we're talking about, that every AI agent essentially needs to do? So it's figuring out the steps that you have to take to reach your goal. And when do you want to use it? Is when there are no straightforward problems, essentially complex tasks, and that it requires parallelization and explainability.

Rather than ReAct, where you can't really understand what happened, you can just see all the thoughts that happen throughout, but you don't really know why. You can talk about forms-based planners such as text-based as Magentic One by Microsoft, or code-based planners by small agents from Hugging Face.

What do you mean when I say dynamic planning? Is that there is an option to do replan. Essentially, it's not just have one plan and go all the way in this plan. In the middle, you can ask yourself, is this plan good, or should I do replanning and go somewhere else?

And when we talk about efficiency, we want to put smart execution into that. Essentially, every planner needs an execution engine, because an execution engine can analyze dependencies between steps, which enables parallel execution, and can also have some sort of a trade-off between speed and cost.

For instance, you can use branch prediction for faster systems. Let's take a look at a simplified version of AI21 Maestro, which is a system that uses both a planner and smart execution engine. So you can see here the very simple case of instruction following.

Really, we have a prompt, which is essentially some context and the task, and the requirements are all those: shouldn't be more than 10 paragraphs, the tone should be formal, all mentions of our brand should be positive, and so on and so on.

The thing that you usually just shove in the prompt, now they are separated so it will be easier to validate. And you can see that in every step here, we have some sort of an execution tree, or execution graph, if you will, where in every step the planner and the execution engine chooses several candidates and only continue to fix and improve those who seem promising.

And really, there are several techniques here that you can use for any system, such as using best-of-n, where essentially instead of one generation, we sample from any LLM with high temperature, several generations, or maybe use different LLMs. We also ditch the candidates that do not seem like they will be good, and only pursue the best candidate or candidates based on predefined budget.

And also, we have some sort of validation, so iterations, and keep fixing, and so on and so on. Here is how it looks in something that may be a bit more complicated, where again, we have the input, the original input of the system, and then we have those tracks where the execution engine can say what is the expected cost, the expected latency, and the success probability, and the planner will choose accordingly theright path to go.

Where at the end, we will have some sort of a reduce, where we will have a lot of results, and we want to take the best one or combine all of them for a complete answer to the query that we had in the beginning.

And you can see that it actually works,right? You can actually see that for AIfeval, for instance, hey, we get really high results for GPT-4.0 or Claude Sonnet 3.5 or R3 Mini. We're getting very, very high results by just improving them and using our planner and smart execution engine.

Also, for requirement satisfaction, which is a dataset that we build internally based on customer data, again, you can see that we improve upon just a single call for LLMs. We are paying a little bit here with,right, more runtime and more money, but we get higher quality.

So this is really the takeaway here, that LLMs alone are not always enough, even for what we call simple tasks such as just instruction following. I will always say, start simple and then use whatever works. So if you can use SLMs, use SLMs.

If you can just take a model and put some tools into it, have at it, that's great. Use ReAct if you can. And if you really need, if your task is so complex, you have to use planning and execution engine, this is where you should go and explore.

And I do invite you to join the Maestro waitlist to check it out and see if it works for you. Thank you.