Training Agentic Reasoners — Will Brown, Prime Intellect

Hi everyone, I'm Will Brown, I'm at Prime Intellect. Today I want to talk about training agentic reasoners, just kind of as a very high-level overview. I think a lot of people here are really excited about reasoning, and a lot of people here are really excited about agents, but I feel like a lot of the conversations between these two topics are kind of different.

Where people are like, "Oh, reasoning is this one thing, and agents are this other thing," and the considerations of reasoning are very different from the considerations of building agents. And I think the high-level thesis of this talk is, no, they're kind of the same thing.

And you'll see why as we get into it. First, just to start, RL kind of works now. I think for a long time people were like, "Oh, is RL going to work? Is it not going to work? How hard is it going to be?"

And DeepSeek, I think, took a lot of people by surprise for many reasons, like the costs or whatever, and how good it is compared to the open models, to the big labs, as well as just it being fully open.

But I think it was also just that it was RL applied at scale, working, with surprisingly few tweaks needed. Where you just have a good setup, you have a good signal, you have a model that is good enough to do some learning, and you see this curve where doing more RL results in the model getting better.

And it's also kind of how everyone else is doing it. This is what the big labs are really banking on to drive the next iterations of progress. The o3 release is the one that OpenAI is really excited about, not GPT-4.5.

They stopped serving the big pre-trained model via API, but they have continued to really double down on the scaling direction of doing more and more reinforcement learning and spending more compute on reinforcement learning once you have theright setup to enable progress.

And o3, to me, is a very naturally agentic model. The ChatGPT version has all of these tools. The kind of selling point of it is not just that it's smarter, it's that it's really good at using lots of tools in agentic task settings to solve harder problems that involve interacting with complex systems.

And that is kind of really the selling point of all of this, is that the more complex your system, the more things can go wrong, the more that a generic LLM API is going to be brittle and go off the rails after a certain number of steps.

And RL is kind of the way around it. It's the trick you can do to take the system that kind of works, maybe it works on small scales, but as you go harder, it starts going off the rails, and train the model to be better at that thing.

And so this is a recipe that is still kind of like a research topic that people are not fully sure the best way to do it, especially outside of the big labs. But it clearly is moving in a direction where it's becoming more and more reliable, more and more accessible, and the sort of thing that I think would be silly to disregard as a potential key piece of the future of agentic software and agentic applications.

But it's also complicated. So on the left here, this is the architecture diagram of Veril, which is kind of the most popular software people use in the research world for writing papers to do RL. So if you want to take a model and go do RL, Veril kind of expects that you understand all of this.

On theright, we have GRPO, as presented in the original DeepSeek Math paper back from early 2024. And there's a lot of pieces here. There's a lot of complicated steps going on that I think a lot of people who are used to thinking about APIs, used to thinking about building agents, kind of are hoping they don't have to worry about it, and are hoping that you can just set it aside and something else will work and we'll just use the APIs and it'll all be great.

And I think the reality is somewhere in the middle where I think it doesn't need to be this complicated, but I think you also kind of do have to be aware of it if your goal is really building the most performant agents.

Not necessarily just today you need to know about it, but as a piece of the toolkit to potentially make really powerful agentic software. I think the people who are willing to do this and take the best open models and really RL them for their tasks and configure how to do that well are going to have a huge advantage.

And that's the kind of thing that also allows you to build a moat beyond just being a wrapper API and towards something where it's like, "Oh, I actually have my own model now," but not everyone can be a big lab.

And so we kind of need to meet in the middle somewhere of, "OK, how do we make this a thing that starts to become feasible for startups, for individual researchers to actually do, and at what scale does this become feasible?"

And so agents are the type of product that everyone's excited about. We all love Claude Code and Devin and Manus and o3 and Deep Research. And these are the sorts of products that are really capturing people's attention. They're products that, in their current iteration, happen to work kind of because the models that are being used have been RL to basically do these kinds of things.

Claude is a very good coding agent, probably because it has been RLed on a lot of code. And so it's not very surprising that if you plug Claude into essentially a while loop with some tools, it's quite good at doing these things because it's basically most likely been trained in almost that exact setting.

Same for things like o3. It can do GeoGuessr and whatever, because whether it's literally GeoGuessr or something close to it, they have talked about training it to do this image cropping trick. That's a technique that it didn't just know how to do out of the box.

They said, "Hey, let's give it these tools to do that and use reinforcement learning to train it to do that." And so that is kind of the recipe that we have seen coming from the big labs as if you want a powerful agent that can do a certain type of task, you can use reinforcement learning to train it to do that task better.

And so these are kind of the same thing, actually. Building an agent, the pieces of making an agent in terms of the harness, the environment, the tools, and the iteration is essentially the same conceptual framing as canonical reinforcement learning in the sense of policies, actions, states, rewards, transition probabilities.

And I think the more that we start to view agents as this umbrella, which is not just about static chaining of API calls, but as this interaction loop with evaluations, that framing really is the way to think about RL, which is you build a system where a thing is interacting with an environment and you have some way of evaluating how good it's doing.

And RL is simply an algorithm to improve based on the scores of these evaluations. And if you're building agents and you're tuning your prompts and you're fiddling with your harnesses, this is kind of like doing RL by hand.

What you're doing is you're saying, "OK, currently my evals are saying this. Let's make sure the evals are capturing what I want. Let's look at the data. Let's see if the data matches what my evals are saying." And then, "Oh, let's try a new prompt.

Let's try giving it a new tool. Let's try switching out the model." This is the process which is also being targeted by reinforcement learning in the general sense beyond individual algorithms. About these algorithms, there's a few of them that are very important.

All of them have different implementation details. But in general, the idea is you have a bunch of tasks, like versions of your problem, which are essentially prompts. You have rollouts, which are just completions, potentially involving many steps of interactions, but one sequence of stuff happening.

And then you have evaluations, potentially interleaved throughout or at the end of the sequence. And what you're estimating is the advantage. The advantage here is the idea that sometimes your model will be better than others. These LLMs are all non-deterministic.

You have temperature above zero. You have different things happen in different roles of the dice. And this forking process of saying, "OK, this time it did better than that time, why was it different?" RL is really about saying, "OK, this is the actual thing that changed that resulted in the reward being better, the eval being better.

This is the token at which I went down the good path versus the bad path." And whether you're doing PPO or GRPO, this is the mechanism by which you get the signal of you have something that sometimes went better, sometimes went worse.

Now you can kind of very surgically have the model learn to do more of the good stuff without changing too much overall. I think this is also kind of maybe a reason why DPO, I think people were hoping DPO would really work well.

In my view, DPO does not necessarily have this fine-grained advantage estimate. It's not really clear just from a full good completion and a full bad completion where you're really getting the signal about these complex branching processes. PPO has this, but it's also very expensive.

GRPO, I think, has taken a lot of people kind of by storm in terms of being a very nice middle ground where it's more computationally efficient, it's simple to implement, but also it does have this kind of forking process that comes just from sampling.

There's also just too many papers. So I think a lot of people just see a new paper every day and are like, "Do I have to read this one?" And I feel that too. I think it's difficult to know upfront which of these are going to be important, which of them are just going to be noise, especially because lots of them have very sensationalist titles like, "Oh, Qwen doesn't work," or "Everything only works with Qwen," is kind of true, but there's also more to the story than that.

And I think there's different implementation details of, "Oh, if you change the loss function like this in this experiment, then it works." And I think for most people, it is best to just kind of set this aside and to not get too caught up in the individual details of individual experiments and individual papers, and kind of think more holistically about what is the process of reinforcement learning doing, what implementation details am I willing to kind of leave to other people to figure out and eventually come to me with software that has the knobs set correctly, and which pieces are actually important for solving the problems I care about.

And so for a lot of people, I think the things that are going to be really interesting are things that are relating to actual software, to actual problems that they want to solve out in the world. And agents, I think, are kind of the instantiation of that where this makes sense.

And the thing that makes an agent an agent is tools, the ability to interact with an environment, with a system. A lot of people here are very excited about MCP at the conference. MCP is just tools. MCP is about giving your LLM the ability to interact with stuff, to go solve problems that involve changing files, making requests, editing code, running code.

And so I think these are the papers that I get excited about because there are parts of the puzzle that are not fully solved yet of what's theright way to do all of this. There's still some open questions, but I think those are getting kind of refined.

We're starting to see more and more, but a lot of the code, the tools we have out in the wild to go do this, if you want to go play around with RL, most code bases are very set up for either code and math tasks or things that are quite similar to that.

This is kind of my fault. I had a snippet go viral that was like, "Here's how you do RL on GSMAK," which is a kind of easy math data set. And then I think I've seen a lot of people stick with this as like, "Oh, we're going to RL on math."

And this is also just like math is easy to evaluate. And I think people are writing evals are hard. There's a whole track going on in parallel to this about how to build a good eval. And so I think a lot of researchers gravitate towards things that look like the benchmarks that are also really easy to eval because there's a very clear signal of like, "OK, this thing isright, this thing is wrong, good, OK, we're doing RL."

But real-world tasks are messier than that. We are not going to get great software systems just by hill climbing on whatever question-answer benchmark is popular today. What we're going to do is we're going to have to do is start thinking about the actual systems at hand and the challenges that emerge when we're trying to design these rewards.

And so reward hacking is a real thing. I think this is one of the lessons that RL works, but also it's not always going to work. There are things that can go wrong. And to me, reward hacking is really a message about the difficulty of building good evals.

What you really want with an eval is for it to be easier for your model to do the task than to hack the eval. You want to build a reward signal that actually captures what you care about, where gaming it is more difficult than not gaming it.

If the model can learn to do the task directly just by doing what you want it to do in the spirit of the task, then that is what will happen. It will flow in the path of least resistance.

Models just want to learn, but they want to learn to do better on reward signals. And so your reward signals have to point in the direction of the thing you actually care about. Otherwise, models will find cheats. And I think thinking about these things in combination kind of points a little bit towards a direction that I think is going to be very promising.

And there's some very early signs that this actually can work, which is

when R1 came out, I was kind of speculating what's next. What are the things that are going to unlock this sort of technique being used more generally? And people talk a lot about generator verifier gaps. What are the differences between solving a problem versus checking if you have a solution?

And a lot of problems are much easier to check than solve, but this isn't a binary thing. This is a spectrum of how difficult is it to verify a thing. But there's some kind of signs that you kind of can do evaluations on more ambiguous tasks by just breaking them down into smaller pieces and by using LLMs as subroutines in your evaluations.

Like LLMs that judge on steroids, or maybe you want to actually train a specialized LLM who is really good at doing these fine-grained evaluations. I like using the term rubric as a conceptual general umbrella around reward models, reward functions, LLM as judge setups, the criteria on which you are evaluating a thing.

There's a cool paper from DeepSeek that I found very exciting when it came out a couple months ago about how to train reward models that generate these rubrics on the fly. There was a paper very recently that does this for creative writing and kind of found that, yes, you actually can train reward models that will come up with nuanced, fine-grained evaluation criteria for a task on the fly given the actual problem.

And this gives you something that results in a very fine-grained score that allows you to actually do RL and keep getting better.

And I think this is an area that I'm really excited about to keep watching, but also multi-turn. Multi-turn is probably where we're headed. We want to do agentic search, we want to do tool calls, software, games, long horizon planning, computer use, memory.

Scaling on tool calls lets you solve harder problems. And so how do we actually do this? What's the way to go about building multi-turn agentic systems that we can use RL with? And I think the conceptual pieces here are environments are basically harnesses, rewards are basically evals, tasks are just prompts.

And your policy in the RL sense hopefully should just be as simple as an LLM API. I think the programming interface that makes sense for a lot of people is to have an API that you're writing code as if it's just a normal agent in a loop, but then this is a thing that you can use to go do RL.

And so that's what I've been building over the past couple months. I maintain a repo called verifiers. It's finally on PIP out in the world. You can just install it, but it's been a long time coming. And what it really is, is a toolkit of these pieces to make it so that building an agent that you can actually train with RL feels just like building an agent.

So the interaction protocol here is quite simple. This is the entire rollout function on the left of what happens in the code when you're running an agent to do RL, which is that you kind of set up some initial state stuff, have a while loop for is it done yet.

If it's not done, do a turn. And the thing you're passing here is a client object that's just an OpenAI compatible API. And I think this is the kind of interface that you really want if you want people to be able to go from their agent applications to something that's trainable, something that they can use with RL.

It's been a lot of fun thinking about what are the abstractions, what are the pieces here. And so there's things like parsers and rubrics that I think are nice building blocks that you sometimes want to use. You can also not use them if you don't want to, but I've tried to make it fun and user-friendly.

The other day I was like, let's train a Wordle agent. I think this was like a fun little toy problem where it's not that hard of a game for us as humans, but it's actually kind of tricky to get your code to be this sort of thing where you have this multi-turn interaction protocol that you actually can do learning with.

But now it's much easier. The code to do these things is quite simple. And the reward functions can kind of be relatively simple for this sort of setup where it's like, OK, you want to reward it for solving the thing eventually, but also give it more rewards for doing it in less turns.

And this is a 7B model. It works reasonably well. But one of the reasons it works, which I'll talk about in a sec, is SFT warm-up as a way of kind of lowering the barrier of entry. The code as it is is very much set up so that your environments for RL are also just synthetic data loops or evals where you can plug in Claude or DeepSeek or OpenAI and test.

So you don't have to do RL to debug. You can debug with an API in terms of seeing, is this a good eval? Is this a good reward? Once you're kind of comfortable with it, you can use whatever API you like that you are allowed to use and make synthetic data, do some SFT on it, and now you can start doing RL.

And this helps a lot with small models. I think there's a lot of efficiency challenges that I've been kind of hard at work trying to solve in terms of having all of your computation be utilized effectively, having everything be fully async so you don't have to worry about batching, and that your trainer and your inference can kind of go at the same time.

You can be a little bit off policy. A lot of engineering that I'm hoping if you want to worry about that, great, dig into it, fork the repo, mess with things. If you don't want to, you shouldn't have to.

And the idea here is that this should become something that more people are trying out, more people are having fun with, with exploring and getting a feel for it. Because if it's going to be a thing we have to worry about, if this is the future of building better agent models for your applications, now's a good time to start.

And so this stuff is set up so you can, on a couple GPUs, do a lot of interesting research. The barrier of entry is much lower now than it used to be. I have a lot of fun doing this on a couple GPUs.

We sell GPUs, by the way. Thanks, everybody. I don't think we have time for questions, but yeah.

Thanks.