OpenThoughts: Data Recipes for Reasoning Models — Ryan Marten, Bespoke Labs

Yeah, I'm Ryan. I'm a founding engineer at Bespoke Labs, and today I'm going to talk to you about OpenThoughts, which is our project to create the best open-source reasoning data sets. And I'll be switching tack a little bit from our earlier discussions on reasoning and RL and focus on the reasoning part, and you'll see why.

So just so we're on the same page, we've talked a lot about reasoning, but what's actually going on here? So I like this graph from Jason, which shows this incredible performance that's happened in the last several months, where models are getting much, much, much better on certain benchmarks.

And if you look at that, this is reasoning. This is test time scaling. I think everyone here is quite familiar with this, and it seems that certain tasks like AIME, which are competitive math problems, really respond to models when they're able to think step by step and do these long chain of thoughts.

So let's go back to DeepSeek R1. Now, I think DeepSeek R1 was really impressive for a lot of people for a lot of reasons, and RL was a big part of that. But I was also particularly interested because DeepSeek R1, at the end of the day, is an SFT model.

So the final weights that they've released are actually from DeepSeek v3 base, which is fine-tuned on 800k SFT examples, 600k of which are reasoning. Of course, you can see here that RL was a big part of it, and RL was used heavily to create that model which generated this data.

But at the end, it was SFT and a little bit of RL for alignment. So this is really interesting and surprising. And the other thing that was really interesting and surprising to us was these small reasoning models that DeepSeek released, which were incredibly strong.

And this, for us, was a huge motivation, a huge motivation to try to do this ourselves. And why is that interesting? Because if we go back to here, no additional detail was really given on these data sets here.

So if you want to create strong reasoning models, we now sort of have a training recipe, but we don't have the data recipe. That's the missing link. OK, I want to also include a slide here on why is it interesting to train your own reasoning models.

So I'm partially taking this from Amir's talk yesterday on open source and enterprise, which I really liked. But there are these main points: performance, privacy, speed and cost, and then ownership and destiny. I think using reasoning is a great tool to solve a problem, and you shouldn't limit yourself in your toolbox if you're trying to solve a specific domain task.

So as we talked about before, RL is a great tool in this toolbox to tackle reasoning tasks. But we're going to see here that SFT is, as Nathan put this morning, extremely easy and extremely effective. OK, great. Now, the missing link.

How do we actually solve for this reasoning data recipe? There's all these questions that we had when we started. How much data do you really need? What data curation steps are necessary? What are the optimal choices for each step in that data curation pipeline?

And then how do you even go about figuring all this out? And this is the meat of the OpenThoughts project. So today, we're excited to announce OpenThoughts 3, which is hot off the presses, just came out two hours ago, which is our latest and greatest version of our reasoning data sets.

And yeah.

Thank you. And now, this is the state-of-the-art reasoning data set recipe. So you can see here, these graphs are showing accuracy on three of these reasoning benchmarks: AIME, which is competitive math, Live Code Bench is competitive code, and GPQA Diamond, which are science questions.

On the y-axis, you see accuracy is going up. On the x-axis, you see the data scale is going up. So we heard before that scaling is difficult, particularly difficult with RL. The good news is, for SFT, scaling is quite easier.

You can see here, we compare to other open reasoning data sets. So Nematron Nano, Nvidia released this great model, Nematron Nano. It's an AP model, and they also released the data set to train on it. So we compare directly by training on the same base model between our data set, which is our data set recipe, and the Nematron Nano data, which is the Nvidia recipe.

And you can see here, there's a significant gap. So we've shifted this scaling curve upwards.

Great. So yeah, this is the state-of-the-art 7B open data reasoning model. You can see we have measured across the domains of interest, so science, code, and math, and then a couple held-out benchmarks.

So our original goal was to reproduce, to find the missing link for the DeepSeek distill models. And you can see here, we've crushed that goal. So we're significantly outperforming the DeepSeek R1 Qwen 7B model, which we started off trying to reproduce.

And then compared to the Nematron Nano model, which is trained on a different base model, we are also outperforming on some benchmarks and similarly competitive on some others. So OK, let's actually talk about how we achieved this. This is the interesting part for you.

So if we go back to the scaling graph, you can see once again, on the x-axis, we're scaling data set size. So this is a huge method to increase accuracy. And the thing here is, it gets more and more expensive.

Exponentially, it gets more expensive as you keep going. And then vertically, you can see that we've shifted the scaling curve up. So this is what I was talking about before. This is improving the data set recipe. So given a fixed data set recipe, you can always scale it larger, and you can always have higher performance.

But if you want to push your performance to the absolute maximum, the real question is, how do I create the best data set? And therefore, what is the best recipe for the data set? OK, so

enough teasing here. Let's go into the meat of it. So this is how we approach this problem. We broke down the data set pipeline into sourcing questions, mixing different sources of questions, filtering those questions, filtering out the highest quality questions, generating answers with a teacher model.

So that's distillation. And then filtering out bad answers. And lastly, at the end of this entire experimentation, we looked at what are the best teacher models, which teacher model should we select. So through this entire pipeline, we've come down to this final data set recipe.

Now, this was a ton of work. This is a screenshot of our Hugging Face page. So you can see we created over 5,000 data sets and almost 3,000 models. For this project, it was only around 1,000 experiments. But just to give you an idea of how rigorously we looked at the different decisions in each of these steps of the pipeline, and also, I think this is interesting because it peels back the curtain a little bit on maybe what the frontier labs are doing, finding signal at the smallest scale possible and trying out as many things as possible and empirically choosing the best, and then scaling.

And often, sometimes when you scale, you see, OK, what was the best at the small scale? It doesn't actually work. But if you're lucky and you've done good science, then your YOLO run will be the best possible,right?

OK, so these are the key learnings that we had from our data set recipe, and this is what you can take away. So the first thing is that, pretty surprising, sampling multiple answers, so multiple reasoning traces per question in your data set, works really, really well.

The performance does not go down at a fixed scale. If you take a fixed scale of questions, say 30k questions, or 30k examples, and of those, if you take just 30k questions and you only sample once per question, that performs pretty similarly to if you took 1/16, so 30k over 16, and then for each, you sampled 16 times, which is quite cool.

So this allows you this is really cool because this allows you to scale by 16x, which is more than an order of magnitude. And if you remember the graph from before, that corresponds to a pretty large increase in accuracy.

But the other surprising thing that we found was that a better model in terms of its own performance on evaluation benchmarks does not necessarily mean it's a better teacher model. I think a good way to think about this is a brilliant researcher who's maybe a terrible lecturer,right?

We found specifically Qwen 32B was a stronger teacher model than DeepSeek R1. So we switched to that in our recipe, even though previously everyone has been using R1. We also found that the sources of data that had synthetic questions were actually quite good.

Some of the top sources that we selected were entirely synthetic and better than sources, say, that scraped from forums or had humans manually write things. And this is also really good news because synthetic question generation is scalable. So once again, we go back to the x-axis, and we can push even further, which is accuracy boost.

So question filtering also works well. Here, we filtered questions by asking a language model, how difficult is this question, and then taking only the hardest questions. We also had a language model try to answer that question and looked at the length of that answer.

So these are sort of proxies for the same thing. You can imagine that if a problem is a lot harder, then a language model will think more, and it will produce more text. So its answer will be longer.

And these things worked better than embeddings-based approaches or fast text classifiers, which is interesting so much that those approaches were typical for pre-training. So it seems that the filtering for data and post-training is quite different than pre-training. OK, some things that didn't work that were also quite interesting.

Through our experiments, you saw that choosing a smaller number of high-quality sources was much better than trying to optimize for diversity by going for a larger number of sources. That's very counterintuitive,right? You'd think, OK, I'm always going to go for higher diversity, but this is actually not what we saw.

The last thing that was interesting is that people talk a lot about verification, which is obviously very important for RL. And we actually see for SFT and distillation, it didn't seem that filtering based off of the answer or verifying the answer really helped at all.

This is quite surprising. And I think there's some good research in the literature about maybe why this is, because if you have the hardest problem, it might be still helpful, even if you have an incorrect answer to that hardest problem, keeping it in and seeing how the teacher model attempts.

It's not just the final output that matters.

OK, great. OK, so those are all the amazing learnings that we had from OpenThoughts 3, which I'm super excited to share. But now you're probably thinking, OK, they've done 1,000 experiments. I don't want to do 1,000 experiments. I still want to create reasoning models.

How do I adapt this if I want to create specialized reasoning models? So I guess the first thing I would say is be aware that based off of your domain, these exact choices might be a little bit different.

I would suggest, OK, start with our recipe and then iterate on it. If you have capacity and compute, try a couple different choices for each step in the pipeline. And I think a good example of this is we studied each step in the pipeline differently by domain.

So we studied it distinctly for code, science, and math. And we saw, for example, in the question filtering, which I talked about before, using difficulty labels worked well for code questions. But for math and science, it was response length.

And if you think about that for a second, it makes sense because the response length for coding questions is very different,right? For AIME math, it's literally just a number between 0 and 1,000. So the answer is not considering a large portion of the length.

But you can imagine there's very simple coding questions in which the answer is still a lot of lines of code. So yeah, this is one thing to be aware of. The other thing, which I talked about previously, is synthetic question generation.

Because it works so well, and if you're a specialized domain, if you don't have a lot of data for your particular problem, then go ahead, transform that existing data into questions, expand it, throw those as in-context examples, and generate more data.

So yeah, we built an open-source library for this. It's called Curator, and you can try that out. And then lastly, I feel like everyone says this, but it can't be said enough. The evaluation is paramount. If you don't know how well your models are doing or improving, then you cannot make good, principled decisions about your data set recipe.

We spent a lot of time on this. We also have this open-source library on GitHub called EvalComet, which takes care of this and also takes care of the sharding and parallelism. And the key thing here is for very small evaluation sets, if you only have a handful of questions, you should run your model on those evaluation sets many times on average.

So going back again to AIME competitive math questions, there's only 30 per year. So for our evaluations, we gave the model those 30 questions 10 times, and then we averaged to get the final signal to determine which data strategies were working better than others, because otherwise, there's too much noise.

OK, this is also very, very interesting and surprising and promising for you if you're specializing. It seems that you can actually surpass the teacher in some domains with distillation. This is super cool. Usually, you think about only RL can push the frontier.

Distillation is just about catching up to the teacher. But no, that's not the case. So we have an example. It's in our paper where we looked at the legal reasoning domain, so the problem of classifying Supreme Court decisions.

And what we did is we took 2k unique questions. We sampled five answers per question. And then we did do verification here, which did matter. So we threw away any questions, any answers that were incorrect. And when you fine-tune the 7B model, it surpasses R1, which is a very strong reasoning model and also a very huge reasoning model.

So this is very exciting. I think there's a lot more research and also application to be done here.

OK, cool. So everything's open. It's OpenThoughts. And OpenThoughts means open. Go out and build. We have all of our we've got our detailed paper. It's just out this morning. We've got the weights data set. We have a ton of repos for code, for data generation, for evaluation, and synthetic data.

So check those out. This is the team. It was a huge group of people, a lot of work over many months. I think we're all very proud of what we did, but there's lots of people to recognize here.

If you scan that QR code, it goes to the tweet. And everything about the OpenThoughts project is linked in from there. Yeah, thank you.

Whoo!

Allright. Thank you so much, Ryan. That was fascinating. Looks like we're already getting we have at least one question lined up. Again, we have time for maybe a couple of questions. So if you have questions, please line up, and we'll do it.

Actually, before we get to those questions, I will say, as people are leaving, we are going to be back here at 2 o'clock. We've got an excellent afternoon planned on this track. We've got Nathan Lambert. We've got Christian Seguedy, who's the co-founder of X.

And it's going to be a really great track at 2 o'clock back in this room. Also, one more thing. If you do have questions for any of the speakers from this morning, hopefully, they're going to be able to stick around.

Don't let them go to lunch. They're going to be they're sitting up here at the front. So swarm them as soon as we're done. But for now, let's get a couple of questions for go ahead. Yes, over there.

Thank you. Great talk. So two questions. One is, if you're just using SFT on this data, what's the difference between this and regular SFT?

This is just regular SFT.

Oh.

Yeah.

Oh, OK. So then how is regular SFT able to make the models think longer? Because I thought for the reasoning models, they have a thinking block, and they think for hours and minutes.

Exactly.

So how does SFT make it think for hours?

So you're doing supervised fine-tuning on the questions, and the answers also contain the thinking. So the model learns to use its context window and produce these long thinking traces. So it can do this people call SFT imitation, but it can learn to learn this format in the same way.

Yeah.

Thanks. Allright. We'll take one from this side.

Great presentation, Ryan. One question. Why do you think a smaller model like Qwen 32B was a better teacher than DeepSeek R1? What was your insight in figuring out that a good professor makes a bad lecturer?

Yeah, that's a great question. I think this is something we need to investigate more. But you can see that when you look at charts of the length of reasoning traces, you can see the distributions are different. So it might be the case that you're using more of your context window, using more tokens, more steps.

It also might be the case that you just have a better formatted response, better output. This is another great open research question. Interesting. I'll also say on this point, we also tried Claude as a teacher, which is a good, strong model.

And it was just a terrible teacher. So yeah, it's interesting what actually creates a good teacher. Yeah.

Allright. We'll take one more very brief question from this side. And then those of you still waiting on questions, after we have closed this up, swarm them.

So great talk, Ryan. We're doing a similar kind of thing. But I just had a question. Do you guys have any pattern map as to, in the reasoning chain of thought, when things don't work, at what level in the eval do you find out that things are not working or it's not reasoning correctly?

Is there a pattern map or something that you have in your open-source rep?

Sorry, I didn't catch that. Is there a?

So if there are five steps of reasoning to reach a final conclusion, at what step does the reasoning go awry?

Yeah, this is a great question. We don't do this fine-grained analysis, but there is a ton in the literature about this where, yeah, there's a sort of critical step where it gets things wrong. We did the simplest thing possible,right?

You could also go in and try to do more complicated things at evaluation time where you're doing interventions to maybe detect steps that have gone awry and change. Or you can do this when you're creating the data set.

So you could potentially rewrite things. But everything that we tried in terms of messing with the reasoning trace, it wasn't helpful. So yeah, I think there's still more to explore there. This is really just the start of everything in reasoning.