[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

Hello guys. Hello, hello, hello. Yes, sorry for being a bit late; there's a lot of traffic. But, um, hello. Welcome to AI Engineers World Fair, thanks for coming to my session. And yes, today we're going to talk about the deep dive into RL: kernels, agents, and quantization.

You might know me, or you might not know me, but I'm Daniel. My brother is somewhere. Uh, yeah, somewhere. But yes, thanks for coming again. Oh, we also have stickers and other, like, random stuff later; that's after the talk.

So maybe you might know me, maybe you might not. So we, on Twitter, we tweet a lot. We did, like, a gradient accumulation bug fix last year. We introduced something called async offloaded gradient checkpointing. We also work with the Hugging Face, Google, Meta, Mistral teams to, like, fix bugs in, like, their open-source models, like Gemma, Llama, Mistral, Phi, and more.

Yeah, so, like, we—if you want to follow on the latest stuff of AI, better follow us. We tweet about random stuff. You might even know when next models might be released, you know, we sometimes tell people approximately. So that might be very interesting.

We also do, like, open-source contributions to the entire open-source ecosystem. For example, we contribute sometimes to Llama CVP. We work with the Quen team and Mistral on their releases. We also do, like, you know, Phi for bug fixes, Llama for bug fixes, which increase accuracy by a bit.

So definitely, you know, utilize some of the newer uploads that we do, which fix bugs all the time. We just surpassed 10 million monthly downloads in Hugging Face. And yeah, we also have a GitHub package with 40,000 GitHub stars.

And we do, like, essentially we make fine-tuning faster and reduce memory usage. And yeah, that's the GitHub package. Definitely check that out. There's, like, free Colab notebooks. I'm not sure if many people know, but you have, like, free GPUs that Google offers.

You can just use them. Please use them more. And Kaggle, if not many people know, has 30 hours for free of GPUs per week,right? And there's no restrictions on it. You know, please utilize the free resources as much as possible.

Yes, they won't be unhappy. Just use them more. So yes, we have notebooks. So if you scroll down a bit on our GitHub page, there's, like, free notebooks for Colab, Kaggle. We do reasoning, continue pretraining, supervised fine-tuning, and other stuff.

We also upload models to our Hugging Face page. For example, DeepSeek-R1 0528 released a few days ago. We upload 1.58-bit quants, which are, like, very small. They retain most of the accuracy. So, like, these can run on your local device.

Even if you have, like, very low VRAM or, like, not a very good GPU, it will still work. And we will constantly upload models. So, like, sometimes people complain to us, you know, please stop uploading fixed models. You know, it's kind of annoying.

But too bad, you know, unfortunately, models have bugs, so we do have to fix them immediately. You will see sometimes, for example, the accuracy can increase by 10%. Sometimes, you know, the large model providers won't tell you that they uploaded a fix.

They're not going to tell you. But, you know, be sure to download the latest models. You will get all the fixes. Now, today, let's start off from history,right? Does everyone remember Llama? Although, finally, it got leaked,right? It was just a research paper, you know, Meta saying, oh, we trained Llama.

Where's the weights? You know, it's only research access, and then suddenly it got leaked. And then you kind of, this kind of spawned the entire open-source movement. You know, some of the people who are the authors, you know, are not part of Meta anymore, but Llama is extremely important for the entire ecosystem.

And it was, like, the beginning of open-source, kind of, for large language models. The most famous plot from the paper is this,right? So, like, you know, if you keep making the model train more, it just, the loss just keeps going down.

Well, the question is, will the loss keep going down? That's the question,right? So Llama 1 was only trained at 1.4 trillion tokens,right? So now, 1.4 trillion tokens is actually very less. Most models are trained 10 times more. And you can also see from the trend that the bigger the model, the lower the loss,right?

So 7 billion is the blue line, and then 65 billion was, like, the red line. And you can see that in general, as the model gets bigger, it gets smarter. The training loss, the numbers are correct. So you should see generally these numbers from around maybe a bit higher than 1.

If you see training losses when you do fine-tuning of, like, 8, 13, 30, definitely something's wrong. So you should get at least losses around 2-ish, 3-ish. And so now, or now, Google, you know, Google's new Gemma 3 models are trained in 14 trillion tokens,right?

Which is much more. Llama 4 is trained on 30 trillion tokens,right? So, like, literally, like, at least 10 times more. So Gemma is 10 times more. Llama is, like, you know, 30 times more.

Oh, yes, I forgot. You can also access the slides via the QR code if you want. It's also on the docs as well. So if you go to the docs, there will be a link to all the slides.

I will probably post the slides anyways on Twitter and elsewhere so you can access them as well.

Also, if there are questions for people, like, you know, raise your hand, ask. I will essentially do intermission between, like, some parts, and I will ask people if you have questions. Please ask. Last time in the talk, many people asked questions.

I will answer every single one, even if it's stupid. I don't care. Please ask. Sometimes I get stuff wrong. So yes, just ask questions. Are there any questions? I wish it would be no. Okay, okay, let's go to the, okay.

I was, okay. So I don't know if people have seen this plot, very famous from Maxime. He shows the open-source versus closed-source performance of popular benchmarks. I think this is MLU5 shot. You can see the green line is open-source models.

And then the red line, the orange line is, like, you know, closed-source models. And you can see that in general, the slope of the open-source models is more dramatic than the closed-source models. And I would say in general that, you know, the open-source models and closed-source models, in terms of MLU, they've kind of, like, reached the same accuracy,right?

So, like, you can see that, okay, this is already outdated, but in general, you see, like, Llama 3.1, 405 billion, kind of reached, you know, GPT-4.0 level. So open-source models definitely have caught up to closed-source models.

However, there is a however. Recent, you know, like, since September 2024, I would call this something called the open-source drought. No one wants to talk about it, but I will,right? So, like, September 2024, 01 got released, 01 preview.

And to be honest, the open-source community was shocked,right? So, like, suddenly the capabilities diverged,right? So there's something called the MLU plateau where most models, the open-source models and the closed-source models, they kind of converged. So the open-source models were equivalent to the closed-source models.

But suddenly, 2024 September, you know, OpenAI released 01 preview, and it kind of shocked the entire community because the capability or intelligence kind of skyrocketed,right? So, like, with reasoning, long reasoning traces, it just was a total change of mindset.

And for four months, the open-source community kind of died internally because there was nothing we can't replicate it. We don't know what to do. You know, do we do this? Do we do that? I don't know. Like, and so, like, but then suddenly, in 2025 January, DeepSeek-R1 came along, and they released R1.

And that's when the entire world kind of changed their view,right? So, like, you can in fact train open-source models to be as powerful as 01 or 03 or whatever,right? So, like, that's, that was what I call the open-source drought.

However, there is a previous drought even before that. Remember when ChatGPT got released in 2022 December,right? So, like, before even ChatGPT, most models were base models. They were not really instruct fine-tuned that well. And so most large pretrained models were actually useless, but they were terrible.

But then suddenly, ChatGPT came along, and they did better, you know, reinforcement learning from human feedback, better instruction fine-tuning, better instruction following, and it really changed the world,right? So, like, I think large language models were already here before 2022,right?

They were already there. But it was just ChatGPT which showed that if you have good data,right, good instructions, good answers, good supervised fine-tuning, and good reinforcement learning, you can actually make the model very useful. And yes, again, open-source had a delay, a very long delay,right?

Until, like, Llama 1, I guess. And so always, I would say open-source always tries to catch up to the closed-source models. The next question is, what is after reasoning? Is there going to be something else? I think that's a very good question.

My personal take is it's going to be very hard. I think reasoning was, like, the last, most, the DeepSeek-R1 paper said that most likely the model already has these reasoning capabilities, and we just need to accentuate them. And so, like, I'm not sure if there's going to be some other new, you know, like, step function where we'll get to, like, the next capability.

But in my view, I think, like, every single time, the closed-source models will always do, like, a step function. But who knows? Maybe, maybe now it will plateau forever. I don't know. So that's, like, you have, like, you know, you have, like, long discussions about, you know, if AGI is going to come or not, like, but who knows?

The talk is not going to be about that, but yes. Next. So I call the first jump the SFT or RLHF jump,right? So, like, that's essentially, if you do good supervised fine-tuning, you get this large jump in performance.

And then the second jump is called the RL jump,right? So, like, this essentially can increase performance dramatically if you employ second methodologies, like RL,right? So, like, but the question is, like, what's the next jump? I don't know.

So I'm not sure if you guys saw this picture before. It's very widely known in the community about, you know, by Yann LeCun. He essentially showed this cake. And essentially, unsupervised learning, or, like, just pretraining in general, you know, is just a cake, not that good.

And then supervised fine-tuning is kind of the icing on top of the cake. So, like, it's a bit better. And then the reinforcement learning is the cherry,right? So, like, I'm not sure if people like the cherry, but, like, some people like the cherry.

And so, like, the goal is, how can we get the cherry? But the problem is, there is so less data about this,right? Reinforcement learning is, like, very, very, very less data. And so the problem is, most large model labs will train these large pretrained models, and then they will iteratively refine it to make the model better through supervised learning, through reinforcement learning.

Interestingly enough, this slide was actually shared last year, very popular. But actually, this was from 2016 September. So I had to dig this up on YouTube. And so Yann LeCun actually talked about this back in 2016. So literally, nearly 10 years ago, I was like, how can you, wait a second, that's 10 years ago.

Very long. So this slide was actually very popular on Twitter. I think it was in November last year. People kept tweeting about it. I don't know. I saw this. I was, like, shocked. But yes, so this encapsulates, like, the current AI boom.

And so, like, firstly, like, when we talked about these large models, remember, they started from a base model. And so we call these training stages,right? So when you have a base model, you then convert it to a chat model,right?

So, like, for example, ChatGPT is not a base model. It is an instruct fine-tuned model, or, like, some sort of fine-tuned model from a base model. So actually, OpenAI does have a base model somewhere sitting in their server somewhere.

They're probably not going to serve it ever, but it is somewhere on their computers. And they essentially fine-tuned it to make ChatGPT-4. Claude 4 has most likely a base model, and then they fine-tuned it to become Opus,right? So, like, Gemini also has a base model, and they convert it to Gemini 2.5 Pro.

So this, this phase, when you convert a base model to a chat model, is the fine-tuning phase. And then the question is, like, you know, what do we do in the error,right? Like, you know, is it reinforcement learning?

Is it supervised fine-tuning? Is it, like, some other special source? I don't know. But, like, we essentially will discuss about these topics. Any questions first?

Okay. So, for example, in open-source models, you might have seen Gemma 3 PT, Gemma 3 IT, Llama 4, Llama 4 Instruct, Qwen 3 Base, Qwen 3, Mistral Small Base, Mistral Small Instruct, Llama 2, Llama 2 Chat,right? These, like, terminologies, to be honest, I think the open-source community should standardize the, like, terminologies, like Instruct or Chat or, you know, no, no, not even a word, like, you know, IT and PT.

Maybe they should, like, standardize it a bit. But in general, if you see IT, it means instruction, instruction-tuned. PT means pretrained. Instruct just means, you know, instruction fine-tuned. Qwen 3 just removed it entirely. It's just called Qwen 3.

And then the base model is called WebABC. And so, like, essentially, these naming methodologies, if you see on Hugging Face, hopefully you will now recognize these different types of models.

And so generally what we say for, like, you know, reinforcement learning and fine-tuning is, I would say it's called fine-tuning is everywhere. You start off with pretraining. You then convert it into a supervised fine-tuning model via supervised fine-tuning, SFT.

It's called SFT. You also might hear, like, IFT, which is instruction fine-tuning. They're the same thing. And then we call something post-training, the post-training phase. But actually, recently, it actually kind of changed. So I don't know if you guys have been keeping up with the latest stuff, terminology.

I actually don't really like terminology anymore. But, like, we have something called pretraining, which is you take, like, all of Wikipedia, all of the web, you know, everything, all of the data you can ever see, shove it into the model, predict the next word.

That's called the pretraining stage. We then have something called the mid-training stage, which essentially gives you higher quality data. Like, for example, you can wait Wikipedia more because it's higher quality. You can essentially do long context extension as well.

You shove this during the mid-training stage. So if your context of your model is very short, you want to extend it to very long context, you shove this during the mid-training stage. And then the second stage is the supervised fine-tuning stage, where you want to convert the model to a chat model.

And then we have the post-training phase, which is, like, preference fine-tuning, DPO, RLHF, and stuff like that. And then we have this new thing called reinforcement fine-tuning, or RLVR. If no one knows what RLVR stands for, it stands for Reinforcement Learning with Verifiable Rewards.

And this is, like, a new paradigm, not the same as preference fine-tuning or DPO, where we consider reward functions to make models much better. And so this is how I would envision, like, you know, the whole training phases of models.

Another way to put it is we have some random initialization of the model, like some random weights of the model,right? So, like, 7 billion parameters, literally random numbers,right? Like GPT-4, GPT-4, I don't know, 1.4 trillion parameters, like just random numbers.

And then somehow we move in the space, like the black line,right? So pretend this is, like, some high-dimensional, 1.4 trillion-dimensional space. And then we somehow move in this space, and then we get the final model,right? That's the green job.

The question is, how do we move in this space to get to the final model? That's the question. Most people, what they do is, firstly, you start from a random initialization. You do the pretraining phase, which is very long.

You get to this dark blue dot,right? That's called the pretrained model. And then you do some supervised fine-tuning, instruction fine-tuning to get the blue dot. And notice the line for the light blue line is very short because it is very short,right?

There is not that much data for supervised fine-tuning. And then somehow we get the blue dot. And then we keep doing more iteration to get to the purple dot, which is through preference fine-tuning. And then finally, we get the green dot, which is reinforcement learning via, you know, verifiable rewards, like, you know, 03 or 01.

And so, like, the goal is somehow we have to move from the black dot to the green dot. And essentially, all of large language models, all of AI is just an optimization problem,right? So, like, how do we make this easier to get to the green dot?

You could, you know, you could kind of theoretically guess. You could just, why don't you just go from the green dot, black dot to the green dot, like, skipping all of the dumb phases, you know, just skip it entirely?

Yes, you could do that, but it's not going to be very efficient. You're going to be waiting there for, like, you know, I don't know, millennia. Your loss is not going to go down. So the tricks that we found in AI is, like, you have to do these phases to get to your final green dot.

There is, like, a new methodology where you can actually bypass this supervised fine-tuning stage and the preference fine-tuning stage and directly go to the green dot. There is a way, and that's the dark red line. I think DeepSeek Zero kind of, like, showed that.

You can use a pretrained model, a base model, and directly do some reinforcement learning with verifiable rewards and just skip it entirely. So that's, like, a new paradigm that people want to focus on. In my view, I think you should still do the light blue, the purple, and then the green.

I don't think so you should, like, directly skip over to the green. If you want to waste resources, you can skip to the green. But I don't think so large model labs want to waste resources. Hopefully not.

So I don't know if people have seen this diagram, agents in the old sense. So, like, everyone keeps connecting agents with reinforcement learning. Okay, but, like, why? So in general, an agent is you have some sort of environment.

You have, like, the agent doing something in the environment. You get, like, an action. You do the action, and then you get some sort of reward. And then reward is R, S is the state, so the current, what the environment currently looks like.

And then essentially, RL tries to optimize this loop. You're trying to maximize the reward, giving some sort of action. And that's why, like, you know, that's why RL and agents are kind of connected. Assume the agent is the language model,right?

So assume the agent is, in fact, the language model. And so this, the environment is kind of fishy. Like, you know, it's, it's hard to say what exactly is the environment. It's more like the language model's inference space.

That's the environment, kind of. But, like, pretend this was a game,right? So, like, the agent was the computer. The environment is, like, Mario, for example,right? And so, like, you're playing the Mario game automatically, and your goal is to win the game.

And so, like, the whole goal of RL is to maximize reward.

Another one is, like, Pac-Man,right? So, like, you have the yellow Pac-Man. You can either go up, down, left, orright,right? So, like, up, down, left, orright. That is the action. That's the A. The yellow, the, I think, I don't know, orange or something.

Yeah, the orange little things are, like, rewards,right? So, like, if you eat, if you eat, you know, a yellow, an orange dot, you will get positive reward, for instance, R plus,right? So, like, you have R pluses. If you eat a very big one, you'll get, like, very large reward.

But if you encounter one of the enemies, you will get minus reward,right? So, like, the question is, how do we maximize the reward based on this environment?

For language models, there is a trick. The trick is this loop kind of changes because we don't actually have a continuous loop. The state does not actually change over time,right? So, like, for example, in a game, in a game, if you do an action, the whole state changes,right?

The environment totally changes. And so you have to, like, continuously keep a history of the past steps. But in language models, there is no history,right? So, like, if you do a prompt, what is 2 plus 2? If you ask another question, what is 4 plus 4?

It's, like, totally not relevant to your previous prompt. Okay, well, fine. It is kind of relevant, but you, like, it's not directly correlated. And so, like, you can actually delete one of the lines. That's the next prompt. You can delete it entirely.

And so, for example, what is 2 plus 2,right? So, like, essentially, you have all of these options. It could be 0, it could be 1, it could be 2, it could be infinity, it could be B, it could be D, I don't know.

It could be anything that you like, a symbol. And so the, what is 2 plus 2 is the state. So, like, what is the question? That's the question,right? So, like, the question. The reward, for example, if you choose the one, if you choose 4, your reward is plus 1.

If you choose anything else, your reward might be negative infinity, 0, whatever number you like. You can come up with any number you like for reward. It doesn't have to be plus 1. It can be plus 10. It can be plus 100.

You can do anything that you like. You can also do distance-based scoring. For example, is, you know, is choosing the number 5 better than 0? So that's a question. What do you guys think? Is choosing the number 5 better than 0, or is it worse?

I think it's better.

Yes, okay, better. So what would you do for the reward then? Pretend the model outputs 5 for what is 2 plus 2.

Something that's not 0.

0? Someone said 0? Okay, 0 is fine because it's wrong. Yes? Like, if you, the answer 5 is wrong, so you should probably give a reward of 0. But is there a better answer?

Less than 1.

Okay, yes, less than 1. So, like, some sort of, like, maybe 0.8. I don't know,right? So correct. So, like, you could do, like, the answer divided by the correct answer. So, like, if it's 5, you divide it by 4.

You could do some sort of, okay, no, that's wrong. It's 5 minus 4 divided by 4. So, like, something, some reward like that. Pretend if you're, pretend if the model says A, what is the reward?

Minus 1.

Minus 1? Okay. Or could be minus 10 because that's very bad,right? You should not output a letter. It should be some sort of number. So that's how you design reward functions,right? We just design a reward function. Shove this into, you know, take your reward function.

It's like just if statements. Shove this into a language model fine-tuning phase, and there we go. You have 03. Okay, well, you won't have 03, but you know what I mean. Essentially, 03 is a collection of all these reward functions,right?

So, like, for example, this, what is 2 plus 2 is one question. Remember, it doesn't have to be what is 2 plus 2. It is a general maths question. What is 10 plus 20? What is 10 times 200 divided by 10?

You know, whatever maths equation you ever want. And this function can take your question and convert it into a number. And 03 is just a collection of all of these reward functions.

And so the goal of RL is to make the good ones more good. You want the good rewards increase in value. So, for example, the 4, you want the 4 to appear more. But you want the 3 to decrease, you know, you don't want the 3 to keep appearing in your answer.

But you want the D and the B to be very, very, very heavily penalized. That's the goal of RL,right? So, like, we don't actually have the answer,right? Okay, this, this question is very easy. What is 2 plus 2?

Obviously, it's 4. Yes, okay, that's very easy. But pretend you have some sort of complicated question, like, for example, how do I win the stock market? For example, like, dumb, dumb thing. Okay, how do I win the stock market?

You don't know what actions you're going to take,right? But the point is you have the result,right? You have the result, like profit or loss. But the question is, we don't know how do we get to the good profit.

And so the question is, how do we maximize good actions as much as possible and decrease bad actions as much as possible? And that is RL.

And so OpenAI released something called RLHF in ChatGPT, and they, oh, actually, I think it was in StructGPT. But anyways, for ChatGPT, they showed that you just need some training data. You need some data. You interact with the agent, which is the language model.

You then get some actions, which is your answer to the language model. You feed this into a reward model, and then you get some reward. And then you keep, you know, iteratively doing this step, and you'll finally get ChatGPT.

So remember, the base model you start with, so ChatGPT base, you convert this into ChatGPT-4 via this method.

To expand on a bit, if you guys have heard of PPO,right? What is PPO? Essentially, PPO is just, you just expand the box for the agent,right? So the language model is like an agent. You expand it, and there's just three models inside of it.

There is a generating policy, the reference policy, and then there's a value model. And that's all. Not that special. We will talk about each of these things separately. But PPO is just an optimization algorithm to make RLHF work better.

GRPO, which is the algorithm behind DeepSeek-R1, smartly deletes one of the things, the value model. It just gets rid of it. Now, why would you delete it? We will talk about this. But the trick is, if you delete a model, the value model, you just save parameters.

You save compute, and it's much more efficient,right? Remember, each of these models is kind of like a large language model,right? So, like, you know, pretend you're generating, your model is, like, already 1.4 trillion parameters. What are you going to do?

Make another 1.4 trillion parameter model for the value model? So we just get rid of it, delete it. And that is GRPO. That's the only difference between, okay, well, there's other differences. But the biggest difference is we get rid of the value model.

Any questions? Yes.

So you talked about negative reward.

Yes.

It's confusing because in pre-training, isn't it, it's like, it's.

Probabilities?

Well, it's like negative rewards,right? And then, so I guess it's the, why is the phrase reward used when it can be positive or negative versus comparing it to pre-training where it's always negative?

Do you mean pre-training as in, like, the negative log likelihood or, like, some prob? So when during pre-training, the goal is to maximize probability.

Yeah.

So, like, you know, you output some number from 0 to 1, the probability of the next word, and you want to maximize that. For RL, you want to maximize reward. So if it's a negative reward, you still want to maximize that.

So if it's negative 1, you just want to make this negative 1 go away, and you want to be positive in the positive range. But also, rewards can actually be just negative,right? So, for example, if your reward function can be negative 10 and negative 1, the good one is negative 1, the bad one is negative 10.

Your goal is to move towards negative 1 as much as possible because your goal is to maximize it. So I would say the reward is a misnomer. You could just add 10 to everything, and then, you know, it scales the numbers away.

Does that kind of make sense or?

It does. It was a question of nomenclature, I think.

Okay. Yeah. Most people, I would say, like, to be honest, they don't like negative rewards, actually. In the RL space, people just like to do positive reward. I don't know. I like negative rewards. I feel like it's more, for me, it's more intuitive.

Yeah. Any other questions? Yeah, yeah.

So you've got your language model, and then you've got generating policy and reference policy.

Yes.

Right? What, what models are being used there? Is it the same as your language model? Is that another model?

Trick? Yes, very good question. There are some tricks that you can employ. Most people just make them at the same model. The reference model is like the beginning of the model. The generating model is the model that you're updating.

So, like, essentially, the reference model is like the base model. Okay, that's probably not a good, okay, fine. Just keep it as the base model. The base model. The generating policy is actually the model where you update it.

So, like, the model that you update, the base model's update. So every single time you get a base model, you update 1, update 2, update 3. That is the generating policy. But we will talk about this. So it's essentially the same model, but there are updates to the model.

So the reference policy is the model that is not updated. The generating policy is the model that is actually updated. And they're both the same model. So the language model, one of them's updated, the other one's not updated.

But we will talk about that. Yes.

So the actions, is it typically one token or is it more a series of tokens, like an actual full completion in a latent space or, like, how do you think of it in context of?

That's a good question. In the Pac-Man case, the action will be a string of actions,right? So, like, you can go up, then down, then left orright, you know, some sort of, like, long history. For the, in the language model space, we just generally, this is called single turn and multi-turn.

Generally speaking, currently, single turn is what most people do. It's just one action. So the action will be, the action essentially is saying, what is 2 plus 2? The answer is 4, and that is the action. The action is actually the inference space.

So, like, what is the actual chain of thought? That is the action. And so, like, it's just one, though. But it is the total sum of the chain of thought. So, like, if you have, like, what is 2 plus 2?

I think the answer is 4. You know, let me do some working out, blah, blah, blah, blah, blah, blah, blah. That is the entire thing. If does that kind of make sense?

Yeah.

Okay. Yes.

There was a Claude conversation around how, like, to finish a poem's next line, it has to, like, think ahead to the last letter, to the last word to match the previous one. Do you think, like, reward focus versus next word completion allows you to do more think-ahead inference?

You could.

Really greedy when you're trying to understand the reward function instead of maximizing the next word focus?

I think for pre-training specifically, there are research papers which show that actually pre-training doesn't just predict the next word. It does try to predict many words ahead. And so, like, yes, maybe reward model. In reinforcement learning, you can see, you can essentially, it essentially accentuates a pre-training behavior.

So maybe these behaviors already exist in the model. We just see it more often. So maybe I would say that, I would say that the model itself, it already has this capability. We just want to make it more obvious.

And so maybe the model already knows how to do that. It already knows it predicts 10 words ahead or 20 words ahead. It already knows how to do that. But we just want to make it more obvious. I'm not sure that answers.

Would it be safe to say, like, so if it's a poem generation, you want, like, every last word, every, you want almost a circuit for the last word to match the thesis line's last word?

Okay.

It's more likely to build that circuit with generative GRPO instead of, like, the classic ones which would rely on next 100 word predictions or next 500 word predictions?

I think for reinforcement learning specifically, yeah, I guess yes. I think it essentially, your goal is to maximize a reward. And so, like, however, whatever way you try to get there, it's different from generally pre-training. Pre-training is just maximizing the next, the probability of the next word.

But reinforcement learning is you're trying to maximize reward. The question is, how do you actually maximize a reward? Do you, like, do chain of thought? Do you do what you described, like thinking about the next, you know, in the future or something?

I don't know. Like, I mean, the question is, like, what is the reward function actually doing? I don't know. What is the language model actually doing? I don't know. I don't know if that answers your question. Like, it's, to be honest, yeah.

Yes. Yeah.

I was curious about when you were talking about, you know, the arithmetic, whether 5 is a better answer than, say, negative 100 or something.

Yes.

Plus 2. And, like, given that there are these, like, closed circuits between all these different related, you know, mathematical functions you can do on numbers in the space, like, whether it is, like, in the literature or in, like, the current state of the art, better to train it so that a closer prediction is more accurate or whether just saying, like, theright answer isright and everything else is wrong, which in some logical sense is true, like, tends to produce more performance.

Yes.

In that space.

Yes, you're correct. You should have data, which is, like, getting more accurate data. Is that what you're trying to say? Like, you should have data, which is, like, for example, what is 2 plus 2? You should get more data, which is, like, 4.

It shouldn't be, like, 5 or 10 or minus 100.

Well, like, saying that, like, the practice of saying that 5 is better than 10.

Yes.

Like, in general, is that what people do and produce better answers?

Oh.

They tend to say, like, when there is, like, exactly one correct answer and everything else in a mathematical sense is equally wrong because it's not that answer.

That is a good question. I don't know. I think large model labs won't tell you exactly what they do. In our experiments, when you use our notebooks, we actually showed that if you do distance-based, so the closer your number is to the actual number, you will get better results.

But generally speaking, it's easier to just say 5 is wrong. Just give it zero reward. Everything is zero, and then the good one is, like, 1. It's actually much easier to do. For example, if you want to do execution of code, how do you actually do distance-based scoring,right?

So, like, if you ask it to create a Flappy Bird game, you just have to find an output. But you don't actually know how to verify, like, you know, oh, is this Flappy Bird game better than the previous Flappy Bird game?

It's only in a mathematical sense you can, like, do distance-based scoring. I'm assuming large model labs, they probably do the 0, 1 better. Like, the majority of them just do, like, yes or no, yes or no, binary. But in our experiments, for math specifically, you should do distance-based scoring.

It makes the model learn faster. Yeah.

For verifiable domains like math, is it actually tenable? Because 2 plus 2 makes sense, but it's not going to scale for, like, really large numbers or large multiplications. So are we going to end up, is it end game using tool use to calculate that or the model could potentially be trained to solve math?

That is a good question. In the olden days, before this paradigm came along, we would think that you can just use a tool, like a calculator to, you should actually, I would say you should still use a calculator to calculate 2 plus 2,right?

You should not use a language model. But with RL, you know, VR, the trick is we actually found that actually, wait a second, if you just do 2 plus 2 or you do another question, like 10 times 10 or you do some sort of complicated mathematical expression, you know, like the derivative of x squared or something, I don't know, like some random mathematical equation, it randomly learns to actually solve that equation without actually doing overfitting.

And so, like, I would say that with RL, you can actually make the model actually learn how to do multiplication, how to do addition. So it's actually in the model. Does that kind of make sense?

Yeah. Could we use this in production or?

Oh, yes. Yeah. People use that in production. Okay, maybe don't use it in production. You know, you're not sure if the answer is correct. Maybe it will say 3 plus 3 is 7. Okay. I don't know. It's possible.

So, like, essentially, but it's getting better. Maybe in the future, all of mathematical equations can just be done by a model. I think in the, maybe a few months ago, maybe, like, September, you know, before 01 got released, actually, not even 01, a few months ago, people would still say use a calculator, you know, like some sort of tool calling.

Yes, you should probably still do it. But imagine, you know, as time goes on, as models get better and better and better in terms of, like, training data, just for the math equation, you know, 2 plus 2, imagine in the limit, as we get all of the world's data for just this maths question,right?

2 plus 2, 4 plus 4, 10 times 10 or whatever, it should in theory solve them all. In theory. Yes, it's always in theory. But yes, you don't need a tool calling. It's not necessary. Yeah. Yes.

My question is about the reward model. In practice, are people using large language models as a reward model?

Good question.

For inverse reinforcement learning or is it hand-rolled or?

Yes, I actually.

A viable solver?

Good question. Oh, I was going to go in the next slides. We'll be talking about that. Yes.

How does this change with, oh, sorry, multi-turn?

Multi-turn?

How does this change with multi-turn? I mean, you showed just a single turn. Great. So in a multi-turn, you can sort of loop out, like, a tree in.

Yes. You could do multi-turn. It's a bit more complicated. You just imagine. There are tricks you can do. You imagine that your current step is good. Imagine it. And then you just continue doing inference. You append, like, your next question.

Like, for example, how am I going to, what is 2 plus 2? You say, okay, let me think about this question. What is 2 plus 2? Blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah. The answer is 4.

And then the question is, what is your next question? Maybe the user interacts with it and says, okay, I don't think your question's correct. Oh, I don't think your answer's correct. And then the model says, oh, okay, let me rethink about this.

Blah, blah, blah, blah, blah, blah, blah. I still think the answer is 4. So you could chain this all together and shove it into the, you know, the whole RL step. You could do that. It's a bit more complicated.

I think the diagram will be a little bit more different. Yeah.

I guess the follow-up to that could be, do you assume, like, a loop is a single turn or a loop is a lot of turns and then you only give a reward at the end or you give, like, sub-rewards or.

Very good question. So there is, in the DeepSeek-R1 paper, you could do sub-rewards or you can just do the reward at the very end. I think sub-rewards might actually do better in general. But the question is, sub-rewards is very hard to calculate.

You would rather just wait, you know, all until the very, very end and just give a reward. That's probably the easiest. So it's more about efficiency. It's all, to be honest, all of AI is about efficiency. What is more efficient?

What is more, it's all optimization. So the answer is, like, I would suggest people just to, like, shove a reward at the very end. Yes.

Once you get your reward signal, is it just the reinforce algorithm with the gradients to go back to the weights?

We will talk about that. Yes. Yes. Mostly, yes. Correct.

Reinforcement?

Yes. So we will talk about reinforce. We'll talk about PPO, GRPO, and stuff like that. Yes.

You mentioned with the slide about scalable lines that using RLVR, like, to skip the intermediate steps is.

Yes.

A waste of resources. Can you talk about, like, why it's a waste of resources and, like, how you achieve seeds, like, 100%?

This one?

Yeah.

The problem is, if you skip from pre-training to the RLVR stage, it's relatively hard because your model doesn't actually know how to do instructions,right? So, like, you have this base model. You ask the question to the base model, what is 2 plus 2?

It's not going to say, I think the answer is 4. It might, you might be lucky somewhere in your pre-training data, somewhere on the web, someone asks this question, what is 2 plus 2? And then, you know, the answer was, like, oh, the answer is 4.

But you have to be lucky. So, like, so the problem of this is the whole trick of SFT is you want to force the model to answer what is 2 plus 2 in an instruction way,right? So, like, it will tell the model, what is 2 plus 2?

You want it to say the answer is 4. You don't want it to, like, blabber on and, like, rich jerry jig, like, get some Wikipedia article and shove it as the output. So the whole point of SFT, preference fine-tuning, and stuff like that is to make the model forced to make a more optimal to, like, output conversation style.

If you want to skip, it's also fine. It's just not efficient because, like, you could do this. I'm assuming large model labs are trying to do this. So it's not, like, a you should or you shouldn't. They are trying.

Does that, okay. Yes.

Do you have a question about a couple of questions? One question is that this is RLVR. Is it an online policy optimization or offline? Basically, does the reference model change after evolve or stays the same?

The reference model does not change. So the reference model is just a model that you didn't train. It's like the, it's like the, it's like the base model or, like, the SFT, whatever checkpoint you started with, it doesn't change.

You could change it. I think that would be too expensive though. I think if you change it, that would be more complicated. Remember, all of AI is about optimization and efficiency. So I feel like you don't have to.

You could. I don't know if their paper is talking about it though. Maybe OpenAI does it. I don't know.

Another question is that do we need less examples of, you know, training data for, you know, any of this RLVR compared to, I mean, new thinking and anything, you know, compared to SFT? Do we need less training data or what do you think?

So the trick of RL is you just need a reward function. You need to make that. And you don't need data. You don't need the answer of the data. Oh, actually, you do need the answer. You don't need the chain of thought.

You just need lots of questions. Like, what is 2 plus 2? What is 4 plus 4? Remember, you can actually automatically generate this,right? So, like.

Number of samples, like, do you need less number of samples compared to SFT?

You should do as much as possible. You, most large language models, I think for, like, you know, 03 or 01, I don't know what is the percentage of compute. Maybe they spend, like, 5% or less. But the goal is, what happens if you spend double the compute just on RL,right?

So, like, previously, if you do 14 trillion tokens on pre-training, make RL 14 trillion tokens. And then the goal of large labs is to just do that. So currently, it's very less. But over time, it will increase.

But compared to, like, SFT, you know, like this thing you hear, the number of samples will be much lesser,right?

Currently, yes.

Because it's expensive. Like, you need, like, three models.

Correct. It's expensive. But over time, I think, like, maybe by, I don't know, next year or, like, this year, large model labs, their goal is to do this phase the most. That's their goal. Because remember, you can automatically generate questions now.

What is 2 plus 2? What is 2 times 2? What is 10 divided by 10? I don't know. Generate as many maths questions as you like. But remember, you can also generate, you know, like, coding questions. You can generate any questions that you like.

Or you can use the supervised fine-tuning data itself for the RL step. You can do that as well. Does that kind of make sense? Okay. Any, okay.

How do you protect the SFT from being, like, screwed up by?

Oh, good question. Yes. We will talk about that.

Say, like, you have 2 plus 2 equals 4. 5 minus 1 is also a good answer. 2 plus 2 equals 5.

Correct.

But I don't want more equations. I want the answer.

Very good.

So is there, like, techniques to make sure that we're not violating our instructions?

Yes. We will also talk about that.

Allright.

Clipping and stuff like that. Yes. Yes.

Is there any research that's been done on how, I guess, RL specifically, but maybe training more generally, affects specific circuits in the model? So, for example, you know, there's a circuit that, like, says 2 plus 2 is 4.

It just knows that.

Yes.

But can you incentive, is there anything that's, like, incentivizing the model to, like, learn addition, the concept in general?

That is a very good question. I don't know. I think that's, like, during the pre-training phase. Essentially, somewhere, somewhere in the internet, someone wrote what is 2 plus 2 somewhere. And then somehow, maybe someone did a formulation that, you know, some sort of derivation of, like, what is 2 plus 2.

Okay. I don't think anyone has done. Pretend there is some derivation of, like, some complicated maths equation. And so the model somehow learned to predict all of that entire trace. And if it keeps seeing this, it would, like, accentuate the fact, oh, okay, I've seen this before.

Let's make this even more, more prevalent in the model. So somewhere in the model, it has learned 2 plus 2 is equal to 4 somewhere.

Yes. I guess what I'm saying is, so we can use, like, the RL stuff specifically. You're saying that if you get, like, super bad reward or super small or super good reward, you can.

You're weighting it.

You can kind of coax the model into a certain direction.

Correct. So there is actually two schools of thought. The first one is the model already has this knowledge,right? It already knows what is 2 plus 2. And you're just, RL just tries to, like, maximize. What, if it sees 2 plus 2 is equal to 4, it tries to weight this factor more, weight this circuit in the model more.

So the model already learns, learns. But then the second school of thought is, like, okay, maybe the model doesn't know actually. And RL actually learns a new thing. I'm more in the first one. The model probably already learns.

It already knows it. And we're just maximizing the, you're just trying to make it more accentuated.

Exactly. So the extension of my question is basically, I want, I'm wondering if you can steer, or maybe there's, like, research to this, that you can steer the model away from the 2 plus 2 circuit to the, always do the, like, do addition circuit, if that makes sense.

You mean, like, just do addition?

Yes.

You could, I guess what you could do is, like, get the language model, see which weights are changed during the RL phase, which weights are changed, and you just give it what is 2 plus 2, what is 2 plus 2, what is 2 plus 2.

You just keep doing this question. And you can see which of the weights are changing. And essentially, you can extract this from the model. You could. I don't know if there's research about this, but I'm just making, I'm just making stuff up on the spot.

You could do that, if that makes sense. Oh, maybe that's a research question someone should do. Research paper. Any other, yes.

If you go back to the GRPO slide, the more and more you get, wasn't it basically quality and reward? They save money and compute by just making equal solutions that the reward. You have to answer exactly 4 in your scenario for R1,right?

That's one of the optimizations.

Yes. We will also talk about that. Yes, yes. Yes. Okay.

One question.

Okay. Yeah. Sorry.

UBEXPAT usually changes the RL, changes all the parameters in the model or is it, like, more targeted? Like, this set of RL would be this set of, like, targeted parameters. It's a subset.

You could. There's, like, two large model labs will most likely change all of the parameters. Every single parameter is changed. But there are papers which show that actually not all of the parameters are actually changing that much. Some of them are changed by, like, zero.

Like, the majority of updates to the model is, like, zero. And only some very small updates to the model are seen. And so that's kind of the circuit idea where, like, the model already knows how to do whatever question you give it.

You just, most of the updates are, like, zero.

Are you saying which one is better? Is it better to aim for changing all the parameters versus a subset?

You can also do, like, LoRA, you know, parameter efficient fine-tuning. You can do other things. You don't have to fine-tune every single thing. But I think the majority of large language model labs, they just do everything. Otherwise, this, again, becomes an optimization problem.

You know, what do we, which layer do we select and stuff like that. So it gets more complicated. But yes, you could do parameter efficient fine-tuning. Actually, we're going to show a notebook for that. So you can actually do it on your own computer.

Yeah. It was, yes, one more question. One more. Yes.

I guess I want to add a slide note about the model steering. It's called model obliteration.

Oh, okay.

You can steer models towards a specific output that you want to find that. But I also, my question was, I wanted to, I guess, highlight what you said about the model can build capabilities outside of what it was pre-trained on, for example.

There was a paper from Google Research that talked about how the tail divergence was trained, for example, limits the capabilities that can be learned by the model. You can only have the previous priors of the model that you pre-trained.

I did some experimentation with small language models and found that to be true. For example, I found that when performing the best in the reasoning capabilities might support. So I want to, I wanted to understand what are the implications for, for example, working with smaller language models.

How important is this base model? And I guess, how can we work forward to kind of not make RL a meaningful distillation?

Good question. For the tail divergence term, do you mean, like, removing it? Would that make it better?

Oh, I was saying the tail divergence term. Well, I guess they removed it in the APO paper because they found it didn't have a good effect on the end results. But.

Because the whole point of the tail divergence term is to, like, not make it too stray away from the supervised model.

Reference papers,right?

Okay. Maybe I'm not kept up to date with research papers. But anyways, so your point was, like, if we remove the tail term, it will be better? It learns new capabilities?

Well, it was more, no, it was more, there was some, there was, like, a proof paper that described you can't build additional capabilities than what you have previously developed in the model. Almost to your point about amplification capabilities.

So I wanted to understand what are the implications of that and how can we prevent RL from making a kind of distillation, for example. Or how can we kind of, it's almost like the essence of your talk,right? Just understanding how can we build on top of that.

Do you mean, like, do you want to have more capabilities into the model?

Yeah. So how can, are there strategies you thought of, for example?

Strategies.

Hard to say. I'm assuming the large model labs will probably know strategies. We will show, like, I will show examples of how to, like, make RL better. Like, how to, like, reach higher reward faster. I'm not sure about new capabilities.

It's actually very hard. It's actually very, very, very, very hard to elicit new capabilities in the model. The question is, like, is this new or not new? I think that's a question. Like, is this actually part of the model or not part of the model?

And most research papers are, like, hand-wavy. They say, oh, most updates are sparse. You know, like, so most likely it's not, you know, new capabilities. But what happens if, you know, one year later, all of the model updates are, like, not sparse?

Is this considered new capability? I don't know. Like, you know, those are the questions. It's more like, I don't know if that answers your question. I probably didn't answer your question, but.

No worries.

Maybe, maybe the other parts of the talk maybe might answer some part. Yeah. Okay. I will keep going on. More questions later. Okay. So, like, the reward model,right, was actually a language model that, like, some sort of model, some neural network, some AI model that predicts the reward.

In RLVR, we delete this entirely and we just call it the reward function. So, like, the ground truth reward, you know, if it's correct, you plus one. If it's bad, it's just zero. Right? So you essentially delete another part.

GRPO essentially deletes another part. Right? So, like, you remove, remember, GRPO, you delete the value model, totally remove it. And then you delete the reward model and it's just a reward function.

And yes, as a reward model, you could use LLM as a judge. So you could ask a language model itself to say, is the answer good or bad? You could do that. You could do a regular expression check.

You know, like, is the formatting of the answer good or bad? Is the maths equation good or bad? You know, is the final output good or bad? You can do distance scoring, stuff like that. You can also execute the Python code and then you can see if it actually executed.

Right? So, like, is there, like, import errors or, like, format errors or, like, some sort of Python error? And you can use this as a reward. And so this blue box, the reward can be anything that you like.

It just needs to output a number. You know, minus one, plus one. I don't know. It just has to be a number. In fact, you can make a dumb reward. Just everything, it just does random. You know, plus one, 50% of the time, minus one, 50% of the time.

I don't know. And confusingly, a paper recently showed that actually random rewards works. So, like, yes, go ahead. You can try it. But also, why does someone say why? Probably read the paper. But why? To be honest, actually, I think the paper might be a bit, I don't actually believe there was, yes, there was an update showing that actually this was wrong.

That actually it's because the model, the benchmarks are incorrect. So when you say that you actually increase accuracy, like, from 20% to 50%, but actually the model itself was already 50%. They just didn't check the accuracy of the correct model before.

So there was a recent rebuttal to those types of papers. But, you know, interesting results. Yeah. I don't know.

Okay. So remember, in RL, the goal is you don't know the best action to take in the space,right? When you're doing Pac-Man, I don't know if going left,right, up, or down is the best. I don't know. But at the very, very, very end, you will either, you know, win or, like, you know, get some reward or you will die.

Yes. But the goal of RL is to maximize the best action you can ever take. Right? So, like, what is a better action than all of the other bad actions? So RL just tries to maximize the best, well, not the best action, the better action.

Normal pre-training, you already know what is the best answer. It's like, you already know what is the next word. Right? If you want to predict, you know, hello, my name is Daniel, you already know the next word is going to be Daniel.

Right? So you already know it. But RL, you don't know in advance what is the actual correct reward. So the only thing you can do in RL is to maximize the, you know, one of the better options. And so yes, okay, now more maths.

The goal is to maximize this equation. That's the goal of RL.

So what is this equation? The J is like the total gradient. Well, actually, it's more like the, it's more like we want to maximize this. It's not actually the total gradient. Okay. Maybe I miswrote that. Anyways, pretend I didn't write that.

We want to calculate the gradient with respect to the policy language model. And the action is given a state. And the R is a reward. If you want to write this down in, like, English, it's like, we want to take the derivative of the log probability of the action given the state times the reward.

Now, I don't know if you guys understand what that means, but I did, like, an example. Pac-Man case. Okay. So you are Pac-Man. The red is your enemy. You don't want to go there. Right? So, like, you definitely don't want to go to the red thing.

Right? But you want to eat the two gray dots. Right? That's, remember, you can only go up, down, left, orright. Right? You only have four actions. Remember, the action space is just up, down, left, orright. So if you do rewards, I just randomly made some rewards up.

If you go to the red thing, you will get minus 10 reward. Well, actually, it should be minus infinity. You die. But anyways, minus 10. If you eat the gray dots, you get plus one or plus one. And if you go up, it's just zero reward.

There's nothing there. Now, when you get this language model or, like, some sort of model, it has to tell you what is the next action. Right? It tells you what to do to the next action. For now, we will just assign every single action up, down, left, orright as one quarter probability.

Right? So, like, you go up 25% of the time, left 25% of the time, and so on. So these are your numbers. Right? This is the entire state. So the goal of RL is you want to do that red, going towards theright.

You want to do this, you want to go towards theright less. You want to do it much less. Right? So, like, you want to push the probability of the 0.25 of theright much less. And you want to go, like, you know, down and left much more.

Right? So you want to push the probabilities much more. And the top are not really that important. And so RL, essentially, your goal is to avoid doing the bad thing. And you want to do the good thing much more.

That's kind of RL. If you convert this into a table,right, you have the probability of the action given the state. Right? Remember, up, down, left, orright. We just assign 25% chance. Right? We just, just pretend 25% chance. The reward, which we can calculate.

Right? We calculated the reward. We just made some numbers up. Right? As 0, 1, 1, and minus 10. The probability times the reward, we get some numbers. Right? So, like, 0, 0.25, 0.25, minus 2.5. And then if you take the log of the probability times the reward, you get some number.

Right? So, like, 0, minus 0.6, minus 0.6, and 6.02. So from this table, does anyone know which row do we want to maximize? What is the goal? Like, what do we want to maximize? Which row?

Bottom.

You want to maximize the bottom row? What is the reward of the bottom row?

You want to minimize it.

Correct. You want to minimize the bottom row. Remember, the reward is minus 10. We do not want to maximize the last row because the last row is the worst. And so that means the 6.02, we want to actually decrease this number dramatically.

Right? That's way too large. We want to decrease it. The other rows we want to maximize. And so the goal is, okay, we just take the sum of all of that. Right? We take the sum of the four numbers and it's 4.8.

And so remember, okay, let's try. Right? So, like, by hand, by hand, we show, remember theright, remember all the probabilities are one quarter, 0.25. By hand, we show do the bad action even more. Right? We actually do the worst thing.

What happens to the, what happens? Right? So, like, the reward, the probability times the reward is now minus 4. It used to be minus 2.5. And so the reward, the log probabilities times the reward, the sum actually decreased.

Right? It decreased to 2.58. Before it was 4.81. Is 2.58 smaller or bigger than 4.81? Obviously smaller. So actually this is worse. You should not do this. Right? This is actually bad. Remember, the goal is to maximize, maximize this equation.

Maximize it. Right? Maximize. And so 4.81 is actually better. The original state is actually better than the 2.58. So this, the thing that we just did is worse. So do not do this. However, let's do theright thing less.

Right? Let's not go to theright and actually maximize the rest. You shall see that if you do the log probability times the reward, sum of them all, you will get 8.9, which is a larger number. And so the goal is to maximize this as much as possible.

You could say, wait, wait, we know the answer. Right? You should go towards theright. Just make this 100% probability. Let's just, you know, and you will have an infinite reward. Okay. Okay. Not infinite, but you will get maximum reward.

Right? Why don't we just do that?

But you should not do that because you actually force, if you do this, your model will be like, learn, oh, okay, let's just keep goingright. Let's keep goingright. And it just gets stuck. And it just becomes very bad for optimization.

So definitely don't do that. Now, there is someone who's talking about reinforce. We don't just multiply the reward. Right? Remember this equation we did. Where is it? The probability of the action given the state times the reward. We don't actually multiply the reward.

We should not do that. You actually multiply by something called the advantage. And what is the advantage? The advantage is the reward minus the average reward, the base reward. So you shouldn't actually just see you want to maximize reward.

You want to actually maximize the reward, but also looking at the average reward across the entire model. So it's called the baseline. And so this B, this baseline model is what is the value function, the value model. Remember, GRPO deletes the value model.

This was the value model. And this value model essentially estimates what is the average reward if we just see the current state. It does not take an, it does not look at, like, you know, what is the next, next step.

It does not look at, you know, what is the next action. It just takes a snapshot of what you're currently. So essentially, it looks at this, it looks at this, and just guesses what is the reward. Right? It doesn't, you're not supposed to give it the rewards.

You're not supposed to give it minus 10, plus 1, plus 1, or 0. It just looks at the current state and produces a number. And this number is called the average reward.

And so the goal is now we don't actually want to maximize this, you know, just the reward. We want to maximize the advantage as well. So, like, we multiply all this together and the goal is we want to maximize this new equation.

Does anyone have any questions? There's lots of baffs, but questions? Yeah. Yes.

So in terms of probability, how do you, is the large language model emitting an estimated probability or this is a known probability of all the possible states? But how do you get that in practice in the normal world?

So a large, so a model, a large language model predicts the next word. So, for example, you take the entire Wikipedia and then you're like chunking it into small little tokens and then the output is just what is the next word.

So, for example, my name is Daniel, but it could also be my name is Michael, my name is Bob, my name is whatever, whatever. Right? You have all of these probabilities for every single word in the entire language possible, like 128,000 words.

You assign a probability for every single one.

So probability is based on the token.

Yes, correct.

But there'll be, okay.

The trick of this for language models is you can utilize the probabilities directly. That's the trick. And so, like, that essentially makes everything easier.

Any other question? Yes.

What's kind of the intuition behind this? You're kind of almost like normalizing the gradient to kind of give it a little perspective.

Yes. Yes. Yes, correct. Correct.

Yes.

What about multimodal models?

Oh.

Do you have a world model that is not only based on text, but can be also based on visual?

Multimodal models. Do you mean like doing RL multimodal models? Oh, that is more harder. I would say you could, you could look at the Sudoku puzzle, just convert the text model into a vision, just cheat. I guess you could do that.

You could, like, say, oh, you know, I guess you could give it the Pac-Man, you know, give it the Pac-Man thing and tell the model, what should I do next? You could. I mean, vision, it's kind of the same thing, but it's more, does 03 do vision plus reinforcement learning?

I think it does. Yes, you could. I think for open source, I don't think I've seen, yeah, I don't think, yeah, I don't think I've seen open source models do that very well. It is still very hard. Yeah.

Any other questions? No? Okay.

Do you want to ask a question?

Did someone ask a question? Oh.

So what is the.

Yes. Oh, sorry.

So the average, what is the average? The average of all the actions.

Oh, what is the B? What is the base model?

Yeah. I understand average, but like, what is the average on reference model for what action?

So it's just like, your goal is to, your goal is you see this current state of the model, like whatever the environment currently looks like, and you just want to produce a number that approximates what is the total average reward.

Okay. I'll give you an example. Pretend you're playing chess or like Go or remember AlphaGo. You look at the board, the current state of the board, and just say, what is the probability of the white player winning? What is it?

You're not supposed to do any prediction. You just have to predict what is the probability of the white player winning by just looking at the board. That's kind of the average reward.

It's always low.

It's always low? Yes. Yes, correct. But remember, at the very, very, very end phases, like, you know, you might get higher reward, but that's the goal. You want to, you essentially want to predict what is the probability. Always, you know, for example, in chess, I'm sure there are like some steps you can take to make the reward higher.

The question is, when the model sees this, you need to essentially, you need to say, is this board better than the previous boards? And so this model you have to train as well. You have to train this model.

It needs to output a probability of it winning. That's for the chess example. Does that kind of make sense? Or no? Yes?

So there's a policy model and reference model. Do we use reference model for these?

The value, no, no, no. The value model is totally different. There's three models. There is a value model which predicts the average reward of the state. The reference model is just the model that you started with. And then the policy or the actual model that you're changing is the out, the final result of your model, like the actual chat model.

So there's actually three models.

So you would use policy model to get the B as well.

This one, the B.

Okay. Got it.

You will see the current state. You will see the current state. And then you will see, okay, what is the actual? I think you do use the policy. No, you, no, you just look at the current state. You look at the current state and then you output a probability of whether this chessboard is good or bad.

I see.

Of some, like 0.8% you're going to win. Okay. Yeah. Something like that. Yes?

So when there is maybe an advantage in DPO, they tell you to take like the ratio of the new policy to the old policy to sort of make sure that you're within.

Oh, yes, yes, yes. We'll talk about that. This is just a general, simpler formula. Yes, we'll talk about that.

Okay. Cool. I guess my question would be relevant then. Do you want me to ask it then or not?

Oh, you can ask. Yeah, go.

Okay. I guess because the policy is predicting the next token and we're, that's the probability we're trying to reduce or increase based on the reward.

Yes.

Are you taking the ratio per token on the advantage while you're estimating the advantage or are you taking the ratio across a turn?

That is a good question. And that is an active area of research because you could either normalize by all of tokens or the entire just one turn. Remains to be seen which one's better. It's actually still people talking about that.

Because if you keep adding it, sorry, keep taking the

multiplication of all the tokens across your context center, you probably get the super low numbers,right?

Correct. Yes. So generally speaking, normally people just assume this, assume this rollout is correct, assume this chain of thought is correct, and they just do the very end. But then you do have the multiply probability. Wait, I don't, yeah, you do have the multiply probabilities.

So there is a multiplication somewhere. You will get very small. Yes. But, you know, you'll get very small, but the numbers are relative. Right? So everything is very small, but then the smaller one, the bigger ones are still very small, but it's still better.

So they're all relative.

Any, was there one more? Yes?

Yeah. It's related to the reinforce slide.

Yes.

So are you referring to the classical reinforcement learning algorithm reinforce or is it like some new thing?

Oh, no, no. It's very old. Yes. Very, very old. Yes.

I wonder if you can like give some like advice on like how to think about this at a sort of framing or abstract level about like error propagation between like if you have a trained model which does the scoring or does the value function or whatever that itself is trained from data and has some like error margin and that, you know, you have some softmax function, for example, that like only one in a hundred times will produce the wrong thing, but like it has that probability.

Yes.

Like how do we think about the like development over time of these models and like to what extent that error propagation is something that you can observe and measure or like, you know, systematize and engineer around? Like just like I don't really understand like what the state of the mindset is in this sort of, in this processright now around that.

In my view, I think all of these formulas are just made up. And so like the goal is to maximize reward.

Yeah.

But the question is you need to like, you can't just maximize reward because otherwise you might make the model really silly. Like you might say, okay, what is two plus two? It just says four. Pretend your data set was just what is two plus two.

Right? So like you literally just cheat. What is two plus two? What is two plus two? What is two plus two? Just make the model just say four, four, four, four, four, four, four, just four forever.

Yeah.

Do you want this as a model? Definitely not. So like we want, it needs to learn, okay, if I give it the next question, what is eight plus eight? It should not just say four. Or what is two minus two?

It shouldn't say four. And so the goal of all these algorithms is to somehow force the models not to like overfit to your question. And so like these formulations are trying to do these things to like not overfit.

Yeah. Well, I'm thinking about like the chess example.

Okay.

You were saying, I think which scores the board and produces this like.

Number.

A number which is like, is this good or bad?

Yes.

Like sometimes these well-trained models have these theoretical novelties that, you know, where they say like, make this move and it's like not, the new state is not obviously good or whatever, but they somehow have like figured out to do this.

Yes.

And suppose that your training mechanism for the value function model, you know, hasn't picked up on something like that. In fact, there's like some error in the tendency of the value scoring model.

Okay.

Like its probability of producing like, you know, some sort of like perfect scoring of the board's position.

Yes.

Or like black versus white.

Yes.

You know, is not always exactlyright.

Yes. Always notright.

Up into your training process.

Yes.

In some sense.

Correct.

Yes. And I'm like, how do you think about that? Or like how do you, like what is the.

The value model, you have to train it together. So it's like a combination of the entire algorithm. So the value model predicts what is the probability, but you actually have to train this as well. And so that is actually the problem.

Some people, you could train this, you could train this separately. You know, you can like get all the chess possibilities and then output what is the final number. I think that's what some people do. You could train this in tandem with the model actually.

I think that's actually more harder. I don't know if that, you, so there is always error in the value model. Always. But you have to train this model as well. So you will reduce the error. But there's always error.

So I think there's like some numbers you can like force the value model to be like less, less prominent. Like don't forcibly utilize the value function. But in GRPO, we just get rid of the reward model, the value model anyways.

So totally gone. No, no. So you don't need to worry about that anymore.

Great.

Okay. I will keep going on. Let me just check time actually. Okay.

Okay. So remember the goal is the advantage, we want to maximize advantage, not reward anymore. Advantage is the reward minus the average reward or the base, the base reward. Right? If the advantage is less than zero, it means that it is worse than average.

If the advantage is more than zero, it means it is better than average. And so the goal is we want to do the action more if it's better than average on general.

Now to PPO. Right? So like, I don't know if you guys have seen the PPO formula. It is ugly, but this is the PPO formula. Right? So like it looks, it's more confusing because there's like a clip and then there's epsilon and blah, blah, blah, whatever.

But we could just strip everything away. It's just, it's just the probability of the action given the state times advantage. Right? We literally just discussed about this. Okay. Minus the log. Okay. The log's gone. But anyways, it's just that.

And then the rest is, the rest is trying to reduce overfitting.

And so remember, so essentially this, there is a thing called the division of the old model. And essentially it's the model that created the action. And the goal is we now want to maximize this likelihood ratio. We don't just want to maximize the probability of the model.

We don't want to maximize the probability of the action given the highest reward. We actually want to maximize the likelihood instead. But what is this likelihood? So I did some numbers. I just made some numbers up. So this is the pack.

So pretend the top, the numerator is 0.01 and the denominator is 0.01. 0.01 divided by 0.01 is one. If the top is 0.01 and the bottom is 0.99, remember these are all probabilities. You divide the top from the bottom, you'll get 0.01 again.

If the top is 0.99 and the bottom is 0.01, you'll get 99 and so on. Right? The last one's one. And so the goal is 0.01 divided by 0.99 is 0.01. This means that the action that you do is actually very likely.

Right? Because the bottom, the bottom thing is 0.99, but we actually don't like this. Right? Remember the top is 0.01. We do not like this. So the ratio is 0.01. And then the bottom is this, this action, the bottom, the denominator is 0.01.

It is actually not likely, but we actually like this because the top number is 0.99. And so when you multiply, when you do the division, you get 99. So this is actually good. And so actually we're not actually trying to maximize the probability.

We're actually trying to maximize the likelihood now.

And so the question is, why don't we just maximize the probability? Right? The first equation, why do we need to do the division thing? Because if we maximize just the top, you will have reward hacking. What is two plus two?

It might say to solve this question, we need to do blah, blah, blah, blah, blah, blah, blah, blah, and then suddenly it says, hello, hello, hello, hello, hello, hello, hello, hello. And then it says four. Is this good?

I don't think so it's very good. We don't want it to say hello, hello, hello, something or like some weird trace in the reasoning model. It does something weird. We don't want this to happen. And so actually this, this, you know, hello, hello, hello, hello is actually very not likely.

And so the goal of the division is to reduce these issues.

The epsilon part is called the trust region. Essentially, we don't want to make, we don't want to do large steps for PPO. Right? We don't want to do large steps. And the trick is we want to restrict them.

Right? So like you don't want to overfit the model. So now we restrict the model. And so epsilon could be like 0.2, 0.1, 0.3. And the one minus epsilon is 0.8. One plus epsilon is 1.2. And the trick is we just want to not move the direction of the gradient that much.

Right? We don't trust the model that much. We don't trust the algorithm that much. So we want to constrain it.

And then also the PPO, there's also a KL term. There's another term. And essentially what this does is we want the model to be as close to the supervised fine-tuned model as much as possible. We want it to be, we don't want it to go so far away from the base model or the supervised fine-tuning model.

So essentially if it deviates too much, we want to tax the, we want to tax it. And so this beta is like 0.05 and the KL divergence is the dist, okay, it's not a distance. It's like the distance between the current model and the pre-trained model.

And essentially we want to also shove this into the equation. So you can see like PPOs has many moving parts. It's, who cares about the equation? It's not that complicated. The point is all of these extra add-ons are just to reduce overfitting and not to make the model like randomly go to some weird state that like, you know, overfits to like your questions.

And so the trick of PPO is they just added all these terms in to make training more stable. And so the final equation is like this, this again. Hopefully you will, to be honest, no one even cares about the formula.

It's not that important. But I just tried to like break it down into pieces. And the goal, remember, the goal is to maximize this equation. Right? We want to maximize it. And normally I just like to think about this one.

Right? You just need to learn this one. Right? You want to maximize the probability. So it's just this equation. Remember we did the table. Just this is enough. You don't need to learn the rest of the formulas. It's not very interesting.

Yeah. Any questions? Yes.

I have a question. I know like the reason why PPO was preferred over like reinforcement learning is because of the stability.

Yes. Correct.

Previous one. Could you go more in depth like what causes this instability and why this minimizes the instability?

So the biggest problem is pretend you are like, pretend you just started RL. Like you, you have the pre, you have like the base model, you have like a supervised fine-tuned model, and then you do RL. The gradient updates at the very beginning are going to be gigantic.

Right? You're like, what is two plus two? It says four. But if it says five, you want to like penalize it dramatically. And so the problem is you don't actually want to do large steps. And so the goal is you want to constrain it.

And so the constraint factor is like, you know, if the, if the, if the num, if the gradient update is extremely large, you just want to like constrain all the numbers. If that makes sense. The goal is just to constrain the update, not to make it too large.

And then, so that's the clipping. What about the ratio?

Oh, the, the ratio, it's the KR divergence. Oh, sorry, not the KR, the likelihood. To be honest, I think I need to do more research. I would ask Gemini exactly what it is. That's my answer. I'm probably not the best person to answer every single question.

Yes. Any other questions? Yes.

So for the dominator, you know, so that this problem coming from like a reference model, is that the quotient is that polysonal generated between that you see on the down there, yeah, the old one.

Yes.

So is that a, for the correct answer probably?

It's the model that actually created the action. And so the top one is all of the numbers that actually. Like how do I explain this? The bottom one is the model that created the action. So for example, you, the model says you want to go up, down, left, orright.

Is that a, is that a correct action or the action?

Oh, it just created the action. Like it could be anything.

Okay.

It could be the, so it's the, it's the maximum, it's whatever action the model says currently.

I see.

It might be wrong, it might be good, it might be bad. It's just any action.

Okay.

Any other questions? Okay.

One question.

Yes.

Has this been tried in latent space instead of the probability

or for different probability? Have people tried this in like thinking in latent space

or reinforcement PPO in latent space?

I don't think so I can answer that question. I don't know. That's why I don't know. Maybe research papers show it. I'm not sure. Eddie, other. Okay. I will. Okay. So GRPO, the trick from PPO is we remember, remove the value model.

We get rid of it entirely. We do not want to estimate the average reward. It's totally removed. And the reward model is now removed as well for a reward function.

So we get, yeah, we remove it. Right? Remember the value model is removed. B is a reward value model. We get rid of it entirely. But what do we replace? So the trick of GRPO is we do rollout or inference sampling.

We get the answer. What is two plus two? You literally make four inferences. You just literally call the model four times. It could say the answer is zero, the answer is one, the answer is two, or the answer is four.

You can do, you do like, you just literally call the model four times and you take the reward. Right? Zero, the, what is two plus two? The correct answer is four. So you want the last number to be one, but the rest is all zero.

And the trick is you literally just take the statistics of your current rollout. You take the statistics of all of this. You literally take the reward minus the mean divided by the standard deviation. You get the Z score.

And this is your, this is your base model. This is your value model. Right? This is, there's no more value model anymore. It's just a number. And so I did this on a table as well. Right? What is two plus two?

If you think it's zero, remember the prediction could be zero, one, two, or four, and your reward could be zero, zero, zero, or one. And if you take the mean or the average of all the rewards, you get 0.375.

If you take the standard deviation, you take 0.43301. And then you do the reward minus the mean divided by the standard deviation. You get some numbers. Remember the number four is correct. That is why, you know, reward minus the mean divided by the standard deviation is 1.44.

It's the largest number. And so that is why we need to like max, we need to essentially maximize that good answer and we want to reduce the bad answers.

But why is it called group relative in GRPO? Because it's not just one question. It's many questions. It could be what is two plus two? What is four plus four? Okay. Well, my graphs are all the same. My plots are all the same.

But anyways, imagine there's like four different tables. What is two plus two? What is four plus four? How do I create this Python function? You know, whatever. And there'll be four tables. And so the goal, group relative just means you, for each question, we take the statistics within each group.

For example, what is two plus two? You create four, you literally call the model four times and you get some, you know, answers. What is four plus four? You call it four times. Right? Create Python code, you call it four times.

Yes, there are other factors of GR. So essentially we already explained what GRPO is. Right? Everything you need to know about GRPO, we already explained in the total mathematical formula. It looks kind of like this.

There's some rearrangement. For example, the minus beta, the KR divergence is just taken out of the reward function. That's the only other difference. Hopefully it makes more sense about the parts of the GRPO formula and stuff like that.

It's actually not that complicated to understand. The majority is just trying to reduce overfitting. Right? That's the whole goal. Right? Minus beta times the KR divergence is to reduce overfitting. One minus epsilon, one plus epsilon is to reduce overfitting.

The division, reduce overfitting. Everything's reducing overfitting. Right? So that's all of machine learning and AI. It's just to make the training more stable and to reduce overfitting. I would highly suggest there is, these are the two things that I really highly suggest.

Nathan Lambert's Policy Gradients book. It's online though. Very, very, very, very helpful. And Yannick's video on GRPO and stuff. Very, very, very helpful as well. Yeah. And then now I will go into a collab demonstration of GRPO. Before that, like does anyone have any questions?

Let me just check time. Questions? Yes.

So looking at this formula, the GRPO formula, it looks like everything there is about exploitation and not so much exploration. Yeah, you can say that the exploration between the new model and the old model, but how about if the model can look in a totally different part of the search space?

Because maybe the answer is there or a better answer is there. Because if it looks to me that all of, all of this is make more salient whatever information is in the base model in the end.

Yes.

That can mean memorization. So you make more salient theright memorization. In this case, the question of the two plus two.

To answer your question another way, I think it's actually because GRPO itself is the problem. Remember the goal of all these algorithms is to force the model not to detract too much from the original model. Right? With this like minus beta KR divergence, you know, one minus epsilon, all of this is trying to make the model not go towards too far away from the original model.

And I think that's the problem because you're essentially forcing the model not to go too far. And so maybe there might be some new algorithm, I don't know, something, some other formulation which, you know, you want to go very far away.

You could do that. I don't know if there are any, I don't know if there's any research papers about that. I don't know. But you could, yes, you could do that.

Thank you.

Yes. Any other questions? Yeah. Yeah. Yes.

You probably heard that Yann LeCun is saying that like don't believe in LLMs,right?

Okay. Yeah.

And that's why he's pushing towards the JEPA.

Yes. Yes. Yes.

For the energy-based models.

Yes.

So what do you think about that?

What do I think about it? I can't really comment. I mean, he definitely, you should listen to what he says.

But you don't see anything like movements in open source in that regard?

I don't think so. I think open source, we kind of got captivated by RL, GRPO. I don't think so open source people are doing whatever he's talking about, unfortunately. I think he needs to talk about it more. Yeah, JEPA.

Yeah. And energy-based models. Unfortunately, I don't think so open source. Yeah. Maybe we should talk about it more, but yeah, I don't think so. Yeah. Yeah. Yes.

Yeah. So for group relative, so GRPO,

so it's removed the value function.

Yes. Yes. You got rid of it.

And then the, so for the rewards, the sampling of these rewards, I always thought it was very interesting. Like because you, with the LLMs, you can actually create different types of samples.

Yes. Correct.

My question is like, are there strategies related to understanding whether you want to have a larger variance or you want to have a larger group and make sure that this mean is more related to like the value function, the true value function of like that LLM and that, let's just say at that state?

Because I'm not sure, like, because it seems like when you constrain the group to be very, very small, you're going to almost certainly go through some kind of bias that the LLM has in front of it. Unless you have like maybe a larger enough sample that might, one of these larger traits would be way, way more valuable than let's say a trait that you never get if you have a very small batch size.

So this is more about an optimization question. So in theory, you should make the, you should make, for example, I just selected four. Right? What is two plus two? Create four examples. You should do, you know, as many as you like.

You know, 3,000, whatever number you like. You should do as much as possible. But remember, AI is about optimization. This is going to take forever. You know, it's all about efficiency. So probably don't do as many as you like.

But you should, in the limit, you should do that. But, you know, everyone can't just wait there waiting for the computer to spin. So yes, you should do as much as you like. That's it. But yes, also for like recommendations, when you do inference sampling, you should set temperature as like, you know, 1.2, 1.5, set min P to be 0.1, you know, something like that.

If you set temperature to be zero, you'll have the same answer every single time. So definitely don't do that. But you should have high temperature numbers to make the model, you know, produce new output as much as possible.

Maximize variability?

Yes.

Or distribution?

You should try your best to maximize variability. Your outputs should not be all the same. If it's all the same, I don't think so the model's going to learn. So you should make it as different as possible. So that's why you should set temperature to be 1.2, 1.5, whatever, some large number.

Don't do too large though.

Any other questions? Yes.

Yeah. I'm sure though, like the situation where like all of their rewards are none or zero, like the model doesn't have any way of learning anything.

Yes. Correct.

So you're starting like, you mentioned like, yeah, you can kind of pivot them all away from their behavior. Maybe, yeah, you were going to talk about KL later, but like, yeah, if the model doesn't have the capability to answer the question.

Yes. Yes.

Like all the first steps are only, all the rewards are basically giving no signal at all of the underlying thing you're trying to train. Like how do you deal with that? I have faced that a lot of times with GRPO, like moving to a larger-based model helps sometimes, but if the model doesn't have any of that knowledge of the training data, like there's no way to learn that.

Yes, that's a good question. So essentially you're saying the model, if the model starts off with like no reward, like every single update is like zero, zero, zero, zero, zero, zero, zero, zero, it's not going to do anything.

Yes, that happens all the time. But by chance, just by chance, you know, you have like some small, little, little, little probability, just by chance, you will have some reward. So that's the trick. You will see this after 10,000 inferences, what is two plus two?

Suddenly the model says four. Suddenly. Okay. Just suddenly. Just by random probability. Let's make this small. That's all. That's all of GRPO.

Yeah. Like this is addition, like it's a super simple task. If it was like a proof of a mathematical concept, it may never come with theright solution.

Yes. May never. But remember, you're not doing this one question. You're also shoving this together with other questions. What is two plus two? What is four plus four? What is, you know, derive the derivative of blah? Do this Python function.

This step is very large. You essentially shove this all together. And the trick is, in general, it works. In general. Maybe by bad luck, it might not work, but I feel like the bad luck won't last forever because remember, you're changing the samples.

Right? So like the question, what is two plus two? You're changing it. The next phase will be some other question. And so the trick is, just by chance, you will have a good reward. Just by chance. And we just force that to be more.

Does that kind of make sense? So it's all, to be honest, it's all luck. Yes. It's all luck. We're just guessing. Oh, you know, we're praying that there's going to be some positive reward somewhere in the model. There will be negative reward.

Right? So like if your model is really, really bad, you can do negative reward. And so you just don't want to do the negative one. You just want to do the negative one less. And by miraculous probability, you know, just relying on probabilities, you will get a good reward somewhere.

Just by chance. Does that kind of make sense? I mean, all of the large model labs are just literally relying on the fact that that's what they're doing. They're just guessing. We're just praying for the GPUs to work and then suddenly the reward comes out.

I'm being serious. That's exactly what they do. They're just waiting for the algorithm to work. And if suddenly, oh, okay. That's why people do random seeds as well. So for example, the initialization of the model might not be good.

So you just kill the training run. You do like 500 training runs. Oh no, 499 of them are like zero reward. Oh, just kill them all. Don't release them.

To extend into that topic, like maybe it's worth it. Like I have seen on my trainings that like GRPO, most of them stays flat forever.

Yes. Very common.

And spending some more time on doing as small as 50 steps so the model may get like 5% of those rewards.

Yes. That's the core. Yes. I was going to show you guys that.

Take off.

Exactly. So like you could force the model to answer some question. Like for example, you ask a question, what is two plus two? It's very easy. It's four. You just force it to learn, oh, okay, it should be four first.

And then you do other steps. That is actually why remember, okay, I have to go back to all the slides. I don't remember. Okay. Where is it? How do I, like how exit? It's the same as this problem.

Where is it? This one. Right? Someone asked about why didn't you just start from the blue, you know, the pre-trained model to go to the green one? It's the same thing. Essentially, the trick is we want to do some supervised fine-tuning to make it know some instructions.

So it knows something and then you want to go to the reinforcement learning phase. But if you want to start from nothing, like just a pre-training phase, that's the hard part. Right? Your reward might be zero, zero, zero, zero, zero, zero, zero, like zero forever.

And then suddenly one, you know, suddenly.

Oh, but I start from ping.

Oh, if you start from ping. Okay. That's just unlucky then. I think like if you see zero rewards, most likely either one, your reward function's not that good. Two, yes, doing priming or like, you know, make the model learn a little bit about your data actually does work, does help.

So there are tricks to make it work. But generally, I would just say it's bad luck. Just bad luck. And unfortunately, you can't do anything. It's not your fault. It's, yeah, just unfortunate. Yeah. Yes.

How should we think about like when or like what to expect from GRPO? Like is it going to be just that this is the way open source catches up with closed-source models? Or is it something that is another tool for like your average or like your competent ML engineer to be able to like specialize a model for a task?

Like I just don't, yeah, is there consensus on like where, what this is going to bring us to? Smart people like you going to give us a really good open source model? Or is it that, you know, we should think about this as a new tool for specialization?

The algorithm is not special. The hard part is actually the reward functions itself and the data that you're going to shove into the model. That's the hard part. So like I think there is a misconception like, you know, the algorithm is important.

No, it's not. It's useless. Who cares about the algorithm? You can literally just use, you know, the general, the function which I gave. You can just use any algorithm that you like. But the problem is actually the reward function itself.

I gave you some examples, you know, like what is two plus two? The answer is four. Yes. You can do distance-based. But that's just one example. Can you like, can someone make a function, can someone make a reward function for like trading stocks?

That's, you do that, do that. Right? And then there you have like a model for trading. Go ahead.

So you think it's going to be more that because people are able to create reward functions like a little bit more easy, like similar to like you could create a prompt. It's an easier thing for most people to like iterate on.

It's actually quite hard.

Coming up, but it's easier than coming up with the algorithm itself or whatever.

Actually, I think collecting, so in the olden days, large model labs will ask like, you know, large data providers like Scale or whatever to create data. Like what is two plus two? You literally have someone sit there and write, okay, the answer is four.

But then you also have to do the chain of thought. Like, oh, I think the answer is four because of blah, blah, blah, blah, blah, blah, blah. Or like, you know, this is my working out. You literally have to ask someone sitting there to make the data.

The trick is no more. You don't need the data labeling step anymore. It's totally gone. You have the answer and you have the question. The middle step is totally removed. But you still need to make the reward function.

You need to say, is the answer four good or bad? For maths, it's very easy. For code, for code, it's somewhat easy. You know, you can check, oh, did you import the correct function? Did you import the library?

Did your code execute, you know, some other reward functions? But it's still hard to verify if your actual function's correct. For example, let's say your question, let's say the task was create the Flappy Bird game. How do you actually know that the output's good?

How do you actually know? We don't. You could, again, ask a human to verify. Oh, you know, let's test the Flappy Bird game and then give it a good reward, a bad reward. Or the trick is, did the game actually run?

If it ran, plus one. Did you see the word Flappy Bird in the, you know, Flappy Bird inside of the functions? If yes, plus one. Did you see the, you know, the image of the Flappy Bird sprite be used?

If yes, plus one. Yeah, something like that. So like, you don't, it's still, I would say the hardest part is writing the reward functions. And for open source specifically, if, you know, the whole open source community starts writing reward functions, we can probably beat 0301.

Like, you know, okay, plus compute. You still need compute. That's the problem. You still need compute. But if you write good reward functions, you'll probably catch up in no time.

So the end state here is that you want an open version of the closed model. It's not, I guess my original question is like, is this a thing that people are going to use like prompts to have specialized models separately?

Or is it more that we want one or many good open models?

It depends. Okay. That is a good question. It depends on which school of thought you're in. If you're in the thought that large language models already have the capability and you're just trying to accentuate it, then there'll be just one model.

Yes. But this model can only learn some facts because otherwise you're like overfitting and like it's not. But if you're in the second camp that the model actually learns something new, wait, did I say itright? I think it's the opposite way around.

The first one is you have many models. I saw, sorry, I said it wrong. The first one is you have many models because the model doesn't actually learn that much. But if the second one is the model actually learns everything, then you have this one gigantic model.

I think OpenAI probably subscribes to that point. You know, like most large model labs think that actually RL can get you to AGI. Right? It will know everything about everything. Any single question you ask, it already knows. And so like that's, I think that's where they're trying to go for.

For open source, it's more harder. I think the open source community consensus for now, for now, is the model, it already knows your questions. You're just trying to accentuate it. And by doing reward functions, you're trying to wait the circuits more.

You're trying to wait the model to know how to do these equations and stuff like that. So I think like the goal of open source is, you know, if the entire community comes up with good reward functions, write them all, then the problem is we need compute.

That's the second problem. Right? If you shove both of them together, you will get O10 or something. I don't know. Right? Imagine if every single person writes a reward function once per day. Okay. That's probably too hard. Once per day, we'll have like, you know, 7 billion reward functions more than OpenAI can ever come up with and they will defeat OpenAI.

But, you know, you need the compute part. That's the only problem.

Okay. Any other questions? Yes.

Oh, sorry. I'm curious when you said that like, yeah, you know GRPO is going to get traces that may get correct or those get rewarded.

Yes.

How do you feel about like saving those traces and just doing SFT on those traces?

Very smart. That's what, yes, yes, yes. I don't know if large model labs do that. You could do that. Yes.

Because I feel like, yeah, we're just mining for those.

Good examples.

That are correct.

The only problem I would say is pretend the question was, what is two plus two? Right? And then the model says, okay, pretend the model just says this. Okay. It says, oh, let me work out what is two plus two.

I think it is, oh, you know, the number two means two apples and I want to add two more apples. I think the answer might be three. Hmm. But let me rethink about it. Wait a second. It's like four.

Is that a, should you fine-tune on that? I mean, you could, but maybe it's like cheating. Maybe it just says four by chance. Maybe like, okay, I'll give you an example. Let's say that by chance, okay, the question was, what is two plus two?

It says gibberish. Like, I like to go to Paris for fun or whatever. I don't know. I like to go to the, you know, this event. Blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah. And then suddenly it says four.

Just by chance. Remember, we're still rewarding this. We're literally rewarding this as good. But this is not good.

Yeah, but that's how we reward. That's like.

No, so we reward it at the very end. Remember, we see the number four. It is good. The question was, what is two plus two? The model can generate anything it likes. It could.

You can reward that part too.

You could, but that gets harder. So the trick is people don't actually reward the steps in between. They just do the final step because otherwise it gets too complicated. Right? So like, what is intermediate steps reward? It's way too complicated.

So what you do is you just reward the final step. So if you see the number four, it's good. But we don't know how we got there. So you should, so yes, you could maybe at the very, very, very end step of RL, you can then use some data to do fine-tuning.

Yes, you could. But I think like in general, it's because we don't know what the process is in between.

You say we don't train on that thought of random.

Oh, no. We don't train on the thought. Yes, we don't. But remember.

Is that the problem with all traits there?

We don't train on the intermediate step in between. You don't have to.

No, you don't have to.

You could. You could. But remember, we don't know if the traces are good or bad. We don't know. So you can't just take this trace and then do supervised fine-tuning because pretend the answer was four is good, but we don't know the intermediate steps unless if you read the data.

Right? You could ask, you know, some human labelist to like, oh, you know, please verify if this trace is good. You could, but then that kind of defeats the whole purpose of RL. So like you don't, you don't want to do this.

Does that kind of make sense or not really? Okay. Yes.

Can we see the demo?

The what? Sorry.

The demo.

Oh, yeah, yeah, yeah. Yes, we will do that. Yes, yes. Yes.

If we have multiple categories of reward functions, like one for math, one for Python.

Yes.

What's the normalization among the rewards that are like best practices?

What do you mean by normalization amongst rewards?

If we have, you know, for four, you know, for the math one, two plus two, four gets a one, five gets a 0.5, A gets zero, then you have one for Python that might just be a does this compile?

It's a binary reward function.

Yes.

If we're running this in one big compiler.

Yes. One batch.

How do we normalize the fact that.

Good question.

We have two different types of rewards.

Correct. It could be like minus 10.

One wrote a plus 100 to negative infinity. The other person wrote a binary.

Very good question. That's your choice. Unfortunately, that's the problem of RL. It's all about human choice. Like you will have to decide, you know, is the Python one more important than your, than your maths? What is two plus two?

Then you can wait it more. For example, your make two plus two, the reward as minus one and one, and the Python function as like 1000 and zero. Right? You have to decide on the waiting functions. That is, that is your choice.

Unfortunately, it is a, it's kind of like an art. You could like dumbly, you could just do everything as the same scale. I think that's what most large, I think that's what the large model labs probably do. It's like everything, all the reward functions have the same scale, you know, plus one, minus one, plus one, minus one, not plus 10 and the minus 1000.

So it's up to you.

Okay.

Yes.

And what about stuff like, you know, is this a good summary? Is this a poor summary? How are we, how are we creating reward functions?

That is the question. So like now you want to like analyze, okay, that's where the LLM as a judge comes in. So like there is a school of thought that you can use a language model itself to, to make a number.

You can ask ChatGPT, is this a good summary or is this a bad summary? Please give me a score from minus 10 to 10. Ask ChatGPT. You could do that. That's called the LLM as a judge thing. There is a paper which shows that you can do this for some time, but then it breaks down.

So you can't just keep calling the language model, you know, like it's kind of like cheating if I would say like you're trying to call ChatGPT to train ChatGPT. Like it will work for some time, but then it will break down.

There was a paper, I need to find the paper, but the paper showed that if you keep doing this, your actual reward actually goes backwards. So like you will get more, more, more reward. And then suddenly it just, I don't know, by bad luck, again, it's always about bad luck, the reward just goes back.

So yes, I mean serious, like all of AI is about bad luck and good luck. And, you know, optimization, trying to do efficiency. That's what everyone, yeah, unfortunately. So yes, you can use LLM as a judge. Is that kind of.

If I use LLM as a judge, doesn't it just end up being a teacher and student model or?

Yes. That's why like, that's why sometimes essentially the problem is I need to find the paper. If you keep doing this, it will actually do bad. I mean, intuitively it kind of works, but then at some point there is actually another way.

You could ask a language model to generate reward functions. That is actually another school of thought. You can actually ask a language model to generate 10, 7 billion reward functions. But the question is, is the reward functions good or bad?

I don't know. So like now you need to like rely on the fact that the models are good or bad. You could then ask another language model to verify the reward functions. So yes, you could do this. Maybe that's what OpenAI is doing.

I don't know. Maybe OpenAI's goal this whole time is like, oh, let's generate all these reward functions, verify each of them, and then shove it into the function. Let's see what happens. Maybe that's what they're doing. I don't know.

But yes, it is a student teacher. Yes.

What's your opinion on how to make reward models at scale efficient for the open source realm? Is it a bunch of distributed experts? Is it inverse learned? Is it, is it something else?

Do you mean like how to make reward functions more efficient in general?

No, I mean scalable in the sense of going over many different domains, prioritizing past just coding and math.

Yes. The majority of reward functions currently, that's why it's called verifiable rewards, is like maths and coding. To be honest, I think coding is also hard. I don't know why people lump. So coding, you can't actually verify technically it's correct.

You can just say it ran or the output is most likely correct,right, for some functions. But for example, the Flappy Bird game, tell it to create a Flappy Bird game. How do you actually verify if it even is the Flappy Bird game?

I don't know. But you could,right? That's the whole point of the LLM as a judge. You could take the output of the Flappy Bird game, ask the language model, does this look like the Flappy Bird game? And if it's yes, okay, plus one.

If no, minus one. You could do that. But you can only go so far. If does that kind of.

I just wonder what your opinion is to scale beyond that.

Oh.

Hard to say. I think most, I think large model labs, they're currently just trying to use their own model to, to, to like literally reward, reward it. Like as I like describe, you know, ask it, oh, does this look like the Flappy Bird game?

If yes, plus one. If no, minus one. And I think maybe that's what I think large model labs, their view is if you keep doing this, you'll get to AGI. That's their view. I mean, if you think about it, you could maybe, but then I always go fall back to, oh, but you might be bad luck.

It's not going to work. So like I think in general, I think bad luck will just, it won't work. You will only get so far and then suddenly it just doesn't work. Does that, okay. Any other? Yes.

How do you think about the scope of the task that you're instilling into the model? For example, you can, you can train for the math, you can train for the coding, you can train for the planning. Do you think if there are the tasks are diverse, does it work well or should we specify the specific task?

That is your choice again. So like if you want to specify, for example, you just want to make a legal bot, you give him some sort of court case, and if it's like, you know, the plaintiff wins or the defendant wins, I don't know, you could just do law.

Yes, you could. You could do that. But in my view, you should combine it with other sources. You should combine it with some maths. You should combine it with some programming. Because the point is you don't want the model just to know.

Like you just, you don't want the model just to like overfit just to just law. Maybe maths might be helpful just by, you know, by chance again. Maybe, you know, maybe coding might be helpful. Probably not. But like, you know, in general.

So like you should combine other source, other domains together. I feel like, you know, all the large model labs, their goal is to do every single domain possible,right? Like mine every single reward function in the whole world, make all the reward functions, shove it into the model and just learns.

So like, yes, you should do more domains. If, yeah.

Yes, another. Yeah.

Yeah. I hope the board question is more of a practical point of view. So, so what, what is the latest research in. Should we fine-tune? Let's say for inference we want a smaller model, should we fine-tune the smaller model first or go with teacher instilling to smaller model?

Is that better for this particular thing, you know, using GRPO and thinking those things?

So the notebook I will share will showcase you should probably do some supervised fine-tuning fast. It's called the priming stage. Otherwise, you'll remember the plot over here, the, the, the, this one,right? You don't want to be in the situation where like you're starting from like some bad, you know, pretrained state and you're trying to go to the RL stage.

Very not efficient. But remember, AI is all about efficiency. You don't want to do this step. So we do have to do some priming, you know, the SFT stage and the other stages. If that's your question or.

Should we use bigger teacher model first?

Oh, if you want to, yes. The bigger the model, the better. Yes.

I mean, it's extra cost. So this one, if you, if you can just do, if you just train the smaller model, is it still efficient?

That's the trick. So essentially the research papers show that small models actually do work, confusingly enough, because essentially these small models, it just does longer thinking. It does longer reasoning traces if the model is smaller. And if it's a larger model, maybe the reasoning traces are like smaller in general.

So like I, I feel like the small models actually do work. They do break down though. If you want to do like very complicated reasoning traces, then maybe the small models might not work because, you know, there is only 7 billion parameters.

There's not that much space you can move. And so the large models, you just have more space to move around. And so that's why large models are better. I don't know if that maybe kind of, I don't know if that answers your question, but.

Oh, yeah, yeah. That makes sense. Another question I often pumped up that is like why we, if you want to fine-tune for our use case, like we want task, can we use the, we can also use the one, the distilled DeepSeek model, you know, which is already fine-tuned on the task.

So we can, instead of working on the base model, we can take that and then fine-tune.

Yes. Correct. Exactly. Yes. That's what you should do. Yes. You can take a distilled model, like an already reasoning model, and then further fine-tune it. Yes, you could. I would say it's a bit more complicated because you could do that, but remember the reasoning model itself is already a reasoning model and you're trying to fine-tune it to become other, like you're trying to do some other domain.

It might be easier. It might be harder. It's all about luck again. I don't know. So you have to try. It's all trial and error. Yeah. Yeah. Yes.

Two questions for you. One, like, is this all things pretty empirical? So it's again, it's like trying and seeing what works and what doesn't and seeing what sticks.

Yes. Correct.

Okay. And then the other side, how are you keeping up with all the papers and all the content that's being put out? Like I'm sure it's a lot. How are you learning like some of the principles that you're using to follow like the latest work?

To be honest, I don't, you don't need to follow. That's my view. Don't try to follow the latest research because sometimes I may like in the next day is like rebuttal of the previous paper and then the next paper says, oh, it's a rebuttal of the rebuttal.

I don't know. So I would not try to keep too much up to date with the latest research. I think the field has kind of matured and it is mostly stable now. You might have like some algorithm increasing accuracy by 1% or 2% or some efficiency.

Remember, all of the papers are about efficiency. It's always about efficiency, making the training more stable, reducing overfitting. It's always these similar, similar papers. So I would say don't, you can keep up to date with papers. Twitter is very good as a resource.

Sometimes I tweet about papers, although I don't suggest the Nathan Lambert paper, the Nathan Lambert book is very good. He keeps updating it. Where is it? Where did I put it? This one, the RIHF book. That is very good.

So definitely read that. He updates it all the time. And so maybe follow Nathan Lambert. He's actually very good Twitter and like the latest research. So he's very useful. In general, there's a lot of noise in the RL space as well.

You don't know if the research is good or bad. Like, you know, rebuttals on top of rebuttals. So I would suggest people just to like try, it's just trial and error,right? Try to see if your reward function is good or bad.

You know, is the loss not, you know, is the reward just zero, zero, zero, zero, zero? Like unfortunately something's wrong or you just, it's bad luck. Try again. So it's just empirical. Yes.

Are you going to put these slides up?

Sorry?

Are you going to put these slides?

Oh, yeah, yeah, yeah, yeah, yeah. Yes, these slides should be up. I was supposed to make a Bitly link. I'll probably do that later, but I will share the slides. Yes.

Maybe post it on Slack in the engineer slide.

Oh, yeah. Okay. I'll do that then. Okay. Okay. Any, yes.

Yeah. I have two questions. So with the idea that the capabilities already exist and during reinforcement learning, you're not adding new capabilities, when you're talking about the value model and reward model, is that something that you are creating and then discarding after the training?

In the old PPO sense, you are, the value model is a new model. The reward model is a new model. Yes. But remember in GRPO, we delete the value model. The value model is totally gone. We, we create the value from just statistics from the distribution.

We essentially just create, you know, four, what is two plus two? Just create four examples, four trials, and then find the mean, find the standard deviation, and then that is your, that is your value model. It's not even a model anymore.

And then the reward model is no more as well. It's just reward functions. And that is why we call it reinforcement learning with verifiable rewards. It's not normal RL anymore. It's like you replace a reward model as well.

If does that kind of make sense?

That does make sense.

Okay.

My follow-up question is, so is it sound intuition that then there are some circuits within the existing model that have this idea of reward? And so we're kind of weighting those circuits to approximate the reward that we are getting during training?

Yes. So the, again, there are two schools of thought. The first one is like the question, what is two plus two? Somewhere in the model, somewhere, I don't know where,right? There's high dimension of 1.4 trillion parameter space somewhere.

It knows to calculate it as four,right? There is some sort of circuit inside the model. And then the goal of RL is just, just to maximize the circuit somehow,right? Via these formulas and stuff. You're just trying to maximize it.

But that's one school of thought. The other school of thought is, oh, you know, RL is actually learning something. You know, like it's actually learning how to do two plus two is equal to four. And it's not actually in the model.

Thank you.

Okay. Any other, yeah. Yes.

When you say capabilities of, of a model that already has them inside.

Yes.

You mean knowing actually the answer to a question or knowing how to reason to get to a question?

That is a good question. Maybe both. I, I think it depends. It probably knows how to do the re. It might, for example, I can try the example. You get all of the entire world's data,right? Like, you know, 30 trillion tokens, get all of the data and you just make a question that is not part of the data,right?

You, you could do that,right? What is 10 billion, you know, some random number times some random, you can make a maths equation which is not in the data. But somehow the model has learned to do multiplication, has learned to do addition somewhere.

And so maybe this circuit for addition, for multiplication, you know, for whatever, some, you know, many, many, many circuits of these like functions, we just want to accentuate them all. And that is what kind of RL is trying to do.

And yes, there's a reasoning circuit. So like somehow the model also learns how to do reasoning. And so we also want to make that more important. And so, you know, two plus two is very important. Addition is very important.

Multiplication is very important. And so on. We're just trying to like make all of these circuits more prevalent. But that's only one half of the AI community,right? That's only one half thinks like that. The other half is like, oh, but the model is actually learning.

You know, we're actually training the model to learn and the base model actually doesn't know how to do reasoning. Does that kind of.

Okay.

Okay. Yes. Okay. Hopefully. Okay. Any other questions? Yes.

So ask question that support RL framework, what framework do you use? Like I see in notebook you use a Hugging Face one, but there's also popular one world.

Yes. There is VAL, there is TRL. Unsloth, like we, we also make a, it's not called a framework. It's more like we showcase that you can do GRPO and reinforcement learning with very low resources. So we are the only package which allows you to do GRPO on like a free collab.

And so like that's the only difference between us and everyone else. So for VAL, VAL was very good for large training runs. But for now, Unsloth, like if you want to do like small experimentation, you want to try stuff out, you don't know what reinforcement learning is, you don't know how to make reward functions, you don't even know what reward function to do, you should utilize our notebooks.

And that's what I was going to demo. Yeah. Yes.

So one question is RL, like let's say you don't want to use a thinking model or do chain of thought. You just have a regular task,right? Like which say call some tool, et cetera. So what are your, like if you've done any experiments on that, what is the like effectiveness of that?

Like using RL to just improve like, let's say tool use for your use.

Yes. Yes, you can do that. Exactly. It should increase accuracy by quite a bit. If your accuracy, if your accuracy before tool use was not very good, RL should definitely help. And I feel like the trick of RL is it reduces overfitting.

I think that's the trick because you do, you do multiple inferences. You don't know which one's correct, but you're trying to like maximize some good ones,right? Some good ones. The problem with general fine-tuning is you're kind of like overfitting the model.

And so the trick of, the trick of reinforcement fine-tuning is you can essentially reduce overfitting. So the model actually learns how to do tool hauling, not just, oh, you know, I see someone's trying to do a restaurant order.

I just want to call DoorDash to do an order or something,right? But it actually learns, oh, okay, because the person wants to order food, I should order DoorDash. It's like reverse thinking. So it should definitely help.

Yeah. Because some of my experiments said unless, like I tried not explicitly asking the model like a regular instruct model,right? When forcing me to like just do some task,right? And it does not, or unless it does not automatically start the thinking process unless you explicitly prompt it that, okay, first think and then answer.

Yes.

I was trying to check if like just without explicitly asking it to start the thinking process, just to improve the tool use accuracy itself, does it start thinking? And that, I think that did not happen because it, yeah.

You don't need to do it. So GRPO, generally people utilize GRPO and like reinforcement learning algorithms to create the thinking process. That's because it's like an artifact of GRPO. Just by chance, they see the reasoning process. You don't need it.

So maybe by chance, by luck, somehow it learns how to do tool calling and it's not some thinking process. It could be some weird symbols maybe, you know, I don't know. It could be like using some other, you know, like sometimes models have like different languages suddenly.

It could be like that, you know, randomly it learns how to do tool calling. It made some new programming language internally. I don't know, but it could have done that.

In this case, there was no thinking process.

Yes.

It just directly gave the output because like let's say you sample like 10 trajectories and none of them have a thinking process. Then it's never, it never explores those paths at all.

For now, it will never. But remember, it's all about luck. Over time, you will have a thinking process. Just by miraculous chance, there is a thinking process somewhere and then, oh, you should do this more and it will just do this more.

But to make that more probable, like we should prompt it probably.

You can prompt it. So essentially in the system prompt, you can say, please put your working out between this box. You could force the model to create the working out. You could do that.

You, you could. But is this the most efficient? I don't know. You could, you could say, oh, please create a new language that I don't understand which does tool calling and then it does some weird symbols and then, oh, it does tool calling.

I don't know. But yes, you, you should prompt it. It should make it more effective. Okay. Any other? Yes.

Is there secret sauce of Unsloth to make the training fast and efficient?

Oh, we, we utilize traiting kernels. We do like kernel optimizations. We reduce memory usage by 70%. There are lots of optimizations that we do to make training faster and more memory efficient. Yes. Later. Yes. It's not the main focus, but yes, we will talk about that.

For VLM specifically, we use, okay, that's actually the notebook. Oh, before that, do we have any other questions? Yes.

We're trying to focus on best time scaling these days and also papers hinting that even with best time reinforcement learning can get efficiencies in performance. Do you see any kind of reinforcement learning that kind of improves the best time learning capabilities?

Actually, the DeepSeek paper talks about this. There is parse at K and majority at K. I think they said that if you do test time scaling, I think it improves majority. I think that was correct. You need to read the, I'll have to revert back to the DeepSeek paper.

But they did say, remember, test time scaling is different from reinforcement learning. There are different methodologies. Test time scaling is calling the model 10,000 times and then, you know, then you just check, you know, by average what it, okay, you, for example, you ask ChatGPT, what is two plus two?

It might say four. You know, it says four, four, four, four, four, and suddenly it says five. I don't know. By chance, it says five. It says zero. And you just take the most likely answer. That's called test time scaling.

And then reinforcement learning is kind of different. It's more like, oh, we want to like make, we want to actually train the model to actually do the whole trace. And you don't, you don't need to do, you don't need to output 10,000 examples and get the best answer.

You just do one.

There's also best time optimization. Maybe a few short examples kind of optimizing your, things like DSPY that does like, you know, prompt optimizations, things like that,right? So there's optimizations you can build on a test time.

Yes, correct.

As well.

Yes.

Is there anything that is feeding into the reinforcement learning methodology itself that improves the later test time optimization?

Yeah. So the trick is you do the RL step and then you do test time compute. You will actually make the accuracy much higher.

There's no feedback there.

That's a good question. You could. Kind of GRPO is kind of like that,right? So GRPO, you do test time scaling in, in the actual reward function. You literally call the model, what is two plus two? Do test time scaling and then you aggregate the results.

So it's like GRPO itself is doing test time scaling internally. You could, that sounds like a new research paper. You could do that, I guess. Yes. Okay. I will have to go to the notebook now. So in order to access the notebook, you can go to our GitHub page,

which until my internet actually loads. So if you go to Unsloth,right, the GitHub page Unsloth, there is a button called Qwen3 GRPO,right? And you can click start for free. That's how you get the notebook. Or you can go to our docs, which have the notebook.

So remember, go to the GitHub page and then click start for free. Yes, for Qwen3 GRPO. And then you will get this notebook.

Generally, it's dark mode, but I know in presentations, you know, presentations, I don't think so people can see that. So I will change this to light mode.

So we utilize VLLM behind the hood. So VLLM, does, does anyone not know VLLM? I think that's a good question. Who does not know VLLM? Okay. 100% you must use VLLM,right? So like for all open source, how do you serve a large language model?

Please use VLLM or SGLang or, you know, I think Hugging Face has one as well. So like these are the best open source libraries to serve open source models. You know, like you have a GPU, how do we actually serve Llama 3?

You know, how do we serve Llama 4? You use VLLM to serve it. The trick of Unsloth is, so Unsloth is a package for fine-tuning, for GRPO, for reinforcement learning, for whatever you like, continue pre-training, whatever. And the trick is we just optimize it.

We make it much faster, you know, use 70% less memory, make it fit on a free collab. Remember, please use free collab resources. You know, Kaggle has 30 hours for, I already said this again, Kaggle has 30 hours of for free per week of GPUs.

Please utilize them. They won't be unhappy. You know, please utilize them. And yeah, so you install Unsloth in VLLM and we have this thing called the fast language model class, which essentially you can call a model. For example, we will now utilize the Qwen3 base model,right?

So like remember I told you not to do this,right? But we are, anyways, we're going to do it. This plot, where is the plot?

We are going to do the dark blue dot

to the dark green. Yeah, that's what we're going to do. We're actually not going to do what I suggested not to do. But anyways,

we're going to do that. You also have to set a max sequence length. So for example, if you want to make it longer, you can set it for longer,right? If you want longer reasoning traces, you can increase the maximum sequence length.

We set it as 2048. If you set it for larger, the free GPU will run out. So that's the problem. You can also load in 4-bit. So if you want to do 4-bit quantization, you can make the model go to 4 bits.

You can reduce memory usage by quite a bit. So you can do that as well. And remember, we are utilizing LoRA, which is a parameter efficient fine-tuning method. You don't need to fine-tune every single weight inside the entire model.

This would be very, very, very costly. Instead, we add small weights to the model to fine-tune it. And so that's the trick that we do. And because we utilize VLLM directly, we do a trick. We actually reduce memory usage by 50%.

The trick is we share VLLM's weights directly. Other training frameworks, what they do is they have to copy VLLM's weights because you have to, you have the model for fine-tuning and you have the VLLM weights. The trick that we do is we actually share the VLLM weights directly so you can reduce memory usage by a further 50%.

We use something called the Unsloth gradient check quantity, which reduces memory. Okay. Essentially everything in AI is about reducing memory usage, more efficiency. You know, everything that we do is just efficiency. So everything that we set is for efficiency purposes.

Your LoRA rank for, if you do LoRA, please set it to be the alpha to be two times the LoRA rank. It speeds up training dramatically. So please do that. Some lots of stuff,right? Compiling. We do like automatic compiling and stuff like that.

You don't need to read all of this. And here is the bulk. This is the most important part. Someone was asking about, you know, about a prompt. You make a system prompt,right? You are given a problem. Think about the problem and provide your working out.

Place it between reasoning start and reasoning end,right? So like the reasoning start is start working out and end working out. So it should look something like this. If you, you know, does start working out and end working out,right?

You are given a problem. Think about the problem. Provide your working out. Place it between start working out and end working out. Then provide your solution between solution start and solution end. So it should look something like this.

And this is the system prompt that we're going to use for reinforcement learning. Remember, you can customize this to however you like,right? You don't have to say you are given a problem. You are given a legal case,right? Think about the case and provide, provide your, I don't know, legal thinking.

I don't know. I'm just making stuff up. I don't know. Whatever. Provide, place it between, I don't know, it can be literally anything. Thinking, I don't know. I don't know. You can even make spelling mistakes. It doesn't really matter,right?

End thinking. The whole goal of RL is you can design your reward. You can design the system prompt to whatever you like,right? I think that's the main problem is like people think, oh, you must follow DeepSeek's think,right? Think,right?

You need to, people see in DeepSeek think,right? And close think,right? You do not need to follow, you do not need to follow this,right? You do not need to follow this at all. You can make it up entirely,right? This is customizable to whatever you like.

The hard part is because we are using a base model, remember this is a base model, you have to make a chat template as well. This is the more annoying part. You can just copy and paste this chat template.

You do not need to do anything else. Just, you know, literally copy and paste it. The base model does not have a chat template,right? A base model, when you call it, you can't actually call it for conversation. You can't, it's not ChatGPT,right?

It's just a base model. It doesn't do anything. So you need to specify a template for it to understand how to do conversations. And so this is kind of like a template that we did. It's a very, it's very generic.

You can just copy and paste this. It should be the same for anything.

And then we show, so after you do the chat template, we show an example of how to actually utilize the chat template and the tokenizer,right? So for example, if you ask, what is one plus one? You do the reasoning process.

It will say you are given a problem. Think about the problem and provide your working out, blah, blah, blah, blah, blah. And then the question, this is the question. Your question is, what is two plus two? Remember, the answer is four.

And so start working out. This is what you give to the model. You give all of this to the entire model,right? All of this, okay, I can't really highlight. All of this you will give into the model. And the goal of RL is you want to create the working out process automatically,right?

The RL algorithm will automatically create the, you know, the working, the working out or the thinking process. And then finally it will say four. Well, hopefully it will say four. And the goal is if you see the, if you see four, you want to make the reward higher just for that.

Now someone was talking about, you know, fine-tuning with the, you know, instruct fine-tuning first. Again, remember we go back to this diagram. We wanted to start from the blue dot to go to the green dot, but we found it doesn't actually work.

So don't actually do this. The trick is we go back to this diagram. We actually want to take the pre-trained model, do some fine-tuning, do some supervised fine-tuning, and then go to the green dot. And so this part we show that you should actually do some supervised fine-tuning.

You need to do some fine-tuning to prime the model,right? The goal is you want to learn, you make the, you want to make the model not just output zero reward forever. And so this data set allows you to prime the model to do supervised fine-tuning.

So for example, the problem is, you know, what is the sum of all the real numbers, blah, blah, blah, blah, blah. And then you use DeepSeek-R1. So this is a trick. This is a hack. You use DeepSeek-R1 to create some examples and then you shove this during the fine-tuning step.

And essentially the model already learns how to do some reasoning. And that's the trick. This data set is very small. It's only 7,000 rows,right? You don't need to have that much data for just this first step. You can have like, I think I only used 600 rows.

Very, very, very less data.

So this is just data preparation step, not that important. I need to skip to the reward function. This is the most important part of the model. So this is the supervised fine-tuning step. So all of this is the supervised, supervised fine-tuning step.

So this, this part, this part of the model, that's this part. So not that important. Okay, we skip all of that. This is the most important part. The reward function creation is the most important part. And I feel like, you know, the majority of people like neglect this part.

It is the most hardest part to do.

Okay, let's see. Where is the reward function? Oh, here it is. Okay. For example, okay, no, no. This one is a regular expression to match if your format is correct. For example, remember we have, we asked the question, we asked the model to say, please put your working out between start working out and end working out.

This regular expression essentially, essentially rewards the model to see start working out and end working out. If it doesn't have this, you will actually penalize it. And so this is one reward function that I created.

For example, I give it an example. If you say, let me think, end working out, it extracts two. Yes, that's good,right? Remember we forced the model to say, we forced the model, you must generate the answer between this and this.

And it successfully extracted two. So that's good. But, you know, also sometimes the model might generate some random spaces. It's possible. You know, the model might not actually, the model might not follow your exact format. We still try to match it,right?

Even if it generates extra spaces, we still successfully match the number two. So that's good.

This is a reward function,right? So essentially what we see is if it matches the format exactly, we add the score by three. If not, we just put zero. And remember this match format essentially matches regular expression. We had to create it by hand for matching the format.

This number does not have to be plus three. It can be plus 300, whatever you like. It can be plus one. I don't know. It can be anything that you like, but I just found plus three to work fine.

So you can do whatever you like. Yeah, anything. And remember the score is zero if you don't see it. You can also do minus one. For example, if it's else,right? If it's not good, you can also do score minus,right?

Minus three. You can minus three points from it. So up to you. You can design your reward function as whatever you like.

But remember, if the model, if the model output is not exactly following your format, we should still at least reward it a little bit,right? Otherwise your reward would just be zero, zero, zero, zero, zero. So the trick is if we see a keyword, we plus one.

And if we don't, sorry, plus 0.5 if you see the keyword. And if you don't see the keyword,right? If you see the keyword this, if you see this in the output, you should at least plus 0.5. But if you don't see it, then you minus one.

And so this essentially allows you to partially reward the model.

More reward functions now gets more complicated. This, this large reward function essentially allows you to calculate the distance-based scoring. For example, remember we said over here, where is it? This one,right? What is two plus two? Four is correct, but three is also a better answer than D,right?

If you, if you output D, it's definitely wrong,right? If you output five, it's okay, but it's wrong. And so this function essentially allows you to take the good answer. So, sorry, this is the guess divided by your true answer.

And it's like a ratio. And this ratio essentially allows you to reward. If your number is close to the actual answer, you give it higher reward. And if your, if your answer is very, very, very far off, then you penalize it by minusing reward.

And so this essentially allows you to do that. If it's exactly correct, you also add five points. And so this, this is probably the, this is probably the most important reward function, but this is only for maths. For other like code and stuff like that, you have to create more reward functions.

Now we test if our reward functions actually work. And yes, it extracts the numbers. This is just format of reward. Sorry, this is just extracting the solution. And you can see that it extracted 0.34. It extracted this number.

It extracted this and extracted this. If your reward function is not working very well, you probably did something wrong in the regular expression. So please like edit that. And then this is helper functions. Oh, this is another, another reward function.

If you see, if you see the number 1, 2, 3, 4, 5, 6, we want to remove the comma because you can't actually convert this into Python. So you want to remove the comma. And then if it's equal to the true answer, you plus 3.5 reward.

And if it's not, then you minus 1.5 reward.

More data set preparation functions, not that important. And here is the meat of the code for training,right? We call VLLM. Top P is 1.0. You don't, it's not that, you can probably, it's probably not a good 1.0 just means you're sampling the entire space.

So that's good. You can set this to like 0.8 or something else, up to you, but I generally set it to be 1.0 to be like full sampling of the entire space. Min P is 0.1. I suggest people to use this because otherwise the model might like go into like, it might do inference of like random outputs.

So, you know, use 0.1. And temperature, I did suggest people to increase temperature to 0.1.2,right? Or you can do one. Try to increase your temperature as much as possible. The more temperature you increase, the model becomes very, very creative,right?

It like creates random outputs. If you increase the temperature too much, like, you know, two, your model will be like gibberish. So probably don't do too large numbers. So I normally suggest 1.0, 1.1, or 1.2 or somewhere around there.

You should utilize min P together,right? You should utilize min P together with high temperature numbers. There is a paper about using temperature 1.5 and 0.1 min P. You should utilize that. There are some, some other things that we utilize.

Num generations is very important. I set this to be four. This number is, is this thing? Where is it?

This number is this. How many, how many, like, how many roll outs do you want to do? How many inference steps do you want to do for GRPO,right? We chose four. And so four will just means what is two plus two?

It will create four, four options. And so that is this number. If you increase this number too much, you will use much more memory. You should increase this as much as possible if you can. And there's like gradient, there's like batch size.

We set this to be one. The trick is the batch size times the gradient accumulation is equivalent most of the time. GRPO is not. But essentially what this does is if we do one, it just means we're doing one.

What is one, what is two plus two? If you set batch size to be three, then we shove all of these three examples together into one. Generally, you should set batch size to be much larger. The problem is if you set batch size to be too big, you're going to use more memory.

So the trick is instead you do gradient accumulation. You set this to be 16. That's the trick. Gradient accumulation essentially allows you to do addition of gradients over time, and you can skip using too much memory.

And then there's like evaluation. If you want to do evaluation, there's some functions for that. And then we shall see the training.

You will get a large table of numbers during the training process. This took two hours and 54 minutes on a free collab. Look at the reward column,right? The reward column. Minus 7.5, minus 5.5, minus 5.5, or very bad.

Oh, and then suddenly plus 13. Just by chance, suddenly it's plus 13. Remember GRPO, the trick is if you see this plus 13, let's maximize this even more. And then it's, oh, but then it didn't really work. So it goes back to minus 7.5, minus 5.5, minus 7.5, and so on.

And then plus 11. You see another good reward. We want to maximize this as well. And so on, so on, so on,right? That's, that's the trick of GRPO. By luck, by chance, literally by luck, you will have good answers,right?

Good answers. You want to maximize this. And if you keep looking down,right? If you keep scrolling down, in the end, okay, I need to make a plot, but in general, your reward will increase over time,right? Look, these are all positive numbers now,right?

These are all positive numbers. Your minus numbers are getting less and less and less and less. I think if I can, I don't know if I can plot this, but okay, I'll plot this later. But essentially, if you plot this over time, your reward will actually increase over time.

There is also other numbers like completion length. So essentially, remember when you use a reasoning model, the reasoning trace can be extremely long. So this column just tracks how long the reasoning process is. It's over time in general, the reasoning length should get longer and longer, but sometimes not always the case.

Yeah, not always the case. There is also another column called KR divergence. This essentially tells you how far the model is, the, you know, the final model from the original model. And the larger the number, it means it's getting very, very, very far away from the original model.

In general, this number should get bigger over time. In general, sometimes it doesn't move, but you should make this number go as much, much higher as possible. And then we also, we made separate reward functions. Each of those reward functions also has their own reward.

The most important one is the last column,right? Or the second last column. These are the two numbers. So in, in RL, there is a problem. Most RL, most RL training runs just follow the format and it doesn't actually learn.

So the format columns are not important. Do not look at the format columns. These are useless. You need to look at the last two columns. And if you look at the last two columns,right? Rewards check, check numbers. This essentially checks if the output is good or bad.

You see that it's minus 2.5, minus 2.5, not very good. And then suddenly 3.5,right? 3.5 is good. We want to maximize this. If you keep looking over time, in general, if you take the, if you take like a rolling average, in general, the model gets better and better and better.

Obviously, we only trained this for two hours and 50 minutes. You know, if you train it for 20 days, it might actually do very well. But remember, this is a free collab GPU. So in general, remember, so like the goal is, the goal of GRPO is suddenly we see a good answer with a good reward.

We want to maximize that. And that's the whole point of GRPO. It's, it's nothing fancy. It's just like by luck we see it and we just want to maximize it. We can also see some output from the model,right?

At the very beginning,right? At the very beginning of the model, let's see, where is it? An example.

Compute the number of positive integers that divide at least two of blah, blah, blah, some question. And then it does some reasoning trace. Remember, we already fine-tuned it a little bit. So it does something,right? It does something, but the answer, it just goes on and blah, blah, blah.

It just blah, blah, blah. It just keeps going on, blabbering on. But then if you look at the actual answer, where is it? If you keep, okay, there's a lot. We print out every single, we print out a lot.

Okay, it just keeps, okay, whatever. It keeps going on and on and on. This is the output of the GRPO algorithm. And you will see over time, if you inspect this, you will see that the model actually gets better and better and better.

This is an example,right? Let's say we ask the model, what is the square root of 101,right? It's not, we don't just say, what is the square root of 100? That's just 10. We say, what is the square root of 101?

If you do not train the model, this is what you get. It will say answers, education, math, and arithmetic. What is the square root of 101? Wiki user. Oh, wiki user. This is what it actually will say,right? That's actually what it will say.

Just do, where do you think this data comes from? Does anyone know? Can take a guess. Where do you think this data comes from? Probably Wikipedia,right? So if you ask the question, what is the square root of 101?

It doesn't do anything,right? Remember, this is the base model. The base model is useless,right? You're not going to get, it's not going to answer the question.

But after we do GRPO,right? What is the square root of 101? We ask the question again. It says, okay, so I need to find the square root of 101. Hmm, let me think. I remember there's a square root of numbers between the perfect numbers that are rational, blah, blah, blah, blah, blah, blah, blah,right?

It just says blah, blah, whatever, some, whatever. And there's a solution. 10 point,right here. Solution. 10.049875. I think that's correct. I don't know if that's correct, but probably it's very close. And so the goal of, so the whole point is GRPO allows you, the GRPO algorithm produced all of this,right?

This reasoning trace. In the olden days, you actually had to have a human write all of this, and then you have to fine-tune the model. GRPO, you skip, you don't need to make this anymore. It's automatic,right? That's the trick of GRPO and reinforcement learning.

All of this reasoning phase is automatic, totally produced from nothing. And in the end, it gets a solution. Yes.

You had that 7,000 world.

Yes, that's the trick. If you do the base model, the trick of doing this is if you just do the base model, go into this space, you'll still get this, but it'll be too long. Otherwise, you'll wait there for like 20 days.

And for demonstration purposes in the collab, you have to do the supervised fine-tuning step. That's the trick.

Okay.

Yeah. Yes.

Where's the advantage of doing this 7,000 examples fine-tuning versus using an instruct model out of the books?

You can use an instruct. We actually have notebooks for that. So if you go to GRPO in general, we have notebooks for using instruct model. Okay, the internet's very slow. GRPO, we actually have other notebooks. For example, if you use Llama 3.2, 3 billion, that is using instruct model.

You don't need to use a base model, but we show that you can use a base model. Yeah, you can use, I suggest people to use instruct. You probably shouldn't use base. It's all about efficiency as well. Yes.

Can you quickly touch on the KL divergence? Like maybe GRPO will be beta equals zero.

Sorry, what would it?

You mentioned the KL divergence.

Yes, yes.

Is drifting away from the original.

Yes, correct.

Can you talk a little bit about like why using that or putting beta equals zero so you get like a non-KL divergence GRPO?

Yes. So the goal of KL divergence is you want the model not to stray too much away from the original model,right? KL divergence is, I shouldn't say distance, but KL divergence is like a distance between the current model, the current model that you're training, and the previous very, very, very beginning of the model,right?

And so essentially, if the model is too far away, your KL divergence will be very large,right? If you look at the plot, where is the table? I'll scroll up a bit.

Right, which one is the KL divergence? Oh, here. This is the column,right? This column is a KL divergence column. Over time, it should get larger and larger and larger over time,right? The number should get larger and larger and larger because the model is straying away from the fine-tune, the original model.

If you set the beta to be zero, then you remove this char. Maybe this might make the capabilities of the model more, maybe, because you're essentially not forcing the model to be as close as possible to the base model.

Active error of research. So yeah, some people might set it to zero, some people might not set it to zero. You know, I think it's 0.05. I think the default is 0.05 or 0.03. If that.

Yeah.

Okay. Any other questions for? Yes.

Just took the, what's the, the 7,000 rows or whatever, where between the gibberish to the proper reasoning would it come? Like what would the output look like for the same question? Write them before the GRPO.

So for fine-tuning, actually fine-tuning is actually very helpful already. Where is the loss? Where is the loss? The base model, so the base model already is very bad. If you do

here, there's a loss. There's a loss. This is using the fine-tuning step, the pre, the priming stage,right? You use like a data set to firstly prime the data, to prime the model. The loss does decrease. Remember, if you see a loss of 0.64, that's good.

If you see a loss higher than 30, definitely something's wrong. Higher than three is very bad. You can see the loss definitely decreases over time. So yes, doing the fine-tuning stage does teach the model a little bit to do reasoning, and it learns how to do some stuff.

Also, a good, a very interesting fact is we used DeepSeek-R1, some of the reasoning process to do the fine-tuning step. And interestingly, if you just call the model without doing GRPO, it kind of does reasoning already by doing 7,000 examples,right?

It already says, remember the question was, what is two plus? Oh, wait, this is just a general question,right? It kind of learns how to do reasoning somewhat, but it's not perfect. And so the goal of GRPO is to forcibly make it perfect.

Okay, not perfect, but as much as possible. So actually the fine-tuning step already kind of learns a little bit.

Is the 7,000 the minimum examples that require?

No, we, no, actually you don't need to use 7,000. I think I only used, I think I used 118.

Oh.

It's, so it's two training epochs. I think it's, yeah, it's 118. I only used 118 rows. You don't need to, you can use 10 rows. You can use, yeah, use as less. You must use more than three rows though, because when you do LoRA, the gradients become a zero, so you must use more than three.

But, but anything more than three is fine. Yeah, so even if you use 118, it does fine. Yeah.

Yes.

Is there any things to consider if we need to scale this training for, to start with the bigger base models? Like apart from taking a bigger GPU, are there any tricks or strategies?

Do you mean like a small model versus a big model? What's the difference?

Can you tell any tricks if you want to do this training on a bigger base model?

Oh, yeah, go ahead. You can, you can take the notebook. You will need a better GPU though. Take the notebook, edit, edit this here. Not four. You can do 14 billion. Wait, I think there's a 14 billion. I think.

Or is it 12 billion? I can't remember. You can do whatever you like. You could even do, you know, Llama 3.3, 70 billion. I don't know, up to you. Do whatever you like. And, but the goal is for a collab demonstration because it's a small GPU.

I use like a small model. We actually have notebooks for free collab, which fits 14 billion. So Phi for 14 billion actually fits in a free collab. So you can do big models in a free collab. Kaggle, again, I said Kaggle.

Use Kaggle, free GPUs. There are like, so essentially this whole page has like notebooks. Where's Kaggle? Kaggle. Kaggle has like notebooks for GRPO as well. So you can do whatever you like for large models. Yeah.

Do you have examples on sampling and training, separating them on different machines?

Oh, do you mean like for VLLM rollouts? Like, oh, so the trick, what we do is you, we co-locate. So you use the same machine for inference and fine-tuning. And the trick is you can reduce memory usage because you're sharing the VLLM weights.

So some other trainers like Verl and TRL, you do have to put the inference on another server, and then your training is like a separate server, and they have to do communication. We don't, there is no communication for us.

There is none. So we do, it's very close to asynchronous training, nearly no delay in training. But yes, we, we don't support it yet, but we do plan to support like, you know, larger training runs. Yeah. Yes.

Quick question. What's the TPU roadmap look like for?

Yes, people have asked that. No roadmap. I don't know if we're going to support it. It's a bit more complicated. You could use, I think, XLA, like they, they do have PyTorch converted down to TPU, so maybe it might work.

I don't know if it works. I've never tried it.

Maybe later. Yeah, maybe later.

That's a valid answer.

Okay. Yeah. Yes.

What's the reason why you chose the base model to instruct model?

Oh, you don't have to. You, you, the, the whole point, then like this thing

here,right? We chose the base model because you can show that you can do a base model going to the green dot. But then unfortunately in the collab, we do have to do some supervised fine-tuning. Otherwise, you'll wait there forever.

The reward again will be zero, zero, zero, zero, zero. We just want to remember all of AI is about efficiency and speed. So we just want to showcase, okay, you do need to do the light blue step. You still need to do the supervised fine-tuning step.

Yeah, yeah.

So if you took an instruct model, would you need to?

Oh, no, you don't need to. You can take an instruct. So the notebooks over here, so for example, if you go to the, the Llama 3.2, 3 billion notebook, we don't do any fine-tuning step at all. You skip directly because it's an instruction model already.

It already learns how to do chat. It already learns how to answer some questions. You can skip directly to GRPO if it loads. But yes, the notebooks, okay, you'll have to wait for it to load.

Whatever. The internet's very bad. It is loading. Yes. Any other questions? Yes.

Over in the reinforced algorithm, you had the log probability of a state, of an action given a state. Is that happening inside the sampling model? Where's that in the notebook happening?

Oh, the, the, the algorithm itself or GRPO? Oh, it's like behind the scenes, like.

Yeah. Is that happening over in the GRPO trainer?

Yes. It's inside the trainer itself, like somewhere in the code, somewhere it does that.

It's figuring out like what is the probability of a given token versus all the tokens?

Oh, the calculation is inside the trainer. So like somewhere, you know, on the GPU, you're doing this calculation, but you do get the probabilities. Remember the language model, you get the probabilities already. You just get the reward function and you just want to maximize it.

So.

So do you, do you take the logits that come out of the large language model and turn them into pseudo-probabilities and then just assume that's the distribution?

I think so. Yes, that's correct. I think it's exponential of, yes, I think that's correct. There is, if you go to like the code, there is like some derivation for it, but yes, you're correct. Any, oh, okay, the notebook loaded.

But yes, there is another notebook which does the instruct here,right? There is the instruct model. And there is no fine-tuning step at all. It just does the reward function and stuff like that. Will's notebook, for example, is also very good.

So like if anyone wants to check other notebooks out, Will's notebook also utilizes, I think also the instruct model and then does GRPO.

Okay. The GR, okay, kind of time is running out, but okay, technically the GRPO portion is done. Oh, there's actually more portions. I will have to breeze through them. There's only 10 minutes left. Whoops. Any other, I will take questions at the very end.

Anyways, I'm going to stay here anyways afterwards. Quantization, we'll now shift over to quantization. So I don't know if you guys know about the DeepSeek-R1, 1.58 bit quants that we did, but you can essentially download these models. DeepSeek-R1 is 730, I think 730 GB.

You can quantize them down to 140 GB without that much loss in accuracy. Okay. There's obviously loss in accuracy, but the trick is you can quantize them down to be very small and miraculously they work.

Llama 4 Scout, for example,right? You can't really see the accuracy plot. The smallest number is 80% accuracy on MLU 5 shot, 80%. And the highest accuracy is 81 point something,right? So it's actually only 1% difference. And the one on the left is a one bit quant.

It's very small. It's tiny in comparison to the full precision like Float8. And so essentially you can make the model eight times smaller and you only decrease accuracy by 1%. So that's very interesting. And so essentially we showcase that you can actually quantize the MLU layers, the mixture of experts layers very heavily, but you must leave the attention layers, the shared experts, and other layers in higher precision.

And that's called, that's what we call the dynamic quantization methodology. If you see there was like a benchmark of Llama 4 Scout, for example, if you use a two bit, a two bit quant, it actually gets high accuracy than other, other full precision providers, which is very interesting,right?

So like, for example, the two bit quant gets 73% accuracy and then other inference providers get 65% accuracy, 67,right? There is a very large difference. And so, okay, there are like some bugs in the models and there's maybe they quantize it incorrectly, but the goal is to show that if you quantize the model down to be very small bits, it still works.

We showcase this with an example. For example, if you take a vision model like Qwen 7, Qwen 2 billion, if you naively quantize all the layers to be four bit,right? You ask the model, what does this image show?

It will say the image depicts a vibrant and colorful scene of a coastal area, which is totally wrong,right? The answer should be the image shows a train traveling on tracks or something like that,right? If you quantize everything to four bit, it's 1.36 GB, but it's definitely bad.

So the trick is you must quantize some layers to be, you must leave some layers to higher precision. And you only need to increase it by 500 GB or so to 1.8 bit and it works. The image shows a train traveling on tracks.

It suddenly works. But the question is, which layers do you not quantize? That's the question,right? Which layers? You could do an exhaustive search,right? You can check, oh, let's not quantize layer zero. Let's check layer one, layer two, you know, check every single one, but it'll take forever.

So definitely don't do that,right? You have 70 choose one or something. 70 choose one plus 70 choose two. Horrible,right? And remember, all of AI is about efficiency. So don't do that. The trick is you can check the activation quantization error and the weight quantization error and you will see these large outliers.

For example, for Qwen, if you quantize the first few layers, it's extremely bad. So you must leave the first few layers not quantized. And also this gigantic jump for the weight quantization error. This means you probably shouldn't quantize that layer as well.

There are some other plots that we show. For example, for Llama 3.2, it's interesting. All of these graphs are very different from each model. You will notice Llama 3.2 has these weird, you know, continuous spikes. It's because they use attention and then they put the attention back to the vision module.

I think every single three, I think it was every single three layers. So every single three layers, it has these big jumps. This means you should not quantize those layers.

Pixtral, for example, is also a different graph again. Pixtral seems like you can't quantize many layers, unfortunately. And so like the whole vision module must not be quantized.

There is a very popular, there is a very important paper talking about like, you know, why, which layers you should quantize and which layers you should not quantize. It's called the Superweights paper. You guys definitely should read that.

Essentially, it says that in all language models, the first few layers of the down projection, there is a very, very, very important number in one of the numbers, one of the numbers in the models, very, very, very important.

And you should never quantize it ever. But the trick is that the interesting finding is it's not actually a very large number. So there is a trend in the, there is a trend in language model space where people think that you should not quantize outliers.

The problem is, you know, these models have these big outliers. Like suddenly in the model, there's like this big number, like 3000. And if you quantize it, it essentially ruins the model. But actually this paper shows that it's not actually the outliers that are the problem.

These numbers could be very small. And if you look at the plots, if you remove, if you select these numbers and if you make them zero, the accuracy, the accuracy decreases dramatically. And so like you see, like for example, if you remove one of the numbers, you know, the activation value totally decreases, very bad.

They have very large activation values. And then if you remove them, it's very, very, very, very bad. There is another trick that you can do. If you have a model that has 7 billion parameters, make every single number go to zero.

The first parameter, make it go to zero. Check accuracy. The second number, make it go to zero. Check accuracy. The third number, go to zero. Check accuracy. You can do this 7 billion times and then you can see which number is the most important.

You could do that as well. But remember, AI is very efficient, very not a good idea.

More later research, for example, the new Blackwell chips, instead of doing quantization to like one bit, two bit, three bit, four bit, NVIDIA chips also have this new architecture, this new format called FP4 or MXFP4. And essentially this is Float4.

And Float4 is very, it's, it's most likely going to be very used a lot in the future. And then there's these like new formats for quantization as well, which, which essentially allows you to train models in very low precision.

I also made this plot going from Float32. So the question people always ask is, why is GPUs getting faster and faster and faster? My take is actually this year's probably the last year you're going to get GPUs that is actually faster.

There is no more faster GPUs. Why? Because the majority of GPUs getting faster is because of numerical precision. From Float32 to Float16, you get five times faster. Why? Why is it five times faster? Because, because the calculation of faster is when you use transistors, it's the exponent plus the mantissa squared,right?

And Float32, you have to use 23 numbers for the mantissa and 23 squared is very large. Float16, you reduce the mantissa to 10. And that is why you get five times speed up from Float32 to Float16. It's because the number itself is getting smaller.

The representation inside the models for each of the weights is getting smaller. And then we moved from Float16 to be BFloat16. It is again maybe around two times faster than Float16. We then have Float8. Float8 is even more faster, you know, uses even less space.

But then there is a problem. Float4, we get to Float4 and it's around two times faster than Float8, around. The problem is, what's next? Do we go to Float2, Float3, Float1? You know, I, there's, you can't push anymore in terms of numerical precision.

There is no much more to go in terms of that space. And so like you can only get maybe 180 times faster than Float32, maybe 200 times faster. But essentially my take is Float4 might be the final flop that's getting faster, you know, the final precision, numerical precision.

And in the future, GPUs are not going to get faster. So maybe if, you know, people want to buy Blackwell GPUs, that's probably, you should probably buy them. It's most likely not going to get faster anymore. That's kind of my take.

And also for, okay, I was going to talk about kernels and stuff, but I don't think I have enough time. You must use Torch.compile. You know, every single function that you see, wrap it in Torch.compile, try it out.

You know, like I, I always tell the PyTorch team, please make it by default. You know, definitely use Torch.compile. Why? Because it makes your training faster. Sometimes, only sometimes, not all the time. It reduces memory usage most of the time.

If you see bugs, they'll probably fix it. But remember, Torch.compile is not as easy as you think. You don't just do Torch.compile the model. There is actually many options you can tune,right? I just listed a few options. This is, this is literally just a few.

There is like 10 more pages of options. I'm being serious. 10 more pages you can tune. Imagine if you can like use Torch.compile and tune every single one. And that's why I highly suggest people to use Torch.compile more effectively.

It's probably the biggest thing that can change your entire, you know, training run. Make it more, more memory efficient, make it faster. So definitely look through this. Okay. So in general, yes, thank you. Definitely start us on GitHub.

Join our Discord if you want to have any questions on RL and stuff. We have a website as well. And finally, we have stickers. Yes, there are some limited time stickers as well that we have somewhere, I think over there.

And remember, if you have any questions, I'm still going to stay around and ask. Yeah. Thanks a lot.