Fixing bugs in Gemma, Llama, & Phi 3: Daniel Han

Hello everyone. So today I'm going to talk about how to fix bugs in open source models. Thanks for coming again. We had a talk yesterday, the three-hour workshop, and thanks for coming again. So we have slides. It's at tinyurl.com/unsloth2 for the workshop slides, which we did yesterday.

You can also access that now, tinyurl.com/unsloth as well. So you might know me from, like, the Gemma bug fixes that we did. So Gemma was an open source model by Google, and there are a few bugs in there, and we fixed a few of them.

And we just did some tweets about this, and, you know, like, there's many bugs in there. Like, you know, the activation function had to be approximate JLU, not exact JLU, and there are some other issues that we talked about for Gemma.

We also have, like, some a few stickers, which you can, you know, get them from, like, when we're outside. But this is, yeah, we won't be handing them during the talk. But yeah, they're very cool and cute. And also there's, like, tokenization problems as well in language models, which we also helped to fix.

Today I'm just going to be talking about Llama 3 bugs. So yesterday I talked about Gemma and Phi-3, and today we're just sharing all the stuff that we found with Llama 3. For Gemma, you can access, like, all the bug fixes that we did in our blog post, and we have a collab notebook for all of the Gemma bug fixes as well.

For Phi-3, for example, we talked about this yesterday, and I just pasted some slides again if you want to, like, review this in your own time. For example, like, the sliding window should be 2048, not 2047. You can also, like, you should unfuse all of the QKV matrices, otherwise LoRA fine-tuning will not work that well.

But we'll be talking mainly about Llama 3. So there are actually eight bugs in Llama 3. Some of them are not announced yet. We will be announcing these later. So this is like a pre-release, and we'll be going through each of them separately.

The first one is you must not use double BOS tokens. So this is actually a very common theme in fine-tuning Llama 3. Some people don't actually know that you're adding two beginning-of-sentence tokens to the fine-tune, and this will actually ruin your fine-tune by making the accuracy of your inference time lower.

So please, like, check before you fine-tune if you're using double BOS tokens. In Unsloth, we do this, we check this automatically, and we'll remove the extra BOS token automatically for you. So this will actually cause your model to lose accuracy because if you trained on two BOS tokens and you do inference on one, then your model template will be incorrect.

So please check this. It's not just a Llama 3 problem. Other models like Mistral and Gemma also have problems like this, so just be careful of this issue. So a very easy way to check if you have double BOS tokens, if you use the apply chat template from Hugging Face, if you do the first one, your chat template must have a BOS token, otherwise it won't add it.

Llama 3 does require a BOS token. If you do the second one, you're actually having two BOS tokens if you do this, so please do not add a BOS token to the chat template. The second issue we found was you must not use the Llama 3 base model if you're using the Llama 3 template.

There are some untrained tokens in Llama 3 base. The instruct version actually has these tokens trained. So please be careful when you want to use the Llama 3 base model when you want to do your fine-tuning because some of these tokens will cause NANDs for your gradients.

These tokens include the reserve special tokens from 0 to 250, the end-of-turn token, the start header, and the end-of-header. And the graph I showed shows you the mean of the embeddings versus the other tokens, and some of them actually are zero.

So the Llama 3 team made some of these tokens go to zero purposefully because these tokens are not actually used for the model. So just please don't use some of these tokens when you do fine-tuning as well. If you want to fix them, set them to the mean of the entire tokens.

And in Unsloth, we do this automatically as well for you. So we showed some code where you can take the mean of the trained tokens and set them for the untrained tokens. Just be careful. Don't do this, like, incorrectly as well.

If you want to take the average of all the tokens, don't just take the average. You must remove the untrained tokens from the average. If you do not do that, you might actually have an incorrect average. Right? If there's, like, 10,000 tokens which are untrained, if you divide it by 10,000 plus the number of trained tokens, your average will be incorrect.

So you have to do this more complicated method of masking out the untrained tokens and then take the average. Also, reminder, because of this issue, the Llama 3 chat template will not work for the base model. I have known, like, many fine-tuning people have used the Llama 3 instruct chat template for the base model, and your fine-tune will actually be incorrect.

You will get NANDs in your gradients, and your whole fine-tune will be broken. So please do not use the Llama 3 instruct chat template for the Llama 3 base model. Only use this for the instruct model itself. Another way to fix this is to actually train the LM head and the embed tokens, which will actually learn and remove the NANDs in your models.

Another interesting fact, and not just a Llama 3 problem, but for other models, is the pad token and the EOS token must not be the same. If you do this the same, your model will have infinite generations. The reason is because the pad token gets masked out during the error, during the cross-entropy loss.

And if you use the same pad token as the EOS token, then your EOS token, the end-of-sentence token, will be masked out. So just be very, very careful when you do fine-tuning to check what is the pad token ID and the EOS token ID.

For example, if you look at Phi-3, they're the same. So technically, Phi-3, when you do fine-tuning, it will be infinite generations. So just be careful and look, you know, before you do the fine-tune, check what is the EOS token and what is the pad token.

They must be different.

For Unsloth, we also do this automatically. We fix this for you. And we essentially check if there are any unreserved tokens, and we just select one which is untrained. If there are no untrained tokens, then we will add an extra pad token ourselves.

Be careful. Do not add a pad token which has the same, like, vocabulary as your current vocabulary. So what we do is we actually check the tokens inside the vocabulary and add, like, extra hashes to see, you know, to make a new pad token.

Another issue we found for fine-tuning people is, like, when you finish your fine-tune, you don't actually know how to export it to Ollama, and that is because the chat template for Ollama must be exactly the same as your fine-tune.

And this was actually very complicated to do before, and now we can actually automatically generate the model file for you during the fine-tune. So we have, like, two collab notebooks for you to use for Ollama. One of them is the Alpaca dataset, and one of them is a you can upload a CSV file to make Ollama work after you finish fine-tuning.

Now, there are some community contributions for Llama 3 bugs. There are, like, three of them. The first one is someone noticed that you can only use CPU conversion and not GPU conversion when you convert to GGWAF or Llama CPP.

So be, you know, be careful when you convert to Llama CPP that you must use the CPU version. I think the main reason is because the precision is different in a GPU than a CPU. The CPU, when you do float 16, is different from when the GPU does float 16 conversion.

So just be careful on that as well.

Another issue is, remember we talked about the double BOS tokens. Through a community contribution, Llama CPP now has a warning for you to tell you that you're using double BOS tokens. So please, you know, take heed of the warning and do not add double BOS tokens to your chat template and when you do inference.

Another point someone found was adding a system prompt could make fine-tuning much better. And so, like, sometimes when you do inference on Llama 3 instruct, if you add an actual system prompt, this could make your whole fine-tuning better.

I think for some people, when they add the system prompt, you actually miss the system prompt. Like, you don't actually add one. And so maybe try your fine-tune with the system prompt, and you never know, this could work.

So we have, like, a GitHub package which is open source, and you can click the button Start Free Fine-Tune to start your first free fine-tune using Unsloth. We already pushed all the Llama 3 bug fixes to our GitHub repo, and so the Start Free Fine-Tune button will redirect you to a fixed collab notebook for all of these issues.

Feel free to star us as well. We also, like, have a Discord channel, so if you have any questions, you can ask, you know, any question that you like about our, you know, how to do fine-tuning, talk about AI, and talk about our bugs as well.

We also have, like, a blog post, so blog posts about all our fixes about Gemma, Llama 3, Phi-3, and more. For example, we talked about continued pre-training. You can do continued pre-training using Unsloth now. You can train on the LM head and the embed tokens, and we show that instead of just training like that, you need to reduce the learning rate of the LM head and the embed tokens by 10, or, you know, maybe 5 to 10, and this will make your training much better.

We also support four times longer context using Unsloth, and this also does not increase the time of completion. So we make it 1 to 2 percent slower, but you get four times longer context using Unsloth. And this was because, like, we use something called offloading gradient checkpointing, where we offload the gradients to system RAM.

There are some other systems which offload the gradients to the disk. Please do not do that. If you offload to disk, then your time of completion of your fine-tune will be extremely slow. So try to offload to system RAM first and then offload to disk.

Although if you don't, if you offload incorrectly, you might actually make this slower as well. So your offloading must be non-blocking calls, and do not do blocking calls to the system RAM.

Yeah, so I will show you. Okay, let's see if I can open up. Let me go to

does this work?

Okay, I'm going to open up a collab notebook for the Ollama one. Yes.

Okay. So for the Ollama collab notebook, you can simply just install Unsloth over here. This is already for free for everyone to use. And essentially, don't forget when you do the collab notebook, you have to select a max sequence length.

This determines how long your model wants to do long-context fine-tuning. You can set this to any number that you like, but remember, your dataset must match the max sequence length. So for example, if you want to set the max sequence length to, like, 10 million or 1 million, but your dataset is only, like, 1 million tokens or, like, less, try to, like, not set that max sequence length to be that large.

Otherwise, your model cannot do fine-tuning on long sequence. Load in 4-bit does 4-bit training, so this actually reduces memory usage by four times. If you do it to false, your memory usage will explode. So please do not try false, especially on a free collab Tesla T4.

If you do false, your memory usage might skyrocket to 16 GB. So do not do that. You only should do this if you use more stronger GPUs. We support, like, Unsloth supports fine-tuning all models, including, like, Llama, Mistral, Gemma, Phi-3, and more.

So this area, like, the model name over here, you can actually try to select any model name that you like. I don't think people know that Unsloth can support other models other than the ones we listed. So please try to put any model, like a Hugging Face model name, in there, and it should work.

So for the get peft model, this is where you add the peft LoRA adapters. The R is the rank. So we set it to be 16, but you can select any number that you like for fine-tuning. So we suggest you normally to use powers of 2, but you can use any number, like 1, 2, 3, like, any number that you like.

The larger the rank you select, you can make the model learn more about your dataset. But if you add too large of a rank, you might actually overfit your dataset, and also your memory usage might skyrocket again. So we normally suggest people to select 16, 32, 64, or 128.

Try not to select too large ranks. The maximum rank you should select is the size of the dimension of the model itself. So if it's 4096, set this to be 4096. For the target modules, be careful. You must do fine-tuning on all linear layers,right?

So QKVO down, up, and gate. Some people have done fine-tuning without doing some of these layers. Please do not do that because this will cause your fine-tune to be not optimal. And the LoRA alpha, there is actually a trick for this.

Normally speaking, select the alpha to be the same as the rank or larger. We found that if you do 16 times 2, so the rank times 2, this can make your fine-tuning much better. You can also use use RS LoRA to be true to set the rank automatically for you, to set the alpha automatically for you.

For the gradient checkpointing, Unsloth is the method which we showed that you can do long-context fine-tuning. You can also set this to be true, but your memory usage will increase again.

We also show you how to do data preparation in an Ollama collab notebook. So this one is we upload a Titanic CSV. So the Titanic dataset, the goal was, can you predict if someone died or survived if you're on the Titanic?

And you get details about the person. For example, their age, their, like, fare, where did they embark from, and so on. With our new collab notebooks, you have to be very careful when you do Ollama chat templates because when you do fine-tuning, you can only have two columns, the instruction and the output.

But what happens if your CSV has more than one, like, more than two columns, the instruction and output? What we can do is you can merge the columns into one column, and with Unsloth now, you can actually do that.

You can merge your columns into one.

And also, we show you that you can do customizable chat templates now. So previously, if you want to do an Alpaca-style fine-tune, you have to use instruction, input, and response for the Alpaca-style fine-tuning. But remember, the problem is, if you want to output to Ollama or GGWAF, you can only have two columns, the instruction and output,right?

If you do ChatGPT, you have to type something, and then the output comes along. You can't have, like, three inputs,right? So what we do is you can actually customize your chat template, and you must include the input and the output, and you must do this repetition twice.

Some people have asked me, like, why do you have to do two repetitions of this chat template? It's because there are dangling new lines. And we found a solution to this is you have to specify two iterations of your chat template.

We also show examples of how to do the Llama 3 chat template using our methodology. So you can see there are two iterations of the chat template. Reminder, if you don't use two iterations, you will actually error out.

And this is the training methodologies. We normally suggest people to use a batch size of 2, gradient accumulation of 4. Remember, the memory usage is only relevant to the batch size. So try not to set the batch size to be very large.

Otherwise, your memory usage will explode. Instead, set your gradient accumulation steps to be larger. So the formula for the effective batch size is batch size times the gradient accumulation. So in this case, it's 2 times 4, which is 8.

Set your learning rate to be 2e-4 or smaller, maybe 2e-5.

And after that, you can also do inference on the model. So now you have to use the apply chat template. Remember, be careful of double BOS tokens, but we in Unsloth fix this.

And finally, you have to save this to Ollama, and, you know, you have to install Ollama first. Saving now, we now support saving multiple GGWAF files. So you don't actually have to save it to one GGWAF file. You can save it to multiple, and we actually allow you to do this now.

Before, if you want to save to multiple GGWAF files, you have to wait 10 minutes extra. You can now do this automatically by, you know, specifying more than one format.

We also can show you the model file which we created. So you can actually copy-paste the model file and put this to custom, like, a custom Ollama as well. So the model file was the complicated part when we had to automatically generate this.

So we have, like, internal code to generate the model file automatically.

And finally, when you want to do inference, you can do Ollama to do inference, and, you know, it works in general. So try that out. The Ollama chat template notebook is in the slides, so tinyurl.com/unsloth2. And remember the workshop slides which we did yesterday is tinyurl.com/unsloth.

And don't forget to join our Discord channel. If you have any questions, I'm outside. You can ask questions and stuff like that. And yes, like, thanks for coming. I much appreciate it. Thanks a lot.