Everything you need to know about Fine-tuning and Merging LLMs: Maxime Labonne

Hi everyone. In this session we're going to talk about everything you need to know about fine-tuning LLMs and model merging. Quick intro: my name is Maxime Labonne. I'm a Staff Machine Learning Scientist at Liquid AI. I'm also a Google Developer Expert.

I write blog posts on these topics. I created the LLM Course, which is super popular on GitHub. I also contributed to the open-source community through models, through tools, and I'm the author of "Hands-On Graph Neural Networks Using Python" with Packt.

So first of all, let's talk about fine-tuning. We saw a bit of fine-tuning in the previous session, so I'll try to not repeat too much. But basically, here's the LLM training lifecycle. You see three stages. First of all, you have the pre-training stage, where you give a lot of raw text to the model.

And the idea is that the model learns to do next-token prediction. The result of that is called a base model. This base model is really nice, but if you ask it questions or instructions, it's going to auto-complete your question instead of answering it, which is why we have the supervised fine-tuning stage, where this time we give pairs of questions and answers to the model.

And we have a similar training objective, but the idea is that at the end of it, it's going to actually answer your questions and follow your instructions. Then we have a third and final stage, the preference alignment stage, where we give human preferences to align the model to how we want it to behave.

And the result is commonly referred to as Chat Model. So when to use fine-tuning? Here you can see a little flowchart that I've made. It's very high-level, but basically there's a conversation about when to use prompt engineering, when to use fine-tuning.

I think it's good in general to start with prompt engineering if you can. And the idea is to have a really robust evaluation structure where you have a lot of different metrics that you're interested in. It can be the accuracy of the model.

Does it answer my question well? You can create a custom benchmark if you have a very niche use case. Or you can reuse open-source benchmarks. But also in terms of cost, latency. Because the question is, is it good enough?

If it's good enough with just prompt engineering, then probably you don't need fine-tuning. The problem is solved. Congrats. Otherwise, the question is, can you make an instruction dataset? So can you create pairs of questions and answers to fine-tune the model?

If it's not the case, it can be for multiple reasons, but it's probably a good sign that you need to rescope the project. Otherwise, fine-tuning is an option, and you can reuse the evaluation framework that you created to evaluate the model.

So that was the technical answer, but you also have a non-technical answer to that. Here's a report from A16Z. And the question is, why do enterprises care about open-source? You can see that the two main items are actually control and customer's ability.

And customer's ability is mostly about fine-tuning models. So even if there's arguments about the technical side and cost and latency, there's also a strong argument for customer's ability and control over these models. So in terms of fine-tuning libraries, I think that you know about Anslo now, but I'm going to talk about the other ones.

So TRL from Hugging Face, a great library built on top of Transformers, very easy to use. You have Axolotl, excellent library, very versatile. You have a lot of YAML config files. And then you have Llama Factory, where you have a really good graphical user interface that is built in.

So to talk a bit more about supervised fine-tuning, here you see an example of a sample that we give to the model. So we have the instruction, which is both the system prompt and the user prompt, and the answer, which is the output.

So in this case, the system prompt is used to steer the behavior of the model. Think like you're answering to a five-year-old. And the user actually gives the task, remove the spaces from the following sentence. We train the model usually, generally, on the output only.

So we mask the rest. It's used as context. And what we want to do is train the model to output the correct answer. Most SFT datasets, I want to say, use synthetic data, and that's perfectly fine. Usually it's generated with Fontian models, and that's a great way of building higher quality datasets.

Then you have the preference alignment. I'm just going to mention it here. There are a lot of different methods: PPO, DPO, KTO, IPO. In practice, direct preference optimization is probably the most popular one. So here you see that you have a different format with an instruction, and you have a chosen answer and a rejected answer.

So the idea here is that you're going to show a positive example, a negative example to the model. And with DPO, the goal is to make sure that the model that you're currently training outputs higher probabilities for the chosen answers than the untrained version of the same model.

I'm not going to delve too much into the details here, but this is the general idea and can be used to either censor the model, how to make a bomb, the chosen answer would be, as an AI assistant, I cannot tell you that.

Or it can also be used to boost the performance of the model in general. How to create SFT datasets? So this is a very fundamental question in the post-training world. And the main question is, OK, what's a good sample?

Human evaluation is quite bad at actually reviewing the samples. But what I like to define is three main features. The first one is the accuracy. We want the samples, the outputs, to be factually correct. Maybe no typos would be good too.

We don't want to compromise the knowledge of the model by giving it fake information. Then you have diversity. And diversity, you want to cover as many topics as you can. Of course, it depends on your use case, because if you do summarization, you won't be as general as if you do general-purpose fine-tuning.

But it's always a good idea to include a lot of different topics, different writing styles in this dataset. And finally, you have complexity. I think this one is a bit less trivial. And it's about giving complex tasks to the model, forcing reasoning.

So for example, the output will have chain-of-thought reasoning, because you want to train the model to have this kind of reasoning. Or it can be a task like summarization, explain to me like I'm a five-year-old. This kind of task really forces the model not to only answer the question like a Q&A with answers you could find on Wikipedia.

It also forces it to reason over the prompt and give a more complex answer. So as a little recipe, you can see here, I would recommend in general starting with open-source datasets if you can. Combine some of them.

Then you can apply different filters. The first one is data duplication. It can be either exact, because you want to remove duplicates, or it can be fuzzy. So same idea. And then you have data quality filters. Here you have different techniques.

It can be rule-based filtering. For example, you want to remove every single row where you have, as an AI assistant, I cannot, because people hate it. But you can also use more clever techniques, like reward models or LLM as a judge, to evaluate the quality of each sample and filter out the bad samples.

And then you can use data exploration with different tools like Lilac, Atlas, text clustering, to have topic clustering, to visualize your dataset, to get ideas on how to improve it. And with these ideas, you can go back to data generation and start the process all over again.

In terms of SFT techniques, we have three main techniques: full fine-tuning. This is the most basic one. You take the base model and you just train it on the instruction dataset. It has the best performance, but it's also very inefficient in general.

A more efficient way of seeing it is LoRA. With LoRA, you are going to freeze all the pre-trained weights and you add adapters to each targeted layer. These matrices, A and B, are these adapters. So you don't train on all the parameters of the base model.

You only retrain a subset of them. So this is a lot faster. But it can still be costly, because you're still loading the entire model in 16-bit precision here. So a more efficient way is to quantize the pre-trained model here in 4-bit precision.

This is QLoRA. And you apply the same idea that you had with LoRA, but this time the weights are heavily quantized, so you have a lower VRAM usage. The problem is that it also degrades performance. So there's a trade-off here.

I want to briefly mention some hyperparameters, but Daniel already talked about a lot of them, so I'm going to be brief. I think the most important one is the learning rate. The learning rate is model-dependent. It requires a few experiments to be able to really tweak it and find the best one.

Generally, I would recommend to go as high as you can until your loss explodes, like in this graph. Then you can reduce the size of the learning rate. Other super important hyperparameters: number of epochs. I would say that depending on the size of your dataset, you can have more or less epochs.

Sequence length is also good, because it's a trade-off with the batch size, because the longer sequence length you have, so the bigger the context window, the more VRAM you're going to use. But you don't need to use a sequence length that is as big as the pre-trained model.

Then you have the batch size. You want to maximize it to maximize the utilization of your GPUs. And then you have the LoRA with the rank. This is quite easy to fine-tune, so I don't want to go into the details here.

Let's talk about model merging now. So model merging is the idea that you can take the weights of different fine-tuned models and you can combine them together. So you just can leverage what the open-source community has produced on the Hugging Face hub, for example.

It doesn't require any GPU, so it's super efficient, and it provides excellent results. So the OpenLLM leaderboard was updated this morning. So we have a version 2 now, but this is the version 1. I haven't had time to update it.

But you can see that for 7B parameter models, the entire top 8 or top 10 is just merged models. So it really shows that this approach is extremely effective at producing high-quality models. And you can find similar results on really a lot of different datasets.

I would recommend using Merge Kit. This is the leading library in this space with a lot of different techniques that are implemented there. So here you can see the family tree of merged models. So you don't really need to see the name of the model, but you see that every node is actually a model.

And we actually merge different merges together until it becomes like a giant family tree. This one is actually quite small. It can get a lot crazier than that, but it didn't fit on one slide, so I chose this one instead.

About the merge techniques themselves, I want to mention a few of them. The first one is called SLERP. It stands for spherical linear interpolation. So the idea is really to apply spherical, but linear interpolation with the weights of different models.

You can only merge two models at the same time with this technique. But you can really tweak it with different interpolation factors for different layers. Here's a model that I've made, NeuralBeagle 147B, which was a really efficient way of leveraging the different models that were created by the open-source community.

And then you have there. So in there, you want to reduce the redundancy of the model parameters. To do that, you're going to use pruning. You're going to select the most significant parameters in your model weights. And you're going to rescale the weights of these source models.

The advantage that it has is that you can merge different models, not just two, but even more together. And I would recommend this technique, and not with just two models, not with three, but with seven or eight models.

It works really, really well. So I strongly recommend that. Then you have a very funny technique called pass-through. And in pass-through, you can concatenate layers from different LLMs. It can also be the same one. We call it self-merge.

And so here you have an example that I've made recently. It's called Meta LLaMA 3 120B Instruct, because I took LLaMA 3 70B Instruct and I just repeated 10 layers six times. So you could say, this shouldn't work at all.

Come on, you haven't even trained the model. This is ridiculous. Actually, yeah, this is ridiculous. People loved it on Twitter and Reddit and online in general. So it shows that there's a lot of things that we can still discover with these merge techniques, with these models.

They can be counterintuitive sometimes. And you can see that this model in particular was particularly good at creative writing. It was also quite unhinged in general, but really good at creative writing. And now it's being used by a lot of people, even though it's super big.

But no kind of fine-tuning at all?

No, no fine-tuning. Nothing. And then I want to mention the last technique, which is called Mixture of Experts. So in traditional Mixture of Experts, you are going to pre-train a model with a router, you can see on the bottom here, and different feedforward network layers.

And you pre-train it from scratch. But you can do something quite smart with merging, where you extract the feedforward network layers from different fine-tuned models and you combine them together like this. So we call this a Franken-MoE. You add a router, you combine the FFN layers from different models.

And this is how you create your kind of Mixture of Experts. It's actually pretty cool. It works pretty well in practice. You can see on the left a Merge Kit config for the Beyonder model. So for this model, I selected four different fine-tuned models.

One as a chat model, one as a code model, one as a role-play model, and one as a math model. You can see that I'm using positive prompts here. So actually, it's a way to initialize the router, because if you go back to the previous slide, we can see that the router is supposed to select for each token and each layer which feedforward network layer is going to be used.

We use two in general. And so how do we initialize it? If we do not fine-tune it, once again, we don't want to fine-tune it. We can, but we don't necessarily want to. In this case, we're just going to use these positive prompts, calculate the embeddings, and use these embeddings to initialize the routers.

And that works really, really well. So those are two models that I've used. For Phixtral, I had to modify it to make it compatible with Phi2. And that outperformed the base model on a lot of tasks. So it's really a good technique to use in general.

But I would say that if you compare it to merging, as we saw with SLERP and with there, I would say that if you want to increase the performance, it's better to use SLERP and there instead of Mixture of Experts, because this is a bit more experimental.

This will not bring you the same level of performance. And here you can see the results of the Beyonder model. You can see that the other models I'm comparing to are the source models that I've used in this merge.

So it's quite remarkable to see that it's actually performing better than the source models on a lot of different benchmarks. So yeah, that's it for me. Thank you for your attention. If you are interested in knowing more, if you want notebooks to run some code, I created the Large Language Model course.

All these notebooks are available on GitHub, LLMCourse. And yeah, thank you.