Domain adaptation and fine-tuning for domain-specific LLMs: Abi Aryan

Hi everyone, welcome to my presentation. So, I don't know what's the best way to start about it. I would probably say something along the lines of, well, you've heard a lot of really good presentations that are focused on one very specific thing.

In this session itself, we'll more so focus on an overview of domain adaptation and fine-tuning for large language models. Because there's so much information out there which is like, oh, take this, use that. So my goal is to sum up all the literature for you, to be able to make an informed decision on how to be able to do domain adaptation for your particular enterprise use case or for your hobby use case, however you're using it for.

So, about me. This has already been spoken, so let's, let's skip this. Why do we care? I think, I think the answer to this is pretty obvious, which is, I mean, there are ChatGPT as a model, or even if you're looking at open-source large language models, they're not trained for every single use case out there.

There are some domains that are underrepresented. There are some domains for which there is not enough data because of compliance or for whatever reasons. And for that, we need to be able to have some sort of method to be able to fine-tune the models or use some other strategy and alternative of fine-tuning, whether that's knowledge bases, whether that's RAGs, or whether that's prompting.

The second is basically you don't want to collect new data for every single domain. One of the best things that has happened with large language models, I would say, is the ability of these models to be able to transition to a new domain.

So there's one paper that I would reference. So, one quick example, I would say, is before, let's say, transformer models, or even while we were having transformer models, to be able to train a model to learn a new language, we needed to collect the data for that particular language and then to be able to do whatever task that we want to do in that language.

One of the best things that has happened is now, because the models are learning via embeddings, they're able to learn on a new language that they've previously not seen as well, because they're essentially learning the structure of the languages instead of like, what is the taxonomy of the language.

Which means there are some languages which are semantically similar. So, for example, English is very semantically similar to Latin. I'm not entirely sure there are a couple of languages that do fall into like that one domain, which is, oh, these languages are similar.

They have semantic similarities. There are other sets of languages that have semantic similarities. So it's very easy to be able to transition between those languages without ever having seen any data or any examples in those languages. The third is basically you want the models to be able to be accessible to a wide range of users.

And what I mean by that is more so like all the work that was happening along in personalization. So, simple reasons. This is something almost everybody is aware of. What is fine-tuning? Fine-tuning is almost a way of us

teaching the model to be able to learn something for which it hasn't already been trained before. So, improving the performance of a pre-trained model. One of the ways we're doing that is by updating the parameters,right? You take some inputs, you have a hidden layer in which you're calculating the weights, you're calculating the biases, and then you have an output layer.

That all stuff, I think, is obvious to almost everybody. You've seen what a transformer model for people who don't know what the structure of a transformer model is. There's an encoder, there's a decoder. The reason I'm referencing this is we'll go a little bit more into details of these while we're talking about different fine-tuning methods itself.

So there's an encoder, there's a decoder. It has a feedforward network, it has an attention network, same for the decoder one. Now, this is, this is how we were looking at transformer models, the way they are. And this is storing the weights and the biasesright now.

But now let's talk about making these models better. So there are a couple of ways that we can fine-tune our methods. We can update all the model weights, or we can update some of the weights. If we update all the model weights, that falls into the category of some of the models that you've seen earlier, which is all the research work that was between 2018, 2016, all those, all those years, which is more around transfer learning, cross-distillation models, in which you have a teacher model and a student model.

The student model is learning from the teacher model. And that's, that's the way you're sort of updating all the weights. But it is very expensive to do that. And it is computationally, it takes more storage as well. The second option that we're now looking at, the reason we're having this discussion today, is how can we update our models?

Because the parameters have gone so big, we cannot keep updating all the weights. So how about we update just some of the weights without making sure, while making sure that we're able to get equivalent performance? And I would, I would put an asterisk on, you know, like equivalent performance, because we may not be able to get the ChatGPT performance.

And that is something we'll talk about eventually. So, in terms of if we update some of the weights, you can break it down into three categories. To be honest, more like five categories, but there are three main ones, which is adaptive tuning, there's prefix tuning, and there's parameter-efficient tuning.

There's instruction tuning, which is basically giving a couple of examples. This is something you've seen a lot at a couple of, so many examples throughout this conference. And the one that was prior to my talk as well, where we were doing instruction tuning, RLHF, obviously not super relevant to most of us, which is we would, it's too expensive to have real human beings to be able to fine-tune your parameters for you.

Or to be able to provide your examples and say, this is wrong, this isright. So we are only left with three techniques, which is adaptive tuning, prefix tuning, and parameter-efficient fine-tuning. We'll go a little bit more into detail of what these are, why are we using these ones, and when do they do well.

So the first one, this is basically adaptive-based tuning. The thing that really happens in adaptive-based tuning is it's really good. What it does, it adds a small number of parameters to the existing model. Those parameters are basically stored in the adapter components that you're seeing over there.

This is the entire model of the transformer remains the same, but we are adding two new components to it that contains the extra weights. So what this does is this exposes the model to the new information. And according to, according to the original paper that came out, you know, it is able to improve the performance of the model.

Or you could say it matches the performance of the model with only 0.15% of the parameters. Where is it good? Where, in which cases would we use something like this? So adaptive fine-tuning or adaptive fine-tuning, both are the same things.

Ideally, you use it when you're trying to learn a new domain itself, which is if you're trying to fine-tune your model for like a very different domain, let's say biochemical engineering. That's, that's more so where you would use adaptive-based fine-tuning.

The second is prefix-based fine-tuning. So prefix-based fine-tuning, what it does is it introduces some prefixes where we are storing the model weights. And what they're able to do is they're able to mimic the behavior of the prefix that we are giving it, which is the couple of weights that we are adding in front of the attention model.

So in a very simple, in very simple words, what it does, it, it, it adds an embedding layer at the front of the attention layer to mimic that behavior. One very simple example to understand this a little bit better is, you know, all of the water that we get comes out of a tank,right?

But the way we are able to access it is using a tap, and water takes the form of a tap, which is it comes out in this quantity. So that's, that's very much like how prefix tuning works, which is it's not changing the behavior of the model, but it's just mimicking or adding a masking layer on top of the existing weights or on top of the existing model that there is.

The third and the final one, which is the prefix, which is the parameter-efficient fine-tuning method. So this one, the one example that you're seeing is basically the LoRA one. There are two commonly known parameter-efficient fine-tuning methods that are out there, LoRA and QLoRA.

The way LoRA really works, what it really is, is basically low-rank adaptation method, where you use it. Any sort of parameter-efficient fine-tuning method is sort of used where you want to compress the model sizes or you want to run it on low-resource devices.

So very ideal for large language models. Biggest reason is because we have massive parameters that we are trying to run on very small devices, which could be our laptops and even smaller devices, which is basically the edge ML devices, the Arduinos and all of that stuff.

So that's one reason the entire community has been talking more so about LoRA and QLoRA, because again, we are looking for efficiency. The way it works under the hood is all of the weights are usually stored as what is basically a matrix,right?

So most of these weights, there are a lot of layers in these weights that aren't unique. And what I would, what LoRA usually does is it identifies the linearly independent layers in terms of the weight matrix itself. So in the matrix, you're looking at all the linearly independent lines or the columns, and you're picking and choosing only those ones.

So what it does is if two things are very similar or two things are almost like you could transform one easily through a mathematical function as like a multiplier of the other one, then storing that one extra layer, which is a copy of the original one, doesn't really make sense,right?

So that's, that's how LoRA works under the hood, which is we are reducing the size of the matrix, which is the size of the weight matrix, essentially.

Practical benefits, obviously, you know, you're, you're able to decrease the size of the model. And you're also, you can also, you can also, you're also using less memoryright now. The second method that we are looking at is basically called QLoRA, which is quantized LoRA method.

The way it works is it changes the model weights to four-bit precision. The way it usually works is you start with the pre-trained models. You collect a dataset with labeled, with labeled data, and you train adaptation matrix and multiply it with the main weight matrix.

And what you're essentially trying to do is you're trying to decrease the distance between the predicted outputs of the source domain and the target domain. That's, that's what's essentially going on in QLoRA. One quick comparison, obviously, I mean, in terms of like the people who are saying, okay, QLoRA is great, should we use LoRA, QLoRA?

One quick thing I'll say on that one is while QLoRA works really good on the original dataset that I was trained on, but to be able to get it to perform really well requires a library, bits and bytes libraries, and some other things which are not available on all the devices.

Not a lot of testing has really happened for QLoRA's efficiency on all of the models. So I would probably say maybe still sticking with LoRA and being able to optimize the performance with the LoRA model is ideally like the better way to go, at least at this current point in time.

So to, to very quickly summarize, which is, again, we have three different methods to be able to do domain adaptation. We have prompting, we have RAGs, we have fine-tuning. For prompting, you can sort of prompt your models, again, with no examples, with one example, with a couple of examples.

When it comes to a couple of examples, I think a good answer would be about 10, which is what ChatGPT says, but obviously the performance is better the more examples you're able to give it. Where it works is in, in the domains that you're looking for more generalizable models, but usually that's just demos, not real-world examples.

Requires less training data, it's cheaper, obviously, but is not as performant as fine-tuning. On fine-tuning, you're looking at three different methods, which is like adaptive fine-tuning, you're looking at behavioral fine-tuning, and parameter-efficient fine-tuning. On each of these ones, you don't need to pick one of these three techniques.

You can also combine them with prompt engineering, you can combine them with RAGs as well, or you can do both of those things, which is you can do adaptive as well as behavioral fine-tuning. The key difference between all those three methods is adaptive fine-tuning really works well on when you have a target domain that you're trying to optimize for.

So for example, like if you have multiple tasks within a single domain, let's say you are, you have a legal company and you're trying to build a model that works really well on five different or 10 different tasks within just the legal domain itself, adaptive fine-tuning works great.

Behavioral fine-tuning is basically where you're trying to optimize the model performance on a target task only. So you're not really optimizing for the entire domain, you're optimizing for just one particular task. The way it really works is you're optimizing for the label space and the prior probability distribution.

So very helpful when you're trying to get to show some sort of like inference and reasoning capabilities. You could also, a good analogy on behavioral fine-tuning is it's very similar to LangChain functions, if you've used LangChain functions. And parameter-efficient fine-tuning is like the standard fine-tuning where we are freezing some of the parameters and we're only updating a very small amount of parameters using the techniques, LoRA, QLoRA, and so on.

But coming to, you know, are these techniques going to really work? Sure, we have all of this available. It would only work depending on how good your data is, which is how it depends on how you're collecting your data, how you're tokenizing your data, how you're cleaning and normalizing your data.

Are you removing the noise and sort of sanitizing your models? Are you doing data duplication as well to be able to remove the duplicate entry? So there was another research that was published, which was basically like the memorization which happens in models is mainly because of data deduplication.

If we're removing the duplicate entries, that, that reduces the probability of a model to be able to memorize certain tasks because again, it's seeing those datasets over and over again in some form or the other. So it's, it's naturally collecting, creating sort of a bias towards those things, and it's naturally outputting those very quickly.

And the last one being data augmentation. Now, let's say you've done all of this. Let's say you've picked theright model. Let's say you've done your data collection thing perfectly. You've got the best data out there. What are still the things that you can think of while optimizing the performance of your model?

So the first thing is do not try to compare whether GPT-4 or GPT-5. It's not going to work comparatively, especially for more complex tasks. It's not a generalized model. While it may be able to capture the nuances of your actual data, but it may not be able to capture the nuances of the new data that it hasn't seen or in newer domains that it hasn't seen before.

So that's, that's one thing which I've seen a lot of companies are trying to sort of in a dilemma with, which is, oh, we've fine-tuned our model, but it's not working as good as GPT-4. The second one is basically using in-context learning with dynamic examples.

And one of the big reasons for that is the, the big problem that we see with the drift in the model, with the data drift in the models. So using in-context learning with dynamic example loading allows you to be able to deal with that particular problem while also making sure that you are able to do cost management as well.

The third thing that also one needs to think of is breaking down this task into smaller tasks. So for example, like if we are working with any sort of language, then instead of trying to train the model for like the entire language, can we break it down into like very specific tasks?

So that's, that's another thing which people need to think of. The final thing I would say is implementing some sort of gradient checkpointing. So what gradient checkpointing essentially does, it, it reduces the memory usage. What it essentially does is it retrains the model and recomputes the weights during the backward pass.

While it may look like, you know, it's, it's not the, it's not the smartest choice to make, but, you know, while the computation is higher, which is yes, the, the weights will need to be recomputed, but the downsides are easily weighed by the memory consumption.

So the memory consumption is very, very less if we are implementing some sort of gradient checkpointing. So another cost effect, cost management thing.

Now, a few more considerations and limitations, which is let's talk about the hyperparameters. Now, choosing a batch size. Ideally, we go with a batch size of 32 or 64. Choosing the number of training epochs. Again, one of the questions I often get is what's theright number of epochs that we should be training with?

If you're, if you're doing a simple test, which is if you're running something in a Google Colab for fonting, maybe having epoch one is nice. But if you're, if you are working with a good model and if you're trying to optimize for like a particular domain, then choosing to go with 100 epochs as like the starting point is, is probably like the ideal choice.

Choosing an optimizer, there are different optimizers that are out there. Adam optimizer is the standard choice because it's general purpose and it works really well with different domains as well. Implementing some sort of regularization, early stopping. Again, one of the things is basically like in, in terms of if, if you're looking at the models that have been trained till now, they're, they're not a lot of, there's not a lot of implementation on optimizing those performances.

While there are bigger models that we are seeing every single day with more and more parameters, they're not essentially squeezing all the performance out of those models. So one of the easy ways to be able to do that is using some sort of early stopping, which is making sure that you're only working with the data that is most efficient.

If the model performance is declining, then you need to reconsider your batch and look into that batch, consider your embeddings. Now, let's say if you fine-tune, fine-tune the model, the next part, which is the hardest part of the process is, you know, how do we evaluate our models?

There are so many

benchmarks out there. There are so many libraries out there. So there's Every by Ray, there's libraries by NVIDIA. But what you're essentially looking for mostly is the loss accuracy and perplexity, but that doesn't really paint the full picture.

So while I say, you know, it is the hardest part, which is there needs to be some sort of adaptation for every single business and every single use case, which is we need to be looking at evaluation from four different perspectives or four different components.

The first is doing some sort of metric-based evaluation, which is something like blue score, rogue score that we were considering before, doing some sort of tool-based evaluation. So I think weights and biases does have a library for doing that particularly, which is their auto evaluate, the debugger one.

And then there's another one, auto evaluator. So that is able to catch the compilation errors very quickly. The third one is using some sort of model-based evaluation, which is using a smaller model to be able to evaluate the other model.

So while this is something which is, I've not seen a lot of performance with this one because again, it's hard to do, but it has a lot of potential, which is it does standardize the process eventually and it automates the process.

And the final one is basically human in the loop, which is something I feel like, you know, this is something that we're, everybody is doing, but not the most efficient. So let's, let's just ignore human in the loop.

Maybe let's, let OpenAI talk about this.

The final thing that I wanted to say on this one for this particular presentation is while fine-tuning is great, yes, you, but you also need to think about the entire pipeline, which is how you're thinking about the data collection, how you're thinking about the storage management, how you're choosing a base model.

So optimizing the performance of your model doesn't really depend on just one feature, while it may work perfectly for like a single one-off demo, but to be able to put a robust application that does sustain the test of time.

And obviously, I'm, I'm not saying, you know, what would be an ideal time that you should be testing on. But in, in the case, the goal is to be able to get the optimal performance of the model and to be able to deal with all the data drift and the prompt drift and all of those things, while also making sure that we're catching a few things early and we're not exposing the enterprise to like reputational risk, compliance risk, and all of those things.

The entire thing has to be thought of. So it is a big picture decision that I would say that needs to be taken. So that's, that's all my presentation for today. I, I hope everybody learned something new. If there is something you would like to go with me in detail, then we can do that after the presentation.

But thank you so much.