The GenAI Maturity Curve or You Probably Don't Need Fine Tuning: Kyle Corbitt

Hi everyone, uh, my name is Kyle Corbitt, um, thank you for the introduction. I work for a company called OpenPipe, which is a fine-tuning platform. Um, the title of my talk is that you probably don't need fine-tuning. This is an awkward title.

Um, given that I am doing a fine-tuning platform, there's a little bit of a catch here. Uh, this is a little bit better. Put a question mark there. Maybe you don't need fine-tuning. Um, allright, I'm going to take it one more thing.

Do you need fine-tuning? Allright, let's start from this basic. We're not assuming anything, and we're going to be talking about that in the presentation. So, um, as we're going through this presentation—now, going back a couple slides—um, I actually do believe, despite the fact that I am in fact working on a fine-tuning platform, that my guess is that most people in this audience, as ofright this moment, probably do not need fine-tuning.

And we're going to be talking about in this presentation kind of what the maturity curve looks like and where theright moment is for you to bring that into your infrastructure and whether it even makes sense to do that, um, in your particular case.

Um, but my hope is that by the end of this presentation, um, a few things have happened. One is, um, you have a better qu answer to that question for yourself. Uh, does this make sense for me? Is this going to apply?

And then, just as importantly, um, if it does make sense, what is theright moment? Um, how will I know that this is something that, like, I, I, I should be doing that's actually going to help my company, um, you know, whatever I'm trying to do, uh, get, get the outcome I'm looking for?

Okay. Um, splitting this up into three pieces. Um, one is why. Okay? So kind of addressing the, the title of the talk. Um, do you need fine-tuning? Um, we're going to talk about what are the benefits, like, why, why is this even something you would put time into?

What potentially could you gain from it? Um, and then the second one is when, uh, again, at what moment in the process does it make sense, uh, to, to start doing this, if it does? And then finally, how.

Um, like, what are the actual steps you take? Kind of like what is the, what is the general process and even some specifics on how to do that. Um, hopefully in that last section we find that it's actually maybe easier than you thought.

Uh, that's usually the case when, when I talk to people, uh, who have not done it before. Okay. First we're going to talk about why. Um, so anyone, this, this is, this is a graph. We're looking at, uh, you know, the, the quality of output across several tasks we averaged, um, to generate this.

And then on the, on the X-axis we're looking at kind of tokens per dollar, basically the cost effectiveness of this model. Allright. Um, can anyone, if you've seen a graph kind of like this before, raise your hand. This is, this, this is a pretty common, um, kind of way to, to look at this.

And I, and I suspect many of you who are, um, practitioners and are using LLMs, you've, you've, um, maybe even generated a graph like this on your own specific use case. Maybe you've evaluated a few different base models.

You've looked at, um, either on your own evaluations or some kind of vibe check or something to figure out, okay, um, for this, for, for these different models, uh, how well does it do on my task? And, um, and you compare them.

Um, and usually, uh, there's a correlation, at least at, at sort of like the frontier level between how expensive a model is and how good it is. Um, typically the larger a model is in number of parameters and the more time and effort that has gone into training it, um, the higher quality outputs it's able to produce.

This is, this is all, um, pretty basic stuff. Um, and then of course as, as you get into smaller models, uh, or, or models that haven't had as much training time put into them, then, then maybe the response quality on sort of a zero-shot basis is, is not quite as good.

Um, okay. All, all pretty basic stuff. Um, one thing you'll notice is, uh, is that you can sort of think of this as like a paradigm frontier,right? Where it's like you, if you figure out how good quality you need, then you can, from that number, sort of like backfill, allright, which, uh, which model hits that bar that's at, at the lowest price point?

Or alternatively, if, um, maybe you've got some, some constraint on your unit economics or like, you know, how much you want to spend on this project. Um, and so then you, you work from the other direction, say, okay, I, I can't spend more than this.

What's the best model I can get? And so this, this frontier kind of maps, allright, given, given a certain constraint, um, what should I choose? Um, there's, uh, this, this, this next slide is a little bit of an aside, but it's actually very important for, for what we're getting to next.

Um, one way you can push this paradigm frontier, especially at kind of the low end in the sense of like the very expensive but high quality end, is, um, by basically using more compute at inference time. So this is, um, I put a gra an example on the graph.

This is actually just a very, like, particular way of doing that. It's, it's called the mixture of agents approach. Um, together AI published, uh, some papers and we've also done some research in that area. Um, but this is, you can think of this as like a very specific prompting, prompting technique that uses extra time at inference.

You're, you're going through several layers and kind of like improving your answer. And then eventually you come out with something that's higher quality than in this case you would have gotten with GPT-4 by itself. Okay. So why is this important?

Um, because it actually does prove it's possible to, to move outside this frontier, um, if we're a little bit smart of how we do it. In this case, by using more prompting. Now this particular approach, um, there's some downsides.

As you can see, it's all the way over on the, uh, left side of the graph, which means it's extremely expensive because you're doing all these calls. Um, but you get higher quality. Okay. That's cool. But if, um, a lot of people can't use that directly, again, because of those constraints where it's, it's just really expensive, latency is really high, you're doing multiple prompts, um, and so that by itself maybe isn't going to, going, going to be that interesting.

What is interesting is if you then take, say you've found some way like using this mixture of experts to get, or sorry, mixture of agents to get really high quality data, um, that is the moment, uh, where fine-tuning can really help you.

And I'm going to put up some more dots on this graph. Um, okay. So let's take a look at what I just put through up here. So this, and by the way, this graph, this, these are not, um, this, these are not conceptual.

Like this is actually, uh, you know, like this is, uh, evaluations that we ran internally on real customer data sets, um, using the evaluations our customers had to find. And, uh, and we used four different data sets. We averaged it.

The quality is, um, an LLM as judge. Uh, and then the tokens per dollar is, is basically just like, uh, industry standard, uh, pricing for those models. So, so this is based in fact. Um, this is not a, a theoretical graph.

And what I've put up here are three additional models, um, which are fine-tuned versions. And two of them are actually fine-tuned versions that are also on the graph. So if we look, yeah, on my next slide, you can actually see through the fine-tuning process for, for LLaMA 370B and then LLaMA 38B, we were able to move the performance on, like, that average across these four tasks that we tested on up substantially.

And not only are they much higher than they were from just prompting, they're actually in both those cases higher than any of the base prompted models. And that's really important because, um, what we're doing here is we are actually moving the paradigm frontier outwards,right?

So it is a different trade-off. If you're willing to do fine-tuning, if you're willing to put in the effort to create a fine-tuned model, you can actually move that paradigm frontier outward and get much more bang for your buck.

So whatever the amount of money you're willing to spend on inferences, um, you can, you can get just much higher quality completions, um, than you would have without this. And that basically, in a nutshell, is the pitch for fine-tuning, is that it gives you this ability to move outside that paradigm frontier and for whatever budget you have, um, you end up, uh, with, with a better model.

Um, so anyway, I've, I've redrawn here. We have, we have a new paradigm frontier, um, you know, which, which kind of takes this into account. Um, and so now all of a sudden the best model you're going to get at any price point is going to be one that you fine-tuned if, if you're willing to go to that work.

So, um, you know, actually I'm going to go. I, I, I really like this, this transition. Just go back and forth, you know? It's like better,right? Cool. Everybody got it? Got it? Okay. So, so this is awesome. Hopefully I've convinced you.

Like, you know, moving that outward is better. Whatever your price point is, um, you're, you're going to get a better completion. Um, okay. So this is awesome,right? Like paradigm frontier, like by definition means that you, we've expanded it.

No matter where you are on that, um, on that point, you're getting a better completion. There are no trade-offs,right?

Woo!

That'sright. Thank you. Okay. I lied. There are trade-offs. There's always trade-offs. Um, the trade-off that you typically run into with fine-tuning is there, there's a couple of different things. Um, the biggest one honestly is just the time upfront it takes,right?

Like this is an extra thing. Um, you've got to actually do it. You've got to gather your data set. You've got to make sure that the, the model's working. Um, and that is a real cost. That takes some time.

Um, and then the second one is, is you get, you lose a little bit of flexibility. Um, when you fine-tune a model, what you're doing is you are making it really, really good at one specific thing. Um, but it's not necessarily really, really good at everything else.

In fact, by definition, it's, uh, it's not good at everything. Um, that's, that's sort of like exactly what we're trying to achieve by fine-tuning. And so, um, because you're losing some of that flexibility, it's really important that you understand before you go into that fine-tuning process what your task actually looks like.

Uh, because you want to be able to specialize on that task and get really great performance on that task. And you want to make sure that when you're actually using this model at inference time, um, you aren't sending it a bunch of stuff that's kind of out of domain that is not related to what it was trained on because you're not going to get that same performance boost on, on anything that was out of domain.

So that's the primary trade-off. Um, that's, that's the reason why maybe it's notright to, to fine-tune immediately. And so that brings us to our second of the three sections we're going to talk about. And the second one is like, okay, when?

How do I make this decision? Um, how do I know when theright moment is to invest the time to do this? Um, and there is a little bit of sort of like, uh, you know, uh, specificity, but, um, there are generally three different reasons why you might want to move from a prompted model to a fine-tuned model.

Um, and the three reasons are, first of all, cost. Um, if you just can't deal with the unit economics, if you're using GPT-4 in production, that's going to be really expensive. Um, the second reason is latency. So as we're moving down towards these smaller and cheaper models, we're also, you know, one advantage of being a smaller model is there's just like less multiplications, less math going on for every single generation.

And that means you can actually bring your latency floor much lower than you can with a large model like, for example, a GPT-4. Um, so cost, latency, and then the final one is actually the, uh, the, the quality, um, particularly the consistency of the quality.

So if you find that GPT-4 maybe 80%, 90% of the time is doing a great job and it is solving your task well, but that last 10, 20%, you know, it makes mistakes, it doesn't get the formatright, it forgets one instruction.

Um, typically those kind of issues where it, it usually gets itright, but not consistently. Um, a fine-tuned model, that's the kind of thing that fine-tuning excels at, and you're going to very consistently get the, get the output you're looking for, um, at a minimum much more consistently than you would just purely through prompting.

So the way I like to think about it is, um, for almost every application, not, not universally, but for most applications I, I see people using, um, if you can get away with a prompted model to start with, um, you should do that.

Um, you, you, you should just eat the cost. This is like as you're first deploying it, you're probably not spending that much money as you're just testing it. Um, and you're going to benefit from the iteration speed you get from being able to, uh, to just change the prompt and change the behavior.

Once you've figured out, okay, this is the, this is the product shape, this is like kind of like, you know, um, what we're actually going to be deploying at scale, then it's worth taking, you know, the, the couple of hours you're going to have to invest to go from that prompted model to a fine-tuned model.

Um, and, and at that point you're going to start getting those benefits from way cheaper, way faster, and higher quality completions. Um, so that's, that's basically the, the way I think about that. Okay. So we have six minutes left.

I'm going to take six minutes and teach you how to fine-tune a model. Okay? And if you have not done this before, it might feel intimidating. I'm hoping by the end of this six-minute section, it is no longer intimidating because honestly we have had engineers that come in with no experience with machine learning, no experience with data science, and have a successful fine-tuned model that's outperforming their base model they're coming from, um, with less than an hour of work.

Um, so hopefully this is something that, that, uh, we come out of this with, with a good understanding at least, um, of what you need. Okay. So yeah, if, if you can prompt, you can fine-tune. Uh, it's, you know, there, there's, we can walk you through this process.

Allright. Four steps. You have to get the data. You have to actually train the model. You probably should evaluate. Technically step three is optional, but, um, I think, uh, we all, we all know, uh, on this track that evals are important.

Um, and then you have to deploy the, the model. Okay. So preparing data. One of the reasons why it's extremely, um, useful if you can get away with it to start with a prompted model like say GPT-4 in production is because it solves like a huge class of problems that anyone who has a background in traditional machine learning is very familiar with, which is it helps you figure out what your actual input distribution is before you have to like actually train the model.

Um, if you're deploying GPT-4 in production, you can actually see, okay, what are my users using this for? Um, like what are the areas where, where like I, it was unexpected, but like this is something people are doing.

And you can actually capture that distribution really well. You just have to log the prompts. Um, and then you can use that to know exactly what your model has to be good at. So this is actually, when people talk about how hard training models like used to be in the dark ages, um, this was actually where most of the effort was, was figuring out what, what does the data actually have to look like?

Like what does my input look like? What do the good outputs look like? Um, and if you can get away, and not every problem can, but most can in my experience, if you can get away with starting with, um, a prompt model like GPT-4, then you're in a great spot because you can very accurately capture that input distribution.

You know what your model's going to need to do, and you can just train a model that's really, really good at doing that. Um, we do that, uh, first party with an OpenPipe. It's, you don't overthink this step, by the way.

I'm, I've talked to tons of people who just like. Whatever database you're using on your application or whatever observability or logs provider you're already using, you just, every time you do a prompt, you throw it in there. Um, the important thing is you're just, you're capturing that somewhere.

Um, and then, and then you can go ahead and use that to figure out what your, what your data actually looks like. Um, and yeah, this is, this is what it looks like within OpenPipe. Okay. Once you've captured that, the next step is to make sure your data set is as high quality as possible.

And the amount of work that needs to go into this step is pretty variable. Um, if you're lucky enough that like GPT-4 is doing a great job on your problem already and you're really just trying to decrease latency or cost, um, then you're in a really great spot.

You can literally just use that exact data that you've been generating from just your normal production. Um, and, and you can fine-tune on do great. Um, if there, it is making mistakes, there may be extra steps required. You may have to do some human review.

Um, or you may have to do some relabeling with, say, a stronger model. Um, this is something I, I mentioned that, that mixture of agents thing that I mentioned earlier. This is something we've seen a lot of our customers be successful with is basically going through multiple iterations.

It's still fully automated, still with an LLM, but it can solve many of those, uh, reliability issues by going through multiple iterations as part of the labeling. Of course, it's expe, more expensive 'cause you're doing this multiple times and it's slower, but this is for generating training data.

You don't have to do it on, on that many. So it's, it's still very manageable. Anyway, um, once that data's prepared, then you actually have to train the model. Um, this part I feel like can be pretty intimidating to people.

This is what people think if they, if they're coming in without familiarity with fine-tuning. This is what people think is going to be the hard part is figuring out the hyperparameters, you know, like how do I actually train it?

How do I get the infrastructure? Um, honestly at this point, like I feel like on this, this is mostly a solved problem. Um, of course, like, uh, we do it just like with one click at OpenPipe. Um, but there's lots of other providers out there.

I've listed some that also do fine-tuning. It's quite simple. There's also some pretty easy to use at this point open-source libraries you can use, um, that I've listed on the other side as well. And, um, and ultimately like this part is actually the, the infrastructure and tooling has gotten good enough.

The architecture's involved have gotten like standardized enough that you don't really have to do that much custom work. Um, so, so the, the training part is actually relatively easy. Um, and then you want to evaluate. So evaluations, obviously there's going to be a lot of discussion on that on this track.

Um, really quickly, I like to think of evaluations as being in two categories, um, which I call the inner loop and the outer loop evaluations. Inner loop evaluations just mean something you can run really fast and get feedback on fast.

So what we do at OpenPipe is we use LM as judge evaluations. We let you define, uh, you know, the criteria and then you can run that evaluation on your fine-tune output versus your data set and see that, um, you know, really quickly, does this seemright?

And then there's the outer loop evaluations, which are incredibly important as well. And that is going to be very business specific, very use case specific, but it's basically what do I care about? How do I measure it? Is this actually achieving that?

And usually that requires actually deploying your model or at least doing some kind of A/B testing or something like that in production and seeing is this driving, is, is this doing what, um, you know, what, what I am expecting it to do?

Okay. Within, um, OpenPipe, we do have that inner loop really strongly integrated. We can, you can define LM as judges. You can see the, uh, basically how your fine-tune models are comparing, um, and very quickly decide if it's something you, you, you want to promote to prod and, and get that outer loop done with.

Okay. We got a minute left. Deployment. Um, this is actually also an area where things have gotten much more mature recently and actually it's not that scary anymore. Um, you can self-host. There's great projects out there that can give you like VLM and TRTLM.

They both give you like an OpenAI compatible API that you can just like host on your own GPUs and kind of call the same way you would. There's also a lot of hosted providers. We will host your model for you if you want.

There's lots of other inference providers out there. Um, this space, like I said, has gotten very mature recently. So, um, not something that, uh, you, you should be too scared of. Um, okay. This is just a recap of the, the, the workflow.

You got to prepare your data, train a model, evaluate it, and deploy it. Um, and then finally, um, this is what you're trying to achieve. Uh, you know, you, you can get for 125th the cost, better performance than GPT-4.

Um, and, and for a lot of use cases, it is worth it. Um, and hopefully, uh, this is enough information so you can decide for yourself when theright moment is to do that. Thank you.