# LLM Quality Optimization Bootcamp: Thierry Moreau and Pedro Torruella AI Engineer · 2025-02-08 Thierry Moreau of OctoAI demonstrates how to fine-tune Llama 3 8B on a PII redaction task using OpenPipe and OctoAI, achieving 47% better accuracy and a 200x cost reduction (from $30 to $0.15 per million tokens) compared to GPT-4 Turbo. He explains that fine-tuning should follow prompt engineering and RAG, and works best for specialized tasks like function calling. The talk walks through building a fine-tuning dataset from the PI Masking 200k dataset, using OpenPipe to train a LoRA for $40, deploying it on OctoAI, and evaluating it to show the fine-tuned model scores 0.97 accuracy versus GPT-4’s 0.68. Moreau emphasizes that this continuous deployment cycle requires monitoring data drift and retraining, but tools like OpenPipe and OctoAI make it accessible even for teams without deep ML expertise. ## Questions this episode answers ### How much can fine-tuning a small open-source LLM reduce costs compared to GPT-4? Thierry Moreau shows that fine-tuning Llama 3 8B for PII redaction with OpenPipe and deploying on OctoAI yields 47% better accuracy than GPT-4 Turbo and a 200x cost reduction: from $30 per million tokens to $0.15, a 99.5% drop. This leverages a smaller open-source model and OctoAI's optimized inference, drastically cutting GenAI expenses for production. [6:36](https://aie.addtry.com/caafe93c-fbc7-4fa5-94be-49efbce0a67f?t=396000) ### What tools can I use to fine-tune and deploy an LLM for production? Thierry Moreau recommends OpenPipe, a SaaS platform that simplifies fine-tuning by handling dataset management, hyperparameter tuning, and weight export, producing LoRAs. For deployment, he uses OctoAI, which offers serverless endpoints with cost-efficient serving of fine-tuned models, achieving low latency and high scale. Together, they enable a streamlined workflow from dataset to production without managing GPU infrastructure. [5:40](https://aie.addtry.com/caafe93c-fbc7-4fa5-94be-49efbce0a67f?t=340000) ### When should I use fine-tuning instead of prompt engineering or RAG? Thierry Moreau advises moving to fine-tuning after exhausting prompt engineering and retrieval-augmented generation (RAG) without meeting quality bars. It suits highly specific, repeatable tasks like classification or function calling where you have ample high-quality data. He frames it as the 'run' stage in a crawl-walk-run quality optimization journey, ideal when facing high GenAI costs or insufficient accuracy for production. [7:23](https://aie.addtry.com/caafe93c-fbc7-4fa5-94be-49efbce0a67f?t=443000) ### Can you show an example of using a fine-tuned LLM for PII redaction with function calling? Thierry Moreau walks through a PII redaction use case where a fine-tuned Llama 3 8B model identifies 54 types of personally identifiable information (e.g., credit card numbers, names) in text using function calling. The fine-tuned model, trained on 10,000 synthetic samples and deployed on OctoAI, achieves near-perfect accuracy (0.97 vs. GPT-4's 0.68), enabling flexible redaction—such as replacing PII with category labels or fake substitutes. [3:19](https://aie.addtry.com/caafe93c-fbc7-4fa5-94be-49efbce0a67f?t=199000) ## Key moments - **[0:00] Welcome** - [0:59] OctoAI customers face GenAI bills of tens to hundreds of thousands of dollars monthly and struggle to reach production quality, says Thierry Moreau. - **[1:16] Pain Points** - **[2:00] Overview** - [2:44] Thierry Moreau presents the 'crawl, walk, run' framework for LLM quality optimization: prompt engineering, then retrieval-augmented generation, then fine-tuning. - **[3:46] Why Fine-tune?** - [3:46] Limited GPU availability and unmet quality bars limit GenAI adoption; the goal is to move from high-cost/low-quality to low-cost/high-quality. - **[5:40] Tools & Preview** - [6:13] Fine-tuning with OpenPipe and deploying on OctoAI delivers 47% better accuracy and a 200x cost reduction over GPT-4 Turbo, according to sneak peek results. - **[7:08] When to Fine-tune** - [7:23] Fine-tuning is not a silver bullet; use it only after prompt engineering and RAG are exhausted, for highly specific tasks like classification or function calling. - [9:50] To consider fine-tuning, you need a focused use case, a large amount of high-quality data, and a strong economic incentive, says Moreau. - [11:37] Thierry Moreau outlines the continuous deployment cycle for fine-tuned LLMs: dataset collection, fine-tuning, deployment, and evaluation—an iterative process that never ends. - **[12:11] Dataset Building** - [12:29] Three ways to build a fine-tuning dataset: real-world human prompts and responses, real prompts with AI-generated responses, and fully synthetic data. - **[14:36] Fine-tuning Choices** - [14:56] Poll shows most audience use proprietary LLMs; Moreau argues open-source models allow ownership of fine-tuned weights and deployment flexibility. - [16:12] Start with a smaller open-source model like Llama 3 8B for fine-tuning—larger models are more expensive to train and serve. - [17:14] Parameter-efficient fine-tuning (LoRA) produces a 50MB adapter vs. 15GB checkpoint, allowing multiple LoRAs to share a GPU and sufficient for specialized tasks. - [19:50] SaaS tools like OpenPipe lower the barrier to fine-tuning by auto-handling hyperparameters, while DIY options like Axolotl require more setup. - **[20:30] Deployment Options** - [21:48] OctoAI serves Llama 3 8B at 150 tokens/sec and 15 cents per million tokens—a 200x cost reduction vs GPT-4's $30/MTok—and handles over 20 billion tokens daily. - **[23:14] Quality Evaluation** - [23:14] Quality evaluation: automated pass/fail tests for tasks like JSON formatting vs. soft evaluations like politeness that may require LLM judging, though bias is a risk. - [24:48] "There is no finish line. Going through a single iteration is something that you might have to do on a regular basis." - **[25:01] Live Demo** - [27:06] For PII redaction, Thierry Moreau uses function calling: the LLM outputs a tool call with redaction parameters instead of redacted text, enabling flexible downstream processing. - [31:06] Thierry Moreau notes that LLMs replace complex regular expressions for PII redaction by using natural language categories and descriptions. - [35:47] Fine-tuning a 10,000-sample dataset on OpenPipe costs about $40, a small fraction of typical GenAI monthly expenses. - [36:36] Thierry Moreau compares his demo preparation to Julia Child, pulling a pre-baked model while training runs in the background. - [40:27] After fine-tuning, OpenPipe users can export the model as a 50MB LoRA or a 15GB checkpoint, giving full ownership of the weights. - [46:09] The accuracy metric for PII redaction penalizes false negatives and false positives; the fine-tuned model significantly outperforms GPT-4. - **[49:12] Results & Closing** - [49:35] Thierry Moreau concludes that fine-tuning yields significant accuracy improvements and a 200x cost reduction, using Llama 3 8B on OctoAI. ## Speakers - **Thierry Moreau** (guest) ## Topics Fine-Tuning, Training Data Curation ## Mentioned Anthropic (company), Hugging Face (company), Meta (company), NVIDIA (company), OctoAI (company), OpenAI (company), OpenPipe (company), Axolotl (product), GPT-4 (product), Google Colab (product), Kaggle (product), Llama (product), MLCLLM (product), Mistral (product), Mixtral (product), OctoStack (product), TGI (product), TensorRTLLM (product), VLLM (product) ## Transcript ### Welcome **Thierry Moreau** [0:14] So, welcome everyone. Thanks for making it to this Lunch & Learn. Uh, my goal today is to make sure that I get to share my knowledge and experience on LLM fine-tuning, and just to get a quick, sort of, poll from the audience here: how many of you are— have heard of the concept of fine-tuning here? Okay, so quite a few people. How many of you have actually had hands-on experience in fine-tuning LLMs? Okay. Allright, that's pretty good. Uh, that's more than I'm usually used to. I mean, this is quite a— fantastic that in this conference the makeup of AI engineers is close to 100%. That's not, uh, something I'm generally used to when presenting at other, you know, hackathons and conferences. So I feel like I'm— I'm speaking to theright crowd. So, just to kind of contextualize this talk, really I'm trying to address two pains that a lot of GenAI engineers face, and to get a sense of where you are in your journey. ### Pain Points **Thierry Moreau** [1:16] How many really identify and can relate to the first one, which is, "My GenAI spend has gone through the roof"? Okay. Yeah? Allright. And how many of you are in this other segment of this journey, which is, you know, you've built POCs that's showing promise, but you haven't yet quite met this quality bar to go to production? Can I get a sense of— allright. So, so I think, you know, we have a good amount of— a good fraction of the audience that can relate to one of these two problems. Uh, myself, I'm a co-founder at OctoAI, and I'm going to talk a little bit more about what we do, but the customers I've been working with, they feel those pains in very real way. We're talking about tens of thousands, if not hundreds of thousands of dollars in monthly bills, and perhaps even having issues trying to go to production because the quality bar hasn't yet been met. So, the overview of this, uh, 50-minute talk, uh, is going to be spent on understanding the why of fine-tuning. ### Overview **Thierry Moreau** [2:19] Really try to understand when to use fine-tuning. It's not really a silver bullet for all the problems you're going to face, but when usedright, in theright context, for theright problem, it can really deliver results. I'm also going to try to contextualize this notion of fine-tuning within the crawl, walk, and run of LLM quality optimization, because there's different techniques that you should attempt before trying to do fine-tuning. But finally, when you're convinced that this is theright thing for you, I'm going to talk about this continuous deployment cycle of fine-tuned LLMs. So we're going to go through today over a whole crank of that wheel of this deployment cycle composed of, you know, model, uh, you know, dataset collection, model fine-tuning, deployment, and evaluation. And really I'm trying to demystify this whole journey to you all, because in the next 15 minutes we're actually going to go through this whole process, and hopefully that's something that you're going to feel comfortable going through and, you know, applying to your own dataset, to your own problems. And so, for illustrating today's use case, we're going to use this, uh, personally identifiable information redaction use case. You know, that's a pretty traditional, sort of, data scrubbing type of, uh, application, but we're going to use LLMs, and we're going to see that we can essentially achieve state-of-the-art accuracy, uh, while keeping efficiency at— at the highest, uh, using essentially very compact, very lightweight models that have been fine-tuned for that very task. ### Why Fine-tune? **Thierry Moreau** [3:46] So again, trying to motivate this talk: what limits GenAI adoption in most businesses today, based on the conversations that I've had in the field, discussions I've had with customers and developers. The first one is, there's a limited availability of GPUs. I think we're all familiar with this problem. It's one of the reasons why NVIDIA is so successful lately. I mean, everyone wants to have access to those precious resources that allow us to run GenAI at scale, and that can also drive costs up,right? So we have to be smart about how to use those GPU resources. And also, uh, when people build POCs, it displays and shows promise, but sometimes you don't reach the expected quality bar to go to production. And so, on this XY axis, you know, on this chart, where Y axis is cost and the X axis symbolizes quality, maybe many people start, uh, on that green cross here,right? On this upper quadrant of very high cost, maybe not having met the quality bar. That's your first POC. But really, to go to production, you need to end on the opposite quadrant,right? Lower cost, higher quality, where you've met the bar, you're able to run this in a way that essentially is margin positive. And many of us are on this journey to reach that point of, uh, you know, profitability. So, we're going to learn today how to use and how to fine-tune an LLM. Now, fine-tuning is a method that we're going to use to improve the LLM quality, but as a bonus, we're going to be also showing how to improve quality significantly. And I use quality as the title of this talk because really, I think many of us AI engineers really care about reaching the high quality bar when we're using LLMs. And hopefully I'm— you know, the goal of today's talk is to instill you with some knowledge on how to tackle this journey. ### Tools & Preview **Thierry Moreau** [5:40] And so, in terms of tools that we're going to use today, we're going to use OpenPipe, which is a SaaS solution for fine-tuning that really lowers the barrier of entry for people to run their own fine-tunes. Uh, you don't need hardware or cloud instances to get started. And we're going to use this to deliver quality improvements over state-of-the-art LLMs. And of course, since I work at OctoAI, I'm going to also be using OctoAI here for the LLM deployments, and that's going to be the solution that we're going to use to achieve cost efficiency at scale. And really, the— the key here is to be able to build on a solution that is designed to serve models at production scale volumes. And just to give you a little bit of a sneak peek in terms of the results that we're going to showcase today after you go through this whole tutorial, and this is something that you're going to be able to reproduce independently, so, you know, all the code is there for you to go through. We're going to be able to show that we can achieve 47% better accuracy at the tasks that I'm going to showcase today, using this OpenPipe fine-tuning. And by deploying the model on OctoAI, we're going to achieve this— I mean, it seems kind of ridiculous— 99.5% reduction in cost. This is really a 200x reduction in cost here from a GPT-4 Turbo to Llama 3. And mostly because this is a much smaller model, it's open source, and we've optimized the hell of this model to serve it cheaply on OctoAI. So I'm going to explain how this is achieved, but I hope your interest at least has been piqued on those results that you yourself can reproduce. So, when to use fine-tuning. Uh, again, it's not really a silver bullet for all your quality problems. ### When to Fine-tune **Thierry Moreau** [7:23] It has itsright place and time. So I like to contextualize it within the crawl, walk, run of quality optimization,right? And as GenAI engineers, uh, many of us have embarked on this journey. We're at different stages of this journey. And really, it should always start with prompt engineering,right? And many of you are familiar with this concept. You start with a model, you're trying to have it accomplish a task, and sometimes you don't really manage to see the result you expect to see. So you're going to try prompt engineering, and there's different techniques of varying levels of sophistication. Uh, this talk is not about prompt engineering, so, you know, you can improve prompt specificity. There's few-shot prompting where you can provide examples, uh, to improve essentially the quality of your output. There's also chain-of-thought prompting. I mean, some of you probably have heard these concepts, but this is where you should get started,right? Make sure that given the model, given those weights, you just try to improve the prompt to get theright results. Uh, sometimes that's not enough. And, uh, there's a second class of solutions, which I like to map to the walk stage. Um, retrieval-augmented generation,right? We've probably seen a lot of talks on RAG today and throughout this conference. So, you know, there's hallucinated results. Sometimes the answer is not truthful. Well, why is that? It's because the, um, the weights of the model that is really the parametric memory of your model is limited to, you know, the point in time at which the model was trained. So when you try to ask questions on data it hasn't seen, or information that's more recent than when the model was trained, it's not going to know how to respond,right? So the key here is to provide theright amount of context. And so this is achieved through, uh, similar research, for instance, in a vector database, through function calling to bring theright context by invoking an API, through search, uh, through querying a database. And so this is something that I think many of us AI engineers have been dabbing in, in order to provide theright context to generate truthful answer,right? Complement the parametric memory of your model with non-parametric information. And that's RAG in a nutshell,right? So you've tried prompt engineering, you've tried RAG, you've eliminated quality problems and hallucinations, but that's still not enough,right? So what do you try next? Well, fine-tuning, I think, is, uh, the next stage. And again, I'm generalizing a very complicated and complex journey. But in spite of your best efforts, you've tried these techniques for maybe days, weeks, or even months, and you still don't get to where you need to be to hit production. And we're going to talk about this journey today,right? Fine-tuning. So when should you fine-tune a model? Uh, again, after you spend a lot of time in the first two phases of this, uh, journey. So spending time on prompt engineering, spending time on retrieval-augmented generation, and you don't see the results improve. And generally what helps is whenever you use an LLM for a very specific task, something that's very focused, for instance, classification, information extraction, uh, trying to format a prompt, using it for function calling. If you can narrow the use case, uh, to something that is highly specific, then you have an interesting use case for, um, for applying fine-tuning here. And another requirement is to have a lot of your own high-quality data to work with, because that's going to be your fine-tuning dataset, that goes without saying. But a model is only as good as the data that it— that the model was trained on, and we're going to apply this principle here in this tutorial. And finally, I think as an added incentive, oftentimes we're all driven by economic incentive in the work we do. Uh, for those of you who are feeling the pains of, uh, high GenAI bills, whether it is with OpenAI or with a cloud vendor or third party, well, this is generally a good reason to explore fine-tuning. So we're going to go over all the steps now that we've kind of contextualized why fine-tuning and when to consider fine-tuning. We're going to consider all the steps here in this, uh, continuous deployment cycle. It starts with building your dataset, then running the fine-tuning of the model, deploying that fine-tuned LLM into production so you can achieve scale and serve your, your, you know, your customer needs or internal needs at high volumes, and also evaluate quality. And this is an iterative process. There's not a single crank of the wheel. This is not a fire and forget situation, because data that your model sees in production is going to drift and evolve. And so this is something that you're going to have to monitor, you're going to have to update your dataset, you're going to have to fine-tune your model. And I don't want to scare you away from doing this, because it sounds fairly daunting. And so, by the end of this talk, we'll have gone through a full crank of that, uh, wheel, and hopefully, you know, it— through these, uh, SaaS toolings that I'm going to introduce you to, it's going to feel a lot more approachable. And hopefully I'll demystify the whole process of fine-tuning models. So let's start with step one, which is to build a fine-tuning dataset. Now, the data of the model has to be trained on ideally real-world, uh, data,right? It has to be as close as possible to what you're going to see in production. ### Dataset Building **Thierry Moreau** [12:29] So there's kind of a spectrum of ways to build and generate a dataset. Ideally, you build a dataset out of real-world prompts and real-world generated res— uh, real-world human responses. So, for instance, you have customer service, you've logged calls with a customer agent, you have an interaction between two humans. That's a very good dataset to work with,right? Because it's human-generated on both ends. This is very high quality, but not everyone has the ability to acquire this dataset. Sometimes you're starting from scratch. So not everyone has the luxury to start there. There's also kind of an intermediary between real-world and synthetic, where you have real-world prompts, but AI-generated responses. And so this is kind of a good middle ground between cost and quality, because you're starting from actual, uh, ground-truth information that is derived from real data, but the responses are generated by a high-quality LLM, say GPT-4 or Claude. And actually, OpenPipe is a solution that allows you to log the inputs and outputs of an LLM like GPT-4 to build your dataset for fine-tuning an LLM. So this is something that, uh, you know, a lot of practitioners use. And finally, there's the fully synthetic dataset using fully AI-generated labels. And oftentimes when you go on Hugging Face or Kaggle, you'll encounter datasets that have been built entirely synthetically. And that's a great way to kind of get started on this journey. And actually, one of the datasets we're going to use today is, uh, from that latter category. And of course, I mean, it probably goes without saying, but in case people are not fully, uh, familiar with this notion, you want to split your dataset into a training and validation, um, set because you don't want to evaluate your model on data that your, um, fine-tune has seen,right? And so, uh, many of you who are ML and AI engineers are already familiar with this, but I just want to reiterate that this is important. And finally, you know, this is used for hyperparameter tuning, and when you're deploying it and actually testing it on real-world examples, you want to have a third set outside of training and validation, which is your test set. Now you've built your dataset, you're ready to fine-tune your model, and there's a lot of decisions that we need to make at this point. And the first one is going to be open source versus closed source,right? And so who here, just like raise of hands, is using proprietary LLMs or GenAI models today from OpenAI, Anthropic, uh, Mistral AI? ### Fine-tuning Choices **Thierry Moreau** [14:56] Okay, good amount of crowd. Who here has been using open source LLMs like Llama, some of the free Mistral AI models? Okay, so maybe a smaller crowd,right? And maybe that's because these models are not as capable and sophisticated. And, but I'm going to walk you through how you can achieve better results if you do the fine-tuningright. So, of course, the benefit of open source, and this is why, you know, I'm, I'm obviously biased, but I'm an open source advocate, is that you have to, you, you get to have ownership over your model weights. So once, once you've done the fine-tuning, you are the proprietor of the weights that are the result of this fine-tuning process, which means that you can choose how you deploy it, how you serve it. This is part of your IP. And I find that this is a great thing for anyone who wants to embark on this fine-tuning journey. With proprietary solutions, you're not quite the owner, or you don't have the flexibility to decide to go with another vendor to host the, the, the models yourself. And so you're kind of locked into an ecosystem. Some people are comfortable with that, others are less comfortable with it. And many of the customers that we talk to, they're very eager to jump on the open source train, but they don't really know how to get started or, uh, you know, where to start on this journey. So hopefully this can, this can help inform you how to take your first steps here into the world of open source. Then there's a question of, like, do I use a small model or a large model? Because, for instance, even in the world of open source, you have models that are in the order of 8 billion parameters, like Llama 3 8B. And then you have the large models with, uh, Mixtral 8x22B. So this is a mixture of expert model with over 100 billion parameters. Uh, very different beasts. And we're going to see even larger models from Meta. And generally, my recommendations here is, well, look, the large models are amazing. They have broader context windows, they have higher capabilities at reasoning, but they're also more expensive to fine-tune and more expensive to serve. And typically, when you have to do a deployment, you're going to have to acquire resources like H100s to run these models. So generally, start with a smaller model like a Llama 3B model. And sometimes you'll be surprised by its ability to learn, uh, specific problems. So that's my recommendation. Start with a smaller Llama 3 8B or a Mistral 7B. And if that doesn't work out for you, then move towards, uh, larger and larger models. And today we're going to be using this Llama 3 8 billion parameter model. There's also different techniques for fine-tuning, and I'm going to go over this one fairly quickly, but there's two classes of, of fine-tuning techniques. One, which is parameter-efficient fine-tuning, it produces a LoRA. And the other one is a full parameter fine-tuning, which produces a checkpoint. A LoRA is much more, much smaller and efficient in terms of memory footprint. We're talking about 50 megabytes versus a checkpoint that is 15 gigabytes. And so you can guess that because of its more compact representation, you're able to serve it on a GPU that doesn't require as much onboard memory. And you can even serve multiple LoRAs at the same time, so multiple fine-tunes on a GPU for inference, as opposed to the checkpoints, which require a dedicated GPU for every single fine-tune. So there's more flexibility in deployment, and we're going to use that today. We're actually going to serve these LoRAs, which are the result of parameter-efficient fine-tuning, on a shared tenancy endpoint with other users who have their own LoRAs, all running on the same server. And that allows us to really reduce the cost of inference. And there is a benefit to checkpoints, though, and full parameter fine-tuning, which is that there are more parameters to tune, so it's a more flexible, uh, fine-tuning technique. It allows the model to have, essentially, achieve, uh, better results at more expensive tasks like logical reasoning. But for very specialized tasks, which is what we're going to look at today, like classification or labeling or function calling, a LoRA is just fine. So we're going to use parameter-efficient fine-tuning. And also, when you're doing fine-tuning, you have to decide, am I going to DIY it or am I going to use SaaS? So I'm sure some of you only like to DIY things. Others like the convenience of SaaS. And here, I'm not going to take a side. I think there's some great toolsright now to DIY your own fine-tuning. Uh, for instance, the open source, uh, project Axolotl. And actually, at the conference, there's the, the creator behind Axolotl who you might be able to catch. Um, and, you know, the challenge here is that you have to find your own GPU resources. You have to understand how to use these libraries, even though they're, they're, they're easier than ever, uh, to, to adopt. And you have to tune and tinker, uh, you know, settings and hyperparameters. Then there's SaaS, which really aims to make it easy to embark on this journey. Companies like OpenPipe, and there's, uh, many folks from the OpenPipe at this conference today, so if you can catch them, please do talk to them. And they're trying to lower the barrier of entry to fine-tuning,right, to make it easy. And they bring all this tooling, all these libraries to make it as seamless as possible to, for instance, move from a GPT-4 model to a fine-tune with, uh, the least amount of steps in, in collecting your data, fine-tuning, etc. And so we're going to use SaaS today, but if you feel more comfortable in this journey, uh, you might want to start with SaaS and then evolve into DIYing it. When it comes to deployment, you have to navigate the same options,right? Once you have a fine-tuned model, now you need to decide, well, how am I going to serve it,right? Because I need to generate maybe thousands, millions, or billions of tokens a day. And so you need infrastructure, you need GPUs, you need inference libraries. ### Deployment Options **Thierry Moreau** [20:43] Some people like to DIY it using libraries like VLLM, MLCLLM, TensorRTLLM, Hugging Face TGI. These are all things that you might have heard of. Uh, these are all solutions to run models on your own, on your own infrastructure. But you need to provision the resources, you need to build the infrastructure to scale with demand, and that can get tricky, especially achieving high reliability under load. That's a challenge that many people face as they scale their business up. With SaaS, you can essentially work with a third party like OctoAI. And obviously, I'm a bit biased again, I work there, so I'm going to insert a shameless plug for OctoAI, which allows users to get these fine-tunes deployed on SaaS-based endpoints. So endpoints very similar to the ones from OpenAI, for instance, if you're familiar with that, or Claude. And, uh, it offers the ability to serve different kinds of customizations as well. And so very quickly, I want to go over the advantages of OctoAI here. First of all, you get speed. So a Llama 3 8 billion parameter model, you get to achieve around 150 tokens per second, and we keep on improving that number because we've been applying our own in-house optimizations to the model serving layer. It also has a significant cost advantage because it costs about 15 cents per million tokens compared to, say, GPT-4, which costs $30 per million tokens. So that's where the 200x comes from. And we don't charge a tax for customization. So whether you're serving the base model or a fine-tune, it's the same cost. There's customization, as I mentioned, you can load your own LoRA and serve it. And finally, scale. Our customers, some of our customers generate up to, uh, billions of tokens per day on our endpoints. I think we're serving around over 20 billion tokens per day. And so we've focused and spent a lot of time on improving robustness. And also worth mentioning, if SaaS doesn't cut it for you, you are working for a Fortune 500 company or, you know, a healthcare company, uh, banking sector, government, you need to deploy your LLMs inside of your environment, either on-prem or in VPC. We also have a solution called OctoStack. Come talk to us at the booth. So that's it for the shameless, uh, plug section. Let's go over to section four, which is evaluating quality,right? We've talked about dataset collection, fine-tuning, deployment, now quality evaluation. And we could have an entire conference just dedicated on that. I'm going to try to summarize it into kind of two classes of evaluation techniques that I've seen. First of all, you know, can your quality be evaluated in a precise way that can be automated? ### Quality Evaluation **Thierry Moreau** [23:29] For instance, you generate a program or a SQL command that can run. Uh, can you, for instance, label or extract information or classify information in an accurate way? That's a kind of pass or fail scenario,right? Or formatting the output into a specific JSON formatting. This is something that you can easily test as a pass or fail test. And then there's more of the soft evaluation. For instance, if I were to take an answer and say, well, which output is written in a more polite or professional way? You can't really write a program to evaluate this, unless you're using an LLM, of course,right? But you have to put yourself into maybe 2000, 2000, sorry, 2020, 2021 mindset before, uh, GPT was around. Well, it'd be hard to build a program that can assess this,right? So generally, you'd need a human in the loop to say which out of, out of A or B is a better answer. Thankfully, today we can use LLMs to automate that evaluation. But keep in mind that, for instance, if you're using GPT-4 to evaluate two answers, well, if you're comparing against GPT-4, it might favor its own answer. And people have seen that in these kind of evaluations. So this is a whole science. I mean, we could have a whole conference just on this. I just wanted to present the high-level, uh, guidelines of this whole cycle of deploying, uh, fine-tuned LLMs. And so really, there is no finish line. That's what I want to convey to you all, that going through a single iteration is something that you might have to do on a regular basis, maybe once a week, maybe once a year. It all depends on your, uh, use case and constraints. Now, let's get a bit more practical. Let's switch over to our demo. And so for those of you who came, uh, a little bit late, there's a QR code here that you can scan, and that will point you to our Google Colab. ### Live Demo **Thierry Moreau** [25:21] And we also have under Slack, uh, let me see if I can pull it. If you're in the Slack channel for AI Engineers World Fair, there is this, uh, quality optimization bootcamp where you can ask questions here if you want to follow along. And so we're going to go, we're going to try to go over the, uh, practical component in the next, uh, 25 minutes. I just want to provide some context here. The use case is, uh, personally identifiable information redaction. We've taken this from a dataset composed by AI for privacy called PI Masking 200k. It's one of the largest datasets of its kind. It has 54 different PII classes, so different kinds of sensitive data, like the name, the, uh, email address, add, you know, address of physical address of someone, uh, their credit card information, etc., etc. Across 229 discussion subjects. So that includes conversations from, uh, customer ticket resolution, uh, conversations with a banker, conversations between individuals, etc. What this dataset looks like is as follows. You're going to have a message, an email. Uh, here we have, you know, something that looks like it came out of an email that contains credit card information, IP address, maybe even a mention of a role or, or, uh, anything that is essentially personally, personally identifiable. And I've highlighted those in red because they will need to be redacted. And after redaction, we should get the following text that shows, look, here is this information that is now redacted, anonymized. But instead of just masking it, we're actually telling it what kind of category this information belongs to,right? A credit card number, an IP address, or a job title. And this is how we're going to redact this text. So where do LLMs come in? The way we would use it is through function calling. Who here has used LLMs with tool calls or function calls? Okay, so quite a few people, you know, and, um, as, as many of us are aware, this kind of what powers a lot of the agentic applications. So this is a great use case for people who want to do function calling and are not seeing the results, you know, out of the box from, say, GPT-4, uh, that they would like to, to, to see. And in this case, we're actually going to see that, that these kind of state-of-the-art models aren't doing quite well at fairly large, uh, and complex, uh, function call use cases. Uh, so to achieve this, uh, redaction use case, we're going to pass in a system prompt. We're going to also pass in a tool specification. The system prompt says, look, you're an expert model trained to do redaction and you can call this function. Here are all the sensitive, uh, PII categories for you to redact. And then as a user prompt, we're going to pass in that email or that message. And then the output is a tools call. So it's not the redacted text. It's actually a tools call to that redact function that's going to contain all the arguments for us to perform the redaction. Why am I doing this as opposed to spitting out the redacted test? Well, that gives us flexibility in terms of how we want to redact this text. We could choose to just replace that information with the PII class. We could also completely obfuscate it. Or we could choose to use, for instance, a database that maps each PII entry to a fake substitute so that we have an email that kind of reads normally, except the credit card, the, the, the names, the addresses are all made up, but they will always map to the same individual. And so that allows us to do then more interesting processing on our dataset,right? So that's why we're going to use function calling here. And let's start to build the dataset. So I'm going to switch over to our notebook here. Uh, this notebook is meant to be sort of, uh, self-explainable, so there's a bit of redundance, redundant context. As part of the prerequisites, you're going to have to get an account on OctoAI and OpenPipe. Um, and, and these are the tools that we're going to use. And if you want to run the evaluation function, also provide your OpenAI key because we're going to compare against GPT-4. So we're going to install the Python packages initially, only OpenAI and datasets from Hugging Face. You can ignore this, uh, pip dependency error here, uh, which happens when you, uh, pip install datasets in a Colab notebook, but that's okay. We can get past that. You can enter your OctoAI token and OpenAI API key at the beginning. And I've already done this. So we're going to start with the first phase, which is to build a fine-tuning dataset. So we have this PI masking dataset, and I'm going to show it from Hugging Face. So PI, uh, masking. And you can see what the dataset looks like. It has the source text information. As you can see, these are, you know, exchange, you know, snippets from emails, for instance. You have the target text that is redacted and the privacy mask that contains each one of the PII and the classes associated to it. So this contains all the data, all the information, input, and labels that we need to build our, uh, our dataset for fine-tuning. And so really what we're going to do is that we're going to use the system prompt. We're going to define our system prompt here, which is again, telling the model you're an expert model trained to redact information. And here are the 56 categories explaining next to each category what that corresponds to. And this is really the beauty of LLM and sort of natural language entry is that in the old world when we were doing PI redaction, we had to write complex regular expressions. And here, this is all done through just providing a category and a bit of a description here, and the LLM will naturally infer how to do the re-redaction. We're also going to, uh, define the tool to call,right? And so this is done as a, uh, essentially a dictionary, a JSON object. And as you can see, there is an array that contains, uh, dictionaries containing a string and a PI type. And the string is the PI information. The type is essentially one of 56 categories that we provide as an enum. Soright off the bat, you can see that this tool call is, um, you know, a bit of a large function, uh, specification. And so let's load our dataset from Hugging Face. In this case, it's going to take maybe a few seconds to load in that dataset of 200,000, uh, entries. And then what I have in the next cell when I'm downloading this, uh, dataset is what I'm going to use to build my fine-tuning training dataset. And here's the thing about fine-tuning is that to build your dataset, you need to make it seem like you've essentially logged conversations with an LLM,right? You're logging the prompts and the responses because that's how you're going to fine-tune it. You need to tell it this is the input with system prompt, uh, tools specification, user prompt, and here's the, uh, tools call response that I expect to see. And so this cell here just sets it up so that we essentially have each training sample as a message from an LLM that's been logged. We're going to see what that looks like in a second. So we're going to build a 10,000 entry training dataset for OpenPipe. And that's going to be downloaded as this OpenPipe dataset.jsonl. And so as I run the cell, it's going to download this, uh, from Colab. And now when you switch over to OpenPipe, we're going to create a new dataset. So once you're on OpenPipe console, you have a project here. I've, uh, generically named it project one. You can access datasets. And already, as you can see, I already have built a few datasets, uh, before, but if you're a first-time user, you're not going to see anything under datasets. So you can create a new dataset here by clicking on this button. And if you go under settings, we can name our dataset. So I'm going to call it, uh, lunch and learn. And today is, uh, June 26. Allright. So this is two days lunch and learn. I'm going to, I'm going to call this my dataset. And under general, I can upload the data that I just downloaded from my notebook, OpenPipe dataset.jsonl. So this upload operation is going to take a few seconds or maybe a couple of minutes because what's going to happen on OpenPipe is not only we're uploading this dataset, but it's going to do some, uh, pre-processing here to split it into, uh, training and validation set. It's also going to get it all formatted in a nice way so we can essentially look into the dataset. Uh, so you can see there's this little window here that shows that you're uploading the dataset and that it is essentially being processed. So while this is happening,right, we've prepared our dataset, and we're going to take a look at it in a second while it's being processed on OpenPipe. But let's see how we're going to do the fine-tuning in the next stage,right? So once we have our dataset uploaded, we're going to have this view on the dataset that shows every single entry that we can peek into and how it's split into training and test set, generally a 90-10% split. And from that UI, we can launch a fine-tune. And this is where we get to choose our base model. And what we're going to choose is a Llama 3 8 billion parameter model with 32k, uh, context width, which is a, uh, fine-tuned from news research called the theta model. And you can see that there's essentially a pricing here that is being estimated for this fine-tune. We have a substantial training set because it can range from, say, hundreds of samples to thousands to hundreds of thousands. And the cost can scale up as you, um, as you feed in more training samples, but it will improve the accuracy. And it also provides an estimated training price of $40. Now, that might seem like a lot, especially when you're tinkering with fine-tuning, but keep in mind some of the people that we work with, they tend to spend tens of thousands or maybe hundreds of thousands of dollars a month on GenAI spend. So this is absolutely something that you can do upfront that will pay off. And I believe that on OpenPipe, if you get started, you get a $100 credit. Uh, so that allows you to, to run some fine-tunes off the bat, uh, without having to necessarily, uh, have to, to pay. So, um, let's go over to OpenPipe. And it is still uploading. I think maybe the network is, uh, is a bit slow, but we're going to essentially start training at this point. And once the training is happening, we're going to then deploy the fine-tune LLM, uh, when training is done. And what happens on OpenPipe is when you're done with training, you're going to get an email when that training job is done. It can take a few minutes. So I'm going to pull a Julia Child here. I'm going to stick the, you know, the turkey in the oven. And in the second oven, I'm going to have a, a pre-baked turkey just so that we don't lose time. But as you're going through this on your own, keep in mind it's going to take a little bit of time to just kick off that whole fine-tuning process, but it's not that long because, um, you know, you're training a fairly small model here. Allright. So this is still, uh, saving, but let's kind of take a look at what we've done so far,right? So we've built our dataset using a synthetic dataset from Hugging Face. Uh, we format each input-output pair from the dataset as logged LLM messages. And this is essentially stored as a JSONL file that we upload to OpenPipe. And we produce 10,000 training samples. We're fine-tuning a model from OpenPipe, and we're OpenPipe uses a parameter-efficient fine-tuning, which produces a LoRA. And we choose Llama 3 8 billion parameter model as the base. And when we deploy, what we're going to use here is OctoAI. So let's see. This didn't finish uploading. So I'm going to go into the one that I uploaded just a couple days ago just to essentially show you what you should see on the, uh, user interface. So as you peruse through the training samples, what you're going to see is an input column and output column. And so on the left, you have the input with the system prompt. As you can see, it's a, it's a big boy, uh, because it has all these different categories,right, that it needs to classify. It also has the user prompt, which is the message that we need to redact, the tool choice, and the tool specification here with all the different categories of PII types. And then the output will be, will be this tools call from the assistance response. And that will have this redact call along with these arguments field to redact as a list of dictionary entries containing string and PII type information,right? And so this is what we've passed into our fine-tuning, uh, dataset into OpenPipe. And, uh, this is still saving. So I'm just going to go ahead and go to the model. So once you have the dataset uploaded, again, you hit this fine-tune button, and this is what's going to allow you to launch a fine-tuning job,right? I can call this blah. And this is where you select under this dropdown the model that you want to fine-tune. Uh, this is again what we saw before. Training size is substantial. I'm not going to hit start training because I already have a trained model, but when you do that, it's going to kick off the training. And when it's done, you'll get notified by email. Now, let's fast forward. Let's assume I've already trained my model. So I'm going to have this model here that's been fine-tuned from this dataset. I'm going to click on it. As we can see, it's an Llama 3 8B model. It's been fine-tuned over these 10,000, uh, datasets split into 9,000, uh, training samples and 1,000 test samples. Uh, we can even look at the evaluation. But going back to the model. And the nice thing is that it's taking care of the hyperparameter like learning rate, number of epochs. It kind of figures it out for you, so you don't really have to tweak those settings. And I find that to be very convenient, especially for people who haven't yet built an understanding of how to tweak those values. And the beauty of using OpenPipe is that you can now export the weights and be the owner of those weights,right? Remember when we talked about open source, it's really important to own the result of the fine-tuning. So you can download the weights in any format you want. You have LoRAs, but also merged checkpoints. So you can have a, a, a parameter-efficient representation as well as a checkpoint. And so we've selected to export our model as a FP16 LoRA, which is what we're going to use to upload our model on OctoAI, which is where we're going to use to deploy the model. So now I can download the weights as a zip file. And it's fairly small, only 50 megabytes, but I can also copy the link, copy the URL. And this is what we're going to need to do in this tutorial. So to deploy the model, what we need to do is copy this URL. I'm going to download in the cell the OctoAI CLI. This is a command line interface for users to upload their own fine-tunes to what we call our asset library. So this is a place where you can store your own checkpoints, your own LoRAs for not just LLMs, but also models like Stable Diffusion, if some of you are developers who also work in the image gen space. And so we can serve these customized models, uh, on our platform. And so we're going to upload this LoRA from OpenPipe to OctoAI. So we're going to log in just to make sure our credentials are good. And here we have a confirmation that our token is valid. And in the cell, we have to replace the LoRA URL from set me to that URL that I just copied here from download weights. And keep in mind this might take a couple minutes to get the link to appear, but once you have that link, and again, I'm kind of skipping ahead because when you're going to run this at your own time, it might take a, you know, a few minutes to run the fine-tune. It might take a few minutes to download the weights. But everything that I'm running here is essentially the steps that you'll take yourself. And what I'm doing here is, uh, passing in this URL here and setting a LoRA asset name in my OctoAI asset library so I can then create this asset from this LoRA as a safe Tensor file. And, uh, based on the Llama 3 8B model, I'm going to name it, uh, let's see. Seems like something has, uh, failed here. So let's try to run it again. And so what this is doing is, uh, let's see. Usually that, that, uh, that should have worked. So what's, uh, what should happen here is at this point, once you've taken the, uh, URL of your fine-tune asset, should be able to host it on our asset library and then from there serve it to start running some inferences. So this, uh, this, this LoRA upload step didn't quite work here. So Pedro, are you able to maybe double-check with product whether this capability is working? Uh, this isn't a good demo unless something fails. And so, uh, yeah, you know, I just tested it earlier today and it was working flawlessly. So, uh, let's see. I might have to list my assets so I can pull an old one. Um, actually, one second. Pedro, can you, can you tell me what the, uh, command is to, to list the assets that are on? I think it might be OctoAI, OctoAI asset list. Allright. Let's. Okay, there we go. So I'm going to pull from an asset that I uploaded earlier. The third one. Allright. Yeah, because. Allright. So I'm going to take an asset that I uploaded earlier. I'm going to have to look into why, uh, that step failed, but, uh, let's, let's try this. Okay. So I'm going to use an asset that I uploaded earlier. I'm not sure why this didn't work, but I'll make sure that this is working for you all to reproduce this step. And I'm going to set LoRA asset name equals this. Allright. So these other LoRAs I uploaded using the exact same steps as I used for this, uh, tutorial. So we'll make sure to get to the bottom of this and, uh, we'll use the Slack channel here for folks who want to run through this, uh, step. But I'm just going to run an example inference here on this asset that I pulled from OpenPipe. And so again, we have our, uh, system prompt. We're going to pass in this, uh, you know, this message, this email as our test prompt. And then when we're invoking this OctoAI endpoint, we're using the standard chat completions, uh, from OpenAI. And what we're passing here is this OpenPipe Llama 3 8B 32K model. And we pass in this argument for parameter-efficient fine-tune and pass in the LoRA asset name that we just uploaded to the asset library. And as we can see, the response here contains the tool calls and the call to the function that will do the redaction. So this is behaving exactly as we intend it to. So now we can move on to the quality evaluation. For quality evaluation, what we've done is use essentially an accuracy metric. Thankfully, we have a ground truth,right, from our dataset. All the exchanges have been labeled with privacy mask information that we can use as ground truth. So that makes evaluating scoring our results fairly easy. We don't have to use an LLM, for instance, for that. We can actually use more traditional techniques of accuracy evaluation. And so we have a metric that we've built. It assigns a score that can be penalized when PII information was missed or mistakenly added, i.e., false negative or false positive. And then we use a similar distance metric to kind of match the responses from the LLMs compared to our ground truth. So for illustration purposes, we have, for instance, this PII information that's been redacted. That's a score of 1.0 because it's the perfect match. Our fine-tune might, for instance, miss the fact that Billy was the middle name and might interpret it as first name. In that case, we're still attributing a high score because it's close enough and probably for a practical use case that would be good enough. But for instance, upon calling GPT-4, it fails to identify two out of the three information that we had to redact. And so the score is about a third here,right? So in this case, what we're going to do here, I'm just going to reduce the test size to 100 samples. And I am going to run this evaluation inside of, uh, this cell. It's going to bring us 100 test samples that we can then run our evaluation metric and get our overall scoring out of. Uh, so if we look at, you know, the, uh, output from the cell, essentially we're just invoking back to back the fine-tune running OctoAI and we're invoking GPT-4 on OpenAI to do the results collection. So we're going to collect some results here. And, uh, once we've collected the results, once we get to 100, I think we're getting pretty close here, we can run the quality evaluation metric. And of course, I invite you to run it on more samples, maybe 1,000 or 10,000. Uh, it just gets more expensive as you're using GPT-4. You know, to run 100 samples, it costs about a dollar in inference. So, uh, then 1,000 samples costs, uh, $10. And now we're going to score it. Allright. So we're going to go through every single entry. We have our ground truth information. We have our eval and labels from GPT-4 and our eval and labels from our fine-tune. And we can see thatright off the bat, the fine-tune is actually better at finding the PII to redact. Here, GPT-4 scored only a score of 0.49, whereas our fine-tune achieved 0.85. And here 0.3 for GPT-4, 1.04 the fine-tune. So the fine-tune overall is performing better. And once we aggregate and average the score, GPT-4 achieved 0.68 out of 1, whereas our fine-tune achieved 0.97. ### Results & Closing **Thierry Moreau** [49:12] And so that's the difference between prototype and production,right? You're expected to achieve somewhere in the single nine or two nines of accuracy. And this is what this technique shows allows you to achieve. And again, I want to reiterate that in terms of cost, GPT-4 costs upwards to $30 per million tokens generated, whereas Llama 3 8B on OctoAI costs just 15 cents. That's a 200x difference,right? So with that, I just want to conclude with some takeaways on fine-tuning,right? Fine-tuning is a journey, but a very rewarding journey. There's truly no finish line here. You need to attempt fine-tuning after you already tried other techniques like prompt engineering, retrieval augmented generation. But once you decide to embark, data is very important. Collecting your dataset, uh, because your model is only as good as the data it's trained on. You need to make sure to continuously monitor quality to retune your model as needed. You also need to, um, you know, thankfully we have solutions like OctoAI and OpenPipe to really make this more approachable and easy to do. And it's easier than ever. It's only getting to get easier, but maybe a year ago it was only reserved for the most adventurous and sophisticated users. And now we've really lowered the barrier of entry. And when you do itright, you can achieve really significant improvements in accuracy as well as great reduction in costs. I wanted to thank you for sitting here with me over the last 50 minutes. I want to reiterate a few calls to action. So go to octoai.cloud to learn how to use our solutions and endpoints, but also come to our booth. And, uh, so we're located at, uh, this, uh, G7 booth. And we're going to be here today and tomorrow if you want to chat about our SaaS endpoints, about our ability to deploy in an enterprise environment. And also I want to give a shout out to my colleague here, Pedro. If you're curious about all the know-how that goes behind how we optimize our model in production, because our background is in compiler optimization, is, is in system optimization, infrastructure optimization. We've applied all of this to be able to serve our models, you know, with positive margins. We're not doing this at a loss. Allright. We're not wasting our VC money here. We're actually building all this know-how into making sure that AI inference is as efficient as it could be. So there's going to be a talk on that. And also make sure if you, if you get a chance, assuming you've joined our, our, uh, Slack channel, which is the following one. So if you're on the Slack org for the event, go to LLM Quality Optimization Bootcamp. You can ask us any questions. And if you fill out the survey that Pedro's going to post, we're going to give you an additional $10 in credits. Uh, so that doesn't seem like a lot, but that's a ton. You know, if it's 15 cents per million tokens, that's a lot of, uh, tokens that you can generate for free. So we can give you an additional $10, uh, for, uh, filling out the survey, which would should take about, you know, 20 to 30 seconds. So I'm going to be around. And also you can find me at the booth this afternoon in case you have any questions. But I'd like you all, uh, to thank you for sitting through this talk. And hopefully, hopefully you've learned something from this and hopefully you feel like I've, I've demystified this idea of trying fine-tuning on your own. Give this notebook a try, assuming of course we've fixed this, uh, LoRA upload issue. And, uh, yeah, thank you all. And, and maybe ask me some questions after this, uh, after this talk. Thanks. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.