Finetuning: 500m AI agents in production with 2 engineers — Mustafa Ali & Kyle Corbitt

Hey everybody. Yep, I'm Kyle Corbitt from OpenPipe, and I'm here with Mustafa Ali from Method. We're going to be talking about how Method has scaled in production to over 500 million agents, and basically all the tricks they use to make that actually work.

So a little bit about Method is that we essentially collect and centralize liability data from across hundreds of different data sources. This includes tapping into the credit bureaus, connecting with the card networks like Visa and MasterCard, and just direct connections with the financial institutions and various other third-party sources.

And we sort of aggregate and enhance this data and serve it to our customers, who are typically other fintechs, banks, or lenders. And they use this enhanced data to anything really to do with debt management, so refinancing, loan consolidation, liability payments, or just personal finance management.

Yeah, and at OpenPipe what we do is we help you build, train, and deploy open-source models for actual usage. We also let you use in production your signals you get from users, from the environment, to improve your model continuously over time.

And that's some of the things we'll be talking about, what we did with Method.

Nice.

So one of the early challenges that we faced at Method while coming up with this aggregation pipeline was that some of our customers basically came to us and said, you know, it's really nice that you can give us the balance and payment information on a specific liability for their end consumers.

But you know what would be really nice is if you could also give us some of these liability-specific data points, like the payoff amount on an auto loan or the escrow balance for a mortgage. And we said, OK, let's do some research.

So we go back to some of our data partners and basically ask them, is there anything we can plug into to get these kinds of data points? And what we found was there's really no central API that we could get access to that would allow us to get some of these data points.

And of course, ideally, we would want to work directly with the banks. But having already worked with banks before and just from initial conversations, we realized that it would easily take up to at least a couple of years before getting anything solid done.

And we were an early-stage company, so we want to build for the customer fast. And so that's really what we're trying to come up with a solution that we can just push into production tomorrow.

And so just to get a better understanding of how some of these companies are operating today, the services that they're providing today, how are they doing that in the first place,right? They must be getting that data somehow. So we go back to some of these customers and basically ask them, how are you guys operating?

And what they tell us is it's kind of interesting. So a lot of these companies, they basically hire offshore teams of contractors. And these teams are basically responsible for calling these banks on behalf of the company and the end consumer.

They authenticate with the banks, gather the necessary information. Somebody has to proof-check it. It gets sent back, and then it gets integrated into the financial platforms. And it gets surfaced to the user, is used for underwriting, stuff like that.

And so that's the status quo that we're dealing with here. And when you think about it, that's a very inefficient manual process,right? When you try to think about scaling, it doesn't really scale. It has a lot of problems.

It's expensive because one person can only do one thing at a time,right? So if you want to scale, you basically have to hire more people. And for the same reason, because it's so synchronous, it's also really slow. And I guess the biggest problem with that is also that there's a lot of human error involved.

And you need to hire a team to fact-check it, to proof-check it. And

the worst thing that you're going to end up with is to surface basically inaccurate financial information. And so conceptually, though, if you think about it, it's kind of like an API,right? You have the request component. You have the authentication component.

You have the response, validation, and all that stuff. So essentially, when you drill this problem down into the core problem, that's really just trying to make sense of unstructured data,right? So if only there was this magic tool or software that we could use that was really good at parsing unstructured data.

And lucky for us, around the time that we were trying to solve this problem, OpenAI announced GPT-4. And as people like to call it, there was this Cambrian explosion of AI or LLM-enabled applications all around us. And the results were just mind-blowing.

And we thought to ourselves, this is the perfect thing for us. This is like a godsend. So we tried to see if there's anything there that we could use. And if there's one thing that we all know in this room is that advanced LLMs, especially post-GPT-4, are really good with parsing unstructured data.

So tasks like summarization or classification, they're really good with that kind of thing. So we wanted to test that theory out and see what that can get us.

And so we put our heads down, hacked together this agentic workflow using GPT-4. And as expected, it worked really well. So we tried to expand some of our use cases because the API costs are high. So we wanted to get as much as we could from a single API call.

And it turned out to be really good at that. So we tried to obviously, this was in a very controlled manner. But this was in production. And so we were testing out different extractions, basically. And everything was going really good.

But as soon as we started to increase a little bit of traffic, what we found was the bill had to come due. And it was a lot. So $70,000 for our first month in production with GPT-4. And this made leadership really unhappy.

But it was something they were fine with because the value that we were getting out of GPT-4 was so immense. And so we actually kept this thing in production for at least a couple more months as we tried to work around this kind of cost problem.

And cost wasn't the only thing that we were concerned with. As we started to scale some of these use cases, we quickly ran into a wall with prompt engineering. It only takes you so far. One thing we realized is that even though GPT is really smart, it's not a financial expert.

So you had to give it really detailed instructions and examples to really make it work with all kinds of use cases that we were trying to target. So it's hard to generalize those kinds of prompts. They become really long, convoluted.

It's always a cat-and-mouse chase with you fix it for a certain scenario, and it breaks for another one. You fix it for that one, it breaks for the previous one. And so you're always going back and forth. We didn't have any prompt versioning.

So we had to figure out a better way to make this work for all of our use cases.

And so the TL;DR here is that we didn't want to adopt that initial solution that I just talked about earlier in the slides because of its scaling challenges and just because it was so inefficient. But we kind of ran into the same scaling challenges with GPT, where it was expensive because we couldn't really optimize for caching because of the variability in responses and the prompt tweaks we were making all the time.

And the baseline latency that we were finding was actually really slow. So overall, we couldn't scale concurrently. And similar to human errors that were kind of in a different nature, we had AI errors, which were just hallucinations that were hard to catch.

And we just couldn't scale with this kind of system. But we still kept it in production because for specific use cases, it was actually really, really good. And so now the problem shifted from solving that core problem of trying to make sense of unstructured data that was solved with GPT.

Now the problem shifted to how do we scale this system? How do we build a robust agentic workflow that can handle this kind of volume reliably? And so some of the ballpark figures that we came up with is that we're going to be at least making 16 million requests per day.

We're going to have at least 100k concurrent load. And we need minimal latency to handle this kind of real-time agentic workflow. So sub-200 milliseconds. And so the natural next step for us was like, we thought to ourselves, do we buy more GPUs?

Do we host our own model? What do we do at this point? So at that point, OpenPipe comes in.

Yeah, so about a year ago, we started working with Method on solving these issues that Mustafa just listed. And we actually found that those three issues he listed, which are quality, cost, and latency, are very common. These are things that across almost everyone we work with, at least some subset of those are really top of mind.

And so with Method specifically, we were working on, OK, how do we solve those problems in a way that makes this a viable business for you? So the first thing we did was start measuring error rates. Like he mentioned, even AI models are not perfect.

These are all probabilistic systems. Getting to a 0% error rate was not really feasible. But we were able to see different models had different performance characteristics there. So on modern models, on the tasks they're doing, these are the rates we're seeing.

On GPT-4o, we're at about an 11% error rate. And with o3-mini, it's much better. It's around a 4% error rate. The way you measure that is going to be specific to your business. And that's actually true to some extent for all three of these things we'll talk about.

In the case of Method, this is actually relatively easy to measure, luckily, because they have this agentic workflow. But ultimately, what the agent is trying to do is fill out, extract all this information he was talking about, bank balances, things like that.

And so you can have a human go through the flow and figure out what the real number should be. And then you can compare an agentic system's final outputs to that and see if it was successful or not, which made this part relatively easy to calculate.

So these are kind of the error rates we're getting. On the latency point of view, we see that GPT-4o is around a second to respond. And then o3-mini takes about five seconds for their specific task. Again, this is somewhat task-dependent, depending on how much, for example, o3 has to think.

As you're measuring this, you also want to make sure that you're using real production conditions, that you're actually doing a real diversity of tasks that match what you're actually doing, and at a reasonable concurrency level that matches your production.

And we also measured the cost. So again, cost, this is something that is going to obviously be specific. And how much it matters is also very specific to your use case as well. Interestingly, o3-mini, even though it has a much lower per-token cost than GPT-4o, if you just look at the pricing page on the API, for their specific use case, we found it was a little bit more expensive because it generates many more reasoning tokens.

So it has much longer outputs. Again, though, this is somewhat task-dependent. So I just recommend actually, just as an aside, I would recommend once you get to the point that you're trying to optimize, that you have sort of that initial proof of concept with some model, something that works, I think it's really worthwhile.

It can be as simple as literally just writing three different Python scripts that are able to categorize each of these for a different model. And then as new models come out, you'll be able to quickly tell how they're doing.

OK, once you've done, or in this case, once we've done this sort of benchmarking of where the models are, the next question is, allright, where do we need these models to be? Where do we need to get to?

And so again, this is very task-dependent. In the case of Method, they do have extra checks that happen after this, where they look and see, OK, are the numbers that came out plausible? Do they match the types of things we're seeing before, all these different kinds of checks they're doing?

And so they didn't need to get all the way down to a 0% error rate. But of course, those checks are still fallible. And so if it's over a certain point, then some fraction of those errors are going to get through.

And that's going to be bad. So we found around a 9% error rate was able to get them what they needed. From a latency point of view, so the way their agent works is a real-time system. It needs to be able to respond quickly, to move through basically through the whole flow to get the information it needs.

And so they did have a hard latency cutoff. We see a wide variety in this, for what it's worth. We have some customers that I talk to who is like, hey, if I get a result back at some point in the next few days, that's totally fine.

This is a background batch process. We have other customers who are doing real-time voice with a human on the other end of the line. And it's like, hey, if I'm over 500 milliseconds, that's not going to work for me.

And so again, you just have to know for your specific case how much this matters. Same with cost. In their case, because of that very high volume, as Mustafa was mentioning, cost is pretty important to them. Again, depending on your use case, usually mostly dependent on how high volume it is will determine how much cost matters to you.

But you should know these numbers for your specific task as you're comparing different models. OK, so we're looking here at this. Of course, as you're looking at this slide, you may see there's a problem here, which is of the two models we're comparing at least, none of them actually meet all three of the requirements we need to be able to deploy this in production.

And GPT-4o, on both the error rate as well as the cost, we're not quite there. And then o3-mini, on the cost, but especially on the latency, it's just not going to work for what we need. So this is the point at which Method came.

And they talked to us. And we're like, hey, we're not able to hit what we need here because again,

these models aren't getting us where we need to be. So what we work on at OpenPipe is fine-tuning. We work on building custom models for your specific use case. And so I'm going to talk about why you would want to do that and how that helps in this case.

First, I would say fine-tuning is a power tool. It does take more time. It takes more engineering investment than just prompting a model. So you don't really want to do that until you have actually benchmarked the production models, just prompting them, and seeing whether they work or not.

So in this case, in Method's case, and in all of our customers' cases, they found that they were not able to hit the numbers they needed. And so that's the time you want to bring in fine-tuning. So let's look at we were able to fine-tune a model and see how that was able to help because it can actually really bend that price performance curve a lot.

So on the error rate, which is basically just the inverse of accuracy if you want to measure it that way, we were able to get to a place where we were doing significantly better than GPT-4o and importantly better than that threshold they needed.

This used to actually be much harder to achieve. It required a lot of manual labeling of data and things like that. It's actually become much easier over time because of the existence of models like now o3-mini, which allows you to just use your production data.

You can use basically the inputs you're using in production. You can generate outputs for them using a model like o3-mini and train on them. We find, like in this case, that often you're not able to quite get to the performance of the teacher model, the model o3-mini in this case that you're using.

But you can get quite close to it and usually do much better than a slightly less good but much, much larger model. In this case, the model we ended up deploying with them is just an 8 billion parameter LLaMA 3.1 model.

And we find that actually for the majority of our customers, a model that large or smaller is good enough and is able to hit the numbers you need from quality. But yeah, the important thing is to be able to benchmark that and to answer that question for yourself.

On the latency point of view, because actually this is sort of the magic of being able to move to that much smaller model, because we've got this 8 billion parameter model, it is way easier to deploy in a low-latency way.

There's just many fewer calculations, fewer sequential calculations with the number of layers. And so you can get just a much lower latency. You can even and we didn't actually have to do this in Method's case. But something you can do is you can train this model.

You can deploy it within your own infrastructure, co-locate it with the application code that's using it, and even completely eliminate the network latency. And then finally, on the cost front, again, just because this is such a smaller model, you end up with a much, much lower cost.

And so that, for many of our customers, is incredibly important is to be able to get that performance number you need while still maintaining a relatively low cost. In Method's case, we were actually able to far exceed the sort of cost thresholds that they were looking for to make this viable, which means that they don't have to worry about this from sort of a unit economics point of view in the way that they did when they were using the larger models.

So just to sort of reiterate what I started with before, this is a power tool. The fine-tuning, it does take a fair amount of work, not an extreme amount of work, but significantly more work than you do for prompt engineering.

However, if you're not able to get to the reliability numbers you need through just prompt engineering with the models that exist out there without tuning, it is a viable way to very strongly bend that price performance curve and get to a much better place, which can help you get to a very large scale in production, just like Method did.

Nice. So yeah, just to wrap up here, one thing that or at least a couple of points that we want to highlight is that the reason we put two engineers in the title is also because it's not that complicated.

You can get away with using we identified a specific use case. And we got away with just using the cheapest model that was out there. We fine-tuned it. We already had the data from GPT in production. So we already had the data.

We didn't have to go digging around for the data in the first place. So we already used that. And we used the cheapest model that gave us the fastest performance. And you don't need to buy your own GPUs.

And the other thing that we realized is that productionizing AI agents actually requires a little bit of some level of openness and patience from the engineering team, from the leadership team. It's because when you write code, we're always writing code that just works.

You push out a feature and never breaks because you're not changing anything. But with AI agents, it takes some time to get to a point where it's like production ready and actually gives you the responses that you're looking for.

And I feel compelled to say something about to mark the top of the traditional software engineering job. So I'll leave you with these last few words. If you're in SWE, pivot to AIE.

Thank you.

Thanks, everyone.