Practical tactics to build reliable AI apps — Dmitry Kuchin, Multinear

Welcome, everyone. I'm going to talk about practical tactics to build reliable AI applications, and why nobody does it this way yet. Uh, a little bit about myself, or why you should trust me. I have about 15 years as a startup co-founder and CTO.

I held executive positions for the last 5 years at several enterprises, but most importantly, I spent the last couple of years developing a lot of Gen AI projects, ranging from POCs to many production-level solutions, and helped some companies to get it done.

And I've learned, or distilled, a way to make these applications reliable. And there are quite a lot of tracks this

conference about evals and reliability, but to my surprise, nobody was talking about the most important things, and we're going to talk about itright now. So, standard software development lifecycle is very standard, simple. You design your solution, you develop it, you test it, and then eventually you deploy it.

And when people start doing POC with AI, it sounds simple, like you can very easily do some prompt and models are very capable, but then you start facing some unexpected challenges. Actually, like, you can easily do a POC that works 50% of the time, but making it do the same reliable work the rest of the 50% is very hard because models are non-deterministic, and it starts requiring a data science approach, continuous experimentation.

You need to try this prompt, you need to try that model, you need to try this approach, etc., etc. And everything in your solution, everything that represents your solution, which is your code, your logic, the prompts that you use, the models that you use, the data that you base your solution on, anything of that impacts your solution in unexpected ways.

People very often come to this to try solving this with the wrong approach. They start with data science metrics. Like, it sounds reasonable,right? So it requires data science approach of experimentation, and people start measuring groundness, factuality, bias, and other metrics that don't really help you to understand, uh, is your solution working theright way.

Does it, does your latest change improve your solution in theright way for your users? For example, I've been talking to an ex-colleague that are building a customer support bot at Wix. I asked him, "How do you know that your solution is working well?"

He started talking about factuality and other data science metrics.

That's, again, I started to dig deeper, and then we just, together, figured out that the most important metric for them is

the rate of moving from AI support bot, like escalation, to a human support. If your solution hasn't been able to answer the user with all this factuality, like it could be super grounded but still not provide theright answer that the user expects, and this is what you actually need to test.

And my experience was to start with real-world scenarios. So basically, you need to reverse engineer your metrics, and your metrics should be very, very specific to what your end goal. So they should come from a product experience, from business outcomes.

If your solution is a customer support bot, you need to figure out what your users want and how you can mimic it. And instead of measuring something average or something generic, you need to measure very specific criteria, because universal evals don't really work.

How do we do it? So, for example, customer support bot, which is, by the way, one of the hardest things to do evals properly. Let's say I have a bank, and the bank has FAQ materials which contain, including, like, "How do you reset your password?"

So what I usually do when I help my, like, companies that I help them to build AI solutions, we start with reverse engineering, like, how do we create evals based on that? So in this case, I use LLM, and in most cases I use LLM to come up withright evaluations.

So here I can take, say, O1, o3 now, and just reverse engineer what should be the user question that we know to answer based on these materials, and what should be the specific criteria that these materials provide an answer for.

And some of these criteria are quite important. So, for example, here it says that as part of the thing, you need to receive a mobile validation, so you receive an SMS code, and it says that if you don't have a mobile number, then you can reach support, etc., etc.

If some of that information is missing from the answer, the answer would not be correct, but you need to be very specific about what exact information you need to see in the answer, and that information is very specific to that specific question.

So you need to build, like, lots of evals

from the materials, in this case, that mimic specific user questions that you need to be able to answer for. How do we do it? Usually, again, I work with smart models, like o3, and I provide enough context, I provide which personas are we trying to represent, because you can ask the same question in completely different ways depending on who is the persona asking, yet you would expect exactly the same answer, so you need to account for it.

So this is an example from the open source platform that we have that just helps to get it done. So if you look it up, Multineer, I'm not trying to sell you anything, I'm not trying to, like, vendor lock-in or whatever, it's completely open source, and if needed I can just recreate it in a couple of days now with Cursor.

The point is in the approach, not in the platform. So, for example, here we see that very same question, "How do I reset my password?" You see the, what was the input, what was the output, and that specific criteria that I measure it, that specific question, "How do I know if the answer is correct?"

And now I can just reiterate and generate, like, 50 different variations of the same question and see if I still get theright answer, if the answer matches all the checklist that I have for that specific answer.

How the process usually works. So, contrary to, like, regular approach, you build your evals not at the end of the process, but in the very beginning of the process. So you just build your first version of the POC, you define a first version of your tests, evaluations, you run them, and you see what's going on.

You will see that in some cases it will fail, in some cases it will succeed. What's important is to look at the details, not just see the average numbers. The average numbers won't tell you anything, won't tell you how to improve it.

If you actually look at the details of each evaluation, you'll see exactly why it's failing. It could be failing because your test is not defined correctly. It could be failing because your solution is not working as it should be.

And, like, in order to do it, you may need to do a change in, like, you may change a model, you may change something in your logic, you may change a prompt, or the data that you use in order to answer a question in our example.

And basically what you do now is experimentation. So you start running your experiment, you change something, you need to define these tests in a way that will help you to make an educated guess on what you need to change in order to do it.

In some cases it will work, in some cases it won't. But even if it works, let's say you change something in your prompt and it fixed this test. In my experience, in many cases it breaks something that used to work before.

Like, you have constant regressions, and if you don't have these evaluations, there is no way you'll be able to catch it on time. So this is hugely important, and what actually happens is that, again, you build your first version, you build your first version of the evals, you match them, you run these evals, you improve something, you improve your evals, or maybe add more evaluations, and then you, like, continuously improve it until you reach some point where you are satisfied with your evals for this specific solution for that specific point of time.

And what actually happened is that you got your baseline, you got your benchmark that now you can start optimizing, and you have the confidence that the tests should be working. So now you can try another model. Let's say, well, what, how can I try to see if 4o mini will work the same way with 4o or not?

Can I use the GraphRAG, or can I try a simpler solution? Should I have to use the agentic approach that, like, maybe better but requires more time, more inference cost, etc., or should I try to simplify the logic, or maybe I can simplify the logic for a specific portion of the application, etc., etc.?

Having this benchmark allows you to do all these experimentations with confidence, but again, the most important part is, like, how do you reach this benchmark. And while the approach is pretty much the same, the evaluations that you need to build and how do you build your evaluations are completely different depending on the solution that you need to build, because the models are super capableright now, so they allow you to build a huge variety of solutions, but each and every solution is quite different in terms of how do you evaluate it.

For support bot, you usually, typically use LLM as a judge, as I made an example. If you're building text to SQL or text to Graph database, then to my experience the best way is to create a mock database that represents the, whatever, database or databases that you need your solution to work with.

They represent the same schema and you have the mock data, so you know exactly what you should expect on specific questions. If you need to build some classifier for call center conversations, then your tests are, like, simple match whenever this is theright rubric or not.

And the same approach applies to guardrails. So getting back to the support, to the example of a customer support bot, guardrails, you need to cover questions that should not be answered, or questions that should be answered in different ways, or questions that the answers are not in the materials.

So all of this you can put into your benchmark, just different type of benchmark, but it's pretty much the same approach.

So just to reiterate the key takeaways, you need to evaluate your apps the way your users actually use them, and avoid abstract metrics, because these abstract metrics don't really measure anything important. And the approach is through experimentation, so you run these evaluations frequently, you, that allows you to have rapid progress with less regressions, because testing frequently helps you to catch these surprises.

But most importantly, what you get if you define your evaluations correctly, you get your solution pretty much as kind of explainable AI, because you know exactly what it does, you know exactly how it does it if you test it theright way.

Thank you very much. Take a look at Multineer, that's a platform that you can use to run these evaluations. You can totally use any other platform. The approach is quite simple, it doesn't require any specific platform. I've built Multineer just because no other platform helped me to do it this way, to help me with the process of evaluation, like, end to end.

I'm working on a startup that does reliable AI automationright now, and yeah, thank you very much.