Understanding AI Stakes to Break Production Code: Philip Rathle

So the idea here is: a lot of you are solving some of the most leading-edge AI, GenAI problems in the world, in this room, and then a lot of you are trying ambitious things. So the main goal was to get folks learning from each other, so think of this as a logical roundtable—not very round, physically.

So come away with insights about specifically your getting-to-production challenges by learning from each other. So two rules of this AI roundtable: one is, we'll all be stronger if we figure this out together. Number two is chatham house rules—that's for Peter's benefit.

For us Americans in the room, what happens in knob Hill stays in knob Hill.

So let me actually kick it off by sharing some of my personal learnings. I'll go through maybe

five minutes or so of what I've learned in relation to getting to production, but specifically around the topic of stakes. And by stakes I mean, like, how high stakes is the problem that you're trying to solve? And I found there's a relationship between the bars and the obstacles that you run into as the stakes increase.

So I'll just share my own journey, and hopefully this is useful and food for thought. But as I'm going through it, think about what you're experiencing and the hurdles. Because what I'm going to ask after I go through my own section is: what are some of the hurdles you are running into?

What are some of the ones that you haven't solved? What are some of the ones you have solved? So that we can learn from each other. Allright, so it all started like this: GenAI is a hockey stick,right? And then we kind of ended up with actual project experience where you can do—and I think Lucas said this earlier—lots of amazing prototypes.

It's so easy to build an amazing prototype, but then getting to production is harder. You have a higher bar. And so what a lot of folks have figured out is you can use RAG with vectors and, allright, now it looks like a hockey stick.

And in some cases it does, in some cases you can get to production with this. But then there's still a class of use cases where that's not enough. And so what you'll hear from the Neo4j folks here, and if you go to the booth and attend Emile's talk and some of the other ones later, is there's another unlock here with knowledge graphs and with Graph RAG.

Again, like, will this be a hockey stick? I think the world moves in S-curves, so this is probably more realistic. And there's probably going to be some other thing. I don't know what that looks like. But why this pattern?

So I've come to see a spectrum of application stakes, and it's a whole spectrum,right? But to kind of paint opposite ends and maybe something in the middle. If something is low stakes, then maybe it's more business peripheral. It's things like summarizing text.

Here I have someone, you know, a children's bedtime story. There's no singleright answer. The stakes are a bit lower. And then on the high end of things, like a wrong answer can have significant impact in terms of dollar value, brand and reputational impact, health and human safety, regulation, fines, bias, all that stuff.

And then there's this in-between zone where, you know, it might be higher business stakes, but I've got some mitigations. Like I've got a human in the loop. So here you have a difference between at the farright, pilot, and in the middle, co-pilot.

So there's kind of an interesting relationship between pilot, co-pilot, and stakes. And, you know, the problem with high-stakes use cases is you bump into the meme,right? Which you've all seen. And specifically it's things like needing enterprise domain knowledge up to the moment knowledge.

This is what RAG solves. Solving for hallucinations. Again, RAG solved this, and depending on how you solve it, you're loading the dice, loading the dice, loading the dice. And then depending on the stakes, your tolerance for error, you know, could be, you know, potentially gets very high,right?

How tight should I tighten my bolts on my 737 MAX 9? Like, let's not have a 0.001% chance of error because that's not good enough.

Cool. So let's look at how we're representing data. So here's an apple, and think about how we conceptualize an apple as a human. It's, you know, this multidimensional, multisensory kind of thing. Now let's look at a vector view of an apple.

It looks something like this. So I can find other fruit by doing a, you know, cosine similarity or spatial distance operation with this representation of an apple. It's useful. Again, good for certain things, not as good for other things.

And if I characterize the behavior that you get with LLMs generally and with vector-based RAG, it's this, you know, kind of world of statistically computing the real world based on word proximity statistics and word frequency statistics. So you get this behavior that's kind of a little bit like theright brain.

You know, it's creative, it's mostlyright, sometimes really, really wrong. I don't really know why it's wrong. It just does its thing. Bit of a black box. Enter knowledge graphs as a way to bump up the stakes on things.

So this gives you a more structured representation that is understandable by humans, can be reasoned upon by machines. And this gives you a bit more of left-brain behavior. So it can't do all the things that LLMs can by any means, but when you combine it with an LLM, you're being able to bring in facts, reasoning, discernment, long-term memory, can create a better together.

So that's my quick sharing around stakes and unlocking things and different levels. So let me turn it over to folks in the room, and Peter will be coming around with a microphone. So let me start with a question.

Does anyone have what's one learning that would have saved you a lot of time if you'd known it sooner?

Come on, all at once.

Anyone just raise your hand. Blurt it out if you're.

Well, maybe if no one's got one, maybe you have one, like just off the cuff. It might inspire other people to have a story, get the scope.

I mean, it comes back to stakes and, you know, understanding what those are up front for different applications. So we've heard a lot today about iteration and how important that is. So definitely iterate. You know, don't overthink things initially.

But, you know, it's also. Think and plan ahead. And that can guide things like what tools you use. Also, locus of reasoning is something I think about. If there's something, if there's a question that has an exact answer and it's a deterministic question, then, you know, maybe you can use the LLM, which is inherently non-deterministic, to complement some other technology that can give you a deterministic answer, which you have.

You've got databases and other kinds of reasoning engines. And then use the LLM to do what it's exceptionally good at, which is maybe take the question, turn it into a query, take that query, formulate it back into some text, and then maybe modulate that text depending on who the audience is.

If it's going to a regulator, then very serious, no jokes. If it's going to, you know, an outbound marketing email, then it can maybe go to be phrased differently depending on who the target audience is. Is it someone in high school?

Is it a retiree? Yes, I see a hand though, over there.

Oh, cool. Let's go. Here we go. Ah, so I'll go here.

I have one learning that I wish I had had before. Instead of asking the LLM to solve a problem,right, like paste in Excel and say, which one's the outlier? Just ask it to write the code to solve that problem, which it can easily do.

And then you can leave the LLM out of the loop,right? You say, find the outlier. We'll do some Python outlier machine learning thing, and then you have the answer.

Cool, yeah. So the LLM might not be good at certain kinds of reasoning, but it can write code that will do the reasoning, which is a pretty amazing hack, actually. Thanks. I think you're.

Oh, everyone's getting involved now.

You.

Oh, wait, I'm sorry. Who had their hands up? Oh, let's go here.

So one of the things we learned was when we were getting excited about RAG and indexing and doing the semantic search, and when we put it to production, that our customers were not used to doing queries like this.

They always resorted back to keywords. It's almost like they were trained to do this. And so getting them along. And so we had to rewrite, like, is this question actually a query for keywords, or is it like a query they ask, like a question?

So that was a learning that, you know, we made while putting it into production. And we put a lot of effort in making the search well, but we didn't ask up front, like, well, is this actually, are you going to use it?

So that was one learning there.

And so did you have a sort of curriculum to help people understand how to ask questions and that they can do more than just keywords?

So, yeah, we didn't put attention to this. We were just assuming, like, yeah, everybody's now going to use, like, putting questions in. So that was, you know, the surprise and the learning there. Like, we needed to ease them in.

Need to understand what the technology is capable of and that can do more than just what you're used to.

Adapt to user, to how the new thing is supposed to work, in a way. Like, that was a learning.

Cool.

I think the lesson is skill more projects than experimentation. Soright now, I think one of the big lessons we learned is we did a lot more experimentation. And if I have to look back, I will focus on few that will go all the way to production.

Because a lot of these experimentations, even with the best effort, will never meet the user expectation on accuracy. So if we can kill a lot of projects but focus on one or two that we can take all the way to production is probably where I will focus a lot more.

And people spend too much time on not being focused.

And when deciding what project to focus on, I imagine there's an element of how achievable is it and how valuable is it. How do you.

Can

we get to the level of accuracy that the user is expecting?

Yeah.

Optimization expenses. We can go to 90, 95. That's not good enough. Why would you spend that money? What are the use cases that will go to that where 90% is good enough?

Yeah, so matching up with user expectation of accuracy, and then those projects are bound to succeed and go with those. Cool. Thanks.

My learning is expect less and use more common sense.

Yes.

It's a funny one.

Yeah.

I like that.

One learning that I've experienced from our customers when they're building deployments is, you know, it's really easy to walk in and say, I want the best model, and I want it to run as fast as possible. But when you're making, you know, actual cost-to-quality trade-offs and cost-to-speed trade-offs, you know, you can really overspend versus what you really need if you don't have a great understanding of what you're actually trying to build, how fast it actually needs to run, that kind of stuff.

Yeah, yeah. In a lab with a small number of questions, you're naturally going to go for the best accuracy, but then enter production. If it, you know, if you're burning a year's budget within a day, which is very possible with this stuff, then maybe you need smaller models or multiple.

Yeah, great one.

Actually, I just had one pop up in my own mind that I remember seeing, which was when Bloomberg were training, I think it was LLaMA 2 at the time, to handle a lot of the financial data that they had.

And it passed really well on all the evals and everything. And then they found out when LLaMA 3 came out, it actually outperformed their fine-tuned model they spent all this time producing. And so I'm not entirely sure what the takeaway from that is, but it's almost this idea that if you just wait for the models to get better, then you kind of save work.

But then I guess that's not necessarily going to fly in the enterprise where it's like, we want something working today.

Well, this is a great I'd love to hear what people in the room are experiencing. Because there's this real tension of, allright, if I just wait a couple years, like this stuff is all going to figure itself out and it'll be way better.

But who wants to wait a couple of years? Like we want to get out and get competitive advantage.

We're seeing crazy advances.

Yeah, yeah. You wait a few weeks.

We're just focused on like the full path and getting an end-to-end and then like in parallel tracks, optimizing every part of that journey,right? So trying to end-to-end, shipping customer value, and then, oh, our data engineering part of that was like a clutch, and that's not going to scale, so let's get people working on that.

And it's like an interface between every step of this chain. And so like if I encounter an issue where my LLMs aren't performing and it's like, oh no, we got to use like 3.5 Sonnet and it's going to cost us a lot, we'll just wait.

You know, it's going to come down a little bit.

Yeah.

And just keep optimizing as these improvements happen across that stack.

Yeah, yeah. The bars I was showing earlier are naturally all rising, like across all those dimensions. And if you understand what the pipeline looks like, then you can just drop stuff in,right?

You have your pipeline first.

Yeah.

And then you can just iterate on each one of those components.

Yeah.

That's what's been working really well for us.

Cool. I like so one question to drill into that is, I can see pipelines getting more complex with more agentic stuff and so on. Like how do you like that one is a moving target too. How do you see handling, how are you handling that?

That's not something we've figured out yet. Honestly, looking for better tools to help orchestrate that.

Yeah.

We are making improvements on various parts of this thing. But like I've had to slow down the team. Like you're making too many changes at once now across the different things. So it sounds like we have to slow down a little bit with interfaces and contracts between the layers so that we can iterate on one side and not affect the downstream on the other.

So I don't know that yet.

Cool. Great. Anyone else on this question?

Just to piggyback.

Yes.

When you're thinking of the latency improvements you can get, if you have a solution that might take you months to implement, you have to evaluate that with the trade-off and say, by the time we finish this, will a new model come out?

So I think the time it takes to implement these solutions, there should be like a max, like a ceiling. Like if it takes more than however many months, you're just wasting your time because it's going to be throwaway.

I love that. Yeah. If it's a two-year project, then why bother? Like anything you do now is going to be irrelevant.

Right. I need another solution, basically.

Yeah. Yeah. Love it.

My dash over here. Yeah, wait. Sometimes work.

Thank you. Yeah, one all of that I really resonate with. And one thing that we've seen is companies that have succeeded have avoided premature optimization. So obviously, you want to be sort of aware of latency in production and, you know, make something that will be realistic.

But the teams that have sort of started with that and gone, we're going to use a small fine-tuned model, often incorrectly conclude that a task isn't possible. And also, people are often able to take data from a larger model and fine-tune it later.

And so I always think of like moving down the model size stack or fine-tuning as optimizations. And you want to validate the product need and whether it works, whether it's possible, usually with a better model first. At least we've seen that be much more successful.

That's a great I've heard that too, like start with GPT-4.0 or I guess now maybe Claude 3.5. And if that can't do it, then the smaller models probably won't. And I guess you have an exception for data that's highly proprietary, and that's where fine-tuning and RAG come in.

Yeah. I think the same thing. Just optimize the best out of the box, and then we kind of slowly bring it down to more performant, cheaper models. Hopefully, we're trying to get the same efforts to get out.

Cool.

I think building up on something that you mentioned as well earlier, when, for example, you're using a large language model to turn unstructured data to structured data to then feed it into like traditional machine learning models. One of the key learnings for us was

if we're building our eval pipeline from the get-go, which is really starting to be echoed, you know, trying to figure out what your eval should be, get it ready from the start, is to understand, do you have theright data to build your eval pipeline?

And if not, start from day zero to make sure that you generate that data for your eval pipeline. Because if you leave it to stay too long and then realize, oh, we've built an LLM model and it's in our app, but we cannot evaluate whether the fine-tuning on the prompts makes it better or worse, you're then spending a lot of sort of long-leg time on figuring out what your eval pipeline should be.

So our key learning was really start as you start building your app, also start with your eval pipeline and make sure that you have the data that's there.

Cool. That's maybe a corollary of the, okay, you need your DevOps kind of framework, but then you need your eval framework, and then your data framework is closely related to that, is making sure you have theright data. Yeah, and the quality is there.

One of the things we learned is not hiding AI from the end user. So we're like explicitly, for example, generating the responses slowly. So as a hint for the user that they should be more vigilant. We're using specific colors in the UI to make it clearer, the icons that they click on.

So kind of make sure that they understand that there is a special thing they need to do. And then when they want to check, we give them as much tools for them to validate the answer that was generated to see what went wrong, what were the sources, what kind of went in there.

So that's one of the lessons that not abstract this away and kind of hope for the best, but.

This is related to your moderate expectations point. If the user can actually see the thing coming out, then that's a trigger that, allright, it's not instant, it's coming from some machine. I need to consider things. Cool. I have I want to ask if anyone

dealing with regulation or something where explainability is a big issue. And if so, how are you solving that?

What sort of industry are we talking? Like a medical something?

Yeah, I could think about medical. Like if you're trying to if the end game is prescribe something or provide some treatment, I can think about equipment maintenance as another one.

We work in the behavioral health space. And a lot of there's a lot of regulation. And the one way we determined to solve it as a kind of a core principle is just always human in the loopright now.

Like we're only in the co-pilot stage. In fact, I tell people, we'll never get to the pilot stage, but that's just to not scare them. But we're very focused on co-pilotright now and making sure there's a human in the loop.

But as part of that, we have to give transparency to the inputs that generated the outputs so they can kind of see things side by side. If we make recommendations or we pick some data out that we think is interesting for a particular case that they're working with, they can trace it back to the root data.

And so do you find the inputs are sufficient for getting the humans in the loop to trust the answer enough? Because I so you've got inputs. Another way to do it is, here are the references that, you know, but then they're not going to read the 350-page papers maybe.

We so if it's doing a lookup, a knowledge lookup, we're showing the source documentation that we find based on their query so they can see it kind of side by side with those citations. If it's doing a generation, so say we're summarizing data that was collected in a therapy session, we show the data side by side with the actual summarization of that.

It's kind of like on a paragraph-by-paragraph basis so that the users have complete control of the resultant summary. They can change it, edit it. It's theirs at that point.

And then are you last question, are you using the

judgment of the human in the loop as like RLHF to improve this?

Yeah, we run our own eval on any of our output with a separate stage just to make sure that like did the input does the output match kind of the inputs? And that's just a quick one. It's also for a safety perspective.

But then we use the whether or not the user edited or and/or just gave us thumbs up, thumbs down.

Yeah.

And then we do our own clinical evaluation at the end on our own with our own team, just making sure like we're just everywhere we need to, we're putting in that eval stage to make sure that we know what we're comfortable with what's coming out.

Cool.

I'm just going to go over here. Just one thing affects whether we can do one more or multiple more is our next speaker here. Because I've not oh, fantastic. Cool. Okay. Unfortunately, this has to be the last one then for now.

But I assume this discussion can continue at lunchtime as well because it seems everyone there's so many shared concerns and stories that people have got in the room. So it would be a great thing for people to discuss at lunch.

Yeah.

Absolutely. Here we go.

I'll keep this short. So a human in the loop is clearly a solution, but that's not always possible. So what we do is give options. So this is in one of our use cases is personal loan recommendation. It's regulated.

And then so we give rather than ever say, this is the best loan for you, we give three options and give explanation for these. And so it becomes less deterministic and less and that's how we take care of the regulation.

Okay, cool. Yeah. So give a few options. And then do you do that directly to the end user or to a human in the loop?

Because we cannot have human in the loop. But I think the three options that we select are clearly say some more logic to it than just coming that's coming out of the LLM.

Cool. Thanks. Well, let's definitely keep this one alive throughout the day as you're mingling with each other and lunch and so on. But thanks. Appreciate the participation.