RAG Agents in Prod: 10 Lessons We Learned — Douwe Kiela, creator of RAG

My name is Douwe Kiela, I'm the CEO at Contextual AI, and I'm here to talk to you about RAG in production, RAG agents specifically, and I'll share some of the lessons that I've learned. So my background is in AI research, but after that I became the CEO of an AI company focused on enterprise, so I thought I would share some of my learnings with you in the hope that that's useful.

So if you look at enterprise AI, if you work in this space, you've probably noticed that there's a huge opportunity ahead of us,right? Everybody wants to grab that opportunity. There's these huge numbers flying around: $4.4 trillion is the estimated added value to the global economy according to McKinsey, so we have this giant opportunity.

But at the same time, if you actually look at what's happening in enterprises, you see a lot of frustration. And it's probably even true for some of the people in the audienceright here. If you're a VP of AI, then you're probably under some pressureright now.

It's like, where's the ROI? We're investing all this money in AI, but where is it actually leading us to? Are we getting something out of this? So Forbes has this interesting study where they showed that only 1 in 4 businesses actually get value from AI.

So why is that happening,right? It feels a bit like a paradox. So to explain it, we can look at a paradox that might be familiar to you. It's something called the Morvex Paradox. It's from robotics. And in robotics, they were very surprised when they found out that it's actually much easier to beat humans at chess than to have a robot that can vacuum clean your house or have a self-driving car.

So the paradox here is really that things that seem hard are actually much easier for computers than you would expect, and things that seem easy actually turn out to be much harder,right? So there's something very similar happeningright now in enterprise AI specifically, and this is around context.

So on the one hand, we have these amazing language models,right? That's why we're all here, basically, because we see this revolution happeningright in front of our eyes. So language models can generate code much better than most humans. They can solve mathematical problems much better than most of us here can do, and we're pretty smart.

So it's really amazing what they can do. But one of the things that they really still struggle with, and that's one of the things that as humans we are very good at sort of without effort, is putting things in theright context,right?

So as humans, we build on our expertise, we build on our intuition that we've developed over the years, especially if we're a specialist. This is something that is very easy for us to do, is to put something in theright context and in theright situation so that you can make sense of the information or the problem that you're solving.

So I would argue that this is really the key observation, this context paradox, for unlocking ROI with AI. And the reason for that is that where we areright now here is in the bottom left,right? So we're mostly focused on convenience.

We have general-purpose assistants. They're very useful, mostly if you're lazy. They help you sort of solve your problems faster. But where you really want to get to is differentiated value. If you're an enterprise, it's nice that you can make things more convenient.

You probably can make people more efficient and more productive. That's great. But where you want to get to is this business transformation ideal,right? That's what all the CEOs are probably telling you as a VP of AI, like, "I want to change my entire business.

How am I going to do that?" So getting to that differentiated value, that's where you want to get to. But the problem is that the higher you go on that axis, the further you go on the context axis.

So the better you need to be at handling the context that exists within your enterprise. So what should we do about that? So that observation is really why I started the company that I'm currently the CEO of, Contextual AI.

And we started this two years ago to try to help bridge this gap, and we've learned some lessons along the way that I thought I would share with you in the hope that they're also useful for you. So the first observation is really that language models are awesome, but often they're only 20% of a much bigger system.

So if you have an enterprise AI deployment, usually that means it's a RAG system. So I think everybody here probably has heard of RAG. RAG is something that I originally pioneered with my team at Facebook AI Research when I was there.

So RAG is really kind of the standard way that you get Gen AI to work on your data. So what happens very often these days is a new language model comes out, everybody goes, "Whoa, new language model. It's great."

Everybody starts to think just about the language model, but very few people actually think about the system around the language model. And that system needs to solve the problem,right? So you can have a relatively mediocre language model but an amazing RAG pipeline around it, and that's going to be much better than an amazing language model with a terrible RAG pipeline around it.

So the basic observation here, or the lesson, is that you should be thinking about systems, not about models. The model is only a small part of the system, and the system is the thing that solves the problem. The next observation is that if you're in an enterprise, expertise is really your fuel,right?

So one of the things that you want to be able to do as an enterprise is unlock all of that expertise. So you have all of this institutional knowledge in your company. How do you get it out? So one way to try to do that is using these generalist kind of general-purpose assistants, but it's very hard to get them to match the expertise of people in your company.

So ideally, what you want to do is to specialize so that you can capture that expertise much better. So at my company, we call this specialization over AGI. AGI is great. There are lots of use cases for it.

If you really want to solve a very difficult problem that is very domain-specific, where you understand the use case, you want to specialize for it, and you'll get much further. So that's, I guess, pretty counterintuitive if you look at the sort of broader interests,right?

Most people are much more excited about AGI, but solving real problems is much easier with specialization.

The next lesson is, at an enterprise scale is your moat. So if you think about what a company really is, is a company maybe its people? Probably a little bit,right? But over time, what the company really is, or what makes a company a company, is its data, because even people are transient,right?

So the data that a company owns that is the company in the long term. So now, as an enterprise, you need to think how you can unlock all of that potential,right? And so one of the big issues that we see a lot is that enterprises think that you need to scrub the data and clean it and invest a lot of time in making your data accessible with AI.

But what you really want to do is make sure that AI can work on your noisy data at scale. And doing that is incredibly difficult, but if you succeed in doing that, that's how you get to differentiated value,right?

That's how you get that moat, because the data makes your company your company, and so that data is really your moat.

One observation, and this is really a hard truth that we've learned and I think that many of you might have learned already or that you're about to find out earlier in your journey, is that pilots are very easy.

Building a demo is not very difficult these days,right? If you want to build a RAG system, you take one of the frameworks, you put in some documents, you have a working solution, it's great. You give it to your 10 users, they all tell you it's fantastic, and then you show it to the CEO and he says, "Okay, we're going to fire half the customer support team and we're going to replace them with AI, and we're going to do that in three months."

And now you're on the hook for productionizing something that is actually much, much harder,right? So getting this to work at tens of thousands or hundreds of thousands or millions of documents, you can't do that with any existing tools that are out there on the open-source market.

It's very, very difficult to do that. Making this scale to thousands of users is very hard. Making it work for lots of different use cases, if you're an enterprise, maybe you have 20,000 different use cases that you want to cover.

So how do you scale if that's the problem that you're solving? And then there's, of course, enterprise requirements around security and compliance. So bridging that gap is much harder than you think, and theright way to deal with that is to really focus on production from day one.

So don't design for the pilot, design for production, and that can save you a lot of time. And that brings me to the next observation, is that speed is really much more important than perfection. What we see in terms of production rollouts of RAG agents, it's all about speed.

And what that means is you need to give it to your users relatively early, real users, not sort of testers who are kind of friendly. You want to give it to real users to get their feedback. You want to do that early.

It doesn't have to be perfect. It just needs to be barely functional. And if you do that, then you can hill climb to actually get to this level where it's good enough. If you don't do that and you wait too long and then you try to design something that is perfect, it's going to be very hard to bridge that gap from pilot to production.

So iteration is really the key to a lot of successful production AI deployments in enterprises.

Next observation is related to this too, which is that if you want your engineers to be fast and if you want to follow that speed maxim I just talked about, then you don't want them working on boring stuff.

Sounds kind of obvious, but it turns out that engineers are working on a lot of very boring stuff. And so one of the things that they have to worry about, for example, is what is the optimal chunking strategy for my RAG system?

And it's different for every use case and it's different for every framework. And then they have to think about what theright prompt is, or really basic things that ideally they don't have to think about too much, because you really want your engineers to think about how am I going to deliver business value,right?

How do I make sure I have this differentiated value and that I'm actually better than my competitors? So make sure that your engineers spend time on the things that matter and not on the chunking strategy or things that can be abstracted away very well these days by state-of-the-art platforms for RAG agents.

Next one is about making AI easy to consume. So what I mean by that is we actually see this happen quite often where companies have Gen AI running in production. And then the next question I often ask them is, "Okay, how many people are actually using it?"

And surprisingly often the answer is zero. Almost nobody's actually using it. They did all this work, but they had to make sure it came through sort of model risk and teams like that. So it was really like kneecapped almost, and now it's barely useful.

So that's one scenario. Or very often people just don't actually know how to use the technology. So it really is a journey that you are on, and the easier you can make your solutions to consume, the better it is.

And what that means for most enterprises is not just thinking about your enterprise data and how you make AI work on it, but also how you integrate it into their workflows. So the closer you can integrate it into a workflow that already exists in your enterprise, the more successful you are going to be with real production usage.

Next one is related to the previous one as well, where it's really about getting usage. It's about sort of being sticky. And so this sounds maybe a little obvious, but the quicker you can wow users or get this sort of spark where they suddenly get it, for me as a CEO of an AI company, that's really the special moment when people suddenly go like, "Wow, I didn't know that it could do this."

So you can try to design your experiences for onboarding users around this observation too,right? So where they get to the wow as quickly as possible. So for us, we had this really nice example with someone at Qualcomm. So we're running in production globally with Qualcomm with thousands of customer engineers, and one of them became so happy when they found this document.

It was seven years old. It was hidden away somewhere. They didn't know it existed. They had all these questions, and they just never knew what the answers were. And suddenly, because they asked our system, they got these answers, and their world was never the same again after that.

So these are the small wins sort of that really matter for evangelizing production in AI.

So that brings me to the penultimate learning, which is that it's not even really about accuracy anymore. So accuracy is almost table stakes,right? So I think as AI practitioners, we probably know that getting 100% accuracy is very hard, if not impossible.

Getting 95% accuracy, maybe you can get there, or 90%. But what enterprises are thinking about much more these days is what about the missing 5%? Or what about the missing 10%? How do I deal with the things that might go wrong,right?

So there's a minimum requirement for accuracy, but beyond that, it's really about inaccuracy. And the way to deal with that is through observability. So you want to be very careful with how you evaluate these systems. You want to be very careful with making sure you have proper audit trails, especially if you work in a regulated industry.

This is incredibly important,right? Making sure that you have an audit trail that says, "This is why I generated this answer. It's because I found it here in this document." Basic things like that. So attribution, essentially, in a RAG system actually becomes very, very important for dealing with the inaccuracies.

And similarly, what you can do is you can check the claims that your system generates. So do a lot of post-processing to ensure that you have proper attributions that you can really back up as evidence.

Struggling with the clicker a bit. So final one that I want to end on, and this sounds maybe a little bit cheesy, but it really is true, is be ambitious. We actually see a lot of projects fail not because people are aiming too high, but because people are aiming too low, where folks are going like, "I have Gen AI running in production, and then what does it do?

It answers basic questions about who your 401(k) provider is or how many days of vacation I get." That's not really where the ROI of AI is,right? So you want to aim for really ambitious things where if you solve them, you actually have ROI and you don't just have a gimmick that people don't really use anyway.

So try to be ambitious, because we really live in special times. We have the astronaut here on the slide. So I think it was a pretty special time to be alive during the moon landing and when all of that happened,right?

We're in a similar momentright now where AI is really going to change everything. It's going to change our entire society in the next couple of years. And so you have an opportunity, being in the role that you're in, to really affect that change in society yourself.

So be ambitious when you do that, and don't aim for the low-hanging, easy fruit. Aim for the sky. So that's really what my lessons were for you here. This context paradox is not going away, but by understanding these lessons that I shared with you, hopefully you can turn some of these challenges that we see everywhere in enterprise AI into opportunities for yourself.

So it's really build better systems, think about systems, not models. Focus on your expertise and specialize for it. Don't settle for general solutions. Specialize for the expertise that you have in your company and be ambitious, and then you'll be very successful.

Thank you.