Forget RAG Pipelines—Build Production Ready Agents in 15 Mins: Nina Lopatina, Rajiv Shah, Contextual

Welcome, everybody. Thanks for showing up bright and early this morning. We're going to make your life easier for RAG, so just hang in there. If you want to get started, we have the linkright there: contextual.ai/ai25. That's going to get you a notebook, it's going to get you the "Getting Started" link, which will get you an API key that you'll be able to use for the notebook.

So feel free to jump ahead and start on those pieces like that. Allright. So let me first introduce the team that's here. So I'm Rajiv Shah. I'm the Chief Evangelist of Contextual AI. So I do a lot of these talks, workshops.

I endlessly talk about AI, which I'm kind of in this position. For background, I've been in the AI space for a while. Before this, I was over at Hugging Face for a number of years as well. So besides me talking, we also have Nina.

You want to introduce yourself?

Hi, I'm Nina. I'll be presenting along with Raj. This is my fourth week at Contextual, so very exciting for me. Thank you all for joining us this morning. I've been working in NLP and language modeling for the past seven or eight years, really since BERT was considered a large language model.

And I'm excited to show you how you can simplify your RAG pipeline with Contextual today.

Thanks. And besides Nina, we also have two other members of our team that are in the back that, if you raise your hand and have questions, they will come to you. So one of them is Matthew, who's a platform engineer.

So if you have questions about how we scale up, what are the things going on in the back end, he's the person for you. We also have John on our team, who's one of the solution architects. So if you're like, "Hey, how would I integrate this into my environment?"

he's the guy who's going to be able to handle that as well. So let's kick this off. What I want to do is for you to all get one of the big pieces out of this is we can treat RAG like a managed service.

We don't go out anymore and train our own large language models. If we're working with embeddings, we don't build our own vector databases. Same thing with RAG. We can treat RAG just like any other managed service. And so what we're going to do today is I'm going to do this quick introduction here, but then we're going to get you startedright away with building this RAG agent with ingesting some files.

And Nina and I will go back and forth, where I'll go and give you a bit more of an overview of Contextual. You'll build an agent, run a number of queries against it, get a feel for it. And then I'll do a deep dive on some of the advanced settings that developers like around extraction, re-ranking, retrieval, like that.

And we'll end with a little bit on evaluation, as well as how you can use MCP, for example, and connect your RAG agent to Claude Desktop as well.

Now, to start with, we all know the value of AI,right? We're all here because of the importance of AI. But for many of you, you also know the struggles of AI, how it's easy to build that demo with 10 documents when you're doing RAG.

But when you go to scale that out, all of a sudden, it's hard to extract across thousands of documents of diversity,right? Your accuracy isn't there. Your users are complaining that they aren't happy with it because they don't know how to query properly.

Now,

we at Contextual have focused on this. So our founders here, Dow and Amman, come from working on RAG for a long time. Dow was an author, was leading the team on the initial RAG paper. Amman was with him at Meta.

I knew both of them from my time at Hugging Face, where I was helping people do RAG years ago on question-answer systems. And I would be able to pull their researchers in to help work with customers as I was cobbling together kind of open-source components to build these pipelines.

Well, they saw the need in that. They want to change how the world works. And so Contextual has come around for focusing on enterprise. And we still have this dedication to AI research. And you'll see that as we go through our product as well.

Now, for again, just the basics, just in case there's somebody in here not familiar with it. When we're talking about RAG, Retrieval-Augmented Generation, the value here is when you have lots of that unstructured enterprise data, we want to be able to understand it.

So RAG allows us to take that information. A very simple RAG pipeline uses a vector database to keep all of that, use cosine similarity to find similar pieces, pass it to an LLM. That's the simplest. We're going to get much more complicated today like that.

And what I want to do is show off our platform to you. The platform is built for a couple of different levels. So one is there's a no-code. So if you're a business user, you just have a bunch of docs.

You want to be able to ask questions to them. We've made it very simple with opinionated defaults on that. Now, if you're a developer, on the other hand, you know what you want in your RAG pipeline. You're spending time evaluating it.

You're spending time tweaking it. Well, we've built a platform where you can then orchestrate and change. How do I want to do querying? How do I want to do generation along the ways? And finally, some of you might already have RAG.

You might already have RAG pipelines in production, where there's just one component that isn't working that well. Maybe your extraction isn't working that well, or you want a better re-ranker. Well, we've made our system modular. So if you just want to use one piece of it, you can.

And all we focus on is RAG. Other companies focus on other pieces. We just think about building RAG. And again, the reason we're here is as you go and try to do this in production, you'll see, once you get to lots of documents, it gets messy.

All of a sudden, then you're orchestrating a big environment full of lots of different models. Now I need a BM25. I need a re-ranker. Putting all that stuff together ends up sucking up your time. It's fun to do the first time.

But after that, it ends up sucking up your time, ends up with slow, kind of costly RAG pieces. So that's enough to get started. I'm going to turn it over to Nina, who's going to get you started on building that first agent.

Allright. We're on the clock. 15 minutes to get our agent up and running. Just kidding. We'll take a little longer than that to explain everything. But how you can get started is you can find the notebook and the Getting Started page at contextual.ai/ai25.

And then the GUI for Contextual is at app.contextual.ai. That's what that Getting Started button will point you to. And so we're going to start by loading a few documents. We're going to load some financial statements from NVIDIA and some funds furious correlations to see how our RAG system treats data that's not fitting with conventional wisdom.

And then we'll try out a few queries, including quantitative reasoning over tables and some data interpretation. And so I'm going to have these two screens here side by side, the notebook on the left and the platform on theright.

So you can get started by signing up in the platform, if you haven't already. That's at app.contextual.ai. That's where you're going to get your API key. So everyone take maybe a minute or so. You'll need to log into that.

And you'll need to set up your workspace. I already have my workspace set up. And I'm actually using

not my work account, just so that you'll see exactly what a general access user would see on their screen on my screen. And just a tip for setting up those workspace names, it's going to have to be a unique name.

So you can't call it ai-demo. I just throw my name into the names just to make it a little bit more unique.

So everyone just take a minute to do that. And to help you get a sense of where folks are, just briefly raise your hand and put it down once you have your workspace set up and you see this welcome screen.

OK. Folks are making fast progress.

Yes, Matthew?

Can we make use of the, especially on the web development? Sorry, on the notebook. Can we make use of the text a little bit?

Yeah.

And maybe also on the platform UI.

Thank you. Thanks.

So yeah. So we can start by, in the notebook, doing our pip installs. Oh, start by making a copy of the notebook. So you can just go to File, Save a Copy in Drive. And then that will create a copy for you of the notebook so that you can save whatever changes you're making to it, if you want to make any changes.

It is a bit slowly making a copy. But yeah, after the pip installs and the imports, to get your API key, you go into your Contextual app, click on API keys here on theright, Save a Copy of the Notebook.

And then in our Contextual platform on theright, you can go to API keys. Click on Create API Key. So Nina, NinaKey2 is my name. And then we'll copy that. And then we can go to our Notebook Secrets on the left here, the key there, and then add a new secret.

And then you'll just add contextual_api_key. Yeah, it's refreshing. But if yours is working, then you can I'll just do it in the notebook without making a copy. contextual_api_key. And then just paste the value there. And then you'll save that.

And yeah, that's all. That's the only API key we need. For those of you that have spun up RAG on your own, it's often an API scavenger hunt just to get started. And this is literally the only API key we're going to use for the whole rest of the workshop.

So we have just finished the hardest part of setting this up.

So once we've saved those credentials, then we're going to set up our client here. So we're just going to run the client in our notebook. So here I'm on the left in the notebook. And then we're going to create our data store.

So you can change the name on this data store name variable. And then it's going to see if there's an existing data store or load that one.

And then after that, which seems to still not be running

OK.

Yeah, after the pip installs and the imports,

that data store is going to be where you're going to be loading all of your documents. And then we have some code here in step two that will download those files that I mentioned earlier to your notebook. So that's going to be a few quarterly reports and furious correlation reports.

And so here in step two, you'll see that it's fetching the data and then uploading it to the data store. And then you can go to your app.contextual.ai. And

this is the ai-demo data store that I made just here. So if you click on your documents, you'll see that they're processing. And once they're done processing, you can click on these three dots. And then once the files are loaded, you'll be able to inspect them and see how those files have been parsed and ingested.

So I'll just take a quick look at the run I did yesterday of all these same exact files.

And so if you click on Inspect once your files are loaded, for example, if you have a table, you're going to see the raw text and then also the rendered preview. So you can see this table has both the structure and the contents of the table in this image below.

And you're welcome to take a look at these. The numbers match very perfectly on the structure. So that's kind of step one of getting your RAG system set up, is having really accurate parsing. And you can look at some of the image files.

We chose some spurious correlations, which are a fun way to test how much a RAG system will actually answer based on your own documents rather than its conventional wisdom. And so you can see there also, there will be raw text and rendered previews with key data points from figures and summaries of the figures in addition to the text and other information extracted.

So while your files are ingesting, Raj will come back and expand on

the platform.

Allright. It seems to me I need to update this slide with Wi-Fi issues as well. But any of you who've built RAG systems before know that there's several components of it. There's a lot of tasks you have to do.

You have to think about building out an extraction pipeline. Besides that, when you're thinking about accuracy, how am I going to chunk out? What am I going to do for chunking? What am I going to do for re-ranking?

And then finally, you have to think about scaling. And there's a lot of these issues that come up. And again, it's interesting the first time you do it. But maintaining this stuff after a while gets tiresome. And that's where Contextual comes in, where it's really an end-to-end platform for managing RAG.

And our platform runs in our SaaS. It can run in your VPC. We have a UI for those business users. But we also have an API, REST API endpoints as well. And within the platform, we manage all those pieces for you.

So you don't have to worry about maintaining a vector store for your embeddings. And the way it works is you'll come with your structured and unstructured data. We bring that in. We have a document understanding pipeline. And that's the extraction part that we're going to walk through here, where we need to cleanly be able to pull all that information out of your documents, your tables, your images.

Once we have that, we'll chunk that up. And then we go through the best practices around retrieving. So we have a mixture of retrievers, so BM25, as well as an embedding model. We have a state-of-the-art re-ranker that we've trained ourselves that's in our pipeline.

And then we pass it all to a grounded language model. Now, we don't use a model from OpenAI or Gemini. We trained our own grounded language model because for RAG use cases, we want the model to be grounded.

We want it not to give its own advice just because it can pass the law exam or know everything about medicine. We don't want it to use its knowledge when it's answering questions. We want it to instead stay grounded, respect the context that we're giving like that.

And finally, there's lots of different ways we can output this. And I'll show you this along the way as well. Now, for each of these parts, we have that academic pedigree, as I mentioned earlier. So one thing we always do is look at the academic benchmarks.

Nowadays, we have lots of customer data set points that we use as well. But whether we think about the end-to-end accuracy of our platform or the specific parts around document understanding, retrieval, grounded generation, each one of those we focused on to make them state-of-the-art for what they do like that.

So once we've done that, when you go to use them, there's a lot of different uses that you can have for this. I think often when we think initially of RAG, we think about, I'm going to build a question and answer chatbot,right?

Like I'm going to ask a question, get back an answer. Well, yes. That's one use of what you can do with this. But we have customers like Qualcomm who use Contextual on their website. So if you go live to their websiteright now and you ask a question there, it's powered by Contextual on the back end.

And you can see similarly the question, the answers, our feedback modules, sources are all available there. Besides that, we have other customers. For example, some folks in the financial spectrum. My audio. And they're thinking about automated workflows. How can I take all the unstructured data I have, structure it to make it useful for folks?

Well, we have APIs. You can hook up those APIs, be able to run them against here.

Is it good?

Ah, OK. No worries. But we can take all that unstructured, use our RAG agents in the context of something like a spreadsheet or some other workflow to be able to use that. Finally, we can connect it with lots of other tools.

I'm going to put this in there, even though I know it's not going to work. But just for all of you, we can show you and I'm going to show you at the end here today how you can, for example, integrate the RAG agents with something like Claude, with MCP.

So that way, you can take advantage of the enterprise knowledge that you have in there.

Allright. So with that, I'm going to hand it back over to Nina. And let's get you started on your agents.

Looks like the Wi-Fi might be down again.

Yeah. It looks like the speaker, Wi-Fi, and my hotspot are not working. So we'll just continue with the notebook. And we'll talk through it. And then I guess, is anyone else's Wi-Fi working? Raise your hand if you have Wi-Fi.

OK. Well, you guys can follow along.

You have the videos too,right?

Hmm?

You have the videos.

I do have. Oh, it's back up. OK, cool.

So yeah, where we were in the notebook on the left here is hopefully all of your

data stores have loaded now. Yes. So assuming you had Wi-Fi to upload them, these are the ones I just loaded in the workshop earlier. And then in the notebook on the left, you can see that you can access these files through the API just the same as you can through the GUI.

And then you can have the document metadata, like when it was created, the name, the status of the ingestion. And now we're going to create our agent. So over here, we have our system prompt. This is kind of just our default system prompt that the agents are loaded with.

And then you can run this next block of code here on the left to set up your agent. You can change the name if you'd like. I called it demo-ai. And then on theright in our GUI, if you click on Agents, there it is.

There's my new agentright here that I just created. And

then you can put in your question based on these documents. So for example, one question that we can ask is, what was NVIDIA's annual revenue by fiscal year 2022 to 2025? And so what you'll notice here is we have our responses.

What we provided was quarterly data. So what the agent is doing is adding that up and then listing it here on theright and then noting that this is based on quarterly data. And then you'll see these little numbers in a circle, the 2 and the 1.

So if you click on the 2 here, you'll actually see the image that this data came from. And similarly, if you click on 1, you'll see the other image. So interestingly, this came from two separate files that were unrelated in any way.

They actually don't even have a standard naming convention for these. I pulled these from the NVIDIA financial reports website. And it knew to reference those documents to pull that information across documents. And then you can do the same exact thing that we just did in the GUI programmatically.

So here in step four, we're running our query. And we're getting the exact same response in the API. So this can make it easier to integrate it into your application. And so we can also here visualize which files were used to reference in the API.

And so now we're printing the documents here. The base64 encoding is getting returned with another API query. So just because it's a more fun and visual demo, we'll ask the next few questions via the GUI here. So NVIDIA used to be a gaming compute company.

So I'm going to ask the chatbot, when did NVIDIA's data center revenue overtake gaming revenue? And so the crossover point was in Q1 fiscal year '23 with data center revenue at 3.7 million and gaming revenue at 3.6. And then here I can click on the one here.

So that's Q1 fiscal year '23. And we can look at our slide here. And here it is. That's actually the exact crossover point where data center revenue overtook gaming revenue. Now, we're going to look at the spurious correlation files that we've also loaded.

So for those of you unfamiliar, this is a website that basically data crawls and finds correlations between things that have no correlation. So we're searching for what's the correlation between the distance between Neptune and the sun and burglary rates in the US.

That would be some extreme astrology there if that actually determined burglary rates. And so what you'll find is that our RAG system that's really, really focused on avoiding hallucinations and really following the data that you've loaded, it will give you that correlation coefficient, which is very high.

But then it will also share that context. So elsewhere in the document, they talk about how they're doing data dredging and that this is not really a valid statistical correlation. And so you can then also reference where in that document these claims were made, both the statistical correlation

and the caveats, which are not loading, I think, from Wi-Fi issues. But you can test this out in your own instance as well. We have another fun one that was, what's the correlation between global revenue generated by Unilever Group and Google searches for lost my wallet?

That's another spurious correlation that we loaded. And it will do the same thing. It will give you the actual answer and then note the caveats. And I'll note that if you run these same queries in ChatGPT here on theright, if you just ask the question without any files, not doing RAG, it will just start with telling you there's no meaningful correlation.

We know that's true. But think about if your own documents don't follow conventional wisdom or have some information that's not out there. It's going to argue with the information in your documents rather than presenting it, even though it may not actually be true in this case.

And then if you do a document upload and use the long context in ChatGPT, you will get that response of the actual correlation. But then it will just say it's a spurious correlation. And it won't really go into why.

So it's not really going to a simple system like this is not really going to hue to the facts that you've presented it with.

And

for a fun question, we just searched about global revenue from Unilever Group and Google searches for lost my wallet. Does this imply that Unilever Group's revenue is derived from lost wallets? And it will answer, no, that's not true, which we know.

And then if we ask another query that relates to different documents that don't have any information shared between them, now we're looking at the correlation between the distance between Neptune and the sun and global revenue generated by Unilever Group.

And it will not answer that question. It will just say it does not have that information. And so these are all pretty straightforward questions and answers. But you set up with our default settings. But if you go to your agent and you click on Edit in the admin panel, there's actually a lot that you can change if, for example, that last question I asked, if there was data that supported it and it wasn't found or something, you can check which data stores are linked.

So you can link multiple data stores. You can adjust the system prompt. And that is also something you can do via API that we'll go over briefly later. There are some settings on query understanding. So we disabled multi-turn in our setup just to have everyone have the same consistent responses as they're trying it to make the workshop easier to follow.

You can set up query expansion or decomposition. You can set some retrieval settings, settings for the reranker, settings for the filter, including the prompt, generation settings. And you can even set up a user experience where you have suggested queries for the user.

And so

going back to our notebook here,

what we have next in the notebook is going to be examples of code for some of the components. So these are the individual components that make up the full RAG system. We won't run through this in the workshop.

I'm just going to share what we have here before Raj does a deeper dive into what these components can do. So for each component, we've shared a link to a more complete notebook that lets you use more features of that component.

We have our benchmarks that compare it to other solutions. And then we have a little bit of example code that will run in the notebook to try out these components alone. And you don't need to set up the it's the same API key that we set up before.

So everything will just run. I will just say you want to let your file process before you display it. But in this example, we're just printing the first page of the attention is all you need paper. We have our reranker, which is the world's first instruction following reranker.

And that's also a standalone subcomponent that you can use independently of the full platform. We have our generate model that is, according to the effect grounding benchmark, the most grounded language model in the world. That's available as a standalone component.

And here's some sample code for that. And then after Raj's deep dive on the platform and components, we'll go over the LM unit, the natural language unit testing model available as an endpoint that we will use for our evaluation.

So yeah, back to Raj for the next portion of the workshop.

Allright. Yep.

Allright. I just need this one.

Allright. If anybody knows me, I'd love to kind of talk and get into the technical pieces and what's going on a little bit deeper. So this is useful for learning about contextual. But if you're even building your own RAG pipelines as well, all these components come into play to start with.

Now, to start with, how many people have had issues with hallucinations in their RAG pipelines? Has this been a few? Have people not had issues? OK. Yeah, I think everybody finds this. Now, when we start thinking about hallucinations, there's a couple of ways we can think about how to prevent them.

The first is making sure we retrieve clean information. If we don't do a great job at extraction, if we don't do a good job at retrieval, well, then we're not going to give good information to that generator model.

It makes it the generator model will then put its own will try to answer it itself, leading to hallucinations. So that's one big piece. The next is that language model itself. And we'll talk about that is how we can ground that language model so it refers back to the context that it was given rather than substituting its own judgment.

And then I want to talk about having some checks in place. Because even if you do that, you still want your end users to be able to trust the system. So I'll talk about groundness checks as well as you've already seen it inside the platform, the bounding boxes for attribution as well.

So to start with, when you first build your first RAG project, you need to extract from some PDFs. Maybe you've used an open-source PDF extraction tool. Have folks used open-source extraction tools like that, PDF Plumber, things like that?

Not that many? Have people been happy with them in terms of how they've worked for tables and charts? Yeah. The document on the left is the easy one,right? That's what you show in the demos. But folks are going to show up with complex tables, multimodal.

And that's when it gets much trickier. Because when you get to these types of documents, if you have a parsing error in that table and things get shifted over, that's going to cause problems that you can't fix later on.

If your vision language model hallucinates and adds some other information in there when it's reading that multimodal chart, you're stuck with that. That's going to lead to downstream problems. So making sure you have very good extraction is top of the list when you get to complex documents.

And so this is how I like to think about kind of what we're doing at Contextualright now. It's an evolving system. We update every couple of weeks. The engineers probably don't exactly like how I've diagrammed it like this.

But at least for me, the mental model works. Where say you have a PDF document, the first thing we're going to do is add some metadata around it. When was it created? What's the file name? Once we have that, then you want to do a layout analysis.

Is this a document that's image only that we need to do an OCR for? Does it have images, maybe technical descriptions, multimodal charts, graphs that we need to add image captioning where we're going to get a text description of those features?

If you have tables, we have a special table extraction mode that focuses on getting high-quality results out of tables. Now, beyond that, the structure of the document also carries meaningful information. What are the section headers? What are the different subsections in there?

Using that information can make your extraction go a lot better. Now, once we have that, we create a nice markdown kind of JSON version of that. I'll show you that in the platform here. But then for RAG use cases, we're going to take those documents and we're going to chunk them down into smaller pieces to work with.

Because LMs, as much as the long context is growing, you can't take some of these large documents and chunk them in. And especially, there's the overall compute cost of trying to use those long contexts fully. So we create chunks.

We use some of that metadata that we've created, inject that in, knowing the hierarchical structure of that. Along the way, we set bounding boxes. This is one reason when we do image captioning, for example, we don't want to take the entire whole page and do that.

We like to know where the images are so we can do bounding box so the user knows when we say sales went up by $10, they know where on the page exactly it was.

So within the platform, and I know you can't get to itright now, we have the components you can see on the left side. Let me see if I can kind ofright there. All the individual parts of our platform, you can work with individually.

And we have a playground for that. So this is the parse piece here where you can just take a PDF document, decide how you want the extraction. I just want text, or hey, I want the full works where I want image captioning as well.

Or I have long tables. I want the table extraction mode. And you can do thatright inside the UI, see what the results look like. Now, of course, we're developers. We also have an API, Python SDKs, JavaScript SDKs as well.

So you can do this programmatically as well. But that's the first step, getting that clean information.

Next, and this is, again, my diagram. The engineers would not be happy with this, the query path. So once you have that query, when your user's asking a question, what's the process that it goes through?

First, we have a translation module for folks for when we need to do that. But the first big step here is the query reformulation. And so here, depending on if you're doing multi-turn, we need to take into account the previous conversations.

Query expansion, maybe the types of users are doing lots of abbreviations inside your company. So we need to take that query, make it a little richer and fuller before we pass that in to get good results. Or on the other hand, maybe your folks are putting in long, complex queries where what we want to do is take that query, break it up into smaller sub-queries, answer each of those.

So this is where you have the flexibility. You can turn these knobs on and off as you want, as Nina was showing you in the edit, to use those. Now, once you've gone through that process, that long query that you have might break into three sub-queries.

Each of those reformulated queries then passes through your semantic search and lexical search, so hybrid search, kind of classic best practices there. And then we give you the option of using what we call a data store filter. Because maybe your query is asking what's the revenue of Apple Computer.

And your data store has Apple, Microsoft, Tesla, all those documents. Well, if you already know they're interested in the Apple company, why not just filter out all the other documents so you don't have to worry about that? That's where that data store filter comes into play.

Now, typically for Retriever, our defaults are something like 100 chunks that will come back. Retrieval's fast. You can do that. But then to get better accuracy, we like to use a reranker. And again, this is best practices around RAG.

We've trained our own the instruction following reranker. And that helps you go from that 80, and the default is 15. But again, you can change it for your application and give you, let's say, 15 high-quality results that you can then pass all the way through to the end.

We have another filter stage if you want to, for example, drop out some of those as well. So all of these pieces are orchestrated with lots of different machine learning models like this. So this is where it's like a pain to build this for yourself.

But we've built this like this. Now, if I had the time, I'd go into all those pieces in depth. But I want to highlight a couple of things. I like our reranker, I think, is interesting, where you have the ability to provide it instructions.

So a common one might be, for example, if you're asking questions about Apple and you want to get the most recent documents. So you can give it a prompt. And then when it reranks those, it's going to look at the contents of the chunks and use that prompt to help rerank it.

So a lot of possibilities kind of with this. And we're kind of excited to have this piece there.

Now, the last step is generation. So we've retrieved back all of the query. Next, we're taking it to our grounded large language model. Again, the notion here is we want a large language model that's been specifically fine-tuned on to respect the knowledge that it's given, not give its own knowledge like that.

Besides that, we have the two pieces that I've talked about earlier, the groundness and the attributions. You've already seen the attributions. And I'll talk a little bit more kind of about groundness. Now, the grounded language model, since we trained it ourselves, we got to kind of do some different things with it.

So one thing we've done is the language model will tell you the difference between facts and commentary. So when you have some text, it will look at that answer and see, hey, this is the important facts in here.

The rest is just other commentary. So this is where when you're working with this, if you want, I need a super grounded model. I just want the facts in my RAG system. I don't want any of the superfluous stuff.

You can turn the setting on to eliminate the commentary and just focus on the facts, for example. So this is where kind of training our own gives us flexibility to do this kind of thing.

The second thing is the groundness check we have. So the groundness check works directly in our UI. You can also use this through the API. Where for every response that comes back, what we do is we look sentence by sentence, essentially, and see, is the claim that's made found in the documents that were given?

So here, if you look, you'll see the bottom sentence is in yellow. And it's in yellow because the claim it's making is a totally BS claim that it just made up. It's not actually in the source documents. I literally prompted it to make up something.

And so here, we'll automatically in the UI highlight that's yellow. So if you have users that you're worried about,ugh, they keep falling for these hallucinations, this is kind of a helper feature inside there. And the way it works is when we retrieve all the context, and again, you have access to all the chunks, all the things that are retrieved, that answer is going to be generated.

But then we're going to decompose that answer into specific claims. So we have a model that just does this decomposition and looks and sees, is this claim grounded back in that context or not? So we can run this live at time, be able to get those groundness scores, show you that with the response as well.

Allright. Nina has got more on evaluation like that.

So for our last step of setting up our RAG agent, we will get to the evaluation step. So we're going to use our LM unit model for natural language unit tests, which may be a little bit different from how you've been doing evaluation of your RAG system before.

So we have a fine-tuned model as a judge. And that has state-of-the-art performance on Flask and BigGenBench. And the way that it works, as we'll get into in the notebook shortly, is that you'll put in the prompt and get the model response as your RAG system would generate.

And then we're going to create unit tests for the prompt that will ask specific questions that you want tested about those responses. Then we're going to use the LM unit model to evaluate these unit tests. And then we can look at those scores.

So we'll be fully in the notebook now. And I have just given up on the Wi-Fi now. I'm just going to go through what I ran earlier so we can just look at the results if you had internet and could run this all on your own.

So

in this section,

we have commented out the code that we used to generate the data set. So that has our set of questions about the data set, six questions. And then this is the section that will generate those results from those queries.

So to save some trees, we just ran this ahead of time and saved it in our GitHub so you could load it directly. And that's at this file here. We have this eval input CSV. And then that just has prompts and responses for six queries from that same agent that we set up earlier.

And the unit tests we're going to ask, I thought these would be interesting ones for this document set and the queries. Does the response accurately extract specific numerical data from the documents? Does the agent properly distinguish between correlation and causation?

That could be useful if you have some sort of statistical analysis in your documents. Are multi-document comparisons performed correctly with accurate calculations? Are potential limitations or uncertainties in the data clearly acknowledged? Are quantitative claims properly supported with specific evidence from the source documents?

And does the response avoid unnecessary information? We know that LLMs like to blab on and on in their response. So this is a good unit test, I think, overall. Some of the other ones you can set based on your own documents if you want to try this out later.

And so we just set up our unit tests here.

And then the LLM unit model will score on a one to five scale. And it's been fine-tuned for this specific task. And so then the next block of code that you would run would be this response generation. So we put in our query, what was NVIDIA's data center revenue in Q4 fiscal year '25?

And then you get your response. And that does have the correct amount, the citation. But then it has a lot of other information. And so the unit test we used was, does the response avoid unnecessary information? And the score was 2.2 out of five.

So for example, if this was an important criteria for us, then we could later update the system prompt or change some of those other settings to specifically target

the results from this unit test. Of course, we don't want to operate from an N of one. So we are now testing this batch of six, which, of course, again, is a small batch. This was meant to run in real time if we had internet.

And this would just run through all of those six queries and all of those six unit tests to get the results. Then this is the line here, run unit tests with progress. This will give you the results from those unit tests.

And so here we can look at those results. We can see the prompt response.

And then we can see the scores here for each of our six unit tests. And those are going to be from one to five. And then we can also then map those long sentences to just a one-word category.

So just accuracy, causation, synthesis, limitations, evidence, and relevance. You can set these as you'd like. And we're going to create polar plots that show us, for example, for our second question, what is the coefficient between Neptune's distance from the sun and US burglary rates?

We can see in the polar plot that the causation was very well addressed by our model. The accuracy and limitations also scored high on the unit test. But the synthesis was not as highly scored. And these other factors were not as highly scored.

So we can look at all six. And we can kind of see some of these queries scored really well across all our unit tests. Others may point to areas to improve. We linked earlier to the full LM unit notebook where you can look at these categories and cluster them.

So for a larger scale evaluation data set, you could actually look for meaningful insights from this unit test. And so your first homework, if you've made it this far, is to improve the agent based on the unit test results.

And so one way that you could do that in the API would be here to update the system prompt. So I have our original one. And you could change something here, like, hey, keep the responses really brief. Only answer

the direct question. And then you can run this line of code here to update the agent.

And yeah, that's our end-to-end pipeline. And now for a bonus, Raj is going to show you how you can connect your agent to Claude Desktop via MCP. And for that, we have links to our GitHub here and the YouTube how-to if you want to try this out yourself.

OK, yeah.

Yeah.

I'm dealing with the challenges of the Wi-Fi and working around that. So let's talk about this. Let's see if we can get this running.

So if we're lucky for all of you, I spend way too much time making videos. So I have a copy of the video integration that I want to show. So we built a RAG agent. You can ask questions.

You want to be able to use it. One of the most common ways of being able to use it that was introduced like six months ago here at AI Engineer was MCP server. So one of the things I like to do is kind of show people how I can connect the RAG agents that we built at Contextual.

And you can use them inside of other MCP clients. And so here's an example of using it inside of Claude Desktop, where now when it has this particular topic, it's going out to the Contextual RAG agent to be able to get the answer to do that.

So you can do this working inside of clients like Claude. You can also be able to do this, for example, if you're working with code. You want to be able to do this in something like Cursor. That works as well.

So to do this, if you're like, hey, I want to do this, I want to be able to use this RAG agent everywhere else, to do that, we have a repo that's out there. So the Contextual MCP server, the link is in the notebook as well.

So you can grab that. It's a very basic MCP server. You could probably build your own. Let us know. Like that. But the workflow is you clone that repo. And then inside that repo, there's a server.py file. And we're just using our Contextual APIs and just pointing to that.

So the important thing, of course, is that doc string that you kind of describe what your RAG agent is about, because that's what your client, whether it's Claude or Cursor, is going to be able to use that. So to make this a little bit more, here's a copy, for example, inside of mine where

you can see in my local environment, I have that MCP server locally hosted. This is what my server.py file looks like, where you can see I've got two different ones set up, one for technical queries, one for financial queries, depending on who I demo to.

And it can have the client then pull into that piece there.

So that's the server that's sitting in the middle. The next piece is configuring it with your client. And every client's a little bit different, like that. But you just want to be able to have that client point to where that server file is.

So if we look at, for example, in the case of Claude Desktop, which is not running now, but luckily we can still do this, and Claude Desktop has a developer piece where you can edit the config for where it should look for MCP servers.

And if I click here, edit config, I can go see the file that it's there. We can open that up. We're not going to open it up in Cursor. But you get a sense. That's the config file that's going to point towards where my server location is.

So again, if you're not sure about this, I have all the directions on the GitHub. There's an accompanying video as well. But it just allows you to use that RAG agent now in lots of other ways as well.

For example, if you're building your own deep research tool, you want to be able to do that. You can do that. Allright. We're going to open it up for questions here in a couple of minutes. And again, if you're having any issues, raise your hand.

We got a couple of people in the back that can help out. So one question often is, what is this going to cost me? You give me the sales pitch. What is it going to cost me? Well, if you use our individual components, the parse, the rerank, the generate LM unit, we've made that all consumption-based.

You pay by the token for those. So the pricing is up on the website. It's pretty straightforward like that. When you've signed up today, you started off with a $25 credit. So you should be able to go start playing around with it as well.

So those are the individual components. We're also making our RAG platform consumption-based pricing as well. So the pricing will be based on the number of documents you ingest and how many queries you do. So based on that, you'll be able to kind of calculate out, know what your workload is like that.

Now, I work with some enterprise customers. For some sensitive places, they're like, hey, we really need a guarantee on latency or amount of queries per second like that. This is where we can work with that team, do a provision throughput.

So that way, you have dedicated kind of hardware for your particular use case like that. But I think one of the great things that's going to really open it up to developers is the consumption-based, because just like you're doing today, you're going to be able to sign up, pass documents through that, and just pay for what you use like that.

Allright. So some final takeaways. I'm hoping that you got out of this, how you can treat RAG just as any other managed service. You don't have to go through, build out all those pipelines yourself like that. You've got the code.

You can get started now building this pipeline individually. Or if you just need parts of it, just the components, try them out as well. Your reranker, you're not happy with, give it a shot. So go try out the app to do that.

All of this stuff is documented over in docs. We have kind of full API docs over there. We also have a bunch of example notebooks showing, for example, integrations with things like RAG us as well if you want to use that for evaluation.

So those example notebooks will walk you through, hey, how do I improve an agent? How would I use all of these settings? If you need other example notebooks, let us know. Nina and I will work on that as well.

So finally, fill out our survey. Kind of share your feedback as well. We've got some nice kind of merch we can hand out to people who have good questions as well. And I think we got a little bit of time for that.

How does that sound?

I've thrown this RAG platform at you. Are you all ready to sign up? OK, yeah. Go ahead.

When you found people using this platform, there's an interesting feedback loop where people can set their own evals at the bottom. And then

is there a mic?

There's a Q&A mic. Or I'll try to repeat back the question to the wider audience. So if you want my interpretation of the question, or I can't massage it then.

Is that on?

Yes.

Yeah? So there's a really interesting feedback you're defining here around you can define evals via prompt. You can see how they're doing. You can let people change the system prompt. Do you find that your users of the platform or the enterprise customers, who is it that's doing that?

Is it technical people, or is it the people that's using it day to day?

So one of the things by putting it in the UI, what you have is you have some of the business users will try to mess around and play with it. And some pieces like a system prompt, like non-technical people have a sense of using ChatGPT and doing that.

But when you get to what's my top K for my retrieval or using query reformulation, those business users are just going to mess up. And their agents are going to kind of not work well like that. So a lot of the settings there, as you get to the advanced hill climbing, because a lot of that stuff comes into like I've built a good RAG agent.

It's hitting 80%. But the business wants us to get to 90%. Now I have to build the evaluation. Now I have to see where are my errors? Are they on the retrieval side? Are they on the generator side?

And this is where having that developer perspective of understanding and being able to do that error analysis is important to figure out which of those settings to do it. Because we give you a lot of settings, but you still have to have some sense of which settings are appropriate to change for what outcomes like that.

The company I work for is relatively new to AI. We're loving what we're discovering. But it's also pretty overwhelming. The information you presented

hints at some of the answers to our major questions, which are we have thousands of PDF documents. We have thousands of Excel documents. We have an MRP system with an SQL backend. Do we need to be writing custom queries for this environment to consume?

I guess, does your environment have a checklist of the approach for different data types and how to package this information for your environment?

Yeah. I can repeat back the question like that. So the question is, and I'll paraphrase here. Getting into it, we have lots of documents. We have Excel documents, PDF documents, structured data. How am I going to make sense of that and figure out what the best way to kind of use your platform is?

Is that fair?

Yeah.

Yes. And so this is where, as a startup, we have a couple of different levels. We have the developer. Go figure it out yourself. We've given you the buttons, the API commands to do that. But this is where we've grown.

And we have a dedicated team that we call customer machine learning engineers. All they do is work with customers. And they do that process of, hey, let's walk you through building your evaluation data sets. Let's help you kind of hill climb and improve that.

So depending on what you need, we have a team that's just focused on helping customers through that process as well. And in the meantime, Nina and I are trying to document it and make it more available for the rest of the users.

But there is that gap of there's a lot of knowledge to effectively use these systems to be able to do that. Is that fair?

Is there a consulting fee or a startup package?

Yeah, talk to us. We got it. We'll find a way to take your money.

A little bit.

Go ahead.

Hey, Rajiv.

We've built a few agents. And I'm very interested in exploring the RAG-only part of what you described. Is there a way to integrate these into our agents that we're building in TypeScript, JavaScript?

Absolutely. So each of the OK, yes. I don't have to repeat. So each of the pieces here, we have kind of APIs. We have a JavaScript SDK like that. So if you just want to use the components of this, this is where when the company first started about a year ago, if you'd asked to kind of do a demo, we would have sold you.

You had to buy the entire platform. You have to get the end to end. But we've realized, if we want to appeal to developers, that lots of times you've built stuff out that you just want some components of it.

So yes, we've kind of modularized that out. So if you just want to use parts of it, you can and integrate with others like that. And in fact, most of our customers don't use the UI that we've showed.

Most of them have other UIs that they're integrating their applications with like that.

In addition to the costing or pricing, how far can the $25 go? Because we weren't able to test itright now. How many documents can we experiment in it?

So the question is, how far will your $25 go? I don't know exactly. I get unlimited use of it. But try it out. If you run into issues, just let us know. We can talk to sales. We can figure out something like that, depending on what you're doing like that.

So don't let the $25 be an issue. We want you to use it. We want to hear your feedback. If you need help, we can talk to you and figure out something to do like that. So yes. Do people like the idea of a managed RAG service?

Does this feel like something? OK, OK. OK, yes.

Well, to that point, I guess I'm wondering how managed is it? So at my company, we build RAG applications for government and other clients who have really strict data sovereignty, data residency questions or requirements. So are you in either of the big clouds where I can decide to be in a GovCloud or some other sovereign data service?

How much control do I have over where stuff lives?

Yeah. So again, we're an early-stage startup. We're starting out. Right now, we have our own cloud, which doesn't help you like that. We have partnered with Snowflake. So we are on Snowflake, if that can work for you. We can also install in VPC.

Butright now, we're limiting it to VPC. We're not doing kind of custom on-premises deployment, just because that takes a lot of upfront work to be able to do like that. Does that help?

Yeah. Can you do VPC in, say, AWS GovCloud or Azure's government solutions?

We have not yet done AWS in kind of GovCloud like that. If you have a strong demand for that, let us know. We can work and try to figure out something like that. But we haven't taken on that yet.

OK. I'll come find you. Thanks.

Yes. Thanks.

Hello. Hi.

Yeah, go ahead. I was curious. How does the performance of your RAG platform vary as the number of documents vary? For example, do you have recommendations of best practices when you're working with millions of documents as compared to hundreds of documents?

What kind of configurations and knobs work better? Would be curious to learn.

Yeah. So I'm going to give the hand-wavy answer of we have customers like Qualcomm, for example, that have tens of thousands of documents. Their documents have hundreds of pages in them. And we handle that fine. All I know is that I have engineers like Matthew in the back that works the platform that makes sure it kind of scales up like that.

So I would say, go grab him if you have questions about scalability. Right now, we've been able to kind of scale with our customers along those lines like that.

So let me take this. And then I'll come back over here.

Hey, this is specifically about the LM unit tool. How deterministic and repeatable is the scoring from that?

I'm looking to see if they have I mean, it's still a large language model. So I think there might be a little bit. But I think it's fairly good.

It's pretty repeatable. There was actually a paper released when this model was released. There's a lot more details in there, too. But there was a lot of analysis done about the correlation with other trusted metrics and how running the LM unit tests repeatedly.

We use kind of a random seed to make sure it's repeatable. So all of our testing has suggested that

it is repeatable. Of course, if you're altering the prompt or altering kind of the natural language unit test, that can have unforeseen impacts on the results. But I think keeping the prompt consistent, it should be pretty consistent with different types of queries.

Awesome. Thank you.

Yeah. Check out the paper, as Matthew said, that it goes into a lot of those pieces like that. So thanks.

Question here.

No.

So is this already integrated with Microsoft Copilot and Azure? And the second question is, are there any APIs that you guys expose in it? Meaning, if we are developing our own custom code, but we want to use your RAG, and what's the approach?

So we have the API. So if you're developing your own custom solutions, you can easily integrate kind of what we have to do that. In terms of installing inside of kind of Azure like that, that's one of those custom VPCs that we can support like that.

In terms of integrations with Copilot, I don't know if we've really done anything. Usually, people are like, I'm sick of Copilot. And that's why they come to our RAG solution like that. So again, we have the APIs, though.

So I don't know what the integrations would require for something like Copilot like that.

Thank you.

Hi. First of all, great presentation, because I have used your product. And the other thing is, regarding RAG, I have a question. You as a company, what are the challenges that you are facing in the RAG? And where do you see RAG would go in future?

So what are the challenges we're facing in RAG?

Apart from hallucination, internet search, or things like that, apart from that, what are the different challenges that you're facing?

Matthew, I might have you chime in on this, too. I think

one of the big challengesright now is around the extraction, where we've spent a lot of time and energy. And there's still, I think, room for improvement when we're talking about working with complex tables along charts. I think there's one there.

I think scalability is always a piece like that, because everybody wants to ingest more documents in a faster amount of speed like that.

I think if you talk about doing generated working with structured data, I think that's a challenge for everybody in the industry when you're trying to do those queries of text to SQL type queries like that. So those are some of the top ones on my list like that.

Do you have?

Yeah, I agree. I think document understanding, making sure that you are correctly understanding long, complex documents with hierarchy and interrelations between sections and really large tables and gnarly figures, I think that has been a consistent thing that we've been working on and seeing improvements on.

But there's definitely still room to grow there. I think another kind of broader change in the RAG space that our research team is very focused on, and it hasn't directly translated into our product yet, but it's kind of coming soon, is moving away from the more static RAG pipelines that Raj beautifully described in his presentation toward kind of a fully dynamic flow with model routers at certain kind of inflection points deciding which tools to use, how many times to retrieve, how to alter the retrieval config in order to correctly answer the query.

So I think those sorts of dynamic workflows will greatly increase what you can answer with any RAG platform. Almost like deep research for RAG is one way to think about it. And I think that's something that our research team is working very hard on and will be coming into our platform in the near-ish future.

Great. Thanks.

Come back in six months.

Do you have plans for a public-facing web search feature, particularly one that could be configured on a company's domain?

So are you looking for kind of a general, I need to be able to search the internet, because there's Tivoli, Firecrawl, places like that?

It's more the companies that have public-facing websites where it makes sense to, instead of running RAG on your CMS database, you're actually just.

Yeah. You want to run it on that live. And so John, I might pull you in here, because I think one of the things John's worked with quite a bit, because a lot of what that requires is the integrations to be able to pull from that web from those sources as well.

And so I know, John, you've done some of those things for customers with Firecrawl pieces.

Yeah, sorry. What was the question before we began?

About ingesting customer websites and being able to use that in RAG.

Oh, yeah. So there's a lot of ways that you can scrape for websites or directly kind of set up an ETL pipeline with pulling the records from the APIs or just pulling the unstructured data from a data cloud or blob storage, and then kind of just setting up an ingestion queue, which is what we do on our side for larger customers or basically any customer that needs a kickstart for lots of data coming in.

Some customers are super happy to write their own scripts or their own kind of functions, our daily cron jobs to update documents. But we're also working on a managed solution on our side so that you can just give us your credentials on our front end.

And then we'll just start pulling and syncing all the data on our back end. But that's something coming on the roadmap in two to three months.

And so please, if there's pieces like that you see that you want us, let us know, because we like to kind of push the product team to kind of make that stuff happen.

Allright. Yeah. Hi. Hi. How do you deal with frequently updated content and document-level permissions if not everybody can see specific documents?

Yes. So those are both tricky things. One of the things we'll do for frequently updated documents is we have a continuous ingestion pipeline. John, if you can talk to in the back, can talk to you more about that piece as well.

So that's one piece for the frequently updated. The other piece is the harder one, entitlements. How do you deal with all the permissions? You're indexing HR RAG stuff next to customer support stuff. How do you make sure that the person doesn't look up the salaries of anybody else like that?

So this is where, within our platform, one of the things we're adding is an entitlements layer like that, because as we've talked to lots of customers, they're like, this is nice when you do it over our customer support.

But inside of a real enterprise, we have governance. We have permissions. We have to be able to respect that when we do these types of searches. And so we're adding an entitlements layer on top of ours to be able to handle that.

So yeah, that's an important piece.

Hi. So my question is regarding in the last six months, what are some of the major breakthroughs in the RAG area?

Ooh, what are the major breakthroughs in the last six months in the RAG area?

So I think the rerankers continue to get better. I think that's an easy place where we've seen that. I mean, I think part of it is just every piece of that has steadily been getting better. I think one of the biggest changes you've seen in the last six months here is the rise of the vision language models and how strong they are at being able to handle kind of images like that.

That's off the top of my head. I see Matthew's busy talking to somebody else. Otherwise, I'd have him weigh in as well. Does that help?

Does the contextual parser module can replace DocLink or GCP invoice parsers and things like this? And if it does, does it have the ability to parse QR codes and barcodes and instruct their metadata to kind of read that data as well?

Yes. So I think one of the things is there's a lot of different types of documents out there in terms of document types. And there's a lot of different solutions. So we're not going to be able to handle every type of thing perfectly like that.

I think there's going to be lots of pieces. But this is where we've made those APIs so you can try that out on those pieces. We haven't specifically focused on, for example, that type of document. We do have the image captioning.

So maybe it would pick them up. Maybe it wouldn't like that. But the idea here is to go in that same space where there's a lot of these other companies doing this parsing thing and making sure that we have a module that you can just stick in and replace and kind of use with ours like that.

How do you deal with domain-specific language?

That's a good question. How do you deal with domain-specific language? And this can get a little tricky. And we've seen it, for example, when we're working with technical companies that have very specific words that they use inside there, where on one hand, you can do things like changing system prompts a little bit for the retriever part.

But sometimes what we've done with some of those customers is fine-tune that grounded language model that's closer to their vocabulary and how they speak. And that's been one technique we've used. Now, this is where at one point in the platform, we had that available to end developers.

But fine-tuning models can get a little complex for using for RAG. So we've kind of put that away now. And that's more for kind of our service-oriented customers that work with our customer machine learning engineers. But that's one thing we've done to if that knowledge is out there and you need to get that into your grounded language model is to do kind of a fine-tuning process step.

Go ahead. Nope. I don't know where the mic is. We're making them run. Getting their steps in today.

Hey. So how about HIPAA regulations like PHI data, this kind of stuff? How do you deal with that? Should we hash it before going to the platform? Or the LLM can understand and deal with that?

So there's two aspects to that. One is we just got HIPAA certified. And people are going to think I'm paying the audience members out there like that. The second is, well, what do you do in extraction? Maybe I have PHI data.

I want you to automatically mask or filter that. So this is one thing that we've been talking with the product team about whether to include that capability for during that parsing piece to automatically kind of flag and identify that behavior.

Right now, we don't kind of have that in the product. But again, if that's something you need or want, let us know. It's not that hard to kind of build on. But you can also do that as a supplement to the parsing and have a second job that runs and looks for that PII information.

Does that help?

I know garbage in, garbage out. But if we have a document with, say, two different numbers on a particular statistic or fact, how does your RAG deal with that? Or how would it answer a question?

Yeah. So what happens in that case is often the retrieval is going to bring both of those pieces of information over to the grounded language model. And it's going to be up to the language model to use its reasoning ability to try to reason out the two differences.

So sometimes if one is, for example, if it's obvious that one is not, let's say it's the size of a mosquito, and one of the answers is 100 feet and the other is 3 millimeters, the language model is going to know one of these is junk and to ignore it.

But if they're both close and reasonable, yes, that can get you in trouble with those pieces like that. I assume this is a pain that you're having now.

Yeah. We've had lots of not updated Excel sheets. And so we're just trying to see how we can organize that, because a lot of my jobright now is just telling people information that should be in a document, but it's not updated.

So one thing that can help with that is taking advantage of metadata and using as much rich information about those documents so when at retrieval time, it can help it kind of figure out, hey, these are the two differences.

And this is why I should prioritize this answer over another answer. Maybe this is the most recent, or this was written by the authoritative person like that. That's one thing that we kind of recommend for those use cases like that.

Thank you.

Allright. OK. Well, thank you all for staying. I've loved kind of the questions like this. Please use the application. Give us feedback on how you're finding that as well, good, bad, negative. Happy to take that all like that.

So thank you again for kind of spending your morning with us like that. Anything else we have, Nina? Or are we good? Oh, you've got the mic off.

I think I still have my mic. Yeah. I've been going around distributing swag to folks that have asked questions. Sorry if I missed some of you that were farther away. I have one more pair of socks. Anyone wants them?

The guy in the back wants the socks.

Allright. Yeah.

Allright. Thank you all. We'll be hanging around for a little while if you may have that.