OpenRAG: An open-source stack for RAG — Phil Nash

Hi there. My name is Phil Nash, and I'm a developer relations engineer at IBM. I've been working on tools around AI and RAG for the last couple of years, and I've got something I'd like to show to you today.

Now, first things first: I've heard that RAG is dead many a time, and I'm sure you have too. Context windows are huge these days, so you might as well just dump all of your information into there. I don't take this kind of thing very seriously.

If every business has less than a million tokens' worth of data, then sure, RAG is dead and probably so are all those businesses. And of course, not everyone is happy paying for a million input tokens every time you want to ask a question as well.

Instead, I sort of hear the "RAG is dead" claims as more of, well, "RAG is solved,"right? We think we understand the process and we can just apply RAG when we need to. You just gather up all your unstructured data, extract the text, chunk it up, embed it, throw it into a vector database, and then when you want to ask your agent a question, you just embed that question, search the database, pick the top K results, and parse them to a model as context.

It's just a footnote in context engineering these days.

But it turns out that RAG is actually hard, and it's hard for different reasons for different projects. PDFs are a pain, chunking strategies are a hassle, and changing them and testing them is difficult. Embeddings keep improving, which is great for the industry, but not very great when you've used something from 6 months or a year ago.

There are new search techniques all the time, and further tweaks that you can add to your pipeline to improve the results, like adding summaries to chunks, performing chunk expansion, using a cross-encoder to re-rank results, query rewriting—there's so much more.

RAG is quite complex. In fact, everyone's documents are different. Every system will have different users, different questions, different interaction patterns, and different expectations. While every RAG system will ultimately be different, there are definitely some core components that are required.

When building a RAG system, it's useful to have a high-quality baseline to build from. And so that's what we've been working on at IBM. We've brought together three existing open-source projects to create a RAG stack that is powerful, easy to use, and easy to extend.

The project is called OpenRAG, and it uses the open-source Docling for document processing, OpenSearch for search indexing, and Langflow for visual orchestration and agents. OpenRAG is an open-source project that you can try out today to build your own powerful, customizable, and easy-to-use RAG system.

But I just want to break down the stack for you so that you understand the components and how they work together, and how they create a stack that is flexible enough for your modern RAG requirements. Let's start by looking at the ingestion side of RAG.

Let's start where it all begins: document processing. Ingesting PDFs, HTML, Word docs, slides, and more can be a pain. But the biggest pain of all is, of course, PDFs. Docling is an open-source project. It was built out of IBM Research in Zurich, and it processes and parses all sorts of documents, from HTML, Markdown, and Word documents, through to slides and spreadsheets, audio and video, and even that enemy of all RAG systems, PDFs.

Docling has a number of different pipelines that handle different file types. This allows it to be flexible in the way it takes in documents and accurate in its output. So there is a simple pipeline that handles those mostly straightforward text documents, like Markdown, HTML, and Word, that just extracts the text, turns it into a hierarchy, and outputs a document.

For audio and video, there is an ASR, an automatic speech recognition pipeline. And for PDFs, there are two available pipelines. The standard pipeline has a number of small, focused models that do different things, like extracting text, tables, and images from PDFs.

You can even choose an OCR backend to read text, which is particularly useful for scanned documents that don't have actual real text in them. So this collection of small models in a pipeline perform things like layout analysis, table extraction, image extraction, and descriptions.

This gives you a wide array of options to get the best out of those documents. There's also a VLM, a vision language model, pipeline that uses the Granite Docling 258 million vision model to extract all of that in one go.

This is a newer pipeline, but it is simpler as it is this just all-in-one model that's trained specifically for this task. Docling extracts text and then produces an intermediate representation, a Docling document, which models the structure of a document in an XML-ish format called DOC tags.

Those DOC tags can then be converted to a number of formats, including Markdown, HTML, and JSON. And the Docling also has a chunker that uses the hierarchy generated by the parsers and built into those DOC tags to produce hierarchically understood chunks of text.

Moving on to embeddings, OpenRAG actually isn't very prescriptive with embeddings at all. It supports a number of external providers, including OpenAI, What's Next AI, and Ollama for locally hosted embeddings. And in fact, the entire of OpenRAG can be run offline using locally hosted models.

Docling itself can be run offline, so it can run in air-gapped situations. It doesn't need those external services. Once you have embedded those chunks, they are indexed in OpenSearch. OpenSearch is, of course, the open-source fork of Elasticsearch and is a powerful database for performing vector search and keyword search, as well as highly configurable.

It also has highly configurable filtering and aggregation. Out of the box, OpenRAG uses OpenSearch for a hybrid vector and keyword search, and exposes that sophisticated filtering for more targeted searching. It also supports vector search over multiple embedding models.

Now, this will slow down your vector search in practice, but it is useful if you decide you need to migrate your embedding models as part of your system.

OpenRAG also sets up OpenSearch with a secret fourth open-source project. The default OpenSearch nearest neighbors plugin gives you options for HNSW or IVF vector indexes, but OpenRAG uses the JVector KNN plugin by default. JVector is an open-source vector index that gives you live indexing, and because it's based on the Disk ANN architecture, it means your whole index doesn't have to fit in memory, giving you more options for scaling the data servers.

All of this is then tied together with Langflow. Langflow is a drag-and-drop visual editor for AI flows, and it integrates Docling, OpenSearch, and all these embedding models, as well as further data enrichment as part of that ingestion process and pipeline.

We'll come back and have a look more deeply into Langflow later. So that's ingestion and indexing. What about the generation side of RAG? On the generation side, we don't normally have to worry about ingesting documents, and we already know that OpenSearch is handling that multi-vector hybrid search for us.

But we do need to point out that OpenRAG uses agentic retrieval in order to perform the search. This is also done in Langflow, and again, gives you access to all the kind of models that Langflow makes available to you.

So out of the box, that's very much OpenAI, Anthropic, Ollama, What's Next AI. But what does agentic search mean? Well, a traditional RAG generation pipeline would take a user query, embed it, users perform that nearest neighbor search over the chunks, and present the top K chunks to the LLM, hoping that the answer is contained within and that the model is smart enough to extract it.

With agentic retrieval, we instead give the user's query to an agent, along with instructions and tools that it can use to perform as many searches as required. The model is actually responsible for deciding what searches to perform and what to do with the results.

So let's actually take a look at this in action. I have OpenRAG running on my laptop, and we're going to have a quick look at what it can do. So once you've gone through the onboarding process with OpenRAG and setting it up, you get dropped into a chat, and the first thing you get to ask is, "What is OpenRAG?"

And as you can see here, it has got an answer, but you can also see that it's done some tool calling already. It turns out that the answer to "What is OpenRAG?" is inside the agent's prompt by default, so it doesn't actually need to do any search querying.

But it did go and get the current date, just in case, as well, which is nice of it. So we can see we get an answer out of it. But we also get these kind of suggestions about the next things.

Those are little nudges. That is also powered by Langflow. And if we were to ask about that, to explore Langflow's role in AI agent construction, then the agent itself will go off, search that documentation, and come up itself with an answer.

And so as you see, the agent has gone and used some tools again. It has come up with an answer. It's come up with more nudges as well. So let's go and look at the knowledge section. This is where you actually upload your data, your documents, and you can do so just by adding a whole file or a whole folder.

There's also a sync button here. We'll see that in a minute. And you can also inspect kind of your objects and your documents here and your chunks, so you can see that they are chunking things as you'd expect.

This also is where you can create knowledge filters. So this takes advantage of that filtering in OpenSearch. You can create filters based on a whole bunch of different options around the data that you have in your system. And then that allows you, in chat, to use those filters to only talk to specific documents.

So that's the knowledge section. And then in the settings, we can dig into the actual customizability of this. Soright at the top, there are cloud connectors, but in order to use cloud connectors, you need a user model, some authentication.

Right now, we set that up with Google OAuth, so you need an OAuth client and secret there. Once you have that involved, you can connect to Google Drive, you can connect to SharePoint, you can connect to OneDrive. And this allows your users to connect to directories of their own documents and allow OpenRAG to sync them directly.

I think that's really powerful. It saves you having to upload things a lot of the time. You can just sync with this external document store, and it will always be up to date. Here we can see our model providers.

You can configure kind of API-based ones or Ollama. Like I said, that's for running things locally. And I'm running Ollama, and you can see I'm running currently Granite 4 3B. That's one of IBM's models. So this is our language model, and you can see the actual agent instructions as well.

So you can set your system prompt there. And then in the ingest section, you can see, again, I'm running Qwen 3 embedding, 0.6b, for my embedding model, also on Ollama. And you can set your chunk size and chunk overlap.

And then these last bits are Docling settings, where we say, "Do we want to capture table structure?" Yes, currently. "Do we want to run OCR?" Notright now. That's turned off. And "Do we want to extract picture descriptions?" Currently, that's off, but that's a useful one if you want to kind of get the information out of images as well.

Of course, adding more models to the pipeline makes things a little slower, so they're off for now. And thenright at the bottom, there are API keys, and this is where you can set up access to OpenRAG as an API, so you can use your search or your agent within your own application.

But let's actually drop into under the hood even further, where we can go and customize things even more. You can hit this edit in Langflow button, and that will take you into the actual implementation of your agent. And so let's actually zoom in.

We can see here is our agent. So this is the chat, the generation flow. And our agent receives its information from this chat input, which goes through a quick prompt template, adding in things about knowledge filters if you've used them.

And then the agent has a bunch of tools. Those tools include this is an MCP server for a URL ingester. That's actually just another flow within Langflow. There's a calculator, because I think that agents and models shouldn't be doing arithmetic.

They're language models, not math models, so a calculator is always useful. And then finally, the last one is the OpenSearch multimodal embedding thing. And so the embedding providers are all here. We can edit this. If you can go into here and just unlock the flow and save that, we can do more with it.

And so, for example, we can take this chat input, and we might want to put some guardrails in place. So we can grab guardrails from our set of components on the left. And then we do need to parse the result of that.

So we can just get a parser. And so if it parses, we parse it through the parser, get the text out, which is the original text that was sent in, and then hand that onto our prompt template. I guess if it fails, we can send an error message to a chat output, and that's fine.

And so now we've added guardrails to our thing. We can use our Ollama models there as well.

And so this is as extensible as Langflow can be for you.

There's one more thing. There is an MCP server available for OpenRAG as well as an API, so you can go and use this and hand this to your other agents as well.

So is RAG solved? Well, that's kind of still up to you, to your data, and to your users. But OpenRAG is built to help. It is an opinionated but agentic and open-source stack for RAG. As I said before, it combines Docling, OpenSearch, and Langflow to create this powerful baseline RAG system made of open-source components.

And it leaves plenty of room to customize that within that stack so that you can build out the best RAG for your data and provide the best context to your agents. It is currently at version 0.4.0, and it's ready for you to play with.

So this link or the QR code will take you to the project. And we'd love if you'd try out OpenRAG. Drop a star on the GitHub and let us know what you think. It's also open source,right? The front end is a Next.js application.

Everything else is a Python app. So if you like the look of OpenRAG, we'd also appreciate your feedback and your contributions to the project itself. And the components to OpenRAG, of course, are open as well. So you can get involved with Docling, with OpenSearch, or with Langflow as well.

And together, we can build a RAG platform that works for everyone. It gives you the choices where you need them and makes good decisions for you where it makes sense. We can do it with open-source components out in the open.

That's what I'd like to see. So thank you very much for listening. Again, my name's Phil Nash. I'm a developer relations engineer at IBM, trying to help build OpenRAG and this open ecosystem of agentic applications. And we can't wait to see what you build with OpenRAG.

Thank you very much.