The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

This is the RAG stack we landed on after 37 fails. Hi, I'm Jonathan Fernandes, and I've been working with language models way before ChatGPT appeared on the scene. So I work as an independent AI engineer, and I help companies build and ship production-ready generative AI solutions.

Now, if you're new to RAG, I'll give you a bit of an introduction, but this is my objective is for this to be the most ROI-packed RAG guide per minute on the internet. Now, if you're pretty familiar with RAG, then I want to give you the details straight away,right?

And these are the headlines for the 37 fails and the learnings from those 37 fails. So I've broken them down into the components that I would use in a RAG stack. So you've got the orchestration, you've got the embedding models, the vector database, the large language model, and so on.

Now, typically I'll break them down into two components: a prototyping piece, which I normally do in Google Colab. One of the reasons I use Google Colab is because I get access to a free hardware accelerator, so that just makes it very easy to prototype.

And then, normally when I'm working in production, because I tend to work with financial institutions, there's a requirement often to have a lot of my data and also all of the processing taking place on-premise. And so I'll often use Docker as my solution, because I can run it either on-premise or sometimes in the cloud if required.

So for the orchestration layer, there are a couple of options,right? So prototyping, I'd use LlamaIndex or LangGraph, and then when using it in production, I'll use LlamaIndex. With the embedding models, there are a couple of options. So you can go with closed models, and this is often helpful, especially when you can just use APIs and it makes it pretty simple to use an embedding model.

Or you can use open models, such as the ones from NVIDIA or BAAI. And again, when I move over to production, then I'll use an open model, such as the one from BAAI and NVIDIA. A vector database quadrant is an excellent solution because it scales really well.

So I could be working with just a couple of documents to hundreds of thousands of documents, and it just works. And so quadrant is my choice for a vector database. And then in terms of language models, again, I like to use closed models here, just because of the simplicity of using APIs.

If I want to go ahead with an open model, I could use the likes of any of the models from Meta, or I'll use a Qwen 3 model. And I'll do exactly the same thing when working in production within a Docker environment.

I normally use Ollama or Hugging Face text generation inference to allow me to serve the Llama 3.2 models or the Qwen 3.4 billion models from Alibaba Cloud. Now, it's really important to be able to monitor and trace your RAG solution.

This just helps in terms of troubleshooting. It helps you to know where the majority of the time is, for example, in terms of the language model and other components. And so the two solutions here are LangSmith and Arise Phoenix for my prototype.

And in production, I'll often just use Arise Phoenix, because I can just very easily use that in a Docker container. In terms of reranking and improving the accuracy of your RAG solution, again, my go-to choice is a closed model, such as the one from Cohere, or I might use an open solution, such as the one from NVIDIA.

And again, in the Docker solution, I'd use the open solution from NVIDIA. And then finally, it's really important to be able to evaluate how good your RAG solution is doing, and I'll use the RAGAS framework there. Now, in our time together, I'll be looking at just a single knowledge base.

And if you want to get the most of your time in terms of understanding RAG, I suggest you just use the next three to five minutes, pause this video, and take a look at this knowledge base. Because we'll be answering a very simple question, which is, where can I get help in London?

And so this knowledge base is for a train or a railway company operating in London. And this is the information in terms of HTML files, where you can go to each of the links, and you'll be able to get information for that train operator.

Now, if you're new to RAG, let me just provide a very quick overview. So RAG is Retrieval Augmented Generation. And so you might have the initial user query, which might be something like, where can I get help in London?

And the first and most important step is the retrieval step. This is where you do semantic search, where you look through a vector database, and you retrieve relevant documents based on that query. The next piece is the augmented piece of RAG.

And this is where the original query is combined with the information that's been picked up from the vector database. And this is provided as context to the language model. This then allows the language model to produce the generation piece, because it now has the required context and the original query.

And so it can provide a response to that original question, where can I get help in London? So a naive RAG solution looks something like this. You have the query, that query is then embedded, and then you want to then compare that to the documents that are available in the vector database.

You then retrieve the relevant documents from the vector database, and you pass that along with the query to the language model. The language model now has that as context, and it's able to generate a response to that original query.

Now, as I said, I like to prototype in Google Colab. So what we'll do over the next few minutes is to take a look at each of these different components in Google Colab and see how they work. So the first thing you'll want to do is to get and view the knowledge base.

So what I'm going to do is I'm going to show you the original knowledge base. And so let's look at the first two articles. So the first one is, are there any wheelchair-friendly taxis in London? And you've got the associated content over here.

So most black cabs in London are equipped with ramps and so on. And let me take a look at the next one. So are there any wheelchair-friendly taxis in Paris? And then it gives you the number for a taxi company.

And there's a reliable taxi firm dedicated to providing accessible taxi services and so on. So if you haven't had a chance to look at this knowledge base, please do take some time to look at it. And with that one question in mind, I'm now in London, where can I get help?

So let's go ahead and grab all of the files that we need, and let's copy it over to Google Colab. And so we have all of this in our data folder, and these are the relevant HTML files. Now, I can look at those HTML files either here, or I can view them in Google Colab like this.

And so these are the two files that we looked at. Are there any wheelchair-friendly taxis in London? And are there any wheelchair-friendly taxis in Paris? Now, let's dig a little bit deeper and look at what exactly is in the documents.

So with LlamaIndex, I can use the simple directory reader, and that'll allow me to read through all of the files in a directory, which are my HTML files,right? And you can see that I've been able to pull them through over here.

Now, let's look at some of the details of some of these documents. So I've got 32 documents in total. If I look at the first document, you can see that I've got a document ID over here. And then you can see I've got metadata, such as the file path and where the information is taken from and so on.

If I wanted to see the actual text of the document, I'm able to do that over here. And you can see that this is the HTML text for that first document. I can then go ahead and move to the second document.

And in the same way, I've got the document ID and metadata, such as the file path and the HTML file and so on. If I want to take a look at the contents of that second file, that's available over here.

And this is what that HTML file looks like. Now, as part of the metadata, I'm able to determine the actual location of that file. So this is taken from this HTML file. And for that second document, this is from the second HTML file over here.

Now, let's take a look at the naive RAG solution. Now, what's really nice about LlamaIndex is that just in a couple of lines of code, I have a basic RAG solution. So in that first line, I'm going to be able to retrieve all of the HTML files from that directory and store them in documents.

I then take all of those documents and store them in an in-memory vector database. And I'm then in a position to be able to query that vector database with my question, which is, where can I get help in London?

I'm at the station. And you can see that the first response using a naive solution is not very satisfactory. You can see that you can get help in London at the station by approaching London black cabs, as most of them have ramps and so on.

Now, if we want to get a more sophisticated RAG solution, then there are a couple of components we'll want to add. We'll want to add a query processing step. And what happens in the query processing is that we might want to remove certain bits of information.

So for example, if there's personally identifiable information, then we'll want to remove that before passing that to our RAG system. And the second thing we might want to do is we might want to improve on the accuracy of some of the documents that have been retrieved from the vector database.

And we can do that in the post-retrieval step, after the information has been retrieved from that vector database. Let's look at some of the building blocks of these systems. So what you have is a cross-encoder. Now, your objective of having a cross-encoder is to be able to semantically compare a query with a document.

You want to send both your query and your document to a BERT model. Now, a BERT model is the encoder from the original Transformer model,right? You then pass that on to a classifier, and then you're going to get a result between 0 and 1.

And that will indicate how semantically similar the query is to the document. Now, as you can imagine, the larger your document gets,right, the less scalable the solution gets, because now you've got a rather short query and a large document,right?

So this solution is excellent for additional accuracy, but it's slow and not scalable,right? So let's think about where we can put that as part of our RAG pipeline. The next solution is where you can actually split things out,right?

So one of the challenges we had in the original with the cross-encoder was the lack of scalability, because we had everything just running with one model. Now, what if we instead have two models, two encoders? So we have a separate encoder and a BERT model for the query,right, which then passes on to a pooling and an embedding layer.

And then we have another model,right? Again, a BERT layer and a pooling model and an embedding layer,right? And we have those as two separate models. And then we can compare how closely the query is related to the document by using something like the cosine similarity.

Now, what's great about this is that by separating out these two models,right, this is a fast and scalable solution, and this is excellent for information retrieval. Now, given that we now have a better understanding of these cross-encoders and bi-encoders, let's try and figure out where we might want to put this in our RAG pipeline,right?

So given that we have a fast and scalable solution, and we're having to retrieve information and compare that query with multiple documents, it would make sense for that bi-encoder piece to be where our vector database is. And similarly, because we're looking to get additional accuracy, but we can't scale, and we are only able to work with a few documents, then the cross-encoder would be a really good solution post the vector database, and so post-retrieval.

And so this is where you normally have your cross-encoder, which is also often known as a reranker. So let's head back to our code, and let's look at swapping out some of the components of our solution. Now, previously, we had an in-memory vector database.

Let's go ahead and use a quadrant solution. Now, this quadrant solution also happens to be an in-memory solution. And so as you can see, we've now got our simple directory reader, and we're using the quadrant client. By default, we will have a built-in embedding system,right?

And so you can see that by default, the embedding solution that we have is the text embedding ADA002 from OpenAI. If we want to, we can swap that out and use a text embedding 3large, again, from OpenAI. But what we have as an option is to use an open solution.

So now we're not going to be using a closed model with APIs,right? We can use an open solution and actually download the embedding model onto Google Colab,right? And so what we're going to do over here is we're going to use the model from BAAI, which is the BGE small model.

Now, if you want to take a look at that, I've provided a link where you can go to Hugging Face and get a little bit more information about this embedding model. And so we're using this embedding model from Hugging Face, and we can go ahead and set that as our default embedding model in LlamaIndex.

And you can see that over the next few lines, we've actually downloaded that model onto Google Colab. And now we've now set that up as our embedding model. Let's go ahead and swap out a couple of the other components in our RAG pipeline,right?

So this is our default settings. You can see that by default, we're using the GPT-3.5 Turbo model as our large language model. It has a temperature of 0.1. And you can see that we've got a couple of other components.

So let's go ahead and swap out that large language model. And let's just use another model from OpenAI, which is the GPT-4.0 model. And this time, let's set the temperature to 0. Now, let's go ahead and make a couple of changes to the bi-encoder,right?

So we know that our vector database is going to be the quadrant vector database,right? We have our storage solution. And now let's go ahead and query using this new language model that we have, which is the GPT-4.0 model,right?

And our new embedder, which is open embedding solution. Let's go ahead and query that, and let's see what sort of response we now get from our RAG solution,right? And you can see that the response has changed. You can get help.

So in response to the question, where can I get help in London? I'm at the station. The response has now changed to, you can get help at the station in London by arriving at least 75 minutes before departure and letting a member of the team know and so on.

Now, this again is not a particularly helpful response, especially if you've taken a good look at the knowledge base. But let's try and see how we can improve on that. Before we do that, let's try and see where the RAG solution was able to retrieve this information from.

Now, in LlamaIndex, this is known as the source nodes, and it provides information about which files were used to provide that response. And if we wanted to take a look at those files, we can do so by looking at the source nodes,right?

And so it's used these two HTML files in order for it to provide that response. So can I get assistance for a connecting UK train journey? And it's taken a look at some of the information available here, and also information from this second file over here.

Now, let's make our next step, which is to use our second improvement, which is where we're going to use a cross-encoder or our reranker solution. This time, we're going to use a closed model. So that's the one from Cohere.

And what we're going to do is rerank the top two results. So again, back to the five lines of LlamaIndex code,right? We want to go ahead and pull all of the information from the directory. We want to then go ahead and use our vector database, which is quadrant.

We then want to go ahead and take those five results from the vector database based on that query. And we want to then go ahead and use the Cohere rerank as our node post-processor. And in response to that original query, which is, where can I get help in London?

I'm at the station. We get the best response so far. So at London St Pancras International, you can get help by going to booth number five next to the Eurostar ticket gates. Now, one thing that we've not looked at, but is a very important part of monitoring and tracing a solution, especially when you deploy it into production, is something like the Phoenix Arise or the Langsmith solution,right?

Now, what this allows you to do is it allows you to determine how long each of the different components takes. So another thing that's really important is, so far, I've just looked at this one question, and I've only asked this one question to the RAG solution, which is, where can I get help in London?

What you'll want to do when working with a RAG solution is to have a RAG evaluation framework that will allow you to test on a whole load more documents. Now, one really good solution is the RAGAS solution. In production, I'm actually going to be using open model, the likes of BAAI or the ones from NVIDIA.

Vector database quadrant scales incredibly well. Few documents to hundreds of thousands of documents. Monitoring and tracing, again, very important to be able to track and troubleshoot your RAG pipeline. So Langsmith or Arise Phoenix in prototyping, and then there is a very good Docker solution available from Arise Phoenix that you can just pull.

Rerankers, I'll use the closed Cohere solution for prototyping, but in production, I'll probably use the NVIDIA solution that allows you to do reranking. And finally, RAG evaluation. RAGAS is great, and it allows you to check the quality of your RAG solutions across a couple of different ways of looking at that.

And it works with the large language model, so it makes that task pretty painless. Now, just a quick look at the production environment. I have a comp YAML file. This will work together with Docker Compose, and then I'll have a couple of the Docker images that I'll use here.

So for example, I'll have an image for ingesting the data that will be connected to the knowledge base, where I'll pull in all of the HTML files. I'll have an image for quadrant. That's going to be my vector database that I can just pull from Docker Hub.

I'll have a front-end app for my solution. Often, I'll use Ollama to serve my models, but you can also use Hugging Face's text generation inference engine here. Phoenix for my tracing,right? And RAGAS to be able to evaluate my models.

And then I can run each of these images and have them as containers within Docker Compose. And then I've got details about the different models that I'd use for the embeddings and reranking and the large language models over here.

So these are my results. These are my lessons from the RAG stack we landed on after 37 fails. I hope that this has been helpful to you. I'm always happy to answer any questions, so you're very welcome to connect with me and chat with me on LinkedIn.

Thank you for your time.