The Hidden Costs of Building Your Own RAG Stack — Ofer Vectara

Okay. Hi everyone, and welcome to the hidden cost of building your own RAG stack. My name is Ofer, I run Developer Relations at Vectara, and I'm a machine learning engineer by training and a software engineer, and I was fortunate enough to work on LLMs since GPT just turned 2 in 2019.

Now, I'm going to talk about the hidden cost of RAG stack. I think we all know what it is, but just for those of you who may not be familiar, it's a way of using LLMs not directly like we use to send in a question and get an answer, but instead you want to point it into your own data.

It could be a bunch of PDF files, it could be data from Notion, or wherever else. And instead of calling the LLM directly, you first retrieve the most relevant facts for the question out of your data using a Retrieval Engine, and then the LLM is grounded in this data when it creates the response.

Now, it turns out that building a RAG platform that is enterprise-scale is much harder than most people seem to realize, and this is what I'm going to talk about mostly in this talk. Before I go there, just to kind of align everybody with a couple of components and things that are included in the stack.

So this is kind of my version of the diagram of how RAG kind of works. It has two main flows. The first one is kind of in the blue arrow is the ingest. So you take data from wherever it is.

It could be a set of documents, it could be a database, it could be a Jira or Salesforce or whatever it may be. And you—this is the data you want to be grounded on or retrieved from—and you ingest it into a bunch of steps that you have to take care of.

You extract the data, if it's a PDF file or a document of some kind. You do something called chunking, which is breaking the text into smaller pieces, shown here. And then each chunk is then encoded using an embedding model, stored in a vector database.

There's also other pieces here, like you have to store the text somewhere and things like this, but for simplicity I'm going to skip those. And then the green arrow is where a query comes in, sometimes with some filtering parameters, but the query text itself is again embedded into a vector.

The vectors run against a vector store, and those relevant facts are retrieved. In many cases, you want a stronger Retrieval Engine than just a vector search, so you kind of want to do sometimes hybrid search, which combines vector and keyword search.

You also might want to do reranking, which can be relevance reranking or diversity reranking or some other business logic-related reranking you want to do. At the end of this retrieval step, you get a set of relevant chunks or relevant facts, and you send them with a prompt to the LLM and say, "Please respond to this query given the facts."

And then—and I'll talk more about hallucination detection—but sometimes what you want to do also is detect hallucination to see if the LLM actually did what it's supposed to do, and use the facts in a proper way. And then once you have the response, you send it back to the user.

So this is kind of a build of a RAG stack, and as you can see, there's a lot of different components here. And just a quick thing about Vectara, what we do is we essentially put all of this in a box.

So think of it as RAG as a service, where all these components end up being part of what we do, what we provide. It could be in a SaaS format when you send all the data to us, or it could be on your own VPC.

It could also be on-premise in your own hardware. But this stack is prebuilt, and you don't have to build it or work on it. All you have to do is use these external APIs for indexing and for querying to first index the data you want the RAG stack to work on, and then when you have some user interface, you can use the query API to run the query flow here.

But all of the internals are hidden from you and saved from the trouble of managing those components. Okay. Now I'm going to get to the meat of what I wanted to talk about, which is there is a very big difference between a sort of DIY RAG stack when you build all of these components yourself.

You put in the vector database and the Retrieval Engine, and you figure out which embedding mode to use and which LLM to use and all those pieces. A big gap between that and using a platform like Vectara, where it's all taken care of for you.

And as we talk to people in the community and customers, we figure there's about 7 things I want to mention today that are kind of pitfalls when you do build your DIY RAG, and I want to talk about these 7.

So irrelevant responses and hallucinations, which I'll talk about. Maintaining low latency, not getting high latency. Scaling up without making the cost be too high. You want to make sure you don't have any issues with compliance and security, of course.

There's something called vendor chaos, sustainable keep, and additional language. So let me talk about each of those separately. So the first one is the quality of responses or hallucinations. We know the hallucinations are still a really big deal, a really big problem, both in pure LLMs but also in RAG.

And at scale, getting theright results first, getting theright facts, is a problem. So you have to invest when you build your own infrastructure in parsing, in chunking. There's a lot of different chunking strategies in parsing of tables and table understanding in general.

Hybrid search. All these things take a lot more work than kind of the initial PLC usually provides. And so you have to realize that when you really want to get high-quality results, you have to invest in all of these components and get them to work properly.

Last but not least, I want to mention there's evaluation of the response quality, which is a complicated topic of its own. And it's not just evaluating on the first deployment. You kind of want to continuously evaluate the results as you add more data, refresh the data, change components.

You want to make sure that your quality of the results remains the same or better. And then hallucinations, again, still remains a big issue. And implementing techniques to fight hallucinations or even correct hallucinations can be quite complicated. The last thing I want to mention here is just explainability.

This means that when you have a result, you want to usually show what the facts are. You have a path for people to look at the facts, click on a link, and see where it came from. Again, that's something that you want to integrate into your UI, but your system has to keep the path or the audit trail of where the results came from so that you can show them.

Okay. The second one is pretty straightforward for any architect or systems engineer,right? You want the latency to be low, and a lot of times it gets much higher than you want, again, just because there's multiple components and not super orchestrated and harmonized.

And so retrieval,right? Simple retrieval of vector store could be fast, but when you want to do, let's say, hybrid search or some other rankers, you may get into a lot more trouble in latency, and it kind of starts to add up.

And your system depends on the weakest link,right? So if one of your components has a latency issue, the net suffers. That becomes the latency for the whole flow of queries. So we all know that users just want to get answers fast.

And just wanted to highlight that this is not as trivial as you think when you come to kind of production grade. Okay. Pitfall number 3: scaling and cost. So you have a lot of components. You have embedding models, you have LLMs, a vector database, a bunch of other things.

You have to sometimes use external services to enhance your PDF parsing or table understanding. And generally, maintaining low latency and high quality just requires good components in your stack. And then on top of that, you have cost of GPUs and CPUs and storage.

And so all of these things really ramp up when you start going production scale with not just 10 documents, but thousands or hundreds of thousands or millions of documents. And the token cost also starts becoming something that's hard to maintain.

Number 4: security and compliance issues. So in security, I wanted to highlight first the idea of attribute-based access control. A lot of times you want to implement that as part of your RAG stack. This means that if some of the documents in your application are only visible in your organization, let's say to the HR or to the CEO, you want to make sure that your RAG stack doesn't leak that information.

You want to have the same access controls implemented in your RAG flow. Similarly, if you have sensitive information like PHI or PII, you want to make sure that that's ingested in theright way and treated in theright way throughout the stack.

A lot of these failures to do these kind of things and similar things in security and compliance can have a pretty large cost in any organization. So you want to make sure that you do thatright.

Okay. This is one of my favorite ones to talk about, actually. Vendor chaos. What I mean by that is, if you remember the diagram I started with that has all the different components and the flows, there's quite a bit of components there.

You have your vector database, you have your LLM, you have your embedding models, you have your maybe keyword search for hybrid search, etc. If you use different components for this and it's not just one kind of harmonized stack, you start having multiple vendors.

So you have multiple contracts, you have to onboard all of them, and the integration can become much more costly than you might imagine. The thing that frightens me the most with these kind of things when I think about kind of real enterprise scale is how do you diagnose issues?

You have multiple vendors, you can have things like finger-pointing coming up, or you really just want to have the problem solved. So it could be here, it could be there. How do you diagnose? How do you know? And who gives you the support?

How do you get the support coordinated? So that could be a pretty big nightmare, so something to be watchful for.

Allright. I'm going to the last two here. So number 6 is unsustainable keep. And again, this means that RAG stack is a pretty unique set of skills and one that moves really quickly. So you have to have in your team, if you build it yourself, experts in LLMs, experts in embedding models, two different expertise areas.

Experts in hybrid search if you implement that, etc. Everything you do, you have to have some specific expertise there. Not easy to get in many organizations. You have to have data engineering expertise to do all the data ingest and ETL flows and parsing the PDFs and parsing the documents and things like this.

Metadata management. And then you have to have somebody running the system, making sure it's up and running all the time. So you got to have your DevOps, SREs, etc., security experts. So it's not always easy to find those resources and keep them as a separate team that can continuously maintain the system up and running to support all of your GenAI use cases.

Takes a lot of work. And then last is a lot of people start with English. It's the easiest one. Everything supports English, and you get a PLC up and running. But when you go to non-English users, which may be a requirement for your organization, ends up that it's not as easy.

You have to make sure that each of the components supports the language you need, and sometimes they don't, and then you have to change, and then your quality suffers or something like this. So you got to think about it upfront if you want to go to a production scale as well.

Allright. So those are kind of the 7 pitfalls and things that we see in many transitions that we see of customers moving from kind of the initial PLC and then starting to think about their production deployment. And again, this is why our approach is to have a platform.

It's a plug-and-play RAG as a service platform. And we've built all of these components: the extraction, the encoding, the indexing, the vector database, the retrieval. All of these pieces are part of the platform. You don't have to create them, you don't have to manage them, you don't have to upgrade them.

We will do all that for you. And as a user of Vectara, what you do is you have APIs around it that you can use to upload your data or index your files and then to run queries or chat or any other agentic RAG applications through that.

So that's really the idea behind Vectara and why it's so powerful for big enterprises that have a lot of use cases. The three things that people usually choose us for are kind of aligned with some of the things I said earlier: accuracy.

We focus a lot on hallucinations and reducing hallucinations, and I'll share in a minute what that means. A lot of emphasis on good retrieval and quality results and retrieval of the facts. Security mechanisms like access controls and prompt attack prevention.

And also explainability to make sure that you have observability of what's going on in the system and you have citations throughout the flow. I talked about hallucinations a little bit. I want to highlight this a little bit more.

So we've seen hallucinations a lot in 2024 and still this year. It continues to be a significant issue for adoption. And this is an example of just an article from last Thanksgiving and a couple of quotes we got from that article.

So especially in regulated environments, hallucinations can become a really big hurdle for adoption and for production deployment. One of the things we focus on in Vectara is called HHEM or Hallucination Evaluation Model, Detection Model. We have this as part of our system.

We've implemented this as part of the flow. And with every call to Vectara, you get your hallucination score, which is really helpful because it can help you understand if the LLM did a good job or not and use that.

This model was also open sourced on Hugging Face. You can see here it's got over 3 million downloads since it started. So it's very popular. It's by far the most popular evaluation for hallucinations out there. And we also built this leaderboard, which might be interesting for everybody here listening.

This we take a bunch of the LLMs and evaluate how likely they are to hallucinate on some datasets. So we use HHEM, sort of the commercial version of it, to build this leaderboard. And this gives a lot of people sort of a good indicator on top of the core capabilities of the LLM, how much they might hallucinate or not.

And I couldn't show all the leaderboard here because there's a lot of models here, but some of them hallucinate at much higher rates. This is just the top. It could be sometimes like 10% or even 14% and things like this.

So you got to choose your LLM wisely. I mentioned that we also

were a RAG as a service, but of course, you can also deploy this in your VPC and on-premises, and that's important to know. Many of our customers require that. And that's it. Thank you so much for listening. I encourage you to try Vectara.

I have a QR code here. You can check it out. We have a 30-day free trial with all the full capabilities of the platform. And if you are interested to talk more, please feel free to reach out. Thank you so much.