Building Production-Ready RAG Applications: Jerry Liu

Hey everyone. My name is Jerry, co-founder and CEO of LlamaIndex, and today we'll be talking about how to build production-ready RAG applications. I think there's still time for a raffle for the bucket hat, so if you guys stop by our booth, please fill out the Google form.

Okay, let's get started. So, everybody knows that there's been a ton of amazing use cases in GenAI recently. You know, knowledge search in QA, conversational agents, workflow automation, document processing. These are all things that you can build, especially using the reasoning capabilities of LLMs, over your data.

So, if we just do a quick refresher, in terms of, like, paradigms for how do you actually get language models to understand data that hasn't been trained over, there's really, like, two main paradigms. One is retrieval augmentation, where you, like, fix the model and you basically create a data pipeline to put context into the prompt from some data source into the input prompt of the language model.

So, like, a vector database, you know, like unstructured text SQL database, etc. The next paradigm here is fine-tuning. How can we bake knowledge into the weights of the network by actually updating the weights of the model itself, some adapter on top of the model, but basically some sort of training process over some new data to actually incorporate knowledge?

We'll probably talk a little bit more about retrieval augmentation, but this is just, like, to help you get started in really understanding the mission statement of the company. Okay, let's talk about RAG, Retrieval Augmented Generation. It's become kind of a buzzword recently, but we'll first walk through the current RAG stack for building a QA system.

This really consists of two main components: data ingestion, as well as data query, which contains retrieval and synthesis. If you're just getting started in LlamaIndex, you can basically do this in around, like, five-ish lines of code, so you don't really need to think about it.

But if you do want to learn some of the lower-level components, and I do encourage, like, every engineer, AI engineer, to basically just, like, learn how these components work under the hood, I would encourage you to check out some of our docs to really understand how do you actually do data ingestion and data query.

Like, how do you actually retrieve from a vector database, and how do you synthesize that with an LLM? So, that's basically the key stack that's kind of emerging these days. Like, for every sort of chatbot, like, you know, chat over your PDF, like, over your unstructured data, a lot of these things are basically using the same principles of, like, how do you actually load data from some data source and actually, you know, retrieve and query over it.

But I think as developers are actually developing these applications, they're realizing that this isn't quite enough. Like, there's certain issues that you're running into that are blockers for actually being able to productionize these applications. And so, what are these challenges with naive RAG?

One aspect here is just, like, the response. And this is the key thing that we're focused on. Like, the response quality is not very good. You run into, for instance, like, bad retrieval issues. Like, during the retrieval stage from your vector database, if you're not actually returning the relevant chunks from your vector database, you're not going to be able to have the correct context actually put into the LLM.

So, this includes certain issues like low precision. Not all chunks in the retrieve set are relevant. This leads to, like, hallucination, like, lost-in-the-middle problems. You have a lot of fluff in the return response. This could mean low recall.

Like, your top K isn't high enough, or basically, like, the set of, like, information that you need to actually answer the question is just not there. And of course, there's other issues too, like outdated information. And many of you who are building apps these days might be familiar with some, like, key concepts of, like, just why the LLM isn't always, you know, guaranteed to give you a correct answer.

There's hallucination, irrelevance, like, toxicity bias. There's a lot of issues on the LLM side as well.

So, what can we do? What can we actually do to try to improve the performance of a retrieval augmented generation application? And for many of you, like, you might be running into certain issues, and it really runs the gamut across, like, the entire pipeline.

There's stuff you can do on the data. Like, can we store additional information beyond just, like, the raw text chunks,right, that you're putting in the vector database? Can you optimize that data pipeline somehow? Play around with chunk sizes, that type of thing.

Can you optimize the embedding representation itself? A lot of times, when you're using a pre-trained embedding model, it's not really optimal for giving you the best performance. There's the retrieval algorithm. You know, the default thing you do is just look up the top K most similar elements from your vector database to return to the LLM.

Many times that's not enough. And what are kind of, like, both simple things you can do as well as hard things? And there's also synthesis. Like, why is there, yeah, there's, like, a V in the. Anyway, so can we use LLMs for more than generation?

And so basically, like, you can use the LLM to actually help you with, like, reasoning as opposed to just, like, pure, just, like, just pure generation,right? You can actually use it to try to reason over given a question.

Can you break it down into simpler questions, route to different data sources, and kind of, like, have a more sophisticated way of, like, querying your data?

Of course, like, if you kind of been around some of my recent talks, like, I always say, before you actually try any of these techniques, you need to be pretty task-specific and make sure that you need a way to, that you actually have a way to measure performance.

So, I'll probably spend, like, two minutes talking about evaluation. Simon and my co-founder just ran a workshop yesterday on really just, like, how do you evaluate, build a dataset, evaluate RAG systems, and help iterate on that. If you missed the workshop, don't worry, I'll, we'll have the slides and materials available online so that you can take a look.

At a very high level, in terms of evaluation, it's important because you basically need to define a benchmark for your system to understand how are you going to iterate on and improve it. And there's, like, a few different ways you can try to do evaluation,right?

I think Anton from Chroma was just saying some of this, but, like, you basically need a way to evaluate both the end-to-end solution, like, you have your input query as well as the output response. You also want to probably be able to evaluate, like, specific components.

Like, if you've diagnosed that the retrieval is, like, the portion that needs improving, you need, like, retrieval metrics to really understand how can you improve your retrieval system. So, there's retrieval and there's synthesis. Let's talk a little bit, just, like, 30 seconds on each one.

Evaluation on retrieval. What does this look like? You basically want to make sure that the stuff that's returned actually answers the query and that you're kind of, you know, not returning a bunch of fluff and that the stuff that you return is relevant to the question.

So, first, you need an evaluation dataset. A lot of people have, like, human-labeled datasets. If you're building stuff in prod, you might have, like, user feedback as well. If not, you can synthetically generate a dataset. This dataset is input, like, query, and output, the IDs of, like, the returned documents are relevant to the query.

So, you need that somehow. Once you have that, you can measure stuff with ranking metrics,right? You can measure stuff like success rate, hit rate, MRR, NDCG, a variety of these things. And so, like, once you are able to evaluate this, like, this really isn't kind of like an LLM problem.

This is, like, an IR problem. And this has been around for at least, like, a decade or two. But a lot of this is becoming, like, you know, it's still very relevant in the face of actually building these LLMs.

The next piece here is, there's a retrieve portion,right? But then you generate a response from it. And then how do you actually evaluate the whole thing end-to-end? So, evaluation of the final response given the input. You still want to generate some sort of dataset.

So, you could do that through, like, human annotations, user feedback. You could have, like, ground truth reference answers given the query that really indicates, like, hey, this is the proper answer to this question. And you can also just, like, you know, synthetically generate it with, like, GPT-4.

You run this through the full RAG pipeline that you built, the retrieval and synthesis, and you can run, like, LLM-based evals. So, label-free evals, with label evals. There's a lot of projects these days going on about how do you actually properly evaluate the outputs, predicted outputs of a language model.

Once you've defined your eval benchmark, now you want to think about how do you actually optimize your RAG systems. So, I sent a teaser on the slide, like, yesterday, but the way I think about this is that when you want to actually improve your system, there's, like, a million things that you can do to try to actually improve your RAG system.

And, like, you probably don't want to start with the hard stuff first, just because, like, you know, part of the value of language models is how it's kind of democratized access to every developer. It's really just made it easy for people to get up and running.

And so, if, for instance, you're running into some performance issues with RAG, I'd probably start with the basics. Like, I call it, like, table stakes RAG techniques. Better parsing so that you don't just split by even chunks. Like, adjusting your chunk sizes, trying out stuff that's already integrated with a vector database, like hybrid search, as well as, like, metadata filters.

There's also, like, advanced retrieval methods that you could try. This is, like, a little bit more advanced. Some of it pulls from, like, traditional IR. Some of it's more, like, kind of really, like, new in this age of, like, LLM-based apps.

There's, like, reranking. That's a traditional concept. There's also concepts in LlamaIndex, like, recursive retrieval, like, dealing with embedded tables, like, small-to-big retrieval, and a lot of other stuff that we have that help you potentially improve the performance of your application.

And then the last bit, like, this kind of gets into more expressive stuff that might be harder to implement, might incur a higher latency and cost, but is potentially more powerful and forward-looking, is, like, agents. Like, how do you incorporate agents towards better, like, RAG pipelines to better answer different types of questions and synthesize information?

And how do you actually fine-tune stuff? Let's talk a little bit about the table stakes first. So, chunk sizes. Tuning your chunk size can have outsized impacts on performance,right? If you've kind of, like, played around with RAG systems, this may or may not be obvious to you.

What's interesting, though, is that, like, more retrieve tokens does not always equate to higher performance, and that if you do, like, reranking of your retrieve tokens, it doesn't necessarily mean that your final generation response is going to be better.

And this is, again, due to stuff like lost-in-the-middle problems, where stuff in the middle of the LLM context window tends to get lost, whereas stuff at the end tends to be a little bit more well-remembered by the LLM.

And so, I think we did a workshop with, like, Arise a few, a week ago, where basically we showed, you know, there is kind of, like, an optimal chunk size given your dataset. And a lot of times when you try out stuff like reranking, it actually increases your error metrics.

Metadata filtering. This is another, like, very table stakes thing that I think everybody should look into. And I think vector databases, like, you know, Chroma, Pinecone, Weaviate, like, these vector databases are all implementing these capabilities under the hood.

Metadata filtering is basically just, like, how can you add structured context to your chunks, like, your text chunks? And you can use this for both, like, embeddings as well as synthesis, but it also integrates with, like, the metadata filter capabilities of a vector database.

So, metadata is just, like, again, structured JSON dictionary. It could be, like, page number. It could be the document title. It could be the summary of adjacent chunks. You can get creative with it, too. You could hallucinate, like, questions that the chunk answers.

And it can help retrieval. It can help augment your response quality. It also integrates with the vector database filters. So, as an example, let's say the question is over, like, the SEC, like, 10Q document, and, like, can you tell me the risk factors in 2021?

If you just do raw semantic search, typically it's very low precision. You're going to return a bunch of stuff that may or may not match this. You might even return stuff from, like, other years if you have a bunch of documents from different years in the same vector collection.

And so, like, you're kind of, like, rolling the dice a little bit.

But one idea here is basically, you know, if you have access to the metadata of the documents and you ask a question like this, you basically combine structured query capabilities by inferring the metadata filters, like a WHERE clause in a SQL statement, like a year equals 2021, and you combine that with semantic search to return the most relevant candidates given your query.

And this improves the precision of your results.

Moving on to stuff that's maybe a bit more advanced, like, advanced retrieval is one thing that we found generally helps is this idea of, like, small-to-big retrieval. So, what does that mean? Basically,right now, when you embed a big text chunk, you also synthesize over that text chunk.

And so, it's a little suboptimal because what if, like, the embedding representation is, like, biased because, you know, there's a bunch of fluff in that text chunk that contains a bunch of relevant information? You're not actually optimizing your retrieval quality.

So, embedding a big text chunk sometimes feels a little suboptimal. One thing that you could do is basically embed text at the sentence level or on a smaller level and then expand that window during synthesis time. And so, this is contained in a variety of, like, LlamaIndex abstractions, but the idea is that you return, you retrieve on more granular pieces of information, so smaller chunks.

This makes it so that these chunks are more likely to be retrieved when you actually ask a query over these specific pieces of context, but then you want to make sure that the LLM actually has access to more information to actually synthesize a proper result.

So, this leads, like, more precise retrieval,right? So, we tried this out. It helps avoid, like, some lost-in-the-middle problems. You can set a smaller top K value, like K equals 2, whereas, like, over this dataset, if you set, like, K equals 5 for naive retrieval over big text chunks, you basically start returning a lot of context, and that kind of leads into issues where, you know, maybe the relevant context is in the middle, but you're not able to find out, or you're, like, the LLM is not able to kind of synthesize over that information.

A very related idea here is just, like, embedding a reference to the parent chunk as opposed to the actual text chunk itself. So, for instance, if you want to embed, like, not just the raw text chunk or not the text chunk, but actually, like, a smaller chunk or a summary or a question that answered the chunk, we have found that that actually helps to improve retrieval performance a decent amount.

And it kind of, again, goes along with this idea. Like, a lot of times you want to embed something that's more amenable for embedding-based retrieval, but then you want to return enough context so that the LLM can actually synthesize over that information.

The next bit here is actually kind of even more advanced stuff,right? This goes on into agents, and this goes on into that last pillar that I mentioned, which is how can you use LLMs for reasoning as opposed to just synthesis.

The intuition here is that, like, for a lot of RAG, if you're just using the LLM at the end, you're one constrained by the quality of your retriever, and you're really only able to do stuff like question answering.

And there's certain types of questions and more advanced analysis that you might want to launch that, like, top K RAG can't really answer. It's not necessarily just a one-off question. You might need to have, like, an entire sequence of reasoning steps to actually pull together a piece of information, or you might want to, like, summarize a document and compare it with, like, other documents.

So, one kind of architecture we're exploringright now is this idea of, like, multi-document agents. What if, like, instead of just, like, RAG, we moved a little bit more into agent territory? We modeled each document not just as a sequence of text chunks, but actually as a set of tools that contains the ability to both, like, summarize that document as well as to do QA over that document over specific facts.

And, of course, if you want to scale to, like, you know, hundreds or thousands or millions of documents, typically an agent can only have access to a limited window of tools. So, you probably want to do some sort of retrieval on these tools similar to how you want to retrieve, like, text chunks from a document.

The main difference is that because these are tools, you actually want to act upon them. You want to use them as opposed to just, like, taking the raw text and plugging it into the context window. So, blending this combination of, like, kind of embedding-based retrieval or any sort of retrieval as well as, like, agent tool use is a very interesting paradigm that I think is really only possible with this age of LLMs and hasn't really existed before this.

Another kind of advanced concept is this idea of fine-tuning. And so, fine-tuning, you know, some other presenters have talked about this as well, but the idea of, like, fine-tuning in a RAG system is that it really optimizes specific pieces of this RAG pipeline for you to kind of better, like, improve the performance of either retriever or synthesis capabilities.

So, one thing you can do is fine-tune your embeddings. I think Anton was talking about this as well. Like, if you just use a pre-trained model, the embedding representations are not going to be optimized over your specific data.

So, sometimes you're just going to retrieve the wrong information. If you can somehow tune these embeddings so that given any sort of, like, relevant question that the user might ask, that you're actually returning the relevant response, then you're going to have, like, better performance.

So, an idea here,right, is to generate synthetic query dataset from raw text chunks using LLMs and use this to fine-tune an embedding model. And you can do this, like, if we go back really quick, actually, you can do this by basically kind of fine-tuning the base model itself.

You can also fine-tune an adapter on top of the model. And fine-tuning an adapter on top of the model has a few advantages in that you don't require the base model's weights to actually fine-tune stuff. And if you just fine-tune the query, you don't have to re-index your entire document corpus.

There's also fine-tuning LLMs, which, of course, like, a lot of people are very interested in doing these days. An intuition here specifically for RAG is that if you have a weaker LLM, like 3.5 Turbo, like LLaMA 207B, like, these weaker LLMs are bad, are not bad at, like, wait, yeah, weaker LLMs are maybe a little bit worse at, like, response synthesis, reasoning, structured outputs, et cetera, compared to, like, bigger models.

So, the solution here is what if you can generate a synthetic dataset using a bigger model, like GPT-4, that's something we're exploring, and you actually distill that into 3.5 Turbo. So, it gets better at chain of thought, longer response quality, better structured outputs, and a lot of other possibilities as well.

So, all these things are in our docs. There's production RAG, there's fine-tuning, and I have two seconds left. So, thank you very much.