RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot

It seems that RAG is so 2023, and everywhere you go you see "my first RAG" and "RAG pipelines" and everything is RAG-related, as if it's already solved. But actually, not really. Hello. I'm Yuval, with me is Niv, we both work at AI21 Labs, and today we're going to talk about why RAG evaluation is broken and how to fix it.

And the reason that RAG evaluation is broken comes from a lot of things, but mostly because how easy it is to build benchmarks the way humans are regular to think. And what I mean by that is that most benchmarks comprise from local questions that has local answers.

There is some sort of assumption that the answer lies in a certain chunk in the data. Makes sense,right? What is the most natural way to build such a benchmark? To go read some long document, find a question that the answer lies somewhere inside, and then this is the golden answer.

And there are a lot of benchmarks who try to overcome that. There's a lot of multiple hope questions, famously FRAMES by Google, but those questions are not really realistic,right? Something like, "If my future wife have the same name of the 15th First Lady of the United States Mother and her surname," what is this?

Does it even mean something? Does it represent something real? Not really. So this is all very manufactured and not really representative of the real world. There's also not a holistic way to test the entire system. Most of the benchmarks are either retrieval-only benchmark, meaning that we try to retrieve the best segments or the best chunks for our database, and we assume that, again, the answer is in one or two or several of them.

Or it's generation-only benchmarks, which is essentially just grounding benchmark,right? Just to see that we can answer a question based on the contexts that are already in the prompt. But what about chunking? What about parsing? What about specific cases that do not actually have this effect of the question directs us to a certain place in the data?

The answer is not as simple. And really, that's the main problem here, that benchmarks do not correlate with real-world data. That real-world data is messier. Each data is different, and this thing is not really generalized as well. We have some sort of a vicious cycle when we are developing a RAG system, where we're building a RAG system for flawed benchmarks, because we have to optimize for something.

And then when we get high scores, we celebrate about it, we write a nice post, whether on Slack, on LinkedIn, on Twitter, and then look at us, we're so good. But then when we actually give it to users or we test it on customer data, we see that they struggle.

It's not as good as we hoped. So we create new benchmarks, optimize for them, and we're showing that we're the best. But those benchmarks probably have the same problems. And this goes over and over and over. RAG pipelines with local questions and local answers.

That's what we have in most of the cases. So how do we fix it?

Um, so we will not say how we will fix everything, but we'll try to think some things. So for example, if you take like a financial data, like FinanceBench or others, like SEC filings or any other financial data, you may imagine that someone would ask aggregative questions such as, "Which company has reported the highest quarterly revenue the most times?"

or "In how many fiscal years did Apple exceed $100 billion in annual revenue?" Or list all of something. So you may imagine how RAG systems currently do on such questions, because they are so limited and they just grab the top K chunks that they get and try to compile an answer by that.

But when you encounter a question like, "All Fortune 500 companies?" So imagine that you bring 10 chunks. Then maybe you will get another Fortune 500 company in the 11th one and the 12th one. And for every N, you can consider the N plus one.

So these problems really make RAG systems struggle.

So we try to estimate how do current systems do with such questions. So we built a corpus, a small corpus of like 22 documents from the historical pages of FIFA World Cup in Wikipedia. So this is our corpus, and we asked many questions about this corpus, such as which team has won the FIFA World Cup the most times, in how many FIFA World Cup did Brazil participate.

And we used the knowledge base, the Wikidata and Kaggle knowledge bases to answer these questions. And as you can see, a common RAG pipeline, like the first one you'll see on LangChain or LlamaIndex, and OpenAI responses, both fail miserably.

It's 5% and 11% of the questions answered correctly, and these are questions that you can find the answer to in very local segments, as Yuval mentioned.

So our idea to handle such corpuses is take this unstructured corpus and convert it into a data structure, and then ask the question on top of this data structure. Because essentially, the questions we talked about are questions, are SQL questions,right?

It's just like, "How many times?" or it's counting questions or max and min questions or calculation questions. And you just, it just doesn't make sense to try to answer these questions by going over the pages or specific chunks of the pages.

So how do we do it? We split it in two, similarly to how a regular RAG flow is done, where you invest the compute in the ingestion and you try to do things quickly in the inference. So in the ingestion phase, we first cluster the document into sub-corpuses.

For example, we may have financial corpus and FIFA World Cup corpus that we do this clustering. Then per each sub-corpus, we identify the schema that represents this corpus. We populate the schema according to each and every document. And finally, we upload the results into an SQL DB.

So for example, if you have the FIFA corpus and you have a schema which is compiled of a year, the winner, the top three teams, the top scorer, and many other attributes. And then you use an LLM. It's not just an LLM, it's like a pipeline.

But its goal is to populate this schema for every document. At the inference flow, whenever we have a query, we identify the schema that is relevant for this query. For example, is it a financial question or a FIFA World Cup question?

And then we just do a regular text to SQL over our data and we return the final answer. So here, for the FIFA World Cup, which team has won the most, the FIFA World Cup the most times, it's just a simple SQL query.

This approach does not come to solve everything, not at all. First and foremost, not every corpus or query is relational DB material. Not every corpus is homogeneous in terms of the attributes. It doesn't necessarily contain a schema underlying it.

But we do see many such corpuses with our clients. Normalization. Even with a toy example such as the FIFA World Cup, we see the struggle with building the correct schema. Because, for example, in the host country, you have West Germany.

But then if I want to ask how many times did Germany won, does West Germany count or doesn't it? A host country, is it a singular attribute or is it a list? Because South Korea and Japan hosted together the World Cup.

So the normalization is an issue both during the ingestion and inference.

Abstinence or ambiguity. What happens if a user asks me, "Did Real Madrid win the 2006 final?" It's not a World Cup, but I do see some things that I can try to do with my schema. And as you probably know, LLMs

tend to try to please the users.

The points in the ingestion where we try to cluster and infer the schema, there's some trade-off here on the complexity and how fine-grained we are and how much compute we'd like to invest during the ingestion time. And finally, the text to SQL, whenever the schemas become complex, is just a complex issue, which is very known and studied.

So the main takeaways are just that RAG is not a one-size-fits-all system, and you have to account for every client separately.

There are, of course, clusters, but we should note that the regular pipeline of chunking, embedding, retrieving, reranking is not good enough for many questions. Existing benchmarks fail to capture these use cases. As Yuval mentioned, they're very limited. And in order to solve the problems, we may need to go beyond standard RAG for specific settings.

If you thought this is interesting and you want to learn more about structured RAG and the problems with RAG evaluation, please check out our full episode of YAAP podcast by AI21. Thank you.