open-rag-eval: RAG Evaluation without "golden" answers — Ofer Mendelevitch, Vectara

Hi everyone, my name is Ofer from Vectara. I run our developer relations, and I'm going to talk to you today about Open-RAG-Eval, a new open-source project for quick and scalable RAG evaluation. You can use the QR code here to get to the repo directly.

So, what is Open-RAG-Eval? It's an open-source project that is aiming to solve one of the major problems of RAG evaluation: the fact that you require golden answers or golden chunks to do anything, which is really non-scalable. It is research-backed, so it's work that we've done with collaboration with the University of Waterloo, the Jimmy Lin Lab specifically, and let me show you how it works.

So generally, this is kind of an architecture view of what it does. You start with a set of queries; it could be 10, or 100, or 1,000 queries that you've collected that are important for your RAG system. We have a RAG connector, and that really collects all the actual information, the actual chunks, the actual answers that are generated by your RAG pipeline.

We have one for Vectara, for LangChain, for LlamaIndex, and a growing number of other connectors. And those connectors generally generate these outputs shown here, which are the RAG outputs. From there, you run the actual evaluation, which runs a bunch of metrics.

Metrics are grouped into evaluators, and that's kind of how the internal architecture works. And those evaluators generate RAG evaluation files, and that has everything you need to know to really evaluate your RAG pipeline. Now, the metrics are where it gets really interesting.

How do we do this thing with no golden answers? Well, there are a few metrics I want to mention here that are part of this package. One is called UMBRELA. It really allows you to do retrieval without the golden chunks.

The other one is called AutoNuggetizer. I'll talk more about this for generation. We have the citation faithfulness, which measures whether citations in the response are really correct, and hallucination detection, which is based on Vectara's hallucination detection model. So let's go into each of these in detail.

UMBRELA is a retrieval metric, as I mentioned, and what it does is it actually takes a chunk and gives it a score between 0 and 3. 0 being that this chunk or passage has nothing to do with the query, and 3 being that it's dedicated to the query and contains the exact answer.

Now, the nice thing about this is not just the scale here, which is pretty self-explanatory, but the fact that the research done by the University of Waterloo lab of Jimmy Lin shows that if you use this approach, it correlates well with human judgment, and that is really, really powerful.

So if you use this, even without the golden chunks, you know that the results will be good. Now, AutoNuggetizer is the generation metric, and again, it doesn't require the golden answers in this case. And it works a little bit differently.

There are three steps here. The first step is nugget creation, where you create these atomic units called nuggets. Then, for each nugget, you assign a Vital or OK rating. You sort of sort these nuggets by getting the top 20.

And then there's a step where an LLM judge analyzes the response you get from your RAG to determine if each of the selected nuggets is either fully supported or partially supported by the answer. Again, you can see all of this in the papers in much more detail.

The third metric is citation faithfulness, which essentially measures whether the citation, the passage, is high fidelity. It's fully supported, partially supported, or there's no support for the citation in the response. And then the last one is really using HHEM, Vectara's hallucination detection model, to check if the entire response aligns with the retrieved content.

So that's the metrics we have, and there's also a very cool user interface for this. So once you finish with running your evaluation, you can, of course, look at the files yourself, but that tends to be pretty complex.

So you can drag and drop those files onto OpenEvaluation.ai, as shown here, and you get this really, really cool UI that shows you all the queries you ran and all the things that you want to compare between the retrieval scores, the different generation scores, etc.

And that's it. So I encourage you to take a look at this. It's a very powerful package. It can help you optimize and tune your RAG pipeline. Again, it's open-source, so all the source is open. You can take a look and see how it works.

This drives a lot of transparency, so the metrics are very clear in how they work. As I said, it includes connectors to Vectara, LangChain, and LlamaIndex. But if you have your own RAG pipeline or some other RAG pipeline you want to contribute, we're very much in favor of contributing other issues or PRs for other connectors.

And let us know if you have any questions. This is about Open-RAG-Eval, and thank you for listening.