HybridRAG: A Fusion of Graph and Vector Retrieval - Mitesh Patel, NVIDIA

To quickly introduce myself, my name is Mitesh. I lead the Developer Advocate team at NVIDIA, and the goal of my team is to create technical workflows—notebooks—for different applications, and then we release that code base on GitHub so developers in general, which is me and you, all of us together, we can harness that knowledge and take it further for the application or use case that you're working on.

So that is what my team does, including myself. In today's talk, I'm going to talk about this project that we did with one of our partners and some of my colleagues at NVIDIA and our partner about how can we create a GraphRAG system, what are the advantages of it, and if we add the hybrid nature to it, how it is helpful.

So that's what my talk is going to be on. I will not give—I will not be able to give you a 10-feet view where I can dive with you in the code base, but there is a GitHub link at the end of this talk which you can scan, and all these notebooks, whatever I'm going to talk about, is available for you to take home.

But I'll give you a 10,000-foot view of if you are trying to build your own GraphRAG system, how can you build it. So a quick refresher: what is knowledge graph, and why are they important? So it is a network that represents relationships between different entities, and those entities can be anything.

It can be people, places, concepts, events. A simple example would be me being here. What is my relationship to AI World Fair conference, AI Engineers World Fair conference? And my relationship is I'm a speaker at this conference. What is my relationship to anyone who is attending here?

Well, our relationship is you attended my session. So this edge of relationship between the two entities becomes very important to which only a graph-based network can exploit or knowledge graphs can exploit. And that is the reason why there's a lot of active research happening in this domain of how you can harness GraphRAG, how you can harness knowledge graph, and put it into a RAG-based system.

So the goal is three things: how can you create a triplet which defines the relationship between these entities that GraphRAG, our graph-based system, or knowledge graph is really good at exploiting.

And that's what is unique about this knowledge graph. So if you think about

why can they work better than a semantic RAG system, well, it captures the information between entities in much more detail. So those connections can provide a very comprehensive view of the knowledge that you are creating in your RAG system, and that will become very important to exploit when you are retrieving some of that information and converting that into a response for the user who is asking that question.

And it has the ability to organize your data from multiple sources. I mean, that's a given no matter what kind of RAG system you're building.

So how do we create a GraphRAG or a hybrid system? So this is the high-level diagram of what it entails. So I broke it down into four components. The very first thing is your data. You need to process your data.

The better you process your data, the better is a knowledge graph. The better is a knowledge graph, the better is the retrieval. So four components: data, data processing, your graph creation, or your semantic embedding vector database creation. Those are the three steps.

And then the last step is, of course, inferencing when you're asking questions to your RAG pipeline.

And at a higher level, this can be broken down into two big pieces: offline, online. So all your data processing work, which is a one-time process, is offline. And once you have created your knowledge graph, which is your triplet, entity relationship, entity two, or your semantic vector database, once you have it, then it's all about querying it and converting that information into a response that is readable to the user.

It cannot be something that, here are the three relationships, and then we as the user have to go figure out what does this exactly mean.

So the top part of this flow diagram is where you build your semantic vector database, which is you pick your documents and then you convert them into vector embeddings and you store them into a vector database. So that piece is how you create your semantic vector database.

And then the piece below is how you create your knowledge graph. And it is much more—there are much more steps that you have to follow, care that you have to take when you're creating your knowledge graph.

So diving into the first step, creating your knowledge graph. How can you create those triplets out of documents that are not that structured? So creating triplets, which exposes the information between two entities, and picking up those entities so that that information becomes helpful is very important.

Here's a simple example. This document is of ExxonMobil's results, I think, their quarterly results. And we tried to pick up the relationship or create the knowledge graph using an LLM. And if you see at the first line, it's ExxonMobil, which is a company.

That's the entity. CUT is the feature of that entity, spending oil and gas exploration and activity. My apologies. CUT is the relationship between ExxonMobil and spending on oil and gas exploration. And activity is the

name of the entity, spending on oil and gas exploration. So this is how the relationship needs to be exploited. Now, the question that comes to our mind is, that sounds very difficult to do. And exactly, it is difficult to do.

And that is the reason why we need to harness or we need to use LLMs to figure out a way to extract this information and structure it for us so that we can save it in a triplet format.

And how can we do that? Prompt engineering. But we need to be much more defined about it. So based on the use case that you are trying to work on, you can define your oncology. And once you have defined your oncology, you can put it in your prompt and then ask the LLM to go extract this information that is oncology-specific from the documents and then structure it in that way so that that can be stored in the form of a triplet.

This step is very important. You might be spending a lot of time here to make sure your prompt is doing theright thing and it is creating theright oncology for you. If your oncology is notright, if your triplets are notright, if they are noisy, your retrieval will be noisy.

So this is where you will be going back and forth figuring out how to get a better oncology. So this is where you will spend—my take is this is where you will spend 80% of your time to make sure you get the oncologyright.

And you'll be going back and forth in an iterative manner to see how you can make it better over time.

And then the next vector database for a HybridRAG system is to create the semantic vector database. And that is very reasonably straightforward or it is well-studied. So you pick your document. This is the first page of attention is all you need, research paper.

And you break it into chunk sizes. And you have another factor called overlap. And chunk sizes are important because what semantic vector database does is it will pick up that chunk and convert that into use the embedding model and convert them into an embedding vector and store it into the vector database.

And if you don't have an overlap, then the context between the previous and the next chunk will be lost if there is any relationship. So you try to be smart on how much overlap do I need between my previous chunk and the next chunk, and what is the size of the chunk that I should use when I'm chunking my documents into different paragraphs.

That is where the advantage of GraphRAG comes into play. Because if you think about it, the important information, which is the relationship between different entities, are not exploited by your semantic vector database, but they are exploited really well when you're trying to use a knowledge graph or create a knowledge graph-based system.

So once you have created this knowledge graph, what is the next step? Now comes the retrieval piece, which is you ask a question. What is ExxonMobil's CUT this quarter that it is looking like? And knowledge graph will help you figure out how to retrieve those nodes or those entities and the relationship between them.

But if you do a very flat retrieval, which is a single hop, you are missing the most important piece that graph allows you, which is exploitation through multiple nodes that you can think about. And that becomes very, very, very important.

I cannot stress how important that becomes. So think of different strategies. Again, you will spend a lot of time to optimize this, whether you should look at single hop, double hop, how much deep you want to go so that nodes, the relationship between your first node to the second node, your second node to the third node, is exploited pretty well.

And the more deeper you go, the better context you will get. But there's a disadvantage of that. The more deeper you go, the more time you're going to spend on retrieving that information. So then latency becomes a factor as well, especially when you're working in a production environment.

So there is a sweet spot that you'll have to hit when you're trying to go how deep you want to go, how many hops you want to go into your graph versus what is the latency that you can survive.

So that becomes very, very important. And some of those searches can be accelerated. So we created a library called QGraph, which is available or integrated in a lot of libraries out there like Networx and whatnot. But that acceleration becomes important so that it gives you the flexibility to get deeper into your graph, go through multiple hops, but at the same time, you can reduce the latency.

So your performance of your graph improves a lot.

So this is where the retrieval piece comes into play, where you can have different strategies defined so that when you're querying your data and getting the responses, you can have better responses.

And the other important piece, I personally worked on this piece, so I can talk at length on this, but I'm going to give you a very high level, is evaluating the performance. And there are multiple factors that you can evaluate on: faithfulness, answer relevancy, precision, recall.

If you try to use an LLM model, helpfulness, correctiveness, coherence, complexity, verbosity, all these factors become very important. So there is a library, a paper stored library, called RAGAS. It is meant to evaluate your RAG workflow end to end.

Anyone who've used RAGAS for evaluating your GraphRAG? Allright, a few of them. Thank you. But it is an amazing library that you can use to evaluate your RAG pipeline end to end because it evaluates the response, it evaluates the retrieval, and it evaluates what the query is.

So it will evaluate your pipeline end to end, which becomes very handy when you're trying to test whether my retrieval is doing theright thing or whether the questions that I'm asking, is the LLM interpreting it in theright way or not.

So you can break down your responses in the RAGAS pipeline, will evaluate all those pieces and see what your eventual score is. So it is a Pip install library. The other is LLM. And RAGAS under the hood uses an LLM.

No surprises there. By default, it is integrated with GPT. But it provides you the flexibility that if you have your own model, you can bring it in as well. And you can wire it up with your API. And you can use that LLM to figure out on these four evaluation parameters that RAGAS offers.

So it's quite complex. I wouldn't say it's comprehensive, but it's really good in terms of giving you that flexibility. The other path is using a model that is meant to evaluate specifically the response coming out of LLM. And that is where this model, LLaMA-Nimotron, 340 million reward model that we released, I think, a few years ago.

At that time, it was a really good response model. It's a 340 million parameter model, so reasonably big. But it evaluates it's a reward model. So it will go and evaluate the response of another LLM and judge it in terms of how the responses are looking like on these five parameters.

But it is meant to go and judge other LLMs. That is how it was trained.

So moving further, I would like to use this analogy that

to create a GraphRAG system, it will take you, which is 80% of the job, it will take you 20% of your time. But then to make it better, which is the last 20%, sorry, which is the 80-20 rule, the last 20% will take 80% of your time.

Because now you are in the process of optimizing it further to make sure it works for the use case good enough

for the application that you're working on. And there are some strategies there, which I would like to walk you through. So one, as I said before, which I couldn't stress enough, the way you are creating your knowledge graph out of your unstructured data becomes very important.

The better your knowledge graph, the better results you're going to get. And something that we did as experimentation through this use case that we were exploring with one of our partners was, can we fine-tune an LLM model to get the quality of the triplets that we are creating better?

And does that improve results? Can we do a better job at data processing, like removing regex, apostrophes, brackets, words that characters that don't matter? If we remove them, does it give you better results? So these are small things that you can think about.

But it improves the performance of your overall system. So that is where I'm talking about 80% of your time, small nitty-gritty of the things that you are, the knobs that you are fine-tuning slowly and steadily to make sure your performance gets better and better.

And I would like to share a few strategies that we did, which led us to

get better results. So the very first thing is regex, or just cleaning out your data. We removed apostrophe as other characters that are not that important if you think about triplet generation. That led us to better results. We then implemented another strategy of reducing the not missing out of longer output, making it smaller.

That got us better results. And we also fine-tuned the LLaMA 3.3 model or 3.2 model. And that got us better results. So if you look at the last three columns, you'll see that by using LLaMA 3.3 as is, we got 71% accuracy.

So this was tested on 100 triplets to see how it is performing. And as it sorry, 100 documents. So as it got better, and as we introduced LoRA, we fine-tuned the LLaMA 3.1 model, our accuracy or performance went up from 71 to 87%.

And then we did those small tweaks. It improved the performance better. Again, remember, this is on 100 documents. So the accuracy is looking high. But if your document pool increases, that will come down a bit. But in comparison to where we were before, we saw improvement.

And that is where the small tweaks come into play, which will be very, very, very helpful to you when you're putting a system, a GraphRAG or a RAG system into production.

The other is from a latency standpoint. So if your graph gets bigger and bigger, now you're talking about a network which goes into millions or billions of parameters or millions and billions of nodes. Now, how do you do search in those millions and billions

in the graph that has got millions or billions of nodes? And that is where acceleration comes into play. So with QGraph, which is now available through Networx, so Networx is also a Pip install library. Anyone who uses Networx here?

Allright, a few. OK. So Networx is also a Pip install library. Under the hood, it uses acceleration. And if you see a few of the algorithms, we did a performance test on that. And you can see the amount of latency in terms of overall execution reducing drastically.

So that is where you can, again, small tweaks which will lead you to better results. So these are two things that we experimented, which led us to better results in terms of accuracy as well as reducing the overall latency.

And these are small tweaks. And it leads us to better results.

So then the question, obviously, is, should I use Graph, or should I use semantic-based RAG system, or should I use hybrid? And I'm going to give you a diplomatic answer. It depends. But there are a few things I would like you guys to take home, which will help you to come up to a decision so that you can make an educated guess that for this use case that I'm working on, a RAG system would solve the problem.

I don't need a Graph. And vice versa, or I need a hybrid approach. So it depends on two factors. One is your data. Traditionally, if you look at retail data, if you look at FSI data, if you look at employee database of companies, those have a really good structure defined.

So those kind of data set becomes really good use cases for a Graph-based system. And the other thing you think about is, even if you have unstructured data, can you create a good Graph or knowledge Graph out of it?

If the answer is yes, then it's worthwhile experimenting

with to go the Graph path. And it will depend on the application and use case. So if your use case requires to understand the complex relationship and then extract that information for the response for the questions that you are asking, only then it makes sense to use GraphRAG.

Because remember, these are compute-heavy systems. So you need to make sure that these things are taken care of. I am running out of time, I think. But as I said before, all these things that I talked about, I gave you a 10,000 feet view.

But if you want to get a 100 feet view where you are coding into things, all these things are available on GitHub, even the fine-tuning of the LLaMA 1.1 LoRA model. And we had a workshop, a two-hour workshop.

So I gave you a 20-minute talk, but this whole workshop is covered in two hours as well. And lastly, join our developer programs. We do release all these things on a regular basis. If you join the mailing list, you get this information based on your interest.

And as my colleague mentioned, I will be across the hall at Neo4j booth to answer questions, if any. I would love to interact with you and see if you have any questions, and I can answer those questions. Thank you for your time.