Practical GraphRAG: Making LLMs smarter with Knowledge Graphs — Michael, Jesus, and Stephen, Neo4j

We are talking about GraphRAG today. That's the GraphRAG track, of course, and we want to look at patterns for successful GraphRAG applications, uh, for, um, making LLMs a little bit smarter by putting knowledge graphs into picture. My name is Michael Hunger.

I'm VP of product innovation at Neo4j.

My name's Stephen Chin. I lead the developer relations at Neo4j and, um, actually, we're both co-authoring. This is fun, because we're both already authors, and finally we've been friends for years, and we finally get to co-author a book.

We're co-authoring "GraphRAG: The Definitive Guide for O'Reilly." So, basically, we didn't sleep this past weekend, because we had a book deadline.

Yep.

So, um, I'm going to talk a little bit about, kind of, at a high level, what GraphRAG is, why it's important, what we're seeing in the media, and then Michael's going to drill down into all of the details and patterns and give you a bunch of takeaways and things you can do.

This is probably, if you want to know how to do GraphRAG, Michael's quick deep dive on this is the best introduction you can get. So I'm also excited.

Awesome. That's good going.

Okay. So the case for GraphRAG is where we're going to start, and the challenge with using LLMs and using other patterns for this is basically they don't have the enterprise domain knowledge. They don't verify or explain the answers.

They're subject to hallucinations, um, and they've got the goal and data bias concerns. And you can see that very much like our friendly parrot here, um, they are all the things which parrots behave and act like, except a cute bird.

So we want to do better than this with GraphRAG and figure out how we can use domain-specific knowledge, accurate, contextual, and explainable answers. And really, I think, like, what a lot of companies and what the industry is figuring out is it's really a data problem.

You need good data. You need to have data you can power your system with. Um, one of the patterns you can do this with is RAG. So you can stick your external data into a RAG system. You can get stuff back from a, um, a database for the pattern.

But vector databases in RAG fall short because it's lacking kind of your full data set. It's only pulling back a fraction of the information by vector similarity algorithms. Typically, a lot of the especially modern vector databases, which everyone's using, they're easy to start with, but they're not robust.

They're not mature. They're not something which has scalability and fallback and gives you that what you need to get into build a strong, robust, um, enterprise system. And vector similarity is not the same as relevance. So results you get back from using a basic RAG system, they give you back things which are related to the topic, but it's not complete, and it's typically also not very relevant.

And then it's very hard to explain what's coming out of the system. So we need an answer. Lifeline.

Yeah. GraphRAG.

GraphRAG. And what GraphRAG is, is we're bringing the knowledge and the context and the environments to what LLMs are good at. So you can think of this kind of like the human brain. Our left brain is, um, ourright brain is more creative.

It does more, like, building things. It does more extrapolation of information. Whereas our left brain is the logical part. That's what actually has reasoning, has facts, and can enrich data. And it's built off of knowledge graphs. So a knowledge graph is a collection of nodes, relationships, and properties.

Here's a really simple example of a knowledge graph where you have two people. They live together. You have a car. But when you look into the details, it's actually like a little bit more complex than it seems at first, because they both have a car, but the owner of the car is not the person who drives it.

Yeah.

This is kind of like my family. My wife does all the bills, but then she hands me the keys whenever we get on the freeway. She hates driving. So knowledge graphs also are a great way of getting really rich data.

Um, here's an example of the stack overflow graph built into a knowledge graph where you can see all of the rich metadata and the complexity of the results. And we can use this to evolve RAG into a more complex system, basically GraphRAG, where we get better relevancy.

We're getting more relevant results. We get more context, because now we can actually pull back all of the related information by graph closeness algorithms. We can explain what's going on, because it's no longer just, um, vectors. It's no longer statistical probabilities coming out of a vector database.

We actually have nodes. We have structure. We have semantics we can look at. And we can add in security and role-based access on top of this. So it's context-rich. It's grounded. This gives us a lot of power. And it gives us the ability to start explaining what we're doing, where now we can visualize it, we can analyze it, and we can log all of this.

Now, um, this is one of the initial papers, the GraphRAG paper from Microsoft Research, where they went through this and they did they showed that you could actually get not only better results, but less token costs. It was actually less expensive to do a GraphRAG algorithm.

Um, there have been a lot of papers since then which show all of the different research and interesting work which is going on in the GraphRAG area. And, um, this is just a quick view of the different studies and results which are coming out.

But even from the early Data.org World study, where they showed a three times improvement in GraphRAG capabilities. And the analysts are even showing how GraphRAG is trending up. So this is the, um, Gartner, um, kind of hype cycle from 2024.

And you can see generic AI is kind of, you know, on the downtrend. RAG is getting over the hump. But GraphRAG and a bunch of these things actually are providing and breathing more life into the AI ecosystem. So a lot of great reports from Gartner showing that it's grounded in facts.

It resolves hallucinations. Together, knowledge graphs and AI are solving these problems. And it's getting a lot of adoption by different industry leaders, by big organizations, um, who are taking advantage of this and actually producing production applications and making it work, like LinkedIn customer support, where they actually wrote this great research paper where they showed that using a knowledge graph for customer support scenarios actually gave them better results and allowed them to improve the, um, quality and reduce the response time for getting back to customers.

Um, median per-issue resolution time was reduced by 28.6%. I mentioned the Data.org World study, which basically was a comparison of doing, um, RAG on SQL versus RAG on graph databases. And they showed a three times improvement in accuracy of LLM responses.

And let's chat about patterns, Michael, because I think everyone's here to learn how to do this.

Exactly. So let's look at how to do this, actually. Right? So, and, um, if you look at GraphRAG, um, there are actually two sides to the coin. So one, of course, you don't start in a vacuum. You have to create your knowledge graph.

Right? So, and we see basically multiple steps to get there. Initially, you get unstructured information. You substructure it. You put it into a lexical graph, which represents documents, chunks, and their relationships. And the second step, you can then extract entities using, for instance, LLMs with this graph schema to extract entities and their relationships from that graph.

And in a third phase, you would enrich this graph, for instance, with graph algorithms doing things like, you know, page rank, community summarization, and so on. And then when you have this, uh, built-up knowledge graph, then you do GraphRAG as the search mechanism, um, either with local search or global search and, um, other ways.

Right? So let's first look at the first phase of, like, knowledge graph construction a little bit. Um, so, like, always in data engineering, there is if you want to have higher quality outputs, you have to put in more effort at the beginning.

Right? So it's basically nothing comes for free. There's no free lunch after all. But what you do at the beginning is basically paying off multiple times, because what you get out of your unstructured documents is actually high quality, high structured information, which you then can use to extract contextual information for your queries, which allows the rich retrieval at the end.

Okay. And so after seeing GraphRAG being used, uh, by a number of users, customers, we've seen, uh, we looked at research papers. We saw that a number of patterns emerging, uh, in terms of, like, how we structure our graphs, how we query these graphs, and so on.

And so we started to collect these patterns and put them on GraphRAG.com. Um, and we want to I wanted to show what this looks like. So we have basically, uh, example graphs, uh, in the pattern. The pattern has a name, description, uh, context, and we see also queries that are used for extracting this information.

Right? So, for instance, here's a mix of a lexical graph and a domain graph. And then we can have the query that fetches, uh, this information. Let's look at the three steps in a little bit more detail on the, um, on the graph model side.

So on one side, we have, uh, for lexical graphs, you represent documents and their elements. So that could be something as simple as a chunk. But if you have structured documents, you can also do something like, okay, I have a book which has chapters, which have sections, which have paragraphs, where the paragraph is the semantically cohesive unit that you would use to, for instance, create a vector embedding that you can use later for vector search.

But what's really interesting in the graph is basically you can connect these things all up. Right? So you know exactly who's the predecessor, who's the successor to a chunk, who's the parent of an element. And using something like a vector or text similarity, you can also connect these chunks as well by, um, a k-neighbor or similarity graph, where you basically store similarities, um, between chunks.

And then you put on the relationship between them and a weighted score, basically how similar are the two chunks. And then you can use all these relationships when you extract the context and the retrieval phase to, for instance, find what are related chunks by document, by, uh, temporal sequence, by similarity, and other things.

Right? So that's on the lexical side. Um, this looks like this. So if, for instance, you have an RFP and you want to break it up in a structured way, then you basically create the relationships between these chunks, uh, or these subsections at the text, do the vector embeddings, and then you do it at scale, and then you get a full, uh, lexical, uh, graph out of that.

Next phase is entity extraction, uh, which is also something that has been around for quite some time with NLP. But LLMs actually take this to the next level with their multi-language understanding, with their high flexibility, good language skills for extraction.

So you basically provide a graph schema and an, um, instruction prompt to the LLM plus your pieces of information, pieces of text. Now with large context windows, you can even put in 10,000, 100,000 tokens for extraction. If you have, you can also put in already existing ground truths.

So, for instance, if you have existing structured data where your entities, let's say, products or genes or partners or clients are already existing, then you can also pass this in as part of the prompt so that the LLM doesn't do an extraction, but more a recognition and finding, um, approach where you find your entities and then you extract relationships from them.

And then you can store additional facts and, uh, additional information that you store as part of relationships and entities as well. So basically, in the first part, you have the lexical graph, which is representing document structure. But in the second part, you extract the relevant entities and their relationships.

If you have already an existing knowledge graph, you can also connect this to an existing knowledge graph. So imagine you have an, um, CRM where you already have customer clients, uh, and leads in your knowledge graph. But then you want to enrich this with, for instance, uh, protocols from call transcripts.

And then you basically connect this to your existing structured data as well. So that's also a possibility. And then in the next phase, what you can do is you can run graph algorithms for enrichment, which then, for instance, can do clustering on the entity graph.

And then you generate, uh, something like, uh, communities where an LLM can generate summaries, uh, across them as such. Right? And, uh, for especially the last one, it's interesting because what you identify is actually cross-document, uh, topics. Right?

So because it's basically each document is an basically temporal, uh, vertical representation of information. But what this is, is actually it looks at which topics are reoccurring across many different documents. So you find these kind of topic clusters across, uh, documents as well.

Cool. So if you look at the second phase, the search phase, which is basically the retrieval, uh, um, part of RAG, what we see here is basically that in a graphic retrieval, you don't just do a simple vector lookup to get, uh, returns, uh, results returned.

But what you do, you do an initial index search. It could be vector search, full-text search, hybrid search, spatial search, or other kinds of searches to find the entry points in your graph. And then you basically, uh, can take, as you can see here, um, starting from these entry points, you then follow the relationships up to a certain degree or up to a certain relevancy to fetch in, uh, additional context.

And this context can be coming from the user question. It can be external user context that comes in, for instance, when someone from, let's say, your finance department is looking at your data. You return different information than if someone from the, let's say, engineering department is looking at your data.

Right? So you also take this external context into account, how much and which context you retrieve. And then you return to the LLM to generate the answer, not just basically text fragments like you would do in vector search, but you also return these, um, more complete, uh, subset of the contextual graph, uh, to the LLM as well.

And modern LLMs are actually more trained on, uh, graph processing as well. So they can actually deal with these, uh, additional pattern structures where you have, uh, node relationship, node patterns, uh, that you provide as additional context, uh, to the LLM.

Um, and then, of course, I mentioned that you can enrich it using graph algorithms. So basically, you can do things like, uh, clustering, link prediction, page rank, and other things to enrich your data. Cool. Let's look at some, uh, practical examples.

We don't have too much time left. Uh, so one is, uh, knowledge graph construction from unstructured sources. So there's a number of libraries. Uh, you've already heard some, uh, today from people that do these kind of things. Um, so one thing that we built is a tool that allows you to take PDFs, uh, YouTube, uh, transcripts, uh, local documents, web articles, Wikipedia articles, and it extracts your, uh, data into a graph.

And let me just switch over to the to the, uh, demo here. Uh, so this is the this is the tool. Uh, so I, uh, uploaded, uh, information from different Wikipedia pages, YouTube videos, articles, and so on. And here's, for instance, an Google DeepMind, uh, extraction.

So you can use a lot of different LLMs here. And then you can also, if you want to, in graph enhancement, provide, uh, graph schema as well. So you can, for instance, say, a person, uh, works for, uh, a company and, uh, add these patterns, uh, to your, um, to your schema.

And then the LLM is using this information to drive the extraction, uh, as well. And so if you look at the data that has been extracted from, uh, DeepMind, this one here, we can actually see from the Wikipedia article, um, two aspects.

One is the document with the chunks, which is this, uh, part of the graph. Right? And then the second part is the entities that have been extracted from, from this, uh, article as well. So you see actually the connected knowledge graph of entities, which are companies, locations, people, and technologies.

So it followed our, um, followed our schema to extract this. And then if I want to run GraphRAG, you have here a number of different retrievals. So we have vector retrieval, graph in full text, entity retrievals, and others, uh, that you can select.

Uh, all of this is also an open source project. So you can just go to GitHub and have a look at this. And so I just ran this before because internet is not so reliable here. So what has DeepMind worked on?

And I get a detailed explanation. And then if I want to, I can here look at, uh, details. So it shows me which sources did it use. As for Google DeepMind Wikipedia, another PDF, I see which chunks have been used, which is basically the full text and hybrid search.

But then I also see which entities have been used from the graph. So I can actually really see from an explainability perspective, these are the entities that have been retrieved by the GraphRAG retrieval, passed to the LLM, in addition, uh, to the text that's connected to these entities.

So it gets a richer response, uh, as such. And then you can also do eval on that with, with RAG as well. Um, so while I'm on the screen, uh, let me just show you another thing, uh, that we worked on, which is more like an agentic approach where you basically put these individual retrievals into a configuration where you have basically domain-specific retrievals, uh, that, uh, are, um, running individual suffix queries.

So, for instance, if you look at, uh, let's say this one, it has, uh, the query here and basically a tool with inputs and a description. And then we can have an agentic, um, loop using these tools, basically doing, uh, GraphRAG with each individual tool, taking the responses, and then doing, uh, deeper, uh, tool calls.

Uh, I'll show you a deeper example in a in a minute. So this is basically what I showed you. This is all available as, uh, open source, uh, libraries. You can use it yourself from Python as well. Uh, I already showed, um, Neo Converse, which was also able not to just output text, but also, uh, charts and other visualizations, uh, networks, uh, visualizations as well.

And what's interesting here in the agentic approach, you don't just use vector search to retrieve your data, but you basically break down the user question into individual tasks and extract parameters and run these individual tools, um, which then are either run in sequence or in a loop to, uh, return the data.

And then you get basically these, uh, outputs back. And, uh, basically for each of these things, individual tools are called and used here. And the last thing that I want to show is the, uh, GraphRAG Python package, uh, which is basically also encapsulating, uh, all of this construction and the retrieval into one package.

So you can build your knowledge graph. You can implement retrievals and create the pipelines here. And here's an example of where I pass in, uh, PDFs plus a graph schema. And then basically, uh, it runs, uh, the input into Neo4j.

And then I can, uh, in the Python notebook, visualize, uh, the data later on. And with that, I'll leave you with one second. Uh, the takeaway, which is on graphrag.com, you find all of these resources, a lot of the patterns, and, uh, we'd love to have contributions and love to talk more.

I'm outside at the at the booth if you have more questions. Yeah. So no, that was great. And I think you're getting it all from the expert with all the tooling. Actually, Michael's team builds a lot of the tools like Knowledge Graph Builder.

Um, very excited you all came to the GraphRAG track and hope to chat with you all more. If you have questions for me and Michael, just meet us in the Neo4j booth across the way. Thank you.

Thank you.