# Stop Using RAG as Memory — Daniel Chalef, Zep AI Engineer · 2025-07-22 Daniel Chalef, founder of Zep, explains why using RAG as memory for AI agents fails: vector databases rely on semantic similarity, not business relevance, so irrelevant facts pollute retrieval and cause hallucinations. He advocates for domain-aware memory using temporal knowledge graphs, demonstrated through Zep's open-source Graphiti framework. Chalef shows how developers define custom entity types (e.g., financial goals, debts) with Pydantic or Zod schemas, register them as ontology objects, and build tools that concurrently search Zep filtered by node type. A live demo of a finance coach agent illustrates Zep capturing structured facts like rent payments into a knowledge graph, enabling precise, context-rich memory for accurate agent reasoning. ## Questions this episode answers ### Why does RAG fail as a memory solution for AI agents? Daniel Chalef explains that vector-based RAG systems lack relational and temporal awareness, so they primarily rely on semantic similarity. This means irrelevant facts—like a dog named Melody appearing when asking about music—pollute the retrieved memory, because the system cannot differentiate between semantically similar but functionally unrelated information. The result is inaccurate responses and hallucinations, since semantic similarity does not equal business relevance. [1:18](https://aie.addtry.com/accd39b1-e745-4bf6-97fd-e4aaa98b99f2?t=78000) ### How does Zep implement domain-aware memory for agents? Daniel Chalef demonstrates that developers define explicit business objects—using Pydantic, Zod, or Go structs—as entity types in Zep's knowledge graph. These objects, like financial goals or income sources, include business rules for fields, and Zep's graph registers them as an ontology. Retrieval then uses filtered searches by node type, running concurrent queries to return only domain-relevant facts, avoiding the noise of unstructured semantic recall. [5:07](https://aie.addtry.com/accd39b1-e745-4bf6-97fd-e4aaa98b99f2?t=307000) ### What does a domain-aware memory application built with Zep look like? In a demo, Daniel Chalef shows a finance coach agent that asks about income, student loan debt, and home-buying goals. On the right-hand side, Zep stores explicit business objects like financial goals, debts, and income sources—not arbitrary facts. When the user adds $5,000 monthly rent, Zep parses the message, captures the amount, and updates the user's knowledge graph, reflecting only the custom-defined entities with their fields. [4:23](https://aie.addtry.com/accd39b1-e745-4bf6-97fd-e4aaa98b99f2?t=263000) ## Key moments - **[0:00] Domain Memory** - [0:16] There is no one-size-fits-all memory — model memory after your business domain, says Daniel Chalef. - **[1:18] Irrelevant Facts** - [1:23] ChatGPT's memory struggles with relevance, pulling arbitrary facts that lead to inaccurate responses. - [2:19] Irrelevant facts like 'I wake up at 7 a.m.' or 'my dog's name is Melody' pollute memory for a media assistant. - [3:16] "Semantic similarity is not business relevance" — Daniel Chalef explains why vector search fails for memory. - **[3:45] Custom Entities** - [4:05] Daniel Chalef demos a finance coach that stores explicit business objects like financial goals in Zep's memory. - **[5:07] Code Walkthrough** - [6:04] Zep's knowledge graph instantly captures a $5,000 rent expense into a debt account entity with developer-defined fields. - **[6:42] Wrap-up** ## Speakers - **Daniel Chalev** (guest) ## Topics Agent Engineering ## Mentioned Redis (company), Zep (company), ChatGPT (product), Grafiti (product) ## Transcript ### Domain Memory **Daniel Chalev** [0:16] I'm here today to tell you that there's no one-size-fits-all memory, um, and why you need to model your memory after your business domain. So if you saw me a little bit earlier and I was talking about Grafiti, Zep's open-source temporal graph framework, you might have seen me just speak to how you can build custom entities and edges in the Grafiti graph for your particular business domain, so business objects from your business domain. What I'm going to demo today is actually how Zep implements that, and how easy it is to use from Python, TypeScript, or Go. And what we've done here is we've solved a fundamental problem plaguing memory, and we're enabling developers to build out memory that is far more cogent and capable for many different use cases. ### Irrelevant Facts **Daniel Chalev** [1:18] So I'm going to just show you a quick example of where things go really wrong. So many of you might have used ChatGPT before. It generates facts about you in memory, and you might have noticed that it really struggles with relevance. Sometimes it just pulls out all sorts of arbitrary facts about you. And unfortunately, when you store arbitrary facts and retrieve them as memory, you get inaccurate responses or hallucinations. And the same problem happens when you're building your own agents. So here we go. We have an example media assistant, and it should remember things about jazz music, NPR podcasts, The Daily, etc., all the things that I like to listen to. But unfortunately, because I'm in conversation with the agent or it's picking up my voice when I'm, you know, it's a voice agent, um, it's learning all sorts of irrelevant things. Like, I wake up at 7:00 a.m., my dog's name is Melody, etc. And the point here is that irrelevant facts pollute memory. They're not specific to the media player business domain. And so the technical reality here is as well that many frameworks take this really simplistic approach, approach to generating facts. If you're using a framework that has memory capabilities, agent framework, it's generating facts and throwing it into a vector database. And unfortunately, the facts dumped into the vector database or Redis mean that when you're recalling that memory, it's difficult to differentiate what should be returned. We're going to return what is semantically similar. And here we have, um, a bunch of facts that are semantically similar to my request for my favorite tunes. Um, we have some good things, and unfortunately, Melody is there as well, because Melody is a dog named Melody, and that might be something to do with tunes. Um, and so a bunch of irrelevant stuff. So basically, semantic similarity is not business relevance. And this is not unexpected. I was speaking a little bit earlier about how vectors are just basically projections into an embedding space. There's no causal or relational, uh, relations between them. ### Custom Entities **Daniel Chalev** [3:45] And so we need a solution. We need domain-aware memory, not better semantic search. So with that, I am going to unfortunately be showing you a video because the Wi-Fi has been absolutely terrible. Um, and let me bring up the video. Okay. So I built a little application here, and it is a finance coach. And I've told it I want to buy a house. And it's asking me, well, how much do I earn a year? It's asking me about what student loan debt I might have. And we'll see that on theright-hand side, what is stored in Zep's memory are some very explicit business objects. We have financial goals, debts, income sources, etc. These are defined by the developer, and they're defined in a way which is really simple to understand. We can use Pydantic or Zod or Go structs, and we can apply business rules. So let's go take a look at some of the code here. We have a TypeScript financial goal schema using Zep's underlying SDK. We can define these entity types. We can give a description to the entity type. Uh, we can even define fields, the business rules for those fields, so the values that they take on. ### Code Walkthrough **Daniel Chalev** [5:27] And then we can build tools for our agent to retrieve a financial snapshot, which runs multiple Zep searches at the same time, concurrently, and filters by specific node types. And when we start our Zep application, what we're going to do is we're going to register these particular goals, uh, sorry, objects with, uh, Zep so it knows to build this ontology in the graph. So let's do a quick little addition here. I'm going to say that I have $5,000 a month rent. I think it's rent. And in a few seconds, we see that Zep's already parsed that new message and has captured that $5,000. And we can go look at the chart, the graph. This is the Zep front end. And we can see the knowledge graph for this user has got a debt account entity. It's got fields on it, um, that we've defined as the developer. And so again, we can really get really tight about what we retrieve from Zep by filtering. Okay. So we're at time. So just very quickly, we wrote a paper about how this, all of this works. You can get to it, uh, by that link below. And I appreciate your time today. You can look me up afterwards. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.