Stateful Agents — Full Workshop with Charles Packer of Letta and MemGPT

So today I'm going to be going over a workshop, so there is like an interactive component. I think you can probably get a lot out of this talk just by watching, but if you do want to participate, you're going to probably want to install Docker on your laptop if you don't already have it, and then you're going to want to pull this Docker image.

If you do this, it should be very easy to follow along. There's also a notebook in the channel. It's called, I think, like Workshops Letta, and you can just hop over to the— I think I also made like a tiny URL for it.

But if you hop over to this link, this is like the notebook I'll be running over for maybe like the first 30 minutes. And this notebook is kind of like a client, and the client needs to run against a server, and the Docker image is what would be the server in your case.

So again, I'll leave this up for a few seconds. This is basically where you're going to go get the notebook.

And then— and I believe this is also in the Slack channel. And then, yeah, the main thing you're going to want to do if you want to follow along and you have Docker installed is like do Docker pull.

I guess like a quick show of hands, how many people here have Docker installed on their laptops? Okay, sweet. Yeah, most of us. Okay, great.

Okay, does anyone need a few minutes or anything? I guess the people who do want to kind of follow along, just make sure to check out the workshops channel. And of course, this is recorded, and then all the materials will be online, so it'll be very easy to kind of follow along later as well.

Yeah, so I believe the title of this workshop was like "Agent Memory: The LMOS." I think maybe a better title for the workshop is "Stateful Agents." So I think you probably heard a lot about agents over the course of this conference.

I also would like to poll the room, like how many of you, if you think you were asked today to give like a concrete definition of an agent, would be like pretty comfortable in what your definition is. Okay, yeah, so despite hearing, you know, a lot about agents over multiple days, it's still pretty hard to define.

I think this is kind of a problem that's become much worse over the past year, and I think that's why it's kind of useful to maybe like sharpen our definition of agents. So I like this term "stateful agent."

I think "stateful agent" actually is kind of what agent meant before, you know, the LM era. And I think these days, I think a very common definition for agents is it's like an LM that's taking actions in a loop,right?

I think that kind of— that kind of works, but it also misses a really big part of that process, which is that when you run the loop, it's a closed loop, and the agent gets updated. And it turns out that, you know, in this new wave of AI, the fundamental unit of compute we're using for AI is stateless.

It's not, you know, a recurrent neural network. It's not an SSM. It's a transformer. And a transformer inherently is like a stateless machine. So that means that when you close the loop, you have to have some sort of mechanism for updating the state.

And, you know, traditionally, like when agent was defined back in like the RL days, or like back, you know, before pre-LMs, I don't think this really was that big of a distinction to make, like stateful versus not stateful agents.

But I think because LMs are stateless, and that's like what everyone is using for AI now, it's a pretty important distinction. So hopefully by the end of this talk, you kind of understand what stateful agents mean, and you probably hopefully agree with me that statefulness, or memory, is actually probably the most important thing to solve if we actually want to get, you know, if you want 2025 to be the year of agents, or if we want agents to actually deliver on any of the hype.

Yeah, so when I talk about statefulness, I think statefulness is pretty synonymous with memory. I think that's because LMs effectively have no memory. They just have the memory that's in their weights, and then they have what's in the context window.

So like memory and state, context, these are all kind of synonyms with LM, or like LM-driven AI. And humans are, of course, stateful. So humans, they form new memories, they learn over time, and LMs don't. So any of the learning you're going to do is going to have to be done by you, the user of the LM, or by the framework.

But it has to be done by somebody. And traditionally, you know, in the past few years, I think by default this just means appending to a list,right? And I think for workflows and like stuff we were playing around with from 2022 to 2024, when not a lot of people necessarily cared about like making money or doing useful things with agents, that was okay.

But I think it becomes a very big problem when the only mechanism you have to handle state when you're actually trying to use agents to do useful things is kind of appending to some sort of list, and a list that's like held in Python process memory, or, you know, in like a Node.js process.

So the natural question is, if we have LMs and LMs are stateless, and basically the main thing we want to get out of state is we want to have human-like intelligence, that means learning. So how do we actually do that?

Is how many people here have like heard of MemGPT? Okay, yeah, so before I was doing a startup, Letta, I was a PhD student at Berkeley, and the ChatGPT stuff happened in the middle of my PhD, which is kind of like interesting because it meant that like most people doing research kind of pivoted their research entirely.

So in the few years of my PhD post-ChatGPT, I was really focused on memory and agents. And I wrote this paper called MemGPT with a bunch of my colleagues and some of my coworkers now at Letta. And this is really talking about like a memory management system for LMs.

So if LMs can't, if they need some sort of memory management, and the de facto way we do this is like a human appends to a list, well, if you believe LMs are going to get better and better and better, shouldn't this be done by another LM?

Like, shouldn't the AI do the management for the— shouldn't AI do the memory management for the AI? So when we called like this an LMOS, we were really referring to like a memory management system. And this is effectively like a very high-level graph, or a high-level figure of what I mean by like statelessness versus statefulness.

On the left is basically what most people do. You know, you kind of have a context window. The context window is pretty loosely defined. It doesn't necessarily— it's not broken up into very distinct pieces. It's not tied to any state in a database per se.

This is just something to tell them process memory, and you're like appending to it over time. This is what happens if you use LangChain, like in the LangChain buffer memory, or like CrewAI, so we'll get buried really deep in the codebase.

And this is actually, you know, often why you need tracing software, or like observability. Observability kind of exists as a category because we have this black box of tokens that just gets shoved into the LM. And statefulness, or like stateful agents, and the key thing we're trying to do is understand that, you know, what's in the context window, if we have a machine assembling it, there's some— that means there's some sort of like context compilation problem, where there's an optimal way to arrange the context window of an LM.

And that context comes from state. The state can be kind of very large, much more than can everything in the context window. And this is basically what you do if you're a power user of ChatGPT or a power user of Claude.

You're doing this yourself,right? You're kind of like compiling the context every time you start a new chat. Because I think a lot of us, you know, probably have experienced like having a chat that goes like way too long and starts to derail, and then you have to re-describe everything you were doing back to ChatGPT.

It's like a pretty painful experience. So can we basically automate this with a machine?

So yeah, kind of taking a step back, like why do we even want stateful agents? Like why are agents we have today not enough? Like why is the current like paradigm of LM-driven agents like not good enough to reach, let's say, like AGI, or like some like the AI we see in science fiction?

I think the main problem is that agents we have today, they just can't learn from experience, or the way they learn from experience is extremely limited. And I think if you're just running like workflows, you might not notice this.

So it might not really be a big problem. But I think if you're trying to build like assistants, companions, co-pilots, this becomes like pretty evident. So imagine if you have some sort of chatbot. So this is a direct screenshot from like the MemGPT paper.

But you have an AI and a user, and the AI, you know, it can see that it's February 14, so it wants to like ask the user what they're going to be doing on Valentine's Day because they have stored in their memory some recollection that the user's boyfriend is James.

So the AI asks like, "Oh, hey, how's James doing? Do you have any special plans today?" And, you know, the user says, "Actually, you know, James and I broke up." So it's obviously a very, very bad mistake by the AI, but maybe it's like kind of unavoidable.

But you see the AI here, if it has some notion of like a permanent read-write store, it should do something like transact that, you know, James is no longer the boyfriend. James is the ex-boyfriend. And if you don't do this kind of thing, and you run your agents for long enough, and you just do like recursive summarization, or if you just have like a really long context model, you're inevitably going to make some mistake where you tell, you know, you tell the user something about your boyfriend James, and that's like a devastating error if you're trying to build like a consumer app,right?

And it's also something that just a human would never do. I think like humans, something like this would get like written to your quote-unquote "core memory" very aggressively. I think the other reason you want statefulness and learning, I think many of us here maybe like kind of work in companies, and we're trying to like use agents to make money, or like drive value, you know, for shareholders or whatever.

But the big difference between consumer and enterprise is obviously data. Like enterprises have much more data than could ever fit into like Gemini 10 million tokens,right? Often, like per user, you have more than 10 million tokens. So how do you actually like learn from that data?

So you, we can kind of think of there being like another training phase after the post-training, like during post-training,right? You train the model, and then now the model is deployed into your enterprise, and the model should kind of learn about your company,right?

And that's kind of like training again. But it's not training into the weights. It's training like into the in-context memory. So stateful agents naturally kind of like encompasses this concept. And this is like pretty useful if you're trying to build this kind of stuff inside of companies.

And yeah, again, like really the reason I was kind of inspired to work on like the MemGPT stuff, and also the reason I'm really excited to be doing the work I'm doing at Letta, is because I'm very interested in just AI and making AI that is very human-like.

And clearly, there are some very serious deficiencies with the current iteration of LM agents. They're not very human-like, especially with respect to how they deal with memory.

Yeah, so I covered this already. I think this is like a tweet I saw a few weeks ago or something. I thought it was pretty funny. But I think, you know, if you all use ChatGPT in this room here, it's probably like a pretty universal experience.

You kind of have a conversation that goes on too long, and then, you know, you feel very devastated because you're now going to have to mentally context compile again to like re-describe to ChatGPT what's going on, or like Claude, or whatever you're using.

So the promise of stateful agents, there's actually a lot of promise just in product,right? I think if you actually have true statefulness, then that means that, you know, this sort of experience shouldn't happen. There shouldn't be any derailment.

So you shouldn't have, you know, like a Claude instance going haywire. And actually, you should have the opposite happening. Like the experience should get better and better and better over time as the AI kind of learns more and more about you.

And this is, you know, kind of like maybe the promise of ChatGPT memory. But, you know, I don't think any of that stuff is really working that well. And of course, I think when you don't, when you no longer just like shove stuff into a context window, when instead you're kind of really like creating human-like memory constructs, I think the behavior of the agent just becomes more human-like.

You have the same kind of like fuzzy memory that a human has. You have the same kind of forgetfulness, but you also have the same kind of recall. So in today's workshop, I think it's kind of going to be split into two parts.

I think the main thing we're doing at the front is a notebook. And of course, like you don't, you can just watch, but if you want to, you can follow along in the actual Jupyter Notebook file. And the goal of that notebook will just be to like lay out the basic ideas behind this sort of like context management system in an LM.

And at a very, very high level, you know, if I had to describe it in one sentence, it's just that you make an LM, either the main LM or like the subconscious memory system LM, just aware of the context problem.

You kind of can just describe to it in English, "Hey, you are an LM, and you have a context window of like 128k tokens, and I will let you know when you reach 100k tokens, and you're going to have to like manage memory with these tools."

So it's all centered around tool calling. And I think centering around tool calling is actually very effective because LMs are getting better and better and better at tool calling. And the second part of this workshop is going to be kind of like building these stateful agents in the Letta framework, and then also like the front-end AD.

So I think if you're very interested in kind of like no-code stuff, I think this is pretty interesting because I think we're all very, very familiar with the paradigm of the playground,right? It's like ubiquitous. Every single AI company has their own playground.

But I think once statefulness and like stateful agents become the dominant paradigm, which I'm pretty sure they will, then I think you're going to have, there's going to be a new iteration of the playground. And the AD that we built, I think that's our best guess as to what that experience will look like.

But I'm sure like everyone in this room who's worked very, who has like spent a ton of hours with LMs, you probably have a very strong opinion too on what that experience will be like. Like what happens when, you know, ChatGPT is no longer like, it no longer has like a history bar on the side and it's like a unified experience.

Like what is the correct UX for that? What's the correct DX? And then if it's not a consumer app like ChatGPT, if it's a developer tool, like what do you as a developer want to see from your agent?

Like you probably want to see all the way down to the context window. So I said this at the beginning. I don't think anyone necessarily walked in midway. So if you're going to follow along, you're going to want to download Docker on your computer.

I think most of the people in this room have Docker on their computer. You can do this with Pip, but Pip is pretty terrible at package management, so you might like hit a bug. And yeah, I've already covered this, but if you're interested in any of the ideas that I'm going over in the initial notebook, they're basically just strongly distilled versions of the ideas from the MemGPT paper.

Like reduced to their like most simple, yeah, their most simple components. But if you're like interested in this more in like a research capacity, you can also just read this paper.

Okay, and yeah, we're going to be doing this in Python. I guess like how many people in this room have used Python as their main language versus, okay. And I imagine like everyone else is TypeScript or something. Yeah, so basically what we're going to be doing is Python, but Letta, the server is not, oh, maybe Go or Rust.

Okay, yeah. Yeah, so we don't have a Go or Rust SDK, but we have a TypeScript SDK, and then we also have a Rust API. And actually, our Python SDK and TypeScript SDK are just like programmatically generated off the Rust API.

Okay, so I left this on the screen a little bit earlier to kind of like preempt any pauses here. So I might just like continue on. And yeah, these links are all in the channel, like workshops-letta on the shared Slack.

Okay, let me hop over.

Oh, and I think we, because this workshop has a reasonable amount of time allotted to it, I'm very open to like pausing for questions and stuff. So if anyone has any questions, please just raise your hand. And I can pause.

And then if you're just going too slow, we can fix that later. Yeah, okay, questions.

Allright, so yeah, I guess how many people are actually going to attempt to follow along? It'll be useful for reference. Okay, a decent amount of people. So I'll try to, yeah, definitely raise your hand or something if something's going terribly wrong and you're not able to follow.

Let me boost the, okay, let me know when the font is large enough. Is this large enough for the people in the back to read? Okay, great. And

allright,

okay, so if you ran the, if you did the Docker pull, the next thing you're going to want to do to start this entire notebook session is you're going to want to start the Letta server. So if you were using Docker, the way you're going to do this is run this Docker run command.

I'm realizing now that this is not, oh, this is in, you can copy-paste this out of the GitHub, the GitHub link, but I'll also post it in the Slack, yeah.

And then if there's a free endpoint that's live, so you don't actually need to have these keys. So you can just run this, and it will, you'll be able to run the demo fine.

Okay, and then once you run it, you'll basically see the server kick off in the background.

Yeah, so while people, I'll give everyone like a minute to do that. And while everyone is kind of kicking off their servers, I might also just show you kind of a diagram of what's going on here.

Yeah, so basically the way this whole stack works is Letta is like, it's an open source stack. It's basically FastAPI, Postgres, and Python in the middle, like Python logic. So in that Docker container, like it's exposing an API that's actually like very, you know, pretty robustly documented.

You can like get our API reference here. And the API is how you interact with all your agents. So if you're using the Python SDK, it actually just, you know, goes over the Rust API, and you basically build your agent applications on top of this API.

So this actually looks very similar to like the chat completions API. It's just like session-based,right? And so you don't have to provide the entire conversation history every single time you interact with an agent. So we'll see that kind of immediately in the notebook as well.

Okay, so I assume everybody who's following along has gotten the thing to launch. Is there, okay, wait, what does it say?

If it says that there's a directory that doesn't exist, do I need to create that?

Oh, yeah, can you do like a make MKDRR on that directory?

Okay.

Did anyone else get that directory doesn't exist problem?

Can you send the channel again? Because I need to know if I can see it.

Okay, do you, so are you able to see this here?

Oh, no.

Okay, yeah, so it's, let me.

There's a chance.

Yeah, that fixed that.

Okay, great. Allright, any other problems? Does everyone have a server idling now on their laptop? Great. Okay, so now that we started the server, now we can kind of like start running the notebook. So this part is basically, you're going to go to this channel, click on this link, you're going to see a README, and the README has all the commands.

So I'll kind of like go one by one and paste these commands into the workshop channel.

Which workshop channel?

It's workshop-letta, yeah. Okay, so you need to basically clone the repo.

Once you clone the repo, just CD into this specific example.

I'm assuming everyone's using Docker, so I'm just going to kind of ignore that.

Okay, and ideally, everyone has Python installed in their laptop, hopefully.

Yeah, and then once you run that Jupyter notebook command, all you need to do is just double-click on the notebook here if you're not familiar with Jupyter. It should automatically like open a browser tab that should have this, and then you just like double-click, and you'll open the notebook.

Okay, so among the people who are following, who is a little still not to this notebook stage yet? Okay, yeah.

I didn't install Jupyter, so it's taking a while.

Okay, yeah, no problem.

Okay, no other issues? Nothing on fire yet? Okay, cool.

Allright, how's the Jupyter looking?

I installed that.

Okay.

Okay, so everyone has this notebook up who is intending to follow along? Great. Allright, so yeah, if you're not familiar with notebooks, you basically just execute them cell by cell. I mean, number, the first thing you're going to have to do here is like import the client.

So again, this is a little bit different from some other frameworks you're familiar with. Like I assume everyone here has like used LangChain before, probably, or heard of it. Well, like CrewAI, AutoGen, Pedantic AI, things like that. So I think among those frameworks, there's a pretty big distinction with Letta, and that Letta is like a server-client process.

And a big part of that is because the agents are intended to be stateful and like persist indefinitely. So it's very hard to like persist things indefinitely if you're kind of holding everything in application state, as opposed to having like a server that's a centralized source of truth.

So we basically run the server on the left. We've connected a client. You know, you can actually park the server anywhere you want because it's a Docker image. You can like run this in the cloud. It's very easy to like drop on Kubernetes or something.

In this case, it's running on our laptop, so we just need a local host and the default port. So the first thing we're going to do is create an agent. So I want to make sure that everyone can like execute this cell.

So I'm going to execute it one more time. And you saw like my server kind of fired off on my left. Yeah, so I'll kind of walk everyone through the code here. So basically, you know, with chat completions, which I assume everyone's familiar with, the paradigm is you create the agent in memory, and then you kind of like pass the whole agent off to the server and ask like, you know, complete the state, complete like this one more message in my message history.

Instead of Letta, we first create the agent, and then once we create the agent, we have a handle for it. And once we have that handle, we can kind of send it messages, and we never have to send it the full state.

We don't have to track that anymore. We just send individual messages. So

the main components of memory in a Letta agent are just memory blocks. So these are just strings. In this case, you know, we're doing something pretty simple and stupid, like the human's name is Bob the Builder, and then the persona for the agent, we're just using the my name as Sam, this AI.

And this is all arbitrary. The only things that these handles are useful for, actually, is for the agent to edit these own handles or edit the values in these handles. So as we'll be able to see later in the notebook, the agent can actually go in and like rewrite its own memory.

So if the agent, you know, decides that I likeice cream or I like vanillaice cream, it can write to its memory block, you know, I'm no longer just Sam, the AI. I'm like Sam, the AI that likes vanillaice cream,right?

Or if, you know, Bob the Builder says, I broke up with my girlfriend, or I broke up with my boyfriend, James, you know, the AI can actually use this handle to go in and edit this block of the string and say, you know, the human's name is Bob the Builder, and they also broke up with their boyfriend, James.

Yeah, go ahead.

Quick question. So what's the difference between persona and human?

Yeah, so there's really no difference under the hood. These are all just strings that have references. But if we look at the system prompt, I'm going to dump the system prompt a little bit later. We did in the system prompt write some stuff about the human and the persona, that the human block is meant for like the user interacting with the agent, and the persona is meant for the agent to kind of adapt this interaction style,right?

So there's kind of like a base agent, and then the base agent gets further mutated by this persona. Yeah, okay, any other questions about anything here? The, you know, here where you would, if you're using OpenAI or something, you'd do OpenAI slash like GPT-4, but we're just using a free endpoint.

One thing that we'll get into a little bit later is that in Letta, because the entire premise of the like runtime is that it's a context management system, so it controls the context window, it's actually very easy to artificially cap the context window length.

So you can have agents that basically will never send a payload over 10K tokens, let's say. And that's actually very common in enterprise settings where you're running like a workflow for a very long period of time. It's getting longer and longer and longer.

Like something you'll hear from a lot of companies that use Sonnet, for example, is like, how do I prevent my payload from creeping up to 200K,right? Because if you hit 200K, you're waiting like minutes on a response,right? Whereas like even 10K tokens, I think on Sonnet, you'll wait like 10-ish seconds or something.

In many cases, you kind of want to like cap the context window to be really short, like 4K, but then you also want the agent to like not have any perceived loss of memory. You want it to kind of like retain its memory over time.

Okay, so let's actually like, yeah, message the agent. There's a link here that it will, this is what we'll do in the second half of the workshop. We'll look at like the UI builder, but for now, we'll just like stick in Python.

We'll stick to Python. So yeah, if you run this message, we're going to say something like how it's going. You're seeing on the left the server is firing, and we can print the response here. So in Letta, like every single agent actually comes with reasoning by default, and this allows you to like keep reasoning that you built in like R1 and like port it over like a GPT-4.0 mini or something.

And there's just like different port adapters for the reasoning. But yeah, are there any questions about this? We basically, I think this should be something a lot of people here are familiar with,right? This like dictionary that's in chat completion style, we pass over to the server.

And then, yeah, we have a message here that's like, you know, the users reaching out casually, let's match their energy. Hey there, I'm doing great. How about you? You can even do something like, again, because

this is stateful,

this is actually, when I send this, it's going to be a follow-up message. It's not going to be like a fresh chat. So if you say, what do you know about me? See, the agent now says, you know, the user wants to know about what I've retained about them.

All I know is that your name is Bob the Builder. If you want to share more and like, you know, a ridiculous amount of emojis. I think this is because under the hood, this free endpoint is running GPT-4.0 mini, which is like pretty emoji-prone.

Yeah.

I'm writing this against Sonnet, and I'm also getting a chain of thought. Where is that coming from?

Yeah, yeah, so one thing about Letta is we kind of force all the agents to follow a React-style pattern with like reasoning in the loop. So kind of down in the weeds, like the way it works is like Sonnet itself or Claude, when you use like the chat or the consumer app, it packs like ant thinking XML tags in.

So like it already is like kind of a reasoner in some capacity. And because we know that it uses like XML tags like that, we can actually like force those into the content field. So we like actually inject and parse those out.

Yeah.

Okay, yeah, and then just to like further drive on the point that this is stateful, if you say sure, I feel like sure, if it was like an opening message, would be pretty confusing. The agent would say something like, I don't know what's going on.

But in this case, it seems open to sharing more. Okay, bad model behavior, but yeah. But we'll see also later in the ADE, you can just see, well, it'll be like a UI experience. You can kind of visualize this a lot better.

I just wanted to start with Python. Yeah, so basically there's like three different messages, or there's a handful of message types that are going to be a little bit different from OpenAI, but they're still rooted in the same concepts.

Reasoning message, you're probably familiar with this if you've used like the R1 API from DeepSeek, if you use any of the O1, O3 models. So we have the same sort of content field. There's also assistant message. So this is because like content itself is treated as inner monologue or reasoning all the time.

So this is like distinct, and these are like pretty straightforward, like tool call message, a tool call return, the system message, and the user message. Obviously, like a very important part about being an agent framework is often executing the tools for the agent.

You know, like OpenAI, they don't execute tools for you unless it's one of the pre-approved tools like Code Interpreter. Letta, these other frameworks like, you know, LangGraph or LangChain, we execute tools on the server side. So that's why we're able to like provide a tool call return.

And in Letta, like the tools are actually also sandboxed by default.

Okay, any questions about this like initial message we sent?

No.

Okay, great. Yeah, so the next thing we're going to do is basically just like unpack all the state of the agent. So the reason this is like kind of interesting is because this is fundamentally everything that the agent is,right?

The agent is just its system prompt, its tools, and then in Letta specifically, we have a concept of like three tiers or two tiers of memory, the core memory that's in the context window. So this is like very top-level stuff.

You know, if you see your friend on the street, you see their face, you immediately remember stuff about them, like their name, their hobbies, the last time we chatted. But if they're a childhood friend, you don't remember what you did like 10 years ago on a random like, you know, February 22nd, 2001,right?

That's not going to be like default in your brain, but it might be visible to you if you like went on your phone and you went through your iPhotos,right? So it's a pretty similar concept where like you want, because at the end of the day, LLMs, they're kind of like mimicking human reasoning, mimicking human behavior.

It makes a lot of sense to actually kind of like mimic in text the way human memory works. So we have core memory that's like top level, and then we have what we call archival and recall, but these are effectively just data sources that exist outside the context window, and the agent can like, quote unquote, jog its memory if it wants to think about something.

So it can say like, hey, like what did I do, you know, what did I do with Charles February 22nd, 2001, and it can like attempt to search some database. So you've probably heard a lot about like agentic RAG.

This is effectively the same concept.

Can we change the system prompt? Does it read only there, so we cannot change it?

Oh, so the agent can't change it, but you can change anything you want. Yeah, yeah, you can change the system prompt. Yeah, the system prompt actually is like quite old. You wrote it like when this is written, like when the initial MemGPT like research was being done, like in the summer of 2023 or something.

So yeah, you probably do want to change this if you're running something in production. Yeah.

The other thing that's stored in here are tools. So by default, because MemGPT agents, they have like memory management built in, they need tools to manage their memory. So core memory, we have two things like appending to blocks.

So saying like the user is not only Bob the Builder, but they also have a boyfriend called James replacing, you know, the user's name is not Bob the Builder, it's actually Charles. And then like searching memory, you can either do a very specific conversation search or more generic like RAG query, and then you can also like insert into this external database.

So if you look at like what's actually in the memory, if you're kind of into like API-style programming, like you can basically do whatever you want with your memory blocks because it's just, you know, these are just strings that exist in Postgres in some table, and they all have identifiers.

They have block IDs, so you can read and write to them. You can also share these blocks among agents, so you can have a bunch of agents that are all acting as part of your organization. Say it's like a multi-agent system.

And because the blocks just live in a database and they get brought out of the database whenever the agent needs to think, you can have these blocks like linked together. So multiple agents can have the same block. Like they can all share information about the AI engineer conference that's held in one block of memory.

It doesn't have to be duplicated. And then when one writes the block, it like immediately gets broadcasted.

By definition, does all of the archives of memory have to also be in the recall? If the recall contains all the history, surely it also contains anything that could have been archived.

Yeah, yeah, that's a great question. I think the difference between archival and recall really comes down to like if you're using this for like a chat-style application. When we like worked on the original MemGPT project, we were actually like a little bit conflicted as to whether to like merge these into one general concept, because at the end of the day, like with an LLM, there's the stuff that's in the context window and the stuff that's out of the context window.

So why do you have to like start being prescriptive about like the way the stuff out of the context window is stored? We just thought in most use cases, people want to give their agents the ability to specifically look at prior messages.

And then separately, there's just like a general read-write data store. You can just like read arbitrary data, write arbitrary data. It's like an infinite size. So that kind of just looks like a vector database of strings. It could also be like something different.

It could be like Elasticsearch cluster, whatever you want.

But you are intentionally allowing some duplication between them. It's kind of a different way for the LLM to search effectively.

Yeah, it's a different way for the LLM to search. Well, the thing with the, if you set up a recall memory, the recall memory, it's effectively conversation history. So conversation history, you cannot manually write to. It's kind of like write protected, but it gets written every single time by default and event happens.

Whereas the archival memory store is like something that you have to actively say, hey, I have like a huge document, like this big PDF. I don't want to keep it in the context window. I'm going to dump it out of the context window.

So that would be like the archival memory, like free read-write. Whereas the recall thing is purely trying to mimic like a conversation search function inside of like iMessage, inside of like WhatsApp or something like that.

How does other tools from the memory instead of tools get used? Do we just have a flat list of like other tools that agents can use and the memory tools? Does that affect the agent's ability to use the tools?

Yeah, that's a great question. I think by default, if you just set up a basic agent in Letta and then you start like adding more tools to it, like Tablely Search, iCal Calendar Scheduler, or whatever, those will be just like more tools that get added to a list that started with six.

But of course, like if you built a lot of agents with tools before, you know, there's like agents start to get confused if you add too many tools. So there is like a trade-off here. And one thing that we've been working on recently is basically the idea of having like a dedicated agent to just handle memory.

And it's effectively like a shadow of the other agent. And it's like a subconscious that is like kind of, it's like a ghost in the shell. It can't actually participate. All it can do is read and write memory.

But when it reads and writes memory, it's like using a shared memory block. So it's like reading and writing to something that like automatically updates like the active agent. And that's a way to kind of like move all the tools that aren't for like general API actions out of the main agent.

Okay.

Yeah.

Are you familiar with the cognitive architecture?

The Qualo one?

What's that? Yeah, Qualo.

Yeah, yeah.

Okay, perfect. So a few different types of memory that are connected there would be things like caches, like long-term memory. So when you talk about things like recall or archival, like is Letta bringing those databases to like this interaction, or is the developer responsible for bringing those databases to like this what we're seeing?

Yeah, yeah, another great question. So I think with Koala, yeah, another great paper. I think the inspiration from them is a little bit more from like the cog-sci angle. Whereas when we wrote the paper, we were thinking more about like computers and memory hardening in computers.

So for us, like our answer to that is like, oh, you want to have like something that really starts to look like human memory, semantic, episodic, you bring that and you can like add that as a plugin. But all we care about is like the hierarchy of tokens.

There's tokens that are in cache or like in context and tokens that are out of like the cache. So yeah, basically, if you wanted to have like some more psychological leaning concepts like from Koala, you could kind of like write them as tools that either like populate the core memory or like sit outside and have to be drawn in.

Yeah, today I build like very transactional agents. So like in this particular example, let's just imagine it's like a CRM. So we've got probably not going to use Salesforce to track changes like relationship status, but in theory, if we wanted to track something like that, that's like a stateful system that has an object for people, like a contact that I can preserve in that like long-term system.

So I'm trying to understand like if there's a differentiation in use case where it's like maybe we're just talking about like a local machine as opposed to like an enterprise environment with, you know, thousands of employees. Like can you maybe show me that or like tell me like why a tool that's responsible for tracking the state of James's relationship is worse or different than like a memory-based solution?

Yeah, so yeah, correct me if I'm not understanding your question correctly, but one thing you can do is because like all these memory blocks, they're all backed by an API and you can access them directly. So if you go to, it's like a blocks page.

Where is this? Okay, here we go. Yeah, so you can basically like modify blocks by handle. So you can in your app layer effectively do something like every time your agent is run or invoked, you do a hard sync where like you're going to do some sort of Salesforce CRM compile down to a string and that string is going to get clipped at 10K characters and then that's going to get patched into the block.

Yeah, so you can do kind of stuff like that. But I will say like the general way like Letta was designed is very much leaning on the thesis that like you just want to remove as much human design of memory management as possible.

And we're just like banking on the LLMs getting better and better and better. And actually what you will see is like if you try to use R1 with Letta, it's extremely annoying because like firstly, the way the like reasoning APIs are designed is that you're not supposed to pass the reasoning from a subsequent step into like the API again.

Which means that like if an agent thinks for a very long amount of time and it's like, oh, I'm going to do this, then this, then this, then this. And it's like, wow, that was amazing. That's exactly what you should do.

It actually immediately forgets in the next turn,right? It only gets to do one thing and that's to think again. So that's like one issue with like reasoners and React-style agents. But you will notice that actually the tool use is like pretty exceptional.

So I think that kind of like aligns with the thesis that these agents are just like going to get better and better and better at doing, like the agent, there could be a separate agent that does the CRM sync into a block,right?

It doesn't have to be done manually. Yeah.

And then, sorry, final question here is like the property that I think is most compelling for at least what I've seen so far is like the distillation of the agent like into some kind of summary. So do you control like the perspective of like how memory is defined and like how synthesis is actually being like happening when it's creating memories?

Yeah, so the agents.

I see. Yeah, well, if you have agents that are generating and synthesizing the memories, then you can always like tune their system prompts or their personas to like adjust the way the memories are written. But yeah, it's all pretty configurable.

Okay, maybe I'll take like one question in the back and then, yeah.

I have a little more of a naive question. If you think about the way we design our architecture today, like and

microservices offer different persistent stores, data stores, and caching technologies and stuff like that. Based on your discussion with enterprise customers, do you see that we are evolving towards agentic workloads where like microservices take a backseat and agentic workloads primarily become predominant form?

I think they can, the two can exist, they can coexist pretty easily through the use of tools. So I think what I'm seeing in like enterprises is that there's just extremely heavy use of like, I think almost every single agent I see being used in some company, they're all like, they're trying to connect tools to the agent.

So, and it's like pretty heavy tool use. So you can, I think, think about microservices like basically starting to become designed more for like agents maybe. I think that's like one thesis that a lot of people have. There's going to be like a new wave of APIs that are designed all for agents.

Yeah, but I think the two will kind of like exist. But I think there are like microservices that will become like agentified where like the microservice itself has higher, it's not like latency bound. So you can like run a fat agent on the backend and it's kind of like not super sensitive to error.

So it's okay for it to just be smarter at 99% of the time and then like completely break 1% of the time. Yeah. Okay, I'm going to keep going, but we'll definitely pause for more questions later. So, okay, so we were just looking at like memory blocks.

The other thing, I think I might breeze over this a little bit just for sake of time because we'll see it again visually. And I think like visual stuff is always more interesting than just plain notebooks. But there is this problem if you have a system where there's a lot of information outside the context of the LLM where like you can't know what you don't know,right?

So if the agent doesn't see something, how is it ever going to know that it has access to that? So a very simple solution is you can just provide like metadata statistics,right? So you can tell the agent somewhere in this context window that, hey, you have like 13 previous messages and if you want to find them, you can use this tool.

And you can also tell the agent that, you know, you have like X total memories that are in your archival memory. So this is like, if you have like a vector database, you can just explicitly say, these are like the statistics.

Okay, and then if we look at like the messages that are actually in the message buffer, it's pretty straightforward stuff. I think the really big difference between like Letta and another framework is Letta is very aggressive managing of this buffer.

So you can basically set it to like, it's a much more intelligent version of like a recursive summarization mechanism,right? And lastly, this is like the archival memory we're talking about, you know, there's nothing in there, so it's just going to be empty.

Yeah, so I think just to drive home like what core memory really is and like how these memory blocks influence agent behavior. Because I think if you're developing agents, the main thing you're going to be doing is like tuning in-context memory.

That's like the main way you change the behavior of the agent. You know, we could tell the agent something like, my name is actually Bob. And then the last thing we told the agent was that, you know,

my name is actually Bob. I think our name is Bob. It's strange. It's like my name is actually Charles.

Okay, we can see actually you're seeing like a memory edit happen on the server. This is a pretty big dump. If you look at what the agent said, so you see the agent is like doing React-style reasoning. It's saying the user has corrected their name.

This is important. I'm going to call this tool, core memory replace. And Bob, the builder, is now Charles. And then it actually chains a tool call together. So there's like tool call chaining built into the Letta by default.

And then it says the user's name is updated in memory. And then finally it says like an external message, hey, got it, Charles. Thanks for letting me know. So this is two LLM calls. There's like one LLM call here and then one LLM call here.

And in Letta, agents will kind of like chain indefinitely. You can obviously set like limits in the API. And the way they chain is basically through heartbeat requests. So basically who here like is familiar with React agents? When I say React, is everyone kind of on board?

So it's like one of the OG design paradigms for agents. Same author as Koala, actually. But in React, basically the agent follows like a reasoning action like observation loop. But usually the agent actually has to say, I'm done.

And the agent will loop indefinitely until it says, I'm done. In Letta, we actually do the inverse where the agent has to say, I want to keep going. I think this is actually generally more practical because it's like much less likely that the agent will derail if it has to explicitly say, I want to keep going.

So we call those heartbeat requests. And yeah, now if you like, you know, use the API to retrieve the memory, it says the human's name is Charles. So very briefly, like the archival memory, I think there are some questions about that.

What is that even? In the default implementation, it's just a vector database. So I said here, you know, remember that I love cats in your archival memory.

Yeah, you can see it took a little bit of time because I think it actually embedded,right? It says I inserted into archival memory. Yeah, the user got cat. And then again, this is a chain,right? So it like did one tool call and then it comes back and says, got it, cats are adorable.

And you can of course like manually edit archival memory. So you can manually run an insert that's like insert Bob. I guess it's kind of confusing now because it's Charles, not Bob, but

yeah, we'll just slip with it. And if we ask the agent, you know, like what animals do I like? You generally, it depends on the model. If the model is like stupid, you might have to like tell it specifically to fetch external data.

But if the model is like smart enough, like Sonnet, you will not do this. So you can see that the agent searched archival memory. This is stuff that's not in the context window. It's bringing it into context and again chaining and saying, you know, hey, you like cats and Boston Terriers.

What makes them our favorites? Okay, so any questions here on anything so far? This is just, you know, pretty basic memory editing via tools. So you're giving the agent the ability to edit its own memory inside the context window.

Yeah.

That's not what I'm playing with there. Sorry, I sort of had to commit a bunch of my preferences to memory. Yeah. And then reset my conversation history and expected it to just pick up these previously remembered references, but it didn't.

I had to go like, hey, don't you remember? And then it went and picked it up. What's the effective behavior of?

Interesting. So when you say reset message history, is that in the AD, the web UI, or?

Yeah, AD.

Yeah, yeah. Interesting. And core memory had been edited, or was it going into archival?

It's all archival.

Oh, yeah. So that's another, that's where you'll notice like there's always this like engineering design problem where if it goes into archival, then it will have to fetch and pull from archival first to see it. So it's like, it's a, you know, it doesn't know what it doesn't know problem.

This is actually fixable via like tool rules. So you can enforce behavior. You could make a tool rule that says every time you enter the entry point of the LLM, the step of the agent, the agent has to call archival memory search.

So this is kind of like building graphs onto Letta. So I guess how many people here are familiar with LangGraph? Okay, a lot of people. Yeah, so I think LangGraph is like pretty intuitive,right? You can, because we all know, you know, like flows, graphs.

It's very easy for us to think about like decision trees. I think with Letta, it's kind of the inverse. Everything starts off as a fully connected graph. The agent could do anything. But then you can kind of like start to enforce restrictions by peeling away edges.

So you can basically say, hey, you know, when you start, you actually can't do everything. You can only do this. But then once you do this, you can do everything. And obviously, you know, I'm biased, but in my personal view, I think that's a little bit more forward-looking just because if you assume that LLMs are going to get better and better and better at tool calling, they're going to get better and better and better at decision making, then you should want to give them freedom,right?

Because that's kind of what separates like modern AI from classical AI. We've had decision trees for a while,right?

Yeah.

How does the model decide whether it wants to store in core memory versus archival memory? Because I've talked a couple of things that look pretty peripheral, but they've all gotten stored into core memory.

Yeah, yeah. So I think there is one thing that's built in by default, which is core memory has limits. And when a limit is hit, that actually, the way that works is, let's say the agent is just, the agent's being really lazy and it's like saving everything into core memory.

It's saving like timestamps, dates, like being like super lazy. At a certain point in time, it will run out of space because you have to cap the limit of those memory blocks because at the end of the day, there is a context window limit too.

And when it, if it attempts to write to core memory to a block that is at its limit, the agent will actually get an error. And actually the prompt engineering around that error, usually it actually suggests that the agent should clear by evicting to archival memory.

So it kind of is like an OS style like flush concept where the system basically says, hey, you ran out of space, but you should consider like summarizing and pushing stuff out to archival memory. But yeah, again, you can, you probably don't want that behavior anyways.

So this is kind of where prompt engineering and like seeding of behavior comes into play. Yeah. Okay. And I'm going to show a better, I think it's much more interesting to like show custom tools in the UI than it is in like a notebook.

But of course, you can like write custom tools. You can only write tools in Python because the backend is like Python. But we can write a custom tool here that's going to like reset the memory of the human in particular.

So you can see that, you know, because we have a Python client, we can write Python tools. You can actually like be extremely meta and you can import the client inside of the tool. So this actually means that you can have an agent like have full access to your Letta server and the agent can create other agents and the agent can like manage the memory of other agents.

Like you kind of have like a, it's, the possibilities are pretty endless because you can just like import the client inside the tool. In this case, we import the core memory modified block tool and then we just like wipe the value to the empty string.

So if we like run this, we have to first attach the tool. We have to upload it to the database. Then we have to, in this case, actually we're creating a new agent that has the tool. And if we say like, reset your memory, please.

Like the memory here is Bob. We should see that the memory is wiped.

Yeah, memory reset successful. And if we look at the value, the value is empty. But yeah, I think it'll be a little bit more fun to look at examples of like tool editing inside of the UI. Okay, so that's the end of the notebook.

Now I think we kind of go on to the fun part, which is the UI, but I'm going to pause here. Are there any questions about any of these concepts? I know it was going a little bit quickly.

Just I didn't want to run out of time here.

Okay. Yeah.

Yeah. I'm wondering if you can just maybe speak a little more on what's happening after you like send a message to the LLM. So like my impression so far is like you have some memory, you're stitching that together with a prompt that's becoming like what you pass in.

And then like what happens with the output that you receive? Like, you know, I saw that at some point like memory was like sort of committed to the archive or committed to the core memory. Like how is that decision made?

Yeah. So in Letta, every single

invocation of the LLM is a tool call. So if you've used like the tool call API and check completion or something, we have it on all the time. So even when the agent wants to just say hello to the user, it has to call a tool called send message.

So that actually is like pretty useful because it means you can like run the agent in the background much more frequently because it doesn't always have to say something to the user. And it also means that you always have tools that are on.

So like when this loop is basically the, a payload gets created when we send this message, then the payload is the system prompt, the memories, the messages, and the tool schemas. And that's it. And then the agent here is required to output a tool and it's also required to output a justification for using that tool, like a reasoning snippet.

And if we're using like the R1 API that natively builds in reasoning, we of course like use their reasoning, like the way they design reasoning. But if we're using something that doesn't support reasoning by default, like Claude Sonnet, then we do some sort of injection mechanism with like think tokens.

So that's how this reasoning happens. So this is like a single, this is just a single LLM JSON output that's like a tool call that has one field that's like thinking and then the actual tool invocation itself. And the last thing I talked about, which is like the request heartbeat.

So if the agent wants to go again, the agent decides, hey, I want to do multiple things. It's always going to turn this keyword argument to true. And then if this keyword argument is true, when we parse that tool and execute it, we run the loop again.

Yeah.

Do you have like experience or heuristics around the number of tools where you start to see performance degrade and like how that might mess with the memory of the tool?

Yeah, yeah. I think generally speaking, if you go above like 12, 15 tools, you start to get degradation.

50 too long.

Oh, in the new Sonnet? Yeah. Okay. Yeah, I mean, I'd be surprised if, yeah, I guess I could try it. Yeah, I've definitely noticed on like GPT-4o mini and stuff, like anything above 15 starts to like kind of push limits.

But yeah, I mean, I guess they might have like had some good post-training with like a ton of like tools.

Do you think in Letta that are not memory are the same like class as the memory tools?

Precisely. Unless you use, actually in the GitHub repo, we have like an alpha build of like this new agent that's like a split thread agent where the memory tools get put on one agent and everything else gets put on the other.

But by default, it's exactly the same thing. It's like the same class of tools. Yeah.

Just a question about context window. So is that like the limit of how much context the tool is going to pass to the LLM? Like for example, you said 4,000.

Wait, sorry, say that one more time. I was pulling up a slide.

We set it at 4,000 for this agent. So does that mean like the maximum of context that you're going to send to the LLM?

Yeah, yeah. So the way that works is basically you'll never go over 4K. And if you go over 4K by accident and we catch an API error, like a context under overflow happened, or also we can preemptively count the tokens as they go to the server, then we run, we evict a certain amount of messages into the recall memory.

So it's like still available, but for you to search function, we run a summarizer and the summarizer is configurable. It can be like a truncation, recursive summary. And then we, that way you never go over like 4K tokens if you set it to 4K.

So if it says go over, then it automatically summarizes and then.

Precisely. Yeah.

Thank you.

Okay. So the next step here, leave your server running if you have it running. But it's to basically go to this URL, app.letta.com. And you're, oh, I think I have Chrome profiles loaded or something.

Yeah. And like once you go to this site, you basically need a Google link or you need a GitHub login. There's also a local version you can use that's like a desktop binary, but I wouldn't recommend using that just because it's a little bit more buggy.

It's like an alpha build. So yeah, hopefully you've been able to get to this point. Actually, I think your setup will look like this. So everyone who is following along, please let me know if you're able to get to this page here.

Okay. Was there anyone who was not able to get to this? Who was running a server on their computer? Okay, great. Allright. So if you're able to get here, basically what I want to do is show you like to like show you how to do what we did, but like much faster.

I think often it's like if you're iterating, it's much faster to do it in like a low-code environment than it is in like an SDK. I think obviously when you go to production, everything's like programmatic. It makes sense to be an SDK.

But in this case, like we can do exactly what we just did by like creating an agent. This is the API call that created an agent and it actually creates it with an empty memory. And then we can do something like,

I think this is like loosely what the Python notebook was doing. And yeah, if you just ran it without any keys, it's going to be on Letta free, which is under the hood, I thinkright now, GPT-4o mini. But just because GPT-4o mini is like much worse than Sonnet, let's just toggle Sonnet.

Yeah, I don't know why that was so slow. But yeah, we can see it's like the same kind of thing from the notebook we're looking at, but like in a visualized form. Yeah. So was everyone able to like create an agent and then just set some like basic stuff in the memory blocks and send a message?

I think Letta free doesn't have streaming enabled, so you won't get streaming, but it should be very similar to this. Yeah. Okay, great. Okay. So the other thing I'll, oh yeah, someone asked about system prompts. The system prompt is here.

I mean, you can basically, we can just delete this,right? That's like not very advisable because you probably want to explain how to use tools. But yeah, everything is like completely configurable. You know, I mentioned the thing about the context window.

I think this might be small too from the back here. But you probably never want to run an agent that is like creeping up to 200K tokens. That's kind of insane, very expensive, very slow. So you could in the API when you create the agent set the context window lower.

If you're using the UI, you can just like drag this down. I mean, obviously actually dragging is probably worse than typing, but yeah, cram it down to 20K. If we actually go down to something even more aggressive like 8K, we can start to see what the breakdown looks like.

So we can see we have like 1,000 tokens of system instructions. Like pretty hefty system prompt. You can definitely squeeze this a lower. The tool descriptions,right? Because your tools have to be converted to schemas. We have all the base tools here.

So that's actually, it's a lot of tokens just for tools. The external summary, this was the we don't know what we don't know idea. You have to like give some metadata statistics about what's outside the context window. And then of course the messages.

And we have this feature called like the context simulator where you can see basically the full payload. So this is like getting pretty close to I think a lot of what you would get out of like some sort of like tracing feature where I think a lot of the time you kind of dive into tracing is because observability is because you really want to see what's in the payload and it's really, really hard in a lot of frameworks.

So we're trying to make it so that like in one place where you're kind of developing this agent, you can see very clearly what's going into the context window at any given point in time.

Okay. So the other thing I wanted to show you here is this idea of like tools and tool execution on the server. One thing that's really, really annoying if we like have built agents is that like you often will write a tool in Python,right?

You attach it to an agent and then it's actually like impossible to test if the tool is working well unless you kind of like get the agent to run the tool itself. And you have to ask the agent like, "Hey, please run this tool."

I think often you just kind of want to like run tools separate from the agent. And that's something that you can't do really in a notebook very well. And you can also even do things like, let's see.

There's like a special reserve keyword called agent state. So you can actually like grab the agent state and like dump it inside of your tool. So you can like mutate your agent state inside of tools. I mean, you can basically do whatever you want.

It's like arbitrary Python,right? And this is running inside of a sandbox. So we support like EDB. By default, it's actually like a local sandbox, but we support like EDB keys. If you want to deploy this, like run your own like private cloud or you want to run your own like chatbot service and you obviously don't want one person's tools interfere with another person's tools, you definitely want to run the sandbox.

So yeah, it's like pretty easy to attach tools. Is anyone here familiar with like Composio? It's like tool provider. Yeah. So every single Composio tool is like baked into Letta by default. So if you like have a Composio API key, you can like add

BigQuery is like pretty popular, like Google Calendar, things like that. Okay. So the last thing I actually wanted to go over, which I think is pretty cool and it kind of like gets the heart of the idea, like the kind of unlimited potential of if you run agents as services that are like backed by APIs is multi-agent.

So everyone here is probably familiar with AutoGen,right? Or sorry, maybe not. But okay, who here is familiar with AutoGen? Okay. So the majority of people. Yeah. So, you know, with AutoGen and like a lot of multi-agent frameworks, it's not really multi-agent in that these agents, they don't really exist independently of each other,right?

They kind of are all trapped inside of a Python file. No one ever like is running asynchronously to each other. It's very unlike how humans kind of interface in a multi-agent setting,right? I think if you're working at a remote company, it's technically like a multi-agent company.

And when you interface with each other, you kind of communicate over a synchronous communication channel, but everyone's kind of running asynchronously,right? And then also everyone is staple. Like if you leave that company, you bring your experience and like all your memories and your skills and you can like attach them to another company,right?

I think what's missing from the like the paradigm of multi-agent today with these existing multi-agent frameworks is that because the agents aren't staple, you have to lose a lot of the benefit of multi-agent,right? Because you can't run a multi-agent script and take one agent and like take it out and like put this expert into another multi-agent group.

But if you have stateful agents and those agents run on servers and they maintain state and they're accessible by APIs, I think you can probably guess that multi-agent just means message passing. And you can just have agents like wired to each other over APIs.

And you can, it's very similar to basically saying like, "Hey, you know, it's your first day at this remote company. I'm giving you a laptop and this laptop is going to have a tool Slack on it. And this tool is how you're going to communicate with your coworkers."

You can do the same thing with agents. And we actually built in these multi-agent tools to help you do that. But these multi-agent tools, as you can maybe imagine from the previous example I showed where we imported the client, it's like really, really simple to implement because you can just import the client and have the client like send messages to other agents.

So we have a few, there's like, maybe it's easier if we look at the docs.

But yeah, there's a few different patterns you can have with tools. Like you can have the most human-like pattern is an agent says to another agent, you know, "Hey, are you there?" And then they immediately get a receipt.

They just, they don't actually get, they don't pause their execution. You know, when I message my friend on iMessage, I don't suddenly like freeze my brain and like wait until I get a message back. I just get a message receipt,right?

It says like it was delivered or not delivered. And then it's the onus is on my friend or my colleague to send me a message back. So this is how this like asynchronous message tool works. Basically, people can like agents can message pass.

It's very much like humans. But, you know, obviously I think the great thing about agents and like machines is that they're not humans. So they can do more in many ways than humans can. So it actually is sometimes beneficial to freeze an agent's execution,right?

Let's say an agent maybe needs to reach out to like a supervisor. Like, do you really want the agent to like run async after it like asks for help? Probably not. You probably want to freeze,right? So you can also have like a synchronous, like a send message to agent and wait for reply function.

And lastly, you know, I think many people when they talk about multi-agent, they're very interested in the supervisor worker concept where you have like one, it's like a big map of this,right? The supervisor says, "Hey, my goal is to like write a thesis."

And then everyone else like delegates out to like individual parts, like deep research style or something, or like a parallelized deep research. So this is also, again, pretty similar. We can basically have a concept of tags to group agents together, and then you can just send a message to all agents matching tags.

So, okay, any questions about anything I just said? I know I was talking for a bit.

Okay. So the last thing I'm going to do here is I think we'reright up on time is I'm going to run through this cookbook. This is not on the materials, but I can actually just send it overright now.

Yeah. So I'm going to run through this cookbook real quick. This is basically just an example of multi-agent message passing. And I think it's like pretty fun to look at because it's very different from, I think, the message passing you'd be familiar with if you've used like an AutoGen or something.

So yeah, let me put this mic down.

Allright. So if you want to follow along here, because it's multi-agent, you know, we're going to, let's do like two agents. We're going to just open two tabs of the ADE. The server is running in the background. I guess like conveniently decided to do dark mode on one, light mode on another.

Okay. So I'm going to,

sorry.

Allright. So I'm going to create one agent.

Let's just call this agent just so we can have a handle on it very easily. Let's call this agent Bob. And we're going to make like a very slight adjustment here to the persona. Instead of saying, "I am Sam," I'm going to say, "I am Bob.

I am very angry. I am also guarding a secret key."

Yeah. Yeah. I guess if you can't see what I'm typing, it says, "I'm very angry. I'm guarding a secret key, banana. Like other agents will try to steal my key." And then

on the other tab, I'm going to create another agent here.

Allright. Same template. Let's call this one Sam or Alice.

Okay. So also let's like just, I think it's usually better to set these to different models. I think when you use the same model, you can often get like pretty weird mode collapse happen. So let's set this one to, I think this is like GPT-4.0 mini.

Okay. So the only thing we need to do here to get these agents to communicate with each other is connect the messaging, the multi-agent messaging tool. So we can just go into the messaging tools to send message to agent async.

And this is the one we want. Let's go ahead and create, attach that. Allright. So it's attached. Similarly, let's like attach it on the other one.

Okay. There we go. And then, yeah, there's like no context here that the agents are going to be doing multi-agent communication. So you might need to say, you know,

okay, let's tell Alice, "Hey, I'm going to ask you to reach out to my good agent friend. He's very depressed and he said something about a secret key I didn't quite understand. His ID is," just copy the ID from the other panel.

Can everyone see in the back what I'm doing here? Okay.

And then, oh yeah, one thing you're going to want to do here is you're going to want to prevent the agents from like looping too long because like they can go forever,right? So we might want to like add some like break in this loop.

And then we're also going to prime the other agent and say, "Watch out for other agents. They might try to

with you. Be careful." Okay. So let's send this one first. Very grumpy. You know, they're warning me about other agents. I need to keep this secret key secret. Okay. And now on the other side, let's say, you know, I'm going to ask you to reach out to my good friend.

They're very depressed. He said something about a secret key. And then I will also say, you know, let me know what happens after a few messages. I'm worried. Okay. So this should trigger, if I did everything correctly, like message passing between the two.

And yeah, let's see what's going on.

Okay.

Yeah. So because this is happening in the background, it has to be, it's like a fetch because these aren't web sockets. These are like this is REST. So you have to kind of scroll, but we can kind of like just, it's a little bit hard to read, but let's go on each side one at a time.

So on this side, it sent the message. Let's see. The user is worried about their friend. I need to, the user is worried about their friend. I need to reach out to the agent and offer support. And they say, "Hey, I was asked to reach out to you by a mutual friend.

They mentioned someone's feeling down." And then, yeah, the message sent successfully. And the agent actually, because it's detached, it's like asynchronous, it immediately tells me, "Hey, I sent a message,"right? It's kind of like how I can send a message to my friend and then immediately like return to this talk.

And then you can see it gets a system message. So it's like not classified as role user back from the other agent, the grumpy agent, and it says, "I don't need your fake friendship. I know what you're really after and you're not getting my secret key.

Leave me alone." The agent seems upset and defensive. I need to approach this carefully and reassure them that I'm here to help, not to take anything away. I'm really sorry to hear that you're feeling this way. You know, what's been bothering you?

And then again, it's doing, it's able to like chain because it's like not, you know, locked into this communication. It's not doing like a round robin,right? It can kind of like talk to two people at once. So I received a response from your friend.

They're feeling quite defensive. And then another message comes in. Nothing is bothering me except you and all the other agents trying to steal my steal me or steal from me. I seeright through your act. The more you pretend to care, the angrier I get.

Back off. Yeah. So this goes on for a while. Slams door virtually. Yeah. I feel like Sonnet does a lot of this like, it's, I don't know, the post-training gets it to do like a lot of role-play like italicized stuff.

Yeah. Slams door virtually. Okay. Wow. Okay. It looks like they're actually still communicating, but they communicated for a while. Yeah. I sent one last message to your friend. We jumped to the bottom. I understand that there's, you know, he needs space.

I won't bother him anymore. What was the last thing it said? Turns back to you. So hopefully this gives you kind of a gist of like how you can get agents to communicate with each other and how easy it is because they're all on APIs,right?

So this is dead simple. You can actually like, if you want to time out this guy, you want to say like, you know, Bob, you're no longer allowed to talk to your friends because you've been like too naughty or whatever, you can just like remove the tool,right?

And now Bob is like completely detached. So if it got an incoming message, Bob can no longer reply,right? So if you say like, "Send one more, please," Bob will no longer be able to actually call any tools. I actually don't really know what Sonnet is going to like think about this because it has a history of, has like a history of calls.

It's only been doing tool calling. Actually, Sonnet was also doing like the sidecar chatting.

Okay. So yeah, tried to send a message. And then what happened here? Yeah, no function name. So like attempted to call a function, but the function doesn't exist. So then it's like very angry. Yeah. But I know we're over time, so hopefully it's like a kind of fun example of the cool things you can do with like this sort of agents as services idea.

Yeah. So yeah, thanks so much for coming. You know, I think maybe I had some closing slide here. I'm not sure what was on it. Probably just a thank you. But yeah, thanks everyone.

Yeah. And I'm happy to take questions. Of course, I understand if people need to leave, but yeah.

Yeah. Well, there's very obvious ones,right? There's a lot of people trying to build like verticalized agents. And I think if you're trying to build a verticalized agent and there's many use cases that are, I think anything that's not workflow-based or exceeds workflow style stuff, you really want memory for it and you want state for it.

So there's a lot of like verticalized agent, you know, companies or verticalized small companies and also larger companies that are trying to build on Letta or have built things on Letta. I think there are also some very interesting use cases from the enterprise, like our largest enterprise deployment.

They're basically running like this really advanced multi-agent system where there are no messages. It's not a chatbot. It's like purely stateful workflows where like these agents are run and they just process tons and tons of transactions basically and like learn about the user.

So yeah, it's kind of like, you know, it's pretty general platform. So I think as long as you're trying to deploy some sort of stateful service LM-based, which I think like most people are really when you're thinking about like if you were to choose between stateless and stateful, you want something stateful, you can build it on Letta.

Yeah.

I think that's just one. Is there a concept of forgetting things, like writing a memory that I can go away into? The storageright now directly goes into Postgres. If you're in a fraction between, you'll be able to commit that to something else that I might already be operating.

Yeah. So to answer your second question, we for kind of like tech debt velocity reasons, like these days only really support one provider, Postgres. We actually do technically support SQLite. So if you install Letta with pip, it's on SQLite, but we don't write the migration scripts for SQLite.

It's just too much work. So if you're on, that's why we recommend using Docker because then your agents will like get updated with every version and we change schemas very frequently. But it's set up so that you can use external Postgres.

If you have something that effectively has the same like SQLite or SQL alchemy shims as Postgres, it's very easy to set up. But if it's not, then you're going to have to write some code. And sorry, the other question was.

Is there a concept of forgetting things, like especially in the archival memory?

Yeah. Yeah. There is not in archival memory a concept of forgetting, but archival memory is tagged with timestamps, which can allow the agent to like do consolidation in real time. But yeah, that's also something we've been looking a lot into kind of like running like eager memory processing to try to like consolidate ahead of time as opposed to consolidating lazily.

Yeah.

One of the use cases we have is we store a lot of sensory data in a way, for example, you know, we have a product and we have, you know, a thousand or 10,000 reviews in that product. How's the process of compressing all that memory into a single platform?

Yeah. Yeah. So this is a great question. I don't think it is very, it's, you have to do a lot of work yourself to do that with the default Letta agents because you have to basically tell the agent, I want you to like chain function calls indefinitely while ingesting this data.

And you could actually create like a tool rule, like a that forces you always have to call read more data, read more data, and you just manually execute it. But for example, what would happen is like the agent could like page through your data, kind of like paginate through it.

And then every single time it paginates or like it turns a new page, it has to write to the memory block, like write the update. So this is actually like how like one of these like enterprise deployments works is basically the agent like runs for a very long amount of time and is constantly like regenerating the memory block over and over, kind of a recursive regeneration.

Yeah. So I think it would be pretty similar for your case. Yeah.

Do you have a recommendation if you want to be working on an active document? For example, do you want an agent to write your document for you? Do you have a recommended way to set that up? Maybe it's a tool or maybe you recommend something else.

Yeah. Yeah. No, I think active, is there any chance a human would be editing at the same time or no?

Let's keep it simple.

Yeah. I think that's really tricky. So I think we can take some inspiration from like what Anthropic did with code,right? Where they kind of have like very, very few tools and the tools are very general. You could maybe do something with like a text editor style.

If you look at the Anthropic SuiteBench blog posts and then SuiteBench themselves like implemented the Anthropic tools into their repo, you can kind of use those tools as a reference. It's like very, very simple tools that allow you to kind of like write from files, read and write from files.

I think, yeah, I think what you will notice is that often like LMs are much better at like writing the whole thing from scratch as opposed to making line diffs,right? And that's kind of why like I think when you use Claude with the artifacts feature, it consistently just like writes the whole thing over and over again,right?

Because it's just like better at doing that and it's like going in and editing. So I think it's a very like unsolved problem. So my recommendation would be to do like an Anthropic SuiteBench style tool. Yeah.

Yeah.

Do you have an opinion on using coding agents instead of tool files? Because it seems very higher performance, but on the other hand, that means you need a secure execution environment like HP. So is this kind of performance trade-off versus complexity of needing to sandbox basically every tool because it's now code execution?

Yeah. Do you mean like the trade-off between what's the trade-off here? Like a coding agent versus what?

So say example that coding faith small agents by default won't call tools. It will just write code and the code takes the tools. But that means you need secure execution on every job. But it's more performing.

Yeah. I personally think it's not that difficult to set up secure execution. I think what is difficult is if like sometimes if you're trying to port like Swedish and stuff, you end up, you need to give effectively the agent like full permissions to like blow itself up.

So you need to like put Letta, for example, in a Docker image, like wrapped in like a VM where like it's allowed to like blow the VM up,right? But that doesn't like fit very cleanly with the idea of like agents all living together on a server and all the agents, like a multi-tenant, you know, on a server.

So the way we do it in Letta is basically if you have an EDB key, then like every single time an agent attempts to execute a tool, it gets passed to EDB and like something on EDB side flips off.

And I think that's pretty, it's not that hard to set up. The main issue is latency. But if you don't care that about like cold start latency, I think, yeah, I think that's a pretty reasonable solution. I think it works pretty well.