Architecting and Testing Controllable Agents: Lance Martin

Good to be here. Good to see a fair number of people. It's early, so I wasn't sure if anyone would come, but thank you for coming. Um, one quick note: I just put my slides on Twitter. I wasn't sure if it was the best way to access everyone.

I'm@rlancmartin. If there's another way I can get everyone's slides, then yeah, I see some people opening. So the slides will link to a few different, uh, it'll link to a collab and other notebooks I'm providing, so all the code will be available for you.

Okay, good, I see people finding it. That's fantastic. If there's a better way, let me know, but I figured this is somewhat easy.

Well, it's great to be here. We have a bit of time, so I think the format is I'll, like, lay out some slides to kind of set the preliminaries and, like, kind of give the big picture. And then there'll be a bunch of time where I can just walk around, talk to people, and because we have, I guess, three hours, so I think the idea of this will be a hands-on workshop.

I provide a bunch of starter code, and also one of my slides shows it'll be kind of a choose-your-own-adventure format. So why don't I kick it off? I think maybe one or two people are still coming in, but.

So the theme here is building and testing reliable agents. And let me go to slideshow mode here.

And maybe I'll just kind of start with, like, the very basics. You know, LLM applications follow a general control flow of some sort. You start, usually it's a user input, there's some set of steps, and then you end.

And you've heard a lot about chains. You know, when you build applications, oftentimes we talk about this idea of chains. And chain is just basically, you know, it is some control flow set by the developer. Again, you start, proceed through some steps, and you end.

So retrieval augmented generation is a super popular application many of you may be familiar with. Basically, it refers to retrieving documents from an index and passing them to an LLM. This is a good example of a chain. It's a control flow set by a user.

The questions, you know, provided, vector store retrieves them, pass to an LLM, LLM produces an answer. So this is kind of a classic chain. Now, when you get into agents, there's a lot of different confusing interpretations. What is an agent?

Here's the way I might think about it, which is just a really simple kind of framing is, agent is just one of the control flows set by an LLM. And so you can imagine we talked about this process of you start your app, step one, step two.

In this case, I have an LLM in there. An LLM looks at the output of step one, makes a decision. Do I go back and do I proceed? So that's, like, the way, a simple way to think about an agent.

So again, chains, developer-defined control flow. I set it ahead of time. I follow some set of steps every time. An agent, an LLM kind of determines the control flow. An LLM makes a decision about where to go inside my application.

That's one simple way to think about it. Now, you hear about function calling a lot, and this is kind of a confusing topic, so I want to talk through it kind of carefully. Agents typically use function calling to kind of determine what step to go to.

So usually the way this works is, what you do is you basically give the LLM awareness of some number of tools or steps that it can take. So in my little example here, I define this tool and this little decorator, it's a line chain thing, but the point is, I have some step, it's some function, I'm defining it as a tool, and I'm binding it to the LLM.

So then the LLM has awareness of this tool, and here's the key point. When it sees an input, like what is the output of step two, it actually produces the payload needed to run that tool. Now, this is often confusing.

Remember, LLMs are just string to string. They don't have the ability to magically call some function. What they can do is produce the payload or arguments needed to run that function and the function name. So really think about tool calling or function calling as just an LLM producing a structured output.

Still, you know, obviously a string, but it's a structured output that then can be used to call a tool. So that's all that function calling is. And you might have heard of React agents. So the way to think about this is, it's just basically binding some set of tools to my LLM.

And again, we talked about tool calling. So the LLM makes decisions about what step or what tool to use. And you have some node that will call that tool. So the LLM says, okay, run step one. I have some node that runs step one and passes the output of step one back to my agent.

React typically stands for, like, basically action. So the LLM chooses the action. A tool is run, it observes the output, that's what goes back to the agent, observes that tool response, thinks about what to do next, maybe runs another tool, and this runs in a loop until you end.

And usually the end condition is the LLM just outputs a string response, not a tool call. So this is the way to think about a classic React agent.

And it's really flexible. That's the nice thing about a React agent. So basically, it can implement many different control flows. It can do step one only, step two, one, two, two, one. That's the beauty of these open-ended style React agents.

And these have a lot of promise. These kind of flexible tool calling agents were really hyped last year. They're still really hyped. It's really exciting because they're flexible. They're open-ended. You can give them a task, give them some tools, and you can just execute arbitrary control flows given those tools to solve open-ended problems.

The catch is, and this is kind of the crux of what we're getting to with this workshop, is they do have poor reliability. They can. So you can get caught if you've played with agents. Some of those you can see, they kind of get caught on one step and they keep calling the same tool.

And, you know, really this is often caused by LLM nondeterminism. LLMs are not deterministic. And also errors in tool calling. So tool calling is kind of tricky. If you think about it, an LLM has to basically pick theright tool given the input and has to pick theright payload.

So it has to produce theright inputs needed to run the tool. And these both can break. So here's a good example. The tool I'm passing is step two, and the LLM is saying the tool name to run in step three.

So that's obviously wrong. Or I'm passing what is step two of the input three, and the LLM says, okay, pass four. So both these errors can happen. Tool calling is a tricky thing, and it's exacerbated. If you pass an LLM five tools, ten tools, it actually gets worse.

If you have very long dialogues, it gets worse. And so this idea of open-ended tool calling agents is really promising. It's really exciting, but it's really challenging because of these issues that we were mentioning here. So this is kind of the build-up here.

Like, so can we envision something in the middle? So again, we talked about chains. They are not flexible, but they're very reliable. This chain will always run step one two in order. We talked about, like, React agents on the other extreme.

They're extremely flexible. They can run any sequence of tool calls. You know, it can run step one two, step one only, two only, two, one. But they do have reliability issues. So can we imagine something in the middle that's both flexible and reliable?

So here's kind of the setup. And this is, like, kind of the intuition. Like, a lot of times in many applications, you have some idea of what you want the thing to do every time. So some parts of the application may be fixed.

Like, the developer can set, okay, I always want to run step one, and I want to end with step two. And you can inject an LLM in certain places that you want there to be some kind of branching or kind of optionality in the control flow.

Okay. So this is the motivation for what we call LangGraph. So LangGraph is basically a library from the LangChain team that can be used to express control flows as graphs. And it is a very general tool, and I put out a bunch of videos on it, and we're going to use it today.

And by the end of this, you will all have an agent that runs reliably using LangGraph, hopefully. And we'll see. So you should test me on that. If things don't work for you, then we'll work it out. But the idea is kind of this.

This graph has some set of nodes and edges. So nodes you can think about are basically, well, maybe I should start with this. This graph has something called state. So it's like short-term memory that lives across the lifetime of this graph that contains things you want to operate on.

Nodes modify the state in some way. So basically, each node can, like, call a tool and can modify the state. Edges just make decisions about what node to go do next. Okay. So you basically have this idea of memory.

And this is the same as common agents,right? Agents are characterized by having tool calling and short-term memory as well as planning. Those same things are present in LangGraph. Memory is this state that lives across your graph. Tools exist within your nodes.

And planning, basically, you can incorporate LLM-dictated decision-making in the edges of your graph. So, like, why is this interesting and where has this been cropping out? We've actually been seeing this theme crop up a lot of places. So there's a really interesting paper.

There's actually a few I really like. This one's called Corrective RAG. And the idea is pretty simple. Like, with a naive RAG pipeline, you're doing a retrieval, you're taking retrieved docs, and you're generating your answer. Corrective RAG, like, is doing one step more where it's saying, well, why don't we reflect on the docs we retrieved and ask, are they actually relevant?

You can have lots of issues with retrieval. You can reflect on the documents, see if they're relevant. If they're not relevant, you can do different things. You can kick out and do a web search. So it makes your application a lot more dynamic to poor quality retrieval.

So this is one of the first videos I put out on LangGraph back in February. It was very popular. And I basically showed you can build Corrective RAG inside LangGraph, and it's super simple. This is what the graph looks like.

I do retrieval. I grade my documents. And we're going to actually, we're going to do this today, and I have a bunch of code for you that does exactly this. So we're going to go way in detail on this one.

But this is kind of the setup. And I showed this working. I showed it works locally with Ollama using, at that time, it was Mistral 7B. And it works really well. So this is, like, one simple illustration of how you can use LangGraph to build kind of a self-reflective or corrective RAG application.

Now, another cool paper was called SelfRAG, which actually looked at the generation. So basically, we're all familiar with the idea of hallucinations. It's a real problem. Instead of just allowing hallucinations to propagate to the user, you can actually reflect on the answer relative to the documents and catch hallucinations.

If there's hallucinations, you can, like, basically do different things. And they propose a few ideas here. I implemented this, and this is actually our most popular video of all time. So this was showing LangGraph and LLaMA 3 implementing three different things: Corrective RAG, which we just talked about, the SelfRAG thing of hallucination checking, and this Adaptive RAG thing.

So I can kind of walk through it. This all runs in LangGraph locally. And I have the notebook here. If you want to test that today, you definitely could. So that's the point. It's reliable enough to run this whole thing locally.

So what's happening here is, I take a question, I route it either to my index or to web search. I then retrieve documents. I grade them for relevance. If any are not relevant, I kick out and do a web search to supplement my retrieval.

If they're relevant, I generate my answer, I check it for hallucinations, and then I finally check it for answer relevance. So basically, does it have hallucinations relative to my documents? And does it answer my question? If all that passes, I finish and return that to the user.

So this is kind of like a complex RAG flow, but with LangGraph, you can actually run this on your laptop. It is reliable enough to run on a laptop with LangGraph. And the intuition, again, is that you're constraining the control flow.

You're allowing the LLM to make certain decisions but at very discrete points. If you implement this as a RAG agent, this can be very open-ended and a lot of opportunities for breakage. And so that's the real intuition here.

Now, a final theme is Karpathy kind of mentioned this idea of flow engineering related to this Alpha Codium paper, a really nice paper on code generation. And the intuition here is, produce a code solution. They tested this on a bunch of coding challenges.

Produce a code solution and check it against a number of unit tests, auto-generated or pre-existing. And basically, if it fails unit tests, feedback those errors to the LLM and try again. Really simple idea. I implemented this in LangGraph.

Again, the code is here. And this works really well. So I basically, I share a blog post as well. I ran this on our internal coding. We have an internal application for RAG at LangChain, and we're actually working on implementing thisright now in production because the performance is way better.

And a common thing this can fix with code generation and code solutions is hallucinations and imports. So we see that a lot with our RAG app. So what I did was, I very simply implemented a unit test for import checks.

Just run that. It significantly improves performance relative to without doing it. And so we're actually working on implementing this in our internal RAG system. So super simple idea that can really improve code generation. So, you know, if I kind of, like, back up, what did we talk about?

We talked about chains. They are not flexible, which is fine in some cases. But a lot of interesting newer papers with RAG, for example, this idea of self-reflection is really beneficial. The ability to kind of self-correct applications can be really beneficial, not beyond RAG as well, for coding.

So chains are very reliable, but they're not flexible. Now, if you go to the other end, like a classic React agent is very flexible. It can implement any sequence of control flows through your different tools, but it does have reliability problems due to things we talked about: nondeterminism, tool calling errors.

And LangGraph kind of sits in the middle where you can actually implement these user-defined slash LLM-gated control flows, and they can actually be extremely reliable because of that constraint. They are less flexible than a classic React agent, so that is true.

So for very open-ended tasks, I do agree. Maybe you do need, you know, a very open-ended, more like autonomous style React agent. But for a lot of applications that our customers are, like, working on and seeing, these kinds of, like, hybrid flows are sufficient, and you gain a lot of reliability.

And so we've talked to a lot of companies that have implemented LangGraph successfully for agentic flows for this reason, because reliability is just incredibly important in a production setting. So this gets into, if you look at the slides, I have a few different notebooks.

And what I show is, we talked about Corrective RAG. These notebooks show how to build Corrective RAG yourself. And I thought that's a fun starting application. It's a really popular one. It's super simple. There are not many dependencies.

You can use your own web, whatever tool you want to use for web search. You can use other things as well. Just have a look at the notebooks, and I'll kind of walk around. And we're going to keep going.

I just want to, like, this is just like a placeholder here. But so if you want to test this locally, if you have a laptop capable of running things locally, then we have a notebook to support that. I use Ollama.

I can talk a lot about that. That's a really cool thing. If you don't, then I have two options for you. So one is a Colab. So that's probably the easiest. If there's issues, let me know. But I've tested it.

So if you have a Google account, you can just spin up a Colab. All you need is a few API keys depending on what models you want to use. It's all kind of there. You can set those accordingly.

And I also have a notebook. So this just kind of is like a kind of a gives you a roadmap of the different things you can try today, since this is a workshop format. And I'll just be walking around, and we'll do questions for a while.

But I want to talk about the second half of this story.

So one of the themes we're seeing a lot, and I think you're going to hear a lot at this conference, is the challenge of testing and evaluation. And this is a real pain point. Like, for example, how do I actually know that my LangGraph agent is more reliable than the React agent?

How do I know what LLM to use? How do I know what prompt to use? Right. So testing agents is, testing in general is really hard, and agents in particular is challenging. So there's kind of three types of testing loops I like to think about.

One is this in-app error correction, and that's actually what we just talked about. So LangGraph and LangGraph agents are really good for that. So basically, in-app error handling, where you can catch and fix errors, is really useful for code generation.

For RAG, we just talked about that. So that's, like, placeholder one. Now we get into this idea of pre-production testing. And then finally, production monitoring. And I want to introduce a few ideas on the latter too. So we just talked through this.

Here, we're going to build Corrective RAG a few different ways, and I just showed the choose your own adventure stuff. And so this is just kind of reiterating that. But I want to show you some other things. So LangSmith is a tool from the LangChain team that supports testing and evaluation as well as monitoring and tracing.

And so we've seen a lot of interest in this, and it's quite popular. It is really useful for doing these types of testings and evaluations. So the key idea behind LangSmith and the notebooks actually have this. So this is totally optional.

If you just want to build an agent, that's totally fine. If you want to also test it, you can use LangSmith. You don't have to, of course, but I have it all set up to use LangSmith if you want.

It's free to use, of course. And so the idea is, there's kind of four components that I like to think about when it comes to testing slash evaluation. You have some datasets. That's some set of examples you want to test on.

So say you have a RAG app. That's like a set of ground truth question-answer pairs you've built. Like, you're testing your system. You have question-answer pairs that you know are correct. Can your system produce those answers? How many will actually getright?

You have your application. That's your agent. That's your RAG app. That's your code app. Whatever that is, that's your application. Now, the thing that's often the trickiest, you have this evaluator thing. And the notebooks show you in detail, but this evaluator is something as simple as a user-defined function that can implement a few different things.

You can think about using an LLM to actually judge your outputs. So in that case, let's take RAG as an example. My application produces an answer. I have a ground truth answer. You can actually have an LLM look at those two answers jointly and reason: is it correct?

And this is often very effective. It requires some prompt engineering. I have some nice templates in the notebooks to show you. But this thing is very popular. This idea of LLM as judge evaluators is very interesting. A lot of people, actually, you'll probably hear about it this week.

It's a really good theme. It's still kind of in development, but that's, like, one placeholder to keep in mind. So for one option for kind of testing is this idea of using LLMs themselves. The other is building your own heuristic evaluator, so a custom evaluator of some sort.

And actually, the notebooks that I share have both.

And so we're actually going to, the notebooks actually show how to evaluate an agent specifically. And there's a few different things you can look at with an agent. So one is, if you go to that farright in blue, the end-to-end performance.

So our notebooks are basically going to be a RAG agent. The eval set has five questions. And I basically have a set of question-answer pairs. So basically, I'm going to compare my agent answers to reference answers. And we'll walk through that in the notebook.

But that's kind of one thing. I just want to introduce the ideas. So one big idea is you can evaluate the end-to-end performance of your agent,right? You don't care anything about what's happening inside the end-to-end performance. The other, which I actually like to look at a lot, is this thing on top.

What are actually the tool calls that the agent executed? This is how you can actually test the agent's reasoning. So what you see often with agents is, they can make some weird trajectory of tool calls that's highly inefficient but still get to theright answer.

You don't get that if you only look at the answer. You say, "Okay, it's got theright answer." But if you look at the trajectory, it's some crazy path. And so you want to actually look at both, like, how efficient, how correct is the trajectory, and does it get theright answer,right?

And so the notebooks I share actually do both. Now, this is actually an evaluation that I ran, and this dataset is public. This is on the agents that we actually just talked about. So this is kind of what you see when you open LangSmith.

So these are different experiment names. This is just saying, like, I've run three replicates of each experiment. And these are my aggregate scores. So this first score is basically the answer correctness thing. And the second score is, like, the tool use trajectory or, like, does it use theright reasoning trace?

And I can go through my experiments. So this top one is actually, this is kind of cool. This is actually my local agent running on my laptop with LangGraph, okay? It's a five-question eval set, small eval set, just a bunch of, it's some very small test examples.

But basically, my local agent does not, does fine. It does 60% in terms of the ultimate answer. So that's not amazing. But it does do very well in terms of the tool calling trajectory. So it's very reliable in terms of reasoning.

It's an 8 billion parameter model. So I basically, I think the quality of its outputs are a little bit lower than you might see with larger models. Now, Fire Function V2 is another option. It's basically a fine-tuned Llama 70B from Fireworks.

This one with LangGraph, so this is actually showing this top, actually gets up to 80%. So very strong performance in terms of answers and 100%, again, in terms of tool calling. So the key observation here is, the tool calling or reasoning is consistent, whether you're using a local model or a 70 billion parameter model with LangGraph.

So you get very high consistency in your tool calling. The answer quality degrades. That's more an LLM capacity problem. But the reasoning of the agent is consistent. So that's the key point. Now, here's where it gets interesting. Fire Function V2, again, that's Llama 70B.

This is with a React agent. What you can see here is the answer quality is degraded. But here's the interesting thing. The tool calling trajectories are really bad. And this, again, gets back to that problem with React agents.

They're open-ended. They can choose arbitrary sequences of tool calls, and you can deviate really quickly from your expected trajectory. So that's the key intuition here. Now, the final two are GPT-4.0. So that's obviously a very flagship model. It's maybe number two now, again, sawn it on the chatbot arena at least.

You know, again, answers ultimately are strong. The tool calling, though, even here is degraded. So basically, it follows some weird trajectories to get to its answers that are unexpected. So what's the high-level point here? The high-level point is, LangGraph allows you to significantly constrain the control flow of your app and get higher reliability.

And if you look at these tool calling scores, it's very kind of consistent, going all the way down to local models. It follows the same sequence every time. React agents kind of go off the rails much more easily.

The answer performance is really a function of the model capacity. So using an 8 billion parameter model locally, the answer quality is lower than a 70 billion. That's to be expected. But the reasoning of my app is consistent and strong.

So that's the key thing that you kind of get with LangGraph. And this is all public. And hopefully, some of you will actually implement this or reproduce this today. And this is just walking through those same insights I just mentioned.

And then deployment, we're going to be talking later this week. We have an announcement related to deployment of LangGraph. So this is actually a very good setup. If you're playing with LangGraph, you enjoy working with it, we're going to have some really nice options for deploying later this week.

And so Harrison will be here on Thursday to give a keynote on that one. And if you've deployed, we also have some really nice tools in LangSmith to actually monitor your deployment. And this is not as relevant for this workshop, but it's something just to be aware of.

I can talk about it if you're interested. So maybe to close out, so this is a really nice write-up. These guys are actually going to give a keynote later this week. It's Jason and company, Hamel, and others. And they kind of made a really nice point that the model is not the moat.

Like, LLMs are always changing. The moat is really the systems you build around your application. That's what we talked about today. Like, do you have an orchestration framework, for example, like LangGraph? Do you have an evaluation chassis, like LangSmith?

And again, you don't have to use LangGraph. You don't have to use LangSmith for these things. But this workshop will introduce these ideas to you. And frankly, I think it's important to just understand the ideas rather than the implementation.

Whether or not you use LangGraph, whether or not you use LangSmith, I think understanding these principles is still helpful. But you know, an evaluation chassis, guardrails, data flywheel, these are like the components that give you the ability to improve your app over time.

That's really the big idea. That's the goal. And I think you'll hear more on that later this week. The goal here is, how are you measuring improvement of your app and ensuring it always gets better? That's what we're actually trying to achieve here.

And that's kind of what evaluation is giving you. And yeah, this is kind of my last slide. And then maybe we can just move into maybe some Q&A. I can actually show the notebooks themselves if you want to walk through them together.

I mean, I'll just do that, and I'll let you guys kind of hack on them in parallel as I walk through them. And then I can just go walk around and talk to people, something like that. So the three types of feedback loops, you have this design phase feedback, something like LangGraph, in-app error handling.

That's kind of step one we talked about. Cool examples there for coding, for RAG, a lot of nice papers, really promising. I'm very excited about anything you can do here with terms of kind of agentic self-correction, self-reflection in your app itself.

Pre-production testing, we just talked through that. Building evaluation sets, running evaluations, testing for an agent, like your tool use trajectory, your answer quality, all really interesting and important. And then in production phase, production monitoring, this gets into, we didn't talk about it too much, but basically this stuff.

So basically, you can have evaluators running on your app in production, looking at inputs, looking at outputs, tagging them accordingly, and then you can go back and look later. So that's kind of the setup here. I know that's probably a lot, and we went about half an hour.

So if there's any questions, I can just open it up, and we can kind of talk through stuff. I can also start ripping through some of the notebooks just to kind of give you an overview of the code itself.

But if there's any questions here, maybe happy to take a few, give you a minute to digest all that. Yeah.

Is there a non-Twitter link to your slides yet?

That's a good point. Let's see.

Is there a question?

Is there a non-Twitter link to the slides? Let's see if the conference organizers give me some. I don't know if I have an email list for everyone in here.

Just post to the general channel.

Is there a, okay, is there a Slack? Yeah, if someone can put, I actually didn't know there's a Slack, so that's very helpful. Yeah, if there's a Slack or an app for this conference, then please someone post. I appreciate that.

I actually didn't know that. Thank you for that question. Yes, sir.

So about the testing and the evaluation, does it really scale to predict the exact sequence of these agents? If it's smart enough dealing with a complex problem, it's hard to say exactly how it achieves the task.

Okay, this is a very good point. So I'm going to repeat the question now. How do you evaluate an agent's, like, reasoning trajectory? If it's a large and open-ended problem, that can be solved in many different ways. In that particular case, you areright.

It is hard to enumerate a specific trajectory that is actually reasonable. For really open-ended, long-running type problems, trajectory evaluation may not be appropriate. One thing I would think about is, can you define a few canonical trajectories of tool use through your application?

So it depends on the stuff. If it's a very long-running agent, I think it's probably infeasible. If it is a shorter-run agent where it's like, you expect something in the order of maybe 5 to 10 steps, you can probably enumerate some set of reasonable trajectories and basically check, does it follow any of these?

You can also do things, so you can do things like this. You can do things like check for the repeat of certain tool calls. You can be very flexible with this. I mean, it's kind of open and fuse.

You can look for, like, is it repeating certain tool calls? You can look at recall, like, is it for sure calling this tool or not? So you can actually be very flexible. Actually, the way we set up LangSmith evaluator for this, it's just a simple function that you can define yourself.

So it's a very good point. You can be arbitrarily creative about how you evaluate that. But I would say, for very long-running, you'reright. You can't really articulate step one, two, and three. But I would then think about more like evaluating, is it repeating steps?

Can you evaluate for clearly aberrant behaviors, excessive number of tool use repeats, excessive number of overall tool calls? So kind of like guardrails related to, like, kind of clear aberrant behavior. It's very short. You can actually enumerate the trajectory specifically.

That's a good question. So in the code, you can see we actually lay out a custom function. You can define that yourself. So that's a very good point, though. Yeah. Yep.

Can we also mention the cases where you have a human in the loop?

Yeah. Human in the loop is a good one. The workshop notebooks I share do not have that, but LangGraph does have some good support for human in the loop. And I can share with you some notebooks that showcase that.

Also, what we have shipping on Thursday has very good support for human in the loop. So I will share some notebooks with you for that and wait for Thursday for even more there.

So if we're building a RAG-like application,right, pre-production, we can do this testing framework using a known set of, this is a question, here's the proper answer to it, then evaluate from zend to end. What I've been struggling to figure out is, like, theright way to approach that in production where we don't know what theright answer is.

And I'm wondering if LangSmith has materials for that.

Yeah. So the question was, this is actually a really good one. For RAG, I'm just going to go to our docs because I actually wrote a doc on this recently. So

you can still see my slides. For RAG, in a pre-production setting, it's easy to define, or not even easy, but you can define a set of question-answer pairs and evaluate them. When you're in production, though, how do you evaluate your app?

Because you don't have a ground truth answer. So what are other things you can actually evaluate for a RAG app that don't require a reference? Yeah. So there is a conceptual guide that I will share. So this is actually our RAG section.

I have kind of a nice overview of this. There's actually a few different things you can evaluate for RAG that don't require a reference that are very useful. Yeah. So it's thisright here. So this is like a typical RAG flow.

So I have a question. I retrieve documents. I pass them to an LLM. I get an answer,right? What we just talked about and we showed is comparing your answer to some reference answer. Now, this is, to be honest, pretty hard to do.

You have to build an eval set of question-answer pairs. Very important to do, but it's not easy. So what else can you do? So some of these we've seen that are really easy and actually pretty popular, there's three different types of grading you can do that don't require a reference, that are like internal checks you can run.

I mean, you can run them online. So one is retrieval grading. So basically, looking at the document, you retrieve documents relative to your question. So it's like an internal self-consistency check. So this is actually a great check to run.

And actually, the corrective RAG paper that, or the corrective RAG thing that is in the cookbooks that I share here does this. So you can play with the prompt and all that. But basically, this is just checking the consistency.

You retrieve docs relative to your question. You can do that. And we have some really good prompts for that. Another one I like is just comparing your answer to your question. Have an LLM look at, here's my answer, here's the question.

Is this sane? Are they related? And this is a really nice check just for, like, of course, you don't have a reference answer, but, like, you can still sanity check and say, does this deviate significantly from what the questioner is asking?

The other, this is a great one, is hallucination. And this is super intuitive. Compare your answer to the retrieved documents. So if the LLM went off the rails and didn't ground the answer properly, and you hallucinated, you can catch that really easily.

And so

I need to get on this Slack because I want to share this link with you. I'll figure that out. I'll find you. But this is in our LangSmith docs. If you search LangSmith evaluation concepts, we have actually a bunch of videos that showcase how to do this.

And I have a bunch of code as well. But those are three things you can do that don't require a reference. And we do run those as online evaluation with our application. So yeah. Yep.

Do you have any thoughts on unit testing?

Yeah, unit testing. So,

yeah, okay. So do we have any thoughts on unit testing? So LangSmith supports PyTest as unit tests. But basically, it depends what you mean by unit tests. Typically, like conventional software engineering unit tests

are very effectively done in things like PyTest. There's a lot of frameworks for that. What I like to think about in unit testing with respect to LLM apps is kind of like what we show

in this code generation example. Here, we use some really simple unit tests just for, like, imports and code execution. Simple unit tests like this can actually run in your app itself. So basically, one place you can think about putting unit tests are actually in your app itself within LangGraph for self, for kind of in-app error handling or self-correction.

So that's like one place for unit tests that's kind of interesting and new with LLM applications. They can live inside your app itself. Another good one for unit tests within app is if you're doing structured output anywhere in your application, which is like a really common thing people like, confirm the schema is correct.

That's another good use case for unit tests within your application. Both of those things could also be done independently, like in CI outside of your application. So we are going to have more integration support for CI with LangSmith soon.

I will check with the team on that. But I think the interesting idea for unit tests with LLM applications is this idea of inline within your app itself. Because LLMs are so good at self-correcting, if you run unit tests in your application, they can often catch the error and then correct themself.

And unit tests are fast and cheap to run. So it's actually a really nice kind of like piece of alpha that, in fact, that's exactly what Karpathy was mentioning here, that running unit tests inline with your application is actually really quite nice and produce significant improvement in performance and alpha codium.

Cool. Yep.

Is it that you can use a small LLM to do a bit of work?

Okay.

Does it need to be fine-tuned? Can it be fast?

Yeah, yeah. So what the question was, if I wanted to do some of this in-app error correction stuff, so let's take this example, the corrective RAG thing. If I actually want this to run in my application, it obviously needs to be super fast.

So that's actually what we've seen. The tricks we've seen here are basically use very fast, smaller LLMs. So you mentioned, for example, even the ability to fine-tune. That's actually a good idea. If you have a judging task that is very consistent, it's a very good use case for fine-tuning, actually.

Fine-tune a small, low-capacity, extremely fast, and effectively very cheap to deploy model. That's a very good idea. We've seen people do that. Also, use very simple grading criteria. Don't have some kind of arbitrary scale from 0 to 5 with high cognitive load that LLM has to think about, yes, no.

Very simple binary grading. Even for some of the stuff, you can even be old school and fine-tune a classifier. But basically, really simple, lightweight, fast LLM as judge style. Classifier wouldn't be an LLM necessarily, but basically, very simple, fast tests for in-app, anything like kind of in runtime you will need or want.

Another cool use case for this, or another interesting option for this, is Grok. It's very, very fast, kind of with their LPU stuff. And they actually would be a very interesting option. We've done some work on that with Grok, basically, for any of these kind of in-app LLM as judge error correction things using something like Grok, which is extremely fast.

But it's a very good insight. Fine-tuning your own model is actually a really good way. We've seen people do that for these types of anything with LangGraph and in-app error correction.

Cool. Yep.

Does it support multi-agent and the continuity of multi-agent?

Yeah. We have some good cookbooks talking about multi-agent, which I, again, will need to find a way to share with you.

Yep. So we have this. If you go to LangGraph, LangGraph GitHub examples, multi-agent, there's a few different notebooks here that are worth checking out.

Yep.

Does LangGraph allow you to define like cyclical graphs? And if so, how do you prevent yourself from getting stuck in cycles?

Yeah. Yeah. So actually, LangGraph is specifically designed for cycles. So some of the examples, like what we're showing today is only a branch, so it's a simpler graph. But for example, the React agent we'll show today is a cycle.

So it's basically going to continue in a loop just like this. And what you do is you set a recursion limit in your LangGraph config. So you basically tell it to only proceed for some number of cycles. And this is default set for you.

I believe it's like 20 or something like that. But that's what you're going to, that's what you're going to want to do.

Yep.

What about like timing of the responses? Is there any way to control that?

So the question was about timing of the responses. So do you mean like if you're implementing some kind of self-correction, how long that takes? Well, that's kind of a, that gets back to the question that this gentleman asked.

It depends a lot on the LLM you choose to use for your judging. And it's latency. So that's kind of where if you're, so maybe there's two sides of this. One side of it is choosing an LLM that's very, very fast, and that's very important to do.

Could be something like Grok. Could be a fine-tuned like deployment that you do yourself. Could be a GPT-3.5. So that's like one side of it. The other side of it is how do you actually kind of monitor and measure that?

And so, again, LangSmith actually does have very good support for tracing and observability. And we do have timings all. In fact, I can go ahead and show you very quickly if you want to see. So this is my LangSmith dashboard.

I'll zoom in a little bit. These are my experiments. Now, if I zoom in here, I can open up one. The Wi-Fi is a little bit slow. These are my replicates. I can open up one of my traces.

And what I can see here is over here, I get the timing. So this is the timing of the entire graph. And I can go through my steps. So this is like the retrieval is really fast. That's good.

Less than a second. Okay. Now, here's what's interesting. My grading in this particular case is like four seconds. That's not acceptable in a production setting, most likely. But again, this is just like a, this is a test case.

In fact, I'm using, what am I using to grade here? GPT-4.0. So it is, there's ways you could speed this up by using different models or different prompts or grading everything in bulk. There's a lot of ideas. I actually grade each document independently.

Oh, actually, you know what? This is using Chat Fireworks. So anyway, but you can look at your timings in LangSmith. That's a really nice way I like to do it to kind of see,

to kind of monitor the timing of my applications.

Yep.

So the case you showed, you still know like testing it from scratch,right? When the agent is running from the beginning and trying to accomplish the task. But in reality, usually we will have some context or history that's being passed to the agent.

Do you have any suggestion in terms of how to test that scenario? Because say they don't have to do a function call because that's already in the history versus like reality.

Let me make sure I got the question. So the question was, with agents, you typically pass them a message history. That is absolutely true. And in fact, the React agent that we implement here, like I can even open up one of the traces.

We can look at it together.

So here's a React agent with GPT-4.0. Here's one of my traces. Let's open it up and actually see what's going on here. So what happens is, first, my assistant isright here. So this is OpenAI. Here's the system prompt,right?

So your helpful assistant, you're answering questions. Here's the human question,right? So again, this is the start of our message history. Okay. And

also, these are the tools that the LLM has. And what's pretty cool, we can see this is the one it called. So what happened is our LLM, it looked at, here's our system prompt, here's the human instruction. And it says, okay, I'm going to retrieve documents.

Great. So then it goes, and this is the tool call. It goes and retrieves the documents. Now that goes, so you look here, we can actually open up the retriever. What do we get? Here's our documents. Cool. Now it goes back to the assistant.

So back to this looping thing. It started with our assistant. It made the tool call. The tool ran. We got documents. Now we get them back. Now let's go back to our LLM. So now our LLM, this is pretty cool,right?

Here's the message history, like you were saying. Instructions, question, document retrieval. The documents that are retrieved areright here. And then now the LLM says, okay, I want to grade them. It calls the grader tool. And this is its reasoning, and this is its grade.

So anyway, you areright that as this goes through, you basically accumulate a message history. And the LLM will use the message history and the most recent instruction to reason about what tool to call next. That's exactly how it works.

I think I answered the question. Is there anything that isn't clear about that? So it is true. Like let's look at an example,right? So in this particular case,right, the LLM sees the retrieved documents from the tool. And then it makes a decision.

It says, okay, I have this tool response. It's retrieved documents. What should I do next? And it says, okay, well, why don't I go ahead and grade them? And it calls the grade tool. So it looks at the message history and reasons about what tool to call.

And that's exactly how these React-style agents work. And the whole issue is, that's kind of a noisy process. Like it can look at that whole trajectory. It can get confused. It can call the wrong tool. Then it's on the wrong track.

And that's exactly why these more open-ended tool-calling agents fail.

Let's say like in the second follow-up question to the agent, that

technically it should follow similar path to tool calling,right? But because sometimes the data is retrieved from the first

batch of tool calling, is there a good way to kind of structure your testing so that you can evaluate the performance on multiple operations or questions? Not just like a single question and get a lot of.

Okay. Right. So I think the question is, well, let's say there's a multi-turn conversation where the user can go ahead and ask a second question, of course. And that whole kind of message history will be propagated. Yes, that

is a common pattern. And that, let's see. I mean, what's the question on that, though? Like it could use context from its initial trajectory to answer the second question for sure. It'll probably look at that jointly when it's deciding what tool to call.

So for example, if it receives a question and in its message history it sees the context and needs to answer the question, the agent could probably decide, okay, I don't need to retrieve documents again. I have the documents and need to answer the question.

I'll answer that question directly. So it is true that in a multi-turn conversation, the agent can look at its message history to inform what to do next. That's definitely true. Here, I don't consider evaluation to multi-turn conversations, but that is a good topic, actually.

I don't quite have a tutorial for that yet, but I could think about putting that together. Yeah, that's a good point. I'll make a note of that, actually.

So multi-turn is a good one.

Cool.

Okay. Yep. Yep.

So testing is cool because they reduce the hallucinations. You can actually run within the app and so on and so forth. But have you experimented with tests used in more like single code and writing tests and then implementing that for an open-ended function tasks?

So the question is, I believe,

have I tested like the ability to kind of online auto-generate unit tests?

Yeah.

Yeah. Okay. So that's a big topic. So basically, the Alpha Codium paper

that Karpathy references here does that. I have not tested that because it does ramp up the complexity because then you're relying on, and I don't think, I mean, that would be aggressive for a production setting because basically be relying on an LLM to auto-generate you unit tests, testing against things that are auto-generated.

There's a lot of opportunity for error there. I think it's interesting, particularly in terms of like offline challenges like this, but in terms of like a production application, that feels pretty hard and risky. But it's an interesting theme.

The thing I've tested more on, I found to be very effective is super simple, crisp, lightweight, free, effectively unit tests. Like, again, the good use case I found was our, so LangChain, we have an internal RAG application called Chat LangChain.

And it indexes our documents and provides QA. It occasionally hallucinates the imports. And that's a really bad experience,right? If you take a code block, copy it from this app, and then this import doesn't exist. It's like, what the hell?

That's really annoying. So I incorporate a really simple check where I have a unit test that just, it does a function call where it extracts the imports from the code block from the answer. And it tests the imports in isolation.

If they don't exist, there's an error. I feed it back to LLM and say, look, this isn't a real import. Try again. And you can do other tricks like then context stuff, relevant documents or something like that. Anyway, you can handle that differently.

But that, just that little alpha significantly improved our performance. So I kind of like simple, lightweight, free unit tests. The idea of online generator unit test is interesting, but it opens up a lot more surface area for errors.

The idea is that you think about how unit tests look like. They oftentimes seem easier to write than the code. And you got to invent a structure. So we use that for like not reviewing the outputs of some things, but have the unit tests down it and then reduce the hallucination.

I'm wondering ifright, even going further and writing the test before you actually implement would lead to a better implementation.

Yeah. So the follow-up there was, let's see, trying to write the test before the app is implemented. I see. Yes. So they also do that. So basically, for each question, they do have, so the Alpha Codium work references existing unit tests for a given question as well as auto-generates.

So you areright that it would be interesting to have a bunch of pre-generated unit tests that you know are good for certain questions and to run them. Absolutely. Hard to do in a production setting with an open-ended input, but potentially very useful in, well, okay.

Even in a production setting, you could maybe have some battery of unit tests and based upon the question type, pull related unit tests that you know are going to be relevant. It's a good idea. Yeah, it's a good idea for sure.

So basically using some battery of dynamically chosen pre-existing unit tests based upon the question type or the documentation, whatever documentation they're asking about, that's a good idea.

For like larger projects, if you want to test for regressions, it's still very difficult to do it otherwise.

Yeah. He was saying, so for larger projects, you can also use that to test for regressions. Yes, that's definitely a good idea. And this paper does incorporate that idea as well as this auto-generated unit test thing, which is a little bit more aggressive.

Cool. Yep.

So how some of the self-correct, for instance, like LLM checking, LLM is often corrected quick. There are some other frameworks that kind of do this with different agents. Microsoft does that,right? Or they are the only one. Microsoft has one of those, but it's added complexity.

Yeah.

Added latency. I was wondering if you've seen those frameworks give

better results to justify the latency complexity or whether you specifically have seen kind of this one checker, one reflection of this.

Yep. So the question was, in some of these self-reflective applications, like let's say this one, the self-reflective RAG,right? We're doing a few different checks here. We're checking documents. We're checking hallucinations. We're checking answer quality. Instead of having some hard-coded single prompt to do that, can you have another agent, like kind of a checker or a grader agent that's a little bit more sophisticated?

And he mentioned a few frameworks. I have not played with them. I think it's interesting. I think it's one of these things where it's really good for kind of academic papers. It's very interesting. It's really good for, it could be good for offline testing in an online or production setting.

The latency and complexity is probably a bit much. I think in a production setting, it goes back to what I think this guy was referencing. You probably would want something that's extremely fast, lightweight. And I would not think about like a multi-agent system in a production setting doing any kind of grading.

This whole idea of LLM as LLM graders is kind of a new idea. So I think this idea of like more complex agent graders is interesting, but we're kind of in the, we're taking baby steps at this point, especially in a production setting.

So I'd probably shy away from that for now, particularly if you're thinking about production. But for something offline or experimentation, it's probably interesting.

Yeah.

What's the best practice that you've seen so far for performing tests or evaluations on those cycles? Because obviously, you can end up with many, many different sort of types of input and that sort of thing. Obviously, LangGraph like can provide some tracing there.

But what about, hey, here's an interesting test case. What are kind of best practices you've seen for actually performing those full-turn evaluations?

Yeah, exactly. So actually, the notebook shared here go into it a little bit. So the way I like to do it is, and I can even, why don't I just show you one of the notebooks? So basically, so at the bottom, I have kind of all the different evaluations.

So this goes back to a question that someone mentioned down here as well.

Which one is this?

So this is, so there's a collab and then there's a notebook. And they are both the same. So they have the same evaluation section. This is rag-agent-testing.ipnb. And there's also a collab, which is, it's the same notebook basically.

It's in the slides. Just make sure you got those links. But to your question, so the way I did it, and you could be very flexible with this, is basically I define the expected trajectory through the nodes in my graph.

And in this particular simple case, the trajectories that I expect are basically retrieval, grading, web search, generate, or retrieval, grade, generate. Those are the two expected trajectories that I want to take. Now, in this case, I don't do cycles.

If you did cycles, you could just incorporate more steps here to say, okay, here's kind of the expected number of steps I want or expect to see through that cycle. And I think someone mentioned here, if you have a really open-ended, challenging task, then it may be hard to enumerate exactly.

But for a lot of like more production setting applications where these are not extremely long running, you can enumerate, here's the steps I expect to take through the cycle. And the way I do the evaluator, it's as simple as this.

There's not much code. It's all in the notebooks. But basically, I compare the tool calls that the thing did to these trajectories. That's it. Super simple. So that's how I would think about it. I keep it really simple.

I would basically try to enumerate, here's the steps through the difference, the cycle I want it to take. And yeah, go ahead.

Just to make sure I understand.

Right.

Basically, you're using, so in the LangGraph concept.

Yep.

Basically using the nodes to say, hey, I expect it to be calling these tools along the way.

That's it.

And that's how I'm performing my evaluation.

That's it. That's it. So it's actually really simple. And this is actually a good point. In the LangGraph case, for this custom agent, you'll see, all I do is, for every node, and we can get into the code if we want.

Maybe people have already explored. So, well, I'll just answer the question directly. Then we can back up and go through all the code if we want. But basically, each node in my graph, I just append this step name,right?

So like retrieve, I just say, retrieve documents. This is my generate node. I just append this thing, generate answer. And I return that in my state as steps. So then I have this record of the steps I took in my graph.

That's it. And I can go ahead and fetch that at eval time and compare it to what I expect. That's all you're doing. So that's how it would work with like the custom LangGraph thing. And now with the React agent, actually, it's a little easier because the React agent uses a message history.

So I can just go to my message history. And that's exactly what I show here. You can strip out, I guess I'd do it up above. But basically, I have this function in the notebook. Let's see, where is it?

Yeah. It's find tool calls, React. So it's this little function that will basically look at my message history and strip out all the tool calls. So

yeah, it's a nice little

thing. With a React agent, it's really easy to get the tool calls out. With LangGraph, I just log them at every node. And then at eval time, I just can extract them.

Yep.

So can you elaborate on the open-turn scenario when there's context switching? Where is it actually the cognitive ability for it to say, I don't have the answer in a multi-term? It's going retrieve. Where's that cognitive ability? Is it in the query router or is it in the graph?

Yeah. So the question was, with an agent in a multi-turn conversation setting, how does it know whether or not it has the answer to a given question?

So it becomes to know where to go.

Where to go. Exactly.

It may be multiple vectors.

Yeah. That'sright. So there's a couple different ways to kind of break this down. So with these agents, there's a few levels of instruction you give it. First, you give it an overall agent prompt. So if I look at the notebook here, we can go look at the React agent as an example of this.

So the React agent is definedright here. This is kind of like the planning step. Here's my like naive prompt. Okay. So your helpful assistant, this task of answering tasks, use the provided vector store to retrieve documents, grade them, and go on.

Now, let's take the case that's more complicated. Let's say I had two vector stores. So one thing I can do is I can explicitly put in the agent prompt, you have two vector stores, A, B. A has this, B has this.

And then you're implicitly having the LLM, giving it the ability to kind of reason out which one to use. Now, this is where the second piece comes in. You also bind it to a set of tools. This is really where the decision-making comes in.

When you create this tool, so here's retrieve documents,right? This tool description is captured by the agent. So the agent knows what's in this tool. And this is really where that decision to use this retriever tool versus another one would be made.

It'd be a combination of the prompt you give to the agent and/or the tool description. So if you had two vector stores, you could basically say, like retrieve documents one. This vector store contains information about X. Another one contains information about Y.

Then the agent is deciding what tool to call based on that description and maybe based on its overall prompt. But to your point, it's not easy. So actually, with the custom agent, this is with a React-style agent. With the LangGraph custom agent, you can do it a little bit differently, where I actually don't have it in this notebook, but I have other cases where you actually can build a router node.

And I mean, I'll show you actually. So this particular notebook, so this one, this self-red, it's in the slides. If you open this up, we did this with Llama folks. So this is actually a trick I really like.

If you go to LangGraph RAG agent local here,

let's see, Wi-Fi is a little slow. What I define here is a very specific router at the start of my agent that decides where to send the query. And this is something I really like to do because, like we saw with the React thing, it has to kind of decide theright tool to use, which can be kind of noisy, versusright here.

So here's my router,right? This is reliable enough to run locally. And what I do here is I run this at the start of my graph. And I have the agent, or yeah, the agent explicitly take the question and decide what to use.

And based on what decision it makes, I can send it to either web search or, in this case, a vector store. So to answer your question, if I pull all the way back, I personally like to do explicit routing as a node at the start of my graph because it's pretty clean.

And you can see in the flow of this overall graph, this router runs first. And it looks at my question. It sends it to one of two places. And this can be more complex. You can send it to one of n places, but I see one of two here.

This is with like a custom LangGraph agent. This is what I like to do. If you're using a React agent, it is then using a combination of the tool definitions and the overall agent prompt to decide what tool to call.

But you can see it's more squishy because it has to call theright tool. And as opposed to giving it a router and saying, always run this router first, it has to kind of make theright decision as to what tool to call based on the context question, which is harder.

And that gets back to our overall story about these kind of LangGraph explicitly defined agents are more reliable because you can lay out this routing stepright up front and have it always execute that step before going forward.

If you can change this routing thing, it's not looking back at its history. Like it's deterministic,right? So here's like, hey, I actually have the answer based on previous. So this seems to be a very deterministic thing. Everything's going to route and go retrieve documents.

But actually in the graph, you would say, do I have a history of the answer? Can I answer it? That's the thing that creates challenges is that you got to not only look at the retrieve, but you also have to look at the cognitive history of can I answer this question.

So I'm just curious.

Yep.

How do you orchestrate that?

Okay, got it. So the question was, how do I incorporate the idea of routing with history? So here's what you would do. It's actually kind of, it should be pretty straightforward. You can define this router node in your graph.

And that router node, I'll actually go down to it. So basically, here's my graph and here's the

route question node,right?

Yeah. Actually, in this particular case, it's an edge. Don't worry about those details. Basically, what you could do is you could have a node that takes in the state. Now, that state could include your history. So what you could do is in that router prompt, you could really easily

here include another placeholder variable for like your message history or something. And what you could say then is make a decision about where to go next based upon the question and based upon something in a history. And so you actually would plumb in your message history here and use it to jointly decide what to do next.

So that's actually really easily handled in LangGraph using a node. And you can reference the history that you can pass into that node as state.

Cool. Yep.

Is there some notion of state?

The notion of state. Yeah. Let's talk about state in a little bit more detail. So let's actually go to the notebooks that we're working with here that I shared.

So here's this RAG agent testing notebook. So if you go down to the custom LangGraph agent, the way you do it is, we'll find the state. Yeah. So here's what I call graph state. So the graph state is basically something that lives across the lifetime of my graph.

I typically like to do something as simple as just a dictionary. So basically, this is a RAG graph. And here, I'm basically going to define a number of attributes in my state that are relevant to what I want to do with RAG.

A question, my answer generated, whether or not to run search, some documents, my step list. And basically, the idea here is that I define my state up front. And then at every node, I exactly accept state as the input.

And then I operate on it in some way and write back out to state. So basically, what's happening is I define state generally up front as like a dictionary or something like that. It's a placeholder for things I want to modify throughout my graph, throughout my agent.

And every node just takes in state, does something, and writes back out to state. That's really it. So basically, it's a way you can think of it as a really simple mechanism to persist information across the lifetime of my agent.

And for this RAG agent, it's things that are really intuitive for RAG. It's like question, it's documents. And so let's take an example. Like, okay, here's a fun one. So my great documents node,right? What I'm doing here is I'm taking in state.

And from my state, it's just a dictionary. So I'm extracting my question, I'm extracting my documents, and I'm appending a new step because I'm notifying, hey, here's my new step. And basically, I'm doing some operation. I'm iterating through my documents.

I'm grading each one. If the grade's yes, I keep it. If the grade is no, so yes, no means like is it relevant or not, basically. So if yes, it's relevant, I keep it. I put it in the filter docs list.

If it's not, I set the search flag to yes, which means I'm going to run web search because I want to supplement. I have some docs that are irrelevant. And I write back to state at the end. My filter docs, my question, search, and steps.

That's it. So state's a really convenient way to just pass information across my agent. And I like using a dictionary, just like nice and clean to manage, as opposed to a message history, which is like a little more confusing.

Like in any node, if you use a message history, it's like a stack of messages. So if you want the question, you have to like, it's like usually the first message, you have to just index it. It's just kind of ugly.

I like using a dictionary. It's like I just get the question out as a key in my dictionary.

Cool.

Let's see. We're about an hour in. I can also

let people just hack and walk around, talk to people, stay up here, whatever's best. And you can keep asking questions if you want to. I think people are just working, doing their own thing now anyway.

So I might ask one question just for fun. Is anyone interested in local agents? We didn't talk about that too much. It's a big theme. Yeah. So I shared a notebook for that. And by default, I am using Llama 3.

You can absolutely test other models. So this is set up to test just Llama 3 with Ollama.

Try other things. I have an M2, 32 gig, so 8B runs fine for me. If you're something bigger, you could actually bump that up a little bit. So that can be a nice idea.

I've got 70B on my demo one and I don't recommend it.

Yeah, exactly. 70B is unfortunate that, yeah, I actually want a bigger machine so I can run that. Because I found for tool calling, 8B is really at the edge of reliability.

So that's actually why you really can't run the React agent locally with an 8 billion model reliably. You can run the LangGraph agent very reliably because it doesn't actually need tool calling. It only needs structured outputs. You'll see in the notebook.

So that's actually a really nice thing.

But React agent won't run locally, reliably at least.

Question?

Yep.

Regarding RAG, what is your typical tool? What do you chunk the documents?

You say chunk documents?

Yeah.

Okay. Yeah. So this question always comes up. With RAG, what is the typical chunk size? Yeah. If you ask 10 people, you'll get 10 answers. It's notoriously ad hoc. You know what? To be honest,

I did kind of a talk on the future of RAG with long context models. I'm kind of a fan of trying to keep chunk sizes as large as possible. Actually, I think this is a whole tangent, but I actually think one of the nicest tricks, let me see if I have a good visual for it.

Basically, let me try to find something here.

This is a whole separate talk. But basically, I think, yeah, this one, RAG in long context. So this is a whole different thing. But

yeah, this idea. So I think for RAG, one of the ideas I like the most is

decoupling. I'll explain this. I'll just say it and then I'll explain it. Decoupling what you actually index for retrieval from what you pass the LLM. Because you have this weird tension,right? Smaller semantic-related chunks are good for retrieval relative to a question,right?

But LLMs can process huge amounts of context at this point, up to, say, a million tokens. So historically, what we would do is you would chunk really small, like very, very, try to get it as tight as possible.

All these tricks, semantic chunking, a lot of things to really compress down to just compress and group semantic-related chunks of context,right? But the problem is then you're passing to the LLM very narrow chunks of information, which has problems in recall.

So basically, it's more likely that you'll miss context necessary. So it's good for retrieval, but bad for answer generation. So a nice trick is for retrieval, use some chunking strategy, whatever you want, like make it small. But you can actually use like a doc store to store the full document.

And what you can do is retrieve based on small chunks, but then reference the full document and pass the full document to the LLM for actual generation time. So you decouple the problem of like, are you sure you're passing sufficient context to the LLM itself?

Now, I also understand if you have massive documents, it's going to be wasteful in terms of tokens to pass full documents through. But there's some preto optimum here where I think indebt being too strict with your indexing approach doesn't make sense anymore, given that you can process very large context in your LLM.

So you want to avoid being overly restrictive with your chunk size. So this is a nice way to get around it, basically, to summarize. You can choose a chunking strategy, whatever one you want. But I like this idea of referencing full documents and then basically passing full documents to the LLM for the answer itself.

It gets around that problem of an overly aggressive chunking strategy that misses context needed to actually answer your question. And I think with long context LLMs getting cheaper and cheaper, this is starting to make more sense. I'm trying to think.

I had a whole kind of like slide on, yeah, this one is kind of like balancing system complexity and latency. So it's kind of like on the left is like maybe the historical view. You need the exact relevant chunk.

You can get really complex chunking schemes, a lot of overengineering, lower recall. Like if you're passing 100 token chunks to your LLM for the final answer, you might miss the exact part of the document that's necessary. Very sensitive to chunk size, K, all these weird parameters,right?

On the other extreme, just throw in everything, throw everything into context. Google, actually, I think this week will probably announce some interesting stuff with context caching. Seems really cool. Maybe that actually could be a really good option for this.

But higher latency, higher token usage, can't audit retrieval, security authentication, like if you're passing 10 million tokens of context in for your answer generation. So something in the middle is what I'm advocating for. And I think

this kind of document-level decoupling where basically you index and reference full documents and pass full documents to your LLM is like a nice trick. We've seen a lot of people use this pretty effectively. So yeah.

Yeah. Kind of a thinking back without the back. Is there any way or any guarantees or tricks that you have to make sure that you're not actually missing any of the elements of the context throughout the entire window?

Like how do you measure that you're actually using all of the context as opposed to just beginning or end or anything like that?

Okay. That's interesting. So the question was, how can you evaluate the amount of context you are using? Okay.

Yeah. Like does it definitely use the whole state? I don't want to miss anything.

Okay. But I guess, so in a RAG context, you have a question, you have an answer. So you have a question, you have an answer, and you have some retrieved documents. So you can evaluate the question relative to your document.

That's one way to get at this. Like how much of the document is relevant to your question? So that's maybe one approach. And actually, the notebooks show a few prompts to kind of get at that. And actually, a good way to think about that is you can think about document precision and document recall.

And this is a little confusing maybe, so I should explain it. Basically, document recall is, does the document contain the answer to my question anywhere? So let's say you have a 100-page document. On page 55 is my answer.

Recall is one. It's in there. Precision is the other side of that coin, which is, does it contain information not relevant to my question? In that particular case, huge amount of irrelevant information. So recall would be one. Precision would be very low.

So that's one thing you can do. You can actually look at your retrieved docs, measure precision and recall. That's like one thing I think I would probably like to do there. And that's probably the best way to get at this question of like, how much of my documents am I actually using?

Now, with this approach, your recall would be high, your precision would be kind of low, and you would say, I don't care. It's fine. If I have a model that's super cheap to use a large number of tokens, I'm okay.

Maybe I'll frame it another way. I care more about recall than precision. I want to make sure I always answer the question. If I pass a little bit more context than necessary, I'm okay with that. Versus if you're a precision-gated system, then you would say, okay, I'm going to miss the answer sometimes, so I'm okay with that because I never want to pass more context than necessary.

I think a lot of people are moving towards higher recall because these LLMs are getting cheaper and they can process more context. So that's kind of how I might think about it. And I think this approach is a nice idea of indexing full documents or indexing chunks, but then referencing full documents, passing full documents to your LLM to actually generate answers.

So yep.

I was wondering, it seems like there's a couple of options for where exactly you inject the context into the conversation history. Like just for one example, you could stick something into the system prompt and say, hey, this is your knowledge that you know.

And then the user just asks a question and there's no prompting from the user. The other thing is you could completely modify what the quote-unquote user is, as in saying, answer this question with this context and then put it in there.

And then like a further idea on top of that, you ask a question and then you ask a follow-up question. Maybe if you say, please elaborate, there's no need to actually retrieve anything, but you're just having a chat.

Like you chat or you ask something else to be able to retrieve. Do you have any observations as far as things that work well or have worked well for continually appending things into the system prompt or like adding another user?

Like where do you stick that context? How much of it do you keep for history?

Yep. Yeah. Okay. So this is a great question. The question was related to kind of like agentic RAG and where to actually put documents. So let's walk through the cases. So case one is I have a fixed context for every question you're just going to ask.

Let's say I have like a RAG bot against like one particular document and that document is always going to be referenced. So you make a very good point. In that case, what I would do is, let's go back to our agent example.

You can put that in the system prompt itself, like you said. And actually, I'll mention something else about this. It'd be likeright here. So here's your system prompt for your agent. You just plumb that whole document in. You say, every question you're going to reference this document, you don't need retriever system, you're done.

That's a very nice case. No to retrieval complexity. You just context stuff your whole document. The thing that Google is going to announce this week, I believe this context caching seems really interesting for this. Because basically what they're saying is, if I get itright, I think Logan will speak to this, but I think they have a context window of a million to 10 million tokens.

It's huge,right? So basically, you can take a large set of documents and stuff them into this model effectively. And they house them for you somehow. I think you've some minor data storage fee. But then for every inference call, they don't charge you for all those tokens that are cached, which is pretty nice.

So basically, here's the use case. To your point exactly, I have some set of documentation. It's 10 million tokens. That's a lot of pages. Hundreds of pages. I have it cached with the model. And every time that user asks a question, I don't get charged 10 millions of tokens to process the answer.

They just cache it for me. Really nice idea. So that's your point of like the first thing. That's like not quite in the system prompt. It's in like the cache, but that's the same idea. So you have cached or system prompt fixed context.

That's like case one. So case two is you want to dynamically retrieve. So you can't stuff your context. Maybe you have a few different vector stores like we were talking about here with routing. In that case, yeah, you have to use an index of some sort, maybe a router to choose which index to retrieve from.

So that's kind of case two.

And I'm trying to remember the, oh, okay. Yeah. So like in that particular case, for follow-up questions, how do I kind of control whether or not I re-retrieve or not? So that's the nice thing about either one of these agents.

It has some state. So the state lives across the lifetime of the agent. So basically, the agent, and this actually gets exactly to what the other question was on. Let's say I built my agent with, I'll show youright here.

So let's say I have a router node at the start of my agent. Okay? And that router has access to state. What I can do is then, given a question, this could be, let's say it's a multi-turn thing.

This is a second question in my conversation. I have an appended state from the rest of my discussion here. The agent knows it returned an answer. So basically, when a new question comes in, you could pass like the entire state back to that router.

And the router could know, okay, here's the docs I've already retrieved. And it can basically then decide to answer directly because I already have the answer to the question. So that's a long way of saying you can use state either message history or explicitly defined in your LangGraph agent to preserve docs that you've retrieved already and then to just use them to answer the question without re-retrieving.

So that's kind of what these RAG agents can be really good at. That was kind of like storing that in short-term memory and reasoning about, hey, do I need to re-retrieve or not? So that's exactly the intuition behind why these RAG agents can be pretty nice.

Yeah.

Yeah. Is there a problem that it seems like there's this idea like RAG is a hack,right? Like we just kind of took some model that's designed to do chat and then we found that it's stuff.

Yeah.

And it seems like there's a little bit of a back and forth going on where the model is designed in this general purpose way where the coders hack something together. And now you go back to the people training the models and they say, oh, well, we can actually fine-tune the model to specifically be good at this.

And I'm curious, it almost seems like you're going to propose all these ideas and then they're just going to converge on whatever the people fine-tuning those models are doing because that's what's going to work well,right? Like can we stick it?

Like do you see it in the industry moving in a direction of like stuffing things into the user context versus the system prompt? And where are like the indications of which one's just going to work better simply because that's what it seems like fine-tuning?

Yeah. Yeah, exactly. Okay. This is a really good discussion and a whole debate. So the highest level framing of this is how do you want to, how do you want your model to learn? So one option is you can modify the weight of the model itself with something like fine-tuning.

Another is you can use what we call like in-context learning through your prompt. So RAG is like a form of in-context learning. I'm basically giving it some documents. It's reasoning about those documents, producing answers. Not touching the weights in my model.

Fine-tuning would be taking the knowledge I want to run RAG on, fine-tuning your model and updating the weight so that it has that knowledge. There's a lot of debates on this. And actually, I think Hamel has a whole course on fine-tuning.

Oh, yeah.

I just want to clarify. I don't mean fine-tuning information in. I just mean fine-tuning so that you have the idea. Like if you could fine-tune it so that it knows, oh, I should always retrieve this information from my system prompt and use that when answering questions.

Or you could fine-tune the model to really use whatever extra context the user is providing. So just as a simple example of that, if you have the system prompt at the very beginning of the chat, you would be training the model to focus more on the information at the beginning of the chat.

Whereas if you were fine-tuning the model to be good at using the context that the user provides, then it's focusing more of its attention on the end of the chat. So it's not getting the information in. It's just how do I use these hacks and tools?

I got it. Okay. I'll repeat that. So the clarification was thinking about using fine-tuning more to kind of

govern the behavior of the agent rather than to kind of encode facts, which is a good, it's a very good clarification because I was going to say using fine-tuning to encode facts, I think a lot of kind of literature has pointed to that being a bad idea for a lot of reasons.

It's costly. You have to continually fine-tune as facts change. So let's dispatch that. I think that's kind of not a great idea. But you make a very interesting point about fine-tuning to govern behavior. Now, there's a paper called RAFT that came out kind of recently.

And actually, as far as my understanding is, I haven't played with it at length, it's fine-tuning to kind of do what our notebooks show today of this kind of like look at the documents that are retrieved, reason if they're relevant, and then don't use them if they're not automatically filtered.

It's doing exactly what we're doing in this LangGraph thing, but it's kind of achieving that same outcome through process of fine-tuning. So that's a very good insight. You'reright. It seems promising to have these like kind of fine-tuned RAG agents, so to speak.

Or it wouldn't be an agent. It'd be an LLM fine-tuned for RAG that incorporates this kind of logical reasoning. Or what you're saying, like maybe some kind of reasoning about if you have a multi-turn conversation, like avoid recency bias, whatever it is, that seems like a very good trend and interesting.

The challenge is a little bit, if you're fine-tuning yourself, fine-tuning is hard and somewhat advanced and all that. Alternatively, if it's like a very niche use case, like the RAFT system, basically it fine-tunes to kind of do this.

As models get changed or update all the time, you kind of need to keep your fine-tuned model up to date. If you see what I'm saying. So I think I'm still a little bit queasy about using fine-tuning even in that context because of the challenge of keeping it kind of up to date with the state of the art.

But it's interesting. I think the RAFT paper is a good reference in this direction and does exactly what we do in this workshop, but I believe it fine-tunes this into the model or attempts to, which is a very intuitive thing to think about.

Basically, let the model reflect automatically on retrieved documents and like automatically filter them for you. Seems like it should be able to do that. Seems like a good idea. But my hesitation would still be like, well, if I want to switch my models, I need to like refine-tune.

I want to use LLaMA 3. I have to fine-tune LLaMA 3 on this task. I can't use ProPriet. Maybe I can fine-tune GPT-4.0. Might have fine-tuning now. I'm not even sure. I could fine-tune GPT. So again, if you have fine-tuning myself, that's hard.

It still feels a little bit like I'd rather just sit up a simple like orchestrated agent that does it rather than rely on fine-tuning is my sense. Yeah.

I guess I'm not really imagining the person doing or implementing RAG doing that fine-tuning. I kind of think of.

Yeah. The provider.

An analogy, yeah, like Tesla's data engine that Android Pro passed away years ago. He's adding into that long tail. Like OpenAI is going to have some kind of data engine. They started with answer this question usefully. But it's going to start to become that data engine.

There's sort of like long tail if you want to call it that analogy. The point could be answer this with this context. Answer this with this tool. And then as we just pack things together, whoever's training the model with a separate entity who just fine-tunes to that.

But there's got to be some kind of like back and forth. Definitely, there isn't quote unquote like this is theright way to do it, but if that's the way that it's fine-tuned, then it does become theright way.

Yeah. Okay. So that's a very good point. I think this is also a very big debate. So OpenAI just did an acquisition this week on a retrieval company. I forget the name. Rocket, I believe. So I think they are moving more in the direction of retrieval.

I could absolutely see them offering an API that potentially does retrieval for you and incorporates some of these ideas for you. So how much of this does get pushed behind APIs and they take care of whatever is necessary behind the scenes for you, that could absolutely be the case.

I would not be surprised if at all they move in that direction. And I think there's always, you know, it's an interesting trade-off. Like how much are you willing to kind of

abstract behind an API versus not? I think there's always a lot of companies and developers that want to kind of control everything themselves and build it themselves, have full transparency, and others that don't. And so, you know, it's an interesting question.

But of course, for certain functionalities, multimodality, very few people are going to stand that up themselves. You kind of let that live behind an API. So what do you allow to live behind an API or not? My only concern is I think for some of these kind of things, they could be very domain-specific.

Like what you consider relevant or not could be very relevant to you and your application. You kind of want to be able to control that. That's the only thing I could imagine. It could be kind of hard to abstract that all behind an API, which I think is maybe why OpenAI hasn't done too much in retrieval yet.

It's just a hard beast. I know they've been trying for a while. I don't know. It's a great debate, though. Yeah, we can discuss more if, yeah, it's a good topic for sure. Yep.

You mentioned about long context windows LLMs.

Oh, yeah.

So there's a problem of like lost in the middle. For example, the precision, like if, for example, in the context, let's say 95% is relevant information and 5% is somewhere in the middle. And somehow LLMs kind of message it at that point.

Like do you have any suggestions about that?

Yeah. Yeah. So the question is on lost in the middle of long context. I actually did a whole study on this with Greg Cameron. Yeah, it's a really interesting topic. So the insight was basically that long context LLMs tend to have lower recall or like factual recall for things in the middle of the context.

Okay. So that was one observation. At least that's what their paper reported. So I actually looked at this with Greg and we did something a little bit even harder. We actually tested for multiple-factor retrieval. So we tested, can you retrieve one, three, or 10 different facts from the context?

And this was using GPT-4, GPT-4 Turbo, single turn.

And on the X-axis, you can see the fraction of the needles that it basically can get. And then on the Y is the number of needles. So basically one needle, three needles, 10 needles. Green versus red is basically just retrieving versus retrieving and reasoning.

So it's like reasoning is a little bit harder than just retrieving. These needles were actually pizza ingredients. So basically the background was, you know, this was 120,000 tokens of Paul Graham essays and three secret pizza ingredients or however many, one, three, or 10, but I injected in that context.

And I basically asked the LLM, what's the ingredients needed to build the secret pizza? So it'd have to find them in there. And basically, as you ramp up the number of needles, go from one to 10, it gets worse.

So with 10, it's actually retrieval itself is only like 60%. So then I looked at, okay, well, where is it failing? And that's what I look at here in this heat map. So basically this is telling you like how long the context is.

So 1,000 tokens all the way up to 1,220,000. And then here's like the needle placement, so one to 10. So this red means you couldn't retrieve it. And what I found is actually it doesn't get them towards the start of the document.

So the retrieval gets worse if the needle is at the front. So it's like this. I read a book. I asked you a question about the first chapter. You forgot because I read that a month ago or something.

Same idea. And actually, I put this on Twitter and then someone said, oh, yeah, it's probably recency bias. And that's a good point that basically the most informative tokens in next token prediction are often the more recent ones.

So, you know, basically these LLMs learn a bias to tend to recent tokens and that's not good for RAG. So that is all to say I'm a little wary about long context retrieval. I wouldn't quite trust basically high-quality RAG across a million tokens of context.

You can see, look, if it's 1,000 tokens, no problem. If it's 120,000 tokens of context, you know, it depends a lot on where those facts are. If they're towards the start, you actually can have much lower recall. And so that's a real risk, which is kind of why it goes back to this whole thing of

like I don't really buy just stuffing everything into context that farright side. I think there's too many issues with bad recall, recency bias, like you said. And so I think until we have very, and by the way, I also don't really trust, you know, when they show those needle on the Haystack charts, it's like perfect.

I don't trust any of that. I did my own study. I found there's like a lot of errors. And I think it depends a lot on a couple of different things. One, how many needles? So in this case, you see with one, it's okay.

With 10, it's really bad,right? So how many needles? And then I saw an interesting study saying that like the difference in the needles relative to your context makes it easier. So like in these studies, it's like pizza ingredients in Paul Graham essays is really different.

But if it's just like related slightly, it's actually harder still. So that is to say I don't really trust the needle and Haystack studies. I don't particularly trust stuffing, you know, passing a million tokens of context and counting that and counting on that to just work effectively.

I'd be very wary about that. That's kind of my thing. But, you know, in these studies, look, 1,000 tokens of context already, you know, that's, that is, if you're stuffing 1,000 tokens, eh, I mean, that's actually still pretty small.

So yeah, I'd just be wary about retrieval from very large contexts. Yep.

What about the number of tools? Is it better to have small unit tools versus complex ones?

Okay. Yeah, that's a great question. With agents, the number of tools, this is a really big issue I hear mentioned a lot. So if you recall, if you go back to the agent stuff,

so you're basically binding some set of tools to your LLM,right? And that's what we show here,right?

I've seen a lot of issues with a large number of tools. So I don't know exactly know what the exact cutoff is, but this is one of the big problems with open-ended tool calling agents is if I am basically selecting from 20 different tools, I actually, maybe the Berkeley Leaderboard has data on this.

If someone knows, feel free to mention it. But

reliability of tool calling, even with a small number of tools, like on the order of five is already challenging. If you're talking about hundreds of tools or dozens of tools, I don't think there's really, yeah, I think it's quite challenging, which is why I've seen more success in, A, not using these open-ended style tool calling agents, laying it out more explicitly as a LangGraph where the tool calls live inside nodes and you're not relying on your agent to pick from like 20 different tools.

So you can lay out more of like a control flow where you route to different tool nodes based upon the logic. So that's kind of one thing I've seen. And the other thing I've seen is maybe multi-agent type things where you have different agents with subtasks with each having like maybe a small number of tools.

But basically what I've seen is it seems to be that, okay, maybe it's two things. Selection from a large number of tools is definitely challenging. One of the most interesting things I saw is something like you can use something like RAG where basically you can take a description of your tools, create a natural language description, embed it, and then use basic like semantic similarity search, your query versus the embedded summaries to select using semantics.

That's actually not a bad idea. I would actually use that more than I would trust an LLM to just like do the tool selection from a list of a hundred. That's not going to work. So actually, I think that like RAG for tool selection is a cool idea.

I was going to do a little like test that out and do a little tutorial. So actually, maybe I'll just make a note of that. That's a great, a great question.

To do RAG for many tools.

That brings me to a bit of a follow-up question because I think a similar approach is often used for data querying.

Right.

Like semantic similarity. Were you thinking about something like that for the tools as well?

Yeah. Well, I think, I think using semantic similarity for tool selections is a good idea. Definitely.

What about data querying?

What do you mean by data querying though?

SQL is a natural language. If you plug in some tables as one of your tools or source of information.

Right.

Let me make sure I understand. I think the way I would think about it is, so you know how in your, in the notebook, like in the code for all of our tools,right, you have this little tool description,right?

Like retrieve documents, grade them, run web search. I would actually write verbose descriptions for all my tools and then index those descriptions,right? Or embed them. And then I would do, and I would probably create like very, very verbose high-quality summaries of what the tool actually does and then do semantic similarity search against those summaries.

I think that could actually, I haven't done that yet, but I think that could work really well

because it's a very tall task to ask an LLM to differentiate between like 20 different tools. Whereas you could do something like semantic similarity, that would actually probably be very effective.

Yep.

Can we go back to the multi-agent kind of problem where it's like it seems like it's, you know, pinned to a single tool and you're kind of saying like you can have like an agent with five tools and one of its tools is called an agent tool or tool.

Right.

Have you done any like experiments with that? Like in terms of, or knowing it, like any research you wrote to do like to figure out that problem?

Yeah.

We talked earlier about like agents like finding a good idea for example or two agents and one.

Yeah. Yeah. So the question was about multi-agent context when you want to orchestrate a large number of tools having sub-agents with specializations that manage some small set of tools each and how do you kind of move between them.

So if you look at our LangGraph repo, we do have a subdirectory. It's under, it's under LangGraph examples multi-agent. We have a few different notebooks that have multi-agent style kind of layouts, which I would encourage you to look at.

I haven't personally done too much work on it. It seems promising, but I haven't played with it.

Multi-agent in general, and these all reference papers. You can also look at the papers, but multi-agent in general for production setting feels quite aggressive. Although that said, as far as I understand, I remember looking at the code a while ago, Devin and some of the software agents do use like multi-agent style setups.

So maybe have a look at the Devin repo or there's open Devin. Have a look at these notebooks. Those could all be useful if you want to learn more about multi-agent. Yep.

So I'm wondering like, and I stepped out for a minute.

Yeah, sure.

I already asked this, but I'm wondering, like we've been talking a lot about kind of the variability and unpredictable nature of having the LLM decide which tool to call.

Right.

So given that, I'm wondering how do you think about when it makes sense to wrap RAG inside of an agent versus just making a chain on its own?

That's a classic question. Yeah. So the question was, when do I use a chain versus an agent? So that's very good. So we kind of touched on it a little bit kind of here. So I think the intuition behind where and why an agent can make sense is simply that sometimes you want your application control flow to be variable.

And if you want some flexibility within your application, an agent is a nice idea. And so all this self-corrective type stuff we're talking about, the corrective RAG thing, those are all kind of agentic flows where the control flow depends upon the grading of the documents.

And so

you know, historically people have largely been building chains and chains are very reliable and they're easy to ship and all that. I think with things like LangGraph and of course I work at LangChain, so I'll speak my book about LangGraph, but I've really used it quite a bit and I found it to be very reliable.

And we are seeing a lot of people starting to deploy with it because you can actually ship and deploy a reliable agent with LangGraph. And so I think a blocker to the ability to kind of ship agents has been reliability.

And I think we're, I would actually encourage you to play with the notebooks and look at LangGraph because it does allow you to have that kind of reliability that would be necessary to ship something in production. And we do have customers that have LangGraph in production, whereas a React agent in production is not recommended.

But let's imagine my options aren't.

Yeah.

LangGraph agent or React agent.

Yep.

Rather just traditional RAG versus LangGraph agent.

100%.

In that case, would it only make sense to go with a LangGraph if I wanted something like deep retries or ranking of documents or something of that nature?

So, okay. So I think, yeah, so the different, why would you ever want kind of like an agent, be it LangGraph, React, or otherwise versus not? And again, I think it goes back to do you want your application to have any kind of adaptability?

So, okay, here's one we can talk about. Routing. I have three different vector stores. I want to be able to route between them. That is kind of a quote unquote agentic use case because the control flow depends on the question.

So that's one. You might want routing. You may want self-correction. So that's kind of what we talked about here a whole bunch with the corrective RAG stuff. So you want routing, you want self-correction. I mean, those are two obvious ones in the context of RAG itself.

I mean, that's one thing I've often found a problem with the RAG systems is the routing thing is a real issue. Like you want your system to be flexible enough to deal with questions that are out of domain for your vector store.

And you need some, you need some kind of dynamism in your application to handle that. So looking at the question saying, okay, just answer this directly. Don't, don't use a vector store. So those are like the most popular ones.

Self-correction or routing.

Great. Thank you.

Yeah. Yep.

I'm wondering if there's anything to be said about building evaluation data sets, like question answer pairs are so domain specific.

Yeah.

I'm wondering if there are like general best practices, mental models, like things to think about when sitting down to build an evaluation data set.

Yeah. Yeah. Okay. So the question was about kind of building eval data sets. Okay. That's a great question. It's often a very, very challenging part of app development. So if you're, if you have a RAG application that's domain specific, then oftentimes you have some set of canonical question answer pairs you care about.

You know, it's hard to define like, you know, very, very general rules for that. I think it depends on your application. I think there's kind of this, this hurdle. Any evaluation is better than no evaluation. So small scale eval sets that you can use and just work are already better than not doing any evaluation.

So, I mean, for this particular case, I just looked at the document. Now, maybe, maybe I'll back up and answer. So one thing I've seen, and I've done this a little bit, is you can use LLM assisted QA generation.

So here's one thing you can do. And I've done this a little bit with LangChain docs. I can build a prompt that says, given this document, produce three high-quality question answer pairs from it,right? And I can just basically load my documents and pass them into that LLM.

I use a high capacity model like Sonnet or 4.0 and have it generate QA pairs for me and then audit them. That's a nice trick. I've used that. It actually kind of works. Now you have to be careful with it.

You only would pass it like usually one document at a time to keep it really like, you know, restricted. And you audit them, but actually that's a nice way to bootstrap your eval sets. That's like idea one. And this, that gets into the whole idea of synthetic data sets.

But if you're building, you know, domain specific synthetic QA pair data sets, that's a nice trick. So basically using LLM to help bootstrap. I think that's one idea that can help a lot.

Yeah. And otherwise, I think that basically trying to stand up a small evaluation set, for example, for RAG is even this case, five questions, but you can already see I can get some nice insights. It's very simple to set these up.

I have my little experiments all over here. And again, it's only five questions, but it gives me some immediate insights about the reliability of React versus, versus LangGraph agent. So keep it small, potentially use synthetic data, start with something, and then like kind of build it out over time.

Now, a whole other thing here is we didn't talk about this too much, but if you have an app in production then, the way this whole thing kind of comes together is you can actually have different types of evaluators that run on your app online.

We call those online evaluators. Okay. So this is with our internal app. And this gets back to the question I think he mentioned of you can have a bunch of evaluators for RAG that don't require a reference. So like, I don't show it here, but, but basically I can look at like document retrieval quality.

I can look at my answer relevance or hallucinations. I can run that online. I can flag cases where things did not work well. And I can actually roll those back into my eval set. So if I do that, then I actually have this self-perpetuating loop, like Karpathy talked about, like the data flywheel where actually I'm running my app in production, I'm collecting cases of bad behavior that I'm tagging with like online evaluation, and I'm rolling those back into my offline eval set.

So what you do there is look at the case that the app is doing poorly in production, audit them, correct them, so build like a canonical question answer pair from that and put that back in your test set.

And that's a good way to build, bootstrap and build it up. So I'd start cold start, synthetic data, small scale examples, online evaluation, or some system to check online where it's failing, loop those back in and build it up that way.

That's like your data flywheel. Yeah. Actually, I even had a slide on this in one of my older talks. I used to work in self-driving for many years, and I actually was a big fan of Karpathy's stuff at Tesla.

And actually I've had his, his thing here of this is like the, the data engine thing of like, you know, you ship your model, you do some kind of online evaluation, where's it failing, capture those failures, curate them, put them back in your test set, run that as a loop.

That's like he called it operation vacation because you can go on vacation and the model keeps getting better. And this was more in the context of like training models because basically all those, those failed examples, once they're labeled, they become part of your, your training set.

But the same thing applies here with LLM apps.

Cool. Yep.

Yeah. So I just wanted to ask a question. You had just mentioned text to SQL generation,right?

Oh yeah.

The tables and everything. So I just had a question where like, for example, all that, all that happens when we send a table schema and you generate the SQL query and then we run it on the database. And if the result set is too, too large or something, so we cannot send it for another LLM to generate a user-friendly query or something, like user-friendly answer or something,right?

How do we handle that? Like what do you suggest on like the query? The result set is so big that we cannot fit it into the context and we can, and I want to generate a user-friendly answer for that.

Yeah. Yeah. The question was on text to SQL.

We actually have a pretty nice text to SQL agent example here. So it's in LangGraph examples. I think it's in, is it SQL? Where is it?

I'll find it here.

Where is it?

Tutorials. Oh yeah, it's in tutorials. So SQL agent here. I think a lot's in the prompting. So basically, in this particular case, I believe Ankush from our team set this up. You prompt your, you can do a couple things.

So you can prompt your SQL agent to, where is it? It's somewhere where he tells it to, yeah, it's basically in all these instructions.

Be very careful about, let me just find it. But in short, you basically can instruct the SQL agent when it's writing its query to ensure not to extract an excessive amount of context. And I can't remember exactly where it does that.

Okay. Yeah. It'sright here.

Limit your, always use a limit statement on your query to restrict it to like whatever it is, five results. And that's the hard-coded thing here. And then also in your SQL agent, you can incorporate a query check node

to actually look at the query before you execute it to sanity check for things like this. So basically I would have a look at the LangGraph examples tutorial SQL agent notebook to have, I've, I've actually, I ran this and evaluated it and I found it did work pretty well.

So that's one thing that I would look at.

Oh yeah. One of the questions is just because a lot of times like the text to SQL, they do well on like the group by type of questions, but how do you handle like filtering questions? Like where it's not like when you have a high cardinality column, like when you have a question,right?

Like if it's a cardinality of SQL query and it needs to use a WHERE clause on SQL, WHERE is actually translates to like any column value, like an empty value from the column. But like especially if you're having something with very high cardinality, like maybe 500 values, like how do you handle that?

That's what I meant.

Hmm. Yeah. So the question was related to like how do you handle high cardinality columns? And I guess that is related to restricting the output size. I mean, I'm actually not really a SQL expert, so I'm probably not theright person to ask about very gory details of text to SQL.

But in general, I would kind of consider, can you prompt the LLM effectively? Like, okay, two different things. One, the general idea is here where basically upfront prompting of your LLM to kind of follow some general kind of query formulation criteria.

And then two, an actual query check explicitly to review and confirm it doesn't have some of the issues like extracting excessive context. But for anything more detailed than that, I'm probably not the person for those insights, but there might be a specific text to SQL deep dive at some other point in this conference and you should definitely seek that out.

But I haven't done that much with text to SQL. Yeah.

I think we've had a lot of, you know, question answer RAG patterns, but one thing I'm quite interested in is with more long-form document generation. So, you know, writing a report or something like that.

Yeah.

Like you see any interesting patterns of like what you think about in terms of architectural solutions with maybe LangGraph or other types of things to achieve that type of outcome?

Yeah. Okay. So the question was related to kind of document generation. That is a really good theme. So we actually have kind of, there was an interesting paper. I did this a while ago. I need to find it.

Where is it? So we have a notebook. If you look in LangGraph examples storm. So this was actually for wiki article generation. Here's kind of the diagram for it. We actually have a video on this too. I'm actually trying to refresh myself.

I did this like three or four months ago, but it was basically a multi-agent style setup in LangGraph where if I recall correctly, I'm just looking at the flow here myself, but basically what it did was you give it a topic and it'll kind of do initially this kind of like generation of related topics and actually uses the multi-agent thing of like editors and, and experts.

The experts go and do like web research. Don't worry too much about the details. So the point is it was an interesting paper in flow for wiki article generation using LangGraph in a multi-step process. And actually the wikis are like pretty good.

I think at the bottom of the notebook I have an example wiki. So you can see all the code here. That's the graph. Yeah. And then here's the final wiki that you get out from this, this type thing.

So it's pretty good. Have a look at that notebook. I think, yeah, Jason Liu also had a post on this recently, this idea of like report generation is a theme like we're going to see more and more of.

This was a, this is one idea that was pretty sophisticated though. You can probably simplify it a lot. I've done a lot of work just on like a simple kind of distillation prompt, like perform RAG and then have some generation prompt, give a bunch of instructions for how I want the output to be formatted.

That also is really effective.

Yeah.

Since you said that you're facing data issues, what is a suggested way to implement clarification tactics also during the process? Because during the reasoning process, say you might, you know, I might get to know that whatever RAG information that you provided, like say node three or node four, you might understand that there is some information that is very specific to the domain and that we do not have.

What is the way to implement or ask back to the users so that they can provide information that can be used subsequently?

Yeah. So the idea, the question was related to user feedback. That's a really good one. So we do have, I think I mentioned previously, if you look at LangGraph, where is it? We, I believe we have some user feedback examples.

Thursday we're definitely going to be announcing something that has a lot of support for user feedback, and I would encourage you to keep an eye out for that. So Tarrason's going to launch that here on Thursday.

I will look for, I know we have some user feedback examples in LangGraph, but I will need to find them. Let me see. We probably,

I'll check LangGraph. We probably tweeted about it at some point. I haven't actually done anything with user feedback though. Let's see.

Yeah.

Hmm. Yeah. Maybe I might have to get back to you on that. I thought we had some nice examples with LangGraph.

Hmm.

Yeah. I'll have to get back to you on that one.

Feedback.

Customer support might be in here.

Hmm. Let's try something else. Look at LangGraph docs.

Hmm.

I'd poke around the LangGraph docs that we have a bunch of tutorials in the docs here. Just Google LangGraph documentation. I'm just poking around here for user feedback. Ah, here we go. Look at this. So LangGraph how-tos, human in the loop.

I would have a look at that. I have not played with that myself, but yeah.

Why would, why do you think, like, why from good, asking for user feedback, what would be a good way to judge whether there is missing information and then ask for the user feedback? Basically looking at the information that is missing and looking at what user has asked for and basically try to judge whether we have all the information that is required to answer the question else put the human in the loop.

Yeah, yeah, yeah. Okay. Right. So for like mid to longer term problem solving tasks, how do you incorporate user feedback to ask for more information? So I would have a look at this documentation because I would imagine it will cover examples along those lines.

I haven't personally done that. I also believe that

I would have a look at the customer support bot because that's an example of, of a kind of multi-turn interaction between a user and a support agent. So I would look at the customer support bot, which Will and my team did, as well as the documentation on human in the loop.

So those are two things I would check out there.

Nice.

Yep.

Is there any research into model architectures or training models for agentic reasoning? Optimized for agentic reasoning?

For agentic reasoning?

Yeah.

Yeah. So that's, that's kind of an interesting question. So the question was related to training models specifically for agentic reasoning. If anyone, so I mean, there's a lot of, there's a lot of work on prompting approaches for different types of reasoning for sure.

I'm a little bit less familiar with like efforts to fine-tune a model specifically for a particular like agentic architecture or use case, but you could imagine it. Most of the work that I've encountered though is just using generalist like high capacity generalist models with tool calling and specific prompting techniques.

So like React is kind of a particular orchestration flow and prompting technique rather than, you know, and you can interchange the LLM accordingly. I think the main thing typically or historically for, for agents has been the ability to perform high quality and accurate tool calling because agents, that's one of the central components of agents.

And so that's kind of been the gating thing. And I think model providers have been focused a lot on just high quality tool calling, which helps kind of like all agent architectures. I haven't seen as much on like fine-tuning for one particular architecture.

I think it's like high capacity generalist models with tool calling and then prompting. So it's more like in-context learning. That's kind of the trend I've seen at least.

Yeah.

Can you talk a little bit more about LangGraph checkpoint? Because I just saw that in the notebook and am Iright in thinking that it's more about sort of production friendly feature as in sort of saving the state?

Yeah.

How much else does it save? Can you just talk a little bit?

Yeah. So the checkpointing stuff, actually this Thursday, that's going to be a lot more relevant because we're launching some stuff to support deployments for LangGraph, in which case you, you can do a bunch of different things, but you can have a single state that persists across many different sessions.

You can also have checkpoints or you can return to state and revisit an agent from a particular point. Don't worry about that too much for now. I think there'll be a lot more documentation and, and kind of context for that on Thursday when this stuff to deployment comes in.

But it's good to be somewhat aware of. I would poke around the documentation for a little bit more on checkpoint thing, but it really becomes relevant on the stuff we're announcing on Thursday. So I would have a look then.

See if there's, if we update our docs. Yeah. Yeah. So there are, there is some documentation on it now, but it'll become a lot more interesting and relevant come Thursday when we have a lot more support for deployment.

Yep.

How do you suggest we deal with issues when the agent is not, is not able to, to call theright tool or makes small mistakes in.

Yeah.

Tool, for instance, instead of YouTube video download, it says YouTube video downloader.

Yeah.

Do we stop the process or do we try this?

Okay. Yeah. So that, that's a, that's a really good question. And so the way it works with the existing, so it depends on the architecture. So using the React architecture, let's see if I can find an example of it.

So here's with React agent. Let's look at one of the traces. Let's see if I have an example. So basically the tool call itself will return like an error and the LLM then is expected to self-correct from that error.

It has to kind of self-correct. So that, that's kind of one approach that at least we do with the React agent. So actually you can see it in the notebook

if you go to, and if I can find some traces that have that example, I will pull them up. But

I think it's in utilities somewhere. Yeah. So basically this tool note with fallbacks, basically what happens is in this tool error. So this, if there's an error in the tool call itself, it'll return that error. And usually the agent will then, or the LLM, the system will look at that and like self-correct its tool call.

So that's, that's typically how it's done. And this actually is reasonably effective. But again, you know, the nice thing about the, the other implementation, the custom agent I called in the notebook is you don't rely on tool calling in this way.

And so you can get around this type of issue. But basically catching the errors in the tool call itself with this code is, is what's currently done. Let's see if I can actually find an example.

Yeah. I mean, if I can look for one where it gets the answer wrong. Yeah. Let's see this one. Let's see if we can find a tool call failure. So here's the trace.

Let's see.

Okay. It didn't have a tool call error. Yeah. So basically what you'll see in the message history is like the tool itself will return this error message and then the LLM will say, oh, okay, I need to retry and then it'll retry and hopefully get itright.

Yeah. Yep.

Follow up to that question is I use Python, Pydantic, and Jason Liu's instructor.

Oh yeah.

Which is great for that exact problemright there because like the instant revalidation of like the output, especially for those simple little errors.

Yes.

I'm wondering, well, first of all, that's probably workbook does. LangChain, I haven't used it much. Like could you use it with Instructor and Pydantic in a similar way?

Okay. So this is a very good point. So yeah, I'm a big fan of Instructor. I haven't used it as much, but what you're saying is one particular type of tool call. So basically that pertains, I believe, more to structured outputs, which is indeed a kind of tool call.

And when you're using something like a Pydantic schema, you'reright, it's very easy to check and like correct errors. So I found catching errors like with schema validation, like using Instructor is, is really good. And we have some other things you can use within LangChain to do the same thing.

So, so that's one type of error that's actually particularly easy to kind of detect and correct. What we show in this notebook here and the code I showed is more for any general tool. So, so this code here will operate on any tool you call regardless.

So it doesn't have to do with structured outputs or anything. And so it's just a more general check for tool call errors. Now, in terms of Instructor with LangChain, now maybe I'll just back up a little bit. So LangGraph does not require LangChain at all.

So that's kind of point one. And neither does LangSmith. So actually everything we're doing here does not need to use LangChain. So actually that could be a pretty interesting thing to try for like kind of the choose your own adventure thing.

But basically in the custom agent part, I use with structured outputs to do the grading. So if you go to, yeah, if you look at the,

the retrieval grader here, so this is using LLM with structured output and here's my grade schema. Try that one Instructor. That should work great. You don't need LangChain at all for this. And that'll fitright into LangGraph. So actually I think it'd be great to use Instructor with LangGraph for this particular use case.

And I do agree that Lang that Instructor is really nice for those kind of like schema validation error correction.

Plug and play?

Plug and play. That, that I'm going to make a note of. That's a really good kind of choose your own adventure case.

Where should I put that? Try Instructor with LangGraph for grading. Yeah, I like that a lot.

Yep.

Yep.

Kind of more of a meta question just because it's really aligned with what you guys are trying to tackle. What's the, what are the path forward to continue to make this technology better? And specifically where's the not, where's the RAG pipeline to be able to like grow?

Like what do we still not do that? And your guys' intention to do so like down in the mid to.

Yeah. So question was related to just RAG in general and where is RAG?

Like RAG agents.

RAG agents. Yeah, sure.

Well,

to be honest, a lot of the problems with RAG, I think about our own internal application chat LangChain. A lot of the problems with RAG, actually our retrieval problems, retrieval is just hard. I'll give a good example. Like LangChain, we have, I'm trying to remember, 5 million tokens of context across all our docs, something like that.

We have all sorts of, we have a very long tail of integration docs. You want very high quality coverage and questions across all of that. There's a lot in how you index all that stuff to ensure that you

boost retrievals from more canonical how-to guides that are much better documented, but still having coverage over long tail content for like, you know, long tail questions. For example, if you're using raw semantic similarity search, you could have relevance to, you know, say your how-to guide, which is really well developed, and three random long tail documents that are not well developed and they'll all get returned.

And so how do you overlay different systems? It could be re-ranking to basically promote content that you believe to be more accurate or better based on some criteria. So that is all to say, I think with RAG, the challenge is actually just domain specific retrieval for your application.

That's just a hard problem. And there's been a lot of work on this. It's been around for a long time. I think that's really the limiter. And actually there's kind of no, no silver bullet. Like in our case, we're having to look at the structure of our documents very carefully, design our retrieval strategy based on that doc structure.

Like in particular, we're thinking about applying certain post-retrieval ranking to docs of certain types based upon their importance. We're thinking about retrieving a large, like a large initial number of docs and then boiling them down with, with kind of re-ranking based upon importance.

So I still think retrieval is very hard. It's very domain specific. It depends on the structure of your documentation. There's kind of no free lunch. I think the things that are good about RAG is context windows are getting much larger for LLMs.

And so back to that point I was making before, I think we're seeing and we're considering this ourselves, less worry about the exactright chunk size. You can think more about chunking in, you know, different ways and then passing full documents to your final model.

So I think that part of it's really good. But still, even like, even in this particular case, you probably still need some

re-ranking to promote the most important documents. So I think retrieval is still quite hard in particular. Like looking at the LangChain docs in particular, the overlay of document importance on top of raw semantic similarity search,right? Take a case of like I have a question, semantically it's similar to 10 different documents.

Those documents though vary widely in their quality and their relevant, like more like higher level relevance. Like maybe that passage is related, but like it might be a general question about how to build an agent. And then some random integration doc talks about building an agent for integration X.

And I want to make sure that the more canonical, well-developed agent, you know, overview doc gets promoted and passed back in the end answer, stuff like that. Sorry, it's a long answer, but basically RAG is, it's hard. I mean, I think retrieval is really the hard part.

The generation part is getting better and better as long contexts grow.

Yep.

So this pre-ranking approach for your documents.

Yeah.

What's, what's the metadata? Do you have a relevancy to a particular topic as well as numerical ranking?

Yeah. Okay. That's a great question. So the question was, when we talk about this re-ranking, how do you assign this relevance to your documents? Like what is that? So I'll just give you what we've been thinking about. I actually think it is for us going to be a hand-tuned kind of relevance score based upon our doc structure.

So if you look at the LangChain docs, like go to LangChain

documents. Yeah. So LangChain documentation, we have these sections up here, tutorials, how-to guides, conceptual guides, which are like really well developed, more recent, well-curated. These you can imagine have some kind of relevance or importance ranking of one or highest.

So these are documents that contain very high quality, well-curated answers that we want to promote and serve to users in the generation phase. However, what say someone asks a question about one particular integration,right? If you go to integrations, we have all these pages,right?

Components, go to retrievers, look at the, you know, Zep Cloud Retriever. This is some stuff related to Zep Cloud specifically. If someone asks about Zep Cloud, you do want to be able to retrieve that doc,right? And so

some ability to differentiate between questions that need, you know, general answers, in which case you would promote your more canonical how-to guides, conceptual docs versus questions that require retrieval from very specific integration docs, in which case you would still promote this information.

That's kind of the crux of it. And I think we'll probably use kind of manual or heuristic scoring to up, up weight or up rank

our core like how-to guides and conceptual guides

over longer tail integration docs. And we might have a router that will indicate whether the question is general or specific. So those are the two things I'd probably do. So routing on the question side and then some kind of heuristic relevance or importance grading or quality grading on the document side.

And that can be packed in the metadata that you pack along with your, your index chunks.

Yep. Maybe not.

Yeah. Oh yeah.

So let's say a typical RAG application where there's a question-answer pair, but we kind of maintain a multi-turn. Like for example, we maintain the conversation history of the user to kind of create a conversation. So the problem is like, let's say for example, a question is asked and then the, the retrieval chunks are like, let's say five.

Right.

And then a subsequent question is asked. So let's say, no, but it's related to the first question, but still somehow in the first node you transform the query and then the retrieval chunks are still the same. So like I would just get an answer which is like more of this first answer.

So like how do you, my question is like how do you make sure that let's say he wanted to deep dive into the document, like into the more context? How do you make that happen in like typical?

Yeah. So the question I guess was like in a multi-turn RAG context, let's say the case where a user asks an initial question

and you retrieve some documents, you produce an answer, and they ask a follow-up that says, "Give me more information about this." Now, do you want to re-retrieve or do you want to re-reference those same docs?

No. So what happened in my case is like.

Yeah.

I kind of, I try to rewrite that question.

Okay. You do a rewriting. So you rewrite the question. Okay.

And then go search the documents. So most often the document could be the same as the first.

The same as before.

Yeah.

Okay.

So the answers would be like mostly the same.

Okay. Okay. Interesting. So the problem there is more of a retrieval problem. You're doing a rewrite. You're still retrieving the same set of documents though. Now, what do you want to have happen? You want to, do you actually want to retrieve different documents or do you want to.

It's kind of like deep dive. Like for example.

Yeah. But that's the question. What do you mean by deep dive? Like you're retrieving, let's say it's a chapter of a book. You're retrieving only the first page. You want to retrieve the whole chapter.

Yeah.

Okay.

Details, intricate details in that chapter.

Okay. Then I think actually a question rewrite would probably not sufficient. What I would think about more is for that second pass, you could actually do something like metadata filtering on your chunks if you have your data or your documents partitioned by like chapters or, or some sections.

I would just do a bulk retrieval of the whole section or something like that. So it's more like a trick on the, on the retrieval side rather than a rewrite of the query. 'Cause I hear, I see what you're saying.

You rewrite the query, you might get the same docs back. If you want to guarantee that you actually get like a deeper dive in your docs, then maybe it's something in your retriever itself. You can increase K, so retrieve more docs.

You could use metadata filtering to like ensure you get all the docs in a given chapter. So I think it's more a retrieval thing, but that's kind of an interesting point though. Yeah.

Cool. Well, I know it's been two and a half hours almost. So there we go. It's good. Yeah, yeah. Yeah. I'm, I'm, I'm hanging out for another till noon, so.

You can do the local tutorial.

Oh, cool.

And it's working fine. I was just wondering about the sports questions. Is that like control group questions?

Yeah, yeah. Okay. Okay. This is good. So see the question was he's doing the, the local, the local agent tutorial and the question's on the eval set. So actually that's a fun one. Modify them any way you want.

The key point was I wanted some questions that are definitely outside the vector store. So I asked something about like two things about sports 'cause I know it's not in my vector store about agents. So I think I indexed three blog posts about like agents and prompting and adversarial examples.

I just wanted someorthogonal questions that'll force web search. So that's the only thing there. But you actually play with those and you can modify them and all that. Yeah. But that's cool to work. Are you using Llama 3?

Yeah.

Cool.

Yeah. I was doing the 70B.

Oh, you've, you have, you have a laptop big enough for 70B. That's.

Oh, really?

Okay. Okay. You're at the edge of.

There's a new one that's taking like a minute.

Okay. Okay.

And then I switched to the 8B.

Yeah. 8B. 8B is.

That's fast.

Yeah, exactly. I mean, it's actually kind of nice you can even run the 70B, to be honest. I'm not sure I can even run it, but yeah, that's cool.

Nice.

Let's see.

Yeah. Well, I can just hang out and, oh yeah.

Sure. What would be the best way to interpret for questions? Is it prompting or?

Yeah. Well, if you have a chat application, so the question was related to how do you incorporate multi-turn. So if you look at the React agent, it uses a chat history as it's state. In that case, follow-up questions will be captured just in the message history as part of chat.

The, I think the current layout of the custom LangGraph agent though is a little bit more single, single turn. So it'd be kind of like question answer.

That's, oh yeah.

I meant like that the agent actually has to come back with a question to.

Oh, okay.

About details later.

Okay. Got it. So the, okay. Got it. So the question was, how do you modify the agent so that it, it will actually return like if it needs more clarification from the user?

Yeah. These particular agent examples don't do that. But again, I think

that's maybe a good takeaway for me. I should add that to these tutorials. Incorporate a simple example of multi-turn.

I will, I will do that and get my contact and I will send that to you.

So now I have like a simple eval or follow-up questions.

Yes.

I think that should be a bit smarter than them.

Yes. Exactly. So you want, so I mentioned previously if you look at LangGraph,

let me find it. It's, it's one of our notebooks.

The customer support agent,

this, so LangGraph examples, customer support is an example of an agent that has like multi-turn dialogue, but it's complicated. So I'd like to maybe augment these tutorials with a simpler example.

I will, yeah, I will, I'll follow up on that if you give me, give me contact info and I'll, I'll send you something.

Cool. Well, I'll just, I'll sit up here. Anyone can just come and grab me. Thanks for everything. Hopefully the cookbooks are working. Yeah, it was good. Made it two and a half hours, so.

Thanks.