Lets Build An Agent from Scratch — Kam Lasater

Hi, I'm Cam, and I welcome you to my talk on how to build an agent. So, I have 3 goals for this talk: I want you to experience the simplest version of what an agent could be, and I want you to feel comfortable in running and breaking the included code.

I think the running and breaking is a very important part to the learning, and I also want you to take away an intuition for how agents work, or how other frameworks work. If you choose to go with some agent framework, that's great; I want you to have some understanding of maybe the underlying, the underpinning building blocks that they used to make it work.

So, with that, let's get to some slides. Okay, great. This is where you can find me on LinkedIn. This is where you can find the slides for this talk and the code. I really encourage you to go and grab the code, try it, run it, break it, see where this code begins to turn from deterministic outcomes to a feeling that you're starting to play with an agent.

So, what is an agent? I really like both of these definitions: agent, LLM, memory, planning, tools, and a while loop. So let's break that down a little bit more. So, mathematically, we could take memory and say, "Oh, actually this is both a read and a write operation," and also on the while loop it's really a conditional and a looping.

So we can break those apart, reorder them a little bit, and come up with the plan for what we're going to do. And so, with all of that out of the way, let's jump in and get to the code.

Finally. Alrighty, so let's jumpright in. So, the first step here is calling an LLM. This is straight from the OpenAI Hello World docs,right? Standard chat completion. And we can run it here. Step 0. Okay.

This should be pretty standard for most people so far. Okay, great. Then let's jump to step 1, the condition. So, here we're going to have the same completion call, checking for the prompt, but then once we get the answer back we're going to make another LLM call that's going to work as an LLM as judge,right?

We are going to ask: "You're a strict critic. Given the following question, determine if the answer is a full answer to the question, question, answer." Okay. And then we give it a forced JSON strict response of whether it's done or not.

And you can see the object here is whether it's done, it's going to be a Boolean. And the original question is, "What is the average wind speed of a swallow?" So we can go in and we can run step 1 to see how it works with a condition.

So, it gives me: "LLM as judge" gives me a thumbs up, that's great. Here's the answer coming back from the LLM. So, so far very deterministic, very mechanistic in its outputs, and then the judgment, and then the response to the user.

So, the next step here is tools, and we are incorporating SERP AI, I think I'm pronouncing that correctly. This is a Google Search API service, and they basically make Google Search API easy. JSON API versus dealing with whatever the Google API nonsense is.

So, I'm asking about buying a hoodie in New York and near Times Square and the like. Again, going to have the same conditional. I'm going to now have this search Google tool,right? And you can see the SERP GET JSON call here.

Engine is Google, passing in the API key with a query and a location. I'm defaulting to Philadelphia from the location of where to start the query from. And then I'm going to print the results. The tool config going into OpenAI looks like this.

And so I'm handwriting out this JSON so that we can inspect it a little bit. We're defining the name of the function we're going to call, giving it descriptions, strict, whether it strictly adheres to the schema here, and then the parameters into the tool call.

So there's a query and a location, and which elements are required, and if there's any additional ones. So this is the JSON that gets generated and passed in when we make our call. The other thing to note with tool calls is that we call the LLM, the tool call kicks back, and tells our local code what tool to call, or which tools to call, and what parameters are included in there.

So we need to handle that. That's not handled by the OpenAI SDK, and we need to look into the tool call. If there are tool calls, in this case we're only doing a single one, we'll get to parallel in a few minutes.

And then we go and we make that tool call off of an array here on tools, which includes the search Google, passing in the args. And then we push those, both the tool call and the response from the tool call, back to the conversation.

And then we can complete with tools again, meaning in this case we are recursing through and calling back to the LLM with those responses passed in as those args, which will then make another determination. So, main loop here.

Complete with tools, the prompt, search Google tool config, and then it responds back, and then it'll see, it'll internally loop on tool calls until it is satisfied and the LLM is satisfied, and then result, come back out. And then here the critic will come and perform a condition again.

So let us go here, let us run step 2 on the tool.

Unhandled rejection. Uh-oh.

Okay, it looks like we failed there due to validation constraint on the location parameter. It doesn't take just any string, it takes a specific set of strings. In this case, the LLM as judge gave us a thumbs up again.

Again, this feels very deterministic. We are asking an LLM to determine that it needs to call a search API, and we're asking the same LLM to go and evaluate whether that search API returned text that was a reasonable answer to where I could buy a fur-lined hoodie near Times Square.

So, of course, still very mechanistic, very deterministic, very straightforward in its outcomes. Okay. But then let's take the next step. Let's go to step 3. And in this case, a little refactoring is in order. So, same prompt around wanting to buy a hoodie, fur-lined hoodie.

We've pushed the complete with tools into a utils folder, so into another file, just to get it out of that main. This still has that same looping, but in this case we've written this so that it can do the parallel tool calling.

These tool calls could come back with multiple,right? This could be an array of tool calls that the LLM is asking us to perform, the client to perform. And so we're promising all over the tool calls, and we are passing in the function arguments for each of the tool calls into the tool functions here.

And then we are then pushing the response from this local function that we have called back into the conversation. And this will have a tool call ID so that the LLM is able to trace, "I asked for this tool to be called, and here's the ID, and then here's the response of that tool call."

So sometimes it asks for the same tool to be called multiple times with different input parameters,right? So that's how that works. The tools

are configured. So, for example, in the search Google example, we're passing in this query as an object, and it's performing the query, and it's returning a string result. In my mental model, the tools are best handled by thinking of them as text transformations.

You might have a couple of different parameters going in, but usually string to string is sort of the core of what these tools are performing. So, that was the AI and the completions, the, sorry, the complete with tools on the refactor.

And so that is here. And then we are outputting the result, and then we are running the same LLM as judge. So, we can run that again. That'll give us the step 3. That'll give us some additional looping and parallel tool calling.

Alrighty, LLM as judge, thumbs up again. Okay, great. Here are a couple of those sites. Again, though, still feels very deterministic. We just reshuffled things around. Shouldn't have been too big of a difference. This is where we are going to really feel an inflection here.

So, the biggest change on step 4 for planning is creating a to-do list. And this now solves both the read, the write, and the planning aspect. The while looping aspect that we talked about is covered by the LLM itself,right?

In making a tool call and then having us reply back with the result of that tool call, the LLM is able to keep iterating and keep operating on the code that we are, or the prompt that we gave it, and keep working through towards a solution.

There are cases when that LLM can just iterate forever, calling tools and never converging, particularly if we keep pruning the context window and we never hit some error or boundary condition. Some guardrails I've seen get put on are the number of iterative loops that they're able to go through before client code kind of cuts them off and says like, "Okay, LLM, you're drunk, go home."

So, in this case though, we don't have that set up, but we have added this to-do list, which has what you would expect from a standard Hello World to-do list,right? You can add things to your to-do list. And in this case, we have also the tool config for the to-do list of adding new to-dos.

That's an array of to-dos,right? There'll be an array of to-dos, and we'll have an array of done. So you can add new to-dos, and they get pushed onto the to-do list. And then we're just printing it out, and then we're returning the to-dos that were added to the to-do list.

Then we can also mark to-dos as done. And so if a to-do is included in the to-do list, mark it as done and pull it out of the to-dos that are there so that we don't have to do it again.

As well as the config for that. And then the check done. We can see if we have completed all of the to-dos. If the length is 0, then it'll say no tasks have been marked to done. And then the config for that as well.

And then check to-dos. So it's able to get all the to-dos that are on the to-do list. So, again, sort of standard to-do list. And now with all of those tools, the LLM as judge, the search Google, and I guess I should cover this browsed web.

Very similar to the search Google,right? It takes in a URL. It's using Cheerio and Turn Down, and we are requesting that URL here. And if the response comes back 200, then it's just taking the text, using Turn Down to turn it into Markdown, and returning that Markdown out of the tool.

So, very simple operation given that URL,right? So, given that, the planning will then take the to-do list that is generated based on the prompt,right? So this is where the programming of the agent begins to take over,right? You're a helpful assistant working for a busy executive, your tone is friendly but direct, they prefer short, clear, and direct writing, you try to accomplish a specific task you're given, you can use any of the tools available to you.

Before you do any of your work, you always make a plan to use your to-do list,right? So that's driving the planning towards the to-do list. You can mark to-dos off of your to-do list after they've been complete, you summarize the actions you took by checking the to-do list, then create a report.

You always ask your assistant to check goal done. That drives towards LLM as judge. If you say you are done, you send the report to the user. If your assistant has feedback, you add it to your to-do list.

And then I've added it to today's date because sometimes the LLM doesn't quite know when in time it is, which is quite amusing. And in this case, since we've pushed LLM as judge as a tool,right, the condition can be a tool itself, the loop is handled by the LLM itself, then this main function is just a single call to this complete with the tools, and then we're going to respond with our answer.

So, oh, and the default here is I want to learn about building agents without a framework. So let us try that. We can npm run step 4 and see how it goes. I want to learn about building agents without a framework.

Okay, it's calling the add to-dos. So here it came up with a plan. These are the news to-dos,right? Search for information about building agents without a framework, summarize key points from the search results, and then he asks the assistant to check if the explanation is sufficient.

It's calling check to-dos, calling search Google about this. It's browsing this blog post, it's browsing this PON data, browsing the web, browsing the web. Here's a bunch of Markdown from that website. It's marking done, summarizing key points from the search result.

It's checking to-dos, it's checking goal done. It's giving it LLM as judge gives it a thumbs up. Okay, great. Thumbs up. We are done. Oh, so here is a summary of building an agent without using a framework. To build an agent without a framework, follow these steps.

Understore core components, recognize that an AI agent is a language model capable of tool use and maintaining conversational context. Define tools, tools or functions for environmental interaction, like database queries, web search. Okay, that's tools, that's memory, a loop, input processing tool decision, execution, response formulation, prompt engineering, direct API calls.

This is a good one instead of trying to pass through the LLM, but just directly store state and call it for yourself. So I feel like this is pretty decent. You know, where I've started to see some

real interest in this LLM, or this agent that I've built, is doing some things like I am planning a date with my wife this Saturday

near Ardmore, yay. Please

help me find

some activities and dinner. Be between 6:00 PM and 11:00 PM.

Let's see how it does.

So, searching for local activities. Here's the plan again. Finding a good dinner option.

Searching Google. Best restaurants. Browsing the web.

Lots of Markdown. Marking things as done. Checking the goal.

Okay.

Here are some of the activities. Here are some of the dinner options. And hopes that I have a lovely evening. So, it's not that

this agent is perfect by any means, but hopefully you can see how adding each of these components from the LLM call, the conditional, the tool use, adding some planning, some read/write in that to-do list, and how we can really leverage some of those things coming together and start to produce something that's quite interesting.

You can see how next steps could be adding a vector database or injecting these browsed web pages into, you know, a Chroma DB in memory or into something, into a more RAG-like system. So, hopefully this was really helpful, and hopefully this gives you some interest or excitement in how to dive in and do this yourself.

Okay, great. Thanks for coming to my talk. Again, I'm Cam Lasater. You can find me on LinkedIn here. The slides are up on my personal site, and the code is up on GitHub. I encourage you to take it for a spin.

Let me know what you think. Let me know if there's some improvements you could make or if something like this is exciting. We're always building and looking for people who are interested to build agents for our customers. So, again, see you online.

Cheers.