Why the Best AI Agents Are Built Without Frameworks (Primitives over Frameworks) — Ahmad Awais, CHAI

Well, hello there. I'm Ahmad Awais, and I'm going to wipe code an AI agent built with AI primitives while I deliver this talk,right? So let's start here. One of the most common AI agents that we've seen in production is: there's some data, and there's a chatbot as an agent, and you're trying to chat with that data.

Because, well, all of these LLMs are self-attention algorithms anyway,right? So you go to chai.new and you say something like, "Chat with PDF," and let's see what happens. Chai is now going to wipe code an AI agent for you, and it's going to build it on top of AI primitives instead of an AI framework.

And as that is happening, it just takes, like, a minute. How about I tell you a little bit more about who I am and what are we talking about today,right?

So, uh, look at Perplexity, Cursor, v0, Lovable, Bolt, and even now Chai. You will see one common theme across all these production-ready agents that are, you know, building millions of agents, millions of generat- you know, millions of people are using them.

And this is one—this one really good trend that you can pick up: that all these AI agents in production are actually not built on top of any AI frameworks. Because, well, frameworks do not really add that much value.

They're bloated, they move super slowly, and they're filled with these abstractions that nobody really needs. Instead, you should be building on top of AI primitives. This is what my entire talk is about today. I am Ahmad Awais. I've been around the block for quite a while.

If you use WordPress, Next.js, Node.js, React, you're probably using my code because I've contributed to all of these software. I have also built hundreds of open-source packages, mostly automation CLIs with Node.js, and created a shades-of-purple code theme, all of which are downloaded like 40, 50 million times, you know, a year.

And I've gone up to, you know, as technical as you can imagine. I've contributed to NASA Infinity Helicopter Mission, and in the past, I've been a VP of Developer Tools, VP of Engineering, Google Developers Advisory Board member. All I'm trying to say is: I'm deeply technical.

I've gone through this phase of working with and building frameworks. And now, why do you think I'm talking about primitives? I think primitives have this native ability of working really, really well in production. Like, Amazon S3 is a really good example here.

Amazon S3 is a primitive where you can upload data and download data, and they scale it massively. They're not building a framework for object storage. It's a very simple thing. It's a low-level primitive you can use to build lots of things,right?

And that is what we are talking about here today. My journey in LLMs actually started in 2020 when Greg Brockman himself gave me access to GPT-3. GPT-3 was just, like, what, maybe a month-old model, and I had already started building, you know, something like GitHub Copilot, which launched in 2021, a year later,right?

Even now, you know, we've been building things in agents for, like, I don't know, 5 years. Building and deploying and scaling AI agents remains to be the biggest pain there is. And I think everybody has a different definition of AI agents, but this is my take at it: I think AI agents are just a new way of writing code.

Everything we know of how we used to build code, how you used to build coding projects, or SaaS, all of that is changing because of AI and because of agents. And it's just big enough. It's not—it's just a new way to write code.

It's big enough, you know? If you try to put it inside of a framework, that abstraction might not be enough. Instead, how about we build small building blocks that are useful across a stack? Like, something like threads. You know, every agent needs to store some sort of context or history of conversation.

So threads would be an awesome primitive. And we built that,right? So why not build that? Why I think that is important is because—here's my belief—I think most of engineers are going to become AI engineers. This you can already see with full-stack AI engineers, web developers, and front-end developers.

They're quickly transitioning into this AI engineering role because they're shipping a lot of product with AI, and they're building stuff with different LLMs or whatnot, with vector stores and this and that. Even DevOps engineers and ML engineers are now shipping product.

So when everyone is building as an AI engineer, what we are trying to do here at LangBase is improve their experience. We want to become the fastest possible way for you to build a production-ready AI agent. A lot of people, I think, start with this painful way of building AI agents, where they pick up a framework which is filled with, you know, obscure abstractions which are really, really hard to debug, and then they have to figure out how to deploy and scale those agents.

We think the other way. I think if you are building on top of predefined, really good, highly scalable AI parameters, especially composable parameters that come with a piece of cloud in it, like, you know, memory. Memory is an AI primitive which is, like, you know, like memory which has a vector store in it.

You can throw in, I don't know, terabytes of data inside of memory, and it will automatically scale. So if you build an AI agent with that memory, or with that parsing, chunking, or threads or tools infrastructure, what you get is a serverless AI agent that automatically can do the heavy lifting for you.

But what does that actually look like? Let's take a look at what is happening here. Today, in this talk, I'm going to share—and by the way, like, I'm going to probably, you know, it's already done—I'm going to deploy this before, you know, we move forward.

I'm going to go over 8 different AI agent architectures that are built purely with AI primitives instead of a framework. Look, like, look at this. You know, I said, "Chat with PDF," and it figured out that, well, you need memory for storing this PDF, which will have a vector store, and you need an AI agent, an LLM, which will be used to ask questions from this PDF,right?

So it went ahead and did this. You know, we need—it created a memory, created a way for you to generate answers, an agent, on top of that memory,right? The flow of this is very simple. I want to retrieve some PDF content and then generate the answer,right?

And, you know, it's already done. As you can see these lines, the first step is to build that memory. And we are using a primitive here, langbase.memories. This primitive is where we will put the user input, the, you know, the question that the user is trying to ask.

Is the name of that memory, the PDF document memory? If you go here, you will find that memory. And this—all of this can be built with API as well. What I'm going to do here is I'm going to go to, I don't know, amodawais.com about and amodawais.com talks.

This page has information about me. This one has information about my talks, like the one I'm givingright now. And I'm going to throw all of these as

PDF files in it. So I have already saved them on my desktop, so I'm going to just quickly grab them

and upload here. I've also—I'm going to also going to do this: api-key langbase.

So I'm also going to throw a PDF version of this particular file into the memory. As you can see, this is the doc about LangBase, how to get API keys from LangBase, my talks, and my about page. All of these files areright now being processed, which means a parser primitive is converting these files from PDF to text.

And then a chunker primitive is going to chunk these into small pieces of context, which are going to be used in a similarity search. If I refresh these, you will see that all of these files are now ready,right?

Now, now let's go to this agent code and see what is there,right? So we are going to ask a question from this memory,right, which will return back some memories. And these memories are—in the second step, as you can see here, this is what we are going to use as context,right?

This is generate answer step. And in this step, we are basically going to create an AI agent that is going to use that context to answer the questions you have,right? Very simple. It has also—Chai has also, you know, built an agent app for you, which makes it really easy to use.

But you can also go ahead and use, you know, an API with whatever programming language you prefer. Let's try the agent app. Let's ask it something very simple. So who's the founder and his last 3 talks? There you go.

Right now, this information is in 2 different files in our memory, which was parsed, chunked, and embedded automatically using those primitives. And you can see the answer is already here. I'm the founder. Here are, you know, the last 3 talks I did.

And how do I get an API key? This should probably also give me an answer on how to get that API key from that other file that we had added in there. There you go, how you get an API key.

There's a bug. You can actually go to the app mode and wipe code. A fix for this bug. And this app will be fixed. You can also, you know, make it public or whatnot. What's happening behind the scene here is most interesting.

You know, I think every agent needs all these primitives that we are building,right? Primitives are like, you know, memory, which is an autonomous RAG engine, a workflow engine that is built—purpose-built—for multi-steps agent. Threads, where you store and manage this context and conversation.

A parser to extract the context and a chunker to split this,right? And using all these primitives, you can build almost any AI agent. We actually did a lot of research on stateofaiagents.com, where you can read a lot about how people are building agents, what type of primitives they are using, and what type of LLM is required and doing really well in which type of industry,right?

Let's now go over different AI primitives and different agent architectures that use different AI primitives. So the first most common architecture that we see is an augmented LLM. Augmented LLM is basically an agent. It's an agent that is going to get in, you know, some input, and it's going to generate some output, which will have an LLM.

It will have some way to automatically call tools so it can connect to MCPs or call different APIs. It will have access to threads. Threads is another AI primitive, which is where you can store the conversation users are having with this agent or the context of this agent, you know, which might be, you know, completely asynchronous,right?

Thread of context, a scratchpad. Like, for example, when you're booking a flight, there's a certain set of information that you kind of keep in your head or on a scratchpad. Like, I'm going to land here, I'm going to go do this and that.

And that is useful when you are, you know, booking that particular flight. So you can use thread to store that, you know, memory-like context. And then there's memory, the long-term memory of, you know, different events or whatnot. This could be terabytes of data that you want to search when you're using this agent,right?

Which obviously requires more context and requires a, you know, vector store. So in this example, in this augmented LLM, we see tools as primitives, threads as a primitive, and memory as a primitive. And you can basically build almost any type of AI agent with this augmented LLM architecture.

As you can see here, LangBase pipes or agents provide you with that primitive. Now let's look at what type of agents and what type of architectures can you actually build using that augmented LLM that I just showed you.

Here's another one. So prompt chaining and composition, where you use multiple agents, as you can see the purple blocks here, working together. You get an input, an agent creates an output, and based on that output, you basically decide if you want to go forward,right?

Maybe you got an email and you figured out if that was spam or not. And if it was not spam, then you use another agent to write a draft email in response. And you can take a look at, you know, what's happening here.

There's a summary agent, there's a features agent, there's a marketing copy agent. This code is plain JavaScript or TypeScript, if you will,right? You can see here a summary agent. This is a feature agent, and this is a marketing copy agent.

They're all going to work together to generate an output that you need based on an input. A more interesting one is something like agent router,right? Where an agent or an LLM router that you built basically decides which other agent is needed to be called next,right?

Let's see what this is, what this architecture looks like in production.

None of this—we are going to build all of this with AI primitives. None of this is built with a framework. We are going to basically create 3 specialized agents for different tasks. One is a summary agent for summarizing text.

The other one is a reasoning agent for analyzing and explanations. And then a coding agent for obvious reasons, you know, when you need to write code. And all these are built for different LLMs. So summaries from Gemini, reasoning is with DeepSeek Llama 70B, and coding is with Claude Sonnet,right?

Now let's look at this code. This coderight here is only going to use AI primitives, and you will build your own AI framework on top of it instead of using a bloated AI framework,right? So as you can see, here's a routing agent.

Router is being told what the job is. It has access to all these agents, and it is supposed to respond back with valid JSON and picking which of these agents is going to do the job for us,right? And here is the documentation of a very simple set of agents.

A summary agent, a summary agent, a reasoning agentright here, and a coding agent,right? Now all of these are going to run together. All of this, as you can see, is just plain old code. There's nothing new here. You can easily write this.

There's no magic. There's nothing to learn. It's basically you're just calling these agents together, and one of these agents is going to respond back with which next agent you need to run, you know, for the task. So the task is very simple.

Why days are shorter in winter? Now let's go ahead and run this code.

So the main agent, the router, the scene agent, has decided that it needs to run the reasoning, or you need to run the reasoning agent for this particular task,right? Obviously, you're not writing code here. You're not writing summary.

You can see how the scene maker made theright choice. And then you pick up the same reasoning agent that you had built based on this answer and then throw that input in there. And here's the, you know, answer of why, answer of why days are actually shorter in winter.

Don't get distracted by this answer, by the way,right? So. Another very common architecture for building agents we see is running agents in parallel. This is absolutely simple to use. There's no abstraction needed. For JavaScript, in JavaScript, you can basically build a set of agents, as you can see here.

Here's a sentiment summary in the scene maker agent. And you can promise.all all of these agents, and they will run in parallel. My favorite, however, is this agent orchestrator worker, where an agent basically decides to orchestrate and build n number of worker agents that are going to solve a problem, which is then synthesized by another agent,right?

This is exactly what a deep research agent architecture looks like. So this one is really good. What we are going to do in this is we are going to create an orchestrator agent that will plan and create subtasks for worker agents.

And these worker agents are going to work on those subtasks,right? Let's see how that looks, what that looks like. So this is the orchestrator. And the entire thing it is going to do is give us back this type of response with subtasks, which are going to be things that, you know, these worker agents are going to do.

It's a very simple worker agent. It's going to get a subtask, and it will complete it,right? So let's look at the main input here. The input is, "Write a blog post on benefits of remote work. I care about productivity, work-life balance, and environmental impact."

Let's see what happens here. So the orchestrator here is going to generate a bunch of subtasks. As you can see, that is what it has done. So write the introduction. Write a section on productivity, on work-life balance, on environmental impact, and then write a conclusion.

So as you can see here, there are like 1, 2, 3, 4, and 5 subtasks. So this is going to take 5 workers, worker agents, to complete. As you can see here, this is the first worker agent. Second, third, fourth.

And here's the fifth one that is writing the conclusion,right? And finally, we are going to synthesize all of this into one thing. Look how simple this is. This is like what, like 90 lines of code, and there's no framework here.

You're basically building very simple, you know, agents, worker agents, and an orchestrator agent. And you're promise.all running all of them, and the data is just flowing through that,right? There's no need to build a complicated abstraction layer on top of this because maybe, you know, in a couple of months, you'll see that agent, this agentic workflow will get even better with LLMs understanding how to run these worker agents.

If you are using any AI framework and abstraction that you are, you know, kind of stuck with, when these agents and when these LLMs become much more better at these agentic workflows, you'll have a very hard time migrating off of it.

So it's much better to build on top of primitives instead of building on top of a pre-built abstraction,right? We'll probably also going to take a look at this evaluator optimizer where an LLM is used, an agent is used to generate a response, generate something.

And this something, this is probably, I don't know, let's say a marketing copy. This is going to be evaluated by an LLM as a judge,right? Which will either accept it or reject it with feedback. I'm just quickly going to go and run this.

You know, here's a generator which is generating, you know, some product description. Here's an evaluator which is going to either say accepted or provide with feedback based on the prompt you write here,right? And what we are doing is we are writing an eco-friendly description for an eco-friendly water bottle for conscious millennials.

So the first agent takes a stab at it, and the evaluator actually says, "No, you are basically missing, you know, the point with this particular type of audience, the eco-conscious millennials,"right? And provides very specific feedback on what to do and how to improve this.

And this evaluator should be built by probably the best possible LLM you can think of for your space of what you are trying to evaluate. If it is related to health, you're probably calling a memory, and, you know, you're doing all that you need to do to make sure this evaluation is really good,right?

And the second iteration, as you can see, is very well, is generally well done based on this particular feedback.

Finally, obviously, you can call tools with this. We can skip that. I think you can probably easily understand that. The most interesting one is memory. Memory is when you upload data, you create a memory agent, you upload the data, and then you retrieve that data and ask questions, which is exactly what we saw here in this code where we created a memory.

Let me go to the agent code where we created a memory, uploaded the data manually, which you can also do through an API, as you can see here. And then an agent was used to answer questions related to that data.

This is also a very common pattern for building agents. Now that you know all of this AI primitive patterns, you can build pretty much 80% of the most complicated AI agents out there. And I've done just that. Like, I've basically asked Chai to build me a deep researcher-like perplexity where it actually went ahead to build things to analyze the query, do a web search, consolidate the results, and then create a response.

It is using EXA here, and all the code is here. I can actually just probably go ahead and ask it something. I don't know, "What's the latest with OpenAI?" Something like this. And it's probably going to go ahead, send this basic query, and start doing that deep research.

I've also built things like, for example, oh, let me go back. I've also built things like I wanted a receipt checker which would do OCR. And since Chai or LangBase didn't have an OCR primitive, I found one from Mistral.

So I asked it to use Mistral OCR and gave it an example of how to use it. And that is what it exactly did here. So it's processing the image with OCR and using GPT-4.1right now, extracting from the user input whatever is needed, and also using Mistral OCR latest model on top of it to do a double fallback and really improve the OCR based on top of the image we are going to send it,right?

Let's see, you know, I have an example here. I think this is Palo Alto $15 delivery or something. Yeah, there you go. Total paid $15 in city of Palo Alto. It's a parking ticket, I guess. It's analyzing the resultright now.

And there you go. City of Palo Alto and $15,right? Similarly, I built an agent where I was like, "Let me add an image URL and let me chat with that image,"right? So I just added this image URL. What is the expression

of the person in this image? Okay, let's give it a go and see, you know, what happens.

There you go. Eyebrow is raised, quite skeptical and curious, which is what I generally look like in real life,right? So and the code for this is pretty easy. Analyze the image using GPT-4.0, a vision-capable model, takes the image here, takes the image URL from here, passes the input, and that is pretty much it,right?

So pretty simple flow as if you ask me. So, well, this is pretty much it. And we've seen like, you know, it's May 30th, and you can see almost every other minute there is a new agent being built.

I saw somebody building a dental cost advisor, a business swamp pro, I don't know, a task refiner. All sorts of amazing agents are being built with Chai. And Chai is building all these on top of AI primitives instead of building it on top of a framework that would have some abstraction that you probably don't even need when building agents,right?

So the idea is very simple. All the production agents that we know of, when they are not built on top of frameworks, what good a framework would do to you in a fast-moving space when every other week there is a new paradigm, a new LLM, a new problem being solved already that used to take a lot?

Why not build your AI agents on top of really good AI primitives? And you can either build those AI primitives or you can use the pre-built AI primitives that we built or some of our, you know, friends are building over at different companies.

And also, if you really enjoy, you know, vibe coding, give Chai a, you know, a try. It will try to use the primitives it knows, and it will also try to make it really, really easy for you to quickly build an agent, deploy it, and use itright away instead of building a silly demo that may or may not scale,right?

I am Ahmad Awais, and I am always hanging out on Twitter. I would love your feedback. I would love to know, you know, what you folks think and what you prompt ship and ship with our AI primitives or not.

Take care. Ciao. Use your code for good. Peace.