Building a Smarter AI Agent with Neural RAG - Will Bryk, Exa.ai

Allright, so I was going to give a live demo of coding, but—well, I will, but I know you all are actually here to hear a cool story. So I'll tell you a story about web search built for AI, and then we do some coding at the end.

This story will end with this slide: one API to get any information from the web. And you'll know what this means by the end. But the story starts in 1998, and what you're looking at is the state-of-the-art in information retrieval in 1998.

You type in a word, "Australia," to this new search engine called Google, and it magically finds you all the documents that contain the word "Australia" from the web. It's crazy. And the big insight of Google was they had this PageRank algorithm, so the results are ranked by authority based on the graph structure of the web.

And this was a clever algorithm, and it was really cool. I was 2 years old at the time, so if I was conscious, I would have thought this was cool. OK, and now our story skips 23 years to 2021.

By this point, I was conscious, barely. And

I noticed that GPT-3 had recently come out, and it was this magical thing that you could input a whole paragraph explaining exactly what you want, and it would really understand the subtleties of your language and give you an output that exactly matched.

And it's hard to remember how magical this was, but it was really magical in 2021. And at the same time, I noticed there was Google, which you type in a simple query like "shirts without stripes," and it would give you shirts with stripes, which is crazy.

It doesn't understand the word "without," because it's doing a keyword comparison algorithm. And so I decided that for the next at least 10 years, I'm going to devote myself to building a search engine that combines the technology of GPT-3 with a search engine, to make a search engine that actually understands what you're saying at a deep level, and understands all the documents on the web at a deep level, and gives you exactly what you ask for.

This is a very big idea, and we've worked on it for four years, and a lot of progress. But it would change the world if you actually solved this problem. And so in 2021, we joined YC. Summer 2021.

We raised a couple million dollars, and we did what every YC startup should do. We spent half of it on a GPU cluster. I'm joking. You shouldn't do that.

And then we also followed YC's advice, where we didn't talk to any users or customers for a year and a half, and we just did research. Again, you shouldn't do that. You should talk to users. But in our case, it made sense, because we were trying to solve a really hard problem, which is redesign search from scratch using the same technology as GPT-3, this next token prediction idea with transformers.

What if you could apply the same thing to search? And this is actually one of our 1DB training runs. The purple one, I believe, was a breakthrough where it really learned. There were a few breakthroughs along the way involving random data sets and different transformer architectures that we were trying, and this purple one really started to work well.

And the general idea we had was, OK, so what is a search engine? You have a trillion documents on the web. And traditional search engines, on a very high level, will create a keyword index of those documents. So for each document, you ask what are the words in those documents, and you create this big inverted index where you map from words, like "brown," to all the documents that contain that word.

And then at search time, "shirts without stripes" comes in. You do some crazy keyword comparison algorithm and get the top results. That's obviously a simplification of what Google does, but at a fundamental level, it's doing a keyword comparison.

But the idea was, what if you could actually so with transformers, the big thing is, what if you could turn each document not into a set of keywords, but into embeddings? And these embeddings can be arbitrarily powerful. An embedding is just a list of numbers, and it could represent lots of information.

So an embedding, it doesn't just capture the words in the document, but also the ideas in the document, and the way people refer to that document on the web. And an embedding can be arbitrarily big, and so of course, in the limit, it would just destroy keywords.

And so you have this arbitrarily powerful representation. And now the fundamental idea was just the bitter lesson. What if we could train transformers to output embeddings for documents, and if we keep getting more and more data that's high quality, we could make a search engine that actually understands you?

And the way it would work at search time is, a search comes in, a query comes in, like "shirts without stripes." Traditional search engines would use the above thing, where they would do a very fancy keyword comparison and a bunch of other things.

And then instead, we would just embed the "shirts without stripes" and compare it to the embeddings of all the trillion documents.

And after a year and a half, we actually had a new search engine that worked in a very different way. And you search "shirts without stripes" on Google, I'm sorry, on Exa, and you get a list of results that actually do not have stripes.

It's a simple example, but it could handle way more complex queries, like paragraph-long queries.

And when we launched this in November 2022, we got a lot of excitement on Twitter. This is a very new paradigm for search. You could do all sorts of interesting queries that you couldn't do before. And then two weeks later, this happened.

It was a small tweet. And this is a visual depiction of San Francisco at the time. You guys probably all remember this. And then this is a visual depiction of the Exa team at the time, because ChatGPT completely changed the way we interact with the world's information.

Everyone can now use an LLM to just talk to their computer and get information. And we were thinking, wait, is there even a role for search in this world? These LLMs are so powerful. And then very quickly, we realized, yes, there is a role, because LLMs don't know everything on the web.

So for example, if you ask an LLM like GPT-4, "Find me cool personal sites of engineers in San Francisco," it can't. It just doesn't have that in the weights, so it'll apologize, whatever. And there's a very simple information theory argument here, where it's like, there literally isn't enough information in the weights of GPT-4 to store the whole web.

GPT-4, we don't know exactly how many parameters. I think someone leaked it on YouTube once. But it's like, a couple trillion parameters. You could call it less than 10 terabytes in the weights of GPT-4. And then the internet is over a million terabytes, and that's just the documents on the web.

There's also images and video, and that's way more. Actually, I did a tweet recently about the size of the web, and it's in the exabyte range. And our name is Exa. It's not a coincidence. Anyway, so LLMs need to search the web, just from this simple argument.

And they're going to need to do that for a long time, which if you talk to ML researchers, they'll say the same thing. It's just like, it's too hard. Also, the web is constantly updating. That's another problem. It's not just the size of the web.

It's the constant updatingness of the web that makes it very tricky. So LLMs always will need search. That's great. And so when you combine an LLM with a search engine like Exa, you can handle these queries. So like "find me cool personal sites of engineers in SF," the LLM will search Exa, get a list of personal sites, and then use that information to output the perfect thing for the user.

You're all very familiar with this. LLMs plus search, it's obvious now. Everyone knows about it. But now let me tell you a secret about search that most people don't know.

And the secret is that traditional search engines were not built for this world of AI. Traditional search engines were built for humans. And humans are very different from AI. So every search engine, like Google, Bing, you name it, was built in a different era for this kind of creature, this slow, flesh human that's typing keywords and wants to read a few links, and really cares about UI of the page and all these things.

It's a lazy human. They type simple keywords. Google is great for this creature. Google was optimized for this creature. It gives you exactly the kinds of things you would click on. But AIs are very different. An AI can gobble up information like crazy.

This is a much slowed-down version of what our AIs probably feel like inside. And so AIs are very different. They want to use complex queries, not simple ones, to find not a couple links, but just tons of knowledge, as much knowledge as they could get, because they actually have the patience to just analyze it all extremely fast.

And so the search algorithm that's optimal for this type of creature is not the same algorithm that's optimal for the human. That would be crazy if the same algorithm that was optimal for humans was optimal for AIs. And so a lot of the tools, the search tools that we're talking about these days on Twitter and stuff like that, they're still using the old traditional search combined with AIs.

It's just not theright puzzle fit. So Exa, we're really trying to think of what is theright search engine for this AI world. And so just a few examples we could dive deep into of how AIs are different. Well, AIs want precise, controllable information.

So by the way, when I say AI, I'm usually talking about an AI product. So imagine, in this case, a VC that's using an AI system to find a list of companies because they want to invest. So they're looking for what's the next big thing?

What's the next big thing that feels like Bell Labs? Well, when they tell their AI what they want, the AI will then go search a search engine. And if it searches a search engine like Google, it'll get a list of results that humans like to click on.

But it's not very information dense, and it doesn't even match what the person asked for, what the AI asked for. The AI asked for startups working on something huge that feels like Bell Labs. It should get a list of startups.

It's kind of a crazy idea, but what if search engines actually returned exactly what you asked of them, and not what Google knows you'll click on? And so with AIs especially, they just want a search engine that returns exactly what they asked for.

Because what really the world's going to look like is you're going to interact with your AI agent, and you're going to ask for something, and then it's going to make tons of searches. Like, OK, maybe they want startups working on something similar to Bell Labs.

Maybe they want startups working only in New York City that have this quality and that quality. And they'll do all sorts of searches. And it just wants a search API that just does what it asks. And so you need a search engine like that.

So Exa's like that. Another difference between AIs and humans is AIs want to search with lots of context. Again, if you have an AI assistant, and you talk to it all day, and then you ask for restaurants or apartments or what have you, the AI has lots of context on you.

So it should be able to search with this large, multi-paragraph thing, saying, my human is a software engineer, and it likes these types of things, and I like these types of things, and can you give me restaurants that match those preferences?

And so you need a search engine that could literally handle multiple paragraphs of text. But traditional search engines like Google were not meant to do that, because humans would never type in multiple paragraphs, because they're too lazy. So Google was optimized for simple keyword queries.

So Google, I think, has a few dozen keyword limits, whereas Exa can handle multiple paragraphs of text. Another big one where AIs are different than humans is AIs want comprehensive knowledge. If you give a human 10,000 links or 10,000 pages, it doesn't know what to do with that.

It would take 10 days of extreme patience to process all that. But AIs can do it in three seconds if it's parallelized. So if I'm a VC and I want a report on all the companies in a space, I want literally all the companies, and there's a huge amount of value to getting truly all of them, and not just the 10 or 20 that Google is able to find.

And so you need a search engine that exposes the ability to return 1,000, 10,000, whatever it is, and also has this semantic ability to when you say, every startup funded by YC working on AI, you actually can get all of them.

So Google literally just can't do this at all. OK, I hope that through these examples, we see that the space of possible queries is actually way larger than people realize. And until 2022, we were kind of in this top-left blue world.

So this circle is the space of possible queries, and the blues are specific subsets of that space. And so we were all in that top-left corner of blue for a long time, where search engines could handle basic keyword queries, like Stripe pricing, or someone's GitHub page, or Taylor Swift boyfriend, or whatever it is.

After 2022, everyone started to want the top-right blue circle, where it was like, hey, actually, I want to make queries like explain this concept to me like I'm a five-year-old, or here's my code, can you debug it? This is a form of query.

It doesn't require search, but it's another type of query that was introduced to the world in 2022. And then there's other types of queries, like these semantic queries. People in San Francisco who know assembly, as far as I'm aware, Exa is kind of introduced this kind of query and does really well on those queries.

And then there's these really complex queries, like find me every article that argues X and not Y from an author like Z. And we're starting to now have systems like Exa's WebSearch product that can handle these things. And I think this is actually a huge space, because this turns the web into a database.

You can filter however you want. And that's really what AIs want. They want this full, controlled, database-like query system that they could just get whatever they need for their user. And then there are the queries that no one has thought of yet.

Every week, we get tons of queries, and like, oh, wait, that's a really interesting type of query that no search engine could doright now, and eventually, we'll try to handle all the queries that are possible. But there are so many new types of queries now, because we have these AI systems, and the stakes, like the expectations, have just gone way higher.

OK, so now we end our story with the same slide, a one API to get any information from the web. So again, Exa's trying to, if you go back, handle not just the keyword queries, but also the semantic queries, and also the super complex queries, and eventually all queries.

We want one API that could give these AI systems whatever knowledge they want. You have the AI, and you have Exa providing the knowledge. Oh, I only have four minutes. OK. OK. So let's see. Oh, how do I go to a different part of my computer?

Hmm.

If I change to the code editor, how do I do that? Let's see. What? Oh, it's there. Ah, but I can't see it. That's so weird.

You need to use the mirror or the download?

Oh, cool. OK. OK.

Are you using the mirror or the download?

There we go. OK. Cool. Well, first of all, just very quick explanation of this is our search dashboard. We could try different queries. I would just point out, in the search API endpoint, we expose lots of different toggles.

So first of all, you just try out a query and get it shows you the code, and it gets you a list of results. And it exposes tons of different types of filters that you might want to do.

For example, number of results, 10, 100, 1,000, whatever it is. You could have date ranges, or I only want to search over these domains. And it's a lot of toggles, but I think the point is actually you want the toggles, because your AI is actually going to be calling this.

You want a search engine that gives you full control. And we have Neural and Keyword Search, so you could try different ones. OK, let me quickly jump to the code. OK, so I prepared this code, agent.py. So we made this agent, agent Mark.

And Mark loves to make Markdown out of things. Anything you give it, it will make Markdown. And so in this case, we're going to here. Well, I guess, in this case, let's try this query, personal site of engineer in San Francisco who likes information retrieval.

Well, this is the kind of query that Neural would be a lot better at. OK. Save it.

Oh, we're running the wrong agent.

OK, so it's making a query to get a list of personal sites of engineers in San Francisco who like information retrieval. And Mark, the agent, is just making a Markdown output of that. That's a very Neural-type query. You also might want to do a different type of query, which is a more keyword-heavy one.

Let's see.

Demo

Will Bryk GitHub. Let me and.

My GitHub. OK, so here I would want to make a keyword query. So you just change the keyword. Search. So it's going to get information from my GitHub using keyword search, because this is a very typical Google-like search that would work well.

Oh God, I'm running this wrong one again. OK.

Cool. That's information about Will Bryk's GitHub. And then, OK, so when you're actually building an agent, you're going to be combining lots of different types of searches. So Neural searches and keyword searches, and all sorts of other searches that Exa exposes.

So theright agent in the future is going to be this system that decides what type of search it needs for whatever the user says. It'll be like, oh, OK, I'm going to make a Neural search to get a list of things, and then for each one, I'm going to do a keyword search.

You want to give the agent just full access to the world's information in however way it wants, not just keyword search, but also all these other things. And so here I one-shotted with O3 a GitHub agent, which combines these two queries.

So first, because I want to get the GitHub of every engineer in San Francisco who likes information retrieval. So the agent will make a Neural search to get a list of people, extract the names, and then search those using a keyword search to get their GitHubs.

And then if you run that, here it's just getting 10 results. But with Exa, we could do 100 or 1,000 if you're on an enterprise plan.

So now it's getting all the GitHub info.

Cool. So that's just an example. And yeah, there are lots of other things that you could do with Exa. We actually just today launched this research endpoint, where it will actually do as much searches and LLM calls in the background to get you that perfect report or that perfect structured output for the thing you asked for.

So it's kind of like a deep research API, and it's state-of-the-art deep research API. Cool. That is the talk. I hope that was interesting. Thank you.