Building AI Agents that actually automate Knowledge Work - Jerry Liu, LlamaIndex

Okay, hey everyone. I'm Jerry, co-founder/CEO of LlamaIndex. It's great to be here, and today my topic—uh, the talk title—is building AI agents that actually automate knowledge work. So, basically, a big promise of AI agents these days is making knowledge workers more efficient.

I'm sure you've heard the, like, high-level business speak of this, and I copy-and-pasted a bunch of b2b SaaS vendors on theright, in terms of screenshots. You know, "increased operational efficiency," "better decision-making through more data." But what does this actually mean,right?

Does knowledge work automation actually just mean building RAG chatbots? And if not, what is the stack, and what are the use cases that AI agents can actually do, in terms of automating knowledge work? So, for us, a lot of our use cases, and a lot of our core focus areas, is basically automating knowledge work over unstructured data.

90% of enterprise data lives within the form of documents, whether it is PDFs, PowerPoints, Word, and, you know, as you'll soon see, Excel. But humans have historically needed to basically read and write these types of docs,right? You have, you know, an investment banker, or someone, you know, kind of on the customer support side, reviewing a lot of just unstructured data and using that documentation to basically make decisions and take actions.

For the first time, AI agents can actually reason and act over massive amounts of unstructured context tokens and, you know, do analysis, do research, synthesize these insights, and actually take actions end to end.

And so, for us, when we think about the use cases and the types of agents for automating knowledge work, they really fall into two main categories. There's what we call assistive agents, so those that are kind of more like a standard chat interface.

They help humans get more information faster. And then there's automation-type agents, agents that automate routine tasks, can run in the background, maybe require a little bit less human-in-the-loop, and can take actions that automate the routine operational stuff. When we think about the stack that's required to actually build either the assistive or automation-type agents, there's two main components.

There's really, really nice tools, and then there's a really nice agent architecture. With MCP/808 these days, a lot of people are thinking about, "How do I build really nice tools that allow agents to interface with the external world, to basically surface relevant context and let the agent take external actions?"

And a lot of the agent architecture, you know, there's very general reasoning loops as well as more constrained loops. It's basically, "How do I encode the business logic through an agentic workflow to help achieve the task?" So, for the purposes of this talk, we'll talk about three main things.

A lot of stuff to cover, so I'll probably pick up my clock speed a little bit. But basically, there's building a document toolbox, which is, "How do I build really nice tools to allow, you know, AI agents to interact with massive amounts of unstructured documents?"

Two is agent design patterns. So, thinking about, just at a high level, the two categories of agents, from assistants to automation. And three is bringing it together in terms of document agent use cases. So, first step is on building a document toolbox.

Basically, if you think about agents interacting with tools, and as LLMs get better, you're going to have these very general front-end interfaces, like Claude or ChatGPT. Agents need access to theright tools to basically interface with the external world.

And for the purposes of, you know, massive amounts of unstructured enterprise data, they basically need theright toolbox to interact with this data. It's basically a generalization beyond naive RAG,right? RAG is just retrieval. I know this is a RAG workshop, but naive RAG is just, like, retrieval and then one-shot synthesis.

A lot of what agents can do over your documents includes retrieval, but also includes other operations, like file-based search, manipulation, and more. And one of the points I'm trying to make is that to basically create these tool interfaces in the first place, you need a really nice pre-processing layer.

So, you need, you know, actual data connectors to your data sources that basically sync data from your data source into a format that your agents can access. You know, could be SharePoint, Google Drive, S3, Confluence. It needs to sync permissions too and theright metadata.

You need theright document parsing and extraction piece. More on this in just a bit. But you basically need actual, actually good understanding over your documents, over tables, charts, and more. And of course, you know, if you have a large collection of docs, you need to index it in some way.

It could be vector indexing into, you know, vector search. It could also be indexing into a SQL table. It could be GraphDBs. It could be anything. So, basically, to ensure the data is high quality, you need this layer to actually process and structure your documents and expose theright tool interfaces.

In terms of theright tool interfaces, this is what I want to kind of define a term. It's basically called, like, a document MCP server. Again, it's like a generalization of this idea of RAG,right? If RAG is just one-shot vector retrieval, you kind of need, like, a set of tools to basically equip an AI agent with, to basically understand and manipulate different types of documents.

It could be, you know, doing semantic search to fuzzy-find the relevant source of data. It could be file lookup to basically look up theright file metadata. It could be manipulation to actually do operations on top of the files.

And it could be structured query,right? Querying a more structured database to get aggregate insights over the types of data that you've extracted out.

One, you know, top consideration when actually building this type of toolbox is complex documents. For those of you who follow our socials, we talk a lot about this type of issue, where a lot of human knowledge is in the form of, like, really complicated PDFs and other formats too.

Embedded tables, charts, images, irregular layouts, headers, footers. This is typically stuff that's designed for human consumption and not machine consumption. And so, you know, if the documents are not processed correctly, no matter how good your LLM is, it will fail.

So, we were probably one of the first people to actually realize that LLMs and LVMs could be used for document understanding. If, in contrast to more traditional techniques where you use kind of, like, hand-tuned and task-specific ML models to achieve kind of, like, document parsing over a specific class of documents, LLMs actually have a much general layer of accuracy that you can use to your advantage in just, like, understanding and inhaling any type of document with any type of complexity.

Obviously, the baseline these days is you can just screenshot a PDF, feed it into ChatGPT or Claude. It doesn't actually give you amazing accuracy, but it's a good start. And so, one of the kind of secret sauce, like, magic tricks we found was figuring out how to interleave LLMs and LVMs with more traditional parsing techniques and adding kind of test-time tokens in terms of agentic validation and reasoning to really get a higher level of accuracy.

And so, you know, we have a cloud service that does document parsing and is a core step of this document toolbox. We basically benchmarked our modes where we adapt, you know, Sonnet 3.5, 4.0, Gemini 2.5 Pro, 4.1 from OpenAI, and it basically outperforms all existing parsing benchmarks and tools out there, in terms of open source to proprietary.

Yeah.

So, some of you might know us as a RAG framework. That's basically how we started. You know, for those of you who don't know, we have this managed platform that is basically this GenAI-native document toolbox. Contains a lot of operations that you need to do on top of your docs.

It could be document parsing, document extraction, uses some of those, you know, kind of capabilities I just mentioned, and allows you to parse, extract, index data for all the set of tools I just mentioned.

One of the special releases I actually want to highlight today, and we just announced this in a blog post a few hours ago, is Excel capabilities to help complement this document toolbox. A lot of knowledge work happens in Microsoft Excel and also Google Sheets and, you know, Numbers, and basically it's spreadsheets,right?

But it's been unsolved by LLMs. If you look at the document to theright, neither RAG nor Texas CSV techniques will actually work over this, because it's not really a structured 2D table. There's a bunch of gaps in the rows and gaps in the columns.

So, we basically built an Excel agent that's capable of taking unnormalized Excel spreadsheets and transforming them into a normalized 2D format, and also allows you to do agentic QA over both the unnormalized and normalized versions of the Excel spreadsheet.

It's a pretty cool capability. I'll describe how it kind of works in just a bit. But it's going to complement our toolbox,right, in terms of more traditional document parsing, extraction, indexing, and it's available in early preview. So, if you just take a look at the video, it's also on our blog post.

We basically uploaded that example synthetic dataset, transformed it into a 2D table, and you can also ask questions over it to basically get insights. And it's really doing the heavy lifting of deeply understanding the semantic structure of the Excel spreadsheet, and then using that and plugging that in as specialized tools to an AI agent.

The best baseline is not really RAG or Texas CSV. Those both suck. It's really just an LLM being able to write code. So, an LLM with a code interpreter tool is a reasonable baseline, gets you to 70, 75 percent accuracy.

Over, like, a private dataset of synthetic Excel sheets, we basically were able to get this up to 95 percent. It actually surpasses human baselines of 90 percent of a human trying to go and do the data transformation by hand.

A brief note on how it works. It's a little bit technical, but, you know, more details are in the blog post. First, we do some sort of structure understanding of the Excel spreadsheet. So, we do a little bit of RL, reinforcement learning.

You know, we actually kind of adapt dynamically to the specific format of the document and learn a semantic map of the sheet. By learning a semantic map, we can then translate this into kind of a set of specialized tools that you provide to an agent.

And so, from an abstract perspective, you can kind of think about it as an agent could just write code from scratch. As LLMs get better, that will certainly become an ease, like, a kind of higher performing baseline. But in the meantime, we're helping it out by really providing a set of specialized tools over the semantic map so it can reason over an Excel spreadsheet.

Great. The next piece here is, so we talked about a document toolbox. We talked about a lot of operations basically make this document toolbox really good and comprehensive. So, now that you plugged it into an agent, what are the different agent architectures and what are the use cases that are implied by them?

As many of you probably know from building agents yourselves, agent orchestration ranges from more constrained architectures to unconstrained architectures. Constrained is basically you kind of more explicitly define the control flow. Unconstrained is like a React loop, function calling, codecs, whatever.

You basically give it a set of tools and let it run. Deep research is kind of the same thing.

For us, we basically noticed there's two main categories of UXs. There's more assistant-based UXs that can basically surface information and help a human surface information or produce some unit of knowledge work through usually a chat-based interface. It's usually chat-oriented.

The input's natural language. The architecture is a little bit more unconstrained. You know, it's basically a React loop over some set of tools. And it's inherently both unconstrained but also with a higher degree of human in the loop.

So, the goal is, or the expectation is, that the human is supposed to kind of guide and coax the agent along the steps of the process to basically achieve the task at hand. There's a, I mean, there's, I'm sure many of you have built these types of use cases, and so this is just a very small subset.

But it's basically just, you know, your generalization of a RAG chatbot. There's a second category of use cases that I think is interesting, and I think a lot of folks are actually starting to build more into this space, which is this automation interface.

So, being able to actually, instead of providing some assistant or co-pilot to help a human get more information, processing routine tasks in a multi-step end-to-end manner. And usually, the architecture is a little bit different. It takes in some batch of inputs.

It can run in the background, or it could be triggered ad hoc by the human. The architecture is a little bit more constrained, which kind of makes sense,right? If you want this thing to run more end-to-end, you need it to not just go off the rails.

And there's usually a little bit less human in the loop at every step of the process, and usually some sort of, like, batch review in the end. And the output is, like, structured results, integration with APIs, decision making.

After approval, it'll just go route to the downstream systems. Some of the use cases here include, you know, financial data normalization, data sheet extraction, invoice reconciliation, contract view, and more.

I'll skip this video, but, you know, there's some fun example of some community-based open source repos we built in this area, like the invoice reconciler by Lori Voss.

A kind of general idea that we've emerged, that has emerged and we've noticed as a pattern is, you know, oftentimes the automation agents can serve as a backend. Because it runs in the background, you know, can do the data ETL transformation.

They're still human in the loop, but it's kind of the doing the thing where it needs to process and structure a lot of data and do decisions in the background. And then assistant agents are kind of more front-end facing,right?

And so, automation agents can structure, process your data, and provide theright tool interfaces for assistant agents. Not every tool depends on agentic reasoning, but for a lot of these use cases, like for a very generalized data pipeline where you're processing a lot of unstructured context, you might have automation agents go in and process your data, provide theright tools for some sort of more research user-facing interface.

So, we talked about building a document toolbox. We talked about, you know, the different categories of agentic architectures and putting it together. Here are some real-world use cases of document agents. And these are basically examples of agents that actually help automate different types of knowledge work.

So, one of our favorite examples is a combination of both automation and assistant UXs for financial due diligence. Carl L is one of our favorite customers and partners. You know, they basically used some of the core capabilities that we have to build an end-to-end leverage bio agent.

You know, it requires an automation interface to inhale massive amounts of unstructured public and private financial data: Excel sheets, PDFs, PowerPoints. Go through some bespoke extraction algorithms with human-in-the-loop review. And then, once that data is actually structured in theright format, providing a co-pilot interface for the analyst teams to actually both get insights and generate reports over that data.

If you look at any enterprise search use case, that typically falls within the assistant UX. Semex is one of our favorite customers in this space, where, you know, just being able to define a lot of different collections to different sources of data and providing more task-specific, specialized agentic RAG chatbots over your data,right?

You know, it's basically RAG, but you add, like, an agentic reasoning layer on top so that you can basically break down user queries, do research, and answer the question at hand.

And on the pure automation UX side, we noticed a lot of kind of use cases popping up around automation and efficiency. And so, one example is actually technical data sheet injection. You know, we're working with a global electronics company.

They have a lot of data sheets that need to be automatically processed and reviewed, and historically, it's taken a lot of human effort to actually do this. So, by creating theright end-to-end automation agent, you can basically encode the business-specific logic for parsing these types of documents, extracting out theright pieces of information, matching it against specific rules, and outputting the structured data into SQL.

There's human-in-the-loop review, but if we're actually able to do this end-to-end, it transforms weeks of just, like, you know, technical writer work into an automated extraction interface. So, that's basically it. You know, for those of you who are less familiar, LlamaIndex is the most accurate, customizable platform for automating your document workflows with agentic AI.

Our mission statements evolved a little bit since the past few years. We're a very broad, horizontal framework, oftentimes focused on RAG. But if you're interested in some of the capabilities, come talk to us, and then please come check us out at booth G11.

Thank you.