RAG for VPs of AI: Jerry Liu

Hi everyone, I'm Jerry, Co-Founder & CEO of LlamaIndex, and I'll probably spend the first 10 minutes just giving a brief overview— I mean, of RAG and also just, like, LlamaIndex— how we see the enterprise developer space and how it's progressing, as well as give an overview of the product offerings.

And then I think in the next 15 minutes, happy to, like, you know, generally field questions and kind of answer— actually, maybe like, have a discussion on what's top of mind throughout enterprises today. So, let's get started. You know, throughout the enterprise, what we're seeing— and this might resonate with some of you— there's a lot of different use cases that we're seeing pop up, and a lot of it's around RAG,right?

I'm pretty sure we all probably know what RAG is. You know, you point it at some, like, directory of files, and then you get the LLM to somehow understand these files and then generate answers from them. Some other use cases that we see include, like, document processing and extraction, being able to maintain conversations over time.

And then this year there's a lot of people, like, you know, building agents. We haven't seen as many, like, fully autonomous agents in production; they typically are a bit more constrained. But actually curious to get your takes as well, so happy to discuss that.

So, obviously RAG has been a very popular, like, set of techniques, basically, for helping you build a question-answering interface over your data. That's really the end goal, is to help you build a question-answer interface. And what are the main components of RAG?

I won't go into, like, the super technical details, but, you know, you need an LLM to do the final synthesis, you need an embedding model, you need a vector database, or you need some database. Could be, you know, a document store, could be a graph store, could be, like, a SQL database or a vector database.

And then here's the thing that's interesting, is that you basically need a new data processing stack to handle the data parsing and injection side. This is different than traditional ETL, which is primarily for kind of, like, you know, analytics workloads, as well as there's a lot of, like, technologies that popped up around that.

Here, you know, you're really, you know, at a very basic level, say, taking in a PDF, slicing it up into a bunch of chunks, figuring out how to do that well, and index it and represent it in a bunch of different storage forms so that LLMs can have easy access to it,right?

And a lot of what LlamaIndex is trying to solve is on that data processing piece. So, at a very, you know, a big pain point that we see for a lot of companies building LLM applications is going from prototype to production.

Unlike traditional ML, it's actually really easy to build a prototype with, like, some of the tools that LlamaIndex offers. It takes, like, 10 minutes to build a RAG pipeline that kind of works over your data. But going from kind of works to something that's production quality is a lot harder.

And so, as we see you scale up the number of documents, as the documents get more complex, as you try to add more data sources, you have a higher quality bar that you need to meet. And then some of the general, you know, pain points that we see include accuracy issues, knowing how to tune, like, a bunch of different knobs, and then also scaling to a lot of data sources.

Oftentimes this either takes a lot of developer time or they just don't know how to do it, and so what ends up happening is that POC you're building for the higher-ups just ends up not, like, really working, and so therefore, like, the value of that overall project is diminished.

The other problem that we see is that, generally speaking, most of these— a lot of these larger companies that we talk to have a lot of data, and there's this, like, general issue of just, like, data silos,right? You have unstructured data, structured data, semi-structured data, APIs, and somehow, you know, like, this is— a similar problem occurs during the LLM application development where you want to somehow bring in all this data into some central place so that your LLMs can understand it,right?

And when they're able to understand it, and ideally, you know, somehow if you had this magic tool that made that happen and made it work well, then you're able to have this kind of, like, holy grail of RAG, just being able to synthesize answers and do stuff over, like, any of your knowledge anywhere in the enterprise.

A thing that we talk about a lot, both during the keynote yesterday as well as more generally, is the importance of data processing and data quality. Like, you know, we've probably all heard the term, like, garbage in, garbage out, and this is true in machine learning, but this is also true in LLM application development.

If you don't have good data quality, and I can go into an example of what that means, you're not going to get back well-represented information so that even if your LLM is very good, oftentimes if your data quality is bad, this leads to hallucinations within your application.

And so we believe in developers. Like, if you're kind of, you know, leading AI at one of these, like, enterprises, you do want to make a bet on developers and, like, you know, I think generally speaking, and I say this, like, pretty often, you should generally bet on probably building a little bit more than just, like, buying pure out-of-the-box solutions.

And there's a few reasons why this is the case. First, the AI space is moving really quickly. The underlying technology is shifting. Developers are the best positioned to translate that technology into enterprise value that are custom to your use case.

If you, you know, go through the procurement process and purchase, generally speaking, like, out-of-the-box tools that will solve maybe, like, the current pain point that you have around that, like,right, and provide a solution for that. But it will probably be a lot slower to basically adapt it as new techniques pop up, new workflows are possible.

And so we care a lot about developers, and we want to basically provide the tooling and infrastructure to enable developers to build LLM applications over their data. This helps you get applications with high data response quality that's actually ready for production.

And importantly, it's, like, easier for developers to set up and maintain so you don't have to keep throwing developers at it and kind of, like, banging their heads against the wall to figure out how to actually make this thing generate good responses.

And you can scale this to more data sources.

Great. I'm not going to go through, you know, kind of all the different features of LlamaIndex, but I'm just going to quickly run through some of the main components. Our main goal as a company is to help any developer build context-augmented LLM apps from prototype to production.

We have an open-source toolkit,right, and this is an open-source framework. That's a very popular framework to help you build— help developers build production LLM apps over your data. A lot of the use cases that we've seen in the past year have been around, like, you know, productionizing RAG.

In, you know, the next six months, we anticipate a lot more agentic use cases to arise as well. And it's primarily focused on orchestration around, like, retrieval, prompting, agentic reasoning, tool use. The other piece that we have is Llama Cloud, which is a centralized knowledge interface for your production LLM application.

It unifies your data sources, starting with unstructured data, is able to process and enhance that data for good data quality so that you actually have, you know, good quality data from your very complex, like, PDFs and PowerPoints, for instance, and spreadsheets, and helps you build managed pipelines so that you, as a developer, don't have to worry as much about that and can basically worry about building the actually interesting stuff around the orchestration of that data with LLMs.

Yeah, I think I mentioned this already. Open-source toolkit, a lot of people using it. Going to skip this. And then Llama Cloud is, again, this, like, centralized knowledge interface for your production LLM app. You spend— like, the idea is to help manage a lot of the data infrastructure so that developers, generally speaking, have to spend less time wrangling with data and spend more time building some of the core, you know, prompting, agentic, retrieval logic that makes up, like, the custom use case that they want to build for.

I'm not going to run through all the features that we have because this is basically just, like, one of the, you know, some of these things are upcoming. But one specific thing that I think has actually gotten a decent amount of interest from users is Llama Parse, which is a specific component of Llama Cloud.

It's basically our advanced document parser that helps solve this data quality problem. Basically, if you want to build LLM applications over, like, a complex financial report or a PowerPoint with a lot of different messy text layouts, like, tables, images, diagrams, and so on and so forth, we provide a really nice toolkit to basically help you parse that data specifically so that LLMs can understand it and don't hallucinate over it.

So far, you know, we released this, like, a few months ago. There's been some impressive usage metrics so far. Basically, half a million monthly downloads on the client SDK, like, tens of millions of pages processed, and a lot of, like, important customers basically using this throughout the enterprise.

And, yeah, generally speaking, maybe just in terms of, like, discussion topics, I'm happy to talk about any of these components. I'm very interested in, generally speaking, like, kind of the enterprise, like, data stack and how that translates into LLM applications.

I'm also interested on the use case side, basically the kind of, like, advancements from simple QA interfaces into more agentic workflows that can actually take actions and automate more decision-making from different teams,right, either internally or externally. And just a quick shout-out is, you know, we have, like, a general waitlist for Llama Cloud.

That's already gotten pretty popular. There's been a decent number of sign-ups, but there's— the goal is to basically help enable more users to kind of, like, process and index their unstructured data so again, they can help, like, manage that and still build a lot of the kind of, like, important use cases as enterprise developers.

Cool.

Go for it.

I understand Llama Cloud requires people to upload documents to your cloud. How do you deal with customers who are very sensitive to privacy?

Yeah, that's a great question.

Please repeat part of the question because I didn't catch it with the microphone.

Yeah. So the question was about the enterprise product, Llama Cloud, where the understanding is that you upload documents to our cloud, so how do we deal with, like, data privacy? There's two kind of answers to that. The first is that we have both a cloud service as well as a VPC deployment option.

I'm happy to chat about that if you sign up on the kind of, like, contact form. So we deploy in AWS and Azure with GCP coming soon. And then the second is we're all, like, kind of a data orchestration layer, so we actually intentionally don't store your data.

We try to integrate with the existing storage systems. Yeah.

You made a comment on, like, the differences between traditional ETL and, you know, kind of the new skills and tools, et cetera, required. Can you expand on that a bit so that, you know, in my company where I get, like, asked maybe, "Hey, let's have this ETL person who's done a lot of other ETL do it, what kind of instruction would I give them on, like, hey, these other skill sets or tools might be necessary?"

And if there's any other sort of gotchas around that, if you could highlight those, that'd be great.

Totally. I think just on a very technical level, the steps you actually take are just different. Basically, instead of writing, like, SQL or using dbt, you just, you know, this is how you, like, set up a RAG pipeline.

You have a PDF. First, you need to parse that PDF, so either using Llama Parse or another document parser. That parsing step, if you don't get itright, then that leads to a lot of kind of, like, downstream failure modes in your LLM application.

After you parse the document into some representation, whether it's text or increasingly we're seeing, like, multimodal representations as well with, like, image representations of a document, you then need to chunk that document,right? And so the very naive approach is you basically set a chunk size of, like, 1,024 tokens and you split every 1,024 tokens,right?

And that specifically also, you know, introduces a bunch of complexities because if you split, like, tables down the middle, you split pages that, like, there's, like, a section that spans multiple pages or something, you somehow need to better, like, semantically join them together so that, like, most information is preserved within a chunk and that you add, like, theright metadata to that chunk.

And then you need to figure out a good way to index it. And this is where, like, a vector database or a graph store or document store comes in. There's a lot of different ways to index it. So just very fundamentally, it's just, like, a different set of, like, steps you need to do.

And the issue here and the difference actually with traditional ETL is all these steps are kind of, like, fuzzy to understand without the end-to-end performance. Like, with traditional ETL, you know, it's kind of like you do some step and then you know exactly what you want.

Here, like, it's really hard to tell what the chunk size you need to set is without having an eval dataset and having a rigorous end-to-end testing and eval flow. Yeah. Oh, sorry. Oh, just want to make sure I think I saw a hand over there.

Yeah. Do it. Yeah.

How do I integrate audio files?

We'll figure it out.

So with LlamaIndex pipelines, would it be a reader or a parser? I'm building something specifically that does semantic chunking and paragraphing from transcript. So what would be the desired route to integrate that?

Yeah, so I think we have a few audio loaders. So I think the default is just take—so the question was basically how do you integrate audio sources into your RAG pipeline using, you know, LlamaIndex or other frameworks. The simplest is probably just, like, you just directly, like, parse that into text and then ingest it.

I think in the future, as models become more natively multimodal, you might just be able to represent audio as, like, a specific entity,right, and then as a chunk almost and directly feed that into a model. But I don't think we're there yet.

And then, okay, I'm going to go this way. Yeah.

Really a question to go about how do you determine whether your parsing is better than other factors? Are there data sets, are there eval sets that are public and you can say, yeah, our Llama parser is so much better than?

For sure. I think the benchmarking is important. It's also challenging because we're actively working on thatright now to basically find a general benchmark. What typically happens is we do, like, just within the enterprise, they just do a bake-off on their own data and then compare it.

And we basically show them a notebook on, you know, here's how you build a RAG pipeline with Llama Parse, here's how you could do it with other parsers. Yeah. Just want to make sure I cover. Yeah.

On my case.

Yeah. My question is, what options do you have, like, for versioning or different promotion across environments to, you know, do staging and production? That's one part. And the other one is, what regions are you available? So that's maybe a little more easier.

Yeah. I think the versioning piece is definitely important. I think at a high level, we are building out features to help you, like, better version your pipelines. We don't have that yet, but it's kind of, like, upcoming and also requested by some enterprise customers.

And then the second question around kind of regions where the SaaS service is in North America. It's just hosted. But we do kind of, like, on-prem deployments as well,right? And so that's part of, you know, generally the enterprise plan that we offer.

Yeah.

Hi. I'm building a RAG system for a big fintech, basically a bank. The struggle I'm having is I'm obviously working with the servicing team, which has other channels,right? I'm working on a chatbot and a WhatsApp chatbot. The servicing team also has, say, like, a help center, an IVR, a bunch of other, like, channels,right?

It's been very tough for me to convince them that maybe the CMS that they're using, you know, to feed these other sources is not the best way to feed a RAG. I'm curious to know if you've seen other customers that have, like, a similar issue where, you know, internally they want to have, like, this single source of truth that kind of feeds into all of these channels where the RAG system's nature is obviously extremely different than, like, a help center or an FAQ or that kind of stuff.

I see. Wait, so why is that CMS not theright tool to basically?

I'm curious to know if you think that could be theright tool or, like, getting a little bit more into the details, that's, like, we have, like, Q&A pairs. That's how the CMS worksright now, which could work for RAG, but we're missing all the metadata, the different clusterings of, like, different documents for different maybe use cases, different credit cards.

It's a little bit tough to explain in a quick question, but, like, have you seen a single system work as a single source of truth and kind of how have you seen that work, like, in real big use cases?

Yeah. So I think the, yeah, I think the full details, there's probably, like, a lot to dive into there. I think generally speaking, what we see is for, like, homogeneous data sources where it's, like, of the same kind of, like, data type, let's say it's all, like, financial reports, you can generally use, like, the same set of parameters to basically parse it because there's, like, an expectation they're roughly the same format.

For very diverse and different data sources, like, if all of a sudden you're bringing in not just, like, PDF documents, but also, like, semi-structured data from, like, you know, Jira or something or what was it, like Salesforce, for instance, like JSONs, you typically need to set up, like, a separate pipeline there.

And then what, you know, we both offer on the open source, but also the enterprise side is this ability to, like, combine all these different data sources and then you just have to, like, combine them together and rerank them,right, and have some reranking layer at the top.

Allright. Thank you.

Yeah.

So I've been using—

Hang on. I'm going to give you the mic just because it's for the—

Sorry.

It's for the recording.

So I've been using Llama Parse for a little bit and first of all, I love it. So it works really well. So thank you for producing it. However, two weeks ago, I was working on a project for a client and all of a sudden I was getting all these failures and I contacted support via the chat and there was a gentleman helping me out and he's like, "Go pass me the job IDs."

Gave me the job IDs and all of a sudden just went MIA and never replied back. So the question is, what are the support options? So in case I get stuck over the weekend, I could actually get somebody to help me out.

Totally. First of all, I'm sorry you ran into those issues. I know we had, like, a cluster of just failures. I think that specific weekend, it was just, it was a good lesson for us,right? Keep in mind we're all, like, 15 people at the company.

And so when you talk to support, it's probably, like, one of the founding engineers just, like, jumping in. So I promise we're making that process more streamlined. Typically on the enterprise side, like, especially for kind of, like, the enterprise plans that we offer, I'm happy to chat about this offline, like, we offer dedicated SLAs,right?

And so this is kind of, like, there's some support option we're doing on the casual, like, kind of, like, self-serve APIs, but we're offering, like, dedicated SLAs on the enterprise.

Hey. So we are building hallucination detection and other evaluation systems for customers that have a very large collection of documents. And typically that's, like, you could have, of course, like, thousands of PDFs and all sorts of PDFs typically, of course, contain a lot of tables and all that.

And then there's a question of how to combine, like, OCRs and other heated processing on that. So the question is, like, what is your general recommendation? Like, does Llama Parse take care of all this or do you recommend, like, building some kind of custom system directly on top of LlamaIndex or how would you recommend handling that?

Yeah. I think, I mean, I guess I didn't actually show the capabilities of Llama Parse in these slides, but maybe if I dig around a little bit, I can try to find the specific slides where it showcases. Yeah, like, basically what you want when you parse these documents is you want some generally good parser that will lay out the text, like, in a spatially aligned way.

And so it doesn't matter if you have all the bells and whistles of, like, bounding boxes and all these things. You generally, like, bare minimum, like, just want the text to be, like, faithfully represented. And that's exactly what Llama Parse does, especially for, like, tables.

So we have a few examples, for instance, where, like, you have tables within a document and then you lay it out in a spatially aligned way. And then when you feed this to an LLM, LLMs generally are trained to respond pretty well to, like, well-spaced, like, just formatted pieces of text so they can understand what's going on in that text chunk.

Whereas if you use, like, a very naive parser, like a baseline PDF parser, it's going to, like, collapse the text to numbers and therefore kind of, it's going to generate a lot of hallucinations. But, yeah. Yeah.

With the increase in size of context windows that are available to us and also the improvements that we're finding for, like, dealing with the Haystack kind of problems, what is your perspective on where we're headed towards RAG?

Yeah. I think there's two general trends. One is longer context windows. The other is, like, multimodality. I do think there's a few things that will probably go away and a few things that will stay. One is good parsing is still important.

The reason is, like, you know, in the end, if your parser is bad, you're just going to feed bad data into the LLM and it's going to hallucinate information. What I think will probably go away is as context windows get bigger, chunk sizes can also get bigger.

So, you know, you are probably not going to need to worry about, like, intra-page splitting, like splitting a single page into a bunch of smaller chunks. In the future, we could see you just, like, putting entire documents as chunks and basically indexing stuff at a document level.

I think that actually makes a lot of sense because documents are typically, like, self-contained entities and I think that'll make it a lot easier for developers. However, in general, for a multi-doc system, which, you know, if you're in a company, you probably have, like, billions of documents, many gigabytes of documents, you're probably not going to feed all billion documents into the context window on every inference call, even with context caching, which I think Gemini has, because context caching isright now super expensive, probably doesn't make sense from a cost perspective, and also is a black box, so you don't get accountability into the data.

You basically store the transformer weights for those of you who, like, kind of are familiar with that, and you don't really get, like, full transparency into what the data is actually being fed into the language model at each step.

So actually, I think for a variety of reasons, the overall idea of retrieval from an external storage system, whether it's a vector database or a graph database, still matters for a variety of reasons. But, you know, the minute chunking decisions will probably go away.

The second thing, which you didn't ask about, which I'll talk about anyways, is multimodal. I think as multimodal models get better, I think it actually makes sense to basically start having, like, diverse representations of the same thing. So, for instance, if you have a PowerPoint presentation, you're able to, like, represent each page, for instance, as an image in addition to just, like, parse text.

And by storing native image chunks, you basically preserve all the information within that data. Anytime you do parsing, it's inherently lossy,right, because you're inherently, like, trying to extract out stuff in, like, a textual format as opposed to preserving the full picture.

And by having, like, different ways of representing the same amount of data, you can basically trade off between, like, cost, performance, and latency.

Just checking the.

Hi. So I see you've done a lot of work improving the accuracy, reduce the hallucination. I wonder if you are working on anything to make the conversation flow better. In my experience, it's so hard to get the conversation to feel natural.

Sometimes I overemphasize the context data while I just want to give it a FYI and just continue talking like a normal human.

So you're talking about, like, basically how to create more natural conversation flows? That's, yeah, I think, so that's very interesting. I think

the overall answer to that is I think the default way most people are building these conversation flows is you have some, like, say, RAG pipeline as, like, a tool,right? And then you basically have an agent as an outer layer that reasons over the conversation history and can basically, you know, synthesize theright answer at a given point in time.

So the knobs basically that you want to tune are the agent reasoning, like, prompts as well as the memory. And I think the memory is actually pretty important becauseright now most memory modules are, like, very primitive. There's not a lot of good things beyond just, like, dumping the conversation history into the prompt.

So happy to chat more about that as well, but I think there's, like, a lot of stuff there that you could probably try. Just want to double-check the time.

We are at time.

Oh, okay. Oh, wait, okay, no worries. Yeah.

How are you using Llama agents internally? What's the most complex task?

That's a great question. So for those of you who weren't at the keynote, we launched this thing called Llama Agents, which is an open-source multi-agent framework basically for helping you basically deploy agents as microservices. Right now, agents primarily live in, like, notebooks, and the idea is to spin them up as, like, API services.

Right now, I think we're mostly just, like, using it to build, like, kind of more constrained, simple RAG pipelines. And it's actually still in an alpha state, so I encourage all of you to basically try it out. There's a lot of things that I already know it can't do.

For instance, have, like, more general kind of, like, there's, like, communication protocols and interfaces that we want to build in, a more interesting message queue system. But, you know, if you have an enterprise use case that's, like, going agentic and you want to basically kind of understand it as microservices so that you can basically reuse and encapsulate it, please check it out, come talk to us.

But, cool.

Fantastic.

Thank you. Yeah, sorry for going over.

No, that's all fantastic.