Effective AI Agents Need Data Flywheels, Not The Next Biggest LLM – Sylendran Arunagiri, NVIDIA

Hi there, my name is Sylendron. I work with NVIDIA on the generative AI platforms team. In this video, we'll look at what it takes to build effective AI agents that always stay relevant and helpful over time. And it's not about powering them up with the next biggest LLM in the market; all they need is simple data flywheels.

In this video, we'll look at what data flywheels are and how we applied them to an internal agent at NVIDIA. I'll also share with you the lessons that we learned along the way, and most importantly, I'll leave you with tools and techniques and a framework to think about how you could build your own data flywheels for your own agentic AI use case.

Let's get started. So, AI agents are generating a lot of buzz recently, and it'srightfully so. They're already starting to make their way into your workforce as new digital employees. They come in different forms and sizes depending on the use case that you apply it to.

They exist as customer service agent, software security agents, research agents, and etc., etc. But what really are agents? If you look at the definition of agents, you will see agents are systems that are able to perceive, reason, and act on the underlying task.

That means they're able to look at data, they're able to reason and come up with a reasonable plan to act on the specific user query, and they make use of tools and functions and external systems to achieve the underlying task.

But in my opinion, what completes that cycle is if AI agents are able to capture and learn from user feedback along the way, and thus learning from that user preferences and data, and constantly refining themselves to be more accurate and useful for the underlying task.

Right, but we all know building agents can be very painful, and scaling them can be increasingly challenging for sure. For once, the data changes rapidly. If you are an enterprise customer and you're using an agentic system for a use case, you constantly see new data business intelligence flowing into your system.

And augmenting that is changing user preferences and customer needs. Right, and if you're looking to deploy large language models, bigger chunkier models, to support the underlying use case, you also see that your inference cost is going to go high.

So increased usage drives increased cost.

That's exactly where data flywheels can help. At its core, it starts with the enterprise data. Data flywheel is a continuous loop or cycle of data processing and curation, model customization, evaluation, and guardrailing them for safer interactions, building state-of-the-art RAG pipelines alongside enterprise data to provide relevant accurate responses.

So as AI agents run in production environments, this data flywheel cycle triggers a continuous cycle of data curation, of ground truth using inference data, business intelligence, user feedback, to continuously experiment and evaluate existing and newer models to surface efficient smaller models that provide at-par accuracy expectations as that of larger language models, but which offer lower latency, faster inference, and effectively lower cost to ownership, total cost of ownership, or inference cost.

Before we go into how do you build your data flywheel, here's something exciting that we announced a few weeks back at NVIDIA. We announced NeMo microservices. It's an end-to-end platform to build powerful agentic and generative AI systems, and also powerful data flywheels around them.

It's a model microservice that has various components for each stage of this data flywheel loop. There's NeMo curator that helps with curating high-quality training dataset, including multimodal data. NeMo customizer helps with fine-tuning and customizing underlying models using state-of-the-art customization techniques like LoRA, P-tuning, full SFT, and we're continuously adding more fine-tuning capabilities.

NeMo evaluator to evaluate benchmark on academic benchmarks, institutional benchmarks, as well as using LLM as a judge. NeMo guardrails for providing guardrail interactions for privacy, security, and safety, and NeMo retriever to build state-of-the-art RAG pipelines. And the best part about this NeMo microservices is the ease of use.

They're exposed as simple to use API endpoints, and with just a single or few API calls, you're able to customize large language models, evaluate them, guardrail them for the underlying use case. And you also have the flexibility to run these microservices anywhere, on-prem, on cloud, on data center, even on the edge.

And with NVIDIA support, you get the enterprise-grade stability and support as well. Now, here is a sample data flywheel architecture leveraging NeMo microservices. I like to think of it as Lego pieces where I can just put together these NeMo microservices components to build this entire data flywheel setup.

So in this sample architecture, you can look at the end user who's interacting with the front end of an agent. Think of it maybe, say, a customer service agent. It's a guardrail for safer interactions behind the scenes. And on the back end, there is a model that is served as an NVIDIA NIM for optimized inference.

So to figure out what's theright model to power this use case without compromising on accuracy, you can set up a data flywheel loop to constantly curate the data, store it in NeMo data store, and use NeMo customizer and evaluator to trigger a continuous cycle of retraining and evaluating.

And once you see a certain model meet the target accuracy, you can use your IT admin or an AI engineer can promote this model to power the agentic use case as the NVIDIA NIM underlying this. So that's a sample architecture.

Let's take a look at a real-world case study on how we adopted and built this data flywheel for our NVINFO agent, for an internal employee support agent that helps NVIDIA employees with access to enterprise knowledge across multiple domains.

It's a customer service or an employee support chatbot agent. It can help answer queries across a variety of domains, from HR benefits, financial earnings, IT help, product documentation, and anything and everything that an internal employee needs access to.

So here's a quick look at the underlying data flywheel architecture that we built for this particular agent. So just like we saw, a user, in this case an employee, when they submit a query to this particular agent, it's guardrail for safety and secure interaction.

And underneath, this most important router agent, which is run by an LLM, is orchestrated by multiple expert agents underneath. So each of these agents do a great job in their individual specific domain, and they're augmented with, you know, a RAG pipeline to fetch the relevant information to answer the user query effectively.

So in order to decide what models to use and power each of these green models that you see, we have a data flywheel loop set up which constantly builds on top of the user feedback and production data inference log that come in when this router is in action.

So using a subject matter expert and human in the loop feedback, the ground truth is continuously curated. NeMo customizer and evaluator are used to constantly evaluate multiple models to promote the most effective model as a NIM to power this router agent.

And if you're curious to know what these red ones are, since this is a multimodal agentic system, the red ones are essentially everything that could go wrong with this multimodal agentic system. But for the sake of this exercise and understanding a concept in detail, let's look at one of these in further detail.

Let's look at the router agent since we've been spending some time talking about it. So it's basically orchestrated by a mixture of agent architecture underneath. So the router agent, depending on the user query, understands the intent and context and guides or routes the query to one of these expert agents, which further uses the RAG pipeline underneath to fetch the relevant information.

So the problem statement is, given a user query, which of the agents, the expert agents, should the router route the query to? And the goal here is to ensure that our agent, the routing agents, accurately routes it to the correct expert agent, but also using faster and cost-effective LLM to achieve this task.

So to start off, we compared and contrasted a variety of models to power this router agent. When we used a 3.170b or 3.370b variants, we noticed a 96% baseline accuracy that is just deploying the foundational pre-trained model as-is, without any fine-tuning or any actions on that.

And we noticed that the baseline accuracy was 96% in terms of routing the query to theright expert agent. But if you see the smaller variants, the accuracy is subpar. The 8b variant is below 14%, and that's not ideal for our use case.

And that's exactly where most of the enterprise evaluations get it wrong. So they see a bigger model deliver state-of-the-art amazing accuracy of 96%, and there's a smaller model that gives you low accuracy, and they figure that out, hey, we need to go with a 70b variant because we're seeing a 96 to a 14% comparison.

I don't think even with fine-tuning we would be able to get there. But that's exactly where data flywheels can help you. So what we did was we ran the 70b Llama variant, and we used and circulated a feedback form among NVIDIA employees, and we asked them to submit a few we asked employees to submit a few queries and capture the feedback in terms of whether the response was useful to them or not.

Based on that, we were able to curate around 1,224 data points. 729 of them were incorrect, sorry, satisfactory responses, and 495 of them were unsatisfactory responses. There was something wrong that our employees were not happy with. So here's what we did.

We used NeMo evaluator and used an LLM as a judge to investigate these 495 unsatisfactory responses. We figured out that 140 of them were due to incorrect routing. And using a subject matter expert and further manual analysis, we found out 32 of them were truly due to this incorrect routing.

So we figured out and set up a ground truth data point, dataset of 8685 data points, and we split them into 60-40 to train or fine-tune smaller models, and 40 to test and evaluate the accuracy. So you will be surprised to know, just with 685 data points, the results that we achieved were outstanding.

And this was possible only because of the data flywheel setup. So the 70b variant, like we saw, provided us or gave us an accuracy of 96% for accurately routing it to theright expert agent. But it came with a latency of 0.26 seconds

to generate the first token response. Whereas the 8b variant gave us an accuracy of 14%, but the latency was almost 78, 70% lower. So on further fine-tuning various variants like the 8b and 3b models, we observed that the 8b, with a bit of fine-tuning, was able to match the accuracy of a 70b variant.

And depending on how critical, the mission-critical your use case is, you can even look at the 1b variant. It's got a 94% accuracy. That's not bad. It's just 2% below 70b. So like I said, you can take a trade-off between how accurate you want your agent to be versus the cost that you want to manage and the resource that you want, resource that you want to manage underneath.

So by deploying, let's say, a 1b model, you're looking at a 98% savings in terms of lower inference cost, and also 10x, 70x model size reduction with 70x lower latency. So that's the magic of data flywheel. And imagine building an automated loop or a cycle of continuous and periodic evaluation and fine-tuning as newer models are published into market and effectively surfacing smaller models that can run your agentic workflow to replace larger models that you already had in production.

Continuously learning from the ongoing production log and knowledge, and using that to train smaller models is the power of data flywheel. So as next steps, I hope you will explore NeMo microservices and NVIDIA NIM to build your next agentic use case and also the data flywheel.

But here's a good framework to think about. How do you build effective data flywheels?

And so the first step starts with monitoring user feedback. Start thinking about intuitive ways to collect user feedback signals,right? Intuitive user experience, privacy compliance, implicit signals, and even explicit signals to understand whether the models that are powering your agentic system is going through model drift or any inaccuracies.

And spend time analyzing and attributing the errors or model drift that you see as to why the agent is behaving a certain way. Classify those errors, attribute those failures, create that ground truth dataset that can be further used for the third and fourth step of plan and execute.

The planning is where you identify different models, generating synthetic datasets, experimenting with them, fine-tuning them, and understanding and optimizing your resource and cost, and putting them into action by executing it. Execution does not just mean triggering a data flywheel cycle, but also setting up a regular cadence or a mechanism to track your accuracy, latency, monitoring performance, and production logs, and truly managing your end-to-end GenAI ops pipeline.

So that's a good framework to think about building data flywheels. And hope this was super helpful for you to get started with building your own data flywheels for your agentic use case. So that's all I have for you in this video.

Hope this was informative and you start thinking about building not just agentic use cases, but also data flywheels around them, and hopefully with NVIDIA tools and frameworks. Thank you. Have a wonderful day.