Production Evals For Agentic AI Systems - Nishant Gupta, Meta Superintelligence Labs

Hey everyone, my name is Nishant Gupta, and I'm a Software Engineering Tech Lead at Meta, working on building the training and inference Infrastructure for the Meta Superintelligence Lab and their Infrastructure Organization. Today we're going to be talking about production evals for agentic systems.

When most people hear the word "evaluation," they think about benchmarks: a model scores 90% on a benchmark, a new version scores 92%, the team celebrates. But agentic systems have fundamentally changed what "evaluation" means. Today, the systems don't simply generate answers; they plan.

They call tools, they retrieve information, they execute workflows, they interact with the production infrastructure. The question is no longer, "Did the model generate theright answer?" The question is, "Did the system behave correctly?" Today I would like to discuss how evaluation is evolving from model benchmarking into production infrastructure.

This is the problem almost every AI organization is encountering today: offline benchmarks continue improving, yet production reliability often remains unpredictable. Why is that? Because benchmarks measure model capability; production measures system behavior. A benchmark doesn't capture tool failure, API outage, context changes, user variability, long-running workflows.

And as systems become more autonomous, the gap between the benchmark performance and production performance grows. The result is what many teams experience today: high benchmark scores, as you can see, but unreliable production behavior.

Traditional LLM evaluation focuses on outputs, but we should ask the question, "Did the model produce a correct answer?" Agentic systems force us to ask a different question: "Did the system behave correctly?" Behavior includes planning quality, tool usage, execution, workflow execution, recovery from failures, decision-making.

In other words, we're moving from evaluating answers to evaluating workflows, and that requires fundamentally different evaluation architectures. Many teams still think hallucinations are the primary AI failure modes. In production, they're often just one category. Agentic systems introduce an entire hierarchy of failure modes.

At the very foundation: the memory failures, retrieval failures, safety failures. As you go up, you have to think about reasoning mistakes, poor planning, incorrect tool execution. At the highest layer, you have to think about multi-agent coordination failures. And this is why evaluating only model output misses the most production risks we observe.

One of the most useful mindset shifts is to stop thinking like researchers and start thinking like an SRE or a production engineer. SREs don't measure success using accuracy; they measure reliability. Availability, latency, cost, recovery. And agentic systems require the same approach.

The goal is not maximizing the benchmark scores; the goal is to maximize dependable outcomes. Reliability becomes the non-star metric. Accuracy becomes the only input.

In this pyramid is how I personally think about modern AI evaluation systems. At the bottom you can see: there are benchmarks. They're useful, they're scalable, they're repeatable, but their operational value is limited. In the middle, there are scenario-based evaluations.

These simulate realistic workflows. And at the very top, you see production telemetry. This is where the highest value evaluation signals come from. The surprising insight is that the most evaluation data often comes from real users interacting with real systems.

Now let's talk about offline evals. So offline evaluation still matters, but the methodology changes. Instead of evaluating prompts, we evaluate scenarios. For example: a customer support workflow, a code generation workflow, a research workflow. The agent operates inside that simulated environment.

We measure the task completion rate, tool correctness, planning quality, resource usage, which becomes exponentially high at high scale. The key takeaway: agent evaluation should be scenario-driven, not prompt-driven.

Once a system reaches production, every interaction becomes a signal. This is one of the biggest shifts in evaluation thinking. Production traffic is no longer just traffic; it becomes evaluation data. We collect execution traces: user outcomes, escalations, failures, feedback signals.

Production is the largest and the most representative evaluation data any organization will ever have.

For many organizations, we use humans as fallback systems. I think that's a wrong framing. Humans are the evaluators. They provide signals that automated systems cannot. They assess correctness, trust, usefulness, safety. These signals become really critical for calibrating evaluation pipelines and identifying blind spots in automated metrics.

The most successful systems combine automated evaluation with targeted human review.

Now, agentic systems drift constantly. Model changes. You have a new version every couple of weeks or months. The prompts can change, tools can change, user behavior can change. The challenge is that no longer a single change appears catastrophic.

Reliability slowly degrades. Success rate declines, escalation increases, tool failure rises. Without continuous evaluation, teams often don't discover drift until users complain. Continuous monitoring becomes essential.

Observability and evaluation are inseparable. To evaluate an agent, we need visibility into the reasoning paths: the tool calls, the memory access, execution timelines, the straight transitions, as you can see here in this chart. Traditional logs are not sufficient.

We need detailed traces, just like with any deep nested microservice architecture for any application or service we're talking about. Agent traces become the equivalent of distributed tracing for autonomous workflows. Without observability, evaluation becomes the guesswork.

Now let's talk about the continuous evaluation loop, because evaluation is an always-running service, not a testing phase. Historically, evaluation always happened before deployment, but now evaluation continues after deployment. Telemetry identifies issues, as you can see in a human reviews the edge cases.

Feedback improves the datasets. Offline scenarios validate updates. The loop never stops. Evaluation is no longer just a phase; it's an operational capability. Now, this is probably the most important slide in this presentation. Every metric shown here maps directly to a business outcome.

Task completion measures value delivered. Tool success measures operational reliability. Escalation rate measures human burden. Safety evaluations measure risk exposure. Latency affects user experience. Cost determines scalability. Recovery rate reflects resilience. And notice, but notice that accuracy is missing. It's not because accuracy doesn't matter, but because business success depends on much more than just accuracy.

Now, this is the architecture where the industry is heading, more or less. Evaluation becomes part of the control plane. Not a separate tool, not an offline process. The control plane continuously, which observes the systems, collects telemetry, runs simulations, coordinates human review, and the execution plane performs the work.

The control plane measures and governs the behavior, and this separation is becoming a foundational pattern for production AI systems.

Now let's summarize the key lessons. First, benchmark remains necessary, but they are insufficient. Second, agentic systems must be evaluated as workflows, not individual outputs. Third, production telemetry is the most important evaluation signal. Fourth, reliability ultimately matters more than raw model accuracy.

And finally, evaluation is becoming the infrastructure. Not testing, not QA, infrastructure. This is the shift every organization building agentic AI will eventually need to make.