Deterministic Infra for Non-Deterministic AI Agents - Nishant Gupta, Meta Superintelligence Labs

Hey everyone, my name is Nishant Gupta. I'm a Software Engineering Tech Lead at Meta, working on building the training and inference infrastructure, and today we're going to be talking about building deterministic infrastructure for non-deterministic AI agents. So, most of the conversations around AI over the last few years has been focused on models: bigger models, more parameters, better reasoning.

But as organizations move from chatbots to autonomous agents, a different problem emerges: the challenge is no longer intelligence; the challenge is reliability. At Meta and across the industry, we're seeing agents move beyond answering questions and beginning to plan, call tool calls, coordinate workflows, and make decisions that affect production systems.

These systems are fundamentally probabilistic; infrastructure is not allowed to be. Today I want to discuss this topic in more detail.

So modern cloud infrastructure evolved around a set of assumptions: requests. Most of the requests are short-lived. Services are deterministic, more or less. Execution paths are known. Failures are bounded. However, autonomous AI agents violate nearly every one of those assumptions.

They're stateful. They're long-running. They make decisions dynamically. They may execute different workflows for same inputs. This is what I call the great mismatch. We're trying to run autonomous systems on infrastructure that was designed for deterministic workflows.

This is probably the most important mind shift: most AI demos showcase capability, but can it solve a problem? Can it use a tool? Can it complete a workflow? Production systems have a different objective. Can it do it reliably?

Can it do it 10,000 times, 100,000 times, million times? Can it recover from failures? Can it operate safely? Can it do it at an acceptable cost, with an acceptable latency, with an acceptable outcome? The majority of the engineering effort moves below the model layer into orchestration, monitoring, safety, evaluation, and recovery systems.

When people hear AI failures, they immediately think hallucinations. In reality, hallucinations are often the least interesting failure mode. What we see instead are infrastructure failures: recursive reasoning loops, overflowed logs, retry amplification, context corruption, memory poisoning, cost explosions. The model makes mistakes, but however the infrastructure turns that mistake into an outage.

That's the real challenge.

So as this slide shows a pattern that distributed system engineers will probably recognize immediately: an agent calls a tool incorrectly. The tool returns an error. Instead of recovering, the agent generates a slightly different but still invalid request. The cycle repeats.

Each retry consumes more compute. Reasoning depth increases. GPU consumption rises. Eventually, you get exponential resource growth. What started as a minor API error became a compute incident. This is why uncontrolled retries are one of the biggest risks in agentic systems.

This is the architecture principle I recommend most strongly: never let the model directly control production systems. The model should generate proposals. Infrastructure validates them. Policy engine approves them. Execution gateway enforces them. The model just suggests. The platform decides.

This separation allows us to build reliable systems even when the underlying model remains probabilistic.

As we know, containers gave rise to Kubernetes. Microservices created service meshes. AI agents are creating something new: an agentic control plane. This layer becomes responsible for scheduling, memory coordination, policy enforcement, valuation, monitoring, workload routing, which is very important.

And think of it as an operating system for autonomous AI. The organizations that build this layer will have significantly more competitive advantages.

So traditional logs tell us what happened. Agentic systems require understanding why it happened. We need traces to capture planning decisions: tool calls, memory lookups, straight transitions. When debugging an autonomous workload, understanding the chain of decisions and reasoning is often more important than the final output.

Observability becomes multi-dimensional. Without it, production debugging becomes nearly impossible. So as you can see, memory is one of the most underestimated challenges in agentic architectures. Once multiple agents share state, familiar distributed system issues appear: stale reads, conflicting updates, context drifts, inconsistent views.

The challenge becomes even harder when memory itself may be probabilistic and retrieval-based. Many multi-agent failures are actually consistency failures masquerading as reasoning failures.

So safety cannot be a single component. It must be layered: prompt-level controls, tool permissions, policy validations, human approvals, audit systems. Each of these layers catches a different class of failures. Defense in depth is a well-understood security principle. It applies equally well to autonomous AI systems.

Many people frame human involvement as temporary necessity. I don't think that's correct. The most successful systems are likely to remain human-supervised. Humans become exception handlers. They review ambiguous situations. They handle novel scenarios. They provide calibration signals. The goal is not to remove humans.

The goal is allocating human attention where it provides the maximum value.

So one of the biggest infrastructure shifts is that AI workloads increasingly resemble cluster scaling problems. Demand is bursty. Reasoning depth is unpredictable. Workflows may run for minutes instead of milliseconds. Resource requirements vary dramatically. As a result, GPU efficiency, workload placement, elastic capacity management, and scheduling become critical.

Inference is no longer just a model problem; it becomes a resource orchestration problem. The good news is that many of these problems are not entirely new. Distributed systems have solved something similar for decades. Circuit breakers become tool isolation.

Rate limits become agent limits. Retries become controlled recovery. Resource quotas become cost governance. Observability becomes agent tracing. Instead of inventing entirely new infrastructure, we can adapt proven reliability patterns to autonomous systems. The industry has gone through several phases.

The initial prompts were the differentiator. Then the models became the differentiator. Now both are rapidly commoditizing. The next frontier is infrastructure. The organizations that won't necessarily have the best prompts. They'll have the most reliable systems. The competitive advantage is moving up the stack.

If there's one thing I want you to remember, it's this: AI agents should be treated as distributed systems. Models are stochastic. Infrastructures must be deterministic. Reliability is increasingly an infrastructure problem. Observability is mandatory. Control planes are emerging as the foundation layer.

And ultimately, the future of the AI won't be won by better prompts. It will be won by better systems.