# Deterministic Infra for Non-Deterministic AI Agents - Nishant Gupta, Meta Superintelligence Labs AI Engineer ยท 2026-06-29 Nishant Gupta, a Software Engineering Tech Lead at Meta Superintelligence Labs, argues that AI agents must be treated as distributed systems and that infrastructure, not prompts or models, will determine competitive advantage. He highlights the 'great mismatch' between deterministic cloud infrastructure and probabilistic, stateful, long-running agents, which leads to failures like recursive reasoning loops and retry amplification. Gupta advocates for a control plane that separates model proposals from production execution via policy engines and execution gateways, and emphasizes layered safety, multi-dimensional observability, and controlled recovery patterns adapted from distributed systems. He notes that memory consistency is a critical challenge, and that human oversight should remain as exception handling rather than being eliminated. The episode concludes that the future of AI will be won by better systems, not better prompts or models. ## Questions this episode answers ### What is the recommended architecture for building reliable AI agents that interact with production systems? Nishant Gupta recommends a strict separation: never let the model directly control production systems. Instead, the model should generate proposals; infrastructure validates them, a policy engine approves them, and an execution gateway enforces them. This means the model only suggests actions, while the platform decides. This separation creates a deterministic foundation around the stochastic model, enabling reliable systems even when the underlying AI remains probabilistic. It prevents the model's non-determinism from causing infrastructure outages. [3:08](https://aie.addtry.com/853de319-4b2e-4836-8a74-9114fb00e79a?t=188000) ### How do retries cause compute incidents in AI agent systems? Gupta describes a failure pattern where an agent calls a tool incorrectly, receives an error, and then generates slightly different but still invalid requests in a recursive loop. Each retry consumes more compute, increasing reasoning depth and GPU usage exponentially. What starts as a minor API error can escalate into a major compute incident due to uncontrolled retries, making retry amplification one of the biggest risks in agentic systems. [2:33](https://aie.addtry.com/853de319-4b2e-4836-8a74-9114fb00e79a?t=153000) ### What memory consistency challenges arise in multi-agent AI architectures? According to Gupta, when multiple agents share state, classic distributed system issues like stale reads, conflicting updates, context drift, and inconsistent views appear. The problem worsens because memory itself can be probabilistic and retrieval-based. Consequently, many failures that appear as reasoning errors are actually memory consistency failures, making memory coordination one of the most underestimated challenges in agentic architectures. [4:16](https://aie.addtry.com/853de319-4b2e-4836-8a74-9114fb00e79a?t=256000) ## Key moments - **[0:00] Intro** - [0:27] Nishant Gupta: "The challenge is no longer intelligence; the challenge is reliability." - **[0:58] The Mismatch** - [1:14] Nishant Gupta: "We're trying to run autonomous systems on infrastructure that was designed for deterministic workflows." - **[1:31] Production Reality** - **[2:06] Real Failures** - [2:33] Uncontrolled retries in AI agents cause exponential resource growth, turning minor API errors into compute incidents. - **[3:08] Deterministic Layer** - **[3:58] Observability** - [3:58] Nishant Gupta: "Understanding the chain of decisions and reasoning is often more important than the final output." - **[4:16] Memory** - **[4:51] Safety Layers** - **[5:10] Human Oversight** - [5:10] Human supervision of AI agents is not temporary; humans become exception handlers, reviewing ambiguous situations and providing calibration. - **[5:36] GPU Orchestration** - **[6:13] Infra Patterns** - **[6:49] Takeaways** - [6:49] Key takeaway: AI agents should be treated as distributed systems, with deterministic infrastructure for stochastic models. ## Speakers - **Nishant Gupta** (guest) ## Topics Agent Engineering ## Mentioned Meta (company), Kubernetes (product) ## Transcript ### Intro **Nishant Gupta** [0:04] Hey everyone, my name is Nishant Gupta. I'm a Software Engineering Tech Lead at Meta, working on building the training and inference infrastructure, and today we're going to be talking about building deterministic infrastructure for non-deterministic AI agents. So, most of the conversations around AI over the last few years has been focused on models: bigger models, more parameters, better reasoning. But as organizations move from chatbots to autonomous agents, a different problem emerges: the challenge is no longer intelligence; the challenge is reliability. At Meta and across the industry, we're seeing agents move beyond answering questions and beginning to plan, call tool calls, coordinate workflows, and make decisions that affect production systems. These systems are fundamentally probabilistic; infrastructure is not allowed to be. Today I want to discuss this topic in more detail. ### The Mismatch **Nishant Gupta** [0:58] So modern cloud infrastructure evolved around a set of assumptions: requests. Most of the requests are short-lived. Services are deterministic, more or less. Execution paths are known. Failures are bounded. However, autonomous AI agents violate nearly every one of those assumptions. They're stateful. They're long-running. They make decisions dynamically. They may execute different workflows for same inputs. This is what I call the great mismatch. We're trying to run autonomous systems on infrastructure that was designed for deterministic workflows. This is probably the most important mind shift: most AI demos showcase capability, but can it solve a problem? Can it use a tool? Can it complete a workflow? Production systems have a different objective. Can it do it reliably? ### Production Reality **Nishant Gupta** [1:45] Can it do it 10,000 times, 100,000 times, million times? Can it recover from failures? Can it operate safely? Can it do it at an acceptable cost, with an acceptable latency, with an acceptable outcome? The majority of the engineering effort moves below the model layer into orchestration, monitoring, safety, evaluation, and recovery systems. ### Real Failures **Nishant Gupta** [2:06] When people hear AI failures, they immediately think hallucinations. In reality, hallucinations are often the least interesting failure mode. What we see instead are infrastructure failures: recursive reasoning loops, overflowed logs, retry amplification, context corruption, memory poisoning, cost explosions. The model makes mistakes, but however the infrastructure turns that mistake into an outage. That's the real challenge. So as this slide shows a pattern that distributed system engineers will probably recognize immediately: an agent calls a tool incorrectly. The tool returns an error. Instead of recovering, the agent generates a slightly different but still invalid request. The cycle repeats. Each retry consumes more compute. Reasoning depth increases. GPU consumption rises. Eventually, you get exponential resource growth. What started as a minor API error became a compute incident. This is why uncontrolled retries are one of the biggest risks in agentic systems. ### Deterministic Layer **Nishant Gupta** [3:08] This is the architecture principle I recommend most strongly: never let the model directly control production systems. The model should generate proposals. Infrastructure validates them. Policy engine approves them. Execution gateway enforces them. The model just suggests. The platform decides. This separation allows us to build reliable systems even when the underlying model remains probabilistic. As we know, containers gave rise to Kubernetes. Microservices created service meshes. AI agents are creating something new: an agentic control plane. This layer becomes responsible for scheduling, memory coordination, policy enforcement, valuation, monitoring, workload routing, which is very important. And think of it as an operating system for autonomous AI. The organizations that build this layer will have significantly more competitive advantages. So traditional logs tell us what happened. Agentic systems require understanding why it happened. We need traces to capture planning decisions: tool calls, memory lookups, straight transitions. When debugging an autonomous workload, understanding the chain of decisions and reasoning is often more important than the final output. ### Observability **Nishant Gupta** [4:16] Observability becomes multi-dimensional. Without it, production debugging becomes nearly impossible. So as you can see, memory is one of the most underestimated challenges in agentic architectures. Once multiple agents share state, familiar distributed system issues appear: stale reads, conflicting updates, context drifts, inconsistent views. ### Memory **Nishant Gupta** [4:37] The challenge becomes even harder when memory itself may be probabilistic and retrieval-based. Many multi-agent failures are actually consistency failures masquerading as reasoning failures. So safety cannot be a single component. It must be layered: prompt-level controls, tool permissions, policy validations, human approvals, audit systems. Each of these layers catches a different class of failures. Defense in depth is a well-understood security principle. It applies equally well to autonomous AI systems. ### Safety Layers **Nishant Gupta** [5:10] Many people frame human involvement as temporary necessity. I don't think that's correct. The most successful systems are likely to remain human-supervised. Humans become exception handlers. They review ambiguous situations. They handle novel scenarios. They provide calibration signals. The goal is not to remove humans. ### Human Oversight **Nishant Gupta** [5:29] The goal is allocating human attention where it provides the maximum value. ### GPU Orchestration **Nishant Gupta** [5:36] So one of the biggest infrastructure shifts is that AI workloads increasingly resemble cluster scaling problems. Demand is bursty. Reasoning depth is unpredictable. Workflows may run for minutes instead of milliseconds. Resource requirements vary dramatically. As a result, GPU efficiency, workload placement, elastic capacity management, and scheduling become critical. Inference is no longer just a model problem; it becomes a resource orchestration problem. The good news is that many of these problems are not entirely new. Distributed systems have solved something similar for decades. Circuit breakers become tool isolation. ### Infra Patterns **Nishant Gupta** [6:13] Rate limits become agent limits. Retries become controlled recovery. Resource quotas become cost governance. Observability becomes agent tracing. Instead of inventing entirely new infrastructure, we can adapt proven reliability patterns to autonomous systems. The industry has gone through several phases. The initial prompts were the differentiator. Then the models became the differentiator. Now both are rapidly commoditizing. The next frontier is infrastructure. The organizations that won't necessarily have the best prompts. They'll have the most reliable systems. The competitive advantage is moving up the stack. If there's one thing I want you to remember, it's this: AI agents should be treated as distributed systems. Models are stochastic. Infrastructures must be deterministic. Reliability is increasingly an infrastructure problem. Observability is mandatory. Control planes are emerging as the foundation layer. ### Takeaways **Nishant Gupta** [7:05] And ultimately, the future of the AI won't be won by better prompts. It will be won by better systems. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.