AI Infrastructure

When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain

Jul 20, 2026 · 27:33

Dmitry Petrov of DataChain argues that AI agents fail on unstructured physical data because their intuitions assume cheap recompute, while large-scale video, sensor, and robot data requires a different harness. He cites Anthropic's finding that agents achieve only 21% accuracy on data projects without specific data harnesses, and OpenAI's need for six layers of context even on structured data. Petrov demonstrates DataChain, an open-source Python framework that uses Pydantic schemas to turn messy binary files into queryable databases, an execution engine for distributed processing, incremental checkpoints to avoid recomputing on failure, and a knowledge base of datasets and source code so agents can answer follow-up questions in seconds instead of reprocessing terabytes. In a live demo with Claude Code analyzing 90 dashcam videos, the harness took 24 minutes to extract 100,000 object records, then instantly answered 'how many clips have people?' without rerunning inference. Petrov emphasizes that the key is organizing metadata into star schemas and sharing data lineage across teammates so no one pays the compute cost twice.

Stop Renting Your Cognitive Infrastructure - Thiyagarajan Maruthavanan, Kalmantic Labs

Jul 18, 2026 · 7:52

Thiyagarajan Maruthavanan of Kalmantic Labs argues that AI teams should stop renting inference from providers like Anthropic and build their own infrastructure, coining "rent to learn, own to earn." He recounts his app UltraSuno costing hundreds of thousands of dollars in inference, a stolen key hitting $10,000, and moving to his own DGX Sparks hardware. Three enterprises—a fund, hospital, and tax practice—each hit walls with renting: control, audit redlining, and reproducibility. He open-sourced JustTokenMax (benchmarked better than Netflix's Headroom) and wrote the book "Peak Inference" on building your own inference infra. He notes the market's conflicting pitches (Jensen's token factory, Nadella's unmetered, NeoClouds' endpoints) but concludes owning is essential post-PMF.

Computer-Use 2.0: Agents Just Got Multi-Cursor — Francesco Bonacci, Cua

Jul 15, 2026 · 16:41

Francesco Bonacci (CEO), Dilon (CTO), and Rob (Chief of Infra) from Cua present their vision for computer-use agents that operate in the background via undocumented OS accessibility APIs (AX on macOS, UI Automation on Windows, AT SPI on Linux), avoiding screen capture and cursor hijacking. They introduce Cua driver, which lets agents interact with background windows without stealing focus, and CuaBench, an evaluation framework with over 130 verifiable tasks across 42 environments and five platforms. Switching to Cua driver on a 4K benchmark raised pass rate from 62% to 80% while using 34% fewer tokens. Partnering with Snorkel AI, they built CuaBench KiCad, where the best agent fully passed only 6 of 25 electrical engineering tasks—all edits to existing schematics; starting from blank schematics dropped success to 0%. Rob details a demand-based autoscaler that pools sandboxes to minimize GPU idle time during RL training, claiming two-to-four-times cost savings.

From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI

Jul 13, 2026 · 44:34

Abhishek Bhardwaj from OpenAI explains the design of a secure, scalable agent sandbox cloud, arguing that microVMs (like Firecracker and Cloud Hypervisor) provide the strongest isolation for running untrusted AI-generated code despite performance trade-offs, and that persistent disk storage is the next frontier for unlocking long-running, stateful agent tasks. He compares runtime isolation technologies from simple fork-exec to containers (with namespaces and cgroups) to gVisor, and advocates for hardware-backed virtualization via microVMs to prevent guest exploits from reaching the host kernel. For persistence, he details incremental block-level snapshotting using copy-on-write filesystems and always-on distributed filesystems, enabling reliable checkpointing and Monte Carlo-style exploration. Orchestration challenges include low-latency sandbox creation through pre-warming or memory snapshots, and intelligent routing based on cached snapshot layers to minimize restore time. The talk positions storage and fast snapshot restore as key enablers for advanced agent capabilities, such as long-horizon tasks and failure recovery.

State of the Union: Why Local, Why Now — NVIDIA, Osmantic, Roboflow, EXO Labs, @matthew_berman

Jul 11, 2026 · 44:29

Nader Khalil (NVIDIA), Joseph Nelson (Roboflow), Alex Cheema (Exo Labs), Matthew Berman, and Ahmad Osman (Osmantic, r/LocalLLaMA) argue that local AI is now useful, driven by stronger open models and better hardware. They cite inflection points like Llama 2, DeepSeek v3, and GLM 5.2, which closed the gap with frontier cloud models. Sovereignty and control are key: enterprises need to choose their own model versions and avoid lock-in. Specialized models, such as Roboflow's fine-tuned vision models for deep-sea fish discovery, outperform general ones for specific tasks. Optimization is critical: EXO Labs achieved 10x performance on the DGX Spark by tuning existing NVIDIA kernels. The panel emphasizes that simplicity remains a barrier—most users need point-and-click solutions—and advocates for open-source AI to ensure freedom and innovation.

I Run a Fleet of AI Agents Across Three Machines. Here's What Broke. - Kyle Jaejun Lee, KRAFTON

Jul 8, 2026 · 9:11

Kyle Jaejun Lee runs a fleet of AI coding agents across three machines daily and reveals the scaling failures that emerged from a hierarchy of CEO, VP, manager, and worker agents. To overcome his own attention bottleneck, he separated context into entity-specific workspaces on disk and replaced context compaction with full resets that read handoff files. When moving beyond one machine, five things broke: agents doing work instead of delegating, TMUX panes too crowded to read, out-of-memory crashes from stacked Claude Code processes, colliding Git credentials across workspaces, and the MacBook dying mid-task. He offloaded long-running work to always-on Linux boxes, used Git commits and SSH to move context between machines, and consolidated review gateways onto a single always-on machine with Discord as a unified router. Unsolved challenges include consistent credentials, local-only tools, and resource scheduling—he plans to layer his orchestration on top of Kubernetes to handle compute, secrets, and tools.

GPU Cloud Deployment Without Leaving Your IDE — Audry Hsu, RunPod

Jun 9, 2026 · 20:19

Audry Hsu of RunPod introduces Flash, a Python SDK that deploys GPU cloud functions from a developer's IDE with a single decorator, eliminating the slow iteration cycle of commits, Docker builds, and server allocation. She demonstrates hot reload, swapping Stable Diffusion XL Turbo for DreamShaper instantly, and a pipeline that chains Qwen 3 for prompt generation, DreamShaper for image rendering, and Nano Banana 2 for photo composition. RunPod's serverless H100 pricing is $0.00116 per second, charged only during active inference. Hsu recommends starting with pods for experimentation and switching to serverless when scaling to hundreds of workers across data centers.

Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

Jun 8, 2026 · 15:50

Max Ryabinin from Together AI presents their research on extending transformer context length to 5 million tokens using Untied Ulysses, which cuts activation memory by reusing buffers across attention head iterations. The talk walks through a stack of techniques including fully sharded data parallelism, DeepSpeed Ulysses context parallelism for an 8x activation reduction, activation checkpointing for another 8x, CPU offloading of transformer block inputs, and chunked sequence training. Even with these, training a LLaMA 3B model with 3 million tokens fits on an 8xH100 node, but 5 million requires Untied Ulysses. Instead of allocating one large buffer per attention head group, it chunks heads further and reuses buffers across iterations, cutting activation memory with negligible throughput impact. At both 8B and 32B scale, results match the most memory-optimized transformer training baselines while pushing sequence length 25% further than prior Ulysses implementations.

Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod

Jun 7, 2026 · 13:26

Audry Hsu of RunPod presents a cloud AI infrastructure platform that lets developers deploy LLM endpoints in under five minutes. RunPod originated from two failed crypto mining rigs in a basement in 2022; the founders offered the GPUs for free on Reddit in exchange for feedback, and the company now has 500,000 developers and $120 million in annual recurring revenue. The demo shows selecting a model from the Hub, configuring the context window, and deploying a serverless endpoint on H100s. The first request queues for 41 seconds due to cold start (container initialization and model download), while subsequent requests execute in about 1.5 seconds. Users pay only while a worker handles a request, making serverless ideal for bursty or batch workloads with autoscaling up to 15 workers.

Scaling the Next Paradigm of Heterogeneous Intelligence — Adrian Bertagnoli, Callosum

May 24, 2026 · 15:13

Adrian Bertagnoli, founding engineer at Callosum, argues that the era of homogeneous intelligence—scaling single models on identical chips—is ending, and heterogeneous intelligence, which routes tasks to optimal chips and models, is the next paradigm. He demonstrates this with two case studies: on the Ulong benchmark, running recursive language models on Cerebras instead of GPT-5.2 cuts cost by 7x and latency by 5x while matching accuracy; on Video Web Arena, a mixture of Qwen 3 VL8B and Kimi K2.5 beats GPT-5.2 and Gemini 2.5 by 18% and 25%, while costing 3.7x less and running 3x faster. The key insight is that complex problems decompose into subtasks—like zooming on a webpage—which require different intelligence levels; offloading those to smaller models yields 11x speed and 43x cost improvements. Callosum builds an automation layer that predicts the best model and hardware for each subtask, and has secured a £3 million grant with the UK's Arya institute to operate the first heterogeneous co-located cluster.

Lobster Trap: OpenClaw in Containers from Local to K8s and Back — Sally Ann O'Malley, Red Hat

May 22, 2026 · 21:56

Sally Ann O'Malley of Red Hat argues that running OpenClaw in containers with Podman and Kubernetes delivers secure, portable, and reproducible AI agent setups. She uses Podman secrets and OpenClaw's secret ref feature to manage API keys, ensuring secrets stay out of logs and configs. O'Malley demonstrates a local installer that spins up an OpenClaw container in two seconds and lifts the same workload to Kubernetes. She cites an Nvidia team of 10 engineers each running their own OpenClaw in Kubernetes for model evals, claiming it replaced the work of six people. Her vision is a team-standard containerized OpenClaw baseline with company-approved MCP servers and skills, enabling reproducible onboarding and personalization across an organization.

What Breaks When You Build AI Under Sovereignty Constraints - Bilge Yücel, deepset GmbH

May 19, 2026 · 19:09

Bilge Yücel, Senior Developer Relations Engineer at deepset, argues that sovereign AI requires explicit control over data flow, model choice, infrastructure, and operations, and retrofitting these pillars breaks existing systems in predictable ways. Replacing frontier APIs with self-hosted models forces re-evaluation from scratch, moving private data across jurisdictions creates multi-database search problems, replacing managed infra reveals vendor lock-in, and adding observability exposes black-box systems. She presents a sovereign architecture with guardrails, MCP tools, and Haystack's swappable components to mitigate these issues. The closing checklist asks whether you can swap models without changing application logic, have compliant run logs, and respond to incidents without calling a hyperscaler.

Why MLX — Prince Canuma, Neywa Labs

May 11, 2026 · 23:10

Prince Canuma argues that on-device AI, powered by Apple's MLX framework, is a viable alternative to cloud-dependent services, especially for users in regions with unreliable internet. MLX, an array framework for Apple Silicon, has reached 1.5 million downloads and 4000 ported models, including day-zero support for Gemma 4 and Qwen 3 Omni. Canuma demonstrates real-time vision models (e.g., RF Deter for object detection), sub-100ms text-to-speech via Marvis TTS, and modular speech-to-speech pipelines that run entirely on-device. Community projects showcased include a native voice app (Locally), a robot with real-time voice cloning, and a video generation system that chains coherent stories on 16GB VRAM. A recent breakthrough, Turbo Quant, reduces KV cache by 4x, enabling 1 million context windows on-device. Canuma emphasizes that these capabilities provide accessibility for his blind father and enable agents that hear, see, and respond without phoning home.

The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

May 5, 2026 · 18:30

Filip Makraduli of Superlinked introduces SAI, an open-source inference engine for small models that addresses gaps in embedding infrastructure by enabling dynamic model loading, hot-swapping, and memory-aware eviction on a single GPU. He argues that provisioning separate GPUs for each small model wastes idle capacity, and that the real challenge lies in supporting diverse model architectures (e.g., BERT, Qwen, Colbert) with different attention mechanisms and positional embeddings. The engine re-implements forward passes with variable-length FlashAttention and handles model swapping via a least recently used eviction policy. Makraduli also explains that context management for agents requires small models to pre-process data, referencing Andrej Karpathy’s graph-based knowledge bases and Chroma’s own model. The talk details the infrastructure layer including routing, auto-scaling with Prometheus, and GPU provisioning using spot instances, all open-sourced as SAI (Superlinked Inference Engine) with Helm charts and Docker images.

The New Application Layer - Malte Ubl, CTO Vercel

Apr 20, 2026 · 18:52

Vercel CTO Malte Ubl argues that AI engineering is the legitimate successor to web development and that the real value lies in the application layer, not the model labs. He identifies four effective agent archetypes—24/7 support, compressed research, surfacing existing information, and eliminating boring work—and reveals that over 60% of vercel.com page views now come from AI agents. Ubl predicts model companies will commoditize, driving costs down and empowering engineers, while citing Europe's leadership in AI engineering through Vercel's AI SDK, Pi (a coding agent from Austria), and OpenClaw. He stresses the need for open-mindedness toward paradigm shifts and new infrastructure, such as sandboxed agent runtimes, and warns of impending security challenges.

Why We Don’t Need More Data Centers - Dr. Jasper Zhang, Hyperbolic

Aug 1, 2025 · 14:18

Dr. Jasper Zhang, CEO of Hyperbolic, argues that the AI industry's focus on building more data centers is inefficient because GPUs already sit idle 80% of the time. Instead, he advocates for a GPU marketplace that aggregates supply from fragmented providers, cutting costs by 50–75%—Hyperbolic offers H100s at $0.99/hour versus $2–$11 from traditional clouds. His HyperDOS software connects any Kubernetes cluster in 5 minutes, enabling dynamic rental and release of GPUs. He illustrates with a startup that can reduce compute costs from $43.8 million to $6.9 million by flexibly scaling capacity and selling idle GPUs, effectively boosting productivity 6x on the same budget. Zhang contends that smarter allocation, not just more data centers, is key to sustainable and democratized AI infrastructure.

AX is the only Experience that Matters - Ivan Burazin, Daytona

Jul 24, 2025 · 15:25

Ivan Burazin, co-founder of Daytona, argues that agent experience (AX) is the only experience that matters, as AI agents will soon outnumber human developers and tools must be built for agents to autonomously operate. He cites that 25% of YC startups say AI writes 95% of their code and 37% of the latest YC batch build agents as products. Burazin outlines three AX pillars—seamless authentication, agent-readable docs (like Stripe's .md and LLMs.txt), and API-first design—then introduces Daytona's agent-native runtime, which spins up sandboxes in 27 milliseconds and includes features like a declarative image builder, network-mounted volumes for large datasets, and parallel execution for agents to fork environments. He concludes that any tool requiring a human in the loop is built for the past, and if agents cannot use a product, no one will.

What every AI engineer needs to know about GPUs — Charles Frye, Modal

Jul 20, 2025 · 19:52

Charles Frye of Modal explains that AI engineers need to understand GPU hardware constraints to optimize inference, arguing GPUs embrace bandwidth over latency and that Tensor Cores for low-precision matrix-matrix multiplication are the key resource. He describes how GPUs achieve 16,000+ parallel threads per cycle on H100, and notes Patterson’s Law: bandwidth improves at the square of latency. The main insight: arithmetic intensity favors N² operations per N memory loads, so matrix-matrix operations are efficient while matrix-vector is wasteful. Frye demonstrates that running a small 8B model 1,000 times on the same prompt matches GPT-4 quality, and that multi-token prediction and multi-sample query become nearly free because Tensor Cores handle expanded batches as matrix-matrix multiplications. He recommends using smaller models that fit on a single GPU and scaling via multiple generations.

Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI

Jul 19, 2025 · 19:03

Keegan McCallum, Head of ML Infrastructure at Luma AI, details how the company's Dream Machine model scaled from 500 to 9,000 H100 GPUs within hours to handle 1 million users in four days, outpacing ChatGPT's initial growth. He explains that their initial Triton inference server setup was brittle and ill-suited for multi-GPU, multi-node video models, prompting a re-architecture to a custom serving stack on vanilla PyTorch. To solve work starvation across user tiers, they implemented an SLO-based aging system that ranks jobs by the percentage of their worst-case wait time elapsed. For managing dozens of model versions, they store immutable full Python environments and checkpoints in object storage, with a YAML file controlling active deployments and enabling zero-downtime rollouts across thousands of GPUs. McCallum also discusses partnerships with Nvidia, AMD, and Grok, and how Luma's broader mission is to build general multimodal intelligence that generates, understands, and operates in the physical world.

Conquering Agent Chaos — Rick Blalock, Agentuity

Jul 1, 2025 · 14:40

Rick Blalock, founder of Agentuity, argues that deploying AI agents remains the number one headache for developers, citing common issues like serverless timeouts (agents running 15–30 minutes), statelessness, and networking complexities. He demonstrates Agentuity’s platform, which treats agents as first-class infrastructure citizens: users scaffold projects via a CLI (supporting Bun, Python/UV, Node.js), write a simple request handler, and deploy with automatic routing, tunneling, and a built-in AI gateway that tracks costs per run. The platform decouples inputs and outputs—agents can be triggered via email, SMS, webhooks, or cron—and provides human and agent-facing telemetry via OTel tracing. Blalock notes Agentuity already hosts 50–60 internal agents and is building infrastructure agents to replace tools like PagerDuty, with plans to add Slack/Discord integrations and service-level reasoning capabilities.

Building agent fleet architectures your CISO doesn't hate — Lou Bichard, Gitpod

Jun 27, 2025 · 13:52

Lou Bichard explains how Gitpod evolved from a managed SaaS to a 'bring your own cloud' architecture that satisfies CISOs in regulated industries by running secure dev environments—and now agent fleets—on customer infrastructure via a simple runner (a single ECS task) instead of complex Kubernetes. The platform, used by banks and healthcare firms for 37 hours per week per developer, reduces operational overhead through a cloud-formation-based setup that takes three minutes. For agents, the same infrastructure provides source code access and audit logging, ensuring privacy and compliance. Bichard argues that vendors should simplify architectures to lower customers' day-two costs, and advises buyers to prioritize security and ownership models when selecting AI tools.

Fun stories from building OpenRouter and where all this is going - Alex Atallah, OpenRouter

Jun 25, 2025 · 18:47

Alex Atallah, founder of OpenRouter, tells the story of how he launched the LLM aggregator in early 2023 after observing the open-source race sparked by Meta's Llama 1 and Stanford's Alpaca distillation under $600. He argues the inference market is not winner-take-all, citing OpenRouter's data showing Google Gemini growing from 2-3% to 34-35% of tokens over 12 months. The platform evolved from a simple model collection into a marketplace with over 400 models and 60 providers, solving architecture challenges like 30-millisecond latency, cancelable streams, and a middleware plugin system for web search and PDF parsing. Atallah explains that OpenRouter grew 10-100% month-over-month for two years, and he predicts future additions like transfusion models that generate images and more powerful geographic routing for enterprise optimization.

The Geopolitics of AI Infrastructure - Dylan Patel, SemiAnalysis

Jun 19, 2025 · 18:29

Dylan Patel of SemiAnalysis argues that despite US sanctions, Huawei has engineered a 384-chip cluster (Cloud Matrix 384) that Nvidia failed to deploy, while accessing TSMC via Softgo and HBM from Samsung via shell companies — all legally. China's SMIC will soon produce 7nm AI chips in high volumes, debunking the notion that China lacks compute. Meanwhile, Middle East players like G42 (UAE) and Datavolt (Saudi Arabia) are building multi-gigawatt data centers, with G42's deal letting it keep 20% of 500,000 GPUs yearly for itself while 80% goes to US companies like OpenAI. Patel highlights the US's 63-gigawatt power shortfall vs. 100 GW of planned data centers, explaining why US companies rely on Middle East capacity and why China's superior power buildout gives it a geopolitical edge.

How agents broke app-level infrastructure - Evan Boyle

Jun 3, 2025 · 13:32

Evan Boyle, founder of GenSX, argues that LLM-powered agents have broken assumptions about app-level infrastructure, as p1 latency jumps from milliseconds to seconds and workflows run minutes to hours. He explains that traditional serverless providers timeout after 5 minutes, lack native streaming, and force brittle Rube Goldberg machines on top of queues like SQS or tools like Airflow. Boyle presents an open-source library that separates API and compute layers using Redis streams for resumability, enabling users to refresh pages or navigate away without losing progress. He demonstrates components with built-in retries, caching, and tracing, and a workflow that automates Hacker News analysis. Key architectural lessons include starting simple but planning for long-running agents, keeping compute and API planes separate, and leaning on Redis for resumability.

How to Build Your Own AI Data Center in 2025 — Paul Gilbert, Arista Networks

Apr 27, 2025 · 23:00

Paul Gilbert, tech lead at Arista Networks, explains the key considerations for building AI data center networks in 2025, emphasizing the stark differences from traditional enterprise networks. He details the backend GPU network (eight 400G ports per H100 server, no over-subscription), the front-end storage network (calmer, 100-200G), and the need for lossless Ethernet with ECN and PFC flow control to prevent packet drops during synchronized GPU bursts. Gilbert highlights power challenges (10.2kW per GPU server, requiring 100-200kW water-cooled racks) and the importance of telemetry like RDMA error monitoring and an AI agent that correlates GPU and network issues. He also covers advanced load balancing (cluster-aware, up to 93% utilization) and smart system upgrades without downtime, while noting the upcoming Ultra Ethernet Consortium (v1.0 in 2025) that shifts congestion control to NICs.

Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

Feb 22, 2025 · 21:38

Eric Bernhardson, CEO of Modal, explains why and how his company replaced Kubernetes and Docker with a custom container system to deliver sub-second cold starts for AI developers. Modal turns any Python function into a serverless function with a decorator, runs on thousands of H100s, and fans out to 10,000 parallel calls. To achieve fast startup, Modal built content-addressable storage for deduplication, lazy file loading with prefetching, and uses gVisor for CPU memory snapshotting, cutting Stable Diffusion startup to seconds. The company built its own scheduler and file system, and uses mixed integer programming to manage a global GPU pool across cloud vendors. Customers like Suno use Modal for AI-generated music inference. Modal offers $30/month free credits.

Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley

Feb 8, 2025 · 20:46

Humza Iqbal of Snorkel AI explains how the company uses Azure AI infrastructure powered by NVIDIA GPUs to fine-tune foundation models for enterprise customers, achieving better performance per dollar by switching from A100s to H100s. He details their distributed training stack (PyTorch, Horovod, NFS) and lessons learned such as balancing node count for batch size and monitoring GPU utilization to avoid networking or data-loading bottlenecks. A cost comparison found two H100s outperformed four A100s on both training and inference, enabling faster iteration through more synthetic data. Azure's dedicated VMs, reliable NFS throughput, and flexible capacity allowed Snorkel to scale experiments from single-node to dozens of GPUs. Future work includes programmatic preference signals and multimodal retrieval algorithms, all planned on Azure.

Hypermode Launch: Kevin Van Gundy

Aug 23, 2024 · 5:03

Kevin Van Gundy, founder of Hypermode, argues that iteration velocity is the compound interest of software and key to succeeding with AI, and presents Hypermode as a runtime that makes AI approachable by integrating models and data into AI functions with minimal friction. Drawing from his experience at Vercel, he explains how rapid iteration helped them win against competitors, and applies the same philosophy to AI development. Hypermode removes common pain points: it offers one-request RAG with in-memory embedding and search, model comparison and fine-tuning export, and strong defaults that work with existing stacks. The platform de-risks getting it wrong by making model switching and prompt changes frictionless. Kevin invites listeners to a workshop to build a demo and offers $1000 in Hypermode credits to get started.