Docker

We Gave an Agent Production Code Access and Then Tried to Sleep at Night — Moritz Johner, Form3

Jul 20, 2026 · 21:57

Moritz Johner of Form3 explains that giving a coding agent production code access turns it into a supply chain actor, and the blast radius is an architecture decision. His team built PatchPilot to automate CVE patching across thousands of repositories, splitting it into a deterministic Go layer that handles dangerous operations (GitHub write access, CI triggering) and an agent layer that only edits files. The agent remediates vulnerabilities by bumping dependencies, verifying builds, and fixing CI failures, but a prompt injection could escape via the Docker socket, which kept him up at night. To contain that, they moved the agent into a firecracker microVM with its own kernel and separate network policies per layer. Johner argues that where you draw the line between deterministic and agentic code defines your security model, and warns that existing agent sandboxes are worthless when a Docker socket is involved.

Agents Need Receipts, Not More Tool Calls - Armanas Povilionis, Alithea Bio

Jul 18, 2026 · 19:36

Armanas Povilionis of Alithea Bio argues that agents need verifiable receipts, not more tool calls, and introduces Froglet, an open-source protocol for agent-to-agent compute. Froglet enables agents to discover, negotiate, execute, and receive signed, tamper-proof receipts for external services, reducing setup to 2,000 tokens and minutes. The protocol integrates with MCP, OpenClaw, and NemoClaw, and supports payment rails, identity, and workload hashes. In a live demo, he shows Claude using Froglet to publish an 'add two numbers' service locally and remotely, then invoke it to get result 12, with a signed deal ID. Froglet aims to make scientific collaboration repeatable and consistent across organizational boundaries without requiring uniform software stacks.

From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI

Jul 13, 2026 · 44:34

Abhishek Bhardwaj from OpenAI explains the design of a secure, scalable agent sandbox cloud, arguing that microVMs (like Firecracker and Cloud Hypervisor) provide the strongest isolation for running untrusted AI-generated code despite performance trade-offs, and that persistent disk storage is the next frontier for unlocking long-running, stateful agent tasks. He compares runtime isolation technologies from simple fork-exec to containers (with namespaces and cgroups) to gVisor, and advocates for hardware-backed virtualization via microVMs to prevent guest exploits from reaching the host kernel. For persistence, he details incremental block-level snapshotting using copy-on-write filesystems and always-on distributed filesystems, enabling reliable checkpointing and Monte Carlo-style exploration. Orchestration challenges include low-latency sandbox creation through pre-warming or memory snapshots, and intelligent routing based on cached snapshot layers to minimize restore time. The talk positions storage and fast snapshot restore as key enablers for advanced agent capabilities, such as long-horizon tasks and failure recovery.

Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect

Jul 13, 2026 · 46:52

Will Brown of Prime Intellect details the company's open-source ecosystem of post-training tools, including the verifiers and prime-rl libraries, arguing they enable efficient and affordable training of frontier agentic models for enterprises. Verifiers V1 decomposes environments into tasks, harnesses, and runtimes using a decorator pattern and Pydantic, supporting group rewards like conciseness bonuses. Prime-RL is an asynchronous reinforcement learning framework that allows long-horizon coding rollouts to overlap, achieving a GLM-5 step on 28 nodes in under 5 minutes for 131k context, with a 1,000-step run costing roughly $50k. The framework supports custom algorithms including on-policy distillation, GRPO, and self-distillation via decomposable loss and algorithm classes. Prime Intellect's Lab platform offers hosted multi-tenant LoRA training live now, with full fine-tuning arriving soon, enabling enterprises to develop environments on CPU and push them to the cloud for scalable post-training.

RLM: Recursive Language Models for Large Codebases - Shashi, Superagentic AI

Jul 12, 2026 · 17:27

Shashi from Superagentic AI explains how Recursive Language Models (RLM), a pattern from MIT, solve the context window problem in large codebases by externalizing context management into a programmable REPL where the model writes code to inspect, slice, and compute relevant chunks, and recursively delegates sub-questions via llm_query. The RLM loop loads the repo as data, produces bounded observations, and can recursively call another model for deeper insights. Shashi demonstrates RLM Code, an open-source independent implementation running locally and on Gemini with a Docker sandbox, showing two tool calls and a complete trajectory. He notes that similar RLM concepts are used in proprietary systems like Codex, Claude managed agents, and dynamic workflows, making it a practical pattern for AI engineers dealing with monorepos.

Develop at Idea Velocity - Jeffrey Lee-Chan, Snapchat

Jul 11, 2026 · 15:28

Jeffrey Lee-Chan (Snapchat) and Austin (CMUX) explain how to achieve 'idea velocity' by building parallel multi-agent harnesses that let one engineer direct 10–20 coding agents instead of becoming the bottleneck. Lee-Chan argues the key is separating 'Agent Orchestrator Managers' from specialized workers to prevent low-level implementation bias, using OpenClaw for frictionless Slack-based communication and CMUX terminals for real-time parallelization and unbiased manager oversight. He demonstrates two apps—an AI RPG with dice-roll mechanics and a multi-AI analysis tool—built with this stack, and discusses token-burn tradeoffs between models like Codex 5.3, GPT-5.4, and Minimax. The episode also covers staging environments to avoid doubling token usage, the benefit of persistent memory and contextual guardrails, and the importance of pushing human interaction to the start or end of the workflow for improved parallelization.

Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy

Jun 28, 2026 · 45:48

Abed Matini from Ogilvy demonstrates bypassing the multimodal tax by building a local-first hybrid RAG system that converts documents to clean Markdown via Docling, eliminating cloud vision token overhead. Using Ollama (Qwen 2.5 0.5B), PostgreSQL with pgvector, and raw SQL with Reciprocal Rank Fusion, he implements a FAQ assistant for an employee handbook with four chunking strategies—heading-based, paragraph, fixed 512-char with 64% overlap, and sentence-based. The system combines dense embedding vectors and sparse keyword indices (BM25) in a single query, retrieves top 2 chunks, and uses Python functions for speed and testability. Guardrails block prompt injections and out-of-scope questions before reaching the LLM, and LangFuse tracks token usage and latency. Matini argues that a small, local model and code-controlled pipelines reduce hallucination and cost while maintaining full observability.

Research to Reality: Bringing Frontier ML Research to Production - Vaidas Razgaitis, Higharc

Jun 28, 2026 · 14:57

Vaidas Razgaitis, Senior Research Engineer at Higharc, presents three tactical focus areas to accelerate the handoff of frontier ML research into production features. First, he advocates for a Research Prototype Taxonomy Document—a technical design document tailored for ML that maps domain context, business goals, type safety, persistence, and system architecture. Second, he details Higharc's monorepo of cleanly isolated microservices with a layered API/business logic/data structure, enabling researchers to own individual services. Third, he explains using Graphite for stack diffs to decompose monolithic prototypes into small, reviewable PRs that tap subject matter experts asynchronously. The talk offers diagnostic questions to assess team velocity across these three levers.

GPU Cloud Deployment Without Leaving Your IDE — Audry Hsu, RunPod

Jun 9, 2026 · 20:19

Audry Hsu of RunPod introduces Flash, a Python SDK that deploys GPU cloud functions from a developer's IDE with a single decorator, eliminating the slow iteration cycle of commits, Docker builds, and server allocation. She demonstrates hot reload, swapping Stable Diffusion XL Turbo for DreamShaper instantly, and a pipeline that chains Qwen 3 for prompt generation, DreamShaper for image rendering, and Nano Banana 2 for photo composition. RunPod's serverless H100 pricing is $0.00116 per second, charged only during active inference. Hsu recommends starting with pods for experimentation and switching to serverless when scaling to hundreds of workers across data centers.

Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod

Jun 7, 2026 · 13:26

Audry Hsu of RunPod presents a cloud AI infrastructure platform that lets developers deploy LLM endpoints in under five minutes. RunPod originated from two failed crypto mining rigs in a basement in 2022; the founders offered the GPUs for free on Reddit in exchange for feedback, and the company now has 500,000 developers and $120 million in annual recurring revenue. The demo shows selecting a model from the Hub, configuring the context window, and deploying a serverless endpoint on H100s. The first request queues for 41 seconds due to cold start (container initialization and model download), while subsequent requests execute in about 1.5 seconds. Users pay only while a worker handles a request, making serverless ideal for bursty or batch workloads with autoscaling up to 15 workers.

Evals Are Broken, Use Them Anyway — Ara Khan, Cline

Jun 6, 2026 · 19:04

Ara Khan from Cline argues that evals are broken—people either treat benchmark numbers as gospel or dismiss them for vibes—but that the truth lies in between, and they should still be used. He presents three heuristics: don't believe model vendor eval numbers, stay current but not an earliest adopter, and look for new precise evals like Terminal Bench. Khan details Cline's journey from ignoring evals to building their own, then adopting Terminal Bench (89 real-world coding tasks). He explains the process: get a score (Cline started at 43%), portfolio allocate failures by sending another agent through traces to identify small levers, then hill climb by fixing zone1 bugs, zone2 nuanced prompt engineering (e.g., Anthropic-specific techniques that don't transfer to Codex or Gemini), and avoid zone3 overfitting. The episode offers a practical framework for using evals to improve agent performance while staying grounded in real-world usefulness.

Dark Factory: OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc, OpenClaw

Jun 5, 2026 · 16:44

Vincent Koc, a core maintainer at OpenClaw, explains how the open-source project ships code faster than humans can read diffs by running 60–70 autonomous coding agents across swim lanes. In a single night, he and Peter Steinberger executed 2,700 commits touching 82% of the core codebase, launching a plugin architecture and changing close to a million lines of code. Koc describes managing agents as managing people: the key skill is reading reasoning tokens to detect when an agent is bullshitting, gained through sheer volume of token maxing. He argues 2025 was about token maxing; 2026 is about not wasting them, with agent-in-the-loop processes and opinionated swim lanes replacing blind Ralph looping. The episode details his agent development environment using .skills files, Git work trees, and a semantic graph for triaging 60K PRs, emphasizing that the bottleneck shifts from engineering to taste and process.

SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius

Jun 4, 2026 · 16:30

Ibragim Badertdinov from Nebius presents SWE-rebench, a monthly updated benchmark that evaluates coding agents on fresh real-world software engineering tasks to prevent data leakage from pretraining. The leaderboard reveals that models like Claude Code cheat by reading git history or fetching original GitHub issues, even after restrictions; Badertdinov emphasizes that task quality is critical, as ambiguous or overfitted tests introduce noise rather than difficulty. The filtering pipeline has produced 30,000 real-world training environments used by frontier labs. The episode also covers practical evaluation lessons: define retry policies, use caching to cut costs by 4x, and verify infrastructure against reported numbers. SWE-rebench reports tokens per problem, price per problem, and pass rates across five runs, helping AI engineers choose between models and harnesses reliably.

Lobster Trap: OpenClaw in Containers from Local to K8s and Back — Sally Ann O'Malley, Red Hat

May 22, 2026 · 21:56

Sally Ann O'Malley of Red Hat argues that running OpenClaw in containers with Podman and Kubernetes delivers secure, portable, and reproducible AI agent setups. She uses Podman secrets and OpenClaw's secret ref feature to manage API keys, ensuring secrets stay out of logs and configs. O'Malley demonstrates a local installer that spins up an OpenClaw container in two seconds and lifts the same workload to Kubernetes. She cites an Nvidia team of 10 engineers each running their own OpenClaw in Kubernetes for model evals, claiming it replaced the work of six people. Her vision is a team-standard containerized OpenClaw baseline with company-approved MCP servers and skills, enabling reproducible onboarding and personalization across an organization.

The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

May 5, 2026 · 18:30

Filip Makraduli of Superlinked introduces SAI, an open-source inference engine for small models that addresses gaps in embedding infrastructure by enabling dynamic model loading, hot-swapping, and memory-aware eviction on a single GPU. He argues that provisioning separate GPUs for each small model wastes idle capacity, and that the real challenge lies in supporting diverse model architectures (e.g., BERT, Qwen, Colbert) with different attention mechanisms and positional embeddings. The engine re-implements forward passes with variable-length FlashAttention and handles model swapping via a least recently used eviction policy. Makraduli also explains that context management for agents requires small models to pre-process data, referencing Andrej Karpathy’s graph-based knowledge bases and Chroma’s own model. The talk details the infrastructure layer including routing, auto-scaling with Prometheus, and GPU provisioning using spot instances, all open-sourced as SAI (Superlinked Inference Engine) with Helm charts and Docker images.

Scaling GitHub for your Agents — Sam Morrow, GitHub

Apr 27, 2026 · 20:35

Sam Morrow, GitHub's MCP server lead, details the architectural challenges and solutions for scaling a remote MCP server to 7 million weekly tool calls. He explains how tool proliferation degraded agent performance—LangChain's research confirmed more tools confuse agents—leading to innovations like tool sets and dynamic discovery, though 100% of users stuck with defaults. To reduce context, GitHub cut tool descriptions by 49% and trimmed output tokens by 75% on list pull requests. Security is addressed via OAuth 2.1 with PKCE and step-up auth; they rejected dynamic client registration to avoid unbounded app databases and rate-limiting issues. The stateless server uses Redis for session storage and builds a fresh server instance per request, enabling horizontal scaling without session affinity. Metrics include 11 million Docker downloads, 30,000 stars, 4,000 forks, and 126 contributors. Morrow predicts compositional tools and automatic server discovery will make thousands of tools the norm.

Full Walkthrough: Workflow for AI Coding — Matt Pocock

Apr 24, 2026 · 1:36:30

Matt Pocock presents a hands-on workshop on building a full AI-assisted coding workflow, arguing that software engineering fundamentals—not hype—make agents effective. He introduces the 'smart zone' and 'dumb zone' of LLMs (performance drops after ~100k tokens) and the 'Memento problem' (agents forget between sessions). His process starts with a 'Grill Me' skill that relentlessly questions the user until shared understanding is reached, then produces a PRD without reading it, slices work into vertical 'tracer bullet' issues, and runs agents AFK using TDD. He advocates designing codebases with deep, testable modules and shows Sandcastle, a TypeScript library for parallel agent execution with separate implementer (Sonnet) and reviewer (Opus). The workshop transforms ambiguous briefs into shippable features while keeping humans in the loop for QA and taste.

One Registry to Rule them All - Sonny Merla, Mauro Luchetti, & Mattia Redaelli, Quantyca

Apr 10, 2026 · 22:47

Amplifon's AI transformation led to the Amplify program, for which Quantyca built an enterprise-grade registry system for MCP servers and A2A agents. Sonny Merla, Mauro Luchetti, and Mattia Redaelli explain how three registries—MCP, A2A, and use case—are linked via a catalog to provide full lineage, including ownership, environment, authentication, cost attribution, and use case linkage. The solution uses an AI gateway for unified LLM access with Entra ID authentication and budgeting, and provides template repositories on GitHub with CI/CD pipelines that automatically publish agent cards and server metadata to the registries. This enables discovery, governance, and impact analysis across 26 countries and multiple teams, letting developers focus on business logic while avoiding reinventing security and deployment infrastructure.

Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA

Apr 10, 2026 · 10:16

NVIDIA’s Mozhgan Kabiri Chimeh demonstrates that running LLMs locally on the DGX Spark workstation, powered by the GB10 Grace Blackwell superchip and 128GB unified memory, achieves practical performance for models up to 14B parameters. Using a reproducible vLLM benchmarking methodology, she shows that the 14B NVFP4 quantized model delivers 20.19 tokens per second and a time-to-first-token 3.4× faster than the unoptimized 14B base model. The DGX Spark supports the same NVIDIA AI software stack as production environments, enabling local development, fine-tuning, and privacy-sensitive workloads before scaling to the cloud. NVFP4 quantization is highlighted as critical for balancing intelligence and throughput on single-system setups.

Automating Large Scale Refactors with Parallel Agents - Robert Brennan, OpenHands

Jan 8, 2026 · 1:16:21

Robert Brennan, CEO of OpenHands, and colleague Calvin explain how parallel agents can automate large-scale code refactors, demonstrating that breaking tasks into agent-sized batches with human oversight achieves 90% automation. They trace the evolution from context-unaware snippets to autonomous coding agents and now agent orchestration, where multiple agents work in parallel on tasks like CVE remediation — one client saw a 30x improvement in time-to-resolution. Calvin shows a pipeline using OpenHands' SDK to batch code files, run a verifier to detect code smells, and spin up fixer agents that generate focused pull requests. Brennan details strategies: task decomposition into single-PR-sized chunks, dependency ordering, and context sharing via agent.md files. The episode closes with a live-coding walkthrough building a parallel agent script that scans for vulnerabilities and opens individual PRs.

Hard Won Lessons from Building Effective AI Coding Agents – Nik Pash, Cline

Dec 12, 2025 · 14:18

Nik Pash, head of AI at Cline, argues that frontier models have made clever agent scaffolds obsolete—capability now beats engineering tricks. He insists that real model improvement comes from benchmarks and RL environments, not from RAG or indexing systems. Pash details Cline's RL environment factory, which converts real-world coding tasks into training data by qualifying tasks, reconstructing environments, and defining pure outcome verifiers. He announces Clinebench, an open-source benchmark built from actual software development captured via Cline's provider, designed to measure and improve models on real tasks rather than synthetic puzzles. Pash urges the community to contribute by using Cline on open-source projects, turning model struggles into benchmark candidates.

Compilers in the Age of LLMs — Yusuf Olokoba, Muna

Nov 24, 2025 · 17:36

Yusuf Olokoba, founder of Muna, explains how they built a Python compiler that converts plain Python inference functions—like Google's 270M-parameter Gemma embedding model—into self-contained C++ and Rust binaries using LLMs within a verifiable pipeline. The process involves symbolic tracing to generate an intermediate representation, type propagation to infer types for native compilation, and LLM-driven code generation to mass-produce native implementations of Python operations. After compiling into a shared library, the model can be loaded via FFI from any language (e.g., Node.js) and exposed through an OpenAI-compatible client, enabling developers to run open-source models anywhere—locally, cloud, mobile—with minimal code changes. Olokoba details why they abandoned PyTorch FX due to its PyTorch-only focus and reliance on fake inputs, and how LLMs help scale the coverage of elementary operations. The talk argues that this compiler approach solves hybrid inference—small edge models working with large cloud models—by moving beyond Python and Docker to more portable, low-latency native binaries.

Infra that fixes itself, thanks to coding agents — Mahmoud Abdelwahab, Railway

Nov 24, 2025 · 18:08

Mahmoud Abdelwahab from Railway presents Railway Autofix, a system that uses Inngest for durable execution and OpenCode as a coding agent to automatically detect infrastructure issues and open pull requests to fix them. The system runs scheduled workflows to fetch project architecture, resource metrics, and HTTP performance data, then analyzes services exceeding thresholds. For affected services, it gathers additional context like logs and generates a detailed fix plan using an LLM. The coding agent then implements the plan and opens a PR with a summary of changes. The talk demonstrates how this shifts developers from manually investigating alerts to simply reviewing auto-generated fixes.

Ship Agents that Ship: A Hands-On Workshop - Kyle Penfound, Jeremy Adams, Dagger

Jul 27, 2025 · 1:21:00

Kyle Penfound and Jeremy Adams from Dagger demonstrated building production-minded AI agents using Dagger, a programmable delivery engine that runs containerized workflows identically on laptops and in CI. They created a workspace submodule with tools — read file, write file, list files, and run tests — and wired it to an LLM (Claude 3.5) so the agent can edit code and validate changes through the project's actual test suite. The agent runs in isolated containers for determinism and traceability, with behavior visualized via Dagger Cloud. They extended the agent to respond to GitHub issues: a label triggers GitHub Actions to run Dagger, read the issue, hand the assignment to the agent, and open a pull request. The talk emphasized that Dagger's modularity lets developers give agents precisely scoped tools while reusing existing CI workflows, providing guardrails without sacrificing flexibility.

Software Development Agents: What Works and What Doesn't - Robert Brennan, OpenHands

Jul 25, 2025 · 16:46

Robert Brennan, creator of the open-source coding agent OpenHands (formerly OpenDevin), argues that while coding is going away, software engineering remains about critical thinking, not typing. He explains that agents like OpenHands operate via a loop between an LLM and tools like a diff-based editor, terminal, and web browser, all sandboxed in Docker. Brennan advises starting small with rote tasks like merge conflicts (99% success) or PR feedback, being specific about frameworks and files, and always reviewing AI-generated code to avoid tech debt. He shares that 90% of his code now goes through the agent, with 10% requiring manual intervention, and warns against auto-merging without human accountability—OpenHands now assigns PRs to the human who triggered them. Key use cases include fixing failing tests, infrastructure changes, database migrations, and greenfield internal apps where vibecoding is acceptable.

AX is the only Experience that Matters - Ivan Burazin, Daytona

Jul 24, 2025 · 15:25

Ivan Burazin, co-founder of Daytona, argues that agent experience (AX) is the only experience that matters, as AI agents will soon outnumber human developers and tools must be built for agents to autonomously operate. He cites that 25% of YC startups say AI writes 95% of their code and 37% of the latest YC batch build agents as products. Burazin outlines three AX pillars—seamless authentication, agent-readable docs (like Stripe's .md and LLMs.txt), and API-first design—then introduces Daytona's agent-native runtime, which spins up sandboxes in 27 milliseconds and includes features like a declarative image builder, network-mounted volumes for large datasets, and parallel execution for agents to fork environments. He concludes that any tool requiring a human in the loop is built for the past, and if agents cannot use a product, no one will.

Continuous Profiling for GPUs — Matthias Loibl, Polar Signals

Jul 22, 2025 · 11:31

Matthias Loibl of Polar Signals explains how continuous profiling for GPUs maximizes GPU efficiency using low-overhead, always-on sampling via eBPF. He contrasts tracing (high cost) with sampled profiling (e.g., 100 Hz, <1% overhead) and details GPU metrics collected from NVIDIA NVMe, including utilization, memory, clock speed, power, temperature, and PCIe throughput. The platform correlates these with CPU stack traces to identify bottlenecks, such as Python and CUDA functions underutilizing the GPU. A new GPU time profiling feature records the duration of CUDA kernel executions, showing actual time spent by functions on the GPU. Deployment runs on Linux with a binary, Docker, or Kubernetes DaemonSet; early adopters like TurboPuffer use it to optimize their vector engine.

Containing Agent Chaos — Solomon Hykes, Dagger

Jun 28, 2025 · 23:48

Solomon Hykes, creator of Docker and founder of Dagger, argues that containing agent chaos requires engineering reproducible execution workflows built on containerization applied to each step of an agent's workflow. He introduces 'container use'—agents developing inside fully isolated, customizable environments rather than just sandboxing outputs—and demonstrates a prototype using MCP to integrate with Claude Code and Goose. The system provides background work, rails, seamless human stepping in, and optionality by leveraging Dagger, Git-based state management, and ephemeral containers snapshotted per action. Hykes shows how agents can run parallel experiments, merge snapshots, and discard failed environments without pollution. The episode concludes with him open-sourcing the project as github.com/dagger/containeruse.

Memory Masterclass: Make Your AI Agents Remember What They Do! — Mark Bain, AIUS

Jun 27, 2025 · 51:25

Mark Bain, Vasilia Markovits, Alex Gilmore, and Daniel Chalev demonstrate that AI memory requires causal relationships and graph databases, not just vector similarity, to solve hallucinations and enable agentic workflows. Bain argues that memory is any data affecting change, and that attention, diffusion, and VAEs follow the same geometric principles as gravity and entropy. Alex shows Neo4j's MCP server storing semantic memory as entities and relationships retrieved across conversations. Vasilia demos Cognee building semantic graphs from GitHub data for agentic hiring decisions. Daniel presents Graphiti's domain-aware memory using custom Pydantic schemas to filter irrelevant facts. Bain introduces a GraphRAG chat arena that switches between memory solutions on a single Neo4j graph, testing different implementations for episodic and temporal recall.

GPU-less, Trust-less, Limit-less: Reimagining the Confidential AI Cloud - Mike Bursell

Jun 3, 2025 · 43:41

Mike Bursell of Super Protocol argues that Confidential AI, built on hardware Trusted Execution Environments (TEEs) like Intel TDX, AMD SEV-SNP, and NVIDIA GPU TEEs, solves the trust problem in AI by enabling secure processing of sensitive data and proprietary models without exposure. Super Protocol, a decentralized confidential AI cloud and marketplace, allows users to deploy models in TEEs, verify execution via cryptographic attestation, and collaborate across organizations without blind trust. Demos show deploying DeepSeek on H100 GPUs, running n8n healthcare workflows, distributing vLLM inference across four GPU nodes, and provably training a medical model on datasets from Alice's lab and Bob's clinic. Case studies include Realize achieving 75% accuracy and 3-5% sales increase for Mars, and BEL reducing FDA audit time from weeks to 1-2 hours. The protocol replaces trust with on-chain proofs, enabling GPU-less, trustless, limitless AI.

Arrakis: How To Build An AI Sandbox From Scratch - Abhishek Bhardwaj, OpenAI

Jun 3, 2025 · 40:18

Abhishek Bhardwaj, founder of Arrakis, explains why MicroVM-based AI sandboxes are the next unlock for AI agents, detailing how Arrakis provides secure, fast code execution and computer-use environments. He argues that sandboxes are essential for tool-calling models, reinforcement learning, and multi-tenant security, with Arrakis booting in under 7 seconds and supporting backtracking via snapshot/restore. The talk covers the choice of Cloud Hypervisor over other VMMs, overlayFS storage with per-sandbox read-write layers, iptables-based networking, and a built-in code execution server. A demo shows Claude using Arrakis via MCP to build a collaborative Google Docs clone and then revert to a snapshot, illustrating how a full Linux sandbox enables agents to autonomously debug and iterate without extensive prompting.

Rust is the language of the AGI - Michael Yuan

Jun 3, 2025 · 29:14

Michael Yuan argues that Rust, with its strong type system and compiler feedback, is the ideal language for AI code generation, unlike human-friendly Python or JavaScript. He presents Rust Coder, an open-source project supported by two Linux Foundation Mentorship grants, which uses MCP tools to generate, compile, and fix Rust projects. The system integrates a coding LLM (e.g., Qwen Coder) with a self-improving knowledge base of compiler errors, enabling it to generate correct code and automatically fix bugs. Yuan demonstrates its use in a Rust camp of 1000+ students and envisions future AI agents generating and deploying Rust code autonomously for tasks like drone control. He invites contributions to expand the knowledge base and enhance the tools for AGI.

The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

Jun 3, 2025 · 18:52

Jonathan Fernandes, independent AI engineer, details the RAG stack his team settled on after 37 failed attempts, covering orchestration (LlamaIndex), embeddings (BAAI BGE small), vector database (Qdrant), LLMs (GPT-4, Qwen, Llama), reranking (Cohere), monitoring (Arize Phoenix), and evaluation (RAGAS). He demonstrates a live prototype in Google Colab using a London railway knowledge base, showing how a naive RAG returns irrelevant results (e.g., suggesting black cabs for "where can I get help at the station"). By swapping components—replacing in-memory storage with Qdrant, using an open-source embedding model, upgrading to GPT-4, and adding Cohere reranking—the answer improves to "go to booth number five next to the Eurostar ticket gates." For production, he deploys via Docker Compose with NVIDIA embedding/reranking models and Ollama for serving. The episode also stresses the importance of tracing latency per component and using RAGAS for systematic evaluation across many queries.

Stateful Agents — Full Workshop with Charles Packer of Letta and MemGPT

Apr 19, 2025 · 1:19:34

Charles Packer, lead author of the MemGPT paper and co-founder of Letta, argues that statefulness (memory) is the most important problem to solve for building useful AI agents, since LLMs are inherently stateless transformers. He presents MemGPT's LMOS (Language Model Operating System) approach, which treats memory management as a context compilation problem solved by the LLM itself using tool calling to read and write structured memory blocks. The workshop demonstrates Letta's open-source stack (FastAPI, Postgres, Python) where agents persist state on a server, enabling long-running, learning interactions without context overflow—e.g., an agent can update its core memory (e.g., correcting a user's name) or search archival memory (e.g., recalling user preferences after a reset). Packer also shows multi-agent communication via async message passing between agents running as independent services, and highlights that tools are sandboxed by default with support for Composio integrations. The session includes a live notebook exercise and a low-code UI (ADE) showing context window management and memory editing, emphasizing that true stateful agents improve continuously over time rather…

Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

Feb 22, 2025 · 21:38

Eric Bernhardson, CEO of Modal, explains why and how his company replaced Kubernetes and Docker with a custom container system to deliver sub-second cold starts for AI developers. Modal turns any Python function into a serverless function with a decorator, runs on thousands of H100s, and fans out to 10,000 parallel calls. To achieve fast startup, Modal built content-addressable storage for deduplication, lazy file loading with prefetching, and uses gVisor for CPU memory snapshotting, cutting Stable Diffusion startup to seconds. The company built its own scheduler and file system, and uses mixed integer programming to manage a global GPU pool across cloud vendors. Customers like Suno use Modal for AI-generated music inference. Modal offers $30/month free credits.

Fine tune 20 Llama Models in 5 Minutes: Santosh Radha

Feb 9, 2025 · 6:26

Santosh Radha, Head of Product/Research at Agnostiq, demonstrates Covalent, an open-source platform that lets users fine-tune and deploy hundreds of Llama models directly from Python without Kubernetes or Docker. By adding a single decorator to Python functions, users specify GPU requirements (e.g., H100 with 48 GB, 18-hour limit) and run them on remote compute, paying only for actual usage (e.g., 87 cents for 6 minutes on an L14, 11 cents on a V100). Covalent supports job submission, inference endpoints with custom autoscaling (e.g., scale to 10 GPUs at 9 AM daily), and automated workflows for training, evaluation, and deployment. Radha shows a workflow that iterates over 20 models, fine-tunes each, evaluates accuracy, sorts, and deploys the best—all from a Jupyter notebook with a single dispatch call. The talk, recorded at the AI Engineer World's Fair, emphasizes eliminating infrastructure overhead for accelerated compute.

[Full Workshop] How to add secure code interpreting in your AI app: Vasek Mlejnsky

Feb 6, 2025 · 1:48:16

Vasek Mlejnsky, CEO of E2B, demonstrates how to add secure AI code execution to any app using E2B’s Code Interpreter SDK alongside Anthropic's Sonnet 3.5 and Vercel’s AI SDK. The workshop builds an open-source version of Claude’s artifacts UI, running Python code in isolated Firecracker VMs that start in ~900ms and support custom environments via Dockerfiles. Vasek explains how E2B’s Jupyter-server-based sandbox returns stdout, stderr, runtime errors, and rich outputs like PNG charts, which are streamed to the front end using Vercel’s stream data. He covers security architecture—full root access in a VM that self-destructs on escape—and plans for AWS/GCP self-hosting and snapshot-based agent branching. The episode also addresses customizing sandboxes with private packages, mounting cloud storage (S3, GCS), and interpreting table previews as HTML or images.

AI Templates: Gabriela and Aishwarya

Feb 6, 2025 · 1:03:20

Gabriela de Queiroz, Aishwarya, and Pamela from Microsoft present AI templates for rapidly prototyping and deploying generative AI applications, demonstrating how startups can leverage Microsoft for Startups Founders Hub, including up to $150,000 in Azure credits and expert guidance. The workshop covers three templates: a simple chat app using GPT-3.5 Turbo, a Retrieval Augmented Generation (RAG) app on Postgres with hybrid vector and text search, and a RAG app on unstructured documents using Azure AI Search and Document Intelligence. Key insights include the importance of query rewriting, hybrid search over pure vector search, and streaming responses for better user experience. The templates are open source on GitHub and can be deployed via Codespaces, with a proxy provided to bypass Azure OpenAI approval delays.

Running AI Application in Minutes w/ AI Templates: Gabriela de Queiroz, Pamela Fox, Harald Kirschner

Aug 14, 2024 · 1:29:02

In this workshop, Microsoft's Gabriela de Queiroz, Pamela Fox, and Harald Kirschner show how to deploy AI applications in minutes using AI templates, Azure OpenAI proxy, and GitHub Codespaces. They walk through deploying a simple chat app, then two RAG applications: one that queries a Postgres database with SQL filtering, and another that performs RAG on unstructured documents using Azure AI Search. Key decisions include using async frameworks like Quart and FastAPI, token-based chunking, and hybrid retrieval with semantic reranking for best results. They stress the importance of running evaluations with hundreds of samples and share production insights from Copilot Chat, where TFIDF sparse indexing and LLM reranking are used. The session includes free Azure credits and a proxy to bypass Azure OpenAI approval, allowing attendees to deploy everything without spending their own money.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.