Modal

Self Driving Products: Product Signals to Pull Requests — Joshua Snyder, PostHog

Jun 10, 2026 · 15:39

Joshua Snyder of PostHog explains how they're building a pipeline that turns product signals—errors, Slack messages, session replays—into automated pull requests. He reveals that off-the-shelf embedding models cluster signals by structural similarity, so they embed LLM-generated queries instead. He argues specificity determines whether the agent produces a useful PR, with error tracking being immediately actionable while Slack and replay usually are not. He advises starting with costly agents to discover patterns, then collapsing expensive steps into one-shot calls.

What Lies Beneath the API — Benjamin Cowen, Modal

Jun 2, 2026 · 12:40

In this episode, Ben Cowen from Modal argues that as AI products mature, fine-tuning becomes essential, citing cases like Intercom beating their frontier API at 1/10th the cost. He identifies three signals it's time to fine-tune: paying more for the API than customers pay you, evaluative plateaus, and latency requirements that shared endpoints can't meet. Cowen explains that supervised fine-tuning now fits in 300 lines of Python and that reinforcement learning rollouts can scale to 50,000 sandboxes using serverless platforms like Modal. He contends that frontier labs aim to win at everything, while businesses need to win at their specific logic, making fine-tuning a natural destination. The episode provides practical guidance on when and how to make the leap, emphasizing that building an agent harness and collecting eval data already sets the stage for training.

Fast Models Need Slow Developers — Sarah Chieng, Cerebras

May 22, 2026 · 18:02

Sarah Chieng from Cerebras argues that the 20x speed increase of models like Codex Spark (1,200 tokens/sec vs. 40-60 for Sonnet/Opus) forces developers to rethink workflows, or risk generating technical debt at unprecedented scale. She presents a practical playbook: validation and linting become free at every step, so must run continuously; developers can generate 75 component variations across five sub-agents and cherry-pick the best; and with context filling in 30 seconds instead of ten minutes, a four-file external memory system (agents, plan, progress, verify) maintains continuity between sessions. Chieng emphasizes real-time collaboration with the model rather than spawning agents and walking away, and advocates using a slower planner model with a fast executor to orchestrate agents effectively.

Claude Agent SDK [Full Workshop] — Thariq Shihipar, Anthropic

Jan 5, 2026 · 1:52:25

Thariq Shihipar of Anthropic presents the Claude Agent SDK, arguing that Bash and file-system-based agents outperform traditional tool-only approaches for autonomous tasks. He defines agents as systems that build their own context and trajectory, contrasting with structured workflows. The SDK, built on Claude Code, emphasizes the Bash tool as the most powerful primitive for composability and code generation, enabling non-coding tasks like data analysis. He demonstrates live-coding a Pokémon team advisor that dynamically fetches API data via scripts, and explains security through a 'Swiss cheese defense' of model alignment, AST parsing, and sandboxing. Shihipar also covers skills for progressive context disclosure, sub-agents for parallel work, and hooks for deterministic verification, stressing that agent building is an art of reading transcripts and iterating on context engineering.

What every AI engineer needs to know about GPUs — Charles Frye, Modal

Jul 20, 2025 · 19:52

Charles Frye of Modal explains that AI engineers need to understand GPU hardware constraints to optimize inference, arguing GPUs embrace bandwidth over latency and that Tensor Cores for low-precision matrix-matrix multiplication are the key resource. He describes how GPUs achieve 16,000+ parallel threads per cycle on H100, and notes Patterson’s Law: bandwidth improves at the square of latency. The main insight: arithmetic intensity favors N² operations per N memory loads, so matrix-matrix operations are efficient while matrix-vector is wasteful. Frye demonstrates that running a small 8B model 1,000 times on the same prompt matches GPT-4 quality, and that multi-token prediction and multi-sample query become nearly free because Tensor Cores handle expanded batches as matrix-matrix multiplications. He recommends using smaller models that fit on a single GPU and scaling via multiple generations.

How fast are LLM inference engines anyway? — Charles Frye, Modal

Jun 27, 2025 · 16:07

Charles Frye presents benchmarks from hundreds of runs on Modal comparing open-source inference engines VLM, SGLang, and TensorRTLM across models like Qwen 3 and Gemma 27B, arguing open weights models have caught up to proprietary ones, making self-hosting viable. He shows, for example, that Qwen 3 (MoE) on VLM achieves ~1 request/sec with 128 input tokens and 1024 output tokens, while switching to a RAG-like workload (1024 in, 128 out) yields a 4x throughput improvement. Frye warns that optimizing for context over reasoning can improve latency without sacrificing quality, and notes that the engines' out-of-the-box performance varies by model—e.g., SGLang underperforms VLM on Gemma due to less optimization. He also highlights the gap between prefill (parallel) and decode (autoregressive) speeds, which a rationalist would expect from transformer architecture. The benchmarks, available at modal.com/llmalmanac, aim to help engineers choose hardware and engines, with contributions welcome for optimized configs like TensorRTLM's knobs.

Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

Feb 22, 2025 · 21:38

Eric Bernhardson, CEO of Modal, explains why and how his company replaced Kubernetes and Docker with a custom container system to deliver sub-second cold starts for AI developers. Modal turns any Python function into a serverless function with a decorator, runs on thousands of H100s, and fans out to 10,000 parallel calls. To achieve fast startup, Modal built content-addressable storage for deduplication, lazy file loading with prefetching, and uses gVisor for CPU memory snapshotting, cutting Stable Diffusion startup to seconds. The company built its own scheduler and file system, and uses mixed integer programming to manage a global GPU pool across cloud vendors. Customers like Suno use Modal for AI-generated music inference. Modal offers $30/month free credits.

The Weekend AI Engineer: Hassan El Mghari

Nov 22, 2023 · 21:49

Hassan El Mghari shares how he built viral AI apps like RoomGPT and AI Commit, arguing that simple off-the-shelf APIs and focused weekend sprints can attract millions. He describes building 11 side projects in a year, which grew from 20,000 visitors to over 8.5 million unique visitors and 2.8 million sign-ups. Key projects include RoomGPT, which used ControlNet for room redesign and reached 6 million visitors, and AI Commit, an open-source CLI tool adopted by 30,000 developers. He emphasizes using tools like the Vercel AI SDK and v0.dev to accelerate development, making apps free and open source to drive growth, spending 80% of time on UI, and launching quickly with minimal fine-tuning.