AI SDK

Why Eval++ Is the Next Great Compute Primitive — Sunil Pai & Matt Carey, Cloudflare

Jun 8, 2026 · 24:51

Matt Carey and Sunil Pai from Cloudflare's agents team argue that Durable Objects — stateful serverless with 15ms London latency — and Dynamic Workers — a safe, sandboxed eval for LLM-generated code — form the next compute primitive for AI agents. They explain how Durable Objects enable resumable streaming, multi-tab sync, and background scheduling out of the box without distributed systems engineering. Dynamic Workers allow running generated JavaScript strings in isolated sandboxes, reclaiming 30 years of avoided eval. The pair tease upcoming talks: one on collapsing Cloudflare's 2,600 API endpoints into a 1,000-token MCP tool, and another on a coding agent harness built entirely on Workers that they are already shipping.

Give Your Agent a Computer — Nico Albanese, Vercel

May 12, 2026 · 1:08:53

Nico Albanese demonstrates building an agent with Vercel's AI SDK v6, centered on the insight that giving an agent a file system transforms its behavior—it follows through on long tasks, stays on track, and builds on prior work. He walks through creating a tool loop agent from scratch, adding provider-executed web search with typed UI components, and integrating Vercel's persistent named sandboxes that snapshot state after inactivity. The agent gains a bash tool for file system access, a memories.md file for persistent memory injected into instructions each turn, and instructions to generate Python scripts for repeatable tasks so it accumulates tools across sessions. Albanese explains how these patterns scale: an internal agent called D0 reduced customer support tickets by 90% with a 95% satisfaction rate, and his personal coding agent ran for 104 minutes in one turn, using 316 tool calls and only 32% of GPT-5.4's context window with zero compaction. The session concludes with a preview of a larger sub-agent system that uses durable workflow steps and background sub-agents to keep the main thread under 7,000 tokens.

Feedback Loops are All You Need — Mehedi Hassan, Granola

May 10, 2026 · 10:11

Mehedi Hassan, a product engineer at Granola, argues that shipping AI features into production requires building feedback loops rather than one-shotting better prompts. Granola's chat feature for meeting notes revealed problems with web search — token costs ballooning to 10p per chat, overnight provider updates silently degrading results — and with prompt personalization, as a single prompt cannot serve salespeople, engineers, and HR managers equally. To close the gap, Granola built custom internal tracing exposing tool calls, search trails, reasoning, and cost in a UI accessible to non-engineers, not just CloudWatch queries. They also refactored their Electron app's renderer to run as a web app, enabling preview links on every PR and allowing Cursor to automatically test changes and upload screenshots. The result is faster iteration and confidence that shipped features actually work for customers.

Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop

May 7, 2026 · 50:25

Zubin Koticha and Danny Gollapalli of Raindrop argue that agent observability must shift from evals to production monitoring because agent failures are non-deterministic and unbounded. They break down explicit signals (tool error rate, latency, cost) and implicit signals (user frustration, refusals, task failure) detected by trained classifiers and regex, emphasizing that aggregate patterns even from imperfect regex are valuable. Experiments let teams ship changes to a percentage of users and compare semantic signal rates, with statistical relevance often reached after a few hundred events. Self-diagnostics—a simple tool and system prompt—enable agents to report their own failures, capability gaps, and even self-correction behavior, as demonstrated in a live coding agent demo where a disabled write tool caused the agent to use bash and then report the bypass. The episode covers alerting, trace visualization, data export to BigQuery/Snowflake, and the challenge of managing fast-paced experimentation at scale.

The New Application Layer - Malte Ubl, CTO Vercel

Apr 20, 2026 · 18:52

Vercel CTO Malte Ubl argues that AI engineering is the legitimate successor to web development and that the real value lies in the application layer, not the model labs. He identifies four effective agent archetypes—24/7 support, compressed research, surfacing existing information, and eliminating boring work—and reveals that over 60% of vercel.com page views now come from AI agents. Ubl predicts model companies will commoditize, driving costs down and empowering engineers, while citing Europe's leadership in AI engineering through Vercel's AI SDK, Pi (a coding agent from Austria), and OpenClaw. He stresses the need for open-mindedness toward paradigm shifts and new infrastructure, such as sandboxed agent runtimes, and warns of impending security challenges.

Building durable Agents with Workflow DevKit & AI SDK - Peter Wielander, Vercel

Jan 6, 2026 · 1:09:49

Peter Wielander from Vercel introduces the Workflow DevKit, an open-source library that adds durability, observability, and resumability to AI agents by wrapping them in a workflow pattern. He demonstrates converting a coding agent into a workflow-supported agent, showing how steps like LLM calls and tool executions become isolated, retryable units. The toolkit enables long-running agents that can sleep for days, resume streams after disconnection, and integrate human-in-the-loop via webhooks. Deployable on Vercel or any cloud, it provides built-in observability through a local UI. The Workflow DevKit is currently in beta with general availability targeted for January.

Evals Are Not Unit Tests — Ido Pesok, Vercel v0

Aug 6, 2025 · 15:22

Ido Pesok, an engineer at Vercel working on v0, argues that evals—not unit tests—are the key to making LLM applications reliable, using the basketball court analogy of understanding your domain (the court), collecting user queries (data points), and scoring passes/fails. He illustrates with the Fruit Letter Counter app, which failed in production despite passing manual tests, because LLMs are inherently non-deterministic. Pesok advises collecting data from thumbs-up/down, logs, and forums, and structuring evals with constants in data and variables in task (e.g., system prompt). He emphasizes deterministic pass/fail scoring, adding extra prompt tags for easy parsing, and integrating evals into CI/CD to detect regressions when changing models or prompts. The episode stresses that improvement without measurement is limited, and evals enable systematic reliability gains.

Vercel AI SDK Masterclass: From Fundamentals to Deep Research

Apr 20, 2025 · 59:52

Nico Albanese from Vercel delivers a workshop on building agents with the AI SDK, demonstrating how to use generateText, tools, and structured outputs to create a deep research clone in Node.js. He shows that switching between models like GPT-4o mini, Perplexity Sonar Pro, and Gemini Flash 1.5 requires only changing one line of code. Tools with maxSteps enable autonomous multi-step agents, as when the model fetches weather data for San Francisco and New York and sums the temperatures across three steps. The deep research workflow generates subqueries (e.g., 'requirements to become a D1 shotput athlete'), searches via Exa with live crawl, evaluates relevance with an evaluate tool that discards irrelevant results, and recursively deepens research using depth and breadth parameters. Finally, o3 mini synthesizes the accumulated research into a structured Markdown report, all in 218 lines of code.