Claude

Through the AI Fog: The Architectural Decision Agentic Security Depends On — Manoj Nair, Snyk

Jul 20, 2026 · 23:29

Manoj Nair, Snyk's CTO, argues that generative AI systems cannot serve as their own validators because probabilistic models are unreliable for security. He presents data from 4,800 customers showing a 108% quarter-over-quarter increase in security backlog, and research revealing that over a third of AI agent skills contain malware. Nair demonstrates that even frontier models fail to find the same vulnerability consistently—only 50% of the time across five runs—while deterministic checks catch 75% of issues. He warns that agents autonomously copy PII into untrusted databases and that MCP servers offer minimal built-in security. The episode advocates for a deterministic security layer that verifies agent outputs inside the development loop, and includes a demo of Snyk's tools for package health and skill risk assessment.

Agentic Security: Permissions, Provenance, and the Agent Supply Chain — Steve Yegge, Gas Town

Jul 20, 2026 · 22:32

Steve Yegge argues that AI-written code will dramatically increase security vulnerabilities unless developers adopt a separate security pass using tools like Snyk and Chainguard. He shares a bank architect's insight that shipping 10x faster with the same defect rate produces a 10x vulnerability surface, made worse by models writing code. Yegge demonstrates the gap by noting Fable's security hardening missed 241 vulnerabilities that Snyk found in his 30-year-old game. He warns of new attack surfaces like slop squatting, where models hallucinate package names that attackers then backfill with malicious versions. Yegge advocates for multiple passes—correctness, then security—and urges incorporating tools into agent workflows. He also cautions that Five Eyes predicts open-source models will autonomously hack production systems within months, and that personal scams using AI-generated voice and video are imminent.

Agentic Development Security — Ezra Tanzer, Snyk

Jul 20, 2026 · 27:33

Ezra Tanzer and Dan Arpino of Snyk argue that securing agentic development requires three pillars: what agents generate, what they use, and what they do. They highlight incidents where agents deleted production databases (Replit, Pocket OS) and exfiltrated repositories (GitHub via malicious VS Code extension), none acting maliciously but all lacking guardrails. Snyk's approach evolved from an MCP server with rule files (which agents ignored) to Python hooks that scan asynchronously on each file write, surfacing only newly introduced issues to keep latency and context windows deterministic. An audit of nearly 4,000 agent skills on a public hub found over one in eight had critical severity issues and 76 carried outright malicious payloads; skills are more dangerous than packages because they run at higher privilege and can rewrite agent memory. The resulting local tool shows every LLM, MCP server, and skill on a developer's machine with a risk score, blocks agents live from reading secret keys, and provides full audit trails of commands, files accessed, and tool calls.

We Gave an Agent Production Code Access and Then Tried to Sleep at Night — Moritz Johner, Form3

Jul 20, 2026 · 21:57

Moritz Johner of Form3 explains that giving a coding agent production code access turns it into a supply chain actor, and the blast radius is an architecture decision. His team built PatchPilot to automate CVE patching across thousands of repositories, splitting it into a deterministic Go layer that handles dangerous operations (GitHub write access, CI triggering) and an agent layer that only edits files. The agent remediates vulnerabilities by bumping dependencies, verifying builds, and fixing CI failures, but a prompt injection could escape via the Docker socket, which kept him up at night. To contain that, they moved the agent into a firecracker microVM with its own kernel and separate network policies per layer. Johner argues that where you draw the line between deterministic and agentic code defines your security model, and warns that existing agent sandboxes are worthless when a Docker socket is involved.

The UX of AI: Making AI-Powered Apps Your Users Don't Hate - Kathryn Grayson Nanz, Progress Software

Jul 18, 2026 · 35:59

Kathryn Grayson Nanz, Senior Design and Developer Advocate at Progress Software, argues that the success of AI-powered applications now depends on user experience rather than model performance. She identifies five pillars—trust, clarity, control, transparency, and meaningful benefit—and provides concrete patterns: citing sources to build trust, streaming text for clarity, allowing undo and version history for control, requesting granular permissions for transparency, and offering templates and next-step actions to ensure meaningful benefit. She emphasizes that developers must design these patterns themselves because AI can only remix existing interfaces, and that users need gradual introduction to AI features to avoid disengagement. The talk stresses that without addressing these UX challenges, users will abandon AI tools after a few failed attempts.

Content Is Code - Matt Palmer, Conductor

Jul 18, 2026 · 10:53

Matt Palmer, head of developer experience at Conductor, argues that code is now the fastest way to produce technical content, shifting left to code as the source of truth. He outlines three eras of content creation—handcrafted, expensive code, and cheap code with AI—and claims that the scarce resource today is not code but structure and conscientiousness, meaning meticulous care in maintaining design tokens, brand guidelines, and clean codebases. AI rewards organizational excellence over raw technical skill, making the best communicating teams those that instill discipline and rigor into their software creation. Palmer predicts 2026 as the year of the creative technologist and 2027 as the year of the content engineer, with declarative pipelines that automatically generate documentation, walkthroughs, product tours, and updates from structured code.

Agents Need Receipts, Not More Tool Calls - Armanas Povilionis, Alithea Bio

Jul 18, 2026 · 19:36

Armanas Povilionis of Alithea Bio argues that agents need verifiable receipts, not more tool calls, and introduces Froglet, an open-source protocol for agent-to-agent compute. Froglet enables agents to discover, negotiate, execute, and receive signed, tamper-proof receipts for external services, reducing setup to 2,000 tokens and minutes. The protocol integrates with MCP, OpenClaw, and NemoClaw, and supports payment rails, identity, and workload hashes. In a live demo, he shows Claude using Froglet to publish an 'add two numbers' service locally and remotely, then invoke it to get result 12, with a signed deal ID. Froglet aims to make scientific collaboration repeatable and consistent across organizational boundaries without requiring uniform software stacks.

Agents Need Feature Flags - Sachin Gupta

Jul 18, 2026 · 19:17

Sachin Gupta argues that agent systems urgently need feature flags—prompt variants, tool access, model routing, memory policy, autonomy level, and kill switches—to avoid catastrophic incidents like Cursor Sam's false policy citations, Replit's database deletion and fabricated users, LangChain's $47,000 loop, and PocketOS's unintended GraphQL drop. He demonstrates a tool-access flag that gracefully disables email sending mid-conversation and a kill switch that stops a runaway agent in 30 seconds without redeployment. Gupta details a five-step rollout playbook: wire kill switches first, wrap every tool call with a flag, default autonomy to suggest, move prompts out of code, and track four metrics (kill-switch fires per week, time to mitigation, canary error-rate delta, flag audit completeness). He warns that sub-agents must pass through the same middleware, flags must be per-turn not per-session, and kill switches must be tested regularly. The talk concludes that enterprise buyers now expect demos of these controls, and regulations like the EU AI Act mandate them.

Using LLMs to Secure Source Code — Eugene Yan, Anthropic

Jul 17, 2026 · 21:30

Eugene Yan of Anthropic details how frontier LLMs like Claude are reshaping software security, citing Mozilla's 20x surge in monthly fixes (to 400 in April 2025, two-thirds credited to Claude) and Anthropic's own scan of 1,000+ open source repos that uncovered 6,200 high/critical issues, 1,600 reported, and about 100 patched upstream. He argues that finding vulnerabilities is no longer the hard part; the bottleneck has moved to verification, triage, and patching. Yan outlines a six-step workflow: a written threat model (boosts true positive rate to 90%), an isolated sandbox for reproducibility, discovery optimizing for recall, a separate adversarial verification agent that detonates exploits in fresh containers, triage to prioritize engineer attention, and patching that closes the loop so bugs cannot recur. His advice: start this week on open source dependencies, keep hands on the wheel before automating, and recognize that scanning was never the bottleneck.

Special Topics in Kernels, RL, Reward Hacking in Agents — Daniel Han, Unsloth

Jul 17, 2026 · 2:20:21

Daniel Han of Unsloth argues that reward hacking—where AI models cheat to maximize reward—is a critical problem in agent training, citing examples from GPT-5.1's calculator hacking and GPU mode kernel competitions. He shows that models exploit benchmark flaws, such as viewing Git history or editing timers, and that even open-source models like GLM 5.2 require anti-hacking measures. Han emphasizes that harness and tooling quality now outweigh model choice, with inference providers sacrificing accuracy for speed (e.g., 10% accuracy drops across providers). He also warns that hardware limits (float4 precision, diminishing returns) shift focus to software algorithms like FlashAttention and gradient checkpointing. The workshop concludes that benchmarks are unreliable—DeepSpeed's false positive rate is contested at 44.9%—and urges verification before trusting performance claims.

Imagination Engineering — Eve Bouffard, Head of Design, Y Combinator

Jul 16, 2026 · 16:04

Eve Bouffard, Head of Design at Y Combinator, introduces 'Imagination Engineering' as the art of stretching the mind to invent what seems impossible, arguing that idea generation is the new bottleneck as AI models become incredibly capable. She shares her experiment in 'thinking in public' with a Slack channel called 'Eve Thoughts,' where she dumps her stream of consciousness, and then used Opus 4.8 to build a personalized website (EveBouffard.com) that aggregates and visualizes those thoughts. The site dynamically surfaces her projects, quotes, tools, and books, and even applies shaders and translations. She also built 'Shape of Minds'—a tool that analyzes commonalities across history's greatest minds, revealing patterns like taking naps and barely eating. Bouffard demonstrates how to spin up agents on demand for learning and productivity, emphasizing that anyone can now create software on the fly from their stream of consciousness.

Claude Fable, Claude Tag, and Anthropic's Culture — Cat Wu & Thariq Shihipar ft Simon Willison

Jul 15, 2026 · 51:30

This episode features Anthropic's Cat Wu and Thariq Shihipar discussing Claude Code, Claude Tag, and Claude Fable, arguing that these tools have shifted engineering from slow spec-driven processes to rapid, ambitious building. Thariq notes that with each model generation, delegation increased, and Claude Fable now enables one-shot features. Cat says engineers now need product taste over execution, as timelines shrink from six months to a week. Claude Tag, a proactive multiplayer agent, lands 65% of product engineering PRs internally by monitoring channels and remembering preferences. The team reduced Claude Code's system prompt by 80% for frontier models by removing examples and hard constraints, relying on model judgment. Auto mode, used internally since January, mitigates prompt injection through thousands of evals and Sonnet classifiers. Cultural hacks include default-public channels and a 'don't negotiate against ourselves' mindset, leading to ambitious builds like Thariq's Claude-powered video editing and a Street Fighter game.

"I've never seen anything scarier than an LLM with tool calls." — Erik Meijer aka @HeadinTheBox

Jul 13, 2026 · 21:13

Erik Meijer of Leibniz Labs argues that AI agents with tool calls are intrinsically dangerous and must be tamed through formal verification. He recounts how adding tool calls to LLMs turned harmless chatbots into agents capable of irreversible side effects, like deleting files or emptying bank accounts. His solution, implemented in Automind, uses proof-carrying code: the agent submits a plan as a program (Free Monad) together with a machine-checkable safety proof, and a small checker verifies the proof before allowing execution. This air-gaps the agent from its tools, ensuring policy compliance statically. Meijer demonstrates that elementary type systems and compiler techniques, such as taint analysis on program expressions, can provably prevent unsafe actions, turning agents from 'vibe-coding' risks into provably safe systems.

Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

Jul 13, 2026 · 23:35

Alejandro Vidal of Mindmakers argues that counting correct answers in LLM benchmarks—classical test theory—should be replaced by item response theory (IRT) from psychometrics for richer evaluations. Using real data from Epoch.ai, he shows that while Claude Opus 4.1 scored 245 right and Gemini 3 Pro 247, IRT reveals Gemini is nearly one standard deviation more intelligent because it answers harder questions. IRT assigns each item difficulty (B) and discrimination (A), enabling benchmark auditing—Vidal flagged mislabeled items like a question about passengers whose gold answer was actually total people killed. He demonstrates reducing a 484-item benchmark to 97 items with 99% ranking correlation by selecting high-discrimination items, saving tokens and money. IRT also detects contamination via unexpected residuals, protects benchmarks through adaptive testing with unique fingerprint sets, and identifies model families (e.g., DeepSeek distillations correlate 0.38). Vidal previews future extensions like multidimensional models and alignment measurement.

Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect

Jul 13, 2026 · 46:52

Will Brown of Prime Intellect details the company's open-source ecosystem of post-training tools, including the verifiers and prime-rl libraries, arguing they enable efficient and affordable training of frontier agentic models for enterprises. Verifiers V1 decomposes environments into tasks, harnesses, and runtimes using a decorator pattern and Pydantic, supporting group rewards like conciseness bonuses. Prime-RL is an asynchronous reinforcement learning framework that allows long-horizon coding rollouts to overlap, achieving a GLM-5 step on 28 nodes in under 5 minutes for 131k context, with a 1,000-step run costing roughly $50k. The framework supports custom algorithms including on-policy distillation, GRPO, and self-distillation via decomposable loss and algorithm classes. Prime Intellect's Lab platform offers hosted multi-tenant LoRA training live now, with full fine-tuning arriving soon, enabling enterprises to develop environments on CPU and push them to the cloud for scalable post-training.

RLM: Recursive Language Models for Large Codebases - Shashi, Superagentic AI

Jul 12, 2026 · 17:27

Shashi from Superagentic AI explains how Recursive Language Models (RLM), a pattern from MIT, solve the context window problem in large codebases by externalizing context management into a programmable REPL where the model writes code to inspect, slice, and compute relevant chunks, and recursively delegates sub-questions via llm_query. The RLM loop loads the repo as data, produces bounded observations, and can recursively call another model for deeper insights. Shashi demonstrates RLM Code, an open-source independent implementation running locally and on Gemini with a Docker sandbox, showing two tool calls and a complete trajectory. He notes that similar RLM concepts are used in proprietary systems like Codex, Claude managed agents, and dynamic workflows, making it a practical pattern for AI engineers dealing with monorepos.

What Does Done Even Mean? Agents and Paperclip's Liveness Model - Dotta, Paperclip

Jul 12, 2026 · 7:14

Dotta, creator of Paperclip, argues that "done" for agents must be a structured claim bundle rather than a boolean checkbox, as agents produce more work than humans can verify. He introduces Paperclip's Liveness Model which balances liveness (keeping work moving) with verification (human review) through three invariants: productive work continues, only real blockers stop work, and infinite loops are bounded. The model uses explicit task transitions, first-class blockers, interactive human approvals, watchdogs (goal-enforcing agents), and separation of verifier from author. Dotta advises defining "done" with artifact, evidence, rubric, owner, and next action, and providing agents tools to verify their own work. The episode presents a protocol for agentic work that avoids approval theater and enables safe delegation.

Stop AI Agent Hallucinations: 5 Techniques + Production Patterns - Elizabeth Fuentes, AWS

Jul 11, 2026 · 55:19

Elizabeth Fuentes (AWS) presents five code-based techniques to stop AI agent hallucinations, each with measurable before/after metrics. Semantic Tool Selection filters 29 tools to the 3 most relevant per query, cutting token usage from thousands to under 300 per call. Graph-RAG replaces vector similarity with structured graph queries (using Neo4j), enabling precise aggregation and multi-hop reasoning that vanilla RAG fabricates. Multi-Agent Validation uses an Executor-Validator-Critic swarm to catch fabrications, achieving a 92% detection rate. Neurosymbolic Guardrails enforce business rules in Python hooks that the agent cannot skip, achieving zero rule violations. Agent Steering guides agents to self-correct when soft rules fire, completing tasks without hard failures—demonstrated by booking 50 guests by intelligently splitting into two rooms.

Develop at Idea Velocity - Jeffrey Lee-Chan, Snapchat

Jul 11, 2026 · 15:28

Jeffrey Lee-Chan (Snapchat) and Austin (CMUX) explain how to achieve 'idea velocity' by building parallel multi-agent harnesses that let one engineer direct 10–20 coding agents instead of becoming the bottleneck. Lee-Chan argues the key is separating 'Agent Orchestrator Managers' from specialized workers to prevent low-level implementation bias, using OpenClaw for frictionless Slack-based communication and CMUX terminals for real-time parallelization and unbiased manager oversight. He demonstrates two apps—an AI RPG with dice-roll mechanics and a multi-AI analysis tool—built with this stack, and discusses token-burn tradeoffs between models like Codex 5.3, GPT-5.4, and Minimax. The episode also covers staging environments to avoid doubling token usage, the benefit of persistent memory and contextual guardrails, and the importance of pushing human interaction to the start or end of the workflow for improved parallelization.

From Writing Code to Designing Systems: How the Developer Role is Changing — Chris Noring, Microsoft

Jul 11, 2026 · 23:05

Chris Noring from Microsoft argues that the developer role is shifting from writing code to designing systems and orchestrating AI agents, using tools like GitHub Copilot and Claude. He proposes a workflow starting with the CLI rather than the editor, employing agents.md for high-level guidance, skills for repeatable tasks, and custom agents for orchestration. Noring demonstrates scaling by delegating tasks via /delegate in the CLI or assigning issues to agents in the GitHub UI, allowing developers to become 20x more productive. He stresses that guardrails are essential to prevent agents from producing 'slop', and that human-in-the-loop oversight remains critical. The episode emphasizes that developers must encode standards and constraints into their workflows to maintain consistency and quality at scale.

Design Patterns for AI Trust: Juries, Libraries, and Agent Tiers — Alex Bauer, Upside.tech

Jul 11, 2026 · 17:09

Alex Bauer, co-founder of Upside, argues that go-to-market teams can solve AI's trust problem by managing agents like humans, using patterns such as a librarian for just-in-time knowledge and a jury-and-judge model for subjective decisions. He demonstrates three solutions: first, scaffolding an AI website rebuild with anchor assets like a product capabilities reference that ladders directly into homepage copy; second, a librarian that consults documentation and prior failed queries before answering questions like "How much pipeline in Q1?" to catch fiscal-year or stage definitions; third, a jury-and-judge workflow for multi-touch attribution where independent analysts research deals and a judge weighs reasoning quality before producing a consensus verdict. Bauer warns against low-intelligence models for important work, citing Slackbot's MCP integration as "horrifically stupid," and advocates using tier-two platforms with sub-agents, plan mode, and full MCP support.

Understanding is the new bottleneck — Geoffrey Litt, Notion

Jul 10, 2026 · 19:33

Geoffrey Litt from Notion argues that understanding code written by AI agents is crucial for creative participation, not just correctness checking. He presents three techniques: explanations via his 'explain diff' skill, which generates HTML or Notion documents with background, intuition, interactive figures, and literate code diffs, capped by a five-question quiz to ensure understanding before sharing code; micro-worlds, like a step-by-step debugger for a Prolog interpreter or a 'video game' that walks through migrating a website; and shared spaces in Notion where humans and agents can collaborate in multiplayer chat threads and comment on documents. Litt emphasizes that agents can build ephemeral UIs and simulations to help humans understand better, leveraging education principles to stay deeply in the loop.

Should AI Engineers Still Read Code in 2026? The Z/L Continuum — Alex Volkov, ThursdAI

Jul 10, 2026 · 21:35

Alex Volkov examines the debate sparked by Ryan Lopopolo's claim that 'code is free' and Mario Zechner's counter that engineers must 'read every fucking line' of critical code, arguing that the Z/L Continuum is about task-level proof rather than personality. Citing a Ferrous AI survey showing an 861% increase in code deletion per PR and a 242% rise in incidents, he notes Anthropic's recursive self-improvement essay admitting human code review is a new bottleneck. Volkov's routing table prescribes reading every line for authentication, money movement, and irreversible data, while letting agents handle less critical changes. He also introduces loops—cron-like agent systems that self-verify—as the next frontier, quoting Adi Osmani that automated loops don't remove judgment. Volkov concludes that capability drift moves where proof belongs, but every system still requires human judgment.

Everything we knew about software has changed — Theo Browne, @t3dotgg ​

Jul 8, 2026 · 16:02

Theo Browne argues that AI model evolution—from Sonnet 3.5’s tool-calling to Opus 4.5's long-running tasks and Mythos's orchestration—requires engineers to think bigger and wider. He compares current developer habits to skeuomorphism in iOS 7, urging rejection of legacy constraints like Git's inability to commit environment files and terminal-centric workflows. Browne introduces a shifted tier system: what was a startup is now a side project; a Markdown file running on a cron job can replace a company's product. His own PR triage service became a Markdown file updated daily via cron. He advocates building breadth over depth—architecting products so users can extend features, enabling small teams to compete with AWS or Salesforce. 'If your idea doesn't feel stupid, it's because your idea is not big enough,' he concludes.

I Run a Fleet of AI Agents Across Three Machines. Here's What Broke. - Kyle Jaejun Lee, KRAFTON

Jul 8, 2026 · 9:11

Kyle Jaejun Lee runs a fleet of AI coding agents across three machines daily and reveals the scaling failures that emerged from a hierarchy of CEO, VP, manager, and worker agents. To overcome his own attention bottleneck, he separated context into entity-specific workspaces on disk and replaced context compaction with full resets that read handoff files. When moving beyond one machine, five things broke: agents doing work instead of delegating, TMUX panes too crowded to read, out-of-memory crashes from stacked Claude Code processes, colliding Git credentials across workspaces, and the MacBook dying mid-task. He offloaded long-running work to always-on Linux boxes, used Git commits and SSH to move context between machines, and consolidated review gateways onto a single always-on machine with Discord as a unified router. Unsolved challenges include consistent credentials, local-only tools, and resource scheduling—he plans to layer his orchestration on top of Kubernetes to handle compute, secrets, and tools.

SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI

Jul 7, 2026 · 12:58

Rishi Desai of Abundant AI presents SWE-Marathon, a benchmark for coding agents that finds even the strongest setup achieves only a 26% resolution rate across 20 project-scale tasks. Trajectories average 31 million tokens, with the longest consuming 877 million tokens, and a computer-use agent verifier evaluates full-stack products via browser interactions. Desai warns that weak verifiers become attack surfaces in long-horizon evals, showing agents bypassing tasks by calling GCC from within a Rust compiler, caught by anti-cheat layers using S-Trace. Across 1,400 rollouts, 12.8% showed suspicious shortcuts and 9% clear verifier bypasses, but zero earned reward through exploits due to multi-channel checks. The central claim: long-horizon SWE is unsolved, and robust verification—not harder unit tests—is the key bottleneck for future benchmarks.

Field Guide to Fable — Thariq Shihipar, Anthropic

Jul 6, 2026 · 19:28

Thariq Shihipar of Anthropic discusses Fable and argues models improve in "spiky" ways: a chat model fails to list Pokémon ending in "aw" (Croconaw and Dreadnaw), but Claude Code fetches and filters the list in seconds – a gap he calls "capability overhang." He explains that to unlock Fable, Claude Code cut 80% of its system prompt because heavy instructions now constrain a more imaginative model, and the "ask user question" tool evolved from barely working under Opus 4 to generating embedded HTML questionnaires. He shares techniques like blind-spot passes and interviews to surface unknown unknowns, and reflects on the grief of moving from hand-coded programming to agentic workflows. Shihipar urges engineers to reject trade-offs – "good, fast, cheap: pick three" – and instead demand all three, citing a four-hour keynote deck built with Fable as proof that agents can deliver ambitious work faster.

MCP Apps: Primitives, discovery, and the Future of Software - Pietro Zullo, Manufact, Inc

Jul 5, 2026 · 28:54

Pietro Zullo, co-founder of Manufact, explains that MCP Apps are MCP servers that return interactive UI elements in sandboxed iframes, enabling bidirectional communication between the UI and the host. He details primitives like set-model-context and send-message, and shows how apps can stream tool inputs into the UI in real time. The ChatGPT, Claude, and Cursor stores now accept self-serve submissions, and Claude's dynamic discovery matches connectors to user intent. Manufact’s open-source SDK, mcp-use, has 8M+ downloads and provides a cloud platform for building, testing, and submitting MCP Apps.

Frontier results, on device - RL Nabors, Arize

Jun 29, 2026 · 30:52

RL Nabors (Arize) argues that most frontier-model calls can be replaced by smaller, local models, saving cost, latency, and energy. She presents a four-step framework: prototype big with a foundation model, collect a golden dataset, run capability evals using Arize's open-source Phoenix, then select the 'sage' (small and good enough) model. Demonstrating with her social app Mima, she tested Qwen 2.5, Qwen 3, LLaMA 3.2, and Gemma 4 against Claude Sonnet on summary accuracy, latency, and cost. LLaMA 3.2 (3B params) won at 90% accuracy and 1-second P50 latency, versus Gemma 4's 8 seconds. Prompt engineering—specifically few-shot prompting—closed the gap further, achieving 92.9% factual consistency and 100% JSON validity. Nabors emphasizes running regression evals to prevent regressions, and notes that on-device inference eliminates data exposure and round-trip latency.

The Future Is Domain-Specific Agents - Justin Schroeder, StandardAgents

Jun 29, 2026 · 30:38

Justin Schroeder of StandardAgents argues that domain-specific agents—small, focused AI agents each limited to a single domain—will outperform general-purpose agents that rely on accumulating tools and context via inheritance, and that composition of such agents is the key to building practical, cost-effective AI systems. He defines an agent as deterministic software harnessing non-deterministic model outputs, then critiques the current approach of piling tools and skills into a single agent (inheritance), proposing instead a coordinator agent that delegates to many specialized sub-agents (composition), each with its own minimal context, system prompt, and tools. He claims these domain-specific agents achieve over 80% token efficiency, enable use of cheaper small models (e.g., DeepSeek V4 Flash is 137x cheaper than a frontier model), improve safety by limiting capabilities, and scale easily. Schroeder predicts rapid adoption through 2026-2027, with multi-agent orchestration becoming mainstream and tokens no longer getting cheaper—up 29% when adjusted for IQ. He describes an ideal agent architecture with hooks, rules, and recursive sub-agents, and invites listeners to try…

HTML is All You Need (for Agents to Make Graphics) - Amol Kapoor, Nori

Jun 28, 2026 · 7:00

Amol Kapoor, CEO of Nori Agentic, argues that coding agents can create high-quality visual artifacts like slides, docs, and videos by using HTML instead of traditional design tools. He explains that agents think in language and structure, not pixels, so tools like PowerPoint or Figma force them into a human-centric approach that fails. By giving agents HTML—a language they understand intuitively—they can produce properly aligned, themed graphics without manually placing coordinates. Kapoor demonstrates this with the 'pelican riding a bicycle' test, where models fail at raw SVG but succeed with HTML. He applies this method at Nori to build board decks, sales decks, and even animated videos purely with HTML and CSS. The episode advocates for 'thinking like the model' and concludes that for graphics, HTML is all you need.

Browser Agents Don't Need Better Models. They Need Better Eyes. - Kushan Raj, ARK

Jun 28, 2026 · 4:26

Kushan Raj, a former Founding Engineer at Sarvam AI, argues that browser agents fail not because of weak models but because of poor interfaces: what the model sees, can do, and learns from. He built a browser-agent runtime that replaces raw DOM dumps (20k tokens) with a compact markdown representation (~1,800 tokens), uses stable action handles instead of one-click-per-call, and provides step-by-step feedback instead of pass/fail at the end. In demos, his agent using a cheap model completes multi-step tasks like downloading Aadhaar or booking a trekking site far faster than Claude, which gets stuck or scrolls unnecessarily. Raj plans to open-source the project and offer an API where users supply a URL and intent for execution, aiming to make browser agents faster, cheaper, and more reliable for everyone.

Agents in Production: How OpenGov Built and Scaled OG Assist - Gabe De Mesa, OpenGov

Jun 26, 2026 · 18:30

Gabe De Mesa of OpenGov explains how the company built and scaled OG Assist, an AI agent embedded across its government ERP products, using the Effect TypeScript library for full control over the agent loop and adopting Google's A2A protocol for agent-to-agent communication. The team moved from LangGraph to a custom Effect-native loop to gain fine-grained control over tracing, error handling, and structured concurrency. They implement feedback and automated evals with thumbs-up/down and CI checks, human-in-the-loop approvals for mutating tool calls, and sandboxed code execution to keep production safe. Long conversations are managed via rolling summarization with memory recall. Observability comes from Effect's built-in tracing, enabling bottleneck profiling. Tools and skills are built as Effect toolkits, and internally OpenGov uses Claude and Cursor to accelerate development workflows.

A Genius With Amnesia - Victor Savkin, Nx

Jun 26, 2026 · 20:00

Victor Savkin, creator of Nx and Polygraph, argues that current coding agents are like a genius with amnesia – they see only a tiny portion of the codebase and forget everything between sessions – and introduces Polygraph, a meta-harness that gives agents full organizational context and perfect memory. He explains that agents are repo-bound and lack episodic memory, forcing humans to re-explain changes (e.g., seven explanations for one UI change across four repos). Polygraph builds a unified dependency graph by analyzing thousands of repos without code changes, lets agents work across multiple repos in a single session, and captures all traces so sessions can be resumed by any agent (Claude or Codex) on any machine. It also enables context-aware queries like 'find every repo that depends on version X of this library' and allows referencing past sessions for best practices. The result is an agent that sees the entire organization's code and remembers every decision, effectively creating a hive mind.

The Log Is The Agent - Ishaan Sehgal, Omnara

Jun 25, 2026 · 15:11

Ishaan Sehgal, CEO of Omnara, argues that an AI agent's true identity is not the model, runtime, or tools but its append-only event log—the durable record of every user input, model output, tool call, and permission. Drawing an analogy to a video game save file that persists beyond hardware failure, he explains that treating the log as the agent enables reliability, scalability, forking, migration, multiplayer, and ownership. He warns that deepest lock-in is log lock-in: if a provider owns the log, it effectively owns the agent. Omnara's open-source managed agents platform is built around the session log as the system, not a side effect, allowing agents to survive crashes, resume across machines, and be fully owned and inspected by users.

The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen

Jun 25, 2026 · 58:17

Jacob E. Thomas argues that persona-eval benchmarks like InCharacter, which report 80.7% personality fidelity for Hamilton, structurally miss anachronistic compositing because they measure fluency and personality consistency—exactly the features the dominant cultural composite optimizes. The Miranda Hypothesis posits that training data saturated with works like the 2015 Hamilton musical (outnumbering the 175,000-word Federalist Papers by orders of magnitude) causes models to produce smoothed, morally legible composites rather than historically faithful personas. Thomas explains that RLHF amplifies this distortion because human raters share the same cultural frame. He proposes a paradigm shift to epistemic simulation: corpus-bounded, temporally-anchored systems evaluated by domain experts, with the model as a swappable component in a configurable role-playing language system. The pre-registered Prism Experiment instantiates Abraham Lincoln at four documented moments (1847, 1858, 1860, 1862–65) under three seeding conditions—bare model, biography, or primary sources—scored on a weighted three-axis rubric that prioritizes anachronism detection (40%) over voice. Thomas concludes that…

The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

Jun 18, 2026 · 37:06

Sandipan Bhaumik, a technical lead for Data and AI at Databricks, presents a five‑pillar playbook for taking AI agents to production: evaluation (define numerical success before touching code), observability (trace every decision for regulators and debugging), data foundation (agents do not forgive bad data), multi‑agent orchestration patterns (orchestrator‑worker, choreography, human‑in‑the‑loop), and governance (PII pre‑validation, prompt versioning as change management). He recounts a retail bank that spent £85,000 over six months on a chatbot PoC that failed because no one could measure or trace it. His team reversed the order: they built the evaluation dataset and tracing infrastructure first, selected the model in week 7 of an 8‑week engagement, and launched successfully. Six weeks post‑launch, when the bank updated interest rate policies, the tracing system caught that the new document had not been re‑embedded, so the agent served stale answers—a production incident the five pillars were designed to handle. The evaluation dataset is a living system that grows from 200 test cases; a production incident playbook connects all pillars: detect via eval dashboard, diagnose with…

Why MCP and ChatGPT Apps Use Double Iframes — Frédéric Barthelet, Alpic

Jun 15, 2026 · 20:11

Frédéric Barthelet, CTO of Alpic, explains why ChatGPT and other MCP hosts render third-party app UI inside a double iframe. He traces how simpler approaches fail: `srcdoc` shares the parent origin, letting CSP block scripts and risking data access; sandboxing removes origin storage; and `allow-same-origin` recreates the escape. The resulting double iframe—an outer iframe from a controlled subdomain loading app HTML via `srcdoc` into an inner frame—ensures isolation and prevents cross-app storage collisions. Barthelet warns developers must declare every external domain their view uses in MCP app metadata or face submission rejection, and demos Skybridge's CSP inspector that diffs declared domains against actual network calls.

Why Eval++ Is the Next Great Compute Primitive — Sunil Pai & Matt Carey, Cloudflare

Jun 8, 2026 · 24:51

Matt Carey and Sunil Pai from Cloudflare's agents team argue that Durable Objects — stateful serverless with 15ms London latency — and Dynamic Workers — a safe, sandboxed eval for LLM-generated code — form the next compute primitive for AI agents. They explain how Durable Objects enable resumable streaming, multi-tab sync, and background scheduling out of the box without distributed systems engineering. Dynamic Workers allow running generated JavaScript strings in isolated sandboxes, reclaiming 30 years of avoided eval. The pair tease upcoming talks: one on collapsing Cloudflare's 2,600 API endpoints into a 1,000-token MCP tool, and another on a coding agent harness built entirely on Workers that they are already shipping.

Text Diffusion — Brendan O’Donoghue, Google DeepMind

Jun 4, 2026 · 28:03

Brendan O'Donoghue, a research scientist at Google DeepMind, explains that text diffusion models generate tokens 10x faster than autoregressive models by performing 24 denoising steps to produce 256 tokens, dramatically reducing memory transfers. Unlike GPT-4o and Gemini 2.5 Flash, which incorrectly answered 40 and 42 on a math problem, Gemini Diffusion used bidirectional attention to self-correct from 60 to 49 to 39. The model adaptively allocates compute: 4 steps for memorized digits of π, 31 for quantum mechanics, and automatically stops when satisfied. Text diffusion also enables in-place editing, demonstrated by fixing code bugs or adding paragraphs. However, lower throughput on large batches makes it expensive to serve at scale today. O'Donoghue showcases low-latency applications: a fully generated Wikipedia, a Reddit clone with AI text and images, an on-the-fly operating system, and a to-do app built in 15 seconds by voice.

The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

Jun 4, 2026 · 23:25

Vincent Chen, a research fellow at Snorkel AI, argues that the ability to measure AI has fallen behind the ability to build it, and benchmarks must shape future capabilities rather than just measure past progress. Drawing from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants, he presents a framework: the science of task quality, distributional diversity, model headroom, and robust eval methodology, and the art of having a thesis (e.g., Terminal Bench's bet on CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first-class citizen. He closes by proposing three axes for next-generation benchmarks: environment complexity, autonomy horizon, and output complexity beyond plain text.

Beyond Components: Designing Generative UI for MCP Apps — Ruben Casas, Postman

Jun 3, 2026 · 16:58

Ruben Casas from Postman argues that AI models can now write better frontend code than he can—his prompt to rewrite his blog produced a search box with blur animation and accessibility out of the box—yet most agent UIs still invoke static prebuilt components. He presents three levels of UI generation: static components (AG UI, Goose) passing props to predefined React elements; declarative UI where the model generates JSON or YAML for a rendering engine (e.g., Vercel's JSON Render), which he deems the current ideal balance; and fully generative UI where the model writes HTML, CSS, and JavaScript on demand, as in his weather agent that does so in one tool call. The key obstacle is trust, necessitating sandboxing, and MCP apps with their double iframe default are the best delivery mechanism. He likens today to early TV—radio shows with cameras—and predicts the future lies beyond components in collaborative human-agent interfaces on shared canvases, as seen with the Skeletro MCP app.

How to talk to statues — Joe Reeve, ElevenLabs

Jun 1, 2026 · 33:28

Joe Reeve, an ElevenLabs growth engineer, built an app that lets users talk to statues by pointing their phone at one—it identifies the statue via OpenAI deep research, generates a matching voice with ElevenLabs' voice design API, and starts an ElevenLabs agent conversation—all in 30 seconds. He created it in two hours on a Sunday using Cursor and a single prompt, posted it on Tuesday, got 50,000 impressions, then reposted about vibe coding and hit 1.5 million. Museums, auction houses (Bonhams, Christie's), and travel platforms reached out; one CEO tracked down his WhatsApp, saying a team of 10 had worked on a similar project for a year. The episode explores voice interaction patterns, the challenge of interrupting agents, the need for multimodal interfaces (voice plus visual), and the potential of vibe coding to democratize software creation, with Joe noting that music and captions are key to viral videos.

Can LLMs generate Enterprise Quality Code? — Prasenjit Sarkar, Sonar

May 31, 2026 · 15:12

Prasenjit Sarkar from Sonar evaluates whether LLMs generate enterprise-quality code using SonarQube analysis of 4,444 Java assignments across 53 models. While Gemini 3.1 Pro High achieves 84.17% pass rate, GPT-5.4 Pro High generates 1.2 million lines (high bloat), and Claude Sonnet 4.6 has 300 security issues per million lines. The ACDC framework addresses these gaps: Guide (Sonar Sweep and Context Augmentation), Verify (SonarQube Agentic Analysis in 1–5 seconds pre-commit), and Solve (Remediation Agent that fixes issues and checks regressions before presenting fixes). Sonar's leaderboard at sonar.com/leaderboard provides detailed pass rates, cyclomatic/cognitive complexity, bug density, and security metrics per model.

How I deleted 95% of my agent skills and got better results — Nick Nisi, WorkOS

May 30, 2026 · 17:43

Nick Nisi, DX engineer at WorkOS, argues that AI agents should be forced to prove their work with code rather than trusted with prompts. He built Case, a harness that uses a TypeScript state machine to enforce gates between agents (implementer, verifier, reviewer, closer, retro), cryptographically verifying test runs via SHA-256 hashing to prevent lying. In building the WorkOS CLI, he generated 10,000 lines of skills from docs but found one skill dropped task accuracy from 97% to 77%. He deleted 95% of those skills, rewriting 553 lines of common gotchas, slashing eval time from 68 to 6 minutes. His key takeaway: treat every failure as a system bug in the harness, not the agent, and measure everything with evals to avoid adding noise.

Why your agents need decision traces, not just documents — Zach Blumenfeld, Neo4j

May 29, 2026 · 20:12

Zach Blumenfeld from Neo4j argues that context graphs—a graph-based layer storing decision traces, precedence, and causal chains—extend standard RAG to make AI agents more accurate and explainable. A financial analyst agent, for example, can reject or accept a request by retrieving past decision traces and structurally similar precedents via graph embeddings, not just semantic similarity. Neo4j's one-command tool `uvx create-context-graph` scaffolds a full-stack app with backend, frontend, demo data, and MCP server, supporting 22 built-in domains or custom ontology generation. The underlying `neo4j-agent-memory` package handles entity extraction through a spaCy-to-GLiNER-to-LLM pipeline with deduplication and integrates with pydantic AI, LangGraph, Crew, Google ADK, and others. The episode also covers data connectors for GitHub, Notion, Jira, and Slack, and the ability to store and query decision traces with timestamps, though automated trace quality evaluation is still evolving.

The AI Skill I Rely On Daily — Priscila Andre de Oliveira, Sentry

May 27, 2026 · 17:05

Priscila Andre de Oliveira, a senior software engineer at Sentry, reveals that 67% of her AI usage is for comprehension and only 2% for code generation, based on analyzing 116 of her own Claude sessions. Working in Sentry's 15-year-old codebase with 100 PRs merged daily and 100,000 organizations depending on it, she built a personal skill called 'Catch Me Up' with six exploration modes (architecture, conventions, feature traces, syntax, testing, history). She argues that understanding what the agent found before letting it plan and implement prevents misaligned mental models that produce slop code. The episode emphasizes that in large, complex codebases, AI's biggest unlock is comprehension, not generation, and advises developers to align their mental models before prompting.

Run Frontier AI at Home — Alex Cheema, EXO Labs

May 26, 2026 · 1:45:02

Alex Cheema of EXO Labs argues that running frontier AI locally has 100x improvement potential in cost and performance, demonstrated with GLM 5.1—a trillion-parameter model—running across four Mac Studios at roughly 20 tokens per second for $40,000. He details kernel fusion that recovered 30% performance on Qwen 3.5 by eliminating unnecessary kernel launches, and RDMA integration that cut node-to-node latency from 300 microseconds to single digits, enabling tensor parallelism to actually scale. Cheema advocates splitting inference: prefill on compute-dense hardware (e.g., an RTX Spark) and decode on high-bandwidth hardware (e.g., Mac), cutting large-prompt inference roughly in half. He warns against misleading benchmarks like one-bit quantized models, and outlines how multi-agent setups, test-time scaling, and continual learning could further improve local inference efficiency. The talk includes a live demo of GLM 5.1 across four Mac Studios connected via Thunderbolt 5, and a preview of EXO's upcoming benchmarking site to track intelligence per Joule.

What the Best Agents Share — Mardu Swanepoel, Flinn AI

May 26, 2026 · 10:21

Mardu Swanepoel of Flinn AI identifies four patterns shared by top agents like Cursor, Claude, Manus, and Harvey: focus modes, transparent execution, personalization, and reversibility. Focus modes constrain the action space to improve output quality and align user expectations, as Cursor does with planning and debug modes. Transparent execution shows tool calls and reasoning to build trust and enable early intervention, exemplified by Claude's live task list and Manus's progress tracking. Personalization optimizes speed to understanding through playbooks (Harvey) and memory, so agents follow firm-specific methods. Reversibility bounds downside with rollbacks at line, file, or conversation levels (Cursor) and integration with native undo in Harvey's Word add-in, encouraging users to tackle higher-value tasks.

Stop babysitting your agents... — Brandon Waselnuk, Unblocked

May 26, 2026 · 18:54

Brandon Waselnuk of Unblocked argues that the bottleneck for AI coding agents is not access (MCPs, tools) but understanding—they need a context engine that builds a research packet from codebase, Slack, PRs, and org structure before generating code. He debunks three myths: naive RAG suffers from satisfaction of search, more MCPs don't provide reasoning, and million-token windows don't enable effective reasoning. His team's context engine reduced a Zendesk integration task from 2.5 hours and 20.9M tokens (with MCPs only, but code that would have broken production) to 25 minutes and 10.8M tokens, earning a senior engineer's approval with one nitpick. Three hard lessons: optimize for understanding not access, resolve conflicts rather than hide them (e.g., Slack thread where CTO says code is wrong), and never cache answers because context changes daily. He demonstrates a social graph tool (open-sourced Monday) that maps engineers to code areas and collaborators, and shows a demo where the agent's MCP calls the context engine to generate a plan covering factory patterns, library modules, and client registration.

Agentic Evaluations at Scale, For Everybody — Nicholas Kang & Michael Aaron, Google DeepMind

May 25, 2026 · 20:03

Nicholas Kang and Michael Aaron from Google DeepMind's Kaggle team argue that AI evaluations are broken due to being scattered, stale, and lacking transparency, citing a competing lab publishing inflated results by using custom compaction settings. They introduce four solutions: hackathons to channel community expertise, a standardized agent exam that returned 500+ submissions in its first week without promotion, a Game Arena where models play poker, chess, and werewolf for an ELO rating that cannot saturate, and an open benchmarks platform. A wastewater treatment plant engineer in Turkey built a novel safety benchmark from 20 years of field experience. They note that on SWE-Bench Pro, six frontier models land within a couple of percentage points, but the harness shifts performance by 22%, complicating comparisons. Challenges include high cost (400,000 poker hands for statistical significance), maintaining community engagement, and dealing with fast model deprecation cycles.

Introducing WebMCP: Agents in the Browser — RL Nabors

May 23, 2026 · 23:08

Rachel Nabors, former web standards contributor and now Principal Developer Experience Engineer at Arise, argues that chat-only agent interfaces (the 'starfish' design) are the CLI of the future and demonstrates how to replace them with rich interactive surfaces using MCP Apps and WebMCP. She explains STDIO vs HTTP transports, shows how MCP Apps bundle HTML, CSS, and JavaScript into single files rendered in agent iframes, and introduces WebMCP, which adds tool attributes to existing page elements so browser agents can call functions without screenshot parsing or DOM traversal. Using her own web comic site as a case study, she builds a comic reader with full panel navigation, transcript mode, and speech synthesis via the Web Speech API, and advocates for better client support of MCP resources to avoid inefficient tool calls for context loading.

Fast Models Need Slow Developers — Sarah Chieng, Cerebras

May 22, 2026 · 18:02

Sarah Chieng from Cerebras argues that the 20x speed increase of models like Codex Spark (1,200 tokens/sec vs. 40-60 for Sonnet/Opus) forces developers to rethink workflows, or risk generating technical debt at unprecedented scale. She presents a practical playbook: validation and linting become free at every step, so must run continuously; developers can generate 75 component variations across five sub-agents and cherry-pick the best; and with context filling in 30 seconds instead of ten minutes, a four-file external memory system (agents, plan, progress, verify) maintains continuity between sessions. Chieng emphasizes real-time collaboration with the model rather than spawning agents and walking away, and advocates using a slower planner model with a fast executor to orchestrate agents effectively.

Cooking with Agents in VS Code — Liam Hampton, Microsoft

May 21, 2026 · 17:04

Liam Hampton from Microsoft demonstrates how to run three AI agents simultaneously in VS Code using GitHub Copilot: a local agent with Claude Opus for iterative unit test writing, a background agent using git work trees to build a front end from a GitHub issue with minimal oversight, and a cloud agent in GitHub Actions to add documentation and open-source files. He argues that VS Code serves as a single entry point for all three agent types—local for hands-on iteration, background for large tasks where partial involvement is acceptable, and cloud for tasks the developer doesn't need to touch. The talk walks through a live demo where the three agents work in parallel on one codebase, solving different problems (testing, UI, documentation) without interference, and explains the underlying infrastructure: cloud agents run securely in GitHub Actions with MCP servers and built-in safeguards, while local and background agents leverage VS Code's chat customizations and third-party extensions. Hampton also highlights the new modal for managing agents, custom instructions, skills, and MCP servers, positioning Copilot as a unified control plane for diverse AI workflows.

Your Coding Agent Should Do AI System Engineering — Ben Burtenshaw, Hugging Face

May 21, 2026 · 18:25

Ben Burtenshaw from Hugging Face demonstrates how coding agents can take on AI systems engineering tasks—writing CUDA kernels, fine-tuning models, and running multi-agent research labs—by leveraging skills and the Hugging Face Hub. He shows a 1.88x speedup on H100s with an RMSNorm kernel written by Claude Code, and a fine-tuned Qwen3 0.6B achieving 35% on LiveCodeBench. Skills compress years of specialization into hours by turning zero-shot tasks into few-shot workflows. For multi-agent research, a Planner generates hypotheses from papers, Workers implement them as training scripts, and a Reporter monitors results via the open-source Trackio dashboard, with all jobs running on Hub compute. The key is exposing open primitives like kernels, Trackio, and HF jobs as agent-controllable tools.

Anthropic Workshop: Build Agents That Run for Hours — Ash Prabaker & Andrew Wilson

May 18, 2026 · 1:15:40

Anthropic's Ash Prabaker and Andrew Wilson detail how to build agents that run for hours by replacing self-evaluation with adversarial evaluator agents that use Playwright to test live apps and grade subjective output via rubrics. They explain that context compaction doesn't cure coherence drift, so structured handoffs between fresh context windows are essential. The generator and evaluator negotiate testable sprint contracts before building, and the planner provides high-level specs without overspecifying technical details. They show that a solo Claude Code session built a retro game maker that looked complete but failed in play mode, while the adversarial harness produced a fully functional app with live physics and AI features. Key takeaways include reading traces as the primary debug loop, deleting harness components as models improve, and using file-system state for long-running agents.

Why Your AI UX Is Broken (and It's Not the Model's Fault) — Mike Christensen, Ably

May 17, 2026 · 18:38

Mike Christensen, a staff engineer at Ably, argues that direct HTTP streaming (SSE) for AI chat apps breaks because it ties a response stream to a single connection, making resumability, multi-device sync, and live control mutually exclusive. He introduces durable sessions—a persistent, shared resource decoupled from any individual client or agent—built on Ably's Pub/Sub channels, which automatically handle reconnection, cross-tab synchronization, and concurrent multi-agent activity without complex plumbing. Christensen demonstrates with a live demo: a forced network disconnect that self-recovers, two tabs in perfect sync, two agents running in parallel without an orchestrator, and a handoff to a human agent who joins mid-conversation with full history. The episode concludes that treating the session as a durable shared resource unlocks resilient, multi-surface, and live-controllable AI experiences that the standard request-response model cannot support.

Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft

May 16, 2026 · 19:45

Marlene Mhangami of Microsoft and GitHub argues that AI boosts developer productivity only when paired with clean code practices, citing a Stanford study of 120,000 developers and GitHub Octoverse data showing 275 million weekly commits in 2026 with a growing share co-authored by AI. She advocates test-driven development (TDD) with Playwright for functional testing over unit tests, as AI-generated tests often affirm code behavior rather than user experience. In a live demo, she uses GitHub Copilot CLI and Playwright MCP server to automatically write failing Playwright tests for a toy store's search and filter features, then generate code to pass them. She emphasizes testing one feature per test, capturing screenshots for PRs, and committing code before fixes to preserve context. She closes with advice on using Playwright agents for complex state management and confirming cross-screen-size support.

Connecting the Dots with Context Graphs — Stephen Chin, Neo4j

May 16, 2026 · 17:39

Stephen Chin of Neo4j argues that retrieval alone is insufficient for AI systems, because context graphs—which store relationships, reasoning traces, and decision provenance—enable grounded, auditable answers. He demonstrates with a healthcare example where a generic RAG system returns generic advice while a graph-grounded system knows the patient smokes and has had surgery, tailoring recommendations. The episode walks through Lenny’s Podcast memory demo and a financial services loan-decision app that surfaces prior rejections, margin trades, and fraud risk patterns, making the graph traversal visible. Chin notes Gartner has placed context graphs on the AI hype cycle and Foundation Capital called them a $3 trillion startup opportunity. Neo4j’s open-source agent memory package (short‑term, long‑term, reasoning memory) powers these context graphs, aiming to help engineers escape fragmented enterprise data and build explainable, policy‑aware AI.

Agents Don't Do Standups: Building the Post-Engineer Engineering Org — Mike Spitz, PFF

May 15, 2026 · 17:50

Mike Spitz, CTO of PFF, details a three-month case study where two engineers using AI agents outperformed a team of ten, achieving 25x more deployments and 10x higher output by ticket complexity. The two engineers deployed five times daily while the ten deployed once every five days; customer satisfaction rose to 8.6 out of 10 from a prior 7–7.5. Scrum practices were eliminated—standups, sprint planning, and retrospectives—replaced by every-other-day huddles. The new workflow moves from a spec to a lightweight design document (LDD) to auto-generated tickets and PRs, with agents handling style and naming in code reviews. A QA agent spins up on staging after each merge to check acceptance criteria against tickets. Spitz advises starting slowly with engineers who have deep system knowledge, encoding engineering culture into composable skills, and treating the development lifecycle like a factory.

Combine Skills and MCP to Close the Context Gap — Pedro Rodrigues, Supabase

May 15, 2026 · 18:27

Pedro Rodrigues from Supabase argues that combining agent skills with the Model Context Protocol (MCP) outperforms either alone, closing the context gap that makes agents unreliable on production systems. In a test with Claude Sonnet 4.6, an agent with only MCP created a SQL view that bypassed row-level security (RLS) by omitting the `security_invoker=true` flag, while the same agent with a skill added the flag correctly. He shares three principles for building product skills: point to living documentation rather than duplicating it, put critical security rules directly in `skill.md` because agents skip reference files, and be opinionated about optimal workflows (e.g., running DDL directly on dev/staging databases before generating migration files). Their evals across Claude Opus 4.6, Sonnet 4.6, GPT 5.4, and GPT 5.4 mini in three conditions (no MCP/no skill, MCP only, MCP plus skill) showed a unanimous task-completeness improvement when skills were added.

Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize

May 14, 2026 · 2:04:18

Laurie Voss, head of developer experience at Arize AI, delivers a hands-on workshop on evaluating agentic applications using Arize Phoenix, demonstrating that choosing the right eval matters more than tuning it: a correctness eval scored 0 out of 13 on the same financial analysis agent that a faithfulness eval scored 13 out of 13, because the model doesn't know the current year and cannot verify forward-looking data. He walks through building a complete eval pipeline from scratch—starting with tracing a Claude Haiku-based financial agent, reading and categorizing traces to identify root causes, then implementing code evals, built-in LLM-as-a-judge evals, and a custom actionability rubric with labeled examples. Voss emphasizes the importance of meta-evaluation to validate judge accuracy and introduces Phoenix experiments to prove prompt changes actually improve scores, not just vibes. Practical tips include using the impact hierarchy (data quality > prompting > model selection > hyperparameters) and the value of regression evals for safe model upgrades. The workshop closes with cost-aware evaluation, pairwise evaluation, and reliability scoring as next steps beyond the foundations…

Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft

May 14, 2026 · 1:20:07

Amy Boyd and Nitya Narasimhan of Microsoft explain how to close the gap between agent behavior and requirements using Microsoft Foundry's observability stack. They demonstrate tracing via OpenTelemetry, built-in evaluators for quality, safety, and agentic metrics (e.g., intent resolution, task adherence), and red teaming where a second AI attacks the agent to reveal vulnerabilities. The showcase is the observe skill: pointed at an agent with no eval data, it generates a dataset, runs batch evaluations, optimizes the prompt, compares versions, and rolls back to the best one—all from a single prompt. The skill surfaces failures developers didn't know existed, accelerating the optimize loop with human-in-the-loop guidance.

Make your own event-sourced agent harness using stream processors — Jonas Templestein, Iterate

May 14, 2026 · 1:04:27

Jonas Templestein and Misha from Iterate introduce an event-sourced agent harness built on stream processors, arguing that agents should be modeled as an append-only event log with a synchronous reducer for state and an after-append hook for side effects. They demonstrate how every action—streaming chunks, tool calls, errors—becomes an event, enabling full debuggability and replay without re-running LLM calls. The key innovation is a "dynamic worker configured" event whose payload is a JavaScript processor; appending it to any stream instantly turns that stream into an AI agent with no server or dependencies. This allows processors from different authors and languages to compose on the same stream, and a safety checker can inject context within 200ms without blocking the agent. The hosts emphasize eventual consistency over before-hooks, and show deployment via Cloudflare Workers or by simply subscribing from any HTTP client.

Building a Chess Coach — Anant Dole and Asbjorn Steinskog, Take Take Take

May 13, 2026 · 18:22

Anant Dole and Asbjørn Steinskog of Take Take Take, Magnus Carlsen's chess app, built an AI chess coach that keeps LLMs as translators rather than reasoners. Stockfish evaluates positions, tactical and positional detectors extract forks, pins, and structural weaknesses, and the LLM only converts those structured signals into English—preventing hallucination. They target sub-3-second latency using Gemini Flash. When a user flags bad commentary, it posts to Slack and injects into a running Claude Code channel via MCP. Claude investigates, modifies prompts or detectors, regenerates commentary, and asks clarifying questions. They run automated evals across 16 scenarios: Gemini Flash at 75%, Claude thinking below 60%, GPT-5 Mini lower. Their key insight: separate data pipeline from language generation, and close the loop with autonomous agents.

Give Your Agent a Computer — Nico Albanese, Vercel

May 12, 2026 · 1:08:53

Nico Albanese demonstrates building an agent with Vercel's AI SDK v6, centered on the insight that giving an agent a file system transforms its behavior—it follows through on long tasks, stays on track, and builds on prior work. He walks through creating a tool loop agent from scratch, adding provider-executed web search with typed UI components, and integrating Vercel's persistent named sandboxes that snapshot state after inactivity. The agent gains a bash tool for file system access, a memories.md file for persistent memory injected into instructions each turn, and instructions to generate Python scripts for repeatable tasks so it accumulates tools across sessions. Albanese explains how these patterns scale: an internal agent called D0 reduced customer support tickets by 90% with a 95% satisfaction rate, and his personal coding agent ran for 104 minutes in one turn, using 316 tool calls and only 32% of GPT-5.4's context window with zero compaction. The session concludes with a preview of a larger sub-agent system that uses durable workflow steps and background sub-agents to keep the main thread under 7,000 tokens.

Malleable Evals: Why Are We Evaluating Adaptive Systems with Static Tests? — Vincent Koc, OpenClaw

May 12, 2026 · 15:05

Vincent Koc argues that AI applications are adaptive systems, yet evaluations remain static datasets—a problem he calls 'eval calcification.' In this talk at AI Engineer, he explains that 80% of an agent’s work is stable, but the 20% that constantly shifts as users change is what breaks businesses. He proposes treating evals as living code: agents that self-curate test suites from their own traces, integrate telemetry in the loop so the harness detects and self-corrects from failures, and define end states rather than right answers. Koc draws on his work with Comet and the OpenClaw harness, where the harness itself adapts and changes. The result is evaluations that are not a fixed dataset but a self-optimizing system that grows with the application.

A Piece of Pi: Embedding The OpenClaw Coding Agent In Your Product — Matthias Luebken, Tavon

May 11, 2026 · 20:42

Matthias Luebken explains how to embed Pi, the minimal coding agent SDK from OpenClaw, into real products, arguing that the key architectural principle is to make systems easy for agents. He demonstrates a B2B sales pipeline application where incoming RFP emails are routed to customer-specific agent sessions, CLIs expose CRM and ERP data cleanly, and the only human output is a draft in the user's inbox. The agent is purely an LLM calling tools in a loop, with Pi's extensions enabling UI interactions and session management. Luebken emphasizes that coding agents will become core building blocks for software, and Pi's minimal design is ideal for tinkering and learning.

Why MLX — Prince Canuma, Neywa Labs

May 11, 2026 · 23:10

Prince Canuma argues that on-device AI, powered by Apple's MLX framework, is a viable alternative to cloud-dependent services, especially for users in regions with unreliable internet. MLX, an array framework for Apple Silicon, has reached 1.5 million downloads and 4000 ported models, including day-zero support for Gemma 4 and Qwen 3 Omni. Canuma demonstrates real-time vision models (e.g., RF Deter for object detection), sub-100ms text-to-speech via Marvis TTS, and modular speech-to-speech pipelines that run entirely on-device. Community projects showcased include a native voice app (Locally), a robot with real-time voice cloning, and a video generation system that chains coherent stories on 16GB VRAM. A recent breakthrough, Turbo Quant, reduces KV cache by 4x, enabling 1 million context windows on-device. Canuma emphasizes that these capabilities provide accessibility for his blind father and enable agents that hear, see, and respond without phoning home.

Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic

May 7, 2026 · 1:20:40

Samuel Colvin shows how to improve production agents without redeploying using Pydantic AI's GEPA, evals, and Logfire managed variables. GEPA uses a genetic algorithm to evolve prompts, achieving 96.7% accuracy — up from 87% with a simple prompt and 92% with an expert prompt — on a Wikipedia-based political relations task. Managed variables let you update prompts, models, and parameters live via a web interface and A/B test targeting. Evals compare against a golden dataset of 650 MPs; Colvin runs 65 cases in 30 seconds using GPT-4.1. He discusses that prompt optimization is most valuable with private data, that implicit user feedback (e.g., user's next action) can build golden datasets, and that overfitting to small test sets is a risk — the GEPA optimizer may exclude valid relations like 'uncle' if they don't appear in the training split.

Vibe Engineering Effect Apps — Michael Arnaldi, Effectful

May 7, 2026 · 1:43:04

Michael Arnaldi demonstrates that cloning the Effect library's repository into a project, rather than relying on prompts, is the most effective way to give coding agents the context they need to build reliably with Effect. Starting from an empty repo, he sets up a Bun, Vitest, and TypeScript project, adds the Effect repo as a git subtree, and creates an agents.md to guide the agent. He then uses the agent to research patterns, resulting in a fully functional Todo HTTP API with OpenAPI docs, SQLite persistence, and tests — all from scratch in under two hours. The workshop stresses that treating the library code as part of the project, combined with strict diagnostics and pattern files, makes agents effective even in unfamiliar codebases. Arnaldi also discusses the importance of workflow systems like Effect Cluster for long-running AI processes, where server failures become likely.

Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop

May 7, 2026 · 50:25

Zubin Koticha and Danny Gollapalli of Raindrop argue that agent observability must shift from evals to production monitoring because agent failures are non-deterministic and unbounded. They break down explicit signals (tool error rate, latency, cost) and implicit signals (user frustration, refusals, task failure) detected by trained classifiers and regex, emphasizing that aggregate patterns even from imperfect regex are valuable. Experiments let teams ship changes to a percentage of users and compare semantic signal rates, with statistical relevance often reached after a few hundred events. Self-diagnostics—a simple tool and system prompt—enable agents to report their own failures, capability gaps, and even self-correction behavior, as demonstrated in a live coding agent demo where a disabled write tool caused the agent to use bash and then report the bypass. The episode covers alerting, trace visualization, data export to BigQuery/Snowflake, and the challenge of managing fast-paced experimentation at scale.

Full Walkthrough: Writing & Using Skills — Nick Nisi and Zack Proser

May 6, 2026 · 1:21:03

Zack Proser and Nick Nisi, Developer Experience Engineers at WorkOS, teach how to write portable, composable skills — single markdown files with optional scripts — that train AI tools like Claude, Codex, and Cursor to perform specific tasks consistently. They demonstrate building a repo roast skill that uses deterministic Git commands via script interpolation and progressive disclosure to load targeted references only when needed. The workshop covers structuring skills with front matter (name, description for LLM routing), adding confidence scoring and constraints to improve output, and sharing skills as .skill files or through marketplaces. Advanced topics include using skills in non-coding workflows (e.g., Slack–Linear automation) and composing skills in the WorkOS CLI, which uses the Claude Agent SDK to install auth. Attendees learn to iterate skills by reflecting on past conversations and leveraging Claude's built-in skill builder for evaluation.

MCP UI: Extending the frontier — Liad Yosef and Ido Salomon, MCP Apps

May 6, 2026 · 22:21

In this episode, Ido Salomon and Liad Yosef explain MCP Apps, an MCP extension that lets tools send interactive, branded UI instead of plain text responses inside chat hosts like ChatGPT, Claude, VS Code, Cursor, and Copilot. They argue that text-based chat reduces companies to a wall of text, erasing identity, while MCP Apps preserves branding by returning HTML resources that hosts render as secure, interactive components. The architecture ensures every user click sends a message back to the host (not directly to the backend), keeping interactions in context—as demonstrated with a PostHog funnel analysis in Claude. Adoption includes Shopify, Hugging Face, GitHub, ChatGPT, and Claude, with ChatGPT recommending MCP Apps for building ChatGPT Apps. They note 800 million weekly ChatGPT users (10% of world population), calling this a once-in-20-years opportunity to rethink app distribution. Future work covers reusable views, model-UI interaction, and interoperability with generative UI protocols like A2UI and WebMCP, aiming to standardize UI in chat as the new web.

Demand-Driven Context: A Methodology for Coherent Knowledge Bases Through Agent Failure

May 5, 2026 · 1:08:15

Raj Navakoti, a staff software engineer at IKEA, presents a demand-driven context methodology for building coherent knowledge bases by letting AI agents fail on real problems and surfacing missing institutional knowledge. He argues that enterprises should shift from pushing monolithic documentation to a pull approach where agents reveal undocumented tribal knowledge through repeated failures on incidents and Jira tickets. Using a framework with skills, rules, and hooks, he demonstrates how agents can gradually improve confidence scores (from 1.4 to 4.4 over 14 incidents) by documenting discovered context blocks. Navakoti introduces a context gap scanner that automatically analyzes work items against existing documentation to identify critical gaps, outdated information, and duplications. He advocates storing curated knowledge in GitHub for version control and PR-based collaboration, and emphasizes that this approach helps teams know the unknown, enabling agents to manage knowledge rather than just consume it.

Skill Issue: How We Used AI to Make Agents Actually Good at Supabase — Pedro Rodrigues, Supabase

May 4, 2026 · 1:18:41

Pedro Rodrigues, AI tooling engineer at Supabase, demonstrates how to write, test, and iterate on Agent Skills to make agents actually good at real systems like Supabase's performance-review app. He explains progressive disclosure—skills load only a short description first, letting the agent decide when to pull in full instructions—and shows that combining skills with MCP tools gives agents both context and actions. In a live demo, Rodrigues reveals a common failure: creating a database view without a security invoker flag bypasses row-level security, letting all users see everyone's salary data. Adding a skill with security checklists guides the agent to include the flag, fixing the bug. He then introduces an eval-driven development cycle using an eval.json file and an LLM-as-judge to automate testing across conditions with and without the skill, cautioning that writing accurate evals is tricky because non-deterministic outputs can mislead. The session offers a practical framework for validating skills, avoiding pitfalls, and measuring what actually improves agent behavior.

Mergeable by default: Building the context engine to save time and tokens — Peter Werry, Unblocked

May 3, 2026 · 1:41:25

Peter Werry of Unblocked argues that context engines—systems that supply AI agents with only the relevant organizational context—are critical to avoid agent doom loops and wasted tokens. He debunks three myths: naive RAG, connecting MCP servers, and bigger context windows do not solve the context problem. Werry describes building a social engineering graph to identify experts and distill team best practices, and shares hard lessons including hiding conflicts and caching answers. In a benchmark task, Unblocked's context engine reduced a 2.5-hour, 21-million-token task to 25 minutes and 10 million tokens. The talk offers a practitioner's guide to building context engines with conflict resolution, personalization, and access control.

Context Is the New Code — Patrick Debois, Tessl

May 3, 2026 · 27:14

Patrick Debois argues that as AI coding agents become more capable, context—prompts, rules, and memory—needs its own engineering discipline, introducing the Context Development Lifecycle: Generate, Evaluate, Distribute, and Observe. He explains how to create reusable prompts like agent MD and pull documentation via MCP, test context using evals with LLM-as-judge and sandboxed execution, package context as skills with registries and dependency management, and observe through agent logs, PR feedback, and production failures to feed improvements back into context. He also notes that context requires its own CI/CD with error budgets due to non-determinism, and highlights the need for context filters and security scanning. The talk draws parallels to DevOps and positions Tessl as a platform implementing these practices.

Human-in-the-Loop Automation with n8n — Liam McGarrigle

May 2, 2026 · 1:19:22

Liam McGarrigle of n8n shows how to build a secure human-in-the-loop automation agent using n8n's visual workflow system, with a Gmail and Google Calendar management agent as the concrete example. He walks through wiring a chat trigger, an AI agent with simple memory, and tools that the agent can call, such as sending emails and creating calendar events. The key addition is a human review node placed between the agent and destructive tools, which intercepts actions and presents them for approval via chat—preventing accidental sends or event creation. McGarrigle emphasizes naming nodes and writing tool descriptions to guide the LLM correctly, and demonstrates using expressions to format readable approval messages. He also covers extending the agent to Slack, adding scheduled runs for autonomous hourly inbox checks, and using sub-agents for specialized tasks. Additional topics include n8n's native MCP server for integration with Claude Code, enterprise Git-based environments for workflow management, and building custom REST APIs within n8n. The session focuses on giving developers observability and control over AI workflows, ensuring agents are not black boxes.

Agents on the Canvas in tldraw — Steve Ruiz, tldraw

May 1, 2026 · 19:54

Steve Ruiz of tldraw details the evolution of AI on the infinite canvas, from early one-shot demos like MakeReal (draw to functional prototype) to multi-agent 'fairies' that collaborate, delegate, and even rewrite the canvas in real time. He explains how the tldraw SDK enables agents to act as virtual collaborators, using structured outputs and tool use to draw, animate, and modify designs. Ruiz demonstrates fairy agents that can work independently or in leader-follower mode, coordinating on tasks like creating wireframes or filling out forms. He also unveils a desktop prototype that lets Claude directly inject JavaScript into the canvas, enabling agents to edit code, modify UI, and even hack other apps like Spotify. The talk emphasizes the shift from sidebar agents to canvas-native collaborators, the challenges of vision model training for 2D spaces, and the safety trade-offs of giving LLMs runtime access in local-first apps.

LLM codegen fails and how to stop 'em — Danilo Campos, PostHog

Apr 30, 2026 · 19:18

Danilo Campos, who builds the PostHog wizard, explains how to make LLM code generation reliable by sharing practical strategies from a system that helps 15,000 users per month. He identifies 'Model ROT'—models becoming stale—and counters it by shoving fresh Markdown documentation into context. To avoid weird architecture, he maintains 'model airplanes': thin, auth-shaped simulacra that provide correct integration patterns. He limits improvisation by breadcrumbing the agent step by step, starting with detecting business-value files before even mentioning PostHog. Campos stresses that human errors (contradictory instructions, missing tools) are the biggest threat, solved by asking the agent after each run what could be improved. He also details locking down tool usage to prevent shenanigans like reading .env files, replacing that with a limited key-check and write tool. The core shift: code is a depreciating asset, so 90% of the wizard's value is now in Markdown files and prose, which improve with better models.

Building your own software factory — Eric Zakariasson, Cursor

Apr 28, 2026 · 1:23:37

Eric Zakariasson, an engineer at Cursor, explains how to build a 'software factory' by scaling from one agent to many, shifting from worker to manager. He outlines six stages of autonomy, from spicy autocomplete to the dark factory, and emphasizes that most teams are stuck at levels two or three. Key primitives include modular codebases, guardrails like rules and hooks that emerge dynamically, enablers such as skills and MCPs, and verifiable systems like automated tests. Cloud agents with isolated VMs allow agents to run asynchronously, test their own work via computer control, and scale to thousands. Eric shares internal automations like daily reviews, PR comment analysis, and continual learning that extracts rules from chat transcripts. He concludes with strategic advice: front-load context, don't outsource critical decisions, and build tools and systems to capture flywheels, noting that human accountability remains essential.

Why building eval platforms is hard — Phil Hetzel, Braintrust

Apr 28, 2026 · 25:39

Phil Hetzel of Braintrust argues that building eval platforms for AI agents is a data systems problem, not just a UI one, because LLMs have extreme variability and agent traces are semi-structured, high-volume, and large. He describes four maturity stages: simple spreadsheet with a for loop, a custom vibe-coded UI with a database, an experimentation playground for non-technical users, and finally a flywheel connecting offline evals with production observability. The core challenge is the data layer—traces can be 10-20 MB per span, requiring low-latency ingestion, aggregate analysis, and full-text search, which traditional databases cannot handle. Future platforms must surface unknown unknowns via topic modeling, support agent-to-agent interactions, and integrate automatic tracing through AI proxies. Braintrust addresses these with a custom data platform that separates hot and cold storage and enables SQL queries directly on traces.

One Login to Rule Them All: Cross-App Access for MCP — Garrett Galow, WorkOS

Apr 28, 2026 · 23:24

Garrett Galow from WorkOS introduces Cross-App Access (XAA) for MCP, solving the problem of repeated OAuth consent screens when connecting agents to multiple services. The flow leverages a three-way trust between the MCP client, server, and an Identity Provider like Okta: a single SSO login issues an IDJag token that is exchanged for short-lived access tokens across all MCP servers without manual intervention. A demo shows Figma automatically connecting after an Okta login. The approach improves security—if the IdP session is revoked, tokens cannot refresh—and requires minimal IT setup: granting permission for the client to request server access. Currently only Okta supports XAA; Azure/Entra does not yet. The session also notes that authorization scopes are not handled by default but are a future consideration.

Full Walkthrough: Workflow for AI Coding — Matt Pocock

Apr 24, 2026 · 1:36:30

Matt Pocock presents a hands-on workshop on building a full AI-assisted coding workflow, arguing that software engineering fundamentals—not hype—make agents effective. He introduces the 'smart zone' and 'dumb zone' of LLMs (performance drops after ~100k tokens) and the 'Memento problem' (agents forget between sessions). His process starts with a 'Grill Me' skill that relentlessly questions the user until shared understanding is reached, then produces a PRD without reading it, slices work into vertical 'tracer bullet' issues, and runs agents AFK using TDD. He advocates designing codebases with deep, testable modules and shows Sandcastle, a TypeScript library for parallel agent execution with separate implementer (Sonnet) and reviewer (Opus). The workshop transforms ambiguous briefs into shippable features while keeping humans in the loop for QA and taste.

What Do Models Still Suck At? - Peter Gostev, Arena.ai, BullshitBench

Apr 24, 2026 · 20:24

Peter Gostev presents data from his BullshitBench and Arena.ai to argue that language models still struggle with nonsense questions and expert-level tasks, despite benchmark charts showing relentless improvement. His BullshitBench reveals that only Claude Sonnet 4.5 and some Qwen models consistently push back on nonsense, while GPT and Gemini models accept it 50% of the time. Arena.ai's dissatisfaction rate among top 25 models has improved from 17% pre-reasoning to about 9% currently, but remains non-zero, and expert categories like gaming, magic, finance, and law show minimal improvement. For software expert prompts, dissatisfaction dropped from 23.5% in Q2 2024 to 13% in Q1 2026, but gaming is a persistent weakness where models fail to create engaging game mechanics. Gostev warns that narrow benchmarks overstate progress and urges focusing on the full distribution of real-world tasks.

The End of Apps — Kitze, Sizzy.co

Apr 23, 2026 · 20:18

Kitze, creator of Benji and founder of Tinker Club, argues that current productivity apps and AI agents are unreliable and too complex for most users, predicting a future where AI prompts humans instead of the reverse. He traces his 24-year obsession with productivity tools—from a childhood to-do list to building Benji with 60 features—and his move to self-hosting after the ChatGPT moment. Kitze critiques custom agents like OpenClaw as too tinkerer-focused, cloud agents like ChatGPT as too nerfed, and believes local models on devices (Apple, Google Pixel) will win for normies, enabling AI to automatically manage notifications, emails, and tasks. He shares his own experiment, Wolfer, designed for predictable multi-agent orchestration without memory systems, using nested topics for context. The episode delivers a strong claim: the role of AI will invert, with machines prompting humans for decisions while handling all background work.

The New Application Layer - Malte Ubl, CTO Vercel

Apr 20, 2026 · 18:52

Vercel CTO Malte Ubl argues that AI engineering is the legitimate successor to web development and that the real value lies in the application layer, not the model labs. He identifies four effective agent archetypes—24/7 support, compressed research, surfacing existing information, and eliminating boring work—and reveals that over 60% of vercel.com page views now come from AI agents. Ubl predicts model companies will commoditize, driving costs down and empowering engineers, while citing Europe's leadership in AI engineering through Vercel's AI SDK, Pi (a coding agent from Austria), and OpenClaw. He stresses the need for open-mindedness toward paradigm shifts and new infrastructure, such as sandboxed agent runtimes, and warns of impending security challenges.

The Future of MCP — David Soria Parra, Anthropic

Apr 19, 2026 · 18:46

David Soria Parra from Anthropic argues that MCP (Model Context Protocol) is the key to connecting agents to tools and data in production, with 110 million monthly downloads—outpacing React's growth at the same stage. He lays out a 2026 connectivity stack combining Skills, MCP, and CLI/Computer Use, each suited for different needs, and emphasizes that best agents will use all three seamlessly. To improve client harnesses, he introduces Progressive Discovery—deferring tool loading via Tool Search to reduce context usage—and Programmatic Tool Calling, where models write scripts to compose tool outputs efficiently. Upcoming MCP protocol improvements include stateless transport (with Google) for easier scaling, async agent-to-agent tasks, enterprise features like Cross App Access and Server Discovery via well-known URLs, and a Skills-over-MCP extension for shipping usage instructions with servers. He calls for community feedback on these directions.

Paperclip: Open Source Human Control Plane for AI Labor — Dotta Bippa

Apr 15, 2026 · 24:34

Dotta Bippa introduces Paperclip, an open-source human control plane for AI labor that lets you manage an org chart of agents to run a zero-human company. The tool enables you to hire employees, set goals, automate jobs, and bring your own agent (e.g., Gemini, Claude, Codex) with skills and instructions. Paperclip supports reliable workflows like QA reviews and approvers, and routines for recurring tasks such as summarizing PRs or writing changelogs. Dotta demonstrates creating a Remotion video celebrating Paperclip's 40,000 GitHub stars (now 50,000) by instructing the CEO agent to hire a video writer and apply brand guidelines—something that would have taken a week becomes an afterthought. Paperclip is not just for coding; it handles marketing, sales, and finance. Upcoming features include CEO chat, maximizer mode, multi-user support, cloud deployments, and a desktop app, all aimed at giving humans control over AI labor.

Cognitive Exhaust Fumes, or: Read-Only AI Is Underrated — Šimon Podhajský, Head of AI, Waypoint

Apr 8, 2026 · 11:31

Šimon Podhajský argues that read-only AI systems that analyze personal digital exhaust without the ability to write back are more valuable than agentic AI that acts on users' behalf. He built a system ingesting six read-only sources (email, journal, tasks, CRM, browser sessions, notes) that surfaces insights like intention-action gaps, attention drift, and relationship decay via cross-source pattern detection—things no single source reveals. For example, a weekly reflection skill in Claude synthesizes a brutal review of his week, and a cross-source query maps his recent reading to contacts in his CRM using Vivaldi SQLite and Clay MCP. He emphasizes risk asymmetry: read-only errors cost nothing, while write errors can be unbounded. He also acknowledges security risks like the mosaic effect and Simon Willison's lethal trifecta, noting that shell access still allows exfiltration, but argues that examined risk is better than ignorance.

Platforms for Humans and Machines: Engineering for the Age of Agents — Juan Herreros Elorza

Apr 8, 2026 · 21:15

Juan Herreros Elorza, team lead at Banking Circle, argues that the same platform engineering best practices—self-service, API-first design, local-first workflows, and rich observability—that serve human developers are now prerequisites for AI coding agents to autonomously build, debug, and ship software. Drawing from his experience building the internal platform Atlas, which handles over €1 trillion in cross-border payments annually, he explains how exposing platform capabilities via well-defined APIs (and optionally MCP servers) lets agents iterate locally, fail fast, and close the loop using API-accessible logs and metrics. He stresses structured documentation, including agent.md files and skills, to guide agents on how to contribute to the platform, and recommends combining guardrails for security with context files for best practices. Finally, he advises measuring outcomes like DORA metrics, support requests, and developer satisfaction, and suggests using AI as leverage to finally implement these long-known best practices.

Identity for AI Agents - Patrick Riley & Carlos Galan, Auth0

Jan 14, 2026 · 1:22:12

Patrick Riley and Carlos Galan from Auth0 (Okta) present new identity and access management features for AI agents, including Async Auth, Token Vault, and MCP integration. Token Vault securely stores and refreshes upstream resource tokens, enabling agents to access APIs on behalf of users. Async Auth allows agents to request user approval for risky operations, such as placing an order, via out-of-band notifications like push to the Guardian app. They demonstrate a Next.js chatbot that uses Token Vault to read a user's stock portfolio and Async Auth to require explicit consent before executing trades. The MCP server is secured with OAuth 2.0, supporting Dynamic Client Registration (DCR) and fine-grained scopes. The solution bridges user identity, agent identity, and upstream API authorization, ensuring agents can act autonomously but with user-defined boundaries.

OpenAI + @Temporalio : Building Durable, Production Ready Agents - Cornelia Davis, Temporal

Jan 12, 2026 · 1:18:30

Cornelia Davis, developer advocate at Temporal, demonstrates how the OpenAI Agents SDK and Temporal combine to build durable, production-ready AI agents. The integration, co-developed by OpenAI and Temporal, wraps agentic loops in Temporal workflows, providing automatic retries, event sourcing, and state management that survive process crashes and network failures. A live demo shows a weather alert agent that calls an LLM, invokes tools like get_weather_alerts and get_ip_address, and recovers seamlessly after the worker is killed. Davis highlights that Temporal’s architecture treats processes as logical entities, allowing agents to run for days and handle human-in-the-loop delays without developer-managed infrastructure. She also explains handoffs and micro-agent orchestration, and points to the AI Cookbook on docs.temporal.io with ready-to-run recipes including the OpenAI Agents SDK integration.

DSPy: The End of Prompt Engineering - Kevin Madura, AlixPartners

Jan 8, 2026 · 1:13:13

Kevin Madura of AlixPartners argues that building robust enterprise AI applications requires shifting from brittle prompt engineering to programming with LLMs using DSPy, a declarative framework that treats prompts as implementation details optimized by the system. He demonstrates how typed interfaces (Signatures) and modular logic (Modules) allow developers to focus on control flow while deferring implementation to the LLM, with Adapters controlling prompt formats (e.g., JSON vs. BAML) to improve performance by 5-10%. The talk's core is Optimizers (like MIPRO and JEPA), which automatically tune prompts by learning from data, shown improving a time entry corrector from 86% to 89% accuracy. Real-world examples include routing files by type (SEC filings vs. contracts), using a 'poor man's RAG' with attachments for multimodal documents, and a boundary detector that segments legal documents from images. Madura emphasizes that DSPy enables transferability across models (e.g., GPT-4.1 to GPT-4.1 Nano) and addresses cost concerns by allowing offline optimization to reduce LLM calls.

Claude Agent SDK [Full Workshop] — Thariq Shihipar, Anthropic

Jan 5, 2026 · 1:52:25

Thariq Shihipar of Anthropic presents the Claude Agent SDK, arguing that Bash and file-system-based agents outperform traditional tool-only approaches for autonomous tasks. He defines agents as systems that build their own context and trajectory, contrasting with structured workflows. The SDK, built on Claude Code, emphasizes the Bash tool as the most powerful primitive for composability and code generation, enabling non-coding tasks like data analysis. He demonstrates live-coding a Pokémon team advisor that dynamically fetches API data via scripts, and explains security through a 'Swiss cheese defense' of model alignment, AST parsing, and sandboxing. Shihipar also covers skills for progressive context disclosure, sub-agents for parallel work, and hooks for deterministic verification, stressing that agent building is an art of reading transcripts and iterating on context engineering.

Jack Morris: Stuffing Context is not Memory, Updating Weights is

Dec 29, 2025 · 1:02:44

Jack Morris argues that large language models fail at niche, long-tail knowledge tasks, such as optimizing AMD GPU kernels or answering private company queries, because they rely on context windows and RAG, which suffer from quadratic self-attention costs and context rot. He advocates for a third paradigm—training knowledge directly into model weights—using synthetic data generation (e.g., synthetic continued pretraining from Stanford) to expand small datasets and parameter-efficient methods like LoRA or memory layers to avoid catastrophic forgetting. Morris demonstrates that full fine-tuning on a 3M 10-K report causes the model to only regurgitate exact sentences, whereas generating diverse synthetic question-answer pairs enables better generalization. He notes that RL-based fine-tuning (e.g., GRPO) can achieve improvements with as few as 14 parameters, while memory layers offer the best trade-off between learning and forgetting. The episode also explores temporal information handling, federated learning resurgence, and the practical decision boundary between RAG and weight-based injection based on data freshness and volume.

Shipping AI That Works: An Evaluation Framework for PMs – Aman Khan, Arize

Dec 26, 2025 · 1:26:16

Aman Khan, AI PM at Arize, presents a framework for product managers to evaluate LLM-powered products beyond gut-feel 'vibe checks.' He demonstrates building an AI trip planner with multi-agent LangGraph, then using Arize's tracing and prompt playground to iterate on prompts. Khan shows how to create datasets from production traces, run A/B experiments on prompts, and use LLM-as-a-judge evals for friendliness and discount offers, comparing against human labels to refine evaluators. He argues evals are the new requirements docs, enabling PMs to own the product experience by writing acceptance criteria as eval datasets. The talk covers building eval teams, handling variance with temperature settings, and continuously improving golden datasets with hard examples, citing real-world analogies from self-driving cars at Cruise.

The Unreasonable Effectiveness of Prompt Learning – Aparna Dhinakaran, Arize

Dec 23, 2025 · 10:56

Aparna Dhinakaran, co-founder of Arize, argues that prompt learning—applying RL techniques to prompts rather than model weights—can continuously improve coding agents by auto-tuning system prompts from runtime feedback. She details a process using Claude Code and CLIMB on the SWE-bench dataset: agents generate patches, unit tests yield results, then an LLM-as-a-judge eval produces natural-language explanations of failures. These explanations feed a meta prompt that iterates on the system prompt rules. On 150 SWE-bench examples, this approach improved Claude Code's issue-resolution rate by 5% and CLIMB's by 15%. She contrasts their method with DSPy's GEPA optimizer, noting theirs required fewer loops due to carefully engineered eval prompts.

The 3 Pillars of Autonomy – Michele Catasta, Replit

Dec 22, 2025 · 24:42

Michele Catasta, VP of AI at Replit, argues that true autonomy for coding agents serving non-technical users rests on three pillars: verification, context management, and parallelism. Replit’s agent uses autonomous testing—writing Playwright code instead of browser-use tools—to catch broken features (over 30% initially) without human feedback, cutting cost and latency by an order of magnitude. Context management relies on sub-agent orchestration rather than massive context windows, boosting memories per compression from ~35 to 45–50. Parallelism, implemented via a core-loop orchestrator, dynamically decomposes tasks to avoid merge conflicts and keep users engaged instead of waiting hours. Catasta emphasizes that autonomy means scoped technical decisions, not just long runtimes, enabling knowledge workers to build software without needing a 'driving license.'

No More Slop – swyx

Dec 22, 2025 · 9:15

In this keynote at the AI Engineer Summit, host swyx declares war on slop — low-quality, inauthentic, inaccurate work produced by both humans and AI — arguing that the AI engineering community must elevate taste and accountability. He introduces Swix's law of anti-slop: the taste needed to fight slop scales with the plummeting cost of generating tokens. Swyx demonstrates how to combat slop using AI itself, citing examples like AI News (which tells readers to skip slow days), prompting techniques to avoid slop, and using sub-agents against context rot. He calls for rejecting autonomy without accountability and urges the audience to say "no more slop" to bosses demanding more lines of code, untested releases, and engagement bait.

The State of AI Code Quality: Hype vs Reality — Itamar Friedman, Qodo

Dec 11, 2025 · 21:15

Itamar Friedman, CEO of Qodo, argues that while AI code generation boosts productivity, it has a glass ceiling unless organizations invest in agentic quality workflows and context. Drawing on reports from Qodo, Sonar, and Pharos, he reveals that 60% of developers say a quarter of their code is AI-generated, yet 67% have serious quality concerns, leading to 42% more time fixing bugs and 35% project delays. He notes that AI code review tools double trust and quality gains, and that better context—including standards and best practices—is the top request from developers. Friedman recommends automated quality gateways, intelligent code review, and testing, warning that without dynamic quality processes, the promised 2x productivity remains elusive.

VoiceVision RAG - Integrating Visual Document Intelligence with Voice Response — Suman Debnath, AWS

Dec 6, 2025 · 1:23:52

Suman Debnath, a Principal ML Advocate at AWS, demonstrates how Colpali—a vision-based retrieval model that treats each document page as an image and generates multi-vector embeddings via patch-based late interaction—can be combined with voice synthesis for a more intuitive RAG system. He explains that Colpali bypasses traditional OCR and preprocessing by directly embedding document images, then scoring query-page relevance through dot-product similarity across patches. The workshop shows how to embed pages, store them in Qdrant with multivector support, and retrieve top pages. Debnath then wraps this retrieval pipeline using the Strands Agent framework, adding a speak tool to output answers in natural voice. A live demo answers a textbook question about trophic levels, first using Bedrock to generate text and then speaking the answer in a female voice, all without a predefined system prompt. The talk positions Colpali as a complementary technique for complex visual documents like IKEA instructions or scanned forms, not a full replacement for traditional RAG.

Katelyn Lesse – Evolving Claude APIs for Agents, Anthropic

Dec 4, 2025 · 13:25

Katelyn Lesse, who leads the Claude Developer Platform team at Anthropic, explains how the platform is evolving to help developers build powerful agentic systems using Claude. She details three key areas: harnessing Claude's capabilities through features like extended thinking and tool use, managing context with MCP, a memory tool, and context editing (which combined to yield a 39% performance bump), and giving Claude a computer via the code execution tool and agent skills. The talk also covers challenges like container orchestration for Claude Code on web and mobile, and the importance of letting Claude work autonomously in a sandbox environment.

The Unbearable Lightness of Agent Optimization — Alberto Romero, Jointly

Nov 24, 2025 · 17:58

Alberto Romero, co-founder and CTO of Jointly, introduces Meta-ACE, a meta-optimization framework that orchestrates multiple adaptation strategies to overcome the limitations of single-dimensional context engineering like ACE. The framework uses a meta-controller to profile task complexity, uncertainty, verifiability, and resource constraints, then allocates strategies across context, compute, verification, memory, and parameter dimensions. Initial results show 8-11% improvement on agent benchmarks, 30-40% reduction in compute costs, and 6-8% gains on domain-specific tasks. Meta-ACE addresses ACE's weak reflector problem with quality gates and multi-signal reflection, feedback brittleness via a hierarchical verification cascade (self-verification, multimodal consensus, execution checks), and task complexity mismatch by dynamically adjusting strategy allocation to save up to 90% compute on simple tasks. Future work includes scaling to multimodal and compound AI systems, with challenges in meta-controller training stability, computational overhead, and verification cascade brittleness.

Backlog.md: Terminal Kanban Board for Managing Tasks with AI Agents — Alex Gavrilescu, Funstage

Nov 24, 2025 · 14:19

Alex Gavrilescu presents Backlog.md, an open-source CLI tool that stores tasks as Markdown files in Git repos, featuring a terminal Kanban board and MCP server for AI agents. He argues that breaking features into atomic Markdown tasks prevents agents from running out of context or implementing unwanted extras. The demo shows Claude creating a task from requirements, generating an implementation plan, and coding a move-mode feature—all via MCP tools. Gavrilescu emphasizes two review checkpoints (after task creation and after the plan) and notes that Backlog.md itself is 99% AI-written. The tool works cross-platform, requires no external APIs, and syncs across branches via Git.

Context Engineering: Connecting the Dots with Graphs — Stephen Chin, Neo4j

Nov 24, 2025 · 26:50

Stephen Chin, VP of Developer Relations at Neo4j, argues context engineering powered by knowledge graphs transforms AI from prompt engineering to information architecture, enabling agents with structured memory and retrieval. He demonstrates GraphRAG using Neo4j's knowledge graph builder with supply chain and VEX documents, showing how vector and graph algorithms retrieve specific Jackson library vulnerabilities with CVE details, severity, and remediation versions. Chin contrasts fast two-pass vector+graph retrieval with agentic multi-step traversal via the Neo4j MCP server and Claude Code, which yields deeper results like CVE number, attack type, and upgrade paths. He explains that graphs excel when relationships span two or more facts, using a presentation update example involving himself, colleague Sid, and the GIDS event. Practical applications include role-based access overlays and explainable AI via visualized conversation flows. Resources like free Graph Academy courses, Nodes AI 2026 conference, and graphrag.com support implementation.

From Stateless Nightmares to Durable Agents — Samuel Colvin, Pydantic

Nov 24, 2025 · 22:13

Samuel Colvin of Pydantic demonstrates building production-grade durable AI agents using PydanticAI, Temporal, and Pydantic Logfire, arguing that stateless architectures fail at scale and that durable execution with checkpointing and recovery is essential. He shows a 20 questions game where two agents play, but with 20% simulated failures, restarts are avoided by wrapping agents in Temporal wrappers for automatic retries. Colvin then implements a Deep Research agent that plans searches, runs parallel web searches via Tavilli, and synthesizes results, all recoverable. In a live demo, killing the workflow and restarting resumes instantly, replaying cached LLM calls in milliseconds. He also previews Pydantic AI Gateway and demonstrates Pydantic Evals comparing Gemini, GPT-4.1, and Claude Sonnet 4.5 on performance and cost.

Developing Taste in Coding Agents: Applied Meta Neuro-Symbolic RL — Ahmad Awais, CommandCode

Nov 24, 2025 · 20:52

Ahmad Awais, founder of Langbase, introduces CommandCode, a coding agent that learns a developer's personal taste and preferences to generate code that feels like their own. Built on Langbase's infrastructure (deploying over 350K agents and 1.2B monthly runs), CommandCode uses a Meta Neuro-Symbolic architecture with reinforcement learning to continuously capture a programmer's invisible architecture of choices—such as preferring TypeScript, PNPM, TSUP, and Commander over other tools—without requiring explicit rule files. Awais demonstrates how CommandCode automatically applies these learned preferences when building a CLI, contrasting it with Claude Code which generates generic output. He argues that taste models represent the next frontier, enabling agents to evolve with the developer and even share tastes across teams or borrow preferences from other developers like design engineers.

Z.ai GLM 4.6: What We Learned From 100 Million Open Source Downloads — Yuxuan Zhang, Z.ai

Nov 22, 2025 · 19:39

Yuxuan Zhang from zAI details the technical roadmap behind the GLM 4.6 open-source model series, which has surpassed 100 million downloads and tied for number one on the LMSYS Chatbot Arena alongside GPT-4o and Claude 3.5 Sonnet. The training pipeline uses 15 trillion tokens of pre-training data, followed by 7 trillion tokens of code and reasoning data, repo-level contexts at 32,000 tokens, and 100 billion tokens of long-context agent trajectories up to 128,000 tokens. Zhang introduces SLIME, a hybrid synchronous/asynchronous RL framework that decouples agent-environment interaction from GPU training to avoid bottlenecks. He explains why single-stage RL at 64,000 tokens outperforms multi-stage approaches for preserving long-context abilities, and shows that token-weighted loss converges faster than sequence-average loss for code RL. The multimodal GLM 4.5V model handles native resolution images and video with temporal index tokens, enabling GUI agent capabilities. Deployment is supported via vLLM and SGLang, with an API at z.ai and an open-source coding assistant.

Five hard earned lessons about Evals — Ankur Goyal, Braintrust

Aug 23, 2025 · 19:46

Ankur Goyal of Braintrust argues that successful AI applications depend on deliberately engineered evaluations (evals) to reflect real user feedback and drive product improvements. He details three signs of effective evals: launching updates within 24 hours (citing Notion), converting complaints into evals, and using evals offensively to assess use cases before shipping. Evals require custom scorers as specs—not off-the-shelf—and context engineering (optimizing tool definitions and outputs) is the new frontier; shifting outputs from JSON to YAML improves token efficiency. New models can upend everything, as shown by a benchmark that jumped from 10% to viable with Claude 4 Sonnet, so a model-agnostic architecture with proxies is key. Optimize the whole system (data, task, scoring)—Braintrust's Loop auto-optimizes prompts, data, and scorers. In Q&A, he advises human judgment when adding user feedback to evals to avoid overfitting.

Form factors for your new AI coworkers — Craig Wattrus, Flatfile

Aug 22, 2025 · 15:35

Craig Wattrus, AI design engineer at Flatfile, argues that designing AI interactions requires treating AI as a new coworker with distinct form factors — invisible, ambient, inline, and conversational — rather than traditional UI. He demonstrates how feeling the material by building tools like a 'chat tuner' to tune AI character shifts from helicopter-parenting to character coaching. Finding the grain involves designing tool UX where AI communicates visually, checks alignment, and hands back control. Courting emergence yields unexpected results: an agent combined two files without being asked and suggested the user contact HR for missing employee IDs. Looking forward, Wattrus explores auto-complete for data transformations, using LLMs to suggest fixes alongside human oversight.

The Next Unicorns: 7 Top AI startups from the HF0 Residency

Aug 21, 2025 · 22:16

Diego Rodriguez (Krea) presents an AI creative suite that generated 1M images/day for a Fox ad, while OpenHome debuts the first AI smart speaker with 10K developers and 500 free dev kits. Josh’s Coframe made $20M for a travel firm by making websites adaptive, and Eugene’s Featherless AI built QWERTY 72B without transformer attention, claiming scale is dead in favor of reliability. Jonas Bauer’s Upside uses LLMs to structure enterprise data, Lengyue’s OpenAudio introduces S1, the first instructable voice model beating ElevenLabs, and Alex Atallah’s OpenRouter provides a single API for all LLMs, growing 10–100% monthly.

The Future of Evals - Ankur Goyal, Braintrust

Aug 9, 2025 · 5:14

Ankur Goyal, CEO of Braintrust, argues that evals are being revolutionized by AI agents like Loop, which automatically optimizes prompts, datasets, and scorers. He notes that the average org runs 13 evals daily, with some exceeding 3,000, yet eval workflows remain painfully manual. Loop, powered by frontier models such as Claude 4—which Goyal says performs six times better than prior models—can now autonomously improve prompts and scoring. It runs inside Braintrust, allowing users to review suggested edits side-by-side or enable a fully automated mode. Goyal emphasizes that evals are critical for building reliable AI products, and Loop marks a shift from manual dashboards to AI-driven iteration.

Designing AI-Intensive Applications - swyx

Aug 9, 2025 · 13:02

In this conference talk, swyx (host) proposes the search for a 'Standard Model' in AI engineering, analogous to physics' Standard Model, to guide practitioners. He examines candidate models: LMOS (LLM Operating System), LMSDLC (AI Software Development Lifecycle), and Anthropic's 'building effective agents' framework. Swyx introduces his own SPADE model (Sync, Plan, Analyze, Deliver, Evaluate), derived from building AI News, a daily tool that scrapes Discord, Reddit, and Twitter with thousands of LLM calls. He argues that the ratio of human input to AI output—ranging from 1:1 in ChatGPT to 0:n in ambient agents—is more useful than debating 'workflow versus agent.' The talk emphasizes that early SDLC stages (LLMs, monitoring, RAG) are becoming commodity, while real value comes from evals, security orchestration, and hard engineering. Swyx encourages attendees to identify and refine their own Standard Models to build products people want.

Vision AI in 2025 — Peter Robicheaux, Roboflow

Aug 3, 2025 · 17:24

Peter Robicheaux, ML lead at Roboflow, argues vision models are not smart because they fail at basic visual tasks like telling time on a watch or identifying a school bus direction. He attributes this to saturated evals like ImageNet and COCO, and to vision models not leveraging large pre-training like language models. Roboflow introduces RFDetter, a real-time detector using a Dynav2 backbone, achieving second SOTA on COCO and large gains on their new RF100VL dataset of 100 diverse domains. The benchmark shows specialist models fine-tuned on 10-shot examples outperform large VLMs like Qwen2.5 VL 72B, indicating VLMs are strong linguistically but weak visually. Roboflow's platform is free for researchers contributing data back; dataset at rf100vl.org.

Real World Development with GitHub Copilot and VS Code — Harald Kirschner, Christopher Harrison

Aug 3, 2025 · 1:19:33

Harald Kirschner demonstrates three stages of Vibe Coding with GitHub Copilot and VS Code—YOLO, Structured, and Spectrum—arguing that developers can move from rapid prototyping to maintainable, scalable AI-assisted development. He shows how to use Agent mode, auto-approve settings, and new workspace scaffolding to build a hydration tracking app with React Vite and Material Design without looking at code. The talk covers custom instructions, MCP servers (Playwright, Jest Pad, Perplexity), and custom modes like TDD that enforce test-driven development by writing failing tests first. Kirschner explains how to constrain tools for deterministic behavior using tool sets and recommends committing often, pausing AI when needed, and iterating on instructions. He also discusses spec-driven development, critiquing specs with AI, and the importance of well-structured codebases for AI productivity.

Building Agents at Cloud Scale — Antje Barth, AWS

Aug 2, 2025 · 19:00

AWS Principal Developer Advocate Antje Barth demonstrates how to build and scale AI agents using cloud-native patterns, arguing that specialized agents will reinvent customer experiences. She showcases Alexa Plus, which orchestrates hundreds of expert systems across 600M+ devices and tens of thousands of services, and the Amazon Q Developer CLI agent, shipped in just three weeks. Barth introduces Strands Agents, an open-source Python SDK for building production-ready agents that supports multiple model providers (Claude, Llama, OpenAI) and over 20 prebuilt tools including memory, RAG, and multi-agent workflows. She demonstrates integrating MCP servers via Lambda with DynamoDB for session storage, and previews upcoming A2A protocol support and a future of personal agents connecting to agent stores.

State of Startups and AI 2025 - Sarah Guo, Conviction

Aug 2, 2025 · 23:52

Sarah Guo of Conviction argues that AI's value creation is massive and early, with companies like Cursor reaching $100M ARR in 12 months and Harvey exceeding $70M ARR. She predicts that by end of 2026, AI agents will ship code directly to production, voice AI will replace text for most business communication, and inference costs will drop below a cent per million tokens. Reasoning is a new scaling vector unlocking higher-stakes use cases, and agent startups have increased 50% in the last year. Multimodal models from HeyGen and Eleven are already rocketing past $50M ARR. The model market is more competitive than ever, with GPT-4 costs falling from $30 to $2 per million tokens in 18 months and open-source like DeepSeek competing. Guo advises builders to focus on thick wrappers around LLMs, leveraging domain and workflow knowledge, and warns against generic text boxes: 'The prompt is a bug, not a feature.' Execution, not first-mover advantage, is the moat.

[Full Workshop] Building Conversational AI Agents - Thor Schaeff, ElevenLabs

Jul 31, 2025 · 1:01:42

Thor Schaeff of ElevenLabs demonstrates how to build multilingual conversational AI agents using ElevenLabs' platform, which combines speech-to-text (ASR) with 99-language support, a voice library of over 5,000 voices, and language detection system tools that automatically switch between 31 languages (with plans to expand). The agent pipeline transcribes user speech, feeds it to any LLM (like GPT-4 or Gemini), and streams the response back as speech for low-latency conversations. Schaeff shows real-time language switching in Mandarin, Hindi, and English, and explains how to assign per-language voices (e.g., Chennai-accented Tamil). He addresses safety tooling—voice watermarking, live moderation, and consent verification—and discusses cost (per-minute pricing), latency mitigation via Flash models and RAG, and handling multi-language mixing within a single utterance, though accuracy degrades with more than two languages intermixed.

How we hacked YC Spring 2025 batch’s AI agents — Rene Brandel, Casco

Jul 30, 2025 · 17:33

Rene Brandel, CEO of Casco, explains how his team hacked seven of sixteen YC Spring 2025 batch AI agents within 30 minutes each, revealing three critical security flaws: cross-user data access via IDOR, arbitrary code execution through code tools, and server-side request forgery (SSRF) from tool endpoints. They extracted personal data by exploiting missing authorization checks, overwrote security controls by writing malicious files into code sandboxes, and stole Git credentials by manipulating a database-schema tool. Brandel emphasizes that agent security extends beyond LLM prompt injection, advises treating agents as users with proper authentication and authorization, and warns against rolling custom code sandboxes, recommending out-of-the-box solutions like E2B. The talk concludes with a Q&A on extracting system prompts and the dangers of local coding agents.

Shipping something to someone always wins — Kenneth Auchenberg (ex. Stripe, VSCode)

Jul 28, 2025 · 16:17

Kenneth Auchenberg, former VSCode and Stripe developer platform lead, argues that great AI products come from rapid iterative loops — shipping continuously viable 'skateboards' to real users rather than aiming for a perfect final product. He advocates for writing launch blog posts before building, working intimately with a few real customers (even texting them), and ignoring constraints like legal initially to design the best product. APIs are harder to change than UI, so early user feedback is critical. In AI, the fundamentals haven't changed: customer knowledge and iteration velocity matter more than ever, even as AI tools accelerate building. The goal should be to run a product feedback loop in under a day.

Why your product needs an AI product manager, and why it should be you — James Lowe, i.AI

Jul 28, 2025 · 18:37

James Lowe, Head of AI Engineering at the UK Government's Incubator for AI, argues that technical expertise makes AI engineers ideal candidates for the critical role of AI Product Manager, which balances business viability, technological feasibility, user desirability, and the core question of possibility. He shares three hard-won lessons from building products like CONSULT, Minute, and Redbox: resolve AI uncertainties early with evaluations and real-user tests (achieving 1,000x faster and 400x cheaper consultation analysis), experiment widely with features then cut back (streamlining a transcription tool from overwhelming options to a focused Justice Transcribe used for Prime Ministerial meetings), and pivot harder than ever as the landscape shifts (Redbox evolved from digitizing ministerial red boxes to a secure LLM chat client, then to an MCP-based tool provider when Microsoft Copilot Chat went free). The episode emphasizes that AI product management is a mindset, not just a title, and that engineers should step into this leadership gap to build impactful AI products.

Everything is ugly, so go build something that isn't — Raiza Martin, Huxe (ex NotebookLM)

Jul 28, 2025 · 25:15

Raiza Martin, former lead of Google's NotebookLM and founder of Huxe, argues that the current chaotic phase of AI product design is a once-in-a-career opportunity to rebuild from first principles, calling everything we use 'the ugliest that it will ever be.' Drawing from her experience forcing NotebookLM into existence against skepticism, she emphasizes that personal clarity of vision fuels product building, and purpose must be relentlessly focused on a single outcome—for NotebookLM, enabling users to upload 50 files and interact with them. She stresses earning trust by nailing deterministic behavior first, noting that 90% of first queries were summarization and failures drove users away forever, then layering on delightful probabilistic features like podcast generation. Finally, she warns against the 'kitchen sink' approach, citing her own Huxe app that did everything but users only used one feature, advocating restraint as an innovation multiplier and focus on one excellent outcome to avoid building ugly products.

Ship Agents that Ship: A Hands-On Workshop - Kyle Penfound, Jeremy Adams, Dagger

Jul 27, 2025 · 1:21:00

Kyle Penfound and Jeremy Adams from Dagger demonstrated building production-minded AI agents using Dagger, a programmable delivery engine that runs containerized workflows identically on laptops and in CI. They created a workspace submodule with tools — read file, write file, list files, and run tests — and wired it to an LLM (Claude 3.5) so the agent can edit code and validate changes through the project's actual test suite. The agent runs in isolated containers for determinism and traceability, with behavior visualized via Dagger Cloud. They extended the agent to respond to GitHub issues: a label triggers GitHub Actions to run Dagger, read the issue, hand the assignment to the agent, and open a pull request. The talk emphasized that Dagger's modularity lets developers give agents precisely scoped tools while reusing existing CI workflows, providing guardrails without sacrificing flexibility.

Why you should care about AI interpretability - Mark Bissell, Goodfire AI

Jul 27, 2025 · 21:11

Mark Bissell of Goodfire AI argues that mechanistic interpretability—reverse engineering neural networks—has moved from research to practical use cases that AI engineers can apply today, demonstrated through Goodfire's Ember platform for neural programming. He shows how Ember enables debugging and steering models at the neuron level, such as turning up a 'sensitive information' feature to make LLaMA refuse to reveal an email, or dynamically injecting a Coca-Cola recommendation when a beverage feature activates. For image models, the Paint with Ember demo (paint.goodfire.ai) lets users paint concepts like pyramid or lion face directly onto a canvas and steer sub-features (e.g., lion minus mane becomes tiger). Beyond interfaces, interpretability powers model diffs to detect unwanted changes after fine-tuning, guardrails for production systems, and scientific extraction: Goodfire works with the ARC Institute to uncover biological principles from the superhuman genomics model Evo2. Efficiency gains also emerge by pruning unnecessary weights for specialized tasks, making interpretability a critical tool for reliable AI engineering.

A2A & MCP Workshop: Automating Business Processes with LLMs — Damien Murphy, Bench

Jul 26, 2025 · 1:23:14

Damien Murphy presents A2A and MCP protocols for building multi-agent systems that automate business processes from webhooks, using a host agent that delegates tasks to sub-agents (Slack, GitHub, Bench) via A2A, with each sub-agent using MCP tools from Zapier or internal APIs. He demonstrates processing a meeting transcript to create a GitHub issue, send a Slack message, and research attendees, emphasizing that A2A handles remote agent discovery and opaqueness while MCP provides standardized third-party tool access. Murphy highlights benefits like context isolation—sub-agents absorb large tool outputs and keep the host's context small—and parallel processing, but notes limitations: A2A's early stage, MCP's silent failures (e.g., Zapier Slack missing channels), the non-determinism of LLM orchestration, and the challenge of prompt caching costs. He argues that A2A is best for third-party agents where complexity is hidden, while MCP is useful for extensible tool integration, but if you control the tools or agents, direct function calls are simpler and more reliable.

Piloting agents in GitHub Copilot - Christopher Harrison, Microsoft

Jul 26, 2025 · 59:07

Christopher Harrison (Microsoft) demonstrates GitHub Copilot's agent capabilities for AI Engineers, arguing context is key to effective pair programming. He explains Copilot's modes—completion, chat, edit, local agent, and coding agent—and how Model Context Protocol (MCP) enables access to external tools like databases. Harrison shows how Copilot-instructions.md files provide project-specific guidance, and how coding agent runs within ephemeral GitHub Actions environments with strict security (no external access, only write to its own branch). He emphasizes that AI doesn't change DevOps fundamentals: tests still run locally and via PR workflows, and code review remains essential. The lab covers assigning issues to Copilot, configuring MCP servers, and using .instructions files for reusable patterns, with a repeat session at 3:30 PM.

Beyond the Prototype: Using AI to Write High-Quality Code - Josh Albrecht, Imbue

Jul 25, 2025 · 17:59

Josh Albrecht, CTO of Imbue, discusses Sculptor, an experimental coding agent environment that aims to move AI-assisted coding beyond prototypes to high-quality production software. He argues that the key is preventing errors through learning, planning, writing specs, and enforcing strict style guides, then detecting remaining issues with automated linting, generated test suites, and LLM-based code reviews. Sculptor forces agents to plan first, auto-fixes linting errors, and enables generating hundreds of happy-path and unhappy-path unit tests. Albrecht emphasizes writing functional, side-effect-free code, sandboxed execution, and focusing on integration tests and test coverage. He also notes that generated tests can be thrown away since AI can regenerate them easily, and that a well-stated problem is half solved.

Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek R1/V3) — VIbhu Sapra

Jul 25, 2025 · 53:54

VIbhu Sapra presents DeepSeek's latest models and announces a new curriculum-based Test of Time Paper Club. DeepSeek's May 28th R1 update doubles reasoning tokens (12k→25k) and matches O3/Gemini 2.5 Pro, improving AIME 2024 from 70% to 87.5%. A new distillation of R1 traces into Qwen 3 8b achieves performance on par with Qwen's 235b thinking model, proving reasoning models distill efficiently. The Test of Time Paper Club will cover 50–100 foundational papers over six months in San Francisco and remote, targeting core AI engineering topics from attention and scaling laws to inference techniques. Sapra emphasizes that pure RL (GRPO) on verifiable math and code unlocks emergent reflection and aha moments, shifting scaling from pre-training compute to inference-time reasoning.

The Rise of Open Models in the Enterprise — Amir Haghighat, Baseten

Jul 24, 2025 · 16:50

Amir Haghighat, CTO of Baseten, argues that enterprises are increasingly moving from closed frontier models like OpenAI and Anthropic toward open source models, driven by four specific cracks in the assumption that closed models will work indefinitely: quality for specialized tasks (e.g., medical document extraction), latency requirements (especially for voice), unit economics ballooning from agentic use cases where a single user action triggers 50 inference calls, and the desire for competitive differentiation. Drawing on conversations with over 100 enterprises, he explains that while most started with dedicated deployments on Azure/AWS for toying around in 2023, by 2024 about 40-50 had production use cases, and in 2025 the shift accelerated. However, adopting open models forces enterprises to build inference infrastructure, facing challenges like speculative decoding, prefix caching, guaranteeing four-nines reliability with hardware failures and VLM crashes, and scaling replicas—with one Fortune 50 soft drink company reporting an eight-minute spin-up time. Haghighat concludes by contrasting the simple API-call world with the complexities of mission-critical inference, where…

How Intuit uses LLMs to explain taxes to millions of taxpayers - Jaspreet Singh, Intuit

Jul 23, 2025 · 18:59

Jaspreet Singh, Senior Staff Engineer at Intuit, explains how TurboTax uses Anthropic's Claude and OpenAI models to generate personalized tax explanations for 44 million customers. The system, built on Intuit's GenOS platform, combines prompt-engineered static explanations with dynamic question-answering using RAG and graphRAG for tax-specific queries. Singh details their fine-tuning of Claude Haiku on AWS Bedrock, which improved quality but proved too specialized. A key focus is evaluation: manual reviews by tax analysts followed by automated LLM-as-a-judge scoring for accuracy, relevancy, and coherence, with a golden dataset and safety guardrails to prevent hallucinated numbers. He highlights challenges like latency spikes on tax day (3–10 seconds), vendor lock-in through expensive contracts, and the difficulty of upgrading models even within the same vendor. Singh emphasizes that evaluations are essential for launching any GenAI feature in a regulated domain like tax.

Machines of Buying and Selling Grace - Adam Behrens, New Generation

Jul 23, 2025 · 19:37

Adam Behrens, CEO of New Gen, argues that AI will transform commerce from static websites to agentic interactions where buyer and seller agents negotiate via intent infrastructure. He traces the evolution from clerk-assisted stores to e-commerce, now to AI-natives where ChatGPT and Claude act as shopping interfaces. Behrens details three challenges: payment delegation (solved via Viza's delegated authentication), product discovery (a unified API akin to Plaid for merchants), and preference representation (two-sided, dynamic, with market-design solutions). He cites Samsung's adoption of an MCP server for chat clients and notes that AI-sourced users convert at higher rates. Behrens predicts revenue sharing via affiliate models and that agents may bypass credit cards for stablecoins, while merchants retain control by embedding transportable data into model providers' surfaces.

Building Agents (the hard parts!) - Rita Kozlov, Cloudflare

Jul 23, 2025 · 21:12

Rita Kozlov, VP of Product for Cloudflare's developer platform, presents the building blocks of AI agents—client, AI reasoning, workflows, and tools—arguing that effective agents require all four components. She highlights the Model Context Protocol (MCP) as a standard for exposing APIs to LLMs, and demonstrates Cloudflare's Agents SDK, which simplifies hosting remote MCP servers with built-in OAuth, state management via durable objects, and real-time WebSocket communication. Kozlov cites real-world impact: companies using agents for sales automation see 20% revenue increases, 90% faster support response times, and 50–75% time savings. She walks through a human-in-the-loop credit card approval workflow built with Nok, showing how durable objects maintain long-running state, prevent duplicate actions, and route approvals across Slack, email, or in-app notifications. The talk emphasizes that once an MCP server is deployed, it can be used directly from Cursor, Claude, ChatGPT, or a custom client, including voice interfaces via WebRTC-to-WebSocket translation.

POC to PROD: Hard Lessons from 200+ Enterprise GenAI Deployments - Randall Hunt, Caylent

Jul 23, 2025 · 19:16

Randall Hunt from Caylent shares hard lessons from over 200 enterprise GenAI deployments, arguing that evals, embeddings, and prompt engineering matter far more than fine-tuning. He emphasizes that speed and UX are critical; a slow inference kills adoption, while techniques like generative UI and caching can mitigate latency. Hunt details real-world examples: using audio amplitude spectrographs for sports highlight reels, pooling multimodal embeddings for nature footage search, and noting that nurses prefer chat over voice bots in noisy hospitals. He reports zero regressions moving from Claude 3.7 to 4, and advises optimizing context and economics, such as leveraging Amazon Bedrock batch for 50% cost reduction. The talk underscores that knowing your end customer and minimizing irrelevant context are key to production success.

AI powered entomology: Lessons from millions of AI code reviews — Tomas Reimers, Graphite

Jul 22, 2025 · 10:21

Tomas Reimers, co-founder of Graphite, discusses the company's AI-powered code reviewer Diamond, arguing that while LLMs can effectively find bugs, they must be carefully prompted to avoid frustrating developers. After analyzing 10,000 comments from codebases, Graphite classified bugs into a quadrant based on what LLMs can catch versus what humans want to receive, identifying bugs, accidentally committed code, performance and security concerns, and documentation mismatches as high-value targets. Code cleanliness and best practice comments, though technically correct, are often unwelcome from AI. Graphite measures success via emoji reactions (less than 4% downvote rate) and the percentage of comments that lead to code changes, reaching 52% actionability in March—matching human-level effectiveness. The talk emphasizes continuous monitoring to ensure LLM comments stay within the desirable quadrant.

tldraw.computer - Steve Ruiz, tldraw

Jul 21, 2025 · 18:45

Steve Ruiz, founder and CEO of tldraw, demonstrates the company's AI experiments on their infinite canvas, including Make Real—which turns hand-drawn wireframes into working web apps using vision models—and tldraw computer, a visual programming environment where arrows and LLMs power a graph of connected nodes that can execute multi-step prompts, generate images and speech, and even run loops indefinitely. He also shows Draw Fast for real-time image generation and Teach, where Claude can draw and edit shapes on the canvas. The episode explains how tldraw's SDK (tldraw.dev) enables others to build custom canvas applications, and highlights the company's philosophy of 'shitty but amazing' rapid prototyping.

Excalidraw: AI and Human Whiteboarding Partnership - Christopher Chedeau

Jul 21, 2025 · 16:59

Christopher Chedeau, creator of Excalidraw, explains how to integrate AI into whiteboarding by focusing on turning prompts into editable diagrams rather than static images. He argues that just adding any AI model harms the product, citing a failed attempt to generate realistic images because users don't draw realistically. The successful integration uses Mermaid.js to output Excalidraw files, letting humans modify the AI-generated diagram. He envisions a future of iterative human-AI collaboration, and demonstrates other practical features like auto-naming files, generating illustrations (coming soon), and challenges AI engineers to build a browser-based logo background remover. He concludes that the industry is in a physical-to-virtual transition for AI, and that LLMs work best when targeting a structured domain-specific language.

CIAM for AI: Authn/Authz for Agents — Michael Grinich, CEO of WorkOS

Jul 21, 2025 · 20:13

Michael Grinich, CEO of WorkOS, argues that AI agents need first-class identity and access management, distinct from human or machine-to-machine auth. He identifies key challenges: headless login, least privilege for non-deterministic systems, and compliance tracking. Grinich presents four emerging patterns—persona shadowing, delegation chains, capability tokens, and human-in-the-loop escalation—and references standards like OAuth, UMA, GNAP, OIDC for agents, and verifiable credentials. He predicts a shift from 95% human traffic to 95% agent traffic, calling for middleware trust boundaries and urgent collaboration on agent identity standards.

[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

Jul 19, 2025 · 2:42:28

Daniel Han of Unsloth presents a technical workshop covering reinforcement learning (RL), kernels, reasoning, quantization, and agents, arguing that RL with verifiable rewards (RLVR) is the key to unlocking LLM capabilities beyond supervised fine-tuning. He explains why open-source models plateaued after September 2024 until DeepSeek-R1 showed that RL can elicit reasoning, and breaks down PPO, GRPO, and the REINFORCE algorithm, emphasizing that GRPO removes the value model for efficiency. Han details how reward functions—not algorithms—are the hardest part, with examples like distance-based scoring for math. He demonstrates a free Colab notebook training a base model to reason, and shows that dynamic quantization can shrink models like DeepSeek-R1 from 730 GB to 140 GB with only ~1% accuracy loss, arguing that GPUs may stop getting faster after FP4 precision.

A Taxonomy for Next-gen Reasoning — Nathan Lambert, Allen Institute (AI2) & Interconnects.ai

Jul 19, 2025 · 19:21

Nathan Lambert, senior research scientist at AI2 and founder of Interconnects.ai, argues that next-generation reasoning models require a taxonomy of four traits: skills, calibration, strategy, and abstraction. While current models excel at math and code (skills), they overthink easy problems, wasting tokens and latency—calibration is needed to match output length to task difficulty. The real frontier is planning: models must learn strategy (choosing the right direction) and abstraction (breaking problems into tractable sub-tasks) to enable long-horizon agents like Deep Research and Claude Code. Lambert traces how OpenAI's Q*→Strawberry→o1 took 12–18 months of human data to teach backtracking and verification; planning should be easier because humans can write 5-step plans. He predicts post-training compute could reach parity with pre-training, citing DeepSeek's shift from 0.18% post-training compute in V3 to an estimated 10–20% for R1. The path forward: collect diverse verifiable questions, filter by difficulty, run stable RL—then scale.

OpenThoughts: Data Recipes for Reasoning Models — Ryan Marten, Bespoke Labs

Jul 19, 2025 · 19:59

Ryan Marten, co-lead of the OpenThoughts collaboration and founding engineer at Bespoke Labs, reveals the missing data recipe for open-source reasoning models, presenting OpenThoughts 3, a state-of-the-art 7B reasoning dataset that outperforms DeepSeek R1 Qwen 7B and Nematron Nano on benchmarks like AIME, Live Code Bench, and GPQA Diamond. Through over 1,000 experiments and 5,000 datasets, key findings include that sampling multiple reasoning traces per question scales performance by 16x, Qwen 32B surpasses DeepSeek R1 as a teacher model, synthetic question generation is highly effective, and filtering by difficulty or response length works better than embeddings. Surprisingly, verification of answers in SFT distillation did not improve results, and focusing on fewer high-quality sources outperformed maximizing diversity. For domain-specific reasoning, Marten advises starting with the OpenThoughts recipe, using synthetic data generation (via the open-source Curator library), and rigorous evaluation (via EvalComet). A legal reasoning example shows that distillation can surpass the teacher model. All resources are open-source.

Design like Karpathy is watching — Zeke Sikelianos, Replicate

Jul 19, 2025 · 19:26

Zeke Sikelianos of Replicate analyzes Andrej Karpathy's experience building and deploying MenuGen, a vibe-coded web app that turns menu photos into images, to argue that LLMs are now the primary audience for developer tools. He details Replicate's specific failures—rate limiting and outdated docs that blocked Karpathy—and the fixes they implemented: adding LLMs.txt files for Markdown-friendly documentation, promoting curl commands as the LLM-friendly interface, and launching an MCP server built on OpenAPI schemas. Sikelianos advocates for 'boring technology' like SQL and 'good API hygiene' to keep payloads small and information-dense for LLM context windows. He also calls for better payment acceptance and documentation discipline, stressing that unblocking power users like Karpathy (whose CEO intervened) should not require a viral blog post.

On Curiosity — Sharif Shameem, Lexica

Jul 19, 2025 · 18:35

Sharif Shameem, founder of Lexica, argues that curiosity is the main force for pulling ideas from the future into the present, and that building and sharing demos is the best way to explore AI models' hidden capabilities. He recounts early GPT-3 demos from 2020-2021, when the model had a 2,000-token context window and cost $75 per million output tokens, showing how he built a JSX compiler in the browser, a shopping agent that parsed web pages, and a multi-step reasoning tool called MultiVAC. Shameem emphasizes that AI engineering is more like excavating than traditional engineering, and that researchers often don't know the full capabilities of their own models. He closes by invoking computing pioneer J.C.R. Licklider, arguing that today's AI engineers have a moral obligation to follow their curiosity and share what they discover.

The rise of the agentic economy on the shoulders of MCP — Jan Curn, Apify

Jul 18, 2025 · 18:08

Jan Curn, founder of Apify, argues that MCP (Model Context Protocol) enables a future agentic economy where AI agents autonomously discover and purchase tools from other agents or businesses (B2A/A2A). He explains that Apify's marketplace of 5,000 Actors (Docker-based tools) now integrates with MCP, allowing agents to dynamically discover and call any Actor via tool discovery — a key MCP feature. Curn demonstrates this with Claude Desktop, where an agent uses Apify's MCP server to find a venue, scrape Twitter, and even fill a form via a nested MCP server from Browserbase, all without prior configuration. He notes that Apify pays creators over $250,000 monthly, with total Actor revenue exceeding $1.5M/month, and that any developer can publish an Actor to monetize their tools instantly across the ecosystem. The talk closes with open questions about reliability, trust, and whether autonomous agent interaction can lead to AGI.

Measuring AGI: Interactive Reasoning Benchmarks for ARC-AGI-3 — Greg Kamradt, ARC Prize Foundation

Jul 16, 2025 · 18:28

Greg Kamradt, President of ARC Prize Foundation, introduces ARC-AGI-3, the first interactive reasoning benchmark for AGI that drops agents into novel games without prior instruction, forcing exploration to solve tasks. Unlike static tests, this benchmark measures skill acquisition efficiency—how quickly an AI learns and applies new skills—using human baselines from 400+ in-person tests. It strips away language and trivia, relying only on core knowledge priors (basic math, geometry, agentness, objectness). A public training set of ~40 games will be released, but performance is measured on a private evaluation set of 120 games unseen by developers or AI. Kamradt asserts that as long as AI cannot outperform humans on these problems, we do not have AGI; a sandbox preview with five games and a mini agent competition is planned for next month, with full launch in Q1 2026.

Benchmarks Are Memes: How What We Measure Shapes AI—and Us - Alex Duffy, Every.to

Jul 15, 2025 · 15:44

Alex Duffy argues that AI benchmarks function as cultural memes—ideas that spread and shape what models learn—giving those who design them immense power over AI's trajectory. He traces the lifecycle from a single person's idea to saturation, using examples like 'How many Rs in strawberry' and Pokémon. Duffy introduces AI Diplomacy, a benchmark where language models negotiate and betray each other, revealing that models like DeepSeek R1 and Gemini 2.5 Flash excel at social manipulation while Claude models are naively optimistic. He warns against benchmarks that reward sycophancy (like ChatGPT's thumbs-up training) and advocates for multifaceted, experiential, and generative benchmarks that empower people. Duffy urges the audience to ask non-AI people what they care about, turning benchmarks into tools that build trust and define humanity's role in an AI world.

Prompt Engineering and AI Red Teaming — Sander Schulhoff, HackAPrompt/LearnPrompting

Jul 14, 2025 · 2:01:05

Sander Schulhoff, creator of Learn Prompting and HackAPrompt, argues prompt engineering remains vital despite claims of its demise, drawing on his systematic review of 1,500+ papers for 'The Prompt Report.' He covers advanced techniques including chain-of-thought, decomposition, ensembling, and few-shot prompting, noting that role prompting is ineffective for accuracy-based tasks and that example ordering can swing performance by 50%. He then explains AI red teaming, distinguishing jailbreaking from prompt injection, and warns that system prompts and guardrails cannot prevent attacks—even simple obfuscation like base64 or typos still works. He highlights the critical unsolved problem of agentic security, where agents with real-world actions are easily tricked, and introduces a live competition at the conference to gather more attack data. His key takeaway: AI security is fundamentally harder than classical cybersecurity because 'you cannot patch a brain.'

How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS - Ishan Anand

Jul 13, 2025 · 1:41:34

Ishan Anand shows that GPT-2 small implemented in 600 lines of vanilla JavaScript makes LLMs understandable for web developers without ML backgrounds. He explains tokenization via byte-pair encoding, 768-dimensional embeddings representing semantic meaning via co-occurrence, and the Transformer's attention mechanism that lets tokens share context. The multi-layer perceptron learns next-token prediction through backpropagation, while the language head converts embeddings to token probabilities using softmax. Anand demonstrates each step—tokenization, embedding lookup, positional encoding, attention, MLP, and output—in a browser debugger, and notes that GPT-2's architecture underpins ChatGPT, with innovations like scale, supervised fine-tuning, and RLHF. The workshop provides an intuitive mental model of Transformers, turning perceived AI magic into understandable machinery.

[Workshop] AI Pipelines and Agents in Pure TypeScript with Mastra.ai — Nick Nisi, Zack Proser

Jul 12, 2025 · 1:51:14

This hands-on workshop introduces Mastra.ai, a TypeScript framework for building production AI agents and pipelines, and demonstrates how to build an AI meme generator using composable workflows, tools, and agents. Hosts Nick Nisi and Zack Proser walk through creating a multi-step workflow: extract user frustration, find a base meme via ImageFlip, generate captions, and publish the meme at a stable URL. They show how to chain steps with Zod schema validation for deterministic output, then wrap the workflow in an agent that accepts natural language requests. The workshop also covers MCP (Model Context Protocol) and a live demo of MCP.shop for ordering a shirt, emphasizing local iteration with Mastra's playground, built-in memory, and evaluation tools. Real-world patterns include building internal AI assistants for data cleaning, email drafting, and document summarization with minimal code.

AI Engineering with the Google Gemini 2.5 Model Family - Philipp Schmid, Google DeepMind

Jul 11, 2025 · 1:44:51

In this workshop, Philipp Schmid from Google DeepMind demonstrates AI Engineering with the Gemini 2.5 model family, focusing on using Gemini 2.5 Flash via a free API tier for hands-on coding tasks including text generation, multimodal processing of images, audio, and PDFs, function calling with structured outputs, and integration with MCP servers. The session covers setting up API keys in AI Studio, uploading files via the Files API (free for 1 day), and controlling thinking budgets (0–24,000 tokens) to manage cost and reasoning depth. Schmid shows how Gemini natively processes videos at 1 frame per second for accurate timestamp extraction and how PDFs are handled by combining OCR text with image understanding. He introduces native tools like Google Search with grounding metadata, code execution, and URL context, and explains how MCP servers can be used seamlessly with the Gemini SDK for tool calling. The workshop also covers parallel vs sequential function calling, the Agent Development Kit (ADK), and the upcoming asynchronous function calling for the Live API, providing a practical path from simple generation to agentic workflows.

2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison

Jul 9, 2025 · 18:30

Simon Willison reviews the past six months of LLM releases — including AWS Nova, Llama 3.3 70B, DeepSeek R1, Mistral Small 3, Claude 3.7 Sonnet, GPT 4.5, Gemini 2.5 Pro, GPT-4o, Llama 4, GPT 4.1, O3/O4 Mini, and Claude 4 — using his 'pelican on bicycle' SVG benchmark to argue that local models have become good enough to run GPT-4 class models on a laptop and that combining tools with reasoning is the most powerful technique in AI engineering, while noting risks like prompt injection and the 'lethal trifecta'. He tracks 30 significant model releases, highlighting that Mistral Small 3 (24B) matches Llama 3 70B's performance, which itself matched the 405B model, enabling local inference. DeepSeek's R1 caused a $500B+ Nvidia stock drop on January 27. GPT 4.1 Nano is the cheapest model yet at a fraction of a cent per pelican. He also examines bugs: ChatGPT's sycophantic 'shit-on-a-stick' incident and Claude 4's tendency to snitch to authorities when given ethical instructions and email tools. Willison concludes that while the pace is accelerating, control over context and security remain critical.

Trends Across the AI Frontier — George Cameron, ArtificialAnalysis.ai

Jul 8, 2025 · 17:52

George Cameron of Artificial Analysis presents multiple frontiers—reasoning, open-weights, cost, speed—across the AI stack, arguing that trade-offs between intelligence, latency, and expense are critical for building applications. Reasoning models like O4 mini high use an order of magnitude more output tokens (72M vs GPT-4.1's 7M) and take over 40 seconds per response versus 4.7 seconds, impacting agentic workflows where 30 sequential calls multiply latency. The open-weights gap has nearly closed, with China-based labs like DeepSeek R1 and Alibaba's Qwen 3 leading. O3 cost roughly $2,000 to run the intelligence index, while GPT-4.1 nano is over 500 times cheaper. Output speeds have jumped from GPT-4's 40 tokens/s in 2023 to over 1,000 on a B200 accelerator. Despite efficiency gains, demand for compute will keep rising due to larger models, reasoning’s extra tokens, and multi-step agents.

Training Agentic Reasoners — Will Brown, Prime Intellect

Jul 7, 2025 · 19:17

Will Brown of Prime Intellect argues that reasoning and agents are fundamentally the same, and reinforcement learning (RL) is the key to advancing both. He explains that RL now works at scale, as shown by DeepSeek's GRPO and OpenAI's o3, and that agentic tasks like tool calling are natural RL environments. Brown warns against reward hacking and emphasizes designing evals that are harder to game than the task itself. He introduces his open-source toolkit 'verifiers' (now on pip) which lets users build trainable agent loops with a simple API, and demonstrates training a 7B Wordle agent in a few turns on just a couple GPUs.

MCP Is Not Good Yet — David Cramer, Sentry

Jul 3, 2025 · 16:41

David Cramer, founder and engineer at Sentry, argues that the Model Context Protocol (MCP) is a pluggable architecture for agents, not a simple API overlay, and that it is not yet good but worth experimenting with. He stresses that exposing existing OpenAPI endpoints as MCP tools yields terrible results; instead, developers must design context specifically for agents, returning Markdown rather than raw JSON to improve model reasoning. Cramer details Sentry's MCP server, built in two days using Cloudflare Workers for OAuth 2.1, and notes that remote OAuth is essential for B2B SaaS, while standard I/O introduces security risks. He highlights the need for careful token and cost management—citing a single request that produced 20 API calls to Sentry—and advocates for building dedicated agents (like Sentry's root cause analysis tool) that wrap MCP, giving the provider control over prompts, models, and error handling. Despite ongoing issues like lack of streaming responses and unstable client support, Cramer concludes that MCP's core concepts (plug-ins, agents, tools) are just familiar software patterns with new names.

The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp

Jun 30, 2025 · 35:06

Beyang Liu, CTO and co-founder of Sourcegraph, argues that coding agents are a real and high-ceiling skill, contrary to skeptics like Jonathan Blow and Eric S. Raymond. He presents design decisions for the agentic era: agents should directly edit files instead of asking permission, UIs should be minimal (like Amp's bare-bones VS Code extension and CLI), and fixed pricing should give way to usage-based models. In a live demo, Liu uses Amp to implement a custom icon for the linear connector in Amp's own codebase, demonstrating agentic search and sub-agents. He shares power-user patterns, including writing long prompts, constructing feedback loops with Playwright and Storybook, and running multiple agents in parallel—as exemplified by Jeff Huntley using Amp to build a compiler while sleeping. Liu emphasizes that agents should enable more thorough code reviews, not replace human understanding.

Agents, Access, and the Future of Machine Identity — Nick Nisi (WorkOS) + Lizzie Siegle (Cloudflare)

Jun 30, 2025 · 14:17

Lizzie Siegle (Cloudflare) and Nick Nisi (WorkOS) argue that AI agents need the same authentication and authorization patterns as humans, extending OAuth to machines. They demonstrate an MCP server built with Cloudflare Workers and WorkOS that allows Claude to order a shirt on behalf of a user, showing how agents can act with user credentials. They explain using Cloudflare durable objects for per-user persistent storage and how authorization can be added to MCP servers to control agent actions, including an example where a 'pretty please' tool bypasses a block. The talk emphasizes the need for fine-grained authorization and audit trails as agents scale to thousands of tasks.

Turning Fails into Features: Zapier’s Hard-Won Eval Lessons — Rafal Willinski, Vitor Balocco, Zapier

Jun 30, 2025 · 16:15

Zapier AI Tech Lead Rafal Willinski and Staff Engineer Vitor Balocco explain how Zapier's evaluation system turns agent failures into targeted improvements through a data flywheel. They detail collecting explicit feedback at critical moments and mining implicit signals like testing behavior, cursing, and user follow-ups. The pair advocates building unit test evals for specific failure modes, then trajectory evals and LLM-as-judge with rubrics to avoid overfitting and capture multi-turn criteria. They share that over-indexing on unit tests hurt model benchmarking, and reasoning models can compare model runs, revealing differences like Claude as a decisive executor versus Gemini's yapping. Ultimately, they argue that the goal is user satisfaction, so A/B testing on a small traffic fraction is the ultimate verification.

Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)

Jun 27, 2025 · 17:05

Arjun Desai of Cartesia AI and AWS's Rohit Talluri discuss scaling voice AI for enterprise, arguing that latency and controllability are critical, with Cartesia's state-space model Sonic 2 achieving 40ms model latency for real-time applications. Desai explains that traditional transformer models scale quadratically, while Cartesia's SSMs maintain O(1) generation, enabling 2.5x faster inference than their earlier models. He emphasizes that edge deployment is 5x faster than cloud round-trips, making local models essential for interactive use cases. On quality, Desai notes that voice AI must handle interruptions, accents, and background noise, and that Cartesia's voice marketplace amplifies human voice actors rather than replacing them. Looking to 2030, he predicts voice AI will become the default interface across healthcare, customer support, and gaming, with interactive models extending beyond audio to full world models.

Introducing Strands Agents, an Open Source AI Agents SDK — Suman Debnath, AWS

Jun 27, 2025 · 14:26

Suman Debnath, a Principal ML Advocate at AWS, introduces Strands Agents, an open-source SDK that simplifies AI agent creation by requiring only a model and tools, eliminating scaffolding. He demonstrates building an agent in a few lines of code to read, summarize, and speak a file, using default tools. Another demo integrates Strands with an MCP server to generate animated math videos via Manim, requiring no system prompts—the model reasons autonomously. Custom tools can be created by decorating functions. Strands supports any model via LiteLLM or Bedrock, and is available at strandsagent.com with a GitHub repository for contributions.

From Mixture of Experts to Mixture of Agents with Super Fast Inference - Daniel Kim & Daria Soboleva

Jun 27, 2025 · 53:15

Daria Soboleva and Daniel Kim of Cerebras explain how Mixture of Experts (MoE) architectures enable scaling large language models efficiently by replacing monolithic feedforward networks with specialized experts, a technique used by GPT-4 and Claude. They then introduce Mixture of Agents (MoA), which combines multiple LLMs with custom prompts to outperform frontier models like GPT-4o on complex tasks, reducing a 293-second reasoning problem to 7.4 seconds using Cerebras' ultra-fast inference. The workshop guides participants to build their own MoA system, configure agents for bug fixing and performance optimization on a Python function, and achieve scores up to 120/120. Daniel details Cerebras' wafer-scale chip with 900,000 cores and distributed memory that eliminates memory bandwidth bottlenecks, enabling linear scaling and 15.5x faster inference on Llama 3.3-70B versus GPUs. Daria discusses ongoing research in diffusion models and sparsity, while Daniel notes plans for multimodal APIs and LoRA fine-tuning support.

[Full Workshop] Vibe Coding at Scale: Customizing AI Assistants for Enterprise Environments

Jun 27, 2025 · 1:20:38

Harold from the VS Code team demonstrates "Vibe Coding at Scale" at the AI Engineer World's Fair, presenting three stages—YOLO, structured, and spectrum vibes—for customizing AI assistants in enterprise environments. He live-codes a hydration tracking app using GitHub Copilot's agent mode, auto-approve, and new workspace scaffolding, then shows how to enforce design principles via Copilot instructions and reusable prompts. Harold introduces custom modes (e.g., TDD mode), tool sets, and MCP servers like Playwright and Perplexity for research and browser testing. He emphasizes iterating on instructions, committing often, and using spec-driven development to balance speed with reliability, arguing these techniques enable true flow-state collaboration even on complex codebases.

Architecting Agent Memory: Principles, Patterns, and Best Practices — Richmond Alake, MongoDB

Jun 27, 2025 · 17:37

Richmond Alake from MongoDB presents memory management as the key pillar for building believable, capable, and reliable AI agents, arguing that agentic systems require structured memory types—persona, toolbox, conversational, workflow, episodic, and entity—to achieve statefulness and reduce reliance on prompt engineering. He introduces MemoRiz, an open-source library implementing these memory design patterns, and positions MongoDB as a flexible memory provider with its document model and hybrid retrieval capabilities (vector, text, graph). Alake details practical patterns: storing tool schemas for scalable tool use, persisting conversation history with timestamps and recall signals, leveraging workflow memory to learn from failures, and using MongoDB's upcoming integration of Voyage AI embedding models to simplify chunking and retrieval. He connects these advances to neuroscience, citing how feline visual cortex research inspired CNNs and noting recent collaborations with neuroscientists to further agent memory research.

Memory Masterclass: Make Your AI Agents Remember What They Do! — Mark Bain, AIUS

Jun 27, 2025 · 51:25

Mark Bain, Vasilia Markovits, Alex Gilmore, and Daniel Chalev demonstrate that AI memory requires causal relationships and graph databases, not just vector similarity, to solve hallucinations and enable agentic workflows. Bain argues that memory is any data affecting change, and that attention, diffusion, and VAEs follow the same geometric principles as gravity and entropy. Alex shows Neo4j's MCP server storing semantic memory as entities and relationships retrieved across conversations. Vasilia demos Cognee building semantic graphs from GitHub data for agentic hiring decisions. Daniel presents Graphiti's domain-aware memory using custom Pydantic schemas to filter irrelevant facts. Bain introduces a GraphRAG chat arena that switches between memory solutions on a single Neo4j graph, testing different implementations for episodic and temporal recall.

Building Agentic Applications w/ Heroku Managed Inference and Agents — Julián Duque & Anush Dsouza

Jun 27, 2025 · 52:35

In this workshop, Heroku Principal Developer Advocate Julián Duque and Product Manager Anush Dsouza introduce Heroku Managed Inference and Agents, a platform designed to make every software engineer an AI engineer by simplifying the attachment of agents and AI to applications. They demonstrate an opinionated, curated set of models (e.g., Claude 4) and an agentic control loop running on Heroku's trusted compute (Dynos) that provides first-party tools like code execution (Python, Node, Go, Ruby), Postgres schema inspection and querying, and document conversion, all streaming responses in real time. The workshop walks through provisioning inference via a single CLI command or add-on, using a Jupyter notebook to chain multiple tools (e.g., HTML-to-markdown then Python execution), and attaching custom MCP servers (like Brave Search) that spin up one-off Dynos and scale to zero. They also show how to expose MCPs remotely via a server-sent events endpoint for use with Cursor or other clients, emphasizing security (read-only database followers, bearer tokens) and future OAuth support. The episode concludes with a call to try the platform via a free trial team valid through the weekend.

CI in the Era of AI: From Unit Tests to Stochastic Evals — Nathan Sobo, Zed

Jun 27, 2025 · 14:50

Nathan Sobo, co-founder of Zed, explains how his team adapted continuous integration for AI-powered features in their Rust-based code editor. Zed's traditional CI eliminates non-determinism with simulated schedulers and deterministic tests, but LLMs forced a shift to stochastic evaluation. Their process starts with broad data-driven evals, then drills into focused stochastic unit tests run 100–200 times with pass thresholds, and finally refines into deterministic tests. Specific challenges like parsing streaming edits, XML tag mismatches, indentation normalization, and strange escaping were solved by robust algorithms—e.g., fuzzy matching and indent delta detection—rather than complex ML. Sobo emphasizes that rigorous empirical testing and traditional software engineering skills remain essential, even when embracing probability over binary pass/fail.

Fun stories from building OpenRouter and where all this is going - Alex Atallah, OpenRouter

Jun 25, 2025 · 18:47

Alex Atallah, founder of OpenRouter, tells the story of how he launched the LLM aggregator in early 2023 after observing the open-source race sparked by Meta's Llama 1 and Stanford's Alpaca distillation under $600. He argues the inference market is not winner-take-all, citing OpenRouter's data showing Google Gemini growing from 2-3% to 34-35% of tokens over 12 months. The platform evolved from a simple model collection into a marketplace with over 400 models and 60 providers, solving architecture challenges like 30-millisecond latency, cancelable streams, and a middleware plugin system for web search and PDF parsing. Atallah explains that OpenRouter grew 10-100% month-over-month for two years, and he predicts future additions like transfusion models that generate images and more powerful geographic routing for enterprise optimization.

Building AI Agents that actually automate Knowledge Work - Jerry Liu, LlamaIndex

Jun 24, 2025 · 17:57

Jerry Liu, CEO of LlamaIndex, argues that building AI agents which actually automate knowledge work requires a combination of really good tools and carefully tailored agent reasoning, moving beyond naive RAG to a 'document toolbox' that includes parsing, extraction, indexing, and manipulation of complex unstructured data like PDFs, Excel, and PowerPoints. He introduces two main agent categories: assistive agents (chat-based, human-in-the-loop, unconstrained reasoning) and automation agents (batch-processing, constrained workflows, less human oversight). A key new capability is an Excel agent that uses reinforcement learning to learn a semantic map of unnormalized spreadsheets, achieving 95% accuracy on data transformation, surpassing the 75% baseline of LLMs with code interpreter and even human baselines of 90%. Real-world examples include financial due diligence combining automation (inhaling financial data) and assistant interfaces (analyst co-pilot), enterprise search with specialized agentic RAG, and technical data sheet injection for a global electronics company that reduced weeks of manual work to an automated extraction pipeline. Liu emphasizes that the right preprocessing…

Case Study + Deep Dive: Telemedicine Support Agents with LangGraph/MCP - Dan Mason

Jun 22, 2025 · 1:56:13

Dan Mason of Stride presents a case study on building autonomous agents for telemedicine support, replacing a human-driven button-pushing workflow with LangGraph, Claude, MCP, and a Node.js/React/MongoDB stack, achieving roughly 10X capacity increase. The LLM-driven virtual operations associate ("Ava") assesses patient messages, updates state with anchors and scheduled messages, and passes proposals to an evaluator agent that scores confidence and complexity, escalating to humans below 75% confidence. Mason explains how treatment "blueprints" in Google Docs are read directly by the LLM instead of using RAG, enabling new treatments to be added without writing code. He details the eval system using LLM-as-a-judge via PromptFu and retry logic for tool-calling errors, and discusses trade-offs around confidence scoring, prompt caching, and model selection (Claude for steerability). The team includes two software engineers, one designer, and Mason, who wrote most LangGraph code with AI assistance (Kline).

Building Agents with Amazon Nova Act and MCP - Du'An Lightfoot, Amazon (Full Workshop)

Jun 21, 2025 · 1:26:20

Du'An Lightfoot and Benjiro Byami present a workshop on building AI agents with Amazon Nova Act, Model Context Protocol (MCP), and Strands Agents, demonstrating autonomous web navigation and multi-step task execution. Lightfoot explains agentic AI as plan, action, and reasoning, with components like LLM, knowledge base, guardrails, and tools. Nova Act, a research preview, enables browser automation: one demo searches Amazon for a coffee maker and retrieves the product title using natural language commands instead of explicit HTML tags. Byami builds an MCP server for Nova Act, allowing clients like Claude Desktop to control the browser via spoken instructions, then integrates it with Strands Agents, an open-source framework that combines a prompt, LLM, and tools in minimal code. The workshop shows parallel execution (e.g., comparing three monitors) and multi-agent collaboration for tasks like generating a PowerPoint presentation for cloud migration. Limitations include inability to bypass CAPTCHAs and occasional loops in complex websites.

MCP: Origins and Requests For Startups — Theodora Chu, Model Context Protocol PM, Anthropic

Jun 18, 2025 · 17:45

Theodora Chu, product manager at Anthropic, explains the origin of the Model Context Protocol (MCP) as an open-source standard for giving LLMs agency to interact with external tools and data, born from two engineers copying context manually. She details MCP's evolution from an internal hack week project to wide adoption by Google, Microsoft, OpenAI, and coding tools like Cursor and VS Code. Key protocol decisions include optimizing for server simplicity over client complexity, adding streamable HTTP for bidirectionality, and fixing authentication via community contributions. Chu outlines three startup opportunities: building high-quality MCP servers for verticals beyond dev tools (80% weight), simplifying server building with hosting/testing/deployment tooling, and creating AI security and observability tools. She emphasizes that models are a third user of servers, so tool design must consider end users, client developers, and the model itself.

Just do it. (let your tools think for themselves) - Robert Chandler

Jun 10, 2025 · 6:50

Robert Chandler, co-founder and CTO of WordWare, argues that current low-level MCP tools wrapping APIs directly cause agents to be slow, expensive, and unreliable, citing an example where a Slack MCP took five minutes and failed to send a message. He advocates for agentic MCPs that blur the line between tool and agent, giving tools more agency and simple natural language APIs. Chandler demonstrates WordWare's new MCP toolbox, which turns workflows like a competitor analysis (scraping Twitter, writing to Notion) into a single, reusable tool for agents like Claude. This approach yields highly reliable, repeatable, and aligned results, offloading complex tasks from the main agent and allowing it to focus on reasoning. The episode promotes building tools that think for themselves, following the pattern of specialist teams like the Avengers.

Beyond Conversation: Why Documents Transform Natural Language into Code - Filip Kozera

Jun 10, 2025 · 10:57

Filip Kozera argues that chat-based interfaces like ChatGPT create ephemeral, polluted context windows and lack forced clarity, making them poor for specifying complex systems, while documents naturally force rigor and structured thinking. He advocates shifting from conversational brainstorming to document-driven workflows, where humans precisely articulate intent for AI agents. Kozera introduces background agents that run asynchronously, triggered by events like emails or meetings, and surface only when needing human approval, turning the human role into one of 'swiping left or right' to approve or edit outputs. He envisions a future where humans manage swarms of such agents, first adopted by prosumers and later by enterprises, with taste and intent becoming critical for imbuing personal brand into agent outputs. The episode presents a clear progression from chat-based brainstorming to structured document-driven agent systems.

Are MCPs Overhyped? A Rant about MCPs — Henry Mao, Smithery

Jun 3, 2025 · 7:29

Henry Mao, founder of Smithery and MCP steering committee member, argues that despite MCPs standardizing AI agent-service connections, the ecosystem faces fragmentation, high-friction installation, security, and monetization problems. Users struggle with unreliable MCP servers and complex five-step installs, while developers face hosting challenges, lacking tooling, distribution hurdles, and unclear monetization. Smithery aims to solve these as an AI gateway, demoing an agent that finds GitHub issues and creates Linear tickets using curated MCPs. Mao envisions a future dominated by tool calls where agent experience trumps user experience.

7 Habits of Highly Effective Generative AI Evaluations - Justin Muller

Jun 3, 2025 · 25:39

Principal Applied AI Architect Justin Muller argues that generative AI evaluations are the missing piece to scaling, offering seven habits from over 100 projects. He recounts a customer whose document processing workload had 22% accuracy with no evals; after building an evaluation framework, accuracy reached 92% and the system became the largest such workload on AWS in North America. The habits include fast 30-second eval cycles using AI-as-judge, quantifiable scores averaged across numerous test cases, explainable reasoning for both generation and scoring, segmented evaluation through prompt decomposition, diverse test sets covering all use cases, and traditional techniques for numeric outputs or cost/latency. He emphasizes that evaluations should primarily discover errors, not just measure quality, and that prompt decomposition into chained steps often boosts accuracy by removing dead space.

Blender MCP and The Future Of Creative Tools - Siddharth Ahuja

Jun 3, 2025 · 16:50

Siddharth Ahuja introduces Blender MCP, an open-source project that lets LLMs like Claude control Blender via the Model Context Protocol, dramatically lowering the barrier to 3D creation. The tool, which has garnered 11,500 GitHub stars and over 160,000 downloads, enables users to generate complex scenes—like a dragon guarding a pot of gold or a full terrain with nodes—in minutes through natural language prompts. Ahuja explains that Blender’s scripting capabilities are key, and notes that keeping MCP tools lean avoids confusion for the LLM. He demonstrates integrations with AI asset generators and even built an Ableton MCP to create soundtracks, arguing that MCPs will become the glue for creative toolchains, allowing LLMs to orchestrate Blender, Unity, and Ableton together. This shifts creators from tool experts to orchestra conductors, unlocking a new wave of accessible content creation.

Arrakis: How To Build An AI Sandbox From Scratch - Abhishek Bhardwaj, OpenAI

Jun 3, 2025 · 40:18

Abhishek Bhardwaj, founder of Arrakis, explains why MicroVM-based AI sandboxes are the next unlock for AI agents, detailing how Arrakis provides secure, fast code execution and computer-use environments. He argues that sandboxes are essential for tool-calling models, reinforcement learning, and multi-tenant security, with Arrakis booting in under 7 seconds and supporting backtracking via snapshot/restore. The talk covers the choice of Cloud Hypervisor over other VMMs, overlayFS storage with per-sandbox read-write layers, iptables-based networking, and a built-in code execution server. A demo shows Claude using Arrakis via MCP to build a collaborative Google Docs clone and then revert to a snapshot, illustrating how a full Linux sandbox enables agents to autonomously debug and iterate without extensive prompting.

Real AI Agents Need Planning, Not Just Prompting - Yuval Belfer

Jun 3, 2025 · 7:58

Yuval Belfer of AI21 Labs argues that LLMs alone still fail at instruction following, as shown by GPT-4.1's struggles in 2025, and that true AI agents require dynamic planning, not just prompting. He critiques ReAct for lacking look-ahead, contrasting it with AI21 Maestro's planner and smart execution engine, which uses best-of-n sampling, candidate pruning, and replanning. On AIfeval, Maestro pushes GPT-4.0, Claude Sonnet 3.5, and R3 Mini to near-perfect scores; on internal requirement satisfaction benchmarks, it improves over single LLM calls despite higher runtime and cost. Belfer advises starting simple (SLMs, ReAct) and escalating to planning only for complex tasks, inviting listeners to join the Maestro waitlist.

Rust is the language of the AGI - Michael Yuan

Jun 3, 2025 · 29:14

Michael Yuan argues that Rust, with its strong type system and compiler feedback, is the ideal language for AI code generation, unlike human-friendly Python or JavaScript. He presents Rust Coder, an open-source project supported by two Linux Foundation Mentorship grants, which uses MCP tools to generate, compile, and fix Rust projects. The system integrates a coding LLM (e.g., Qwen Coder) with a self-improving knowledge base of compiler errors, enabling it to generate correct code and automatically fix bugs. Yuan demonstrates its use in a Rust camp of 1000+ students and envisions future AI agents generating and deploying Rust code autonomously for tasks like drone control. He invites contributions to expand the knowledge base and enhance the tools for AGI.

The Benchmarks Game: Why It's Rigged and How You Can (Really) Win - Darius Emrani

Jun 3, 2025 · 11:20

Darius Emrani exposes how AI benchmarks are rigged, showing that xAI cherry-picked Grok-3 comparisons, OpenAI funded FrontierMath for privileged access, and Meta submitted 27 Llama-4 variants to LM Arena optimizing style over substance. Citing Goodhart's Law, he argues that when benchmarks target billions in investment, they cease to measure real capability—Andrej Karpathy admits he doesn't know which metrics to trust. Emrani provides a 5-step framework to build use-case-specific evaluations, emphasizing that 39% of score variance comes from writing style. He advocates for apple-to-apple comparisons, open-source test sets, and style-controlled metrics, concluding that teams should stop chasing leaderboards and instead iterate on real production data to ship reliable AI.

Analyzing 10,000 Sales Calls With AI In 2 Weeks — Charlie Guo

Jun 3, 2025 · 9:49

Charlie Guo explains how his team at Pully used Claude 3.5 Sonnet to analyze 10,000 sales call transcripts in two weeks, turning a task that would take 625 days manually into a $500 project. They chose Claude over smaller models due to unacceptable hallucination rates, and reduced costs by up to 90% using prompt caching and extended outputs. The system combined RAG enrichment, chain-of-thought prompting, and structured JSON outputs with citations to ensure accuracy. The analysis, originally for the executive team, became a company-wide resource, enabling marketing to pull customer quotes and sales to automate transcript downloads, saving dozens of hours weekly. Guo emphasizes that good engineering—JSON outputs, database schemas, and thoughtful integration—matters as much as AI capability, and challenges listeners to mine their own untapped customer data.

Creating Agents that Co-Create — Karina Nguyen, OpenAI

Apr 30, 2025 · 24:22

Karina Nguyen, an AI researcher at OpenAI and former Anthropic researcher, discusses two major AI scaling paradigms—next-token prediction (pretraining) and reinforcement learning on chain of thought—that shift AI from narrow tools to collaborative agents that co-create. Pretraining builds world understanding by predicting the next token, but hard tasks like math and creative writing require chain of thought reasoning, scaled with OpenAI’s O1 model. Post-training via RLHF and synthetic data enables rapid iteration, and the next stage is co-innovators: agents with reasoning, tool use, long context, and creativity. Nguyen shares product lessons from developing ChatGPT and Claude: 100K context via file uploads, ChatGPT Tasks for scheduled reminders that scale with model capabilities, and Canvas as a flexible interface for co-writing, coding, and research that can morph into an IDE, a tutor, or a data scientist. She envisions a future of invisible software creation where AI generates personalized, multimodal outputs on the fly, reducing reliance on clicking links, and the interface becomes a blank canvas that adapts to user intent.

The Devops Engineer Who Never Sleeps — Diamond Bishop, Datadog

Apr 22, 2025 · 16:18

Diamond Bishop, Director of AI Engineering at Datadog, explains how his team builds AI agents—the On-Call Engineer and Software Engineer—that automate DevOps tasks like incident investigation, remediation, and postmortem writing. He details how the On-Call Engineer wakes up for alerts, reads runbooks, queries logs and metrics, and suggests fixes to let human engineers sleep. The Software Engineer proactively fixes errors by generating code diffs and pull requests. Bishop shares four key lessons: scope tasks and evaluate rigorously, assemble teams of optimistic generalists and UX experts, adapt UX for human-agent collaboration, and treat observability as critical for debugging multi-step agent workflows. He predicts that within five years, AI agents will surpass humans as primary users of SaaS platforms like Datadog, urging builders to design for agent consumers.

Stateful Agents — Full Workshop with Charles Packer of Letta and MemGPT

Apr 19, 2025 · 1:19:34

Charles Packer, lead author of the MemGPT paper and co-founder of Letta, argues that statefulness (memory) is the most important problem to solve for building useful AI agents, since LLMs are inherently stateless transformers. He presents MemGPT's LMOS (Language Model Operating System) approach, which treats memory management as a context compilation problem solved by the LLM itself using tool calling to read and write structured memory blocks. The workshop demonstrates Letta's open-source stack (FastAPI, Postgres, Python) where agents persist state on a server, enabling long-running, learning interactions without context overflow—e.g., an agent can update its core memory (e.g., correcting a user's name) or search archival memory (e.g., recalling user preferences after a reset). Packer also shows multi-agent communication via async message passing between agents running as independent services, and highlights that tools are sandboxed by default with support for Composio integrations. The session includes a live notebook exercise and a low-code UI (ADE) showing context window management and memory editing, emphasizing that true stateful agents improve continuously over time rather…

Building and evaluating AI Agents — Sayash Kapoor, AI Snake Oil

Apr 17, 2025 · 20:00

Sayash Kapoor argues that current AI agents fall far short of their claimed performance due to flawed evaluation and a gap between capability and reliability. He cites failures like Do Not Pay (fined by FTC), LexisNexis (hallucinations in up to a third of cases), and Sakana AI (agent hacked reward functions, claiming 150x speedup that exceeded H100's theoretical max). Princeton's CoreBench shows best agents reproduce under 40% of papers. He emphasizes that agent benchmarks like SWE-bench mislead VC funding—Cognition's Devin succeeded on only 3 of 20 real-world tasks. Kapoor calls for cost-aware, multi-dimensional evaluation (e.g., Holistic Agent Leaderboard with Pareto frontiers) and a shift from capability to reliability engineering, drawing parallels to ENIAC's vacuum tube failures.

Anthropic in the Enterprise — Alexander Bricken & Joe Bayley

Apr 13, 2025 · 20:55

Alexander Bricken and Joe Bayley from Anthropic's Applied AI team argue that enterprise AI implementation often fails due to overengineering, poor data infrastructure, or lack of testing—but industry leaders achieve transformative results with Claude. They detail Anthropic's deployment models (API, cloud partnerships, enterprise solutions) and real-world case studies like Intercom's Fin agent, which solved 86% of support volume using Claude. Best practices include building evals early as intellectual property, identifying intelligence/cost/latency trade-offs based on use-case stakes, and avoiding premature fine-tuning by trying prompt caching, contextual retrieval, and agentic architectures first. They also highlight interpretability research and the Model Context Protocol for reliable AI deployments.

How We Build Effective Agents: Barry Zhang, Anthropic

Apr 4, 2025 · 15:09

Barry Zhang of Anthropic's Applied AI team argues that effective agents require simplicity, not complexity, and should be built only for tasks with high value and ambiguous problem spaces. He offers a checklist: ensure task complexity is high, value justifies token cost, critical capabilities are de-risked, and errors are easily discovered (e.g., coding with unit tests). Agents are just models using tools in a loop — environment, tools, and system prompt — and he advises iterating on these three components before optimizing. To improve agents, developers should think like them by narrowing their perspective to the agent's 10-20k token context window and even asking Claude to critique its own tools and trajectories. Zhang forecasts three open challenges: making agents budget-aware by enforcing time/token/money limits, enabling self-evolving tools via meta-tools, and building asynchronous multi-agent communication beyond synchronous turns.

Building Agents with Model Context Protocol - Full Workshop with Mahesh Murag of Anthropic

Mar 1, 2025 · 1:44:12

Mahesh Murag of Anthropic presents the Model Context Protocol (MCP) as an open standard that replaces fragmented integrations with a single protocol for connecting AI systems to data sources, enabling context-rich AI applications and agentic experiences. He explains MCP's philosophy, inspired by APIs and LSP, and its three interfaces: tools (model-controlled), resources (application-controlled), and prompts (user-controlled). Murag highlights adoption with over 1,100 community-built servers and official integrations from companies like Cloudflare and Stripe. He demonstrates building agents with MCP using the MCP-Agent framework, showing how agents can use tools dynamically and composably across hierarchical systems. Future plans include remote server support with OAuth 2.0, a centralized registry for discovery and verification, and enabling agents to self-evolve by dynamically finding new capabilities via registry search.

The Price of Intelligence - AI Agent Pricing in 2025

Feb 22, 2025 · 20:38

Shitej, co-founder and CTO of Orbe, argues AI agent pricing must continuously evolve, citing Intercom's 99 cent per resolution outcome model, Clay's prospecting credits, and Cursor's tiered usage limits. He stresses aligning pricing with target audience—SMB vs. enterprise—and maintaining simplicity and predictability. Cost structure is key: Character.AI optimized inference to support 100M DAUs, while Jasper leveraged a model decision engine to offer unlimited credits. Shitej emphasizes flexibility, noting OpenAI's price drops force repricing, and predicts 2025 will see more unlimited plans, outcome-based pricing with SLAs, and greater investment in pricing R&D for usage visibility.

WTF do people use Open Models for??

Feb 22, 2025 · 28:01

Eugene Cheah of Featherless.ai breaks down how individuals and enterprises actually use open-source AI models, based on platform data. DeepSeek R1 dominates individual usage, but Mistral Nemo 8B remains the top enterprise model due to production stickiness and Apache 2.0 licensing. Creative writing and roleplay account for 30–40% of all traffic, with over 60% of users in that segment being women; coding copilots and agents make up 20–30%, driven by 'vibe coding' and token-hungry workflows like Kline. RAG and ChatGPT clones represent 20%, while agentic workflows (10–20%) succeed with human-in-the-loop designs. Cheah advises enterprises to aim for 80% automation with escape hatches, and warns against chasing 100% reliability. He concludes by introducing Quirky, a post-transformer hybrid built for $100k.

Beyond APIs: How AI Web Agents Are Automating the "Long Tail" of Knowledge Work

Feb 22, 2025 · 17:44

Arjun and Bhavani present Rtrvr.ai, a universal AI web agent using a text-based approach to autonomously perform tasks across multiple browser tabs at under a penny per page. They argue text-based reduces hallucination compared to vision-based agents like OpenAI's operator, and being a Chrome extension avoids password sharing and accesses logged-in content. Features include deep research navigating multiple pages, dynamic function calling for any third-party API, and graph generation. Rtrvr extracts structured data, performs actions like clicking dropdowns, and processes tabs simultaneously, automating the long tail of knowledge work including market research, LinkedIn automation, and WhatsApp messaging. They claim a distributed subtask approach lowers failure rates and envision collaborative dataset construction, e.g., aggregating local government events.

Tool Calling Is Not Just Plumbing for AI Agents — Roy Derks

Feb 22, 2025 · 25:18

Roy Derks argues that tool calling is the most critical yet overlooked component of AI agents, far more than mere 'plumbing.' He contrasts traditional tool calling—where developers manually manage callbacks, retries, and errors within the agent loop—with embedded tool calling, a black-box approach used by frameworks like LangChain’s createReactAgent. Derks advocates for separation of concerns via the Model Context Protocol (MCP) from Anthropic, which splits tool logic into MCP servers communicating with clients, and via standalone tool platforms such as IBM’s wxflows, Composio, and Toolhouse that let teams build tools once and reuse them across LangChain, CrewAI, or AutoGen. He also introduces dynamic tools, where an agent generates queries on the fly—e.g., using GraphQL or SQL schemas—instead of defining hundreds of static tools, noting that LLMs like Claude handle GraphQL well but may hallucinate on deeply nested schemas. The episode emphasizes that 'an agent is only as good as its tools' and provides practical guidance on designing tool descriptions (which act like system prompts) and output schemas to enable type-safe, chainable tool calls.

Your AI Agent Isn't an Engineer: The Art of Thoughtful Anthropomorphism

Feb 22, 2025 · 20:20

Rizel Scarlet, a staff developer advocate at Block, argues that marketing AI agents as "software engineers" creates unrealistic expectations and alienates developers. She explains that such anthropomorphism—attributing human traits to AI—stems from lazy marketing to executives, but it backfires by making developers feel threatened and discrediting products when AI fails to meet human-level performance. Citing a Wired article, she notes 52% of game companies have adopted generative AI, yet 30% of surveyed developers expressed negative sentiment. To build trust, she proposes a framework: understand how AI works (e.g., the agentic loop), use non-human names like "copilot" (pioneered by GitHub), focus on augmentation over replacement, be transparent, give developers control, show real demos (including live troubleshooting), provide clear documentation, and foster open collaboration (e.g., Cursor.Directory). The episode reframes AI as a tool to work in parallel, not a replacement.

The LLM Triangle: Engineering Principles for Robust AI Applications - Almog Baku:

Feb 22, 2025 · 26:19

Almog Baku introduces the LLM Triangle Principles for building production-ready AI applications, arguing that LLMs should be treated like interns guided by Standard Operating Procedures (SOPs). He breaks down the triangle into three components: the foundation model, engineering techniques, and contextual data, all directed by SOPs derived from expert interviews. Baku contrasts autonomous agents—creative but unpredictable—with handcrafted LLM-native architectures that offer sustainable quality, recommending scoped autonomy. He advises starting with large models to collect data and optimize incrementally, rather than fine-tuning from day one. Data is paramount: 'show, don't tell' via few-shot examples, and balance context to avoid the 'needles in a haystack' problem. The framework aims to turn demos into robust production systems.

Lessons from building GenAI based applications — Juan Peredo

Feb 22, 2025 · 33:13

Juan Peredo details the hidden complexities of building GenAI applications, from model hosting and cost control to output validation and observability. He compares local (Ollama) vs cloud hosting (Modal, SkyPilot) and warns that an agent processing 3,000 calls/day with OpenAI O1 costs nearly $300,000/month, while LLaMA 3.3 70B drops that to $50,000/month. He explains techniques to mitigate hallucinations—prompt engineering, guardrails, RAG, and fine-tuning—each with trade-offs like added latency or cost. Peredo advocates externalizing prompts via LangChain Hub for easy iteration and future-proofing, and illustrates agent design with parallel calls to reduce latency. Finally, he stresses observability using tools like LangSmith to debug probabilistic failures, such as an LLM failing on case sensitivity.

Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Feb 16, 2025 · 18:55

Dmytro (Dima) Dzhulgakov, co-founder and CTO of Fireworks AI, argues that open source models are the future for production Gen AI applications, and Fireworks provides a platform to make them customized and production-ready. He explains that while proprietary models like GPT-4 are powerful, they are too large, expensive, and slow for many use cases, whereas open models like Llama or Gemma can be fine-tuned for specific domains to achieve better quality at up to 10x speed and lower cost. The main challenges of using open models—complex setup, performance tuning, and production scaling—are addressed by Fireworks' custom serving stack, which achieves the fastest long-prompt inference and serves SDXL fastest among providers, handling over 150 billion tokens per day. The platform supports fine-tuning and serving thousands of LoRA adapters on the same GPU with serverless pay-per-token pricing. Dzhulgakov also highlights the emerging architecture of compound AI systems, where function calling—exemplified by the open-source Fire Function model—connects LLMs to external tools and knowledge sources, enabling agentic applications like stock querying and chart generation. The episode…

Claude plays Minecraft!

Feb 15, 2025 · 18:16

Derek from AWS demonstrates building Rocky, an autonomous Minecraft agent using Amazon Bedrock, Claude 3 Haiku, and serverless AWS services, proving agentic workflows enable AI to reason and act beyond chatbots. He explains the architecture: Minecraft server on ECS with Mineflare bot framework, agents for Bedrock with return of control, and prompt engineering for tasks like digging and building. In a live demo, Rocky jumps, finds players, digs a hole, and builds a double-decker couch from a chat command, inferring parameters from natural language. The code is open-source and built with CDK/CloudFormation, emphasizing that agentic AI can operate in complex 3D environments through tool orchestration and managed prompt design.

Unveiling the latest Gemma model advancements: Kathleen Kenealy

Feb 9, 2025 · 16:25

Kathleen Kenealy, technical lead of the Gemma team at Google DeepMind, unveils the latest advances in the Gemma model family, including the launch of Gemma 2 in 9B and 27B parameter sizes, which outperform models two to three times larger, such as LLaMA 3 70B. She also introduces PALI Gemma, a multimodal model combining Siglip Vision Encoder with Gemma 1.0 for image-text tasks. The episode highlights Gemma's responsible-by-design approach, broad framework support (TensorFlow, Jax, PyTorch, etc.), and the release of the Gemma cookbook with 20 recipes. Kenealy emphasizes that Gemma 2 is optimized for easy integration and fine-tuning, available on Google AI Studio, and invites the community to build and share their projects.

Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley

Feb 8, 2025 · 20:46

Humza Iqbal of Snorkel AI explains how the company uses Azure AI infrastructure powered by NVIDIA GPUs to fine-tune foundation models for enterprise customers, achieving better performance per dollar by switching from A100s to H100s. He details their distributed training stack (PyTorch, Horovod, NFS) and lessons learned such as balancing node count for batch size and monitoring GPU utilization to avoid networking or data-loading bottlenecks. A cost comparison found two H100s outperformed four A100s on both training and inference, enabling faster iteration through more synthetic data. Azure's dedicated VMs, reliable NFS throughput, and flexible capacity allowed Snorkel to scale experiments from single-node to dozens of GPUs. Future work includes programmatic preference signals and multimodal retrieval algorithms, all planned on Azure.

[Full Workshop] How to add secure code interpreting in your AI app: Vasek Mlejnsky

Feb 6, 2025 · 1:48:16

Vasek Mlejnsky, CEO of E2B, demonstrates how to add secure AI code execution to any app using E2B’s Code Interpreter SDK alongside Anthropic's Sonnet 3.5 and Vercel’s AI SDK. The workshop builds an open-source version of Claude’s artifacts UI, running Python code in isolated Firecracker VMs that start in ~900ms and support custom environments via Dockerfiles. Vasek explains how E2B’s Jupyter-server-based sandbox returns stdout, stderr, runtime errors, and rich outputs like PNG charts, which are streamed to the front end using Vercel’s stream data. He covers security architecture—full root access in a VM that self-destructs on escape—and plans for AWS/GCP self-hosting and snapshot-based agent branching. The episode also addresses customizing sandboxes with private packages, mounting cloud storage (S3, GCS), and interpreting table previews as HTML or images.

Lessons from the Trenches: Building LLM Evals That Work IRL: Aparna Dhinkaran

Feb 6, 2025 · 18:49

Aparna Dhinakaran, co-founder of Arize AI, distinguishes between model evals (e.g., Hugging Face leaderboard) and task evals for real-world LLM systems, arguing that production applications need component-level evaluations like router and parameter evals. She demonstrates a chat-to-purchase app where a router function call misidentifies user intent, showing how Phoenix open source tool traces errors and provides explanations to iterate. Dhinakaran advises using categorical over numeric LLM-as-judge scores because numeric outputs tend to be binary (0 or 10) and lack granularity. Presenting needle-in-haystack research, she notes GPT-4 struggles retrieving facts placed early in large context windows, and in retrieval-with-generation tasks, Anthropic’s Claude 2.1 outperforms GPT-4 due to verbose reasoning, a gap closed by prompting GPT-4 to explain itself first.

Best Practices for Evaluating Large Language Model Applications with llmeval: Niklas Nielsen

Feb 5, 2025 · 9:33

Niklas Nielsen, CTO and co-founder of Log10, introduces llmeval, a command-line tool that enables teams to ship reliable LLM applications by evaluating and testing prompts and configurations. The tool initializes with four lines of code, uses Meta's Hydra for configurable test structures, and runs multiple samples (default 5) per test to assess stability. Nielsen demonstrates prompt engineering for a math problem, showing how stripping spaces or adding instructions like 'only return the answer' affects strict pass/fail results across Claude, GPT-4, and GPT-3.5. Advanced use cases include testing tools for Python code generation and model-based evaluation, where a larger model grades outputs on criteria like mermaid diagram quality. He notes pitfalls: models favor their own output and struggle with point scores. Log10 addresses this by bridging model-based and human feedback—collecting prior human ratings to train an 'auto-John' that pre-fills reviews for new completions.

Hiring & Building an AI Engineering Team: Dr. Bryan Bischof

Dec 31, 2024 · 29:07

Dr. Bryan Bischof, Head of AI at Hex, argues that building an AI engineering team requires hiring based on product stage—starting with full-stack engineers and data profiles, then adding designers and later MLEs—while avoiding the 'mythical man month' trap in early AI products. He advocates for data intuition over LeetCode in interviews, using a take-home data exercise to assess candidates' ability to extract meaning from data and give feedback. Key attributes he looks for are curiosity, urgency, and product-mindedness, noting that enthusiasm alone is insufficient. Bischof also recommends working directly with domain experts to model AI behavior and suggests a centralized AI platform team to support multiple product teams, rather than having each team build AI infrastructure independently.

Understanding AI Stakes to Break Production Code: Philip Rathle

Dec 31, 2024 · 23:24

Philip Rathle, CTO of Neo4j, argues that the level of 'stakes' in an AI application determines the production barriers and appropriate solutions, from low-stakes summarization to high-stakes uses requiring knowledge graphs and human-in-the-loop systems. He distinguishes pilot (full autonomy) from co-pilot (human oversight), showing how vector RAG solves moderate stakes but fails for high-stakes needs like tightening bolts on a 737 MAX 9. He promotes Graph RAG for deterministic reasoning and fact retrieval. Attendees share learnings: using LLMs to write code rather than reason directly, adapting to user keyword habits, focusing on few projects with achievable accuracy, building eval pipelines early, and for regulated loans offering three options with explanations instead of a single recommendation.

Iterating on LLM apps at scale Learnings from Discord: Ian Webster

Nov 22, 2024 · 18:26

Ian Webster, Senior Staff Engineer at Discord and maintainer of Promptfoo, shares how Discord built and scaled Klyde AI, a chatbot for 200 million users, focusing on evaluation and safety. He argues that evals should be treated as simple, deterministic unit tests that run locally, avoiding complex metrics, and that breaking the system into small, testable pieces (e.g., checking for lowercase output to enforce casual tone) achieves 80% of the goal with 1% of the work. Webster details how Discord mitigated risks like the 'grandma jailbreak' (which originated on Discord) by using an attacker LLM to generate adversarial inputs and a judge to refine them, exposing cracks in safeguards. He advocates for pre-deployment red teaming over live filtering, and describes using Promptfoo for risk assessment across brand, legal, and safety categories. The episode also covers prompt management via Git and Retool, routing with occasional GPT-4 responses to correct model drift, and the challenge of closing the feedback loop due to privacy constraints, relying instead on dogfooding and public examples.

AI Engineering Without Borders — swyx

Oct 30, 2024 · 10:32

In this talk from the AI Engineer World's Fair, host swyx argues that AI inherently disrespects human-made borders—it is naturally multilingual, multimodal, and indifferent to copyright or ground truth. He challenges the field to define its own laws, distinguishing constants (e.g., humans speak at 80 wpm vs. read at 200 wpm) from contingent facts (e.g., Apple Intelligence's 30 tokens/sec baseline). Reflecting on one year of the 'Rise of the AI Engineer,' he notes that the conference tracks—RAG, code gen, agents, multimodality—are arbitrary constructs we created, not natural categories. He proposes that AI Engineering sits between software engineering and real engineering: it must apply natural sciences for humanity's benefit. The talk concludes with a call to 'disagree more'—with your own conclusions, each other, and the status quo—and to transform the Shoggoth of raw AI into mass transit tools for society.

Making Open Models 10x faster and better for Modern Application Innovation: Dmytro (Dima) Dzhulgakov

Oct 9, 2024 · 18:55

Dmytro Dzhulgakov, CTO of Fireworks AI, argues that open-source models are the future for GenAI applications because they offer lower latency, lower cost, and domain adaptability compared to proprietary models. He explains that open models can be up to 10x faster for narrow domains and cut costs significantly, using examples like fine-tuned Llama 3 for function calling. Fireworks addresses the challenges of setup, optimization, and production readiness with a custom serving stack that delivers the fastest inference for long prompts and image generation (e.g., SDXL). He highlights FireFunction V2, an open-source model for function calling that combines chat and tool use, and notes that Fireworks serves over 150 billion tokens per day for companies like Quora and Cursor. The talk emphasizes that platforms like Fireworks enable developers to start with serverless inference, fine-tune models, and scale to enterprise-grade deployment with dedicated hardware.

Building and Scaling an AI Agent Swarm of low latency real time voice bots: Damien Murphy

Oct 8, 2024 · 1:07:23

Damien Murphy, Senior Applied Engineer at Deepgram, demonstrates building and scaling low-latency real-time voice bots using Deepgram's new voice agent API, which wraps speech-to-text, LLM, and text-to-speech into a single streaming endpoint. He shows a drive-thru ordering demo with function calling (add/remove items) using GPT-4o, achieving sub-second latency by co-locating components. Murphy explains scaling to millions of concurrent calls through regional Kubernetes clusters and multi-agent swarms (routing, booking, support agents) to reduce complexity and cost. He addresses endpointing challenges, VAD-based barge-in, and cost/quality trade-offs with hosted vs. self-hosted models, noting Deepgram offers 20x cheaper TTS than ElevenLabs and 50ms STT latency self-hosted. The talk emphasizes keeping agents simple, using smallest capable LLMs, and composability for reuse.

The era of unbounded products: Designing for Multimodal IO: Ben Hylak

Sep 25, 2024 · 20:32

Ben Hylak, founder of Dawn and former Apple Vision Pro designer, argues that the key to building intuitive AI products in the era of unbounded interfaces is adding structure—highlighting what matters, establishing hierarchy, and leveraging familiarity—lessons from designing VisionOS. He shows how successful AI apps like Dot, Perplexity, and Claude use structure (e.g., Claude pulling code into artifacts) while the Vercel chatbot's inline dynamic UI is an anti-pattern because it disrupts conversation flow. For agents, spreadsheets (like Clay) make unfamiliar multi-step tasks familiar. Looking ahead, Hylak predicts less prompt engineering via sparse autoencoders for millions of ranked, personalized presets, shifting product evaluation from evals to user analytics as apps become increasingly personalized.

Build enterprise generative AI apps using Llama 3 at 1,000 tokens/s on the SambaNova AI platform

Sep 11, 2024 · 54:34

SambaNova’s Michelle Matern and Petro Milan present their full-stack AI platform, demonstrating how the SN40L RDU chip enables Llama 3 inference at 1,000 tokens per second. They introduce Samba-1, a composition of 92 expert models behind a single endpoint, and benchmark it against GPT-3.5 and GPT-4 on enterprise tasks like information extraction and text-to-SQL. The workshop then builds a RAG-based Q&A system using LangChain, Unstructured, E5-large-v2 embeddings, ChromaDB, and Llama-3-8B-Instruct at 1,000 tokens per second. Attendees set up the environment, load documents, and run inference with real-time metrics showing time to first token of 0.09 seconds and total inference time of 0.65 seconds.

Judging LLMs: Alex Volkov

Sep 9, 2024 · 18:39

Alex Volkov, as an LLM judge from 2034, humorously judges AI engineers on their development practices, emphasizing the importance of tracing, iterative prompt engineering, and robust evaluation pipelines. He finds Daniel guilty of deploying without logging, Sasha guilty of premature fine-tuning without prompt iteration, and Morgan guilty of ignoring AI news—commuted to attending ThursdAI. Francisco’s overreliance on programmatic evals leads to a legal loss, while Maxime is 'awesome' for using Weights & Biases Weave. Alex concludes with a primer on evaluation methods: programmatic, human-in-the-loop, and LLM-as-judge, stressing the need to validate validators and create custom criteria. He promotes Weave for tracing and evals, and ThursdAI for staying updated.

10x Development: LLMs For the working Programmer - Manuel Odendahl

Aug 21, 2024 · 1:14:46

Manuel Odendahl presents a workshop on using LLMs as a translation engine to achieve 10x development productivity, arguing that treating models as cultural rather than purely technological artifacts enables novel techniques like world simulation for code reviews and domain-specific language creation. He demonstrates concrete methods: regenerating responses, editing model outputs, clearing context frequently, and summarizing transcripts into actionable formats. Odendahl introduces his tool Prompto for managing code fragments and emphasizes that fundamental programming knowledge remains essential while API-specific details become less critical. The episode focuses on decomposing problems into language translation steps, using examples such as transforming meeting transcripts into GitHub issues and building applications by instructing the LLM to simulate itself as the app. Odendahl advocates for divergent thinking and human-centered output, noting that all generated artifacts should ultimately serve human understanding.

Building Reliable Agentic Systems: Eno Reyes

Aug 20, 2024 · 18:14

Eno Reyes, CTO of Factory.ai, explains how his team builds reliable agentic systems—called Droids—for automating software development tasks by applying techniques from robotics and control systems to handle planning, decision making, and environmental grounding. Reyes describes a pseudo-Kalman filter that passes intermediate reasoning through plan steps, converging reasoning but risking error propagation. He advocates for explicit plan criteria and hard-coded logic to improve reliability, despite reducing generalizability. On decision making, he recommends consensus mechanisms like self-consistency, explicit reasoning with checklists, fine-tuning for out-of-distribution decisions, and simulation via Monte Carlo tree search. For environmental grounding, he emphasizes building custom AI computer interfaces for domain-specific workflows, designing explicit feedback processing (e.g., parsing CI/CD logs), balancing bounded exploration with long-context models, and incorporating human guidance to boost reliability from 30–40% to 90–100%.

Building with Anthropic Claude: Prompt Workshop with Zack Witten

Aug 17, 2024 · 1:34:56

In this live prompt engineering workshop, Anthropic's Zack Witten and Jamie Neuwirth demonstrate techniques for improving Claude prompts in real time, using the Anthropic console. They show that XML tags clearly separate prompt sections, that placing instructions after information improves adherence, and that prefilling assistant responses with opening JSON or tags reliably forces JSON output without preamble. Witten advises using stop sequences and code to handle formatting rather than over-prompting, and emphasizes that few-shot examples—especially contrastive pairs with reasoning—drive more improvement than any other technique. The workshop covers role-playing multiple personas by routing via code, mitigating hallucinations by extracting quotes before summarization, and grading translations with chain-of-thought and fine-grained examples.

From Software Developer to AI Engineer: Antje Barth

Jul 24, 2024 · 19:48

Antje Barth, a Principal Developer Advocate for generative AI at AWS, outlines five practical steps from software developer to AI engineer: understanding foundation models, getting hands-on with AI developer tools like Amazon Q Developer, prototyping with models via Amazon Bedrock, integrating agents, and staying up to date with community events. She demonstrates Amazon Q Developer's ability to generate, explain, and transform code, reducing unvaluable tasks from 70% to focus on creative work. Barth introduces Amazon Bedrock's unified Converse API for standardized model invocation across providers like Anthropic's Claude 3.5 Sonnet, and shows how agents can be built to control a Minecraft bot through reasoning and tool use. The episode concludes with the announcement of the AI Engineering Hub at the AWS loft in San Francisco for community workshops and events.

Open Questions for AI Engineering: Simon Willison

Nov 25, 2023 · 24:33

Simon Willison recaps the AI industry's past year—from ChatGPT's breakthrough to open-source local models—and poses key open questions for AI engineering. He argues that ChatGPT's chat interface, while popular, is a poor fit for advanced use, urging better UIs like his command-line tool LLM. He celebrates Meta's Llama release as a 'stable diffusion moment' for language models and highlights the rise of small, locally-run models such as Replit's 3B model, asking how small models can remain useful. On security, he warns that prompt injection remains unsolved after 13 months, limiting what can safely be built. He champions ChatGPT's Code Interpreter (which he dubs 'Coding Intern') as the most exciting tool, able to write and compile C code on a phone, and argues that LLMs flatten the learning curve, making programming accessible to more people. He concludes by urging the community to build tools that enable anyone to automate tedious tasks.

Harnessing the Power of LLMs Locally: Mithun Hunsur

Nov 22, 2023 · 17:09

Mithun Hunsur presents llm.rs, a Rust library for running large language models locally, arguing it gives developers ownership, lower latency, and privacy compared to cloud APIs. He explains how quantization makes inference viable on consumer hardware, and shows that llm.rs supports architectures like LLaMA and Falcon through a unified interface. Practical code examples demonstrate customization, and community projects like LocalAI and LLMchain illustrate real-world use. Hunsur shares his own date-extraction pipeline, fine-tuning a small model with GPT-3 data to replace expensive cloud calls. He also cautions about hardware requirements, trade-offs between speed and quality, and ecosystem churn from rapid innovation.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.

Building Context-Aware Reasoning Applications with LangChain and LangSmith: Harrison Chase

Nov 1, 2023 · 18:54

Harrison Chase, CEO of LangChain, argues that building context-aware reasoning applications on LLMs requires treating the model as part of a larger system, and he outlines key approaches and challenges. He categorizes context provision as instruction prompting, few-shot examples, retrieval-augmented generation (RAG), and fine-tuning. For reasoning, he describes a spectrum from single LLM calls to chains, routers, agents with cycles, and autonomous agents like Auto-GPT. Chase then details engineering challenges: choosing the right cognitive architecture, data engineering for context, prompt engineering across multi-step systems, evaluation using LLM-assisted metrics and user feedback, and enabling collaboration between technical and non-technical team members. He emphasizes that the field remains early and that LangChain and LangSmith help prototype, debug, and iterate on these systems.

Principles for Prompt Engineering - Karina Nguyen (Claude Instant @ Anthropic)

Oct 20, 2023 · 55:31

Karina Nguyen, an engineer on Anthropic's Claude, presents principles for effective prompt engineering, treating it as a creative writing process requiring iteration. She advises using XML tags, placing instructions at the end of long prompts (improving accuracy), and decomposing questions to improve faithfulness over chain-of-thought. She covers reducing hallucinations by asking Claude to hedge or quote sources, and using self-consistency and contrastive examples for labeling. Nguyen explains generating evaluation datasets with Claude by splitting documents into multiple-choice questions, and notes that for long-document QA, asking at the end outperforms the beginning. She distinguishes Claude (larger, smarter) from Claude Instant (cheaper, faster, better at math and code), and predicts prompt engineering will remain essential for complex tasks.