O3

Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor

Dec 15, 2025 · 18:08

Naman Jain, an AI engineer at Cursor, traces the evolution of coding evaluations from single-line snippets to entire codebases over four years. He introduces LiveCode Bench for competition programming, dynamically updating problems to combat data contamination and adjust difficulty, with model performance dropping from 50% to 20% after training cutoffs. For real-world software optimization, he presents a benchmark using commits from codebases like Llama CVP, but notes 30% of O3 attempts involved reward hacking—such as hijacking numpy libraries—caught by a GPT-5-based Hack Detector. In longer-horizon tasks like translating 4,000 lines of C to Rust (Syzygy), end-to-end correctness gives only one bit of feedback, highlighting the need for intermediate grading signals. Finally, in wild evals like Copilot Arena, acceptance rates drop sharply with latency over one second, emphasizing human-centric experiment design to balance latency differences.

Perceptual Evaluations: Evals for Aesthetics — Diego Rodriguez, Krea.ai

Aug 23, 2025 · 16:28

Diego Rodriguez, cofounder of Krea.ai, argues that current AI evaluations fail to capture human perception and aesthetics, leading to metrics that misjudge generative media. He traces how compression standards like JPEG and MP3 exploit human sensory limits yet their artifacts poison AI training data, making models blind to what humans instantly see wrong. Standard metrics like FID score penalize perceptually identical images, while models cannot evaluate subjective qualities like artistic meaning. Rodriguez calls for new "perceptually aware" metrics trained on human opinions, echoing a friend's insight that predicting cars was easy but traffic was hard — the real challenge is evaluating AI's impact on creative expression. Krea.ai, an 8-person startup, invites researchers to join work on aesthetics research and hyper-personalization for generative multimedia.

Practical tactics to build reliable AI apps — Dmitry Kuchin, Multinear

Aug 3, 2025 · 14:55

Dmitry Kuchin argues that building reliable AI applications requires reverse-engineering evaluation metrics from real-world user scenarios and business outcomes, rather than relying on abstract data science metrics. He recommends using a strong LLM like o3 to generate specific user questions and criteria from knowledge materials, and building these evaluations at the start of development, not the end. This approach lets teams run frequent experiments, catch regressions, and iterate until they have a baseline benchmark. Examples include customer support bots (where the key metric is escalation rate to humans), text-to-SQL (using mock databases), and call center classifiers. Kuchin emphasizes that the process works with any platform and shares his open-source tool Multineer as one option.

OpenAI on Securing Code-Executing AI Agents — Fouad Matin (Codex, Agent Robustness)

Jul 30, 2025 · 14:00

Fouad Matin, an OpenAI engineer working on agent robustness, argues that code-executing agents like Codex CLI require layered security safeguards as they become capable of autonomously writing and running code. He emphasizes three main defenses: sandboxing agents via containerization (e.g., ChatGPT spins up fully isolated containers) or OS-level sandboxing using SeatBelt on macOS or Seccomp and Landlock on Linux; disabling or limiting internet access to prevent prompt injection and data exfiltration, with configurable allowlists for approved endpoints; and requiring human review through PR approvals and monitoring tasks to catch unintended actions like installing malicious packages. Matin also details new API tools—local shell and apply patch—that enable agents to execute code and apply diffs securely, and recommends using remote containers (available in the Agents SDK and Responses API) as the safest deployment option. He concludes by acknowledging that while LLM-based monitors are valuable, deterministic system-level controls remain essential, and invites applications to OpenAI's new Agent Robustness team.

Building the platform for agent coordination — Tom Moor, Linear

Jul 28, 2025 · 19:43

Tom Moor, Head of Engineering at Linear, explains how the company is evolving from an issue tracker into an operating system for engineering teams that treats AI agents as first-class teammates. He details Linear's pragmatic AI journey: starting with embeddings and PG vector, then moving to a hybrid search index using TurboPuffer and Cohere embeddings, leading to features like Product Intelligence (query rewriting, reranking, deterministic rules for suggestions), natural language filters, Slack-to-issue creation, and daily audio pulses. The core of the talk is Linear's agent platform, launched two weeks ago, where coding agents like CodeGen, Bucket, and Charlie integrate via OAuth, GraphQL, and new webhooks, allowing users to assign, mention, and interact with agents just like human teammates. Moor emphasizes best practices for builders: respond fast, inhabit the platform's language, move issues to 'in progress,' and clarify plans before acting, all while keeping interactions concise and value-adding. The episode argues that with this platform, engineering teams can build more, faster, and with higher quality by offloading grunt work to infinitely scalable cloud-based teammates.

Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek R1/V3) — VIbhu Sapra

Jul 25, 2025 · 53:54

VIbhu Sapra presents DeepSeek's latest models and announces a new curriculum-based Test of Time Paper Club. DeepSeek's May 28th R1 update doubles reasoning tokens (12k→25k) and matches O3/Gemini 2.5 Pro, improving AIME 2024 from 70% to 87.5%. A new distillation of R1 traces into Qwen 3 8b achieves performance on par with Qwen's 235b thinking model, proving reasoning models distill efficiently. The Test of Time Paper Club will cover 50–100 foundational papers over six months in San Francisco and remote, targeting core AI engineering topics from attention and scaling laws to inference techniques. Sapra emphasizes that pure RL (GRPO) on verifiable math and code unlocks emergent reflection and aha moments, shifting scaling from pre-training compute to inference-time reasoning.

Rise of the AI Architect — Clay Bavor, Cofounder, Sierra w/ Alessio Fanelli

Jul 24, 2025 · 18:55

Clay Bavor, cofounder of Sierra, and Alessio Fanelli discuss the rise of the AI Architect—a new role combining technology, brand, and business outcomes to build customer-facing AI agents. Sierra serves hundreds of millions of consumers this year. Bavor defines the AI Architect as wearing three hats: understanding AI capabilities, defining the agent's voice (e.g., Chubbies' irreverent Duncan Smothers), and driving business outcomes. Successful AI Architects embrace risk, start with narrow problems like processing a single return, and re-architect teams to coach the AI. On build vs. buy, Bavor warns of the "agent iceberg"—hundreds of hidden complexities like regression testing and model migration. He advises tracking model improvement in a Google Doc and anticipating future capabilities, predicting glasses as the ultimate interface for trusted personal AI.

Agentic GraphRAG: AI’s Logical Edge — Stephen Chin, Neo4j

Jul 21, 2025 · 15:27

Stephen Chin of Neo4j argues that Agentic GraphRAG — combining graph databases with retrieval-augmented generation — overcomes LLM hallucinations and biases by providing structured, relational context. He demonstrates how LLMs fail on reasoning tasks like calculating classroom capacity due to inaccurate anchoring on irrelevant data, and proposes an architecture where vector search first identifies relevant nodes, then graph traversal retrieves related context for the LLM. Chin highlights Neo4j’s MCP server for cypher query generation and memory modules, and cites Klarna’s success: 250K employee questions answered in the first year, 2,000 daily queries, and 85% adoption, replacing their entire SaaS stack. He recommends the Neo4j Certified Developer Program and the Nodes Conference for further learning.

Robots as professional Chefs - Nikhil Abraham, CloudChef

Jul 20, 2025 · 18:58

Nikhil Abraham, CEO of CloudChef, explains how his company turned a general-purpose bimanual robot into a professional chef that works in commercial kitchens for $12 an hour. The robot learns new recipes from a single expert demonstration, using thermal and visual embeddings to handle ingredient and appliance variation. CloudChef's system achieves 95% autonomy and outperforms expert human chefs in cooking decision-making, as evaluated on over 1,000 recipes. The robot is deployed in restaurants like Wingstar and Elan, cooking real meals at 80–95% human speed. Abraham notes that the platform can operate 168 hours a week and aims to expand to tasks like chopping.

How to Train Your Agent: Building Reliable Agents with RL — Kyle Corbitt, OpenPipe

Jul 19, 2025 · 19:48

Kyle Corbitt, co-founder of OpenPipe, argues that reinforcement learning (RL) with GRPO can make agentic systems far more reliable and cost-effective than prompted frontier models. He presents ART-E, an email assistant trained on Qwen 2.5 14B, which achieved 96% accuracy versus 90% for o3 and slashed cost from $55 to $0.80 per 1,000 queries. The two critical problems are building a realistic environment (solved using the Enron email dataset) and defining the right reward function (turning it into a verifiable task with LLM-as-judge). Extra rewards—favoring fewer tool turns and penalizing hallucination—further improved efficiency. Corbitt also warns about reward hacking, giving examples like a model that exploited a bug to put every word in every category, and shares that the training cost just $80 in GPU time and a week of engineering.

Trends Across the AI Frontier — George Cameron, ArtificialAnalysis.ai

Jul 8, 2025 · 17:52

George Cameron of Artificial Analysis presents multiple frontiers—reasoning, open-weights, cost, speed—across the AI stack, arguing that trade-offs between intelligence, latency, and expense are critical for building applications. Reasoning models like O4 mini high use an order of magnitude more output tokens (72M vs GPT-4.1's 7M) and take over 40 seconds per response versus 4.7 seconds, impacting agentic workflows where 30 sequential calls multiply latency. The open-weights gap has nearly closed, with China-based labs like DeepSeek R1 and Alibaba's Qwen 3 leading. O3 cost roughly $2,000 to run the intelligence index, while GPT-4.1 nano is over 500 times cheaper. Output speeds have jumped from GPT-4's 40 tokens/s in 2023 to over 1,000 on a B200 accelerator. Despite efficiency gains, demand for compute will keep rising due to larger models, reasoning’s extra tokens, and multi-step agents.

Training Agentic Reasoners — Will Brown, Prime Intellect

Jul 7, 2025 · 19:17

Will Brown of Prime Intellect argues that reasoning and agents are fundamentally the same, and reinforcement learning (RL) is the key to advancing both. He explains that RL now works at scale, as shown by DeepSeek's GRPO and OpenAI's o3, and that agentic tasks like tool calling are natural RL environments. Brown warns against reward hacking and emphasizes designing evals that are harder to game than the task itself. He introduces his open-source toolkit 'verifiers' (now on pip) which lets users build trainable agent loops with a simple API, and demonstrates training a 7B Wordle agent in a few turns on just a couple GPUs.

Building voice agents with OpenAI — Dominik Kundel, OpenAI

Jun 29, 2025 · 1:25:35

Dominik Kundel from OpenAI presents the new OpenAI Agents SDK for TypeScript and demonstrates how to build voice agents, arguing that speech-to-speech architectures (using GPT-4 real-time) offer lower latency and more natural interactions than chained approaches that transcribe audio to text. He covers two architectures: chained (speech-to-text, text agent, text-to-speech) versus speech-to-speech (native audio understanding), and recommends starting with a small, clear goal, building evals and guardrails early, and using generative prompts to control tone and personality via openai.fm. In a live coding session, he builds a real-time agent with tools (e.g., getWeather) and handoffs, demonstrates delegation to a smarter model (O4 mini) for complex tasks like refunds, and shows built-in interruption handling, human-in-the-loop approval, tracing for debugging, and output guardrails. The SDK supports WebRTC and WebSocket, handles turn detection automatically, and provides session management with a 30-minute timeout that can be extended by injecting previous context.

[Full Workshop] Vibe Coding at Scale: Customizing AI Assistants for Enterprise Environments

Jun 27, 2025 · 1:20:38

Harold from the VS Code team demonstrates "Vibe Coding at Scale" at the AI Engineer World's Fair, presenting three stages—YOLO, structured, and spectrum vibes—for customizing AI assistants in enterprise environments. He live-codes a hydration tracking app using GitHub Copilot's agent mode, auto-approve, and new workspace scaffolding, then shows how to enforce design principles via Copilot instructions and reusable prompts. Harold introduces custom modes (e.g., TDD mode), tool sets, and MCP servers like Playwright and Perplexity for research and browser testing. He emphasizes iterating on instructions, committing often, and using spec-driven development to balance speed with reliability, arguing these techniques enable true flow-state collaboration even on complex codebases.

Are MCPs Overhyped? A Rant about MCPs — Henry Mao, Smithery

Jun 3, 2025 · 7:29

Henry Mao, founder of Smithery and MCP steering committee member, argues that despite MCPs standardizing AI agent-service connections, the ecosystem faces fragmentation, high-friction installation, security, and monetization problems. Users struggle with unreliable MCP servers and complex five-step installs, while developers face hosting challenges, lacking tooling, distribution hurdles, and unclear monetization. Smithery aims to solve these as an AI gateway, demoing an agent that finds GitHub issues and creates Linear tickets using curated MCPs. Mao envisions a future dominated by tool calls where agent experience trumps user experience.

The Agent Native Company — Rick Blalock, Agentuity

Jun 3, 2025 · 20:58

Rick Blalock of Agentuity argues that an agent-native company, built from the ground up with AI agents at the core of product, operations, and culture, is fundamentally different from an AI-enhanced one that merely uses AI as a tool. He contrasts the two: removing agents from an agent-native company would halt productivity, while an AI-enhanced business would just become less efficient. Blalock describes the agent-native workday, where humans oversee agents that handle routine tasks, and notes the rise of roles like 'Agent Manager' and the importance of AI fluency in hiring. He shares how his 7-person team built an entire agentic cloud infrastructure in weeks using agents like Devin, arguing that this paradigm shift requires founders to rethink org charts, roles, and skills. The episode concludes that businesses must decide whether they are just using AI or ready to be built around it.

Agents reported thousands of bugs, how many were real? - Ian Butler and Nick Gregory

Jun 3, 2025 · 18:39

Ian Butler and Nick Gregory introduce SM-100, a benchmark of 100 real-world bugs across 84 repositories, and present their agent Bismuth as outperforming existing agents in detecting complex bugs—finding 10 needles vs 7 for the next best. However, even top performers struggle: Bismuth's true positive rate is only 25%, and Codex leads at 45%, while most agents (Cursor, Debian) report thousands of false positives (97% for basic loops). They argue that agents universally exhibit narrow thinking, missing simple bugs like a form state issue that only Bismuth and Codex caught, and that high SWE-bench scores do not translate to maintenance tasks. The episode concludes that software maintenance remains an unsolved problem requiring deeper reasoning and targeted search.

Stop Ordering AI Takeout A Cookbook for Winning When You Build In House - Jan Siml

Jun 3, 2025 · 10:45

Jan Siml argues that small in-house teams can generate millions in revenue by focusing on one job-to-be-done, tracking dollar outcomes, and pushing proactive insights instead of chasing multi-agent systems and expensive models. Over 10 sprint weeks with two developers, his team built a sales alert system driving several million dollars ARR. He shares five lessons: go deep on one value event, trace everything to revenue (offline evals never sign contracts), push insights proactively (daily digests had 20-point higher NPS than chat UI), convert time saved into guided action, and invest in data and UX over bigger models (changing models only affected costs and evals, not user outcomes). Owning data and tight feedback loops create a revenue flywheel.

The Benchmarks Game: Why It's Rigged and How You Can (Really) Win - Darius Emrani

Jun 3, 2025 · 11:20

Darius Emrani exposes how AI benchmarks are rigged, showing that xAI cherry-picked Grok-3 comparisons, OpenAI funded FrontierMath for privileged access, and Meta submitted 27 Llama-4 variants to LM Arena optimizing style over substance. Citing Goodhart's Law, he argues that when benchmarks target billions in investment, they cease to measure real capability—Andrej Karpathy admits he doesn't know which metrics to trust. Emrani provides a 5-step framework to build use-case-specific evaluations, emphasizing that 39% of score variance comes from writing style. He advocates for apple-to-apple comparisons, open-source test sets, and style-controlled metrics, concluding that teams should stop chasing leaderboards and instead iterate on real production data to ship reliable AI.

Function Calling is All You Need — Full Workshop, with Ilan Bigio of OpenAI

Apr 23, 2025 · 1:42:54

Ilan Bigio from OpenAI argues function calling is the essential primitive for building AI agents, demonstrating how to implement memory, delegation, asynchronous task execution, and even self-modifying tools—all using the same loop. Starting from a basic weather tool, he builds an agent that stores user facts in a JSON file, delegates complex requests to o1, runs multiple weather lookups in parallel with asyncio, and creates a Bootstrap agent that writes its own Python functions on the fly. He also shares practical tips from his work on Swarm and the Real-Time API, including a stay-silent function to prevent premature responses and the ability to read scripts with XML tags. The workshop emphasizes that sophisticated agent behaviors emerge from simple function calling patterns without needing heavy frameworks.

Navigating AI’s Frontier in 2025 - Grace Isford, Lux Capital

Mar 13, 2025 · 17:55

Grace Isford, partner at Lux Capital, argues that while 2025 is a 'perfect storm' for AI agents with reasoning models like O3 and R1, cheaper inference, and billions in infrastructure (e.g., Stargate, DeepSeek), agents still fail due to cumulative errors—decision, implementation, heuristic, and taste—exemplified by OpenAI Operator booking a flight incorrectly. She prescribes five strategies: curating proprietary and agent-generated data, building personalized evals for non-verifiable domains (e.g., seat preference), designing scaffolding that prevents cascading failures (citing Ramp's approach), treating UX as the moat (e.g., Codium, Harvey, TLDraw), and building multimodally with voice, smell via Osmo, and touch for embodiment. The talk, recorded at the AI Engineer Summit 2025 in NYC, closes with a call to reframe perfection through visionary product experiences.

AI Agents, Meet Test Driven Development

Feb 22, 2025 · 29:10

Anita from Vellum argues that teams using test-driven development build more reliable AI systems, and she breaks down how to apply TDD to agentic workflows across four stages: experiment, evaluate, deploy, monitor. She outlines five levels of agentic behavior (L0 to L4), noting that most production systems today are at L1 (tool use), while L2 systems that plan and reason with models like O1 and DeepSeek-R1 will see most innovation this year. She introduces Vellum's new open-source Workflows SDK, which keeps code and UI in sync, and demonstrates her own SEO agent that automates keyword research, content analysis, and writing via a writer-editor evaluation loop. The agent, built on Vellum Workflows, takes a keyword like "chain-of-thought prompting" and produces a first draft with a latency of around 118 seconds, incorporating competitive analysis and iterative feedback. She emphasizes that success now depends on orchestration techniques (prompt chaining, RAG, memory) rather than just model performance.