O1

The Golden Age of AI Engineering — Alexander Embiricos & Romain Huet & Peter Steinberger, OpenAI

Jul 9, 2026 · 25:13

OpenAI's Alexander Embiricos, Romain Huet, and Peter Steinberger reveal how AI engineering is entering its golden age, with model releases accelerating from every 15 months to roughly every 6 weeks and frontier-level intelligence now available at $1 per million input tokens and $6 per million output tokens. They demonstrate Codex's evolution from a model that couldn't test its own code to one generating 750 tokens per second, enabling a 10-second pull request, and emphasize that the product is built on the same open-source stack—responses API, AGENTS.md format, and harness—that ships to developers. The trio argue that the future is not about automating engineers but empowering them via value maxing over token maxing, with agents moving from local/cloud distinctions to persistent, long-running managers that orchestrate work and only require human steering at key decision points, as illustrated by an automated open-source issue resolution workflow.

Fuzzing in the GenAI Era — Leonard Tang, Haize Labs

Aug 22, 2025 · 19:12

Leonard Tang of Haize Labs argues that standard evaluation methods fail for GenAI systems due to brittleness (Lipschitz discontinuity), and proposes 'Haizing'—fuzz testing that simulates diverse inputs to uncover corner cases. He details two core problems: scoring outputs via 'judges', where Haize's Verdict library stacks GPT-4 Mini in a self-verified debate ensemble to beat O1 at a third the cost, and RL-tuned judges like a 1.7B parameter model achieving 80.7% on RewardBench. For stimuli generation, he frames it as discrete optimization over natural language, using gradient-based methods and tree search. Case studies include Haizing the largest Hungarian bank's loan calculator, discovering prompt injections in minutes, and boosting a voice agent's ground-truth human agreement by 38% using rubric fanout.

Practical tactics to build reliable AI apps — Dmitry Kuchin, Multinear

Aug 3, 2025 · 14:55

Dmitry Kuchin argues that building reliable AI applications requires reverse-engineering evaluation metrics from real-world user scenarios and business outcomes, rather than relying on abstract data science metrics. He recommends using a strong LLM like o3 to generate specific user questions and criteria from knowledge materials, and building these evaluations at the start of development, not the end. This approach lets teams run frequent experiments, catch regressions, and iterate until they have a baseline benchmark. Examples include customer support bots (where the key metric is escalation rate to humans), text-to-SQL (using mock databases), and call center classifiers. Kuchin emphasizes that the process works with any platform and shares his open-source tool Multineer as one option.

OpenAI on Securing Code-Executing AI Agents — Fouad Matin (Codex, Agent Robustness)

Jul 30, 2025 · 14:00

Fouad Matin, an OpenAI engineer working on agent robustness, argues that code-executing agents like Codex CLI require layered security safeguards as they become capable of autonomously writing and running code. He emphasizes three main defenses: sandboxing agents via containerization (e.g., ChatGPT spins up fully isolated containers) or OS-level sandboxing using SeatBelt on macOS or Seccomp and Landlock on Linux; disabling or limiting internet access to prevent prompt injection and data exfiltration, with configurable allowlists for approved endpoints; and requiring human review through PR approvals and monitoring tasks to catch unintended actions like installing malicious packages. Matin also details new API tools—local shell and apply patch—that enable agents to execute code and apply diffs securely, and recommends using remote containers (available in the Agents SDK and Responses API) as the safest deployment option. He concludes by acknowledging that while LLM-based monitors are valuable, deterministic system-level controls remain essential, and invites applications to OpenAI's new Agent Robustness team.

Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek R1/V3) — VIbhu Sapra

Jul 25, 2025 · 53:54

VIbhu Sapra presents DeepSeek's latest models and announces a new curriculum-based Test of Time Paper Club. DeepSeek's May 28th R1 update doubles reasoning tokens (12k→25k) and matches O3/Gemini 2.5 Pro, improving AIME 2024 from 70% to 87.5%. A new distillation of R1 traces into Qwen 3 8b achieves performance on par with Qwen's 235b thinking model, proving reasoning models distill efficiently. The Test of Time Paper Club will cover 50–100 foundational papers over six months in San Francisco and remote, targeting core AI engineering topics from attention and scaling laws to inference techniques. Sapra emphasizes that pure RL (GRPO) on verifiable math and code unlocks emergent reflection and aha moments, shifting scaling from pre-training compute to inference-time reasoning.

Rise of the AI Architect — Clay Bavor, Cofounder, Sierra w/ Alessio Fanelli

Jul 24, 2025 · 18:55

Clay Bavor, cofounder of Sierra, and Alessio Fanelli discuss the rise of the AI Architect—a new role combining technology, brand, and business outcomes to build customer-facing AI agents. Sierra serves hundreds of millions of consumers this year. Bavor defines the AI Architect as wearing three hats: understanding AI capabilities, defining the agent's voice (e.g., Chubbies' irreverent Duncan Smothers), and driving business outcomes. Successful AI Architects embrace risk, start with narrow problems like processing a single return, and re-architect teams to coach the AI. On build vs. buy, Bavor warns of the "agent iceberg"—hundreds of hidden complexities like regression testing and model migration. He advises tracking model improvement in a Google Doc and anticipating future capabilities, predicting glasses as the ultimate interface for trusted personal AI.

Building Agents (the hard parts!) - Rita Kozlov, Cloudflare

Jul 23, 2025 · 21:12

Rita Kozlov, VP of Product for Cloudflare's developer platform, presents the building blocks of AI agents—client, AI reasoning, workflows, and tools—arguing that effective agents require all four components. She highlights the Model Context Protocol (MCP) as a standard for exposing APIs to LLMs, and demonstrates Cloudflare's Agents SDK, which simplifies hosting remote MCP servers with built-in OAuth, state management via durable objects, and real-time WebSocket communication. Kozlov cites real-world impact: companies using agents for sales automation see 20% revenue increases, 90% faster support response times, and 50–75% time savings. She walks through a human-in-the-loop credit card approval workflow built with Nok, showing how durable objects maintain long-running state, prevent duplicate actions, and route approvals across Slack, email, or in-app notifications. The talk emphasizes that once an MCP server is deployed, it can be used directly from Cursor, Claude, ChatGPT, or a custom client, including voice interfaces via WebRTC-to-WebSocket translation.

Trends Across the AI Frontier — George Cameron, ArtificialAnalysis.ai

Jul 8, 2025 · 17:52

George Cameron of Artificial Analysis presents multiple frontiers—reasoning, open-weights, cost, speed—across the AI stack, arguing that trade-offs between intelligence, latency, and expense are critical for building applications. Reasoning models like O4 mini high use an order of magnitude more output tokens (72M vs GPT-4.1's 7M) and take over 40 seconds per response versus 4.7 seconds, impacting agentic workflows where 30 sequential calls multiply latency. The open-weights gap has nearly closed, with China-based labs like DeepSeek R1 and Alibaba's Qwen 3 leading. O3 cost roughly $2,000 to run the intelligence index, while GPT-4.1 nano is over 500 times cheaper. Output speeds have jumped from GPT-4's 40 tokens/s in 2023 to over 1,000 on a B200 accelerator. Despite efficiency gains, demand for compute will keep rising due to larger models, reasoning’s extra tokens, and multi-step agents.

The Future of Qwen: A Generalist Agent Model — Junyang Lin, Alibaba Qwen

Jun 3, 2025 · 25:14

Junyang Lin from Alibaba Qwen presents the latest developments in the Qwen model series, including Qwen3's hybrid thinking mode that combines thinking and non-thinking behaviors in a single model, supporting over 119 languages and dialects. The flagship 235B MOE model activates only 22B parameters and competes with top-tier models like O3-mini, while a 4B model deployable on mobile devices rivals Qwen 2.5 72B. Qwen3 introduces dynamic thinking budgets for inference-time scaling, achieving over 80% on AIME 2024 with 32K thinking tokens, and enhances agent capabilities with MCP support. The vision-language model Qwen2.5-VL excels in benchmarks like MMU and MathVista, and the omni model Qwen2.5-Omni accepts text, vision, and audio inputs while generating text and audio, achieving competitive vision-language performance. Future directions include scaling reinforcement learning with environment feedback, extending context to at least 1M tokens, and unifying understanding and generation across modalities, marking a shift from training models to training agents.

Creating Agents that Co-Create — Karina Nguyen, OpenAI

Apr 30, 2025 · 24:22

Karina Nguyen, an AI researcher at OpenAI and former Anthropic researcher, discusses two major AI scaling paradigms—next-token prediction (pretraining) and reinforcement learning on chain of thought—that shift AI from narrow tools to collaborative agents that co-create. Pretraining builds world understanding by predicting the next token, but hard tasks like math and creative writing require chain of thought reasoning, scaled with OpenAI’s O1 model. Post-training via RLHF and synthetic data enables rapid iteration, and the next stage is co-innovators: agents with reasoning, tool use, long context, and creativity. Nguyen shares product lessons from developing ChatGPT and Claude: 100K context via file uploads, ChatGPT Tasks for scheduled reminders that scale with model capabilities, and Canvas as a flexible interface for co-writing, coding, and research that can morph into an IDE, a tutor, or a data scientist. She envisions a future of invisible software creation where AI generates personalized, multimodal outputs on the fly, reducing reliance on clicking links, and the interface becomes a blank canvas that adapts to user intent.

Function Calling is All You Need — Full Workshop, with Ilan Bigio of OpenAI

Apr 23, 2025 · 1:42:54

Ilan Bigio from OpenAI argues function calling is the essential primitive for building AI agents, demonstrating how to implement memory, delegation, asynchronous task execution, and even self-modifying tools—all using the same loop. Starting from a basic weather tool, he builds an agent that stores user facts in a JSON file, delegates complex requests to o1, runs multiple weather lookups in parallel with asyncio, and creates a Bootstrap agent that writes its own Python functions on the fly. He also shares practical tips from his work on Swarm and the Real-Time API, including a stay-silent function to prevent premature responses and the ability to read scripts with XML tags. The workshop emphasizes that sophisticated agent behaviors emerge from simple function calling patterns without needing heavy frameworks.

Building and evaluating AI Agents — Sayash Kapoor, AI Snake Oil

Apr 17, 2025 · 20:00

Sayash Kapoor argues that current AI agents fall far short of their claimed performance due to flawed evaluation and a gap between capability and reliability. He cites failures like Do Not Pay (fined by FTC), LexisNexis (hallucinations in up to a third of cases), and Sakana AI (agent hacked reward functions, claiming 150x speedup that exceeded H100's theoretical max). Princeton's CoreBench shows best agents reproduce under 40% of papers. He emphasizes that agent benchmarks like SWE-bench mislead VC funding—Cognition's Devin succeeded on only 3 of 20 real-world tasks. Kapoor calls for cost-aware, multi-dimensional evaluation (e.g., Holistic Agent Leaderboard with Pareto frontiers) and a shift from capability to reliability engineering, drawing parallels to ENIAC's vacuum tube failures.

Trust, but Verify: Knowledge Agents for Finance Workflows - Mike Conover

Apr 9, 2025 · 21:10

Mike Conover, CEO of Brightwave, explains how his company builds knowledge agents for financial workflows, digesting thousands of pages of content for due diligence and research. He argues that non-reasoning models perform only local search, so winning systems must use end-to-end reinforcement learning over tool-use calls to achieve globally optimal outputs. Conover describes design patterns like decomposing research into sub-themes, using secondary calls for error correction, and avoiding the latency trap of long feedback loops. He emphasizes that synthesis—weaving facts across documents—remains a hard problem due to limited recombinative reasoning in training data. Brightwave's product reveals its thought process through interactive citations and structured findings, allowing analysts to drill into any passage. The episode also notes the company is hiring, offering a $10,000 referral bonus.

Why Agent Engineering — swyx

Mar 24, 2025 · 11:45

Swyx argues that 2025 is the year of agent engineering, pivoting the AI Engineer Summit to focus exclusively on agents and explaining why agents are now viable due to advancements in reasoning, tool use, model diversity, and a 1000x cost reduction in GPT-4-level intelligence over 18 months. He highlights that OpenAI's ChatGPT grew 33% to 400M users in three months after shipping agentic O1 models, and projects it will reach 1B users by year-end. Swyx cites Simon Willison's crowdsourced definitions and OpenAI's new agent definition, while calling for a halt to demos of flight booking agents and Reddit astroturfing. He frames agent engineering as the evolution of AI engineering, distinct from MLE and software engineering, with practical use cases like coding and support agents having clear product-market fit.

Navigating AI’s Frontier in 2025 - Grace Isford, Lux Capital

Mar 13, 2025 · 17:55

Grace Isford, partner at Lux Capital, argues that while 2025 is a 'perfect storm' for AI agents with reasoning models like O3 and R1, cheaper inference, and billions in infrastructure (e.g., Stargate, DeepSeek), agents still fail due to cumulative errors—decision, implementation, heuristic, and taste—exemplified by OpenAI Operator booking a flight incorrectly. She prescribes five strategies: curating proprietary and agent-generated data, building personalized evals for non-verifiable domains (e.g., seat preference), designing scaffolding that prevents cascading failures (citing Ramp's approach), treating UX as the moat (e.g., Codium, Harvey, TLDraw), and building multimodally with voice, smell via Osmo, and touch for embodiment. The talk, recorded at the AI Engineer Summit 2025 in NYC, closes with a call to reframe perfection through visionary product experiences.

AI Agents, Meet Test Driven Development

Feb 22, 2025 · 29:10

Anita from Vellum argues that teams using test-driven development build more reliable AI systems, and she breaks down how to apply TDD to agentic workflows across four stages: experiment, evaluate, deploy, monitor. She outlines five levels of agentic behavior (L0 to L4), noting that most production systems today are at L1 (tool use), while L2 systems that plan and reason with models like O1 and DeepSeek-R1 will see most innovation this year. She introduces Vellum's new open-source Workflows SDK, which keeps code and UI in sync, and demonstrates her own SEO agent that automates keyword research, content analysis, and writing via a writer-editor evaluation loop. The agent, built on Vellum Workflows, takes a keyword like "chain-of-thought prompting" and produces a first draft with a latency of around 118 seconds, incorporating competitive analysis and iterative feedback. She emphasizes that success now depends on orchestration techniques (prompt chaining, RAG, memory) rather than just model performance.

The Model Isn’t Wrong—You’re Just Bad at Prompting

Feb 22, 2025 · 8:54

Dan from PromptHub argues that prompt engineering remains critical for improving LLM outputs, covering Chain of Thought, few-shot, and meta prompting techniques. Chain of Thought breaks problems into sub-problems and is built into reasoning models; few-shot prompting works best with just one or two diverse examples, but can degrade performance on reasoning models like O1 and R1. Meta prompting uses LLMs to write or refine prompts, with PromptHub offering model-specific enhancers. For reasoning models, Dan advises minimal prompting, encouraging more reasoning instead of few-shot, and avoiding instructing the model on how to reason. Free resources include PromptHub's templates, the AutoReason prompt, and the Prompt Engineering Substack.

Building Multi agent Systems with Finite State Machines

Feb 22, 2025 · 17:11

Adam Terlson argues that finite state machines (state charts) combined with the Actor model provide a structured, reliable foundation for building AI agents, addressing the predictability, observability, and control that LLMs lack. He introduces five agentic patterns—tool use, human-in-the-loop, feedback, collaboration, orchestration, and chartering—each demonstrated with concrete examples like a recipe-writing agent that checks inventory, a two-agent feedback loop for nutritional critique, and a multi-agent team delivering a meal. For orchestration, an LLM directs state transitions via event emitters, while chartering lets the LLM generate the entire state chart from a privacy-removal process. Terlson uses XState library for implementations and points to his demo 'Red versus Blue' for further exploration.

The LLM Triangle: Engineering Principles for Robust AI Applications - Almog Baku:

Feb 22, 2025 · 26:19

Almog Baku introduces the LLM Triangle Principles for building production-ready AI applications, arguing that LLMs should be treated like interns guided by Standard Operating Procedures (SOPs). He breaks down the triangle into three components: the foundation model, engineering techniques, and contextual data, all directed by SOPs derived from expert interviews. Baku contrasts autonomous agents—creative but unpredictable—with handcrafted LLM-native architectures that offer sustainable quality, recommending scoped autonomy. He advises starting with large models to collect data and optimize incrementally, rather than fine-tuning from day one. Data is paramount: 'show, don't tell' via few-shot examples, and balance context to avoid the 'needles in a haystack' problem. The framework aims to turn demos into robust production systems.