O3 mini

Fuzzing in the GenAI Era — Leonard Tang, Haize Labs

Aug 22, 2025 · 19:12

Leonard Tang of Haize Labs argues that standard evaluation methods fail for GenAI systems due to brittleness (Lipschitz discontinuity), and proposes 'Haizing'—fuzz testing that simulates diverse inputs to uncover corner cases. He details two core problems: scoring outputs via 'judges', where Haize's Verdict library stacks GPT-4 Mini in a self-verified debate ensemble to beat O1 at a third the cost, and RL-tuned judges like a 1.7B parameter model achieving 80.7% on RewardBench. For stimuli generation, he frames it as discrete optimization over natural language, using gradient-based methods and tree search. Case studies include Haizing the largest Hungarian bank's loan calculator, discovering prompt injections in minutes, and boosting a voice agent's ground-truth human agreement by 38% using rubric fanout.

Rise of the AI Architect — Clay Bavor, Cofounder, Sierra w/ Alessio Fanelli

Jul 24, 2025 · 18:55

Clay Bavor, cofounder of Sierra, and Alessio Fanelli discuss the rise of the AI Architect—a new role combining technology, brand, and business outcomes to build customer-facing AI agents. Sierra serves hundreds of millions of consumers this year. Bavor defines the AI Architect as wearing three hats: understanding AI capabilities, defining the agent's voice (e.g., Chubbies' irreverent Duncan Smothers), and driving business outcomes. Successful AI Architects embrace risk, start with narrow problems like processing a single return, and re-architect teams to coach the AI. On build vs. buy, Bavor warns of the "agent iceberg"—hundreds of hidden complexities like regression testing and model migration. He advises tracking model improvement in a Google Doc and anticipating future capabilities, predicting glasses as the ultimate interface for trusted personal AI.

Building Effective Voice Agents — Toki Sherbakov + Anoop Kotha, OpenAI

Jul 20, 2025 · 17:17

Toki Sherbakov and Anoop Kotha from OpenAI argue that speech-to-speech models have reached a 'good enough' tipping point for production voice agents, highlighting the shift from chained architectures (transcription + LLM + TTS) to the real-time API's speech-to-speech approach. They detail trade-offs across latency, cost, accuracy, UX, and integrations depending on use case—consumer apps prioritize low latency and expressiveness, while customer service demands accuracy and tool integration. Key design patterns include delegating complex tasks to smarter models like O4 mini, prompting for voice expressiveness and tone, and starting with few tools. For evals, they recommend starting with observability, using SMEs for labeling, then transcription-based LLM-as-judge evals, audio evals with GPT-4 Audio to assess tone and pacing, and synthetic conversations between two real-time clients. Guardrails should run async with a configurable debounce period (e.g., every 100 characters). Examples from Lemonade (early focus on evals and guardrails) and Tinder (customization for brand realism) illustrate successful approaches.

The Future of Qwen: A Generalist Agent Model — Junyang Lin, Alibaba Qwen

Jun 3, 2025 · 25:14

Junyang Lin from Alibaba Qwen presents the latest developments in the Qwen model series, including Qwen3's hybrid thinking mode that combines thinking and non-thinking behaviors in a single model, supporting over 119 languages and dialects. The flagship 235B MOE model activates only 22B parameters and competes with top-tier models like O3-mini, while a 4B model deployable on mobile devices rivals Qwen 2.5 72B. Qwen3 introduces dynamic thinking budgets for inference-time scaling, achieving over 80% on AIME 2024 with 32K thinking tokens, and enhances agent capabilities with MCP support. The vision-language model Qwen2.5-VL excels in benchmarks like MMU and MathVista, and the omni model Qwen2.5-Omni accepts text, vision, and audio inputs while generating text and audio, achieving competitive vision-language performance. Future directions include scaling reinforcement learning with environment feedback, extending context to at least 1M tokens, and unifying understanding and generation across modalities, marking a shift from training models to training agents.

Vercel AI SDK Masterclass: From Fundamentals to Deep Research

Apr 20, 2025 · 59:52

Nico Albanese from Vercel delivers a workshop on building agents with the AI SDK, demonstrating how to use generateText, tools, and structured outputs to create a deep research clone in Node.js. He shows that switching between models like GPT-4o mini, Perplexity Sonar Pro, and Gemini Flash 1.5 requires only changing one line of code. Tools with maxSteps enable autonomous multi-step agents, as when the model fetches weather data for San Francisco and New York and sums the temperatures across three steps. The deep research workflow generates subqueries (e.g., 'requirements to become a D1 shotput athlete'), searches via Exa with live crawl, evaluates relevance with an evaluate tool that discards irrelevant results, and recursively deepens research using depth and breadth parameters. Finally, o3 mini synthesizes the accumulated research into a structured Markdown report, all in 218 lines of code.

OpenAI for VP's of AI + Advice for Building Agents

Mar 5, 2025 · 16:52

OpenAI's Toki Sherbakov and Prashant Mital explain how enterprises adopt AI through a three-phase journey: building an AI-enabled workforce with ChatGPT, automating operations with APIs, and infusing AI into end products. They detail a Morgan Stanley case study where retrieval methods improved an internal knowledge assistant's accuracy from 45% to 98%. The pair define agents as models with instructions, tools, and self-terminating execution loops, then share four field lessons: build with primitives before frameworks, start with a single purpose-built agent, graduate to a network of specialized agents with handoffs for complex tasks, and keep prompt instructions simple while running guardrails in parallel using fast models like GPT-4o mini for safety and reliability.