GPT-4o

Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek R1/V3) — VIbhu Sapra

Jul 25, 2025 · 53:54

VIbhu Sapra presents DeepSeek's latest models and announces a new curriculum-based Test of Time Paper Club. DeepSeek's May 28th R1 update doubles reasoning tokens (12k→25k) and matches O3/Gemini 2.5 Pro, improving AIME 2024 from 70% to 87.5%. A new distillation of R1 traces into Qwen 3 8b achieves performance on par with Qwen's 235b thinking model, proving reasoning models distill efficiently. The Test of Time Paper Club will cover 50–100 foundational papers over six months in San Francisco and remote, targeting core AI engineering topics from attention and scaling laws to inference techniques. Sapra emphasizes that pure RL (GRPO) on verifiable math and code unlocks emergent reflection and aha moments, shifting scaling from pre-training compute to inference-time reasoning.

From Mixture of Experts to Mixture of Agents with Super Fast Inference - Daniel Kim & Daria Soboleva

Jun 27, 2025 · 53:15

Daria Soboleva and Daniel Kim of Cerebras explain how Mixture of Experts (MoE) architectures enable scaling large language models efficiently by replacing monolithic feedforward networks with specialized experts, a technique used by GPT-4 and Claude. They then introduce Mixture of Agents (MoA), which combines multiple LLMs with custom prompts to outperform frontier models like GPT-4o on complex tasks, reducing a 293-second reasoning problem to 7.4 seconds using Cerebras' ultra-fast inference. The workshop guides participants to build their own MoA system, configure agents for bug fixing and performance optimization on a Python function, and achieve scores up to 120/120. Daniel details Cerebras' wafer-scale chip with 900,000 cores and distributed memory that eliminates memory bandwidth bottlenecks, enabling linear scaling and 15.5x faster inference on Llama 3.3-70B versus GPUs. Daria discusses ongoing research in diffusion models and sparsity, while Daniel notes plans for multimodal APIs and LoRA fine-tuning support.

Vercel AI SDK Masterclass: From Fundamentals to Deep Research

Apr 20, 2025 · 59:52

Nico Albanese from Vercel delivers a workshop on building agents with the AI SDK, demonstrating how to use generateText, tools, and structured outputs to create a deep research clone in Node.js. He shows that switching between models like GPT-4o mini, Perplexity Sonar Pro, and Gemini Flash 1.5 requires only changing one line of code. Tools with maxSteps enable autonomous multi-step agents, as when the model fetches weather data for San Francisco and New York and sums the temperatures across three steps. The deep research workflow generates subqueries (e.g., 'requirements to become a D1 shotput athlete'), searches via Exa with live crawl, evaluates relevance with an evaluate tool that discards irrelevant results, and recursively deepens research using depth and breadth parameters. Finally, o3 mini synthesizes the accumulated research into a structured Markdown report, all in 218 lines of code.

Finetuning: 500m AI agents in production with 2 engineers — Mustafa Ali & Kyle Corbitt

Apr 12, 2025 · 18:44

Method Financial and OpenPipe detail how Method scaled AI financial agents to 500 million daily interactions using fine-tuned open-source models instead of expensive GPT-4. After racking up $70,000 in monthly GPT-4 costs and facing latency and error issues, they fine-tuned an 8B parameter LLaMA 3.1 model to achieve under 200ms latency and 9% error rate (beating GPT-4o's 11% at lower cost). Mustafa Ali explains that the key was using production data from GPT as training data and choosing the cheapest model that met performance goals. Kyle Corbitt emphasizes that fine-tuning is a power tool for bending the price-performance curve when prompt engineering falls short. The episode argues that productionizing AI agents requires patience and openness from engineering teams, and concludes with a call for software engineers to pivot to AI engineering.

How Coding Agents change Software Development Forever - Hailong Zhang

Feb 22, 2025 · 8:50

Hailong Zhang presents how coding agents, particularly the unit test agent Gru, transform software development by automating routine tasks while humans focus on creative work. In the future workflow, synchronous agents like GitHub Copilot and Cursor assist in real-time, while asynchronous agents like Gru autonomously handle tasks such as writing tests, fixing bugs, and submitting pull requests. Gru, built to boost unit tests, detects code changes from pull requests, generates and runs tests, and submits a new PR with a summary and coverage improvement. In production, over 50% of Gru's PRs are accepted by humans, and it handles 80% of unit tests in its own repo, making it the top contributor. To build such agents, Zhang emphasizes defining a clear problem (e.g., unit tests, not generic software engineering), creating datasets and evaluation harnesses, selecting and fine-tuning LLMs per stage, building task-specific context from environment data, and developing an agent operating system to reuse common infrastructure across different tasks.

Patrick Dougherty: How to Build AI Agents that Actually Work

Feb 22, 2025 · 17:44

Patrick Dougherty, co-founder and CTO of Rosco (acquired by Klarity in 2024), argues that building effective AI agents requires prioritizing reasoning over inherent model knowledge, iterating on the Agent-Computer Interface (ACI) rather than fine-tuning or relying on frameworks. He defines an agent as a system that takes directions, calls tools, and autonomously reasons—not a prompt chain. His team abandoned RAG-style knowledge injection for discrete tool calls (e.g., SEARCH TABLES, GET TABLE DETAIL), finding that GPT-4o often hallucinates impossible SQL queries while o1 correctly identifies missing data. Switching tool-response formats—Markdown to JSON for GPT-4o, JSON to XML for Claude—dramatically improved accuracy. For production, he warns against over-dependence on abstractions like LangGraph due to security needs (e.g., OAuth credential cascading). In multi-agent systems, a manager agent should delegate to 5–8 specialized worker agents, incentivized by the overall goal, to avoid loops. The true moat, he concludes, is the ecosystem—UX, connections, and security—not system prompts.

Personality Driven Development: Exploring the Frontier of Agents with Attitude

Feb 17, 2025 · 18:00

Ben, from Perpetual, argues that anthropomorphizing AI agents—giving them personalities, forms, and preferences—transforms user adoption and engagement, despite significant challenges. He presents examples like Tech Lead, an artichoke recruiter, and hamster teams, showing how form factors create instant understanding and enable price anchoring (e.g., 1/20th the cost of a junior employee). Benefits include easy problem decomposition (specialized agents outperform generalized), branding, and fun. However, downsides include reinforcing stereotypes (100% of generated software engineers appear male), raised performance expectations, difficulty rebranding, distraction from core value, and stark reminders of job replacement—illustrated by a CEO introducing an IC as the one who 'knows what they're doing.' Ben highlights personality-driven development where every instance is bespoke, with prompt-driven preferences (e.g., 'I like Ivy League schools'), making debugging uniquely challenging.

System Design for Next-Gen Frontier Models — Dylan Patel, SemiAnalysis

Feb 11, 2025 · 18:29

Dylan Patel of SemiAnalysis breaks down the inference challenges for next-generation frontier models like GPT-4 (1.8 trillion parameters) and upcoming models trained on 100,000+ GPU clusters. He emphasizes that prefill (prompt processing) is compute-intensive while decode (token generation) is memory bandwidth-intensive, creating a systems problem where serving 64 users at 30 tokens/second requires 60 terabytes/second of memory bandwidth. Patel details engineering strategies such as continuous batching to improve batch utilization by 10-100x, disaggregated prefill to isolate noisy neighbors and maintain time-to-first-token SLAs, and context caching (like Google's) to cache KV cache on CPU/storage instead of GPU memory, dramatically reducing prefill costs. He warns that open-source tools like LLaMA.cpp lack these optimizations, making high-performance serving of models like LLaMA 405b infeasible without libraries like vLLM or TensorRT-LLM. On scaling, Patel notes that 100,000 GPU clusters (e.g., Microsoft's Arizona data center consuming 150 MW) face reliability issues — optical transceivers fail every five minutes — and straggler chips (silicon lottery) can degrade training…

Accelerate your AI journey with Azure AI model catalog: Sharmila Chokalingam

Feb 6, 2025 · 23:14

Shubhi and Sharmila present the Azure AI model catalog as a platform offering over 1,600 models including GPT-4, Mistral, Llama, Cohere, and Phi3, with a standardized inference API enabling easy model swapping. They demonstrate deployment via serverless API (pay-per-token) and managed compute, emphasizing that customer prompts and completions are not shared with model providers or used for training. The platform includes model benchmarks, playground for RAG, and Prompt Flow for building generative AI apps with evaluation and variant comparison. Customer success stories include EY's EYQ chat adopted by 275,000 employees, CMA CGM's reduced response latency using Mistral, and Bridgestone's 30% reduction in forecasting errors with Nixtla's TimeGen.

Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

Nov 1, 2024 · 1:39:52

In this workshop, Zach Blumenthal of Neo4j demonstrates how to build a GraphRAG application using Neo4j, OpenAI, and LangChain on an H&M fashion dataset. He argues that combining knowledge graphs with vector search and graph embeddings improves retrieval and personalization for LLMs. The session covers creating a Neo4j sandbox, loading data, performing vector search with OpenAI embeddings, adding collaborative filtering via graph traversal patterns, using graph data science (FastRP) to generate graph embeddings for recommendations, and integrating everything into a LangChain chain that generates personalized marketing emails. Attendees build a Gradio app that, given a customer ID and season, outputs a tailored email with product recommendations, demonstrating how structured graph data enhances RAG systems beyond pure vector search.

Building and Scaling an AI Agent Swarm of low latency real time voice bots: Damien Murphy

Oct 8, 2024 · 1:07:23

Damien Murphy, Senior Applied Engineer at Deepgram, demonstrates building and scaling low-latency real-time voice bots using Deepgram's new voice agent API, which wraps speech-to-text, LLM, and text-to-speech into a single streaming endpoint. He shows a drive-thru ordering demo with function calling (add/remove items) using GPT-4o, achieving sub-second latency by co-locating components. Murphy explains scaling to millions of concurrent calls through regional Kubernetes clusters and multi-agent swarms (routing, booking, support agents) to reduce complexity and cost. He addresses endpointing challenges, VAD-based barge-in, and cost/quality trade-offs with hosted vs. self-hosted models, noting Deepgram offers 20x cheaper TTS than ElevenLabs and 50ms STT latency self-hosted. The talk emphasizes keeping agents simple, using smallest capable LLMs, and composability for reuse.

Lessons From A Year Building With LLMs

Jul 19, 2024 · 35:21

The six authors of the O'Reilly article "Lessons From A Year Building With LLMs" — Bryan Bischof, Jason Liu, Hamel Husain, Eugene Yan, Shreya Shankar, and Swix — argue that the model itself is not a moat and that success comes from continuous improvement centered on evals and data. They stress that AI engineers should treat models as SaaS, quickly swapping for better ones, and focus on product and user interactions. The talk warns against toxic practices like prematurely hiring ML engineers without data or blindly adopting tools, advocating instead for deliberate eval practice and data literacy. On tactics, they compare LLM-as-judge (quick to prototype) vs fine-tuned evaluators (more precise and faster), and emphasize looking at real user data regularly with automated guardrails. Ultimately, they conclude that going from demo to production requires sustained investment in infrastructure and evaluation, echoing MLOps lessons from a decade ago.

Open Challenges for AI Engineering: Simon Willison

Jul 17, 2024 · 18:49

Simon Willison argues the GPT-4 barrier has been broken as GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and open models like LLaMA 3 70B now compete, making GPT-4-class models a commodity. He highlights the AI trust crisis with examples of Dropbox and Slack being falsely accused of training on user data, and notes Anthropic trained Claude 3.5 Sonnet without customer data. Willison warns about prompt injection vulnerabilities, citing the Markdown image exfiltration bug affecting six major chatbots, and defines slop as unreviewed AI-generated content, calling for accountability and responsible use patterns.

From Text to Vision to Voice Exploring Multimodality with Open AI: Romain Huet

Jul 10, 2024 · 23:39

OpenAI's Romain Huet demonstrates GPT-4o's real-time audio, vision, and screen-sharing capabilities at the AI Engineer World's Fair, showing how the omni-model achieves human-like conversational latency, understands emotion, and can be interrupted naturally. He walks through live demos of ChatGPT Desktop recognizing drawings, reading book pages, and pair-coding a responsive travel app using Tailwind CSS. Huet also previews Sora's text-to-video generation and OpenAI's unreleased Voice Engine, which can clone a voice from seconds of audio and translate it into multiple languages while preserving the original speaker's tone. He outlines OpenAI's four focus areas: advancing textual intelligence, faster/cheaper models (GPT-4o is twice as fast as GPT-4 Turbo at half the price), model customization via fine-tuning, and enabling multimodal agents. The talk emphasizes that today's models are the dumbest they'll ever be and urges developers to build for a future where AI works across text, voice, and video.