Voice Agents

Don't Let the LLM Drive - Ornella Bahidika & Joel Allou, Microsoft

Jul 20, 2026 · 6:08

Ornella Bahidika and Joel Allou of Microsoft present their voice tutor Ace, arguing that the LLM should never control the flow in multi-step agents. They built Ace with a state machine that confines the model to narrow contracts per step, letting the harness validate outputs and decide next actions. This design allows them to use a cheaper, faster model (Haiku 4.5) instead of a heavy reasoning model (Opus 4.7 Cloud). They identify three decisions the LLM must never own: when the lesson ends, whether the student answered correctly, and what comes next. By engineering these checks outside the model, Ace achieves reliability in production, avoiding loops and early terminations that plague prompted-only approaches. The pattern applies to any flow agent, including coding agents, runbooks, and onboarding flows.

Your Voice Agent Doesn't Need a Frontier Model - Joel Allou & Ornella Bahidika, Microsoft

Jul 20, 2026 · 5:45

Joel Allou and Ornella Bahidika from Microsoft present ACE, an AI voice tutor that deliberately uses a small model instead of a frontier model, arguing that latency—not intelligence—is the critical constraint in voice applications. They demonstrate that a frontier model's reasoning time of multiple seconds breaks conversational flow, while their system keeps model latency under 950 milliseconds by extracting all logic and planning into a deterministic state machine. This scaffolding handles lesson progression, student mastery tracking, and response generation, leaving the small model (Haiku 4.5) with only the task of speaking. The result: a 900-millisecond response time that feels instantaneous. They acknowledge that scaffolding requires strict rules to prevent drift, but call it a one-time code investment that unlocks cost-effective, real-time performance. The episode’s core lesson: pick the fastest model your latency budget allows, then invest in external scaffolding to make it smart.

Voice Agents That Handle Interrupts - Chintan Agrawal and Daniel Wirjo, AWS

Jul 20, 2026 · 32:57

AWS Solutions Architects Chintan Agrawal and Daniel Wirjo argue that the hardest problem in production voice agents is audio engineering—not AI—specifically turn-taking, the decision of when an agent should stop speaking or start responding. They present three levels of turn detection: Level 1 uses Silero VAD with a silence timeout (e.g., 300ms default), Level 2 delegates to STT providers like Cartesia or Deepgram for built-in endpointing (p50 ~250-300ms), and Level 3 combines Silero VAD with SmartTurn, an open-source 8MB model achieving 58.9% recall and 68.4% precision while falling back to VAD on low confidence. They show how interruption handling (barge-in) flushes TTS/LLM in ~15ms and distinguish real interruptions from backchannel acknowledgments. Latency budgets are tight: 40ms mic encoding, 52ms network/jitter, 300ms STT+endpointing, 500-650ms LLM time-to-first-byte (dominant bottleneck), and 120-190ms TTS playback, totaling 800-1300ms in standard cloud setups. Co-locating models in one GPU cluster can achieve ~500ms voice-to-voice. For LLMs, Nemotron 3 Ultra and GPT-4.1 achieve ~530ms p50 but GPT-4.1 spikes to 1.7s p95, and multi-turn drift (>15 turns) can break prompt…

The Prompt Is Still a Punch Card - Ted Johnson, JoinIn AI

Jul 2, 2026 · 20:13

Ted Johnson argues that AI interfaces still use the batch protocol of punch cards, forcing humans to adapt to machines rather than the reverse. He introduces three concepts—channel, expression, protocol—to show how expression exploded with LLMs but the protocol remained static. Examples include voice mode misinterpreting a side conversation, PersonaPlex's turn-taking, and a meeting AI that follows group dynamics without prompts. Johnson calls for designing interfaces where AI understands timing, ambiguity, and shared context, moving from prompting to genuine communication.

Voice In, Visuals Out: The Agony and the Ecstasy - Allen Pike, Forestwalk Labs

Jun 28, 2026 · 13:05

Allen Pike, co-founder of Forestwalk Labs, argues that voice is the preferred input for AI but visuals are the preferred output, as Andrej Karpathy recently claimed. He explains that while voice-in, voice-out requires sub-200ms latency for seamless conversation, voice-in, visuals-out can work within a more forgiving one-second response envelope. Pike shares three techniques his team uses to achieve this: using a fast model like Haiku with low p95 latency, sending inference every one to two seconds as the user speaks rather than waiting for silence, and leveraging stable prefix caching to keep the first 90% of the context identical, reducing cost and latency. He illustrates with a real example where a voice agent filed a Linear issue in under a second during a team call, demonstrating how the approach creates a natural, non-interruptive experience.

Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI

May 31, 2026 · 24:35

Rishabh Bhargava from Together AI outlines engineering voice agents, explaining that pipeline architecture with colocated models can achieve sub-500ms response times critical for user retention. Speech-to-text targets P90 under 100ms and 6% word error rate, while the LLM must stay within 200-300ms time-to-first-token using 8-30B parameter models—larger models blow the budget, smaller ones break tool calling. Network latency from distant data centers adds 75ms (30% overhead) versus 5ms when colocated in the same building. Pure speech-to-speech models are emerging but still struggle with instruction following and tool calling. The thinker-talker pattern uses a small LLM for fast conversational flow and issues a single tool call to a larger model for complex requests.

Reachy Mini: the $300 open source robot you can actually hack — Andres Marafioti, Hugging Face

May 29, 2026 · 21:16

Andres Marafioti from Hugging Face presents Reachy Mini, a $300 open source robot shipped to 7,500 people unassembled, and explains that its most popular app is voice conversation. To make that app responsive, his team optimized Qwen3-TTS from a below-real-time 0.8x factor to 5.8x real time by adding streaming, switching to a static KV cache, and enabling CUDA graph captures, cutting time to first audio to under 200 milliseconds. The full voice pipeline runs Parakeet transcription every 150 milliseconds with partial results, feeds Qwen 3.5 27B for the LLM, and uses the optimized TTS, while infrastructure round trips are handled by a load balancer that separates LLM endpoints from conversation nodes. Marafioti argues that expensive humanoid robots limit creativity and access, whereas Reachy Mini's hackable, repairable design invites hackers, students, and dreamers to build new interactions. He demonstrates the robot's ability to take photos, show emotions, and even be pet, and emphasizes that all models and software are open source.

Why MLX — Prince Canuma, Neywa Labs

May 11, 2026 · 23:10

Prince Canuma argues that on-device AI, powered by Apple's MLX framework, is a viable alternative to cloud-dependent services, especially for users in regions with unreliable internet. MLX, an array framework for Apple Silicon, has reached 1.5 million downloads and 4000 ported models, including day-zero support for Gemma 4 and Qwen 3 Omni. Canuma demonstrates real-time vision models (e.g., RF Deter for object detection), sub-100ms text-to-speech via Marvis TTS, and modular speech-to-speech pipelines that run entirely on-device. Community projects showcased include a native voice app (Locally), a robot with real-time voice cloning, and a video generation system that chains coherent stories on 16GB VRAM. A recent breakthrough, Turbo Quant, reduces KV cache by 4x, enabling 1 million context windows on-device. Canuma emphasizes that these capabilities provide accessibility for his blind father and enable agents that hear, see, and respond without phoning home.

Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral

May 9, 2026 · 22:26

Mistral AI scientist Samuel Humeau explains why text-to-speech models now resemble LLMs, using an autoregressive transformer with neural audio codecs to compress 200 kbps audio into ~500 tokens per second. He demos Mistral's open-weight TTS model, cloning a voice from a few seconds of reference audio and generating speech with 17 ms first-token latency via frame-by-frame streaming. The talk covers the codec-backbone-decoder pipeline, the model's 4B-parameter backbone generating 37 tokens per 80 ms frame via a diffusion decoder, and a live voice agent answering conference schedule questions. Humeau notes the next latency win is handling streaming text input from an LLM, with no settled architecture yet, while keeping the encoder for voice cloning proprietary.

Voice AI: when is the "Her" moment? — Neil Zeghidour, CEO, Gradium AI

May 9, 2026 · 19:27

Neil Zeghidour, CEO of Gradium AI, argues that voice AI remains far from the 'Her' ideal because cascaded systems (speech-to-text, LLM, text-to-speech) suffer from high latency—tool calls alone add 500ms to 4 seconds—while human response time is ~200ms. Speech-to-speech models reduce latency but are half-duplex, meaning they cannot handle overlapping speech or backchanneling, unlike Moshi, Gradium's full-duplex model. However, Moshi lacked intelligence, tool calls, and paralinguistic understanding—the ability to infer tone, hesitation, or discomfort from voice, which is stripped away in text. Cost is another barrier: TTS bills burn through fundraising before user bases grow. Gradium's solution is Phonon, an on-device TTS model running on smartphone CPUs, offering privacy and eliminating API fees. The path forward requires combining full-duplex natural conversation with the reliability and smarts of cascaded systems.

Give Your Chat Agent a Voice — Luke Harries, Head of Growth, ElevenLabs

May 9, 2026 · 8:12

Luke Harries from ElevenLabs argues that the next upgrade for chat agents is a voice layer, not smarter prompts or RAG, and introduces the company's Voice Engine to wrap existing agents via a few lines of code. He demonstrates how text-in, text-out chat agents can be converted to voice agents using a single prompt, leveraging ElevenLabs' advanced turn-taking, emotion-aware interruption detection, and Scribe for speech-to-text. The Voice Engine provides server and client SDKs, plus Shadcn-based UI components, enabling omni-channel deployment like phone calls and Zoom. Harries also addresses tool calling, noting that the wrapper proxies calls to existing agent logic without rebuilding. He predicts that chat agents will either adopt voice or become obsolete.

Full Workshop: Realtime Voice AI — Mark Backman, Daily

Aug 3, 2025 · 1:09:41

In this hands-on workshop, Mark Backman and Aleś from Daily demonstrate how to build real-time voice AI agents using Pipecat, their open-source Python framework, and Google's Gemini Multimodal Live API. They explain that Pipecat's modular pipeline orchestrates audio transport, speech-to-text, LLM, and text-to-speech services, allowing developers to plug and play vendors like Deepgram, OpenAI, or Cartesia. The session highlights how speech-to-speech models like Gemini Live simplify architecture by handling transcription, LLM logic, and voice generation in one step, reducing latency to under 800 milliseconds. Key topics include the critical role of voice activity detection (VAD) for natural turn-taking, strategies for managing context windows to maintain accuracy, and the trade-offs between speed and reliability when using tool calls in real-time. Despite conference Wi-Fi issues, the team live-codes a functional bot and demonstrates interruption handling and the Word Wrangler game, showcasing Pipecat's production readiness with hundreds of thousands of daily calls.

Pipecat Cloud: Enterprise Voice Agents Built On Open Source - Kwindla Hultman Kramer, Daily

Jul 31, 2025 · 26:46

Kwindla Hultman Kramer, co-founder of Daily, introduces Pipecat, an open-source, vendor-neutral framework for building voice AI agents, and Pipecat Cloud, a deployment layer optimized for real-time voice. He argues that achieving sub-800-millisecond voice-to-voice response times is critical and that frameworks like Pipecat handle hard problems like turn detection, interruption handling, and context management. Kramer explains that Pipecat supports 60+ models and services, including Gemini and OpenAI, and that Gemini is often 10x cheaper for 30-minute conversations. He notes that while speech-to-speech models like Moshi and Sesame are promising for natural conversation, they currently lag in instruction-following for enterprise use cases. Kramer also addresses global latency challenges, recommending deployment close to inference servers or using open-weight models locally, and highlights Pipecat Cloud's integration with Krisp for background noise reduction and a free open-source smart turn model.

[Full Workshop] Building Conversational AI Agents - Thor Schaeff, ElevenLabs

Jul 31, 2025 · 1:01:42

Thor Schaeff of ElevenLabs demonstrates how to build multilingual conversational AI agents using ElevenLabs' platform, which combines speech-to-text (ASR) with 99-language support, a voice library of over 5,000 voices, and language detection system tools that automatically switch between 31 languages (with plans to expand). The agent pipeline transcribes user speech, feeds it to any LLM (like GPT-4 or Gemini), and streams the response back as speech for low-latency conversations. Schaeff shows real-time language switching in Mandarin, Hindi, and English, and explains how to assign per-language voices (e.g., Chennai-accented Tamil). He addresses safety tooling—voice watermarking, live moderation, and consent verification—and discusses cost (per-minute pricing), latency mitigation via Flash models and RAG, and handling multi-language mixing within a single utterance, though accuracy degrades with more than two languages intermixed.

Your realtime AI is ngmi — Sean DuBois (OpenAI), Kwindla Kramer (Daily)

Jul 31, 2025 · 16:30

Sean DuBois (OpenAI, Pion) and Kwindla Kramer (Daily, Pipecat) argue that realtime AI systems must be built from the network layer up, with WebRTC essential for voice AI over WebSockets. WebSockets rely on TCP, causing glitches in 10–15% of connections, while WebRTC uses UDP for sub-second voice-to-voice latency (they demonstrate ~500ms target). Kwindla shows a latency breakdown from a Pipecat app, and Sean notes WebRTC handles packet loss, jitter, and bandwidth estimation automatically. They demo Squabbert, a Raspberry Pi stuffed animal using peer-to-peer WebRTC with MLX Whisper and Gemma 3 for local AI. Yashin, a non-technical mom, shares her bilingual learning app built with Sean's guidance, highlighting WebRTC's accessibility.

Why ChatGPT Keeps Interrupting You — Dr. Tom Shapland, LiveKit

Jul 31, 2025 · 27:03

Dr. Tom Shapland of LiveKit explains why voice AI agents like ChatGPT's Advanced Voice Mode keep interrupting users: they rely on a simple VAD that triggers after a silence threshold, unlike humans who predict turn endings using semantics, syntax, and prosody. To solve this, LiveKit developed a semantic end-of-utterance model that considers the last four conversation turns, extending the VAD's silence window when the user is not done. A side-by-side demo shows dramatic reduction in interruptions. Shapland contrasts this with full-duplex models like Moshi and Meta's SyncLLM, which process input and generate speech simultaneously but lack instruction-following. He predicts commercial systems will improve via smarter VAD augmentations and faster cascade pipelines, not full-duplex. The talk also covers handling backchannels (e.g., 'mm-hmm'), the challenge of benchmarking turn-taking, and why OpenAI doesn't yet use LiveKit's model.

Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)

Jun 27, 2025 · 17:05

Arjun Desai of Cartesia AI and AWS's Rohit Talluri discuss scaling voice AI for enterprise, arguing that latency and controllability are critical, with Cartesia's state-space model Sonic 2 achieving 40ms model latency for real-time applications. Desai explains that traditional transformer models scale quadratically, while Cartesia's SSMs maintain O(1) generation, enabling 2.5x faster inference than their earlier models. He emphasizes that edge deployment is 5x faster than cloud round-trips, making local models essential for interactive use cases. On quality, Desai notes that voice AI must handle interruptions, accents, and background noise, and that Cartesia's voice marketplace amplifies human voice actors rather than replacing them. Looking to 2030, he predicts voice AI will become the default interface across healthcare, customer support, and gaming, with interactive models extending beyond audio to full world models.

Milliseconds to Magic: Real‑Time Workflows using the Gemini Live API and Pipecat

Jun 27, 2025 · 21:43

Kwindla Kramer (Daily, Pipecat) and Shrestha Basu Mallick (Google DeepMind, Gemini API) argue that voice is the most natural interface and that the Gemini Live API combined with the Pipecat framework enables developers to build real-time multimodal voice agents covering the full stack from models to application code. They demo a voice-driven task management app that handles grocery, reading, and work lists, showing impressive context-aware tool use (e.g., consolidating lists, web searching for 'Dream Count' author) but also jagged edge limitations like turn detection errors and persistent misspelling of 'Kwin.' They discuss how capabilities like turn detection migrate down the stack over time. The episode also covers proactivity, multilinguality, telephony integration, and experimental native audio models for emotive, steerable dialogue.

The Voice-First AI Overlay: Designing Conversational Co-Pilots - Gregory Bruss

Jun 3, 2025 · 16:44

Gregory Bruss introduces the Voice-First AI Overlay, an agent that assists live human conversations without becoming a third speaker. He argues that as AI grows more powerful, this paradigm can keep humans in the loop through voice, the most natural interface. The overlay passively listens and surfaces relevant help—like language suggestions in a demo—addressing timing, relevance, attention, and latency. Engineering challenges include jitterbug input, context repair, premature interrupts, and glanceable ghost. Bruss emphasizes design principles: transparency, minimal cognitive load, and progressive autonomy. He also explores future directions like full-duplex speech models and on-device inference for privacy.

How to build the world's fastest voice bot: Kwindla Hultman Kramer

Feb 10, 2025 · 20:38

Kwindla Hultman Kramer, CEO of Daily, argues that colocating speech-to-text, LLM inference, and text-to-speech in a single compute container is the most effective way to achieve sub-500 millisecond voice-to-voice latency for conversational AI. He details how architectural flexibility and low-latency media transport are critical, citing measured bottlenecks like 30–40ms from macOS mic processing and typical voice-to-voice latencies of 600–700ms. To hit faster response times, his team uses Deepgram’s on-premises STT and TTS models via Docker and Llama 3 8B for LLM inference, achieving 500–700ms in an open-source demo. The talk introduces PipeCat, a vendor-neutral open-source framework for real-time multimodal AI that orchestrates components like transcription, endpointing, interruption handling, and text-to-speech. Kramer emphasizes that while frontier multimodal models are coming, orchestration layers remain essential for building production-grade voice bots, and shares that a recent latency demo gained over 175,000 views on Twitter.