Pipecat

Voice Agents That Handle Interrupts - Chintan Agrawal and Daniel Wirjo, AWS

Jul 20, 2026 · 32:57

AWS Solutions Architects Chintan Agrawal and Daniel Wirjo argue that the hardest problem in production voice agents is audio engineering—not AI—specifically turn-taking, the decision of when an agent should stop speaking or start responding. They present three levels of turn detection: Level 1 uses Silero VAD with a silence timeout (e.g., 300ms default), Level 2 delegates to STT providers like Cartesia or Deepgram for built-in endpointing (p50 ~250-300ms), and Level 3 combines Silero VAD with SmartTurn, an open-source 8MB model achieving 58.9% recall and 68.4% precision while falling back to VAD on low confidence. They show how interruption handling (barge-in) flushes TTS/LLM in ~15ms and distinguish real interruptions from backchannel acknowledgments. Latency budgets are tight: 40ms mic encoding, 52ms network/jitter, 300ms STT+endpointing, 500-650ms LLM time-to-first-byte (dominant bottleneck), and 120-190ms TTS playback, totaling 800-1300ms in standard cloud setups. Co-locating models in one GPU cluster can achieve ~500ms voice-to-voice. For LLMs, Nemotron 3 Ultra and GPT-4.1 achieve ~530ms p50 but GPT-4.1 spikes to 1.7s p95, and multi-turn drift (>15 turns) can break prompt…

Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI

May 31, 2026 · 24:35

Rishabh Bhargava from Together AI outlines engineering voice agents, explaining that pipeline architecture with colocated models can achieve sub-500ms response times critical for user retention. Speech-to-text targets P90 under 100ms and 6% word error rate, while the LLM must stay within 200-300ms time-to-first-token using 8-30B parameter models—larger models blow the budget, smaller ones break tool calling. Network latency from distant data centers adds 75ms (30% overhead) versus 5ms when colocated in the same building. Pure speech-to-speech models are emerging but still struggle with instruction following and tool calling. The thinker-talker pattern uses a small LLM for fast conversational flow and issues a single tool call to a larger model for complex requests.

Full Workshop: Realtime Voice AI — Mark Backman, Daily

Aug 3, 2025 · 1:09:41

In this hands-on workshop, Mark Backman and Aleś from Daily demonstrate how to build real-time voice AI agents using Pipecat, their open-source Python framework, and Google's Gemini Multimodal Live API. They explain that Pipecat's modular pipeline orchestrates audio transport, speech-to-text, LLM, and text-to-speech services, allowing developers to plug and play vendors like Deepgram, OpenAI, or Cartesia. The session highlights how speech-to-speech models like Gemini Live simplify architecture by handling transcription, LLM logic, and voice generation in one step, reducing latency to under 800 milliseconds. Key topics include the critical role of voice activity detection (VAD) for natural turn-taking, strategies for managing context windows to maintain accuracy, and the trade-offs between speed and reliability when using tool calls in real-time. Despite conference Wi-Fi issues, the team live-codes a functional bot and demonstrates interruption handling and the Word Wrangler game, showcasing Pipecat's production readiness with hundreds of thousands of daily calls.

Pipecat Cloud: Enterprise Voice Agents Built On Open Source - Kwindla Hultman Kramer, Daily

Jul 31, 2025 · 26:46

Kwindla Hultman Kramer, co-founder of Daily, introduces Pipecat, an open-source, vendor-neutral framework for building voice AI agents, and Pipecat Cloud, a deployment layer optimized for real-time voice. He argues that achieving sub-800-millisecond voice-to-voice response times is critical and that frameworks like Pipecat handle hard problems like turn detection, interruption handling, and context management. Kramer explains that Pipecat supports 60+ models and services, including Gemini and OpenAI, and that Gemini is often 10x cheaper for 30-minute conversations. He notes that while speech-to-speech models like Moshi and Sesame are promising for natural conversation, they currently lag in instruction-following for enterprise use cases. Kramer also addresses global latency challenges, recommending deployment close to inference servers or using open-weight models locally, and highlights Pipecat Cloud's integration with Krisp for background noise reduction and a free open-source smart turn model.

Your realtime AI is ngmi — Sean DuBois (OpenAI), Kwindla Kramer (Daily)

Jul 31, 2025 · 16:30

Sean DuBois (OpenAI, Pion) and Kwindla Kramer (Daily, Pipecat) argue that realtime AI systems must be built from the network layer up, with WebRTC essential for voice AI over WebSockets. WebSockets rely on TCP, causing glitches in 10–15% of connections, while WebRTC uses UDP for sub-second voice-to-voice latency (they demonstrate ~500ms target). Kwindla shows a latency breakdown from a Pipecat app, and Sean notes WebRTC handles packet loss, jitter, and bandwidth estimation automatically. They demo Squabbert, a Raspberry Pi stuffed animal using peer-to-peer WebRTC with MLX Whisper and Gemma 3 for local AI. Yashin, a non-technical mom, shares her bilingual learning app built with Sean's guidance, highlighting WebRTC's accessibility.

Optimizing inference for voice models in production - Philip Kiely, Baseten

Jul 1, 2025 · 15:13

Philip Kiely of Baseten shows how open-source TTS models like Orpheus TTS, built on a LLaMA 3.2 3B backbone, can be optimized for production inference using LLM tooling such as TensorRT-LLM and FP8 quantization, achieving time-to-first-byte (TTFB) under 150 milliseconds and supporting 16–24 simultaneous streams on a single half-H100 GPU. He argues that because TTS models are architecturally similar to LLMs, techniques like dynamic batching, KV-cache quantization, and Torch Compile on the audio decoder can dramatically reduce latency and increase concurrency. He emphasizes that for real-time voice agents, the goal is not raw tokens per second (real-time requires only 83 TPS for Orpheus) but low TTFB and high throughput to minimize GPU spend. However, Kiely warns that non-runtime factors—such as client code using sequential requests without session reuse or sending traffic to distant data centers—can easily add back the milliseconds saved at the model level, and that infrastructure connecting listening, thinking, and talking pipelines is often the dominant source of latency.

Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)

Jun 27, 2025 · 17:05

Arjun Desai of Cartesia AI and AWS's Rohit Talluri discuss scaling voice AI for enterprise, arguing that latency and controllability are critical, with Cartesia's state-space model Sonic 2 achieving 40ms model latency for real-time applications. Desai explains that traditional transformer models scale quadratically, while Cartesia's SSMs maintain O(1) generation, enabling 2.5x faster inference than their earlier models. He emphasizes that edge deployment is 5x faster than cloud round-trips, making local models essential for interactive use cases. On quality, Desai notes that voice AI must handle interruptions, accents, and background noise, and that Cartesia's voice marketplace amplifies human voice actors rather than replacing them. Looking to 2030, he predicts voice AI will become the default interface across healthcare, customer support, and gaming, with interactive models extending beyond audio to full world models.

Milliseconds to Magic: Real‑Time Workflows using the Gemini Live API and Pipecat

Jun 27, 2025 · 21:43

Kwindla Kramer (Daily, Pipecat) and Shrestha Basu Mallick (Google DeepMind, Gemini API) argue that voice is the most natural interface and that the Gemini Live API combined with the Pipecat framework enables developers to build real-time multimodal voice agents covering the full stack from models to application code. They demo a voice-driven task management app that handles grocery, reading, and work lists, showing impressive context-aware tool use (e.g., consolidating lists, web searching for 'Dream Count' author) but also jagged edge limitations like turn detection errors and persistent misspelling of 'Kwin.' They discuss how capabilities like turn detection migrate down the stack over time. The episode also covers proactivity, multilinguality, telephony integration, and experimental native audio models for emotive, steerable dialogue.

Realtime Conversational Video with Pipecat and Tavus — Chad Bailey and Brian Johnson, Daily & Tavus

Jun 27, 2025 · 18:46

Chad Bailey of Daily and Brian Johnson of Tavus explain how to build real-time conversational video bots using the Pipecat open-source framework and Tavus's avatar platform. They argue that beyond models, an orchestration layer is essential for handling input, processing, and output with low latency. Bailey details Pipecat's pipeline of frames, processors, and pipelines that manage audio/video frames, speech-to-text, LLM inference, and text-to-speech in a modular way. Johnson describes Tavus's 600-millisecond response time and proprietary models—Sparrow Zero, Raven Zero—plus future turn detection, response timing, and multimodal perception models being integrated into Pipecat. They emphasize that Pipecat solves real-world production challenges like observability, barge-in handling, and parallel pipelines for tasks like voicemail detection. Johnson admits Tavus initially built its own orchestration but now plans to adopt Pipecat internally, as its customers are already using it.

Voice Agent Engineering — Nik Caryotakis, SuperDial

Apr 18, 2025 · 19:07

Nik Caryotakis from SuperDial argues that in 2025, the key to production Voice AI is reliability over realism, especially for sensitive healthcare calls. SuperDial automates back-office phone calls, saving over 100,000 hours of human calling with a lean team of four engineers. He advocates for the 'say the right thing at the right time' approach, using open-source tools like PipeCat for orchestration, TensorZero for LLM routing, and self-hosted LangFuse for HIPAA-compliant observability. Caryotakis warns that new voice-to-voice models often produce nonsensical audio, favoring a sequenced STT/LLM/TTS pipeline for control. He shares specific last-mile challenges: pronunciation of names like 'Caryotaikis', avoiding confusing bot names like 'Billy', and the need for fallbacks when OpenAI goes down. The talk emphasizes that the unique value of a voice agent lies in conversational design and vertical integrations, not realistic voices.

How to build the world's fastest voice bot: Kwindla Hultman Kramer

Feb 10, 2025 · 20:38

Kwindla Hultman Kramer, CEO of Daily, argues that colocating speech-to-text, LLM inference, and text-to-speech in a single compute container is the most effective way to achieve sub-500 millisecond voice-to-voice latency for conversational AI. He details how architectural flexibility and low-latency media transport are critical, citing measured bottlenecks like 30–40ms from macOS mic processing and typical voice-to-voice latencies of 600–700ms. To hit faster response times, his team uses Deepgram’s on-premises STT and TTS models via Docker and Llama 3 8B for LLM inference, achieving 500–700ms in an open-source demo. The talk introduces PipeCat, a vendor-neutral open-source framework for real-time multimodal AI that orchestrates components like transcription, endpointing, interruption handling, and text-to-speech. Kramer emphasizes that while frontier multimodal models are coming, orchestration layers remain essential for building production-grade voice bots, and shares that a recent latency demo gained over 175,000 views on Twitter.