Daily

Full Workshop: Realtime Voice AI — Mark Backman, Daily

Aug 3, 2025 · 1:09:41

In this hands-on workshop, Mark Backman and Aleś from Daily demonstrate how to build real-time voice AI agents using Pipecat, their open-source Python framework, and Google's Gemini Multimodal Live API. They explain that Pipecat's modular pipeline orchestrates audio transport, speech-to-text, LLM, and text-to-speech services, allowing developers to plug and play vendors like Deepgram, OpenAI, or Cartesia. The session highlights how speech-to-speech models like Gemini Live simplify architecture by handling transcription, LLM logic, and voice generation in one step, reducing latency to under 800 milliseconds. Key topics include the critical role of voice activity detection (VAD) for natural turn-taking, strategies for managing context windows to maintain accuracy, and the trade-offs between speed and reliability when using tool calls in real-time. Despite conference Wi-Fi issues, the team live-codes a functional bot and demonstrates interruption handling and the Word Wrangler game, showcasing Pipecat's production readiness with hundreds of thousands of daily calls.

Pipecat Cloud: Enterprise Voice Agents Built On Open Source - Kwindla Hultman Kramer, Daily

Jul 31, 2025 · 26:46

Kwindla Hultman Kramer, co-founder of Daily, introduces Pipecat, an open-source, vendor-neutral framework for building voice AI agents, and Pipecat Cloud, a deployment layer optimized for real-time voice. He argues that achieving sub-800-millisecond voice-to-voice response times is critical and that frameworks like Pipecat handle hard problems like turn detection, interruption handling, and context management. Kramer explains that Pipecat supports 60+ models and services, including Gemini and OpenAI, and that Gemini is often 10x cheaper for 30-minute conversations. He notes that while speech-to-speech models like Moshi and Sesame are promising for natural conversation, they currently lag in instruction-following for enterprise use cases. Kramer also addresses global latency challenges, recommending deployment close to inference servers or using open-weight models locally, and highlights Pipecat Cloud's integration with Krisp for background noise reduction and a free open-source smart turn model.

Your realtime AI is ngmi — Sean DuBois (OpenAI), Kwindla Kramer (Daily)

Jul 31, 2025 · 16:30

Sean DuBois (OpenAI, Pion) and Kwindla Kramer (Daily, Pipecat) argue that realtime AI systems must be built from the network layer up, with WebRTC essential for voice AI over WebSockets. WebSockets rely on TCP, causing glitches in 10–15% of connections, while WebRTC uses UDP for sub-second voice-to-voice latency (they demonstrate ~500ms target). Kwindla shows a latency breakdown from a Pipecat app, and Sean notes WebRTC handles packet loss, jitter, and bandwidth estimation automatically. They demo Squabbert, a Raspberry Pi stuffed animal using peer-to-peer WebRTC with MLX Whisper and Gemma 3 for local AI. Yashin, a non-technical mom, shares her bilingual learning app built with Sean's guidance, highlighting WebRTC's accessibility.

Why ChatGPT Keeps Interrupting You — Dr. Tom Shapland, LiveKit

Jul 31, 2025 · 27:03

Dr. Tom Shapland of LiveKit explains why voice AI agents like ChatGPT's Advanced Voice Mode keep interrupting users: they rely on a simple VAD that triggers after a silence threshold, unlike humans who predict turn endings using semantics, syntax, and prosody. To solve this, LiveKit developed a semantic end-of-utterance model that considers the last four conversation turns, extending the VAD's silence window when the user is not done. A side-by-side demo shows dramatic reduction in interruptions. Shapland contrasts this with full-duplex models like Moshi and Meta's SyncLLM, which process input and generate speech simultaneously but lack instruction-following. He predicts commercial systems will improve via smarter VAD augmentations and faster cascade pipelines, not full-duplex. The talk also covers handling backchannels (e.g., 'mm-hmm'), the challenge of benchmarking turn-taking, and why OpenAI doesn't yet use LiveKit's model.

Milliseconds to Magic: Real‑Time Workflows using the Gemini Live API and Pipecat

Jun 27, 2025 · 21:43

Kwindla Kramer (Daily, Pipecat) and Shrestha Basu Mallick (Google DeepMind, Gemini API) argue that voice is the most natural interface and that the Gemini Live API combined with the Pipecat framework enables developers to build real-time multimodal voice agents covering the full stack from models to application code. They demo a voice-driven task management app that handles grocery, reading, and work lists, showing impressive context-aware tool use (e.g., consolidating lists, web searching for 'Dream Count' author) but also jagged edge limitations like turn detection errors and persistent misspelling of 'Kwin.' They discuss how capabilities like turn detection migrate down the stack over time. The episode also covers proactivity, multilinguality, telephony integration, and experimental native audio models for emotive, steerable dialogue.

Giving a Voice to AI Agents: Scott Stephenson, CEO, Deepgram

Feb 10, 2025 · 13:08

Scott Stephenson, CEO of Deepgram, outlines the evolution of voice AI from slow, domain-specific systems to fast, open-ended conversational agents powered by LLMs, arguing that speed and accuracy are now solved and the key differentiator is contextual understanding. He explains that state-of-the-art speech-to-text and text-to-speech can achieve round-trip latencies under 500 milliseconds—matching human turn-taking—but notes that current systems fail to pass context between components, causing 10-20% of interactions to feel unnatural. Stephenson introduces Deepgram's 'contextual AI' vision, where models are prompted with the full conversation history, including tone, pace, and background audio, enabling the LLM to generate responses and instruct the TTS on delivery. He cautions against monolithic speech-to-speech models for enterprise use due to controllability and cost concerns, advocating for a modular stack that lets businesses optimize each component (e.g., small LLMs for simple tasks like password resets). Deepgram’s upcoming voice AI agent API integrates all components for low latency and offers $250 in free credits for experimentation.

How to build the world's fastest voice bot: Kwindla Hultman Kramer

Feb 10, 2025 · 20:38

Kwindla Hultman Kramer, CEO of Daily, argues that colocating speech-to-text, LLM inference, and text-to-speech in a single compute container is the most effective way to achieve sub-500 millisecond voice-to-voice latency for conversational AI. He details how architectural flexibility and low-latency media transport are critical, citing measured bottlenecks like 30–40ms from macOS mic processing and typical voice-to-voice latencies of 600–700ms. To hit faster response times, his team uses Deepgram’s on-premises STT and TTS models via Docker and Llama 3 8B for LLM inference, achieving 500–700ms in an open-source demo. The talk introduces PipeCat, a vendor-neutral open-source framework for real-time multimodal AI that orchestrates components like transcription, endpointing, interruption handling, and text-to-speech. Kramer emphasizes that while frontier multimodal models are coming, orchestration layers remain essential for building production-grade voice bots, and shares that a recent latency demo gained over 175,000 views on Twitter.