LiveKit

Building Conversational Agents — Thor Schaeff and Philipp Schmid, Google DeepMind

Apr 30, 2026 · 1:47:34

Thor Schaeff and Philipp Schmid of Google DeepMind demonstrate building conversational agents with the Gemini Interactions API and Live API, arguing that the new Interactions API simplifies agent development by introducing server-side state management, unified content blocks, and remote MCP support. They walk through creating a coding agent with read/write file and bash tools using Gemini 3 Flash, highlighting how the model uses tool calling to execute commands and return results. The Live API, powering real-time voice and video interactions, supports native multilingual audio, tool calling, and Google Search grounding, though the presenters acknowledge its current limitations in production, including lack of session transcript retrieval and shorter context windows (15 minutes audio-only). They showcase a Live Jukebox demo integrating Lyria 3 music generation and discuss business use cases like Shopify Sidekick, Waymo, and elderly companion apps, emphasizing that while the technology is ready for experimentation, production readiness depends on partner integrations for observability and compliance.

Why ChatGPT Keeps Interrupting You — Dr. Tom Shapland, LiveKit

Jul 31, 2025 · 27:03

Dr. Tom Shapland of LiveKit explains why voice AI agents like ChatGPT's Advanced Voice Mode keep interrupting users: they rely on a simple VAD that triggers after a silence threshold, unlike humans who predict turn endings using semantics, syntax, and prosody. To solve this, LiveKit developed a semantic end-of-utterance model that considers the last four conversation turns, extending the VAD's silence window when the user is not done. A side-by-side demo shows dramatic reduction in interruptions. Shapland contrasts this with full-duplex models like Moshi and Meta's SyncLLM, which process input and generate speech simultaneously but lack instruction-following. He predicts commercial systems will improve via smarter VAD augmentations and faster cascade pipelines, not full-duplex. The talk also covers handling backchannels (e.g., 'mm-hmm'), the challenge of benchmarking turn-taking, and why OpenAI doesn't yet use LiveKit's model.