Text-to-Speech

From Transcription to Live Music: Gemini's Audio Stack — Thor Schaeff, Google DeepMind

Jun 9, 2026 · 19:34

Thor Schaeff from Google DeepMind presents the Gemini audio stack—Gemini 3 Flash Preview for deep audio understanding, Gemini 3.1 Flash Live for real-time sound-to-sound multimodal interaction, and Lyria 3 for music generation. He shows how a single API call extracts speaker labels, timestamps, emotions, language detection, and translation, and how speech generation uses a 'director's note' to modify a base voice's accent and tone. The talk culminates in a live demo where the Gemini Live model uses Lyria via tool calls to generate a German techno schlager about the UK startup scene.

Reachy Mini: the $300 open source robot you can actually hack — Andres Marafioti, Hugging Face

May 29, 2026 · 21:16

Andres Marafioti from Hugging Face presents Reachy Mini, a $300 open source robot shipped to 7,500 people unassembled, and explains that its most popular app is voice conversation. To make that app responsive, his team optimized Qwen3-TTS from a below-real-time 0.8x factor to 5.8x real time by adding streaming, switching to a static KV cache, and enabling CUDA graph captures, cutting time to first audio to under 200 milliseconds. The full voice pipeline runs Parakeet transcription every 150 milliseconds with partial results, feeds Qwen 3.5 27B for the LLM, and uses the optimized TTS, while infrastructure round trips are handled by a load balancer that separates LLM endpoints from conversation nodes. Marafioti argues that expensive humanoid robots limit creativity and access, whereas Reachy Mini's hackable, repairable design invites hackers, students, and dreamers to build new interactions. He demonstrates the robot's ability to take photos, show emotions, and even be pet, and emphasizes that all models and software are open source.

Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind

May 20, 2026 · 16:21

Patrick Löber, a member of the technical staff at Google DeepMind, explains how to build native multimodal agents using the Gemini API ecosystem, covering multimodal understanding, native image and speech generation, and real-time interaction via the Live API. He demonstrates constructing a NotebookLM clone as an agentic system where a reasoning Gemini model decides whether to generate an infographic or podcast-style audio using function calls to specialized models like Nano Banana for images and a text-to-speech model for speech. The episode details practical implementation: uploading PDFs, video, and audio files; using context caching to reduce costs by 90%; and generating infographics or multi-speaker audio directly from prompts. Löber highlights that native generation models understand world context—like drawing arrows on a map to produce the Golden Gate Bridge—and that the Live API enables audio-to-audio interactions with a single architecture, supporting multiple languages and accents.

Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral

May 9, 2026 · 22:26

Mistral AI scientist Samuel Humeau explains why text-to-speech models now resemble LLMs, using an autoregressive transformer with neural audio codecs to compress 200 kbps audio into ~500 tokens per second. He demos Mistral's open-weight TTS model, cloning a voice from a few seconds of reference audio and generating speech with 17 ms first-token latency via frame-by-frame streaming. The talk covers the codec-backbone-decoder pipeline, the model's 4B-parameter backbone generating 37 tokens per 80 ms frame via a diffusion decoder, and a live voice agent answering conference schedule questions. Humeau notes the next latency win is handling streaming text input from an LLM, with no settled architecture yet, while keeping the encoder for voice cloning proprietary.

Voice AI: when is the "Her" moment? — Neil Zeghidour, CEO, Gradium AI

May 9, 2026 · 19:27

Neil Zeghidour, CEO of Gradium AI, argues that voice AI remains far from the 'Her' ideal because cascaded systems (speech-to-text, LLM, text-to-speech) suffer from high latency—tool calls alone add 500ms to 4 seconds—while human response time is ~200ms. Speech-to-speech models reduce latency but are half-duplex, meaning they cannot handle overlapping speech or backchanneling, unlike Moshi, Gradium's full-duplex model. However, Moshi lacked intelligence, tool calls, and paralinguistic understanding—the ability to infer tone, hesitation, or discomfort from voice, which is stripped away in text. Cost is another barrier: TTS bills burn through fundraising before user bases grow. Gradium's solution is Phonon, an on-device TTS model running on smartphone CPUs, offering privacy and eliminating API fees. The path forward requires combining full-duplex natural conversation with the reliability and smarts of cascaded systems.

[Full Workshop] Building Conversational AI Agents - Thor Schaeff, ElevenLabs

Jul 31, 2025 · 1:01:42

Thor Schaeff of ElevenLabs demonstrates how to build multilingual conversational AI agents using ElevenLabs' platform, which combines speech-to-text (ASR) with 99-language support, a voice library of over 5,000 voices, and language detection system tools that automatically switch between 31 languages (with plans to expand). The agent pipeline transcribes user speech, feeds it to any LLM (like GPT-4 or Gemini), and streams the response back as speech for low-latency conversations. Schaeff shows real-time language switching in Mandarin, Hindi, and English, and explains how to assign per-language voices (e.g., Chennai-accented Tamil). He addresses safety tooling—voice watermarking, live moderation, and consent verification—and discusses cost (per-minute pricing), latency mitigation via Flash models and RAG, and handling multi-language mixing within a single utterance, though accuracy degrades with more than two languages intermixed.

Serving Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing - Neil Dwyer, Gabber

Jul 31, 2025 · 16:09

Neil Dwyer, CTO of Gabber, details how his startup serves real-time voice AI for under $1 per hour using open-source Orpheus TTS, LoRAs for emotive voice cloning, and vLLM with FP8 dynamic quantization to achieve 95-105 tokens per second on L40S GPUs. He explains the critical latency challenge of 'head of line silence' in Orpheus (600ms in default voices) and how fine-tuning LoRAs reduces it to ~100ms, fitting within a 1.5-second budget for real-time conversation. Gabber batches multiple LoRA generations per GPU and uses a consistent hash ring for load balancing across servers, enabling popular clones to be dynamically replicated. The talk argues that with current open-source tools, building affordable consumer voice AI is accessible to small teams.

Text-to-Speech Data Preparation and Fine-tuning Workshop - Ronan McGovern

Jun 3, 2025 · 34:00

Ronan McGovern walks through fine-tuning Sesame's CSM-1B text-to-speech model on a specific voice, using a YouTube video as the data source. He explains token-based TTS models, including how audio is represented via codebooks and how CSM-1B uses a main transformer for zeroth tokens and a secondary transformer for 31 hierarchical tokens. The workshop covers data preparation: downloading audio with yt-dlp, transcribing with Whisper Turbo, manually correcting the transcript, and splitting audio into 30-second chunks (41 clips from a 30-minute video). Fine-tuning uses Unsloth with LoRA adapters (rank 32, alpha 16) on linear layers, training for one epoch with a batch size of 2 and virtual size of 8, reducing loss from ~6.34 to ~3.72. Evaluation compares zero-shot inference (random speaker), voice cloning (closer but imperfect), and fine-tuned plus cloning (best result, producing an Irish-accented voice with natural errors). McGovern recommends 50+ 30-second clips for noticeable effect and notes that combining fine-tuning with voice cloning yields good performance even with limited data.

State Space Models for Realtime Multimodal Intelligence: Karan Goel

Oct 29, 2024 · 14:26

Karan Goel, founder of Cartesia, argues that state space models (SSMs) are key to real-time multimodal intelligence, offering cheaper, faster, and higher-quality alternatives to transformers for streaming applications like conversational voice and on-device assistants. He contrasts batch intelligence (cloud APIs) with streaming intelligence needed for low-latency tasks, emphasizing that SSMs compress information linearly rather than storing all tokens, enabling efficient long-context processing. Cartesia’s voice generation model achieves instant latency in the data center and is being optimized for on-device Mac and desktop deployment. Goel asserts that compression helps long-context tasks (e.g., 24-hour security footage analysis) more than retrieval, and that SSMs now match transformer quality while scaling better for multimodal data.