Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind
May 20, 2026 · 16:21
Patrick Löber, a member of the technical staff at Google DeepMind, explains how to build native multimodal agents using the Gemini API ecosystem, covering multimodal understanding, native image and speech generation, and real-time interaction via the Live API. He demonstrates constructing a NotebookLM clone as an agentic system where a reasoning Gemini model decides whether to generate an infographic or podcast-style audio using function calls to specialized models like Nano Banana for images and a text-to-speech model for speech. The episode details practical implementation: uploading PDFs, video, and audio files; using context caching to reduce costs by 90%; and generating infographics or multi-speaker audio directly from prompts. Löber highlights that native generation models understand world context—like drawing arrows on a map to produce the Golden Gate Bridge—and that the Live API enables audio-to-audio interactions with a single architecture, supporting multiple languages and accents.
