Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind

Okay.

Yes, you're good.

Awesome, yeah. Thank you, everyone, for joining the session. Uh, if you joined the previous session, we're switching slightly the topic and talking more about Gemini now, but I have 2 slides about Gemma as well, so to make Gus happy.

Okay.

Um, I'm Patrick. I'm a member of the technical staff at Google DeepMind. I work on the Gemini API and AI studio, and today I want to talk about Any-to-Any, building native multimodal agents. So I want to talk about multimodal understanding, multimodal generation, real-time interactions, and then also build an example app together.

So at the end of this session, you should be able to build this for yourself, a little NotebookLM clone. Um, so what does Any-to-Any mean? Uh, these are all the capabilities you can do, uh, with the Gemini API.

So there's a lot of use cases this enables, uh, because Gemini does not only understand text,right? It's natively multimodal, so you can also feed in code, image, audio, video, and then some more, like URLs and also Google search.

And then it can not only generate text, but now we're also able to generate images, uh, speech generations, video generations, uh, function calls, and of course, of course code generation. So yeah, this enables a lot of really, really cool stuff.

Um, but this slide is, is slightly giving the wrong impression because actually there are still different models. It's not one multimodal model yet. Uh, this is kind of a bigger vision that we have at Gemini, uh, at Google DeepMind, to bring more of the generation capabilities also into Gemini.

But currently it looks a little bit like this. This is an ugly slide, I know. Um, but yeah, we have the main Gemini model as series,right now Gemini 3, and it's able to understand multiple modalities, but it only outputs text.

And then we have different specialized native generation models, for example, Nano Banana for native image generation and speech generation based on the main Gemini models. And I want to talk about this in a moment a little bit. And also, I mentioned Gemma here, uh, correct me if I'm wrong, but it allows text, image, and video input, and the smaller models also audio inputs.

So you can also build multimodal agents locally. Um, yeah. So I want to focus on, on 4 things, uh, 4 models. Uh, the multimodal understanding with Gemini, native image generation, native speech generation, and then also, if we still have time, a little bit about, uh, the live API.

Um, and then build something together, or at least give you the building blocks, how you can build this, a little NotebookLM clone. Who has used NotebookLM before? Oh, almost everyone, okay, so I don't think I have to explain it.

Um, but yeah, you can feed in multiple different sources. Um, and then the audio overview is pretty popular, where you can generate a podcast, uh, to explain topics for you. And then also infographics are pretty cool. So we want to build the same thing, and we want to build this as an agent rather than a workflow.

So this means that the agent should be able to decide what to create, rather than where we hardcode the pipeline. Um, here we are having a reasoning model that can decide, um, what to create, and then it's hooked up via tool calls or function calls, and then calls the other specialized models.

Um, so this is roughly how the, the app or the agentic architecture looks like. We have the phase 1 for multimodal understanding, and then we have the phase 2. This is where we have the agentic loop, where we use Gemini as the reasoning model, and it can then call different tools, and these will then generate different modalities for us.

And then it, um, acts in a loop and reasons if we need more assets or if it's good enough. And then in the end, we get text, speech, um, and, and infographics as an output. And yeah, I want to do this as an example with, uh, some learning about attention is all you need paper.

So we want to be able to feed in PDFs, images, videos. This can be a lecture, for example, or a tutorial, and then voice memos. And ideally, we also want this cross-modal understanding,right, that we can, uh, draw information from all the different sources together and make let the model make connections.

And it's actually extremely easy with Gemini to achieve this. This is basically the code you need. Uh, who has built with the Google GenAI SDK before?

Half, half, almost half of the room, yeah. Basically, this is how you set it up. You get your API key, uh, for free. It's AI.studio, and then you install the SDK. We have it available in different languages. And then you can simply upload different files.

Uh, like here we are uploading a PDF, uh, video, and an MP3 file. Or you can also, for smaller files, directly use it as inline data. And as a tip, uh, on theright side, I mentioned, uh, Gemini API skill.

So you don't have to know this code now by heart. You can just, yeah, uh, hook up your agent with the Gemini skill and then tell it, um, to create this, and then it should know how to work with the Gemini models.

Um, yeah, this is basically everything we need. And then we call client models, generate content, and here we're using Gemini 3 Flash. And then we can put everything together into the contents list and tell it, for example, analyze all these resources, give it a little bit more information, what these resources contain, and then it should generate a summary.

And then a little bit of, of practical tips or nice-to-knows for, for understanding. Um, you can also use it to transcribe audio, actually. Uh, Flash, and even the smallest ones, Flash Lite is pretty good at transcribing audio. If you just tell it in the prompt, generate a transcript of this file.

And then maybe nice-to-know is for audio, uh, 1 minute of audio translates to, uh, 1,920 tokens. And Gemini has a token, uh, limit of 1 million. So if you do the math, it translates to more than 9 hours of content you can feed in, audio content.

For video, it's roughly 1 hour. Um, but there are, um, configurations you can tweak that give you more control, and you can even feed in longer audios. Then you can tell it to, um, look at only different timestamps.

So for example, only analyze from minute 5 to minute 15. Um, and then, yeah, you can use the file API that easily lets you upload larger files. You can even pass in URLs, uh, YouTube URLs directly. And what's also nice-to-know is you can, um, combine it with context caching.

This is built into the API. This is especially useful if you're, um, loading longer files into Gemini and doing repeated queries, because then it saves you 90% of the costs. Um, so yeah, this is multimodal understanding in a nutshell.

So here, doing a quick checkpoint, we're now able to use Gemini to understand all these different resources and generate a summary. Um, and actually, the timer is not working, so I don't know how much longer I have, but I think we are still a little bit good.

Uh, so yeah, then the next phase is the, the multimodal generation part. Um, so for this, we're using the agentic loop, where we use Gemini, um, as the brain behind this, and we combine it with function calling. And I will show you how to do this in a moment.

And then these functions call the specialized, uh, native generation models, and then it can reason if these assets are enough or if we need more. And the way to do, to use these specialized models is also basically the same code.

Once you have the SDKs, you call, again, client models, generate content. In this case, we're using Gemini 3.5 Flash image preview. It's not the nicest model, but this is actually Nano Banana 2, the more famous model, uh, famous name for it.

And then we tell it to, yeah, create a picture. Or in this case, we can it's pretty good at creating infographics, which is pretty cool. Just give it in your prompt, uh, create an infographic, and then it's creating these, these nice, uh, slide graphics for us.

And similar for text-to-speech. There we have a text-to-speech model, which currently is still based on Gemini 2.5. Um, and you can combine it with different configurations. You can also do, uh, 2, uh, speaker, um, audio files. So for example, this podcast style.

And here is a nice example if the sound works. Um,

does it work?

Neural network architecture. Introduced in a 2017 Google paper called Attention is All You Need. Transformers are the revolutionary models behind using most powerful AI, like GPT-3.

Transformers in only 2 minutes for you. And

then the function calling, like I mentioned, so basically to combine Gemini with function calling or tool calling. What you need to do is you create your function declaration. So there you give it a name and a description. This helps the model to understand what this function is used for.

And then also the different parameters. So in this case, we only want 1 string, which is, uh, then used for the prompt, so the detailed description of how the image should look like. And then you do the same for, uh, the audio, um, generation function.

And then if you set up, um, your model call, client models generate content. Here you configure the tools. And then you also need to add this to your prompt. So this is a small example prompt, an agent prompt, um, where you tell Gemini, "Hey, here's the study we synthesized before from the different modalities.

And now you're a research agent partner. Your job is to enhance the study guide with multiple materials." And then you do, uh, you tell it the 2 functions. So, um, decide which concepts are complex enough to need a visual diagram, and for this, call generate image.

And which sections would benefit from an audio, audio summary. And for this, uh, call generate speech. And yeah, this is basically everything you need to set up the agentic function calling for the multimodal generation part. And I also quickly wanted to touch on why native generation, uh, matters.

So we call this native image generation models, for example, because they are based on Gemini. So all the training, or a lot of the training that goes into the main Gemini models, are now also available in these models.

And this allows a lot of really, really cool use cases because these models understand the world. Uh, this on the left side is, for example, I found on Twitter and, uh, an example I really like from Nano Banana 1, where you can draw errors on maps and just tell it, "Hey, um, create a picture of what you see here."

And since Gemini understands the world here, it's able to, to correctly, um, create a picture of the Golden, uh, Gate Bridge for you. And then on theright side, this is a nice example on the, uh, from the educational space.

So you can use Nano Banana, in this case it was Nano Banana 2, to directly, um, correct your math homework, for example, and create pictures with the corrections because it understands math. It can even generate code on images for you.

So lots of nice use cases. And for the audio models, um, these are multilingual, and they understand accents and tone. So they have.

Ew, lads and lasses, we're getting stuck into building these multimodal agents today. Not faffing about. Let's just get them sorted so we can all nip to the pub for a proper pint.

Was it a good British accent? Yeah? Anyone, is anyone speaking German here? One, two, three. I have a Bavarian accent because you can actually also tell it to create different accents. Of course, not every accent in the world, but still.

So here's one with a Bavarian accent.

Servus miteinander. Heute schauen wir uns an, wie wir diese multimodalen Agenten zusammenschrauben, gell? Wenn das drum.

Was it a good Bavarian accent?

Am Ende gescheit läuft, gehen wir erst.

So yeah, I would say this is pretty.

Okay. Uh, yeah. And yeah, again, so quick checkpoint. We now know how to do the understanding part and the generation part. And this is basically already the, the NotebookLM clone. And now I quickly wanted to mention now, uh, or we also have, um, a model for real-time interaction with it via our, what we call the live API.

And for this, we have a very new model, which is also based on Gemini, uh, Gemini 3.1 Flash Live. And we call this an audio-to-audio model. So native, um, audio generation, it's only 1 architecture. Uh, audio goes in and audio goes out.

So you no longer have this cascaded pipeline with different models. And this allows a lot of really, really cool, um, natural-sounding interactions. I think I don't have time for a live demo, but you can try it at AI.studio live.

And here is a quick video from one of our colleagues, Thor. Um.

Hey, Gemini.

You kind of got started and start talking to it. And then you can also activate your content.

Up the marker.

And it's also doing great.

Thanks for asking.

Just enjoying the chat, you know.

And how it thinks with you?

Um, can you see me?

Well, it's plain as day. Oh, I see you there with your short hair and beard. Wearing a grand dark jacket over a blue shirt.

But yeah, try it out for yourself, uh, at AI.studio/live. And I think that's almost it. Um, uh, yeah, this is, again, how you can do it in the code. But there, again, we have a skill for it that you can figure, configure.

And this, yeah, now we are at all the 3 checkpoints. And yeah, the pattern is transferable to every, every other field. Um, and maybe a few, uh, shout-outs as well to some other models. I'm not sure if you've seen the keynote this morning, um, but we now have a multimodal embedding model where you can combine all the different modalities into 1 unified vector space, which allows applications like multimodal search.

And then, again, you can go local with Gemma 4 and again have this multimodal understanding and video for, for image, uh, for video with native audio. And yeah, so that's it. Uh, thank you, and have fun building multimodal agents.