Prompt to Pipeline: Building with Google's Gen Media Stack — Paige & Guillaume, Google DeepMind

Hi everyone, my name is Paige. I started doing machine learning a long time ago, um, around 2009, 2010. I was primarily working with, though it feels like forever ago, I was just talking with a friend about this recently.

Back in 2009, 2010, it was kind of wild that companies would even trust open-source software to do business-critical work. And so I was contributing to things like NumPy, SciPy, like little—like little antenna, these microphones.

You guys are—

Yeah. Sure. Cool, cool, cool. So NumPy, SciPy, Matplotlib, which is still just as excruciatingly painful to use. Scikit-learn, kind of the early days of the scientific computing stack. And eventually started working as an engineer. My background is geophysics and applied math for undergrad, and then computer science and carbonate geology for grad school.

And so I started at Chevron doing work in subsurface vehicles and doing a lot of things like velocity modeling, drilling optimization, using very, very basic machine learning models. And also some large-scale compute. So if folks in the audience also have gray hair like me, you might remember Cloudera, which was one of the first companies that made some of the open-source frameworks and tools available for consumption of, like, Fortune 100 companies.

So things like Spark, a lot of the Databricks team came from the Cloudera world. And so I was spinning up clusters of machines there. Eventually, TensorFlow got released in the open-source world around 2015, towards the end of it, and I started contributing to that.

The geosciences world is really big into GPUs. They were even into GPUs before the machine learning world. And

so I had experience working with CUDA and all of the associated tools. And TensorFlow, when it was first released, only supported CPUs. I'm not sure how broadly that's known, but it was, like, distributed deep learning across CPUs. And they needed somebody to help with, like, getting GPUs to work with TensorFlow.

That's also why there are three different code paths in the original TensorFlow one framework, because they had to, like, gut the backend and replace it for CPUs, for GPUs, and for TPUs, both for single node and for distributed computation across multiple nodes.

So that's basically it. Like, I owe my entire career to open-source software and open-source models. And I got hired at Google for specifically that reason. And then I left to go work at GitHub for about a year on VS Code, which is also open-source, and early user experience testing for Copilot, and then came back to work on our large models.

And so I was part of the original PaLM 2, Gemini, and Gemma teams. Yep. So that is journey. I'm not sure. Like I said, I'm not sure it's interesting.

Do you need to leave soon, you said? Because I'd love to have a chat with you.

Oh, 4:15, because I've got meetings. We have leadership team meetings from, like, 4:30 to 5:30.

You got to take over.

Yep. And Guillaume will be here. And then also potentially, potentially our colleague Ian will come to—will come as well.

Yeah, I'm going to be potentially, because I got to say the same thing if I have a half an hour or a full hour.

I think—I think there will be—there will be, like, enough good stuff to—enough good stuff to talk about.

But yeah, like, if you're—if anybody's curious, like, open-source is a great way to, like, work with the team to figure out if you would want to work with them or not. And then to also make sure that your work is public so that other folks have access.

I'd love to show you my projects. It's open-source.

Cool, cool. Awesome.

Guess how many stars it got?

How many?

700.

Oh, wow. Excellent. Good job.

No comment.

Oh, okay.

I'm catching up with you guys.

Sprints,right? Like, they're—usually if you see everybody sprinting to do the same thing, that's a great indication that it's the wrong thing. And so, so, like, as—or like a thing that eventually the model will have that capability to do.

So one of my favorite examples of this is that, you know, when the models were first released, they had context windows of, like, 8,000 tokens, 16,000 tokens. And so everybody was sprinting to build a vector database because they were like, oh, well, we have to work around this constraint that the models have this very small context window.

And then, you know, obviously that's expanded over time. There's another great example of, you know, everybody sprinting to build fine-tunes of models to support different languages. Though now models support a variety of languages. If you've been kind of listening along for some of the Google DeepMind sessions today, you've seen that in action.

And then also many of our competitor models also support multiple languages. And so I think, like, we also saw everybody sprinting to build an agent framework. When the reality is, I think that the—like, all of that will probably be absorbed into the model eventually.

And also everybody sprinting towards things like building MCP servers, whereas now mostly people have kind of moved away from MCP servers and are adopting Skills, which are just fancy Markdown files. And then obviously longer term, you could imagine, like, oh, hey, I just want a little listener watching everything that I do, and it will automatically create Skills behind the scenes for many years.

And we're starting. Yep, yep.

No, I was going to say, I'm not necessarily agree because you guys will always cater the sort of most generalized applications, but very, very specific ones, like for instance Bioinformatics, which is what I'm working on, Google is not going to work on that.

I don't think that they will absorb those kind of functionalities.

So, so counter-counter example. Our first implementation of PaLM 2 and our first implementation of Gemini, we had to have a fine-tune, one called MedLM and one called MedPaLM, to support medical use cases. Now we see that all of the people who had previously needed to rely on those fine-tunes are just using Gemini out of the box with either retrieval or with a custom prompt.

Because all of the data that we used for those fine-tunes is just incorporated into Gemini itself.

But you have the issue of reproducibility. You're not—you're not going to have the same results all the time. You're going to have a lot of stochasticity.

So, so no large language model is deterministic. So, so that will be a problem regardless. But I do agree, like, there are—you know, I think a lot of the magic is going to come from having, like, a very opinionated view of use cases and being able to work directly with customers and solve their problems.

So with that, and that's a perfect segue into the session today. So greetings, everyone. Thank you for being brave and for coming back. I know that there were a lot of people who were unable to join us earlier this morning for some of the sessions that we had.

So just show of hands, how many folks came back this afternoon, were not here this morning, or, like, were—cool, cool, cool. Excellent. So we have a treat for you today. For the sessions that we delivered earlier, we're going to do basically a recap of them.

So we'll be walking through all of the—all of the examples. You won't have—you won't kind of walk away feeling like you've missed anything. And I'll be doing some within AI Studio and Anti-Gravity. And then my colleague—my colleague Guillaume, and you can also see one of my agents that's kind of doing its thing with computer use and browsing.

I'm going to disconnect for a second so we don't accidentally see something that perhaps should not be shared. And then going to go straight into slideshow view.

Awesome. So greetings, everyone. My name is Paige. So I am really excited to be here today. And I am especially excited to show you all of the things that we've been doing at Google DeepMind over the course of the last year, last six months.

It's been a wild ride. And never in my career have I been more excited to be a machine learning engineer working in this space. Just over the last month and a half, it feels a little bit like whiplash.

We've released a ton of different things. So Gemini 3.1 Flash Live, which we'll take a look at in a second, which gives you the ability to have a real-time conversation with the model. Gemini 3.1 Pro and Flash Light, respectively, are largest and are very, very small but very capable models that are also very cost-effective, very performant.

AugmentCode, if you've heard of them, they're a company over in the Bay Area. They've just recently replatted their entire agent infrastructure to default to Gemini 3.1 Pro, specifically because of performance and cost ratios. It can do a lot of really good work in a very, like, small number of pennies.

Also Nano Banana 2 for image generation, image editing, including reverse image search, which we'll also take a look at in a second. And you'll be hearing about from my colleague Guillaume, who's our expert in generative media. Our Embeddings 2.0 model, which allows you to embed video, audio, images, code, and text in the same embedding space.

So you can say, like, show me everything you have related to llamas, and you will see everything from, like, stuffed llamas to pictures of llamas to videos of llamas to things of, like, what does it sound like when a llama makes noises?

Because I don't know, but, like, the model—the model somehow does. LIA 3 for music generation, Genie 3 for world model building. Our full-stack runtime with AI Studio, which allows you to incorporate database, OAuth, custom API keys if you want to use other services, and a whole bunch of other things.

Gemma 4 for open models. We just released it last week under an Apache 2 license, which is really exciting if you care about open models. And then also VO 3.1 Lite for video generation. So massive number of things across a broad spectrum of surfaces.

And part of the reason for this is because Gemini is kind of unique in the industry in a couple of ways. One of which is that it's natively multimodal, so it can understand video and images and audio and text and code and all of the above all at once.

But it can also output multiple modalities. So it can output text, it can output code, just like all of the other competitors on the market. But it can also output images, it can edit images, it can output images and text interleaved, and it can also output audio tokens.

So pretty, pretty compelling use cases. But I think it's a lot more interesting to see it rather than to just have me waxing poetic about it. And for that, I am going to go into AI Studio real quick, and we're going to take a look at some of the things that Gemini can do.

So first off, how many people have used AI Studio before? Excellent. I'm very, very glad that all of the DeepMinders have raised their hands. But for folks who have not, AI Studio is kind of the best place to go to get access to DeepMind's models as soon as they're released.

We have a playground feature. We also have a build feature, which we'll see in a second, which is very similar to v0.dev or Lovable. You can select different models here off to theright. So you can see if I click on the model name, we see some pills, everything from Gemini to live capabilities, image generation, video generation, audio generation, and music.

And then also our Gemma open model family, which we'll also take a look at in a second. You can select different models for the purposes of speed. I am selecting Gemini 3.1 Flash Light preview. I am also on my personal instance of AI Studio, and this is an attempt to drive down costs.

So Gemini 3.1 Flash Light is around 25 cents per million tokens analyzed, which is really, really good, almost an order of magnitude lower than our Gemini 3.1 Pro model. And it can still analyze video, audio, etc. It's just a lot more lightweight, which means that you might not always get the same great capabilities, but it usually does a pretty good job.

And you can do everything from analyzing images to analyzing video. So if I, as an example, was going to look up a—was going to look like a dinosaur YouTube video, we already used this Rexi, the little T-Rex before, so I know it works.

But I am going to find maybe this one, Carnotosaurus,

which is—it looks like a very, very long video, so around an hour long. We might chop it up a little bit. But if you click this plus sign, you can see that you can add different files. You can either add files from Drive, everything from text files to PDFs.

You can upload files directly, you can record audio live, add camera footage, add a link to a YouTube video, which we'll doright now. So I paste in the link to the YouTube video. I'll do a start time of zero seconds and an end time of around 300.

And it samples at around one frame per second. And then what you see is this is, you know, 30,000, 30,900 tokens. I'm going to make sure to turn on Google Search grounding off to theright. And I'm going to say, please create a table with timestamps for all of the kinds of dinosaurs that you see in this video.

Make sure to include a fun fact about each dinosaur. For all of the other dinosaur experts, like armchair dinosaur experts in the room, you might have noticed that it said Carnotosaurus. I am skeptical that that's an actual dinosaur.

So we'll see, we'll see what happens. But what's going on behind the scenes is that the model is getting sent this YouTube video for inference, and it's not just the metadata associated with the YouTube video, it's also frame by frame the video itself.

And question?

Can you just put it in the microphone?

Does anyone else have—does anyone else have issues with seeing the settings? Or, like, could we change the lighting? Oh, is this better? Oh, excellent. Awesome. So that's—thank you for the request. I can see it perfectly on my screen.

Usually my eyes get a little irritated by light mode, but this is as long as it's better. And so it took in the video, it defined the different dinosaurs that it sees.

It says that the name means mediating bull, mentioned Triceratops, and then also mentioned that Pteranodons are not necessarily dinosaurs, they're a group of flying reptiles. And then if I wanted to—if I wanted to get the code that was used to generate this experience, and I wanted to replicate it in my own app, all I have to do is click get code, and it automatically configures the model.

It configures the URL that I have inputted, the kind of the offset for the video, the prompt that I used, and it's in TypeScript, Python, or whatever your favorite language might be. So the TL;DR is that if you can get it working in AI Studio, you can get it working as part of your app.

All you have to do is click the get code button.

So that is Gemini 3.1 Flash Light for analyzing video. You can also use it to analyze images with a couple of other baked-in tools. And if we look over to theright, we can see structured outputs, code execution, things like function calling, even for custom functions.

Also some things like URL context. And all of these are very, very special, but they're also just one-liners if you want to use them as part of your API. So as an example, if I turn on code execution, I'm going to see Gemini 3.1 Flash Light selected.

I'm going to go into compare mode. So I want to compare it with maybe the Gemini 3 Flash preview, also with code execution turned on. And I'm going to look for a picture of Lego bricks. This one. And copy the image.

Andright now we're in compare mode. So I'm comparing two different models, both with the same tool turned on. And say something to the effect of, draw bounding boxes

around all of the green Lego bricks using Python. Make sure or display the image with bounding boxes. And code execution is sort of giving Gemini the ability to sort of stand up a makeshift Python environment that's sandboxed. Use a whole bunch of data science libraries that are pre-installed and use those, invoke them as tool calls, kind of like writing the code and incorporating anything that you might share in.

So I shared in this image. And very, very quickly, I'm not sure if you saw how quickly, but it was able to draw bounding boxes around the green Lego bricks. You could also ask for segmentation masks. And even more excitingly, so Gemini 3 Flash is still plugging along.

But if you look at how much this cost to do to define the bounding boxes, you could have also asked for things like, tell me how many green Legos there are, or tell me, you know, what are the orientations of the Lego bricks?

Tell me how many, you know, what are they called, the little funny rabbit things, Labubus? Tell me how many of those you see in all of the frames of this video and at what timestamp. Like, those are the kinds of tools that can be invoked via the sandbox environment.

And again, like, very, very inexpensive in order to do this work with Gemini 3.1 Flash Light and code execution turned on. And it's just a one-liner to stand up that sandbox Python environment with compute that Gemini can use to do that work.

Cool. And it looks like the Gemini 3 Flash preview was able to do it. It was checking its work. So that was the iteration that it was going through along the way. So it got this first result, and it said, allright, I want to double-check and verify that what I did wasright.

It drew the segmentation mask with, like, define all of the green spectrums that it saw in the image. And then it kind of verified that those were the correct coordinates. And still, if I look, it's significantly more cost with the verification, but still on the order of pennies in order to draw the bounding boxes.

It just took a little bit more time and also a lot more tool calls to invoke. So I strongly, strongly recommend playing around with Gemini 3.1 Flash Light for your use cases, especially if previously you were relying on Gemini 2.0 Flash or 2.5 Flash.

Cool. So we also have a feature called Build. Build, again, is kind of similar to v0.dev or Lovable if you've played with that before. We've recently added a feature where you can add database and auth to your apps within Build.

And so as an example, today, if I wanted to click this guy and say something to the effect of, create an app that gives the user the ability to upload an image of their bookshelf. The bookshelf should have a whole bunch of books on it so we can see their spines, things like titles and author names.

I want you to use Google Search grounding to fill in the blanks for the books. So make sure that you have information about the title, the author name, the description of the book, and then also what kind of genre it might be.

And I want you to save it all to a database. So the user logs in with their Google account, they upload a picture of their bookshelf, and it saves all of their books in, like, this database format so they know what books they have and it's attached to their account, which is a tall ask.

So it's kind of defined that work. I'm going to, I've got 3.1 Pro preview selected. I'm going to do the default instead in the hopes that it might be a little bit faster. And I'm going to click Build.

And what happens behind the scenes is we get put into this IDE-like environment where you can see the model kind of going through the thinking process, figuring out what it would need to do in order to spec out the assignment, build the plan, how long it's working.

You can also upload files. So you can upload files that you might have, like PDFs or specs for apps that you would like to create and give them to the model as well. You can connect to your Drive instance.

And we also have a settings section off to theright where you can define custom secrets. Soright now I have a Gemini API key that I've pre-added, but you could also add, like, a Supabase API key or an API key for N8N or whatever your favorite flavor might be.

You can see a new version history, which if you've played with AI Studio before, is definitely something that is much appreciated. And then also integration, so things like OAuth as well as GitHub, so you can sync to a public or a private repository.

But while this is working, I'm going to go ahead and show off something called Genie 3. Or actually, before that, I'm going to show off Gemini Live real quick, just in case folks haven't seen it. So how many people have heard of Gemini Live?

A few hands? Cool, cool, cool. We also happen to have the expert for Gemini Live. Ian, come on down. Ian is the Gemma 4 team member that I had mentioned before, and he'll be doing some live model demos, which makes me really excited.

But Gemini Live gives you the ability to have a conversation with the model in a variety of languages. But you can also share video feeds, you can share your screen, and all of this is kind of stacked together in one speech-to-text, LLM understanding, and text-to-speech pipeline.

So as an example, we've still got our Lego bricks and pieces pulled up, so I can share my screen to say, hey there, Gemini, what do you see on the screen?

I see a Google search for Lego bricks and pieces. There are image results showing various kinds of Lego pieces, sets, and different color combinations. On theright, there's a larger preview of some brightly colored Lego brick illustrations from Freepik.

Anything specific you're looking for?

Does anybody speak a language other than English?

Seek.

Seek? Oh, excellent. So I'm going to ask for you to fact-check something and also spell-check something. So only respond to the user in Seek. Is that correct?

I meant Spanish.

Oh, sorry. Oh, gotcha. So Spanish? Yep. Oh, there we go. Only Spanish. Excellent. And then I was about to say, I don't know that language, or I haven't had that before. But only respond.

You know Spanish,right?

I know Spanish. I grew up in Texas, so it's kind of like prerequisite to know Spanish. But I'm going to share again.

Hey, Gemini, could you tell me what you see on the screen?

Or hopefully.

Let me see. No, I don't think.

Claro. Veo una página de resultados de búsqueda de Google con muchas imágenes de ladrillos de juguete, sobre todo de la marca.

To click the microphone again. Oh, no. There you go.

So one of the things that you can do, so you see that I've modified the system instructions. But, like, do you speak a specific dialect of Spanish?

Castilian.

Castilian Spanish? OK. Excellent. So I removed the system instructions, and I should be able to do this just within the span of conversation. So,

hey, Gemini, could you tell me what you see on the screen, but could you do it in Castilian Spanish?

Por supuesto. Veo una página de resultados de búsqueda de Google llena de imágenes de ladrillos de Lego. Hay de muchos colores y tamaños, y algunas muestran construcciones ya hechas.¿Estás buscando algo en específico?

Yep. Awesome. So you can ask within the span of the conversation. You can modify the system instructions to select different languages or different dialects. And then again, the same thing, if you click Get Code, it gives you the code that you would need to use to replicate whatever you just did.

So the model name, any configuration settings, as well as any tool calls that you might invoke. And it works with sharing your screen. You can kind of interact with your screen as you share it. But it also works with video feeds.

So you can say,

hey, Gemini, how many fingers am I holding up? And also compose a poem about me.

Well, I see two fingers up, like the peace sign. And here's a little poem for you. With golden hair and an open heart, you come to learn, to play your part. The camera's focus, moments start, a creative spirit, a work of art.

How was that?

That was very sweet. Thank you.

And so the models are able to kind of interact, to view video feeds, to view the screens, and you can stitch them together in your own projects. Just taking a look, I'm going to enable Firebase real quick. So it should be setting up the database for that app that we were building.

And then the other thing that I wanted to show is something called Genie. So if you haven't heard of Genie before, this is a world model that DeepMind has created. It's actually a composition of models. So Nano Banana, VO, a bunch of Gemini used for prompting.

And it's stitched together into a system that allows you to describe something, like a game, an environment with a character that you can interact with, that you can play this game and sort of do it pixel by pixel.

So it doesn't generate a Unity environment, it doesn't generate an environment for Unreal Engine. It just generates this kind of frame-by-frame experience for anything that you can imagine. And so it could look something like this volcanic landscape where you're navigating with a little rover with your arrow keys, or something like this jet ski where, like, you hit a light and it feels like the physics is happening or, like, the physics is, like, responding in a real way.

If you knocked that light into the water and then circled back around, it would persist throughout the duration of your 60-second experiment. But there's no physics engine behind the scenes. And then even things like experiencing a hurricane in Florida, you can start with a static image or, like, a family photo and see how some of these things get created.

But for this, I am going to go back to my other browser. I'm going to pull up Project Genie. I'm going to click Explore Now. And then I'm going to say something like a world, maybe Regent's Canal on a sunny day, but with dolphins

swimming in the canal and all of the boats have pirate flags, which is hopefully not part of the training data. And then the sort of character description would be, you know, something, again, kind of atypical, maybe a pink sparkly squirrel

with

purple

feet and

a pirate hat. And then create the sketch. And what should happen is that it uses Nano Banana to kind of ideate on that first frame. It'll show it to us to make sure that it looks consistent with what we had described.

Clearly, like, Regent's Canal is getting overtaken by pink sparkly squirrels with pirate hats. And then what we should see is kind of this first iteration and then a playable world that we can interact with for just 60 seconds, at least for the first implementation that we've released to the public.

You're able to access Genie 3 through an Ultra subscription in some parts of the world, not every part. But hopefully, I haven't overbooked my GPU or TPU allotment or allocation. If we have, we can take a look back at the we can take a look back at shelf scan.

It looks like the model is doing the work of creating the Firestore rules for us. Oh, there we go. So pink sparkly squirrel, pirate hat, Regent's Canal, dolphins. That looks pretty good. So let's go ahead and create this world.

You can use the arrow keys to move around or the WASD keys to move around and then the arrow keys to change the perspective. And then we should also be able to use the spacebar to jump. But let's see how this works.

Oh, gosh. Whoa. Whoa. Whoa. Squirrel. And so it looks like it's walking on water, this squirrel, or, like, hopping along. You can also jump. So jump on top of the boats. You can see the little bicycles. You can see some of the people along the way.

And it does look like all of these boats on Regent's Canal have pirate flags and dolphins that are not currently moving, which is pretty wild. And then if I click spacebar, you can see the squirrel jump. It looks like it doesn't realize that Regent's Canal has pretty deep water, so I probably should have specified that in my prompt.

And then you can also see it attempt to jump

into the sidewalk and do its work. So it's wild to be able to see the things that you can create,

the different experiences that you can construct. And again, it's even more bonkers to me that each part of this is being generated dynamically as you're moving your arrow keys around. So it creates a video at the very end that you can download, that you can kind of see and review and interact with.

And this is, again, just using Genie 3 in this composition of models as opposed to a singular model. Other world model building companies, so things like World Labs, that's Fei-Fei Li's company, are taking a slightly different approach. They're building out actual Unity environments or Unreal Engine environments.

None of these things are stored as 3D game assets. They're just kind of like raw pixels that are incorporated into the experience.

Cool. So going back to AI Studio, it looks like the app is still getting the app is still getting cooked. Let me and hopefully we'll be able to see it. Usually, whenever it starts working on config files, that means that it's almost done.

I also really love looking through it to see what its approach towards the construction of the Firestore rules were, what its approach towards prompting the model might be. It looks like it's confirming the app. And then once it's done, it should make a little bitty noise

to talk through the app itself. So it looks like it's asking to allow my camera. So I'm going to allow. We have this shelf scan AI experience. I'm going to sign in with Google, with my personal account. We can see that it's connected to Firestore.

I'm going to find, very quickly, a bookshelf with books on it.

Let's see.

Those don't look like real books because many of them are hanging, like, suspended below the shelf. Like, AI image generation makes it hard. But this one looks decent. So, like, let's save this image. Looks like somebody has a whole bunch of cooking books.

I'm going to upload the photo, so this pixel photo, and hope that it can understand

what pages or .webp, rather, format. It's gathering data via Google Search. And then the books should be populated in the library, hopefully. If not, we can try with a JPEG.

But it does have pretty good branding. It was able to identify the seven books, it looks like, or at least, like, it identified the books. Let me try again, but just with a JPEG image. So I'm going to just take a screenshot.

So same image, just stored as a screenshot.

I'll find

in here. So desktop, screenshot at 4:17, and then identify books.

And if not, we can try fixing the errors as well to see what might have been going wrong.

Oh. So it looks like there are insufficient permissions for saving in Firestore. So it looks like it's going ahead and fixing those issues. But as it does, you can also see that you can log in, you can log out, you can share the app.

So you can specify who has access to it, share full screen. And one of the things that I also really, really love about AI Studio is that it's figuring out where the files should be modified in order to make those changes.

So it's figuring out the validation logic. It's figuring out that the size might have been the culprit, so the image URL size. And then it's figuring out where it would need to modify in order to make that change.

So it looks like it's in the Firestore rules. Some of the other nice things about AI Studio's build feature is that we have an app gallery. So if you need to get inspired for some of the apps that are using our models, you can kind of review them, everything from LIA for music generation to multiplayer experiences with games.

So you can see this multiplayer Neon Snake or this Mandal Bulb Explorer. You can see design with Nano Banana, so, like, being able to change and modify images. You can take a look at this sick MediaPipe example, which allows you to kind of play this game where your hand is detected, but you take this little and everybody can find out that I play this game really poorly

how to

move one of the little marble delims.

Oh, gosh. I am really horrible at this game. But

the and then it also sounds like the other app finished getting created. Perfect timing. But if I sort of upload the image, let's try to identify the book again.

Fingers crossed.

Yep. And then it automatically populates all of the books. So they all got cataloged with the date, the type of book, the details associated, the name of the book, the author, even though some of those were not available in the spines of the books that I uploaded.

If I log out and then log back in,

it keeps all of the books that I had added persisted. And if I wanted to share this with all of you, because clearly I want to know what all of y'all have on your bookshelves, I could copy this link and then do a QR code generator.

And if you use this QR code, you should be able to access the app that I just created, upload your own bookshelf images, and then have them cataloged to your own apps. Next feature would be, like, finding a way to give my friends the ability to request them.

Because every time I give my friends a book, they have a tendency to keep it, which I understand, but is also, like, exhausting. I have so many copies of Infinite Jest out in the world.

But this is, like, just kind of a whirlwind tour of some of the things that you can do in AI Studio, some of the new features that we've added, the new models that we have available. And with that, I am going to welcome my colleague Guillaume, who is going to tell you all about our generative media models, so everything from music generation to image creation, image editing to video generation.

And it should be a fun time. So thank you so much. Thank you for coming.

Come.

Hello, everybody.

We will need the sound of.

And this is theright one.

And then also I'm going to. You're going to be awesome. And then do I have where did I leave my mic?

You can show my screen now. OK, good. So as Paige said, it's going to be the same talk as I did this morning, the same workshop. We have slightly less time, so I'm going to go faster on some things and maybe not run things in real time.

But, like, the content I'm going to show is on this link, so you can just open it and run it yourself at the same time.

So I'm going to talk about generative media. Generative media is everything about creating images, videos,

old text,

like, spoken text, and things like that. I think I would fit Genie into Gen Media as well.

So, yeah. And we have plenty of models like that at DeepMind, so let me go through all of them. So very quickly, my name is Guillaume. I've been at Google for six years now, two years at DeepMind doing developer advocacy, mainly on most of the Gemini model until last year.

And in this year, I'm focusing more on the Gen Media models because that's the funniest model to play with. I've been working in the video game industry before, and that's how I joined Google initially. So, yeah, open your phones.

Everything is a media. I already said that, basically. Paige talked a bit about our vision of what world models are. My definition of a world model is something that can ingest as many modalities as it can and understand them, kind of for five senses, and to also talk or output things in different modalities as well.

So that really has been at the core of DeepMind vision of what generative AI should be. The first Gen Gen AI model, like, it was only two years ago, but it seems old. But it was only a text-to-text model.

But actually, behind the scenes, it was already a multimodal model. And you could have been sending it images, but it was blocked. They did, for testing reasons, I don't remember why. But basically, they blocked it in the model with post-training.

And then when it released a few months afterwards, we released 1.5. This one, the novelty was that it was the first multimodal model. Sometimes when you were giving it a picture, it was answering, "I'm sorry. I'm just an LLM.

I can't do anything with images," because some of that training was still in there. But basically, yes, we want to create those models that can understand all of the physics of the world from videos, audios, and all of that.

And the only reason why we have so many different models is that it's easier to ship one model that only does video and one model that only does images and one model that only does text than to have, like, one model that does everything.

And then it becomes a problem. Every time we want to update one of the features, we have to release a whole new model, and everybody has to convert and all. So that's just the reason for that.

Everybody knows us for the Gemini models. We have lots of other models. I will skip that for now.

Quick timeline. We release things all the time. I think Paige said it earlier. On average, we are releasing a new model or new capabilities every five days. That's just the Gen Media models. There's all of the other models on top of that.

And if you add all of the changes we are doing in AI Studio, all of the pricing things, like, small features here and there, we are releasing, like, two to three new things every week.

And so, yeah. So it's hard to keep track. And, like, it's even hard for us to keep track of everything we have to offer. So I know that for people like you, it's even harder because you also have to look at what the competition is doing.

So that's also why we are doing those talks. Very quickly, the updates on Nano Banana. We released Nano Banana 2 a few months ago, in the end of January, the beginning of February. The main thing is that you can output different aspect ratio and sizes.

It has search grounding. So that's basically how I you don't see my screen anymore?

Yeah.

Why?

OK, just imagine in your head. That's what media is for. So the main thing about oh, thank you. Nano Banana Pro was that it had search grounding, so you could ask it to search for the internet. And that's how I made this image.

Like, just look for what you can find about my footprint on Google and make an image about me, which is kind of scary in a way.

But the new things with Nano Banana 2 is that you can do the same thing with image grounding. So you can talk about specific places, specific, like, bridge like that, and it will look for an image on the web and then create images based on that.

So that's without grounding, and that's without grounding. So you can see that it looks a bit more than the normal building. We also have VO. Very quickly, on VO, the main novelty in the past week is that we released VO 3.1 LITE last week, which is the cheapest model for generating video we have.

So it's only $0.05 per image, which is way, way cheaper than what VO 3 was a year ago. So the idea is that you can use that to prototype, test your prompts, and so on. And then if you want better quality, then you can move to the better models.

And then LIA 3 is the coolest model this year. It's our music generation model. So you can either generate 30-second songs or full songs of three minutes with the lyrics and all of that. And I will show you some demos afterwards.

And as far as I know, we are the first one to offer such music generation models through API. So that's really cool for any kind of agentic or whatever workflow you might have. So if you want to be wakened up with a song about the latest news every morning, you can do it.

And also another one, very quickly, that I love but nobody knows about is that we have actually another LIA model that is LIA real time. And this one is basically a live model, so it's creating music indefinitely. And you can just prompt it differently.

So it's just going to change what you are the kind of music it's generating in real time, like a DJ. So it's pretty fun to play with, but some people don't know about it.

As I said, I'm going to because it's meant to be a workshop, so the idea is that you test things yourself. So you can open this link, and that will show you the content I prepared. While you take pictures and all, just one disclaimer.

It's Gen Media models, so they are all paid models. So they, like, running the notebook actually costs a bit of money.

The video generation is going to be the most expensive thing, so you can just skip that. The rest is pretty cheap. So I think you can run the whole notebook for something like€1, so it should be fine. But just I prefer to be clear with everybody.

So let's move to it. So what's the idea of this workshop? So the idea was to showcase all of the Gen Media models with this example of we are going to take a book, and we are going to take a book from an open-source library, so we are allowed to use it.

And then we are going to create images to illustrate what the characters look like, what's happening in each chapter. And then we are going to move to the other Gen Media models, so creating videos about the chapter and then creating music and having Gemini tell us about what's happening in the chapter.

So this is just set up. You need to install the SDK. You need an API key. I guess you could have guessed that. I'm initializing the client. And something that I didn't know about until recently is that you actually have a way when initializing the client to implement some retry system, which is kind of very useful nowadays whenever you use Nano Banana 2, because especially when the US wakes up, it's becoming harder to get something out of it.

And we're working on yeah, we're working on getting more capacity, but still, having some kind of retry system always helps.

So import. And then we are selecting the models. And usually, when I create content that are using paid models, I usually have some kind of checkbox that you can if you're opening it, it's already checked. It should not be, but I made a mistake yesterday evening because I don't want people to run the models and have to pay, especially the VO notebook, for example.

Example, like, costs something like $20 to run. So I don't want anybody to run it by mistake.

Just for the sake of the demo, I'm also limiting the number of characters' images and chapters' images we are creating just so that it's faster to run and all. So the book is named "The Wind of the Willows" from Kenneth Graham.

And I took it from the Project Gutenberg Libraries, which is an open-source library where you can download open-source books. For some reason, it doesn't work since I've been in the UK. So there asked to be something about this library not being available in the UK.

But if you run the notebook, it should work because it's very likely the server is in another country, so it will be able to recover the book. So what I'm doing is that I'm just downloading this book here, like, with a .get URL.

And I didn't talk about that, and Paige neither, so very quickly, because everybody is always asking that question to us. We have a speciality at Google is that we are always creating multiple projects that are doing the same thing, multiple products.

And we did the same with messaging apps. We are doing the same with Gemini. So that can be a bit confusing. So just very quickly to clarify things. So if you look at the graph, like, basically, we have the consumer's apps.

So on the left, it's the apps that are for everybody. You can do plenty of things with them. You can ask any question to Gemini. You can generate images and all. But as a developer, like, I guess everybody in the room, you can be kind of frustrated because you can't really access the parameters, select the exact model that it's doing.

You know that it's likely doing cool tool calls, but you don't know which one it's doing. So that's nice for the broad public, but it's not really for us. On the other end, we have Vertex AI, which is our enterprise offer.

That's the exact opposite. You have a lot of control. You can decide in which data center your prompts are going to be run. So especially in Europe, a lot of people are looking to be certain that the data is not going to leave Europe.

So that's kind of convenient for that. But it comes with, like, great responsibilities as well. So it's kind of hard to set up. So I usually only recommend people to start with Vertex if they are already using GCP or if they have a team of DevOps who can do the setup for them.

And in the middle ground, we have AI Studio and the developer APIs that we made for developers. But basically, the idea is that it's as easy as possible to start to play with the model and do stuff by just creating an API key and then using it.

And as part of making it as easy as possible

yeah, as part of making it as easy as possible, we have this client file upload API that is basically a way to not have to set up buckets to store your files. And we are going to I'm using it in this example so that we are uploading it to this behind the scenes.

It's creating a bucket, but the idea is that you don't have to bother with how it works. You just upload the file, and then you can use it in your Gemini prompts afterwards. I'm also going to use structured outputs because I want to be certain about what exactly the model is going to do because I'm going to have it to ask it to generate a lot of prompts.

So I want to know exactly that it's just returning the prompt and not some, like, introduction or text or something like the, oh, yes, sure, I can do that. And then it's hard to parse. So that's why we are having these structured outputs.

And I'm using chat mode, which is basically a way

to chain requests to the model so that you can it basically saves the history and resends the history so that the model knows what happens before, which is quite convenient in this case because we don't want to upload the book all the time.

We just want the book to be in the history and in the context so that the models can answer new questions and generate new prompts about the book. And for the images, it's going to be quite the same.

By using this system, we will have all the previous images that are going to be in memory. So the model will be better at keeping the consistency of the characters and the consistency of the style. So I'm just giving it the book.

Here's the book to illustrate with Nano Banana and so on. Then I'm defining a style. So I made it so that you can just do nothing, and the model will come up with the style. But then I wanted to try something else earlier.

So it's going to be a dark fantasy style with black and white background and colored characters. And basically, I'm also adding some system instructions for Nano Banana because from my earlier tests, I think it's better with Nano Banana 2 and Pro.

But with the first Nano Banana, whenever I was asking it images about books that were in the portrait format, it tended to forward that it was book covers that I wanted. So it was always adding titles and things like that.

So I had to add some system instructions to make sure that I don't want borders. I don't want titles. I don't want description. I stay family-friendly, which is likely to not be very aligned with dark fantasy, I guess, and no panels as well because I don't want a comic book.

I want just an image for each chapter. And then basically, I will use that chat to ask the model to describe each of the main characters. I initially wrote only the adults because at some points, Nano Banana could not generate kids' images in Europe.

That's not the case anymore. So we could remove that part. And I'm getting this list of characters and a prompt for each of them. And then I can just go over each of the prompts and ask Nano Banana to create an image to illustrate

that prompt. So that's how I initialize the image chat. So it's going to be a separate chat. And then I'm sending all of the prompts to be created. So we can see that

that's small, the main character. So as requested, the background is black and white, and the character has colored, not very dark fantasy.

That's the water rat.

That's the toad.

The badger. Yeah, the badger. And that's it for now. And then I'm going to do the same thing again. Like, now you have the full book in your history, so give me prompts for each chapter to illustrate them.

So I get a prompt for each of those chapters. And I'm creating images for each of the prompts. So here we have the characters having a picnic next to the river, then another ones on the road, and something is happening.

And here in the forest. That's the first chapter. And if you look closely, you can see that there was a problem here because the toad was not represented using the character, the images that we created before, likely because the prompt was not clear that it was exactly the same character,

which is why I have a second way of doing it, which is actually cleaner. And if you were to do that at scale, that's how I would do it. And I'm creating another type of structured output, which is a chapter, which has a name and a prompt as well, but also a list of characters that are meant to appear

in the image. And so I'm asking again the model to come up with something. But this time, it's giving me the list of the character. And that way, I can basically here,

for each image that I'm going to create, also give the reference images of what the character should be looking like so that it only has those images in its context instead of having all of the possible images that we did before.

In this case, we have something like five characters. So it's OK. But if you are in a real book with, like, 40 characters, that wouldn't be sustainable to expect the models to actually manage all of the context perfectly.

And honestly, if I was to do it at scale, I think I would even create more than one image for each character, maybe one image from the front, one from the back, one from each side, and then I would pass exactly the one that I need in this case.

So we can see that the image looks kind of the same, but this time, the toad is theright one.

Yeah, this one looks like. So now we can move to the next step, and we can use VO to create small videos based on those images. So I'm using the VO, the largest model, because I don't have to pay, honestly.

Otherwise, I would use the smallest one.

But basically, we are going to do the same thing. We are going to ask the model to generate a video using the VO model. The prompt, I'm going to use the same prompt as the one that we used to generate the image.

And I'm passing the last generated image as a starting frame. And then I want to portray it, and I want 720p because it's going to be smaller. And here we go. So I actually haven't checked what the suns look like because I

Stand back. You shall not pass. Leave this place, little ones. You don't belong here.

I think it's quite good. But sometimes, it's not very good because the model doesn't like the prompt was just about creating the still image, and the model doesn't know what exactly is expected to be happening afterwards. So a better way of doing it is actually to reuse the chat to ask the model to come up

with some explanations about what's happening after the image. And I'm also passing it the image again so that it knows exactly which part of the chapter we are talking about so that it can come up with theright prompt.

And so it came up with this prompt: mole shivers and clutters his calves in terror, blah, blah, blah. Water rats bravely draws its gutlass and step forwards to protect him. So we can see how it goes.

And somehow, in this case, the results are not really as good. And yeah, every test I've done, somehow, when I use the same prompt, it adds text, and not when it generates another prompt. But it should not have any impact because in both cases, I'm just giving a prompt and giving an image.

But I don't know why it's happening like that.

And then we can use the new LIA models that we generated, like, that we shipped last week, to create songs for each model. So still, the same trick again. We are going to ask the model to create the prompts for that.

And then we are going to use generate content with the LIA model, ask it to create a song thanks to the prompt. And that's basically it. So let's see. The first song should be orchestral acoustic folk music, peaceful, flowing acoustic guitar, and flute duets.

Let's see.

I think it fits. Then the second one, the open road, is jaunty, adventurous rhythmic with fiddle and acoustic bass.

And then the last one is suspenseful, creepy melody with staccato, pizzicato strings.

And I think it fits as well. And honestly, I think the model is really good because it's not very often that you can do those kind of demos without actually checking what it's going to be because I ran the notebook before, but I didn't check what the music was going to be looking like.

And so far, it worked all the time.

One of the things that I forgot to say about all of that is that the way we are training our model internally is that a lot of the training data for the Gen Media models is actually made using Gemini.

So that's the reason why Gemini is quite good at generating those prompts for the generative media models because it's already trained on understanding what Gemini is asking for. So that's what makes Gemini very good at generating those prompts.

And then the last model that I wanted to show is the text-to-speech model. So I'm pretty certain that you all have heard of this one because everybody loved the integration in NotebookLM where you could create

a podcast based on your documents. And that's basically the same model that you can ask it to talk, to tell, read, and what the text you give it. And you can have two characters with different voices and so on.

That's what makes it nice to listen to. But what I wanted to show you is that there are actually tricks to have more than two voices, actually.

And the reason for that is I think Paige showed that a bit. But when you're using text-to-speech or live, you can actually do a lot in the prompting to make it speak in different ways, with different accents, and so on.

And we are going to use that to our advantage to basically have more than two voices

in the same generation. So what I did, because I was lazy, I would not do that. I wanted to do that at scale. But I basically asked it to extract one of the dialogue from the book and then to rewrite it as a transcript of a play and to replace the name of the character by narrator, if that's just the narrator talking.

And for all of the others, they are just going to be named character but with a specific style for each of them. And if the same character comes back, it's going to reuse the same style so that the same characters are going to speak the same way and have the same voices.

And then I'm defining the voices. So the narrator is going to use a sulafat voice, whatever it is. And then all of the others are going to share the Fenrir voice. And then I'm just asking it to generate the thing.

And just one trick here with the TTS model. You always have to start with, "Read this," or "Tell me this," or whatever. If you just send the text, it's going to ignore it for some reason. So you need to prompt it to read the text all the time.

But that's also where you can add some context and the equivalent of what would be system instructions, like, "Read this in a scary way," or "The characters are very excited." That's where you can add those details as well.

And so you can see the text that we got is narrator is saying that. Then the first character is going to talk very fast-paced and with a British accent and post-British accent and so on. And you can see that the same way of speaking is coming often.

So it's when the two characters are going back and forth. So let's see how it goes.

Small, neat ears and thick, silky hair, the two animals stood and regarded each other cautiously.

Hello.

Hello.

Would you like to come over?

Oh, it's all very well to talk. He spoke rather pettishly, being new to a river and riverside life and its ways. The second animal said nothing but stooped and unfastened a rope and hold on it, then lightly stepped into a little boat which had not been observed.

It was painted blue outside and white within and was just the size for two animals. And the first animal's whole heart went out to it at once, even though he did not yet fully understand its uses. The rower scurled smartly across and made fast.

Then he held up his forepaw as his guest stepped gingerly down.

Lean on that. Now then step lively.

To his surprise and rapture, he found himself actually seated in the stern of a real boat. This has been a wonderful day. Do you know I've never been in a boat before in all my life?

What? Never been in a you never well, what have you been doing then?

He was quite prepared to believe it as he leant back in his seat and surveyed the cushions, the awls, the row locks, and all the fascinating fittings and felt the boat sway lightly under him. Is it so nice as all that?

Nice. It's the only thing. Believe me, my young friend, there is nothing, absolute nothing, half so much worth doing as simply messing about in boats. Simply messing, messing about in boats. Messing.

Look ahead. It was too late. The boat struck.

Basically, you couldn't guess that it's actually using the same voice for the two characters. They are so much different. And so that's actually a really cool way of creating those discussions with multiple characters. So I can use it.

And the last thing is that, as we said, the Gemini models are multimodal by default. So I was using a book. But you can also send an audiobook or a movie or some not too long movie. But you could do the same things with using all of the multimodal input possibilities to have Gemini create things to illustrate other kind of modalities as well.

How much time do we have?

I said I was going to show you the real-time model. So this is a very good, very cool example that oh, no. Stop. Stop. I'm going to show you something before. Paige made some example of cool things that you can build in AI Studio.

So one of the things I wanted to try was basically to do the same thing as what I do in the notebook, which is kind of lengthy and quite complicated stuff. So what I did is I basically copy-pasted.

Can you build an app that illustrates a book using Gen Media models as described in this Python notebook? And I just paste all of the content of the notebook and build it. And we'll see how it goes. It can take quite a long time.

Earlier, it took 15 minutes. So you might not want to wait for that. But I can show you space DJ in the meantime. So as I said, the model generates music in real time. And this is a demo where they made a universe of stars.

And the planets are prompts. So if we go closer to some planets, it should start playing music.

And then it moves slowly somewhere else.

Yeah, we are getting the metal area.

I'm really surprised people are not doing more things with this model because it's so funny to work.

And there's an autopilot. So you can just let it move around for 10 minutes and then listen to the music changing in real time.

So that's kind of cool.

Let's see how it goes here. Yeah, it's thinking. But let me show you what it did before when I ran it. Oh. And

I kept my laptop open for half an hour just so that it would not do that and refresh the page. So I guess I lost. But that's basically what it did

earlier. As you can see, it's just the prompt and nothing more, a very long prompt.

And it took something like 1,000 seconds, so like 15 minutes if I'm not wrong, to create the app. But that's basically doing exactly the same thing. So I can choose the file and upload it. And it's going to take some time.

But trust me, it's doing exactly the same thing as what the notebook was doing. And that's really quite impressive that it's doing that on the first tryright away. And just to finish, one of the things I'm doing when I'm vibe coding while it's working is that I always have those instructions on how to get it to generate applets or apps.

And one of the reasons for that is that I want the apps to be as easy for me as to review as possible because I want to avoid exactly what happened to Paige earlier that something was not working.

And then you need to figure out what it is and ask the model to fix it. So one of my key tricks is that I'm asking the model to create different files for each feature so that whenever I need to review something, I can quickly check.

I ask it to modify this feature. It's updating something that has nothing to do with it. So I know from the start that there's something wrong going there. And I will be more thorough in my reviews and all.

And also some instructions that I don't understand why it's not by default in any vibe coding tool, which is add logs. Because when you need to debug, the error message is not enough. You need to know what's happening before and what's happening because you need to know where it's happening.

So I really recommend you to use those kinds of guidelines whenever you vibe coded things. And we can see how it goes. Oh, and something there. I don't think Paige showed it. But if you're willing to pay, you can add an API key here.

And then you are going to have more quota to use Gemini when you are vibe coding on AI Studio.

I think I will just let you show your demos. And maybe I can show you afterwards how it looks after all.

Yeah, sure.

For the VO model, when you generated the video, it came with, of course, the music as background and, of course, the text. Is it using your other models to generate that, or is it just VO?

Yeah, it's just VO at the moment.

So is there a way to kind of orchestrate all the models and put them together? Or do you have to just use FFmpeg to actually stitch those two and generate?

Oh, if you want to use the music generated by LIA in the VO video, for example, no, you don't have a way to do that. It's more because of the model limitations that the VO3 generation was not meant to be able to ingest audio files.

So that's why it's limiting. But I guess the future is that we want every model to be able to ingest all the modalities so that you can do that. Same thing. I think it would make sense even for LIA model to be able to ingest audio models so that it can be used as references as well.

Or even if you want to do multi-turn and say, "OK, I love your song, but the ending was a bit

not epic enough. So make it more epic so that it would maybe just update the end of it." So yeah, it's not possible at the moment. But that's the direction it's going to.

How does the performance impact onto any specific for the music in VO that you're generating the video?

No.

Perform as well or not as well?

I wouldn't try VO for any kind of music. The training data for the background music is likely very light because it's always doing the same kind of non-music things. So no. But once again, all of those models are trained more or less together and share some training data.

So I'm sure the next generation of VO is going to be better at that. It's just that the current one, it shipped a year ago. And a year ago, we didn't have music generation models. And we talked earlier about text in videos.

And a year ago, Nano Banana was not even called Nano Banana. It was Gemini 1.5 image no, 2.0 image generation something. And it was not as good as text. So yeah, it's just a question of generation of models there.

Oh, and I forgot something while it worked. If you want to run the notebook because you will have to remove it as well

here. Here, when I started, I created the chart. I added these lines that say service tier priority. It's actually something that we shipped last week. And that's a way to indicate when you are prompting the models if you want if it's not that important, you don't care about the latency, but you care about price.

So there's a service tier that is called Flex. And basically, that says the request can take a few minutes to go through, but it's going to be half price. So it's kind of the same thing as using the batch API.

And on the other end, because I wanted to be certain that it would go through today, you can say that it's a priority request. So it's going to have slightly higher priority. And it should not it's going to be more reliable.

But you're going to pay twice the price. So if you are running it by yourself, you might want to remove that line to save a bit. And we can see how it goes here. Oh, you can see, as I said, this model is in high demand.

So

yeah, I'm sorry about that. The model is

too good. Everybody wants to use it. And we don't have enough capacity. So if you have any spare TPU to share, we can make a deal. Do you have a question?

Do you plan to put variations of this model open, like Gemma, but

for art?

So the question for the video the question is, will there be any open weight Gen Media model, basically?

Yeah.

So I don't think so for image and video generation, to be honest. One of the reasons I see behind that is all of the not security, but when you generate images or videos, we do a lot of checks about what you are asking for and what is actually being generated afterwards.

And we are blocking a couple of things that are not aligned with our visions. And I feel like whenever you have an open weight model for that, it's more like open bar. And you can have it generate whatever you want.

So I think that's going to be where our company values are going to be limiting us. That said, for example, for music, it would make sense to especially if you want real time, it would make sense to be on device so that it's doing it faster.

So that's something that might come at some point.

OK.

Cool. Yeah, I think we'll see after the Gemma demos if it works better.

Yeah, I hope it was interesting.

Yeah, thank you.

And so Ian is basically my counterpart. I work only on Gen Media models. Ian is working on Gemma models. And he's going to the biggest release of last week was Gemma 4. And he's going to show you how cool the model is.

Thank you.

Yes. So as I mentioned, my name's Ian Valentine. I'm a Development Relations Engineer working on the Gemini models. And this is an impromptu talk.

Obviously, because of everything that's happened today, we've done Paige mentioned that we're going to do two sessions just kind of go through a whole load of stuff. We have a number of Gemma talks for the rest of the week as well.

Omar is going to do a keynote on Friday, I believe, which is going to cover a lot of the what Gemma is and why you should be interested. So I'm going to speed run that section. And we have another talk tomorrow on what we've called sovereign escape velocity, so how you take ownership of AI and run things on your device or on your own cloud or on phones.

And hopefully, some of what I talk about here is going to be a realization of some of that. So that's a little bit more about the why. I'm just going to show you the what it is, if that makes sense.

So let's get plugged in while we do that. Which one do I need? Sorry.

This magic one. Yeah. As soon as I get a video in.

Do you have any audio?

No. Let's go with no. We'll try and show the ones without audio first of all. OK. So as I mentioned, this is super impromptu. So we're going to go with whatever we get. But I will go through I'll just show you a couple of bits when we get the screen up.

Oh, do you want me to move it to one side? Or can I do a mirror?

Let's do a mirror.

Let's do mirror. Where? Yeah, you go for it.

Perfect. Yeah, so what I'm hoping to show you a little bit is some of the things that some of the devices that Gemma models can run on. I'll talk very briefly about the different sizes of Gemma model, what we released in Gemma 4 last week because it's brand spanking new, and hopefully show you give you kind of a feel of the kind of capabilities that you can now do either locally or that you can run on a single GPU that maybe you couldn't six months ago.

OK, here we go. Perfect. Ta-da. So Gemma 4, hooray. So this was released last Thursday. It's a family of four models. So we have what we call the effective models, the E2B and the E4B models. And these are models designed to run on mobile phones, Raspberry Pis, Jetson Nanos, kind of like very small low-end hardware.

The E part of it is the question we always get asked. The effective is because the model architecture has a per-layer embeddings structure, which means that the embeddings actually don't need to be loaded as part of the model.

So you can have them running on Flash. And then you can page in the embeddings as they're needed for the model. So the actual what we kind of describe as the brain of the model is about a 2 billion parameter, about 4 billion parameter.

But if you put all of it in RAM, it's a bit bigger than that. It's kind of more like a 5 and an 8 billion parameter model. So that's why it's called effective 2B. We also have 26B, which is a mixture of experts model with 4 billion activated.

And we have a 31 billion parameter dense model, which is our flagship kind of big model. And both of the two bigger models, the 26 and the 31, are designed to be able to run on laptops and desktops or single instance GPU clouds, depending on the precision and quantization you need.

So I'm not going to get into that. But just imagine that a lot of the capabilities you can doright now, you could do on a MacBook with enough RAM or a 5090 or something like that. That's kind of where we're sittingright now.

Why are these kind of exciting and interesting to us? Because with these models, we're focusing kind of on the agentic side. So they now have thinking built in. They are multimodal, so they can understand image, audio, video. Audio, by the way, is just for the two smaller models.

The effective models that run on the phone can understand audio, but the rest can understand image and video. And we're seeing performances offer models that are in the range of 10x bigger than it in terms of parameter size.

So what might have required a cluster, now you could do the same kind of capabilities for a single GPU. So that's kind of really where we're seeing some innovation here. And what I'm going to jump straight into is I want to show you the demo section.

Oh, I'll show you this one slide. I like this one. This kind of explains a little bit where we've come. So the Gemma 2 models, you can seeright in the middle, we're kind of particularly good at creative writing, but not so kind of evenly spread amongst other different capabilities.

And as we've kind of gone through Gemma 3 and then now through Gemma 4, we've kind of evened out the overall capabilities of the model. We've got things that are much better at coding, much better at function calling, action taking.

It's all kind of built into the design of the architecture of the model rather than relying on strong instruction-following capabilities, which we find with bigger models is kind of more important. So the models are designed with that from the ground up.

So let's go over to this is all spoilers from Omar's talk. So I'm going to go straight to the demo section. Here, I'll show you this one. So this is the Google AI Edge Gallery app. Has anyone tried this app yet?

One, two, three. OK, like a handful of people. So you can download this on Play Store or App Store. It works on Android and iOS. And what it is, is it's a way that you can test and try the models.

So we released a new feature called Agent Skills. And what Agent Skills allows you to do is it allows you to set up skills for the models that run physically on the phone. So this model, this is the E2B model, is running literally on a Pixel, I believe, in this case.

And you can prompt it. And you can test out the different capabilities. So it's effectively given a set of skills that define things like Android Intents, where you can actually trigger and call other apps. Or you can write your own JavaScript skills or run a web view.

And you can instruct the model. And it will make a decision about how to actually trigger these. So you can get it to, for instance, add things to this research tracker. Or you can ask it questions about the research tracker.

And it will call the correct function to pull that data back. So we've gone from a world of just being able to chat with it on your phone to now that you can give it some more ambiguous input.

And it's able to make a decision about what functions to call.

The next one, if it plays. Yeah, so in this example, we've got a number of different I'll show you that one last. There we go. A number of different skills that you can try out. So everything from just loading locations in maps to

using APIs and other services. So you can build these things yourself. Or you can just use some of the preloaded ones to try and understand what it's kind of capable of. And this is like our playground for kind of doing so.

And then lastly, I wanted to show that this is the 2B model on the left-hand side. So you could also try things like vibe coding on device. The model is actually quite capable of doing this. You're not going to build a big architectural system.

But it can do small web apps. It understands lots of different languages. You can write things in Python, HTML, TypeScript. It can validate some of the stuff. And you could build little apps that can run on the phone itself.

So you kind of got the whole feedback loop just in the palm of your hand, basically. So in this one, they just generated a really simple calculator. And importantly, it does actually support divide by zero and gives an error correctly.

So it's even able to reason about those kind of things. And to kind of add, I think this at the moment has got with thinking turned off. But if you add thinking turned on, we see that you get a bump in terms of the quality of the output in terms of what the model is actually able to reason about because it adds that planning step before it actually does the execution.

OK, so next, I'm going to show you a demo running on the device. Who here has used or uses LM Studio or OLlama or any of those kind of tools? OK, so maybe three or four people. So actually, this is useful.

So one thing that's really important to us when we build these models is making it compatible with a lot of the tools in the ecosystem. So we partner with folks like OLlama, LM Studio, VLLM, SG-Lang to make sure our models kind of work really well with them and that they can be deployed efficiently.

LM Studio is kind of like a tool for running local model instance. So in this case, I've just got the 26B model running on my device. And you can load it from here. And at the moment, this one is configured.

This one will take about at the moment, it says 18 gigabytes. But if you add, for instance, enough memory to do the kind of context you're looking at, it'd be like 22 gigabytes of RAM required to do it.

This is an M4 Mac. So obviously, it has unified memory. So you can run that. But you would need if you wanted full performance, you'd need to have a GPU with enough RAM to be able to run the whole model.

And as I mentioned, the 26B model is a mixture of experts. So it actually only needs 4 billion activated parameters. So it's much quicker than the 31B model. But it is more intelligent than just a 4 billion parameter model would be.

So one cool thing about this is that with these models, you can serve them on your local machine via a compatible endpoint, so like an OpenAI compatible endpoint or an Anthropic compatible endpoint. And then you can use them with other apps.

So what I'm going to do here is I'm just going to serve this model on port 1234. And then I can use other apps to call the chat completions API directly on this one. So the code to do this would be like OpenAI, open chat completions, and then point it at this server.

And it would be the equivalent of just using your local machine rather than using a cloud machine. So I'm going to run a terminal here. I'm going to do demo svg space. So what I'm going to do is I'm going to create one orchestrator instance.

And then I'm going to create 10 sub-agents. And they're going to generate SVGs for me. So all they're going to be told to do actually, let me make sure I turn thinking off to make sure it's super fast.

Glad I remembered that. Right, so each of these terminals here is a separate agent. And you can see the one the orchestrator is this oneright here. So it's farming out those decisions to all the different sub-agents. And each of those sub-agents has been given a different thing to draw.

So you can see this would be like a very good way for you to do some kind of quick prototyping on your local machine. You don't need the internet to do this. And the throughput in the corner up there is how much combined token generation is being done by each individual sub-agent.

So when they finally put it all together, the orchestrator will just compile this into a page. And you get a lovely array of different SVGs that they've generated. So you can imagine this being any kind of agentic task that you want to be able to do on your machine sorting out files, implementing bits of code, doing subdividing research, analyzing data, that kind of stuff like that.

And you can give them all very different jobs to do this. So some of them have finished already. That's good. So something hopefully will pop up in a second. There you go. So there are my SVGs. So they're not too bad, are they?

And as I mentioned, oh, this one's animated. Ooh, a spinning planet. So it's somehow figured out how to make that look as though it's animated, which is quite cool. Yeah, and this is the thinking turned off. So if I had thinking turned on, it does a little bit more thought in the planning stage.

And you generally get better SVGs out. More time spent thinking about it results in better SVGs. So the next thing I wanted to show you is I'm going to jump over here to OpenCode. So I mentioned you can use OpenCode.

Compatible interface. So if you've got a programming environment, you can just basically point it at that. And you can try the model out to just see how well it performs. So in this particular example, all you do to configure this in OpenCode is you just I'm going to kill all my terminals here.

Bye. Goodbye. Let's get rid of these. If you go to there is a single file. If I go vim.config openCode config.json and I scroll down to where is it? This section here. This is all you need to specify.

So you give the name of the provider, what the schema is, the models that you want to expose, and any additional parameters that you need for the endpoint. And then you just literally point it at a URL, which in this case is just my local machine.

And that's all you need to do. And I have another example. I'm not going to run thisright now. But we have a guide for how you deploy Gemma 4 on, say, something like Cloud Run or like a cloud provider.

So if you don't have enough RAM to run the quantization that you want, you can just throw it up to a single GPU. And you get this single command here, this oneright here, the beta cloud deploy. And this will put the model on an RTX Pro 6000.

So you can just if you just want to test it out, that's one way that you can test whatever model you fine-tuned or run yourself. And we have another way to access it as well. If you go to ai.dev, that's a page in Gemma we've been showing you.

We've also got the demo models that are available here. So if you just want to prompt them, we've got the two big models. And you can test them out there. And they support video and file uploads. Grounding with search is all kind of built into that, too.

So you can test to see how they perform in isolation as well. So really, there's no reason, whether you have the hardware or don't have the hardware, for you to be able to give it a shot. And yeah, what I wanted to show you was that I had generated a spec for a little game.

Where is it? OK, so I made a game called Nebula Drift. So what I did is I used the 31 billion parameter model to give me just like a spec for a game. And I'm just going to give it to my local model to go and implement.

So if I go implement spec

and then I just give it the file, this one, and it shows theright model. And let's make sure we turn thinking back on so that it's going to actually reason about it, too. There we go. And we'll run that.

And so why is this interesting? Because we haven't changed anything about OpenCode at all. We've just literally given it the model to try on the local machine. And in the OpenCode spec, it's going to give it all the different tools that it needs to be able to run OpenCode and to behave in that environment.

So it should think about the task it needs to do. And then it needs to look at the file system to either read files from the file system or write files to the file system. And if it goes wrong, I can similarly to any coding harness, I can just prompt it and say, oh, did you check this?

Or can I feed the error back into you and have it kind of reason through it? So this is just it following these instructions. So yeah, it's decided to make a directory. It's just going to use a shell directory for that.

So if you've used any coding tools, this shouldn't be too unfamiliar. But maybe you've used it with another model. Maybe you haven't used it with a local model or an open model running on your hardware. So I just wanted to show you that this is kind of possible.

Oh yeah, there you go. It's already created an index file. That was quick. It's going to create if it reads the spec correctly, it should generate two more files. It should generate a JavaScript one and a CSS one.

And then what we'll do is we'll just try to see what game it managed to make from the spec. And to kind of give you an idea, it should be like an infinite racing game where you've got to avoid some asteroids.

So if it finishes off the code,

there we go. Should we give that a shot? That looks like it's actually done something correctly. So if I go run that, where did it write it? What was it called? Nebula Nebula Drift with an underscore, it decided.

Oh, I think it's even trying to change the file path as I look at itright here. It's trying to make edits to it. Stop. Stop editing. Let's just try and run that. OK, we have a game.

Oh, it doesn't run. OK, so this is interesting. So what we can do is if we just go

let's try and run that again. Ah, so there's a syntax problem. So if I say if I just literally copy this so imagine in most harnesses, you'd have a feedback loop where you could feed it back straight into the coding.

But I'm just going to do it manually and see if it can spot its own error. I stop. Stop. I found an error. Let's check that.

Did you actually read that or not? Let's try again.

OK, so now it should do some investigation. So probably reread the files, see if it can spot what the typo is or what the problem is. And then it should try and edit it or fix it. So actually, in OpenCode, they give you two tools, one which can edit individual lines and one that can just rewrite the file.

So it can even make a decision about which one of those it needs to use to do it. So hopefully, it's going to edit it correctly. And then write that back out. So it's updated the game.js. And it's editing

what are you editing? Something else. Oh, it can also kick off sub-agents, too. So if you give it specific skills or tasks to do, it can basically run sub-agents. The 31B is pretty good at that. The 26B depends on the task.

Sometimes it understands what sub-agents to call. And sometimes you have to give it a bit more prompting. But similarly, with an environment like this, you can expose MCP servers. You can create skills. You can do all sorts of stuff to kind of help guide the model.

Shall we see whether that's fixed anything?

There we go. I don't even see. Did that refresh? Nope, not yet. It's trying to edit the game file.

It must match exactly. So OK, so it's trying to edit a part of the file. I could also just say,

no. OK, I think it probably got unloaded. Let's put that back in. Let's load that back up. Try again, Gemma. Right, so if I tell it, just output the full file again for game.js.

Where did it put it? It was in that one. So you can tell it specifically which file it should be reading and writing to. So hopefully, it should listen to what I say and then regenerate that file.

While we're waiting, does anyone have any other ideas for a game to make? Paige is obsessed with pink squirrels. So we could try something in that vein. Any ideas for a game?

A game where you can build your own game within the game.

Whoa, let's try it. So I'm going to use a game.

Yeah.

Right, let's go ask. We'll ask the big model. So if I go, OK, write a spec for a game where you can build your own game inside the game and use this as a reference. So we're going to build a spec first.

Let's just go here. And we'll use yeah, we'll use the spec for the other one as a reference.

Let's try and run that.

While we see what's going on with our OpenCode,

it looks like it's editing the fileright now.

Yeah, you can see it's processing the tokensright there. And if I go to this one, OmniForge. So what I've done here is I've told it to follow a particular spec. So it keeps the same pattern. So for instance, if I had a format that I needed it to do in, it's pretty good at following those instructions and kind of keeping to that record.

This may not be impressive if you've used any model in 2026 other than to know that it's something that you can actually run at home. That, for me, is the interesting part of this. All the top models can do this.

But yeah, up until recently, not that you could run yourself. So

let's leave that doing its thing there. Has it fixed my file yet? It's made more edits. We can go and see what it changed. So it's starting. Oh, it added some more delta time. Oh, it's changing the input.

OK, well, you can see what it's generating here. It's got these little player cases and all sorts of stuff. Should we see if it runs again while we're waiting for that? Right, are you going to run yet? Ah, do you know what I could do is I could just say to it,

let's say, pressing the button doesn't start the game. Regenerate the files.

And hopefully, it's just going to write them back out. Let's do that. OK, thinking about it, see how our spec is getting on. So let's just get 31B. Can you just implement

this spec

but as a single index.html with the CSS and JavaScript included so that it can do that?

OK,

and there's one other thing I wanted to show you. We'll do that in parallel. Ai.dev. One thing that the model is actually quite good at doing, being multimodal, is to understand

the context of the input that it's given. So I could, for instance, take a screenshot of a website. Like if I go to gemma4deepmind, let's go to this one. And if I just go like

I just grab this, and I could just go to the model,

implement this web page

as a single index.html. And if I just upload that file,

that one, there you go.

And try and run this. Yeah, how's our OpenCode getting on? So it's written the game file, hopefully with no mistakes, written the style file, and hopefully written the index file.

So let's give that another whirl.

Right,

there we go. So there's our game. So I don't see any asteroids.

OK, we've got a star field, and we've got a movable ship. So again, this is just what the model has come up with. You could just iterate on this. You could kind of add to it. I'm not going to I could ask it for more asteroids.

But I think we're done with this one. Let's see how the other ones are getting on. So this is the game about making a game. I'm really curious just to see what it does for this. Because that's going to be pretty nuts.

And then this one, this is generating the oh yeah, this is generating the recreated web page. So

this is all free to play around with. So just be aware there's usage limits in terms of shared ownership. But you can just use it to test stuff out pretty easy. That's why it's a little bit slower than the Gemini models.

Because

there's only one poor Gemini model serving everybody. Let's see. Is this one done yet? Still going. I don't know. Maybe it's made like a really epic adventure. I think it might just be this might be ship it as soon as this is done.

What else do I want to show you? Oh, OK. While we're doing that, I want to show you this cool demo. So one of our team is really into their robotics. So they made a version if you've ever used any of the robot simulation tools.

So this is called OpenDuck, which is just like a little simulator of a duck. And you can ask it questions, or you can talk to it via the E2B model, which is running in the browser. So you actually download the model, and it runs in WebGPU straight in the browser page.

So when it tells it to do these actions, Gemma is interpreting what it's asked the model to do. It looks at what the robot can do and instructs it to if you look really carefully, you can see where it says perform action.

Yeah, before it actually does the action, it tells you what command it's trying to trigger. So you can imagine this actually running embedded on the device or, in this case, is in the simulator to just kind of prove the point.

So that's quite a cool one. The other one I wanted to show you was the Android Studio team have integrated it into making Android apps. So you've got a little chat window at the side, and you can build the apps from there.

So again, they have a little agentic builder set up. So if you were just building phone apps, you can use the model. I think they're using the 26B again. Yeah, they're using the 26B for that one. And then we also have it working with the ADK, so our agent development kit.

Again, it exposes different functionality. It's got a thinking loop. It's got a feedback system. So you can give it more longer-running tasks, and it can make decisions about pulling information and stuff like that. So all these different environments that you can kind of run it in.

Should we see how our final OK, this one is done. So this is going to be this will be our web page. Oh, that's not bad. That looks pretty good. I don't imagine these buttons go anywhere. But for what's new?

Oh, I literally thought it had made a video. That would have been a bit that would have been a bit intense. But yeah, you can see even just from the layout, it's pretty close. It's kind of matched the font correctly.

The understanding of the actual page is pretty good. And this is just kind of one-shotted. So point at random web pages and see what it can build. And let's see whether this I am now really interested in this game.

Let's see whether this actually works. Oh my gosh. Right, is this oh, I've got a full editor. So I can build a little oh, I like that. So the goal, can I run the game? I can put things in.

Oh my gosh, please tell me that. Oh, yes, the triangle moves. Oh, I love it. Whoa. OK, there you go. That's a good point to end on, I think. So yeah, Gemini models, they run on your phone. They run on your laptops.

They run on GPUs. Go try them out. Try the AI Gallery if you want to explore. I'll be around today and tomorrow if you have any questions about how. But yeah, really excited to share that with you. And enjoy the rest of the conference.