How Google DeepMind is researching the next Frontier of AI for Gemini — Raia Hadsell, VP of Research

Our next speaker is VP of Research at Google DeepMind. Please join me in welcoming to the stage Raia Hadsell.

Hello everyone. Wonderful. What a lovely full room and good smiles. I heard the dig on Google there at the end; I did catch that. So my name is Raia Hadsell. I've been a part of DeepMind for the last almost 13 years, and I'm very happy to have AI engineer come here to London.

I'm also very proud this year to be a UK AI Ambassador, so I help the government, academia, industry sort of bridge those gaps. And yes, I'm American by birth, but I've been here for long enough that I can count myself among the proud Brits as well.

So I'm going to talk a little bit about Frontier AI and the future of intelligence. To start, a little bit of a longer introduction to who I am. It's good to be as old as I am; you get to look at this by the decades.

So in the '90s, I did my undergraduate degree in philosophy of religion. I was definitely not a computer scientist yet,

but I really enjoyed it. Before you ask, yes, I learned a lot and I'm glad that I did it, and no, it hasn't been very useful since. In the 2000s, I did a bit of a pivot, moved into computer science after some good advice from those close to me, and spent my PhD years in New York City working on convolutional networks, neural networks for robots with Yann LeCun, which was a lot of fun.

I then, in the 2010s, made the decision to join a small group of curious, scrappy individuals working at DeepMind. It was a group of about 30, 40 people at the time, and we spent the rest of that decade working on things like Atari video games, Go, StarCraft, and some robotics.

A lot of fun. And now I am a VP within DeepMind. I help run a group of about 1,200 scientists and engineers across 10 labs, and we're working on a lot of different things. I'll tell you about three of those.

So first, Frontier AI is an area where we really are trying to make sure that we are staying in the front. So we're thinking about what are the next architectures that we're going to use for Gemini, what are the next problems that only AI can really address, and how are we going to build the future of intelligence.

And that's thinking not just about artificial intelligence, but it's create the future of human intelligence as well, and even robotics intelligence as well. We are all on this journey together, and I think that it's important to think about how humans change as well as the technology.

Our approach, we look for root nodes. You know, we're not going to waste time on the leaves. We're going to really find for a big problem space that hasn't been solved, how deep can we go, find the deepest problems, and solve those in order to then enable a lot of downstream impact.

We partner, you know, really with the world. I really think about it very broadly and think about who are the partners that can help us find those root nodes and solve those problems, and also, you know, bring it to the leaf nodes.

And solving problems that are worth solving. The motto, the mission of DeepMind is to build AI responsibly for the benefit of humanity. So I really take that seriously. We want to build, solve problems that are worth solving. Allright.

So we work in a lot of different areas within Frontier AI. In DeepMind, these are sort of some of the different categories. I'm not going to tell you about all of them, so you can just maybe keep those a mystery.

But I'll just pick out a couple. So first, in advanced models, I actually wanted to bring up an embeddings model. So the theme of this talk overall is things that are not directly language models. And in the modeling space, I wanted to talk about embedding models.

To start that out, I'll ask if anyone knows what a Jennifer Aniston cell is. Aha, I got a few neuroscientists in the room. So this is actually a concept from neuroscience where we've discovered that there are not just a single cell, but a small number of neurons that will encode for a specific thing, as in a specific person.

And that those combinations of neurons that only activate for that one person or that one thing or that one place, those cells are actually very robust. They activate regardless of modality. And this is used by the brain, very fast retrieval for recognition and for comparison functions.

So that means that when I say the name, Jennifer Aniston, or if I showed you a picture or a video, or if you even heard her voice, if you knew her, if you were enough of a fan, then all those different modalities lead to the same set of cells activating.

So we want that in an artificial neural network for the same reason. We want fast retrieval, recognition, and comparison. And so we can trade what's called an embedding model in order to encode for those concepts, in order to be more robust to different ways that the information can be presented, and to be very, very good at sort of understanding what is the comparison between these different activations.

I use contrastive losses. One of the reasons why I like this space is because I did my PhD work in part looking at Siamese neural networks, which was an early way of understanding what is a contrastive loss function.

And so these embedding functions are really critical companions to generative AI. Sometimes we want to generate, sometimes we want to retrieve. So the group at Google has been working on this for a long time, and just recently we've actually released Gemini embeddings 2.

So this is exciting to me because it really is sort of the ideal. It is fully omnimodal. It uses, it's derived from Gemini. So it's got sort of that level of knowledge and understanding of the world. And it is, and it allows extremely, extremely good retrieval.

In a little bit more detail then, why is it good that it is unified and multimodal? It means that you don't have to have different steps to try to bring things together and map them together. You can be truly end to end and not lose information by trying to combine audio information, visual information, text information together.

So you can get a single vector that represents text

up to 8,000 8K tokens,

128 seconds of video, 80 seconds of audio, and a full PDF. And together that can give you a lot of information. You can then use that to be able to use it for retrieval, for querying, for agentic logic, and other things.

We also use something called the Matryoshka representation learning, or MRL. And that allows us to be able to have the same network but represent different dimensions. So for instance, you could start out doing a retrieval using only 256 dimensions for your embedding, and then you can expand that to get to more expressive mix.

So this also, this gives us, we can demonstrate that we have, this allows us to have a unified semantic space and really state-of-the-art quality. So just something that's come out recently that I think should, it doesn't get talked about quite as often as language models, but it's really important as that companion, I think.

Allright. Next, I wanted to quickly talk about another thing that is not a language model. This is not a language model at all. There is no language involved. And this is work that we've done on the weather. In London, it rains a lot.

And a few years ago, there was

an informatics scientist at the Met Office, Meteorological Office for the UK, the UK's weather agency, that said, can you predict better rainfall than our physics-based models using AI? And I said, I don't know. Interesting problem. Let me take this back to the team.

Took this back to the team at DeepMind. We started working on this and we discovered, yes, you know what? Predicting the weather, even though it is a very, very hard problem using a physics simulation of the atmosphere, is actually quite tractable for neural network models, given that we have 40 years of data, 40 years of global data on what the weather is.

So a couple of years ago, we came out with GraphCast. GraphCast predicts the state of the atmosphere up to 15 days out, everywhere on Earth, and for many different variables. And this uses a spherical graph neural network. You can think about this encompassing the Earth and having nodes that go all the way from the surface of the Earth all the way up into the lower stratosphere.

And we actually feed in and then predict in an autoregressively 100 different atmospheric variables, for instance, wind speed, temperature, and humidity, as shown here.

And this worked very well. Here's a quick example. We were excited to see this in late 2024. This is Hurricane Lee, sort of comes into the Atlantic, pauses for a moment, and then takes a turn to the north, speeds up, and makes landfall on Nova Scotia.

This total video is nine days' worth. So that's how far the hurricane moves. And this is actually the output of the graph neural network. That's its prediction. And the prediction that it made is accurate nine days out of where that landfall is going to be.

In comparison, the best gold standard models that are physics-based were only accurate six days out as to where that landfall is going to be. When you're talking about a major hurricane hitting land, three days is really important. So with this, we said, okay, this is important, and we're going to keep on pushing the science.

So the team developed the next model. We called this GenCast. And the difference here is that this model, while also based on a mesh, is probabilistic and it has a higher accuracy and a higher efficiency. The weather is fundamentally chaotic, and we want to know what's happening on the tails.

And so having a model that's probabilistic allows us to do that, allows this to be operationalized and used for actual weather prediction.

GenCast also was more accurate. So when we compared it to 1,300 gold standard benchmarked weather forecasts, then this was more accurate 97% of the time. And it was also could be, we could produce that 15-day forecast in eight minutes on a single chip instead of hours on a very large supercomputer.

So much different sort of space of the solution that we were proposing. And just this last year, this team is relentless. They're constantly coming up with new models. And so the latest one is called FGN, Functional Generative Network.

This directly predicts cyclones rather than predicting the weather and then having to add on a cyclone detector as sort of post-processing. This actually incorporates the categorization, the recognition of cyclones, their trajectory, their wind speed, and the formation of the eye directly into the network.

We train for that, which means that it's much better. So this has already been used in the US by the National Hurricane Center, and they are very, very excited by how much of an advantage this now gives. So this will hopefully be used worldwide in the coming years.

Allright. Lastly, I wanted to use the last few minutes to talk about, again, something that is not language model-based. So this is world models. And this actually came out of work that DeepMind has done on games and simulation for a long time.

We've been working on Atari, on Go, on StarCraft, and then on, you know, Mojoko type environments for robotics because we wanted to understand agency and the environment. We started focusing more and more on not just training the agent, but creating an infinite environment.

You know, when I did work on a locomotion here,

then oh, that's not playing. Well,

maybe this will play. I'm going to jump forward to Genie 1. So we wanted, so this is Genie 1. It could only run for a few seconds, but you could say, hey, I want this type of a world.

And then you could produce this little Platformer 2D game environment where you could jump around for a few minutes, and it would actually respond to whether you hit the left or theright. And it could produce a reasonable diversity of different looking Platformer type of worlds.

This was enough to say, hmm, we might have something here. Let's scale up. Let's scale up the data and train again. Improve the method and train again now on 3D games. Then we produced Genie 2. Genie 2 is interactive, but it's not yet real time.

So you need to go awfully slowly. And it can produce 3D environments, but it couldn't do anything that was really real world type of quality and more higher definition. So we were working on that. And then along came VO3.

Allright. Now,

allright. Well, now I am out of time, but I will still take another minute or two to just show you these. So this is saying that, telling Genie, I want a world where I'm walking down a muddy lane in Kent.

This looks not far from my house. The fun thing here is that you look down at yourself and you realize that you actually have a body. You're actually interacting with the world. It's a little bit odd to know what's coming out of this model.

It's really understood not just the appearance of a lane in Kent, but actually what it takes to engage with that, make the water move, and to walk forward. Of course, it's not just scenes that are walking. We can very happily ski.

And so you can create an environment where you can engage with the world in so many different ways.

Here's an example where it says original there. That's we started this. We prompted this with a fragment of video, and now it's changed to Genie 3. So this is an artist. He made those first few seconds. And then we use that to prompt Genie and bring this world to life.

He was so tickled to see that we could take a little snippet of his world that he had laboriously created and bring that to life in a way that means that you can fly through it. You can bounce off of this thing.

And it remembers that, oh, here's that weird structure there and go back to that and fly through there. So these environments are not only diverse and interactive, high quality, they also have memory. So the prompt here was, I'm an origami lizard in an origami world, and this is what you get.

And we use this as a nice little test that I can spend, you know, I can spend a minute running in one direction, run back to the start, and everything is exactly as it was at the beginning because we have a really good memory.

Working in these environments gives us consistency and control. And lastly, we have, we're able to prompt this world as you're in it. So that means that while I'm in a world that might be a little bit boring, here I am, you know, this is a world saying I'm walking down the Camden Canal in London here near the DeepMind office.

Well, what happens if I prompt it at the same time? Then what happens? I've just changed the world that I'm in. I can change it again.

There we go. Immediately, the world is different. And one more time, just for fun, I love the idea of a new form of gaming where I could be adversarially prompting your experience of a world. It just creates a whole different sort of entertainment, a whole world, a whole new frontier that I think can be really amazing, not just for entertainment, but for education as well.

The ability to be able to go into a world in order to learn about it, I think, is incredibly powerful and may well be something that we see more and more of. And with that, I will say thank you.

And just a quick call out that tomorrow morning, my colleague Omar is going to talk about Gemma 4, which is a language model. Allright. Thank you.