How Google DeepMind Runs Agents at Scale — KP Sawhney & Ian Ballantyne, Google DeepMind

Okay.

Hello, everybody. Uh, so my name is Ian Ballantyne, I'm a Developer Relations Engineer at Google DeepMind.

Hi folks, I'm KP Sawhney, software engineer in DeepMind's AI platform team.

And we're going to do an agentic panel today to talk a little bit about how DeepMind thinks about agentic software, how we build kind of our own stacks. We're going to start very briefly by just showing a quick demo, and then we'll go into like a, like a discussion about some of the stuff that KP works on.

And hopefully we can just, you can ask lots of questions, find out all the things you want to know about how DeepMind and Google think about agents, and yeah, let us know your thoughts as well. So, quick show of hands, who's actually used anti-gravity before as a tool?

Okay, so like four or five people. I just want to show you one quick thing, because one of the things that we have in anti-gravity is, a lot of people know that it's like a Visual Studio style interface, but they don't actually know that it actually has a whole agent manager and agent manager framework behind it.

So you can run and spawn multiple agents working on different projects, and it's integrated into the IDE, but it's also, again, kind of like an agentic platform in itself. So you can do things, I've got this project here, and you can have this chat panel on the side, and I can just say like, build an example of this spec, and then I can give it a particular file.

Let's do the spec file. Let's just implement that one. And I'll send that off to Flash. Ugh, experience errorsright now. Let's restart. Should we try that? I am connected to the Wi-Fi, I'm pretty sure I am.

Yes. And if you're not connected to the Wi-Fi, you should use the Gemma models instead. That's my plug. So let's start that up again.

And hopefully I can do the same thing, build, build the spec

marked here. There you go.

Let's see whether we get it this time. Okay, there you go. So again, the model will just go away thinking on the side there. And hopefully what it should do is it should have a look at the spec, it should analyze it, and it should use a bunch of its own internal tools to decide how to do that.

It's actually found that there are already existing files that I have built. It might even tell me that it's already done it, which is interesting. But the tool itself has like built-in to-dos. It has a, like a planning system.

And what it's just done here is it's thrown up a browser so it can actually check how the applications run. And this is also, this is controllable by, by anti-gravity itself. So if I just do that, if I give it two seconds, it should take control over the browser and it should actually try and run.

So what it's probably doing now is it's probably looking at the implementation already to see what's actually already done, and it's going to then analyze it. Should we just have a quick look, see what it's actually done there?

Let's try that again. It'll pick up. So it's in a snag. So it wants to try and analyze it. Oh, one thing that it can do is it can actually inspect the DOM. So it can look at like the actual web page itself as part of that and give and feed that back.

And when it finally finishes doing the task, it'll give you like a report at the end as to what it was able to achieve and what it implemented. So you can then review. It can also capture, for instance, like a screenshot or like a video.

So if it's like an interaction, if you say like, add this feature to my web page, it will go through and then actually try and run it. I don't know how good it is at playing games, but we're about to find out.

See if they're going to try and play the game or, should we have a look, see what it's doing at the moment. It hasn't loaded it yet. Let's go back. Okay, we've got an implementation plan here. So this is what it thinks it should change about the file.

So what you can do is you can just go in and you can like edit a line and say like, actually I want this different behavior or that's not what I meant at all. So it kind of gives you that human in the loop feedback.

And then when you're done, you just say proceed. And then it will go away and do that. While we're waiting for that to happen, do you want to say a little bit about what you work on and like how this kind of relates to your kind of day-to-day job?

Yeah, sure. So one of the things I worked on a few months ago was the deep research agent, which is now available via the interactions API. And so that's, that's been great. But as we continue to iterate on that, my focus has turned now to making best use of this anti-gravity harness internally.

And so of course that, that applies to scaling it for all of the coding we're doing. And you know, we have a gigantic mono repo, so it's pretty complex. But now starting to think about how we generalize this to a variety of other use cases.

So potentially deep research itself, rather than passing around huge, huge blobs of text from the searches that have been done, why not have the different parts of that pipeline collaborate in a shared file system. And so yeah, that's, that's really been the focus for me, really tightening up this harness and making it excellent at not just coding, but a variety of other tasks too.

And do you have any interesting use cases for like how people within Google or DeepMind are kind of using it at the moment? Like what kind of, what kind of things are they doing with the agents in DeepMind?

Yeah, so, so there's a huge amount that's been going on. We have quite a few exciting things that you'll probably learn more about in a few months at I/O, which I can't go into too much detail aboutright now.

But at least internally, there's been a huge amount of focus on building up a huge library of skills that enable folks to do their job better. And you know, skills are great, but in an organization as large as Google, there's a risk of skills really sprawling out of control.

And so that's, that's a big area of focus for usright now is improving those skills, making sure that only the best ones survive really, almost Darwinian nature. But it really is helping folks to deliver good code at a way faster pace, which is obviously awesome.

Awesome, thank you. So you can see this little blue bar around the edge at the moment. That's the anti-gravity taking control of the game. It seems to figure out how to start the game. I don't think it knows what the controls are, so it might end up looking those up.

But yeah, this is, so this is what it edited. The file that was there before was actually generated by a different model. So I can tell you it's not even the same game. It's completely rewritten it from scratch based on the spec we gave it.

And then what you would get at the end, I can see that it's like actually looking at the DOM, looking for any errors, trying to figure out how to actually use it. You should get a video at the end here.

Oh yeah, this is a, it's scratch pad. This is like its notes that it's writing as it kind of goes through and does the task. So you can, like you can actually get a bit of a trace as to what behaviors it's trying to figure out.

Again, you can go in and you can review these things. If you don't like what they're doing, you can kind of interrupt it. So in terms of like the workflow, this is kind of, I guess, pretty common to like a lot of different agent harnesses at the moment.

But this is how we kind of think about using, using it with things like Gemini models as well, so that we can use it for our own development. I will close that off now. So I guess one big question on my mind related to like agents, how do we do things at kind of Google scale?

So if we think, you know, research, deep research is a feature within Gemini app, but then also for everyone within Google to use it, like what kind of challenges come along with that kind of scaling of agents?

Yeah, so I mean, the thing that's kind of top of mind for us at the moment is how token hungry this stuff can be. And so as you can imagine, just managing the quota on a per-user or per-team basis is really quite important.

And so there's a lot of work we're doing around making that more efficient and lower cost. And I think Ian and I were chatting before, and I think what's going to be really interesting for folks like yourselves is mixing and matching between models like Gemma 4, which are effectively free from a quota perspective, just using your whatever GPUs or TPUs you have.

And then, you know, using the more advanced models for specific components of the agentic system. And so also evaluation as well is a big thing we're focusing on at the moment, particularly with these really complicated workflows. How do you actually evaluate that it was successful?

How do you minimize the cost of that? So looking into things like mock TPUs so that, you know, you can test the harness itself and the agentic flow, but not necessarily using up a ton of TPU hours.

Yes, because I mean, I'm sure people are aware of this, but like definitely limited in that capacity at the moment within the world, trying to get enough compute to do a lot of this stuff. Just quick show of hands from the room.

Who's, who at the moment is either building their own agent architecture or harness at the moment? Okay, fantastic. For, so just quick question, like what kind of scale are you looking at? Who trying to, who are your customers?

Anybody?

First, I'm building for myself first, and also the next thing is going to be for the hospitals to use. It's going to be for the radiology department, because they're going to try to combine EHR data with the radiological data.

So that is going to be good.

Okay.

Yeah, probably we're going to use Gemma models because, like we want open source models, because we want to function on the sites and try to use it for our own job.

Gotcha. Is anybody using any harnesses off the shelf or you kind of built your own ones? Like what's the general vibe?

So we're building memory as a harness, infra layer, essentially an ontology graph, deep context graph that developers can use, fully open source, 15,000 GB of storage.

Oh, nice.

Nice, nice.

And how many users do you expect to be able to scale that to?

We know, we know like pipelines, like generation of context graphs, and it's around 3 million per month.

Wow.

Wow.

That's some scale.

You know, like one user can create a thousand within a day.

Yeah. That's the, yeah, the challenge too. I'm sure that's the challenge for us too. How do you stop like one user from, well actually, okay, I'm going to turn that into a question. How do you stop one user from taking down a whole system by spawning multiple instances and multiple, because I'm sure that's like, you know, the more we get better at doing these tasks and like we've got power users kind of spinning stuff up and they've got their team of a hundred people like working for them, a hundred agents.

Like how do we kind of manage like the per-user like behaviors, if you see what I mean?

Yeah, no, it's a great point. And like I said earlier, honestly,right now it's kind of brute force with the quota. So we have some real power users at DeepMind and ultimately it gets to a point where it's like, okay, you've got to just stopright now.

But you know, in general, I think that raises an interesting point about how this stuff is going to be priced in the future as well. Like, you know, you saw Anthropic blocking the open core stuff because these agentic systems are so token hungry and the subscription model doesn't really work for that.

And so yeah, I think that's really top of mind for me as wellright now, how to mitigate that.

Yeah, I always used to joke when I joined Google that, you know, we've got all these resources, you know, available in data centers to use for different projects. Like how do you know when too much is too much?

And one of my colleagues once told me, he just said, "Oh, they'll tell you." And I'm like, "Who's they?" And yeah, sure enough, there's people monitoring these things 24/7, like, you know, looking at spikes and graphs and all our SRE team.

And they do just like reach out to you and say, "Can you just like stop this job running on this one cluster, please?" So yeah, I think, yeah, that's also an interesting one. Any questions from the audience?

Like if you've been able to scale, how do you do observability and traceability? Because that.

That's a fantastic question. So we built a sort of custom web app for that, essentially. And essentially there's one agent backend system that is used for a lot of stuff at Google. And essentially anytime a user issues a query to an agent host on that system, it then sort of automatically appears in this UI where you can drill down at various levels of hierarchy, each of the pieces of the system.

If needed, you can drill all the way down to the sort of raw predict request made to the model and so forth. And so that's been useful. And we also have a concept of like an agent trajectory store as well.

That's more focused around the coding piece where obviously you can have a huge number of steps going on there. And it can be really important to kind of diagnose at what exact point looping started happening or, you know, the model went off the rails.

So yeah, it's all custom internally for now. But I'd be interested to hear what you folks use for observability as well.

Just transporting it to a log file and then basically going forward.

Yes. Any other questions? Yep.

Yeah, the deep research, Gemini deep research is something I've used quite a bit. I really, really love it.

Oh, awesome.

Is there a, is that going to be available in anti-gravity?

Yeah, I mean, that's a fantastic question. Obviously, I can't go into sort of too much detail on sort of like release plans and so forth, but what I can tell you is, yeah, that's something we're actively exploring. You know, we hope that it will make it faster, cheaper, and hopefully better results if we can effectively orchestrate that deep research using the same harness.

Because, you know,right now, without going into too much detail, there's a huge amount of context that's passed all the way through that deep research system, which gets quite expensive and, you know, consumes the context. And so we're really thinking about, okay, how do we make each element of this system more like a collaborator as part of a workspace, which is kind of how it would work if humans were researching something deeply,right?

And I think that opens up a lot of nice potential for things like infographics, additional supporting artifacts, and documents. So yeah, it's definitely an area of focus for me.

Yeah. How, sorry, follow up. How many, I struggled a little bit with this copilot, how many levels of sub-agents can you run within anti-gravity?

That's a great question.

Another one, like really.

Good question. I don't know is the short answer to that. The, I think we, they're not, they're a bit more, what's the word? They're a bit more opaque in how they're kind of presented. The way you kind of think about it is like you have multiple simultaneous ones working on different tracks, but it's not, we don't kind of, yeah, the short answer is it's not as obvious as to which agents are actually working on a particular task for that.

It's not kind of like a massively parallel system in that sense. It's more like you can give them different like trains of operation among a particular project, but you can tell them to kind of work on specific things or you can have jobs that kind of overlap a little bit, but it's not kind of, yeah, I don't have a huge amount of detail on the, like the specifics how the sub-agents work.

I do think that's going to be the future though, is how do we make agent-to-agent communication efficient? And then also how do we give us as the human the ability to really shape that and almost act like a supervisor on a digital assembly line, you know?

So yeah, watch this space, I guess.

Question?

You are next.

There has been a recent debate, of course, you know, skills, MCPs, CLIs, communities going so fast about it. We have a trend every single week. Where do you see with anti-gravity this thing going? Like, is it going to be a combination of skills with CLI, self-improvement?

Like, what's your take on that?

Yeah, for me, I really like skills and they've been working very, very well for me.

Perhaps this is controversial, but I did always think that MCP may be a little bit of a flash in the pan. I like it from the auth perspective. I think that's very powerful. But for me, a combination of skills and guardrailed CLI interactions has worked really well.

It speeds up my job so much. You know, I've got a skill for, to your point, debugging raw logs and, you know, it can do most of that from the CLI. And in a business of our size, the great thing is we have these skills contributed by folks who are absolute experts in that particular area.

And then I kind of, I and the agent get that knowledge for free, you know? So I'm definitely team skills, if that helps.

I mean, we support both of them and I think that's the intention going forward. Again, it's like what the community uses, like, you know, we want to make sure that they work with the harness, work with the models.

So I think, yeah, whatever you guys keep using will probably still be supported. It's probably the way to think about it.

Yeah. At the back, should we go over that?

Yeah, on skills. You were saying earlier that there was an explosion of skills and only the best survived. But how are you evaluating skills and I guess the downstream agent performance?

Yeah, no, that's a fantastic sort of point. Evaluating this stuff is really hard. Even just the mechanical nature of spinning up all of these sandboxed environments set up in the way needed to evaluate a particular problem set. I think the trickiest part is coming up with new data sets, you know?

There's a lot of good, you know, open source ones that are good for benchmarking externally, but you'reright, for specific skills, the onus is almost on like the author of the skill itself to come up with some form of test in that.

But people are also experimenting with the agents designing that as well. So it's a little bit meta. But yeah, a lot of work to do in that space.

Anybody question over here?

Yeah. When I'm using anti-gravity, like browser testing, I love. When it works. Because I'm really struggling with the limits. So you also experience like the retry or like the high effort. Are there any things from my side that I can tweak?

Like how do I know when I hit the limits?

You mean in terms of for like the model and the model usage?

Yeah, I have like AIultra bandwidth.

Yeah, yeah.

I still, yeah, I struggle with like maintaining the limits. And I don't feel anti-gravity is telling me like, hey, it's now not working, not because of the network, but because you hit the limit.

Yeah, that's a good question. It's funny actually, because we have worse limits than you do. Because obviously we prioritize customers and not ourselves. So the fact I was like clicking like a hundred times to keep going is because it recognizes I'm a Googler and that's my fault.

You know, it's a good question. I think there's kind of two parts to this. I think that there will always be limits, especially like within different tiers. I don't haveultra, for instance. Sorry, my corporate account does, but my personal one doesn't.

So I have to change my behavior. But I think we're going to end up in a world, I mean, we have hopefully somewhere around here is Kevin from the anti-gravity team. We could probably talk specifically on that. But I think we're going to be in a pattern whereby it's commonplace for you to run out of credit or run out of capacity on like one model or something and then move to something else, but do it seamlessly under the harness.

So you can give us preference rather than having like, you know, you've hit your limit on tokens for the pro model, so it will automatically put you onto flash, or you've reached your limit on everything you have in your subscription, so you use a local model.

But to your point, like that it will not interrupt the workflow that you're doing or, you know, the notification you get is not the completion of the task. It's like, oh, we've run out of quota, sorry. Like, you know, what if you were off doing something else where you'd sent it off doing a job and you come back to find that it spent the last hour not doing anything because you hit a limit.

So I think the way, when I spoke to him last about this, it was very much more about like trying to make it clearer when you get to that point and making sure that you can. So yeah. We have time for, I think, one more question.

Do you want to have a go there?

I have a question in terms of how guys who do PRs at Google, as we have like thousands of PRs today, and maybe closing the development loop somehow. I wondered if you maybe thought about fine-tuning some like PR models that would go through like your past Google comments and commits and perhaps then better review that code and reduce the load.

Yeah, so you'reright. And I really like the fact that we have some really good stuff in place for that already. So the way it works is, you know, on a per-language basis, we have a specific like auto-review model that has been fine-tuned on all of our style guides and all the rest of it and previous like good examples of code.

But then also on a PA or product basis, folks will come up with their own like specific SIs and prompts and so forth to make sure that, you know, the other reviewers get a good signal for how good the code is.

And just yesterday, actually, I sent a PR for review and I didn't even have to trigger the auto-review thing. I just got an agent that someone had spun up commenting on the PR with quite a good suggestion. So yeah, I think it's important.

And I think to the point earlier about us being supervisors of a digital assembly line, it's like how do we get help with that piece of it as well? Then we can all go sit on the beach.

Yeah. Yeah, to your point, like you can imagine like the scale of all the Google engineers submitting hundred thousand lines of code now being done by agents submitting even more code with more review. They've built a lot of kind of infrastructure for us.

But then also we have like tools like Joules, for instance, like which are, if you've ever played with that, you've got like a web interface where you can go and do that on your own PRs and GitHub. So like, and you get review components of that.

So yeah, I think this is, I think this is an area that's going to be with the ballooning of, I think there was a comment yesterday about like the amount of the trillions of lines that GitHub is getting at the moment generated by agents and like that process.

So as much as we hate our own boring work, I'm sure the agents hate their boring work too. So we've got to figure out a way to do that. But yeah. I think we're out of time for questions, but we will be around to chat afterwards if you want to come and join us or if you want to head down to the DeepMind booth later, we'll be around there too.

So thank you very much for listening.

Thank you all.