Building Conversational Agents — Thor Schaeff and Philipp Schmid, Google DeepMind

Hello everyone.

Hello.

Heute machen wir alles auf Deutsch, ist das okay?

Super.

Perfekt. It's just that Philipp and I were both German, so we thought it was funny maybe we were going to do it in German. Actually, it looks like there's a German crew there, which is nice. No, no worries, we'll do it in English.

We'll do it in a couple different languages maybe, we'll find out. Do we have other languages in the room? Other nationalities? Yeah, what do we have? Shout it out.

Serbian.

Serbian, Italian, Spanish. Any Icelandic? No? Okay, close, close.

Moroccan.

Romanian. Nice. Dutch? Ben je Nederlands? Okay, great. Which?

Hindu.

Yeah, Hindu. Okay. Canada? No? Okay, Bangalore? No.

Farsi. Allright, nice.

Czech.

Czech, yeah. Brilliant. Okay, this is fantastic. Well, thanks everyone for making your way over here with all your languages. Really appreciate it. We can really put the model to the test today, which is great. Yeah, hi, I'm Thor, or Torsten for the German speakers amongst you.

Hi, I'm Philipp, only Philipp, so four German and English speakers, so.

Yeah, it's nice. So we work on the developer experience at Google DeepMind, broadly covering kind of Gemini API and also working Google AI Studio as a tool for developers to try out the models quickly, and then also the API interfaces.

To use the API, you do need an API key. Actually, who has an API key already? Gemini API? Okay, a couple folks. Who has used AI Studio before? Google AI Studio? Okay, a couple folks. Great. Maybe we can quickly.

Anti-gravity.

Yeah, that's good. You actually don't need an API key for anti-gravity. So if you don't have an API key yet, if you have your machine on you, I hope you do, it's a hands-on workshop. I do apologize, they took away your tables.

Yes.

So it's a very laptop situation, like literally laptop. I hope you didn't bring your Mac mini or whatever. But so yes, if you just go to ai.dev, and I mean there's you can also go to ai.studio, you can go to aistudio.com, but we paid a lot of money for ai.dev, so please use that.

It just redirects to aistudio.

Yes.

So ai.dev slash api-key?

Yes, or on the corner left side. There's a get-api-key and then api-keys is where you can.

Oder ein German Schlüssel?

Yeah, it's my personal account and we cannot change language, so. You need only a Google account, so no credit card, no nothing. All of the things we are going to do is part of the free tier. So if you go to aistudio or ai.dev and there's some like sign-in form, you can like just use your Gmail, no worries, we will not charge you.

And then on api.keys you find at the topright corner normally something called api-key create or create api-key. And when you do it for the first time, you might need, so similar for me, I can import my projects or I can create my projects.

I can just click create, give it any name. I mean we can call it IEI

workshop and then create project. We can translate it to English. Okay, takes a few seconds and then you should be able to create key. And this key will be used for the demos or the hands-on things we are going to use later.

Wait, it shows the api-key?

Yeah, I will delete it so don't copy it. But you can like.

Copy it very fast.

You can use that key, you can like put it into your bashrc or zsrc file or later you can like directly inline it, whatever you prefer. If you have one, you can use just one. And since we have a few more minutes time, we really want to make sure everyone who wants to follow along, take your time, create your api-key.

If there's an error appearing or something else is not working, feel free to raise your hand. Thor will come to you and will help you. And.

And yeah, just a reminder, it is a secret api-key.

Don't do it.

So don't do what Philipp is doing there.

That's why I delete my key again.

Yes. So we do get that a lot, that people leak their api-keys. Mostly it's clock code that is like pushing the api-keys to GitHub. So I recommend you don't do that. No, just kidding. Just remember it is a secret api-key.

So treat it like a secret, don't share it with your neighbor. But yeah, do create that api-key now and we'll give you a couple minutes to do that. And while you do that, or you know, once you've finished creating your api-key, I'd love for you to just briefly introduce yourself if you want to, and just sort of let us know what you would like to get out of the workshop.

Maybe there's a specific use case you're working on. Yeah, we'd love to kind of get to know y'all a little bit as well. So while you create your api-keys, if you want to, feel free to just shout out your name or nickname and sort of what you're working on and what you would like to get out of the workshop.

Yeah, Wolfram Ravensburg from the Thursday podcast working for Corby for weights and biases. And big fan of Gemma, yeah, Gemma 4 and Gemini. I'm one of the few who are using it for coding and stuff as well.

Nice.

By HNDQs. And I'm also running it on the classes actually.

Oh, nice.

And I want to see more of this, you know.

Okay.

And I want Google Glasses.

Okay, yeah, we'll see if we can get to the glasses by the end of the workshop.

Thank you.

Sorry. Yeah.

If you have a seat to yourright, can you jump in so we can film up on the side here? Makes it easier to people after 75 minutes.

And while we are waiting, we launched something cool on Chrome, which I haven't activated it, but when you go on your tab bar, clickright, you can now move tabs to the side and we have vertical tabs now, so.

Yay.

So more screen. That's good.

Nice. Okay, cool. So glasses. Are you running what are you running on the glasses? Well, technically on the phone,right?

It's a software where you use Gemini, which calls OpenClaw, and that's the engine for tool calling.

And then that's just using Gemini Live kind of through the WebSocket on the phone,right? Yeah, nice. Yeah, that's cool. Yeah, VisionClaw, if you haven't heard of that, pretty fun open source project.

And I think you can run it, yeah, sort of on the well, now that Meta has opened up the SDK for the Meta Ray-Ban, you can actually hook in something like Gemini Live API into the glasses. Well, the glasses connect to your phone and then the phone actually connects to Gemini Live, which is cool.

Experiment, but it shows what will be possible.

Yeah, nice.

Any problems creating api-keys? Everyone has an api-key? Should we check?

Okay.

Anyone else? Anything specific that, yeah?

Michael worked for Get Your Guide in Zurich. We're building like our first kind of AI customer support agent, so.

Cool.

Just interested, I think, in the general how you guys are doing it, comparing.

Okay, cool. Nice. Last chance, anyone else?

Okay.

Allright.

So we can start. Before we go into the hands-on session, I have like 10, 15 minute slides to give a bit of a background what we are going to use to build. The first session, which I'm going to do, is more on like the building an agent without any live audio input.

That's where Thor is going to take over later, and then we are going to be building some very nice conversational agents. So who of you has used the Gemini API to make an API call to Gemini before? Okay, that's a few.

Have anyone of you used the Interactions API? One, okay, two, okay, at least some person. So the Interactions API is a new API we launched in December in beta, which hopefully will succeed generate content soon. It's a unified API to use with models, with agents, and it's much more aligned with, I would say, the industry.

So much closer to what you are familiar with from Open Models, with the Chat Completions API, or from Anthropic, or from OpenAI. And what we are going to do, the slides, then we build a coding agent, like a small little cloud code with reading files, writing files, running bash commands, and then we have some time at the end if you have questions around it.

And then we do a short break, toilet, drinks, and then Thor will continue. We all have an API key. So as I said, we want to build an API which works for both models and agents. And when we launch the Interactions API, we also launch deep research.

So maybe you have used deep research in ChatGPT or in the Gemini app, where you basically start off a query and then you get back a plan, and then the model goes on and like does deep research for 10, 15 minutes, visiting hundreds of sites.

And the API supports both models and agents, and it's very simple to switch between those. You basically either define a model, which could be Gemini Free Flash, or you define your agents, which could be deep research. And we are working on it for you to bring your own agent or to define your own agent that you can customize all of these behaviors.

And it's the same surface. So we will see later, but when you send a request to Nano Banana to generate an image, you can like basically chain those interactions to a Flash model to do something else, to Luria to have you like generate audio, or even to hopefully soon Veo to generate you a video.

And the interface is very similar to what you see from OpenAI. So we now have like those content blocks, basically. So every input you provide and output you provide is the same type. It has a type field, which could be a function call, a thought signature, a text, audio, video, image, which hopefully makes it for you easier to build with the Gemini API.

And it is less, I would say, Google branded, less Proto specific, less gRPC to make it easier for developers to build. The core primitives of the Interactions API, in addition to making it easier, we also introduced state on the server, which we will use for building our agents.

So we don't need to manage our loop and always send back the whole history. Very similar to Responses API, you now have a previous interaction ID you can provide, which basically attaches to the existing history. So you can like just send a new input.

As mentioned, we have an agent with deep research and background true, so you can start your research, can pull it, or soon use webhooks to get notified when your research is done, so you don't need to keep the connection open.

We have the type blocks, and then also we have the same like streaming pattern, very typical for web development using SSE, which also makes it hopefully easier for you to build. We all support the built-in tools, but we also now have support for remote MCP.

And I think two weeks ago we launched tool combination, so you can now combine Google Search with your own custom function, which was one of the big features people were asking for years, I think. And now you can do this.

And to summarize again, the difference between, so generate content is what we have today, Interactions API is what we will have tomorrow. We will have server-side state management, but you don't need to use it. So if you say, "Hey, I want to manage my turns, I want to manage my context, I don't trust you," or "I need to do like context engineering, I need to remove certain parts," you can do this.

You send, it's way easier to send new input, basically. We have the built-in agents, we have also background support, so you will get asynchronous execution. We all see with agents, when you send a prompt, it might take one, two, three, four minutes to complete.

And keeping HTTP requests or connections open for, I would say, more than like 10 seconds is not very good practice. So you want to like use asynchronous calls to either get notified or like do polling when it is done.

And it is less Proto oriented, which I really prefer. It's much closer to what people know from like the developer ecosystem. And also a side effect of the state management is that the implicit caching for the API is much better.

So for the one of you who don't know what is implicit caching, so when you send a request to the model, the model needs to encode all of your input tokens. And you can cache those encodings for a follow-up request to save cost.

And cached requests, I think, are 90% cheaper for the input tokens. And when you have to manage your state or context yourself, you maybe strip out line breaks, remove certain parts, this breaks the cache. And using the server-side state, the server keeps the context.

So the chances for your cache hit rate is much higher, and we see like two to three times better cache rates from like the startups using Interactions API today. Quick code example on what we mean with making it simpler.

So on the left side, you have like the very Proto specific one-off input parts with inline data or text where the field basically describes the type. And on theright side, almost looks similar to other APIs, I would say.

So it should be much easier if you decide after the workshop to give it a try. Then I guess all of you know roughly what an agent is. We have a brain or our model which decides what it wants to do, calling a tool, generating text, doing something else.

We have tools which basically give our brain hands and eyes to interact with the environment where it is in. We have the context. It's basically all the model knows what it has to do, what it can do, if there are certain preferences, if there are certain constraints.

And then we have the loop which basically combines our model with hands and tools and runs it until the model no longer calls the tools and generates a text. So some quick examples on how to use the API, and then we go into like the nice hands-on part.

So basic chat usage with server-side state becomes very easy because we have our interactions create call. So we define our model, we define our input, what's the capital of friends, we get back the output, and then we can just continue providing the previous ID.

And the model behind the scenes, or like the server behind the scenes, basically has our user input, our model output, and then appends to the user input that you don't need to have like a client-side history object where you append the user turns, the model turns, and then the user turn again.

This becomes very helpful when you build agents where you have a loop and always need to append new user input. And as mentioned before, that also works for agents and models. So we really want to build this unified interface where you can continue your conversations no matter what model you use.

And in this example, we basically run a deep research request on research AI agents in 2026. And then we take the research output and just continue with the Nano Banana model to generate a visual for it. And it's like four lines of code, basically, without for you to, what's the context, how do I provide the input, and hopefully makes it a lot easier.

But as said, you don't have to do it. So like the input field also accepts the same array with role, user, role, model, and all of the inputs.

For tool use, it's also hopefully now much easier. We have a type function, our name for our function which we want to use, describe, and then we have the parameters the model needs to generate. And then here's roughly what we are going to build.

So we make our API call with the tools, and then we check what the output of the interaction is. Requires action basically means you as a client, you need to do something. So the output generated some function call or some object which you need to react to.

We iterate over our output types, check for the function call, execute the function, append the result, send a new interaction until the model no longer decides it wants to call a function and generates a text or something else.

Okay.

The last time I did a workshop, we were all still coding manually. So that's my first time doing a hands-on workshop where at least I don't code much manually anymore. I'm not sure how you do it. So what we are going to do today is we don't code manually.

We are going to use your preferred IDE, agent, CLI of choice. I'm not sure how many of you are following. But to make it easier, we created agent skills for our agents to use. So if you go to the Gemini, you can search for Gemini API docs coding agents.

Can you zoom in?

Ah, yes, sorry, of course.

I can also, can you see it?

Yeah, I think you can also just Google them.

Yeah.

I mean, I can. Gemini coding agent skills.

Gemini

agent skills.

Okay.

Yes, this one, it should be the setup, your coding agent with Gemini MCP and skills. I think that should bring us here.

It's very different results.

Really?

Yeah.

Yes. Or if not, you can go to the documentation. And then on the left side, in the getting started section, there's a coding agent setup.

Are we successful? Okay.

Just one question. Should we install this skill globally or just for the project?

You don't need to install it globally. And that's a good question. So how do you install? So we have multiple skills. We have the Gemini API dev skill. We are not going to use this one. That's for the generate content API.

If you want to use that API or if you are familiar with it, go for it. But we have the Gemini Live API. That's what Thor is going to use later. So we also don't install this. But then we have the Gemini Interactions API.

And here you can either pick the first command or the second command, depending on what you want. And then like just copy it, open your workspace where you are working in. And then I already did it, but you can like add the command, npx install, and then you should get a wizard asking you to install it.

There are many preselected. Don't get confused by it. It just means that all of those agents are compatible with the .agents folder. So you might have a .agents/skills/Gemini Interactions API, and in there we have our skill. Yeah. Let me also wait.

So I tested with Cursor and Anti-Gravity and also Gemini CLI. So if you are using one of those three, you are good. If you are using Claude Code, I think it should work as well. I'm not sure if they follow .agents best practice.

If not, I mean, I'm sure you can.

They invented it,right? Like the skills are in the.

Yeah, the skills, but I think they only look at .claud.

I think.

Yeah, they don't look at agents.

But I think it installs into skills, no?

I don't know. Okay, so.

Sure. Yeah, if you install npx skill with claud is agents.

Okay, yeah, question.

You showed two different commands. One said skills of this agent, the other one context 7.

Yeah, both work the same.

But what is context 7?

So context 7 is, I think, a product based out of Upstash. So context 7 has an MCP server and our skills CLI, which you can use to get access to like skills. It's like a public repository. And skills.sh is the same from the Vercel team.

Okay.

So it's like, and it works, both works on GitHub. So the Google-Gemini-Gemini skills is a GitHub repository. And the GitHub repository includes all of the skills we documented. So you can also go there and find it there. There are also the deals and install commands, and it makes it much easier than cloning it and making sure you have it in theright directory.

And then what we can do is to make sure our skill works, we can just like ask our agent, what skills can you use? And then it should, if you have installed it correctly, you should see, yes, we have our specialized interactions API skill.

So Anti-Gravity here picked up our skill in the .agents/skills folder. Okay. How are we doing?

Can you zoom in just a bit?

Zoom in more?

Zoom in one more, yeah. Just in the back.

Okay.

Yeah, that's good.

Better? Yeah.

I was using Telegram with my agent. Is the result of this being run on the system? So would I have to have it run?

No, you should be able to do it on like your OpenClaw kind of stuff if you tell it, install the interactions API, and then.

Just to find a product of the workshop. Will it run?

Yeah, it should run. It's a Python script. We are going to execute it later.

So because I'm only running it in sandbox.

Yeah, no, it should work. Okay. Any difficulties installing the skill? Any questions what a skill is? Everyone familiar with skills? Okay. What we can do maybe to give you some insights on what skill we created or what it contains,right?

The importance when creating skills is it should be either something the model cannot do reliably, or if you have some personal preferences on like how to do a certain workflow, or I don't know, you always need to run tests using bun or something like this.

And what we did with our skill is we made sure that the agent is aware of which Gemini models are available. A common issue we saw before that is like Gemini always used Gemini 1.5, which is no longer the latest model.

We also included the agents here. We have some like very high-level information on how it works, but we did not include like all of the documentation. What we did instead is, you should see, yes, a link to our documentation, which is available as Markdown.

So instead of the need to always update our skill with like, for example, we added a new feature to Interactions API to combine tools, we would have needed to update our skill. And then every one of you also need to update your skill to be able to use it.

And then we are not making a lot of progress in terms of like knowledge cutoff. Instead, we provide the information as part of the skill. So all of the agents now have like web fetch tools so they can query the information based on the skill.

And then like we only need to maintain like the documentation, which is mostly up to date. Yeah.

How do you find that works with like efficiency?

Sorry?

How do you find that works with like tool calling efficiency and all that sort of thing? Do you find it like having to go and fetch the page to them?

I mean, it's normally you would provide it as a reference on your local file,right? So it either needs to do a read file call or a web file call, which is the same, I would say, in terms of like cost.

And it works very well. Okay. So what we are going to do as a first example, since we are not wipe coding, we want to build something more substantial. We not just say build an agent. We want to be more specific.

So we want to build, create an agent class with a constructor and a run method. The constructor

creates a GenAI client. So the GenAI client is what we are going to use to call our model. We also need

to define a model.

And we also need

a global

previous

interaction

ID, and then add the main method to run an example.

Do all of that in

a workshop. Okay.

So as we can see, the model in this case, or the agent as a first step, read our skill, analyzed our main file, is implementing the skill. Okay, it checks. It should probably fail, yes, because I'm using UV.

So I stop and I tell it, use UV from the workspace.

And then

we let it generate UV. Okay, it checks if we have installed the library. We have. So maybe in your case, the agent still tries to install Google GenAI. If not, you can do it yourself with like pip install or UV, pip install Google-GenAI, and then make sure to, okay.

So

we have our starting agent class. We have our GenAI client. We have our model ID. It defaults to Gemini Free Flash. It would never have defaulted to Gemini Free Flash if it hadn't read the skill,right? Because then we were stuck with Gemini 1.5.

We have our run method, which calls, makes our interactions.create call. We have the input text. We have our previous interaction ID. We set our new previous interaction ID, and then we return the text. And then, nice, as a main example, which we will run in a bit, we create our agent.

We have turned one. So my name is Phil. And then the agent runs it. And then what is my name? Now we can check if our Gemini created Gemini agent

works with multi-turn and our interactions API. You might see a warning similar to the one I got here. Interaction usage is experimental. That's as we are still in beta. We really work hard to get the API out of beta to make sure you can use it in production.

And our call was successful. Hi, Phil. Nice to meet you. Okay, maybe we weren't successful. I don't actually have a name. So what is my name? Maybe you should check. Did we do it correctly? I mean,

my name is maybe, ah, okay, that's still the call. Sorry. I asked you, what's your name? I mean, it's a language model trained by Google. It makes sense. And what is my name? Your name is Phillip. How can I help you?

So yeah.

You are using Gemini 3 Flash as a coding model?

Yes.

Respect.

I mean, it works really well if you are providing good instructions with good skills and good context and don't expect it to, I don't know, like cure cancer. Like if you have a very good understanding of what you are trying to build, Gemini 3 Flash is like very fast.

I mean, it didn't take much long. It didn't consume credits,right? Every one of us is somewhat token constraints at the moment. And it works really well. Okay.

Use it for agenda cues? Yes, but for planning and coding?

Yeah, no, it works. I mean, we will going to use Freewell. I mean, like it also asks like if it wants to run. So we are closing the loop. Even like with Gemini 3 Flash, it tries to run our script to make sure it works.

And then we will continue in a bit. How are we doing? Anyone making successful calls?

Yes? Okay, perfect.

Background.

Okay. Any issues, any errors, any questions? Yeah.

I'm using TypeScript.

Okay. It works?

Yeah.

Nice. Great.

Not so much.

Awesome. Even coding on a phone. So okay. So normally the next step for our agent,right, we now have our like very basic run. We can chat with the model. Now we need to add tools to it. And we want to build some kind of a coding agent.

So first tools we are going to add is a read and write file tool. And we just continue in our main agent thread. It's like, okay, next we need to add a read file and a write file tool.

Create the

basic Python implementation and also the

JSON schema definition. So when you use function calling or tool use,right, we need to create a JSON schema, which we provide to the model so the model understands what it needs to generate. Once it generates that schema, we also need to have some kind of a code implementation, which we can then run on the client.

So we ask it to create the Python implementation and also the JSON schema and map to for the key and

key function schema. Okay. Let's see what it will come up with.

Okay.

It's cheating a little bit. So I have like a solution folder and it looked up the implementation. Yeah. I mean, that's why it's still important to like check your weight. I can like, it found like the solution and then it was like, oh, I got the solid Python example in solution.agent to guide me.

The task is implement read file and write file.

It's a smart model for sure.

Yes. So what we got back is we have two new like very basic, very, very basic file implementations. So we have a read file tool with a file path, which uses Python syntax to open it, to read it.

And we have a write file tool with file path and the content and writes it. And then we have our read file schema, reads a file and returns the content, write file, writes the file and returns the content.

And it made some updates to our agent. So what did it change? Okay, our input is now a text string and a list. Makes sense since we now need to return a function call. We check, what do we do?

Okay, we create a tool definition for our model, which is the

schema. So that's our tools map schema.

And then we have our loop, which you might be familiar with from the slide I showed. So after we run our request, we check the interactions.outputs. So the interactions.outputs include all of the events generated by the model. And since Gemini is a reasoning model, it also includes, for example, the thoughts and the thought signatures, which we need to return.

Since we are using the previous interaction ID in the server-side state, that's done by us. And we only need to check, okay, do we have a function call? We have a nice debug, so we can check it. And for our outputs or function call, we check our tools.

Do we have our tool or not? I mean, maybe the model wants to edit a file, but we don't have an edit a file tool. We would catch it here. It calls our model and then it creates the tool results.

So for function call, we have a function result. That's also part of the change. We really want to make it easy. We will see later when we use Google Search, there will be a Google Search call and a Google Search result to have very the same schema.

And then we use recursion. So if we have two results, we basically call ourself the self-run method again. If we don't have two results, we return the interaction text. And it also updated our example. Write hello from the agent to a file named hello text and return it back.

So we, I mean, the changes roughly look good to me. We can try and run it. So UV run Python workshop main.

What do we get? We get our tool call with write file. We get our tool result. We get the read file call again. And then the final agent response. The file hello agent text was successfully created with the content.

Agents are not only like singleton,right? Maybe we want it to respond. So next step is basically very ambitious, telling it, I want to have

continuous, I don't know, stood in implementation to test. Let's see.

Let's see where it will continue.

I hope at least it should update our main function where we have an input, a while loop, basically waiting for the input using our agent. So yep. So we have our user input. And then we have, we always continue with our agent run method, which then inside the agent runs it inner loops until there are no tool calls anymore.

And if there's a result, we basically get back our response. Okay. Let's quickly wait. Is it too fast or are we still on track roughly?

It's fast.

It's fast?

Yeah.

Okay. I mean, we will share the code later. Also very happy to answer questions. We have many like blog posts and examples of that online. So you can, if you are interested in like rebuilding it later with less fast, but I try to be a little bit slower, but I also want to give Thor enough time to have you speak.

So what we can do now, since we are like our agent implemented our like while loop

to provide input, we can say something like hello,right? Normally we should now not do any tool calls because hello is like, yeah, our agent correctly kind of understood or Gemini in this case understood, hey, hello is nothing I need to solve with a read or write file to write.

So I can say, hello, how can I help you? Can you write, or maybe can you create a CSV with a thumbs up?

Might take a while bit. So it does the thinking and it does the function call. And

okay.

Certainly, here's a CSV file, blah, blah, blah. Okay. Can you write it to disk?

Yes, I can if you learn. Wait, maybe our model is not having our tools. Let's check. There's the tools map. It does. Why is it not using our tools? What tools can you use?

Okay, it has the read file.

Yes. Maybe we are not explicit enough.

What we can do to improve this in one second is we can add a system instruction to tell the model, hey, you are a coding agent. You can use tools to write and interact with the. Okay, there we got our tool call, write file with our CSV.

Cool. Since we saw our mistake, we can now tell the model, hey, add system instructions

for the interactions API call and add an example prompt for a coding agent.

Okay. And what's really nice now, since we loaded the interactions API skill in the beginning, the model still has the awareness of, okay, how do I add system instructions to the interactions API? And what I can guarantee you is that the Gemini 3 Flash has never seen any code of the interactions API because the model was trained before we even released the API.

So all of the work we were doing so far is based on like the skills and like the coding infrastructure and hasn't been part of the training. Okay. So what did we get?

Okay. We can provide it on the run command or we have one when we create it. That's good. And okay. Coding posturna, you are an expert software engineer and helpful coding assistant. You have access to the local file system.

Okay. Let's accept it. Let's start our agent again. Let's say hello.

Okay. Hello. How can I help you with your software engineering and coding tasks? I mean, definitely better than what we had so far. And what did we send before? We said, can you create a SVG with a thumbs up?

So can you create an SVC with a thumbs up? And now let's see if it calls. Yes. Now at this time, we got our write file tool call. And then also we got, hey, I have created a thumbs up SVG file with a simple line art thumbs icon.

So

can I.

I think it should.

Yes. There we go. We got a thumbs up icon. Cool.

And of course, what's missing for a coding agent,right? We need to get some bash tools. That's now not part of the solutions folder. So let's see how we will get our bash tool. Now add a similar run command

tool that allows the model to execute

bash commands.

Okay.

Creates an implementation plan.

It does have a spec tools. You're asking me to do something?

I'll fill in. I think there's a question. Can you.

Hey, I was saying, like I was just testing it, it does hallucinate tools a little bit. Like try to do like less character and then even with some school methods.

Okay. So we have our run command, which uses a subprocess, which in this case, I guess is okay. We don't care too much about security for this example. And then our output is a stood out.

Works. Edit. We have our run command tool. Yeah. It even updated our system prompt. And now let's stop that. Let's clear that. Let's run it again.

Any suggestion on what we should test?

You smell a banana to create an image.

I'm not sure that will work because we don't have any skills or any information for it.

It's fine.

Time.

Time. Yeah.

Get the time.

Tool call, run command, date, Wednesday, April 8th. It looks good. Yeah. Cool. That's our small little coding agent.

Delete all files.

No. I mean, let's not do this.

Any more questions? Any more ideas? We have like roughly five to seven minutes. Yes?

General question. So because the state is kept in the server, is it possible to like fork the history of the super part?

Yes. Yes. So

what you always can do. So what we are doing here is,right, we always use the previous interaction ID from the previous turn. So we basically stack it and you can always go back to any index in the stack and branch from there.

So if you would keep the interaction IDs on your client side, you can always use those to, I don't know, branch out and like, I don't know, have like a first prompt or like do basic web search and then like use this as like a base for like five parallel requests doing some other work.

And you can always get the context. So we have an interactions.get method, which you can use to retrieve the interaction and then also get the previous interaction ID. So you can basically go back until the beginning and get all of your state if you want to save it for later.

And the default for those interactions being stored on the server for free tier is one day. For paid usage is 55 days at the moment. Yeah, you had a question.

You answer my question.

Sorry?

You answer my question.

Okay. Perfect. More questions? Yeah.

Does it mean then it can have infinitely long context window?

No. So once, so Gemini models have a million token context. What would happen now if you reach that, you will get an error. But we are working on context compaction techniques, but it's easier said as done and still something you currently need to maintain on your client side.

So when you say that it will retain, the server will retain for one day. So that means every day it resets to zero and then the.

No, no. So when you send a request, you get an ID, the interaction ID. And the interaction ID stores your input and the output of the model. In the free tier, the input and output and the ID is stored for one day.

So meaning if you send a request now and you continue eight hour latest from that point, the state or like the context is still available. If you would send a request tomorrow, it would basically say, cannot find request with the old interaction ID because it basically is pruned after a day.

But if you use the paid API key, it's stored for 55 days. And the interactions API is also coming to Vertex. And I think there might be a little bit more flexibility in terms of how long you want to store or customize it.

Yes?

Is there a specific API tool or if we work the context first, if you want to do context compaction on the client side?

We don't have one yet, but hopefully soon.

Yes?

Maybe an idea. You could have the API joke being spoken out with the eSpeak tool.

Sorry?

The eSpeak tool. It could use the eSpeak tool to actually speak the joke about APIs.

And we can also use Gemini, which has a TTS model, which can speak to, but speaking and listening, I mean, Thor will show many, many cool things. Any questions regarding the interactions API and the small little agent? No?

Okay. Then you get eight minutes.

I said five minutes.

Okay. Five minutes.

And then we'll be back here.

With.

To make your agent talk.

Yes. Cool.

Cool. Thanks.

That worked really well.

Thanks. Yeah.

I used the old one and super happy that now there's a new version. It was 2.5 before.

Yes. Yeah. It's been a while.

There's a long time. How much better is it?

How much better?

Yeah. Much better.

Much. We didn't even pay him for it.

There you go. Big upgrade.

Big upgrade. Yes.

Yeah. Maybe I know there's a couple more minutes, but.

Can I ask a question while we wait? About caching.

Caching?

Yes.

Yes.

When you input tokens.

Yeah.

Like how is that? Are those cached at the context array element level or the whole context is cached? Like when adding the elements to the interaction, is each one of those individually cached or.

That's a good question. We probably need to find Philip to answer that. I actually don't know. Philip, caching question.

Yep.

The input tokens.

Yep.

What was it?

Is the context cached at the individual interaction level or is the whole context.

It is not on an interaction level. It's more on an object level. So it's.

For example, when you provide an input, like no interaction ID, first input, PDF, 4,000 tokens, and the text input with 10 tokens, and you do a follow-up interaction call, maybe only the PDF will be cached and not the other, like the short text.

And then if you do another one, maybe the PDF and like the follow-up turns will be cached. So it's like more on like an object level. But since you, I mean, how is it? It's very easy to make a mistake in caching if you, like even the slightest change in your prompt, removing whitespace, line breaks will break it.

So like having this rely on the server to keep it, it's more guaranteed that it's secure. And it could be as easy as, hey, my user says there's an empty line break at the end. Okay, I remove it and then I use that history again and then it falls apart.

Do itright. So ideally, every conversation, the previous part of the conversation is cached.

Yes.

Optimally. But I mean, it depends on where your request kind of hits it and like how fast you follow up, but the cache rate should be pretty high.

Cool. Maybe to kick things off, we can sort of look at one of the examples that we can build with the Live API, the new model. So this is now Gemini 3.1 Flash Live, which came out two weeks ago, I think.

Very exciting. It's been a while. I think the previous 2.5 native audio model was December. So it's been a while. The reason being, we did kind of major underlying architecture. We work to, you know, ideally lower latency, you know, better scalability in the future as this ramps up.

So that's very exciting. And then, you know, in parallel, we're working on bringing the Live API to the Interactions API as well. So that's something that Philip and I are working on at the moment. Not there yet. So we still have to use the Live API for now.

But let's do a little demo. And so this demo uses the Live API in combination with Lyria 3. So on the Gemini API, have people played with Lyria? So the new Lyria model is actually a music generation model.

And so Lyria 3 can now generate songs, you know, including songs with lyrics. So this is the clip model, which is kind of a 30-second clip generation. But then also we have a full song model now where you can generate kind of full songs.

So the idea is that we have our conversational agent, which is kind of the DJ that you're interacting with. And then that DJ has a tool call to generate music,right? And so, you know, so I'm originally German, but, you know, growing up, I learned kind of English through Harry Potter and BBC Radio One.

And if you remember back in the day, you could like phone in and like wish for a song. And so that's sort of the idea of the experience here.

Right then, you're on Live Jukebox. What absolute banger are we cooking up today? Tell me, what genre are we leaning into and what sort of vibe are you going for? Let's make.

Can we get some sort of high-energy German techno schlager about the AI scene in the UK?

German techno schlager about UK AI? That's absolutely brilliant, mate. Proper Radio One style. Any specific lyrics or themes you want crammed in there?

Just surprise me. Let's go.

Right, you beauty. Let's get those speakers rattling. Prepare for a burst of machine mayhem. Here comes German techno schlager about the UK AI scene. Let's go.

In London town, the future's bright and keen. The cleverest robots that you've ever seen. The computers wear a proper bowler hat. And the androids all enjoy a lovely chat. They're drinking builder's tea with digital grace. A happy British smile on every metal face.

Oh, wunderbar. The AI is super gorgeous. Yeah, super gorgeous.

Yeah.

The TPUs enjoy when you clap for them. So appreciate that. Anyone have any idea for another one? We can do one more. Just shout it out. Any languages?

Any genres?

Can we have a nursing

in Swahili?

A nursing song?

Song, yeah.

In Swahili? Okay.

Right then, welcome to the Live Jukebox. You're on air with me.

Can we get a nursing song in Swahili, please?

A nursing song in Swahili. Brilliant. That's something we don't hear every day. To make it an absolute banger, what sort of vibe are we going for? Upbeat and celebratory or something more relaxing and gentle? And what specific bits about nursing should we squeeze in?

Let me know and I'll get this track buzzing.

Yeah, techno hardcore. About.

Techno hardcore in Swahili about nursing. Wicked. Prepare to blast those speakers. I'm dialing that in now. Get ready for a total banger, mate. Wauguzi nguvu yetu Ubingwa wenu uponyaji Kila siku mioyo yote Kwa huruma mnahudumia Pigo la moyo sayansi Okoa maisha kwa weledi Nguvu ya wauzi Ubingwa wa ajabu Tunawashukuru sana Tunawashukuru sana

Well, we can feed it into Gemini to, you know, guest the audio out. Okay. That is the Live Jukebox DJ. You, if you want to, it's on. So this was built fully with Google AI Studio. So it's kind of a little vibe coders demo.

I've published it in AI Studio as well, so you can try it out. You will need a paid API key for this one because the music generation actually requires a billing account. So if you have, you know, an account with credits, you can use that, for example.

Cool. Yeah, Gemini 3.1 Flash Live, you know, benchmarks. Obviously, benchmarks don't really tell you the truth as much. They are great for benchmarking things. The real world, especially in kind of live audio, does often look a bit different.

So, you know, ideally we'd just try it out ourselves. So Gemini 3.1 Flash Live, it's the model that is now in Gemini Live in your phone. So if you're using Gemini app on your phone, you're talking to that model.

As well as I think Search Live has it now in there as well. So if you're talking to Google Search, I think that's the same model. And then you can build applications using this model on the Live API.

So the Live API is a stateful kind of WebSocket API. You are able to send real-time text, audio, video feeds to the model. So audio you're sending in kind of, you know, buffer chunks. So if of the real-time audio, you're streaming that in.

Video you can stream in at a maximum frame rate of one frame per second. So this can be, you know, a camera feed. This can be a canvas. So it could be like your screen share,right? So you could share the screen with the model.

So for example, Shopify is using this for Shopify Sidekick, where it's actually kind of like a tech support walking you through, you know, if you're like, oh, how do I set up a custom domain for like my Shopify store?

It would basically like talk you through how to do that. And it can see kind of where you are on the screen by sort of ingesting the frames of the screen. And then in return, the WebSocket gives you kind of real-time events back.

And so these are basically streaming back audio buffers. And then also you can get the audio transcription. So that's kind of the text of it. And then we have tool calling built in. So Google Search grounding is built in by default.

So if you need kind of real-time weather information, you can access that as well. Yeah. Some key features. So what's really cool about this model, again, it's kind of a native audio model. So what that means is we're not going through text.

So it's kind of not a cascading pipeline where you're transcribing the text, running the text through an LLM, and then generating speech. But rather the model itself is, you know, going sound token to sound token. And the intelligence is kind of baked into this audio model.

So it's based on, you know, Gemini 3.1. So decently intelligent. You have different kind of thinking levels that you can enable. And so the great thing with that is kind of the multilingual support. So it's, I think, 97 languages that are kind of supported in preview at the moment, which, and the great thing is because it is kind of a, you know, a native audio model, it can actually, it has sort of the audio understanding of Gemini built into it.

So it can understand a mix of different languages. It can, you know, sort of Denglish, for example, which is like a mix of German, Deutsch, and English,right? So it would be able to sort of naturally switch between kind of different languages as well, which is really great.

Yeah, barge in, you know, obviously there's kind of automatic voice activity detection sort of built into the model. So you can interrupt it. You saw it earlier, the DJ, you know, is kind of trying to have a conversation, but we're trying to get it to, you know, use the tool.

Tool use is the other big thing. So major improvements in kind of tool use and instruction following here with that model. And so you can build some really cool things with that.

So obviously we currently only give you a WebSocket API. So that is kind of a downside. If you use something like GPT real-time before, you know, you get a direct WebRTC kind of infrastructure, which can be helpful. So we have partnered with, you know, a lot of sort of integration partners like LiveKit, PipeCat, Software Mentioners in Poland.

They built a great service called FishGem, Vision Agents, Vox Implant. So these partners have integrated kind of the Live API directly and then give you sort of easy WebRTC integrations if you, you know, want that or need that kind of for your system.

Yeah, let's try it out. So you can try it out yourself. And it's going to be interesting if we all try it out in this room at the same time. So we'll see how that works. But again, ai.studio or ai.dev and then /live, you can try out the model.

And so you can, you know, ingest your webcam here as well. So we can give it our webcam feed allow this time. And then, you know, we can send text as well. So we could send like, how is my outfit?

So in this case, you know, I'm not ingesting any.

You're wearing a green jacket over a blue t-shirt paired with a black cap. The combination looks casual and comfortable. Is there a specific occasion you have in mind?

Yeah, okay. So, you know, that obviously is a bit further away from our upbeat sort of British, Australish, Australian live DJ. So what we can do is we can

kind of adjust our voice through the system instructions. Now, in terms of the bass voices, we don't have that many. There's kind of 30 different bass voices.

Got a project in mind?

And they are, you know, fairly generic sort of. But because, you know, Gemini has very deep audio understanding, what we can do is actually we can modify the voice through prompts, through system instructions. So what we can do is actually we can give it kind of a system instruction here, for example, and we can just say, okay, speak in a friendly Irish accent,right?

And so now we have that bass voice, puck, and speak in a friendly Irish accent. Hey, can you hear me?

Well, hello there. Loud and clear so I can. What can I do for you on this fine day?

What do you think of my outfit?

Well, no, you're looking very smart so you are. That green jacket suits you well, I must say. A grand casual look. Were you thinking of heading out somewhere?

No, I'm just here at the AI engineer. Okay, anyway. So I realized I didn't put on. So what I wanted to show you as well is this grounding with Google Search. So what we can do is, you know, obviously the model itself is trained to a certain knowledge cutoff date.

And so, you know, if we need kind of real-time information like this gorgeous weather that, you know, no one would actually believe us, the model wouldn't believe us, you know, if it didn't have Google Search grounding. But now, you know, we can have Google Search grounding.

We can have different thinking levels as well. So

no thinking is actually very minimal thinking. But it means, you know, the model will respond faster. It's kind of lower latency because the more thinking we have, obviously the more thinking steps are kind of involved and it'll, you know, increase the latency.

So you can see in the benchmarks as well there's kind of thinking high, thinking low, sort of different thinking levels. But so now if we turn on kind of the Google Search here.

Oh, hello there. What range of shenanigans can I help you with today? Anything at all? Just give a shout.

Yeah, what's the weatherright now in London, actually?

Well, no. It looks like it's about 9 degrees and mostly cloudy in Londonright now. Might be a bit chilly, so you'd be wanting a jacket for sure. Anything else you need to know about the goings-on over there?

What are you talking about? Can you check again? What date is today?

Oh, apologies there. I seem to have gotten ahead of myself. Today is Wednesday, April 8th, 2026. Is there anything specific you needed to know about today?

Yeah, now look up today's actual weather.

Right, sure. For Wednesday, April 8th, 2026, in London, you're looking at temperatures between 5 and 13 degrees with a decent chance of rain, so don't forget that umbrella. Anything else on your mind?

Allright. Here we are. It looks like I didn't pray to the demo goths. Okay, there seems to be something going on. I wonder if we messed up the UI there. It should work a lot better than that. I think the Google Search grounding for some reason isn't working.

So, but what we can do now is we can obviously try it out ourselves in an application. So the easiest way we could do that is, you know, like the Live

Jukebox DJ, we could use Google AI Studio to kind of vibe code our integration. So we have this pill here which is called add, you know, voice conversation, real-time voice with, you know, Gemini Live API. And then we could say build kind of a multilingual interview assistant that allows me to, you know, train for interviews in different languages like German and English and Spanish and, you know, what have you.

And so we can now kind of fire this off. So this uses Gemini 3 Flash preview. It is limited to kind of JavaScript full stack environments at the moment. So I think you can choose between kind of Next.js, Angular.

There's like XR building blocks as well if you're building for kind of glasses sort of web VR experiences. So, you know, feel free to kind of fire one of these offright now, or you can also clone the Live Jukebox DJ that I shared with you earlier and you can try that out.

It'll take a little time and you'll hear like a little chime once it is ready. So in the meantime, what we can do is, so if you go to the Gemini Live API docs, Gemini Live API, there we are.

So, you know, we've done this. We tried out kind of the Live API in Google AI Studio.

We also can use the coding agent skills. So Phil showed you this earlier. We have dedicated coding agent skills also for the Gemini Live API. So you can install that. It'll help, you know, your coding agent kind of integrate the Live API more easily, more quickly.

But then also, you know, we have good old example apps on GitHub, which can be very, very helpful. So what you can do is

you can clone these example apps. So in GitHub, you know, you can, like you do in GitHub,right? You can clone this. So we'll open kind of a new terminal. And yes, feel free to follow along. Do that a bit bigger.

We'll make a new directory. We'll call it AI Eng Europe. I like that they call it Europe,right? But it's, I mean, I guess, yeah, the UK is still part of Europe, just not the EU. Fair enough.

And so we'll go in there and then we'll just do a git clone of our app. And so now we have our app in here. So there's actually a couple different examples that we can use in here. So if you're using anti-gravity, there's a handy AGY command to open your examples in anti-gravity.

And so we can, you know, look at our different examples here and how we kind of need to set that up. So we have, you know, two different scenarios. So the Gemini Live Gen AI Python example uses the Gemini Live API on the server.

So it creates a WebSocket connection from your server to, you know, Gemini Live API. And then on your front end, you basically set up a proxy to proxy the WebSocket connection to your client side,right? Because your browser window is kind of your client.

And so that is what is capturing your audio feed, your video feed. And so in this example, we're just using Fast API. And we're basically just setting up kind of a WebSocket here that our client can connect to.

And then we're basically just receiving sort of our, you know, audio cues, our video cues from the client site, or, you know, our text input cue as well. And so we're receiving that. We then setting up a live session.

So we're using our Gemini client. So we kind of abstracted sort of all the Live API stuff into this Gemini Live file here. And you can see like starting the session, we're basically setting up kind of our Live Connect config.

Maybe I close this for now so you can see that. We're setting up some system instructions. So that was, you know, earlier kind of said a helpful assistant. We also said kind of speak in a friendly Irish accent, for example.

You know, this is where we put our sort of system instructions, our guardrails. We can make that pretty long in terms of covering sort of what we want. And then we can define kind of our tools here. And then we're basically setting up our, you know, session.

And our session sort of is, you know, the WebSocket session. And then we're just receiving our audio and video cues from the client side and proxying that through. So that is kind of one approach. That's sort of the server-to-server approach.

And, you know, we're just using kind of UV here. So if we're setting this up for the first time, we can go into, so this is our Gemini Live

Gen AI Python example here. So we can set up our virtual environment. We can then activate our source here. We can install our dependencies.

Okay, I might have, this is a fun part of

a Google laptop security.

Come on. Allright. Just look away. Don't look at that. And then, yeah, install our requirements. And so we'll need an API key. So the API key, you can see kind of here how the configuration is. So we basically just need our Gemini API key and we need to set up an environment variable for that.

So you can see we have an example file here. Not a lot in there because it's basically just that. So we can copy our .env.example into our .env

file here. And then do you remember how to get your API key? Where do you get your API key?

Ai.dev.

Yes, ai.dev. There we are. Fantastic. Love it. Okay, ai.dev. So that is where you get your API key. I think I have a couple of API keys. So it takes a little while to load them here. Which one is, maybe we'll use this one.

So once you've created your API key, you can copy the API key from here. Well, actually, maybe I should create a new one because later I'll need to delete this. So we'll say ai-eng-europe. We'll have a couple of projects here.

We'll just use, we'll just, which one? Which one are we using? Too many projects. Okay, we'll just use this one.

And so now I really don't like us actually returning the clear API key here. I think we learned something today. Okay, we save that. And so now we have sort of our API key set up. And now what we can do is we can run our demo and that was just the main.py.

And so now our demo will be up and running on localhost:8000 here. And so we can see it's just kind of a basic sort of demo. And when we connect.

Tapa the morning tea. I'm Gemini Live, a little demo of what this API can do. Why not try out some fun features like hearing me speak in different accents?

Can't see me seen.

Ah, you can see allright. You're sitting there with your handsome face looking straight at me.

Ja, aber können wir auch Deutsch sprechen, bitte?

Aber sicher. Sehr gerne spreche ich Deutsch mit dir. Wie kann ich dir heute helfen?

不好意思 ， 我不会说德文 。 你可以学中文吗 ？

当然可以 。 没问题 。 想聊点什么呢 ？

你吃饭了吗 ？

我是一个 AI， 所以不需要吃饭 。 你呢 ？ 吃过了吗 ？ 有什么好的推荐吗 ？

I was asking. Sorry? Ah, yeah, yeah, yeah. But it didn't transcribe it.

Allright. We're finding out a lot of things here to improve, which is nice. But yeah, so what we can see is, so this is, you could actually notice that the latency is a bit worse because we're actually having that chump from our client to our server.

I would love to blame the Wi-Fi, but, so with the server-to-server setup, you just have that additional latency of sort of going through your client. So what we can do as well is we can go directly from our client to the server.

And so this is the other example that we have, which is this one here using ephemeral tokens. So

for ephemeral tokens, we can just kind of look into the setup here. Very similar. We'll just go back. Actually, let me get a new terminal up here. So we'll say ephemeral tokens. So our ephemeral tokens are basically short-lived tokens that we generate with our API key on the server side.

And then we send that ephemeral token to our client. So, you know, our phone, our browser to then initiate the WebSocket connection directly from the client to the Live API. Again, similar setup here. We want a virtual environment.

We then activate our virtual environment and we install our dependencies.

We also need our API key again. So I think what we can do is just use the same one. So we'll copy, maybe just copy this one here and then

paste that in. Is it big enough? I don't know. Can people see that? Maybe we'll zoom in a little bit more.

So now we have our key here as well. And so

now that we have our dependencies installed, we can run our server. And so our server here,

we can look at the server real quick.

So this server is basically just a very, you know, slim sort of backend that just has our Gemini API key. And then it generates an ephemeral token for us. So sort of ephemeral tokens currently on the V1 Alpha API.

So you'll need to use actually a different API at the moment for this. And then you would pass in kind of this expiration time because ideally, you know, the token should be short-lived. So should the token ever leak, you know, it shouldn't be too costly because it'll expire pretty soon.

So there we are. That's our token. And then we return our token back to our client. So on our front end, we then have our Gemini Live kind of integration here. And so this is an example of just a pure WebSocket integration without sort of any SDK.

So, you know, if you can use kind of any sort of WebSocket framework here and you can see sort of, you know, how all the different sort of raw WebSocket events are handled. And so you can see here we have, you know, this is kind of our WebSocket API.

So here we need to use kind of the V1 Alpha. And then this is the bi-directional, you know, we're streaming in both directions. And here we pass our access token, which is our short-lived token.

Great. So what we can do now is now our server's up and running. So we can see this beautiful interface here, which was handcrafted, you know, no agent involved in the creation of this one back in the day.

And so we can just see kind of all the different knobs here that we have. So we'll try this kind of Google grounding as well. And hopefully it was just the thing we need to fix in the UI.

Here's our Flash Live preview model. So that's 3.1 Flash Live. And then we can connect here.

And so we see kind of the server events. Actually, let me maybe do that again. So we say enable Google Search Grounding, connect. So what we can see is kind of our WebSocket setup. So if we look into the network, we now see we have kind of this WebSocket connection here.

So we see we have our access token. So our token request to the backend, actually, let me see that. Okay. So you can see we're just getting our token. So the response was just our auth token and when it expires.

What we sent in terms of the payload, where's our payload?

Okay, actually there was no payload. Yeah, we're just getting a token back. So once we have our token, we then set up our WebSocket. And so we can see our messages here. So basically the first message in our WebSocket, we're sending our setup configuration.

So you see we're, you know, sending our model as like Gemini 3.1 Flash preview. We have some real-time input there. We have the tools. So hopefully now the Google Search should work. And so now what we can do is we can start our audio.

Hey, can you hear me?

Yep, I can hear you loud and clear. How can I help you today?

I'm in Londonright now and I'm wondering, do I need an umbrella? What's the weather today?

I'm not getting the current weather for Londonright now. Sorry about that. Your best bet is probably to quickly check a weather app on your phone or search online. That'll give you the latest on rain.

Oh, you're killing me. Just use Google Search.

My apologies. I seem to be having some trouble pulling that upright now. I know that's frustrating. You can get the forecast easily by just asking your phone or searching online. I'll try to get this sorted on my end.

Gemini, why are you doing this to me? That's just.

I understand your frustration. Sometimes technology gives us hiccups. And I'm sorry I couldn't get that weather for youright now. I'm always learning. So hopefully next time will be smoother. Was there anything else I could help you with?

Allright. That was painful. I do apologize for that. Okay, so something is definitely going on that is not nice. Let's see if we have more luck with the custom tools. So we're also passing in, do we pass in some custom tools or is there actually some issue with our.

Hey, can you show me a Hello World alert?

There you go. A Hello World alert has been displayed. Anything else I can help you with?

Yeah, can you change the background of the page to green, please?

You got it. The background color should now be green. Let me know if you need anything else.

Yeah, can you tell me what's the weatherright now in London?

I understand you're asking about the weather in London, but unfortunately.

I did not enable the Google grounding, but for some reason it's. Yeah, so with the Google grounding, it should work. So I wonder if I messed up something.

No worries at all. The weather information isn't something I can grabright now, but if you have any other questions or need help with different functions, feel free to ask.

Let's try this again.

Hey, can you hear me?

Ah, yeah, we need some audio. Hey, can you change the background back to white, please?

I wish I could, but I don't actually have control over your display settings. You might need to look in your device's settings menu to change that. Anything else I can help with instead?

Ah, no, sorry, off the page, the background of the page back to white.

Oh, I see. Sadly, I still can't change that for you. Those appearance settings are part of the website or app you're using.

Oh, yeah, you'reright.

No, but I thought, okay. Well, does it enable custom tools? That shouldn't be the case anymore. Allright. Lots of work to do. But, so yeah, this is sort of how you can get up and running.

Did anyone manage to get it running on their machine?

Yeah? Is Google Search Grounding working?

No.

Okay.

That's a shame.

Allright.

What else do we have? So we need to fix lots of takes away from this. Yeah, if you didn't get the link, so this is the link to the Live API examples. So

they also link from the docs. So if you go to kind of the Gemini Live API docs, you can find them there.

Is that the end of my slides? I do think we have, yeah, the agent skills as well. So I mean, if you just go to the Gemini Live docs,

Gemini Live API, and on the main docs, we've actually linked it from the top there. Try the Live API, Google AI Studio, the GitHub examples, or use the coding skills.

So that is, yeah, that is how you can get started. Cool. That was not how I was hoping this would go. Yeah, we'll figure out what went wrong there. But yeah, I appreciate y'all joining. And yeah, we'll take questions.

Yeah, so I was just wondering how does the context look like?

Yeah, so you can look at kind of the session management. So there's actually sort of, so the Live API has kind of this context window compression. So without compression, kind of audio-only sessions are limited to 15 minutes. Audio-video sessions are limited to two minutes.

So what that means is that sort of you will, the session will kind of terminate. It will give you sort of this go away pain. And what you can do is sort of enabling context window compression. So context window compression is kind of like a, you know, sliding window where you basically say, you know, like, I want to keep that much context sort of in my window.

And sort of as the conversation progresses, it will then actually like forget kind of the previous context before that window. So yeah, so context window compression is something that you can enable kind of, you know, to have that sliding window to make the sessions longer.

But then, you know, there's only so much context depending on the frame rate that you're feeding in in terms of images, depending, yeah, mostly it's images. Audio is sort of audio-only sessions. There's more kind of context you can keep in the window, but then as you're adding kind of video frames to it, it does, yeah, compress it.

Do you have anyone that's?

Yeah. Oh, yeah, yeah.

Is there any real-life use cases that this has been used for that's very creative from a, like a business context?

Very.

I see what you've done today is a lot of fun, but like applying it to business, is there any businesses that are actually doing this really wellright now?

Yeah, I mean, so Shopify has it in production with Shopify Sidekick. There is a bunch of, so like actually one that I really enjoy, if you Gemini Live API block,

we had kind of a case study, which is this startup. Yeah, so I mean, Stitch is using it as well. You can like vibe code, vibe design with your voice in Stitch. But then, you know, Stitch is built by Google.

So, you know, you probably got to discount that.

Waymo is also integrating it. So, you know, we first we got rid of the drivers in the cars and then, you know, you do want to talk to someone in the car, you can then talk to Gemini in the future.

So they're working on that. I like this one. So this is HeyAddo. It's a great startup from Argentina. So they are building these voice sort of companions for the elderly in combination with kind of an app for sort of the caretakers or, you know, the children of the elderly where, you know, they can get notifications, you know, should the elderly mumble about something that, no.

But so it's a really nice interaction where like the multilinguality really shines, you know, because like in Argentina, you know, a lot of it would be Spanish. But there's a nice example you can kind of look at, which is really sweet.

I came to visit my grandma, Maria. I want you to say hi to her.

Yeah, you might. You can look at it in your own time. And then I think there was another one. So yes, it is somewhat more of a future music use case, just kind of looking at, you know, some of the rough edges and limitations.

And I think for now, if you're, you know, like in a real business use case, you might be, you know, better off with kind of the cascading pipeline because it gives you sort of the observability at each step of the pipeline, which kind of with the real-time, you know, native audio,

you don't really have that fine-grained control and observability in terms of, you know, plugging into, say, like rewriting the response before it's being said. You know, obviously there's certain benefits with that in terms of the natural flow of the conversation, but for certain business use cases, it might just not be there yet.

So it is somewhat, you know, a bet in the future of like what kind of real-time conversational interactions will look like in the future. But yeah, depending on your business use case today, it might not be the best fit just yet.

Are the transcripts all stored and auditable?

No, so you, in the session, you can't retrieve them. So you would have to store them on your end.

Again, so that is sort of where the integration partners come in. So if, yeah, so Lifekit, PipeCat, they all have like really good offerings to, you know, store the entire audio as well as the entire transcript and sort of give you additional observability tooling on top of that as well.

So it's not something that is kind of all available sort of, you know, from just the Google site. So that's something where currently we're relying on kind of the partner integrations to sort of give you that additional functionality.

Yeah?

Yeah, make sure I always.

Oh, sorry, behind you. Yeah.

Do you want to go first?

No, maybe first.

Okay. So you said that this is not, you know, like super business ready for more complex use cases, but what are your thoughts, for example, in replacing interviews for recruitment using this?

Yeah, I mean, I'm not sure I would, you know, replace your entire interview pipeline with that. But I think it is a great scenario, you know, for certain steps in the interview process or, you know, being able to screen more candidates, for example.

Yeah?

Do you think it's not ready?

Do I think that's ready?

Because there's a lot of like context that you also want to give it,right? Especially even if it's like a first screen or the quick screens that you're mentioning, you still want to put some criteria that would guide and guardrails on how to guide the conversation.

And then the second part to this is also can it work, for example, with if we make companies in a Google environment, Google Meets,right? Right now we have auto transcript. So can you put these tools together?

Yeah, so it is a preview. So it is something you can use in production. Depending on your use case, I think you need to evaluate like, you know, do you need like SOC 2? Sort of there's potentially a bunch of things that you need to build around it for it to actually fit your business use case.

So yeah, it is ready for experimentation. Does that help?

I mean, I think that's what we're doing a lot in companies that are experimenting.

Yeah.

Oh, yes.

An issue I always have is when I'm demonstrating voice agents, a lot of people are talking and it can't differentiate the speakers between each other. So is there a solution for this already? Like train it on a voice sample or something?

In terms of like identifying the different speakers?

Yeah, and so it only listens to you if it's your agent, for instance.

Oh, interesting.

It only listens to you.

Imagine you have a coding agent and somebody makes a prank and says, delete the file or stuff like that. You don't want that.

Yeah, no, that's interesting. No, I don't think there's any specific ability to sort of like say, just listen to me. So there is sort of

kind of this proactive audio where you can tell it to kind of only respond in certain, you know, to certain contexts. So like ignore things that aren't relevant to the conversation. So to some extent that works, but I don't think it's super reliable at the moment where you'd say like, only listen to me, ignore anyone else.

I've done that with Parakeet from Nvidia where you can train a little 10 seconds and it can differentiate the speaker that way. But it would be nice to have something like you talk to it and then it recognizes your voice and ignores the other voices for the rest of the session.

That's cool. That's a great, great idea. Yeah, thanks.

Yeah, I think in the, yeah.

What does thinking look like? Does it think in text or is also thinking in speech?

Yeah, so you get the thinking only in text. So there's text events on the WebSocket channel that, so you can opt into getting the thinking as text. Yeah, it wouldn't speak out the thinking. Yeah.

Thanks for the demo and for being brave to go up against the demo gods. I had a sort of question around the multimodality side of things. One of the areas that I really want good text to, no, speech to text models is having them be grounded in what I'm looking at.

Like when I'm coding, I want to just, you know, word vomit into cursor, sorry, anti-gravity, and have it understand the context. And, you know, when I say something that is a specific class name, it should just actually use that.

How does that work inside of this framework? Like would this be a way to actually do that grounding or would you recommend some other API for that?

And this is for, so general Gemini models are really good at audio understanding. So if your use case doesn't require like fully real-time transcription, I would actually recommend using like Gemini 3 Flash to basically transcribe, but also guess, you know, like ingest context and sort of basically get contextually aware transcription.

Is that going before the text, before the sample? You can do it at the same time. So when you are in your editor,right, you have maybe like two transcripts with Layer 5 and Windows dot 5 and you can provide, so since we are multimodal, you can provide the files as text input and your audio as audio input to the same request.

And then it has like awareness of both and you can use both. And then it can, I don't know, like generate new code or do something else. I think Stitch does this in a way. I'm not sure like if it can show it or not, but in Stitch you have like Figma-like mockups and you can speak to it.

So you can like it has the visual awareness plus the audio input on, I don't know, a Jetson button to print. And if you only have one button, it knows which button it has to press.

Yeah, there's, so I mean, depending on your use case, if you need it to like be fully real-time conversational, then yeah, so you can kind of use text to sort of ingest additional input or, you know, imagery if it makes sense.

But then again, that, you know, reduces the context window size. Or, you know, if you don't need kind of the fully real-time, then like using Gemini, you know, just Flash to, you know, transcribe is actually pretty good.

But if I'm correct, so we always talk about video,right? We say one frame per second and one frame per second means while I'm talking, every second I send a new image. But if you need like visual awareness and your visual doesn't change, you can send the visual as like.

First input, then speak for five seconds and not send an additional image, then you are not using so much context. I think like one image is around 1,200 tokens. So not too much. So if you are like in your editor, you want real-time, you basically can use the API.

First input is send real-time image and then send real-time audio. And then you stop basically and the model has the image input and the audio input and can respond to it. So there's no need to stream the image consistently if you don't, if it doesn't change or if you don't need like it to react to it.

Yeah.

Cool. Yeah, I think we have a bit more time there in the back. Yeah, how do we get you the microphone? Or did you just want to shout it out? Oh, yeah. We'll do it.

Thank you for your interesting presentation. I have a question about the personalization or adaptation. Can it recognize the speaker's level or the knowledge during the interaction and then based on the speaker's knowledge produce the result or not?

Sorry, can you repeat? Can it?

Can it recognize the speaker's knowledge or the background during the interaction to produce the response based on the speaker's knowledge or something like that to personalize itself?

So you want to ingest kind of initial context? Is that what you're saying?

Not context. For example, suppose you're talking to that about the civil engineering and can it recognize I'm a civil engineer and based on my knowledge produce the result, use the advanced keyword in civil engineering or not?

Based on my knowledge. So you would have to, if it's like specialized knowledge, you would have to give it a way to access that.

That means it doesn't have any memory to recognize the human's background or to find the main context of the information of the interaction and based on that produce the next result.

So you have to somehow feed in that knowledge. So you could either do that sort of before you, you know, like as you set up the session, you can ingest kind of the knowledge as initial context, for example, and then it has that in context to talk about.

Or you would give it kind of function calls to access knowledge sort of during the session as you converse.

Yeah, I see. Thank you.

Is that?

Yeah, no, I was to find that look at the GPT, for example, during the some turns it can find the speakers or the user's knowledge or the main concept of the information. And based on that, the GPT can produce some results.

That means step by step, gradually it personalizes to the main context of the conversation. So my question is, can it step by step personalize itself to the main context of the interaction and then produce the result to point to the?

So like as long as the context stays within the context window during that conversation?

Yes.

Yeah, it would, it can like it can identify different speakers and sort of remember what was said in the conversation. Like if they introduce themselves with their name as well, it can remember kind of that person's name. And so like, yeah, that's kind of the audio understanding sort of within Gemini.

Thank you.

Okay, cool. Yeah, do you just want to pass it forward there?

Go forward.

Yeah.

Thanks for the nice presentation. Could you share maybe some of your experience on how to evaluate these live voice apps? Because I can imagine that this becomes a lot more complicated than typical apps.

Yeah, so it definitely depends on your use case in terms of like, you know, like what are your requirements in terms of, you know, do you have HIPAA, do you have SOC 2? Like what is the amount of function calls, the amount of guardrails?

So there's definitely a lot, you know, these demos are nice and fun, but like to bring that in a business context, there are definitely a lot more steps involved. And so that is kind of where the partner integrations come in.

So, you know, Life Kit has built kind of their entire business around sort of giving you all the batteries around sort of voice agents. And so I would recommend if, you know, sort of looking at the partner integrations for the sort of real business use cases potentially.

Thanks.

Cool.

Yeah.

I hope you don't mind if I just ask a simple question about the Interactions API going back to the previous talk.

Yes, please.

When's that going to be available on Vertex?

Hopefully soon. I mean, if you speak to some Google Cloud person, some Vertex person, the more you tell them, I need it on Google Cloud, the easier it gets.

Will do.

Yeah, it's certainly not in our control.

Okay, yeah, that's fair. That's fair.

I mean, the API will be the same. So you can start today on Gemini API, start testing. If you need higher rate limits, anything else, you can like always reach out. If you need the Vertex enterprise-specific features, then you might need to wait a little bit.

Can I ask like in terms of like PII in any like conversation history? Do you know how that's like stored in terms of like data sovereignty? Can you specify your own data sources or is that all handled like backend within the API?

So you can always disable storing anything. So we have a store equals false flag. So we would not store anything, but not storing means no server-side state. So if you would like to use this, it's a bit difficult for other Vertex features in terms of data sovereignty where you call the model, I would expect them to be like similar to generate content.

So if they have it today, they will have it there in the future as well.

Okay, cool. That's great. Thanks. Appreciate it.

Cool. One last one. Oh, okay.

Sorry.

No, please.

Thank you. So I have a question about hallucinations. So we.

Sorry, the what?

Hallucinations.

Yes.

So you have shown some examples with the weather that didn't work so well. But how do your clients actually deal with that stuff on production? Because I can imagine that for like some examples that we've seen here, this is fine, but in real life, this is a different story.

So can you give some best practices or how to deal with that?

Yeah, definitely. So I mean, for the demos,

there's definitely a lack of best practices in terms of like system instructions. And, you know, there's a lot that you can do sort of with the, you know, better system instructions to have the agent actually follow the system instructions and not, you know, go off and like hallucinate the weather, for example.

So yeah, I think we have like there's some best practices docs. So I'd recommend kind of going through those. We have an example as well sort of, you know, how to sort of structure

your system prompt and sort of put, you know, your guardrails in there, guidelines, and kind of the tool definitions as well. And so once you built that up, the agent gets a lot better, you know, at following the system instructions and kind of staying within those parameters.

Cool. Cool. Yes, thanks so much everyone. I do apologize for the hiccups, but we learned something and we'll improve upon it. But yeah, would love for y'all to test it out and, you know, let me know over the next couple of days.

I'll be around. What did you find? And let me know your feedback. Thank you. Cheers.