AI Engineering with the Google Gemini 2.5 Model Family - Philipp Schmid, Google DeepMind

Okay. Hi everyone, welcome, welcome. Um, so welcome to our workshop, "AI Engineering with the Google Gemini 2.0 Family." So, as it is a workshop, we are going to keep it super hands-on, so please ke- keep all computers open.

You don't need any Google account, um, like Google Cloud account; you can use your personal Gmail. It will be completely free for you to use, so that's the point. Um, before we get started, can you maybe help me understand how many of you have used Google Gemini before?

Oh, wow, that's— that's cool. That's a lot of hands, more than the last time I gave a talk like this. Um, so what, um, we're going to do: we have, like, 3 slides, so don't worry, not too much.

But we're going to focus on Gemini 2.5, so there's a Gemini 2.5 Pro model and a Gemini 2.5 Flash model. We're going to use the Flash model as it's available, uh, for free tier via API access, so we are going to do coding.

And both models are multimodal by default, meaning they can understand text, images, audio, videos, documents, and can generate text. We also have Gemini models which can generate images, which we are also going to use. And we have now Gemini models which can generate, uh, audio, so you can create speech from— from text.

Um, if you are curious where you can find those nice model cards with all of the feature, the model context, the output tokens, it's on the Gemini docs. Um, as mentioned, we have 2 new, um, text-to-speech models since Google I/O last week, not a week before.

Um, those are really cool. You will try and see them later. And now, for the important details: so I created a Slack channel. If you are in the AI Engineering Slack channel, you should be able to find it.

Um, feel free to use it during the workshop. You can ask questions. I try to regularly check, uh, to answer them, or even afterwards, if you have questions, complete the workshop at home or next week. Uh, I will take a look and make sure that you get all of the answers.

And then, uh, we have one QR code. Uh, it's AI Studio. You can also just enter in your browser ai.dev, uh, or ai.studio, which will bring you to AI Studio. And the other link is— so there's a GitHub repository, uh, with the workshop we are going to do, the workshop.

I can, like, now switch directly to it. So, um,

let's hope if the Wi-Fi gives us some freedom. So the good part about the workshop is that we have Google Colabs, so there's not a lot of downloading, uh, happening, and it will all run in the Colab environment if you have a Google account.

And the other thing is, what we need to do is, in AI Studio, to generate an AI key. So the GitHub repository is now loaded. In the GitHub repository, we have notebooks.

Can I use the slides for you?

Uh, yeah, of course. Uh, sorry. So in the, uh, GitHub repository, we have a notebooks, uh, folder which includes all of our 4 workshops, uh, plus, uh, a 01, which is basically some minor instructions: how to set up AI Studio, how to get an API key, and how to send your first request.

And then, uh, we have one— or the beginning section will be all about text generation, um, getting started, getting familiar a bit with the SDK. Uh, the second part will be all about multimodality: how can Gemini understand images, video, audios?

How can we generate images or audio? And then the third, um, section will be about function calling, structured outputs, the native tools, how can I integrate Google Search into it? And then, I guess, um, with all of the hype currently going on, we— we look at how you can integrate MCP Servers together with Gemini, using it as a model to call the different tools.

Um, also very nice, so there's a solutions, um, folder. The solutions folder includes the same notebooks, but with the solutions. So all of the notebooks include to-do texts and also some code snippets, um, and some comments. So there's a— a mix between working code snippets, code snippets which have some pointers, and straight-up exercises with to-dos for— for you to do.

Um, I will work with you through, like, the existing, uh, snippets, and then everyone can work on the exercises. The idea is to— that we try to maybe use 30 minutes per different section. If you, for example, are already very familiar with how— what I can do with text generation, and I would li- rather, like, look at the multimodalities parts or at the function calling parts, feel free to, like, directly jump into the section.

And in general, we want to keep it very open, very dynamic. If you have questions related to the content, maybe unrelated, please keep them coming, ask them in Slack, raise your hands. I'm not sure, maybe we have some microphones here as well, so we can, like, give it to you to, uh, make it super interactive.

So I guess let's get started. So if you go to the notebooks, there's also a Colab, um, button you can click, which opens the notebook directly in Google Colab. And if you prefer, like, a local Jupyter environment, you can try to clone the repository.

I'm not sure if it works for the Wi-Fi or not. Um, I guess Colab will be the easiest. And as mentioned before, the only requirement you basically have is, uh, a working Google account. Can be your corporate one, can be your private one, can be one you create in the next 5 minutes.

And the first step is, uh, what we need to do is basically go to AI Studio. For the ones who— few who are not familiar with AI Studio, AI Studio, um, is our developer platform to quickly test the models, to experiment with the models, uh, and also keep it very similar to the development code you will be used.

So if I try to run a request, like, um, maybe let's ask something. What's the AI Engineering Summit? I can, on theright side, for example, enable, uh, native tools connected with Google Search. I have our Flash preview model.

I can run the request, and we'll see how fast the model is thinking. And the nice part here is, uh, I can also directly get the SDK code from our request as soon as it's ready. So if you are experimenting in AI Studio and you want to convert it into a Python script or want to play around with it, extend it, um, that's all possible.

So the AI Engineering Summit refers to several events focusing on artificial intelligence and engineering. That's great. And that also matches, like, the one from New York, which was done at this February. It's cool. So what you need to do to get your API key at a topright is, uh, I can also make that bigger.

Maybe it's easier. So we have, uh, "Get API Key at a Top." Um, you go to it. Um, sorry for its German, uh, for being its German, but on the topright corner, there should be a "Create API Key," a blue button.

And in there, you should— it should open, um, some, uh, I can click it. It should open some modal where you can either select your Google Cloud project. If there's none, you should be able to create one. If— yep.

Can you switch to whiteboard?

Uh, yep. Of course.

Um,

appearance,right.

Okay. Good idea. Okay. Uh, once you select your Google Cloud project or create one, you should be able to create one. And once it is created, um, you should have it available as a pop-up. If not, you can scroll down a bit.

There are your API keys. Um, and then the second step would be to go into Colab and to go, on the left side, uh, in the navigation— so I can also— let me quickly change that to light mode as well.

Um, but anyways, left side, there's a key which is called Secrets. And then you enter a name, which is Gemini API Key, and then the value of your API key. All of what I walked through is also part of the 00, um, setup and authentication notebook.

So if it was too fast, you can, like, look up. There should be a screenshot of it, where I clicked, uh, and where to edit. If you are running locally, um, you need to expose the, um, API key as environment variable with the same name.

It should be also part of our, um, notebook. So in the first— so basically, what we try here is we check whether we are in Google Colab. If we are in Colab, yep.

Can you please go a little bit slowly so you'll be fine recording?

Yeah, we— no, don't worry. Like, we go through it one, and then you have enough time to, like, 5 to 10 minutes to set it up yourself. Okay? Quickly, that we—

Can you increase the font?

Yep.

Is there any PowerPoint presentation?

No. No PowerPoint. Code only. Um, so we— we go through the API key setup in a minute. You have plenty of time to do it yourself. And if you have questions, I'm very happy to come to your place and help you get it created.

Just to complete the— the setup, so once you have created your API key, made it available to Colab, or made it available in your environment, best is to open the first, uh, notebook. It has a super small code snippet in it which uses your API key and generates, um, uh, uses Gemini 2.0 Flash to generate a first string.

So our goal next 5 to 10 minutes is trying to get this working. Okay? And again, going back, so we have those Q- QR codes. One is for AI Studio. It's the left one. The other one is for the GitHub repository.

You can also go to AI Studio, uh, ai.dev to enter AI Studio, or go through Google Search. And you can find the GitHub repository on my GitHub account. Um, it's like gemini.2.5.aiengineering.workshop. And, um, I will, in the meantime, change the appearance.

Can you do that?

Yep. Sorry. Sorry.

Where, um, where's the Google Colab link?

Um, so there are in the GitHub repository if you go to the notebook section. And each of the notebooks, at the top, there's a button which opens Colab directly with the notebook.

Okay. Thank you.

Okay.

Okay. Quick check. Are we— are we ready?

I think we're going to go by, um, yeah.

Any no's?

Um, this is on Google?

Okay, cool.

Sorry to jump in.

So the first section will be all about the— the defaults, basically. LLM started with being text-only. We generated text. So what would the first section basically covers, um, all of how can I generate text, how can I, uh, tech— uh, generate text and have, like, a streaming response, how can I, uh, count my tokens.

It's, like, always important,right, to understand how many tokens did I use, how much will it cost. And there are, like, a few exercises for you to try out different models, try out different prompts. It will also go a bit into detail on how the SDK works in terms of, like, which inputs you can provide.

So in the Google AI SDK, we have, um, this, uh, concept of a client. And client has the— the model's abstraction. The model's abstraction has the method, uh, "Generate Content," or, uh, "Generate Stream Content." And I can also make it maybe a bit bigger.

And, uh, each of the— or it has parameter for the model. And the model ID is basically the Gemini model we want to use, which is defined at the top. Uh, so all of those cells use the— the same concept.

So all of the workshop section have the same. Um, if you think, okay, 2.5 Flash is not theright model for you, you can change it to a different model ID. If you have, like, an paid account and want to use the pro version, you can also change it.

Um, and, um, contents is basically our way to provide, uh, data or conversations, chats, and messages to, um, Gemini. So the first, um, test basically is, we ask it to generate free names for a coffee shop that emphasizes sustainability.

And we use the client models, generate content. We have our model ID, our prompt, and then we get our response from— from Gemini. And, um, if you have already set up everything, you can try prompting a few things.

Um, ask it to explain some terms, or maybe, like, just change the model ID. And then we continue with, um, counting tokens. So there are exercises in there, uh, which don't have any code snippets. The solutions part of the workshop has the code.

So if you are getting stuck, or if you want to look it up, uh, what I added, uh, feel free to take a look there. But, um, definitely try it first yourself. If you want to get familiar with the SDK, there are plenty of other snippets.

So exercise is basically just to, um, make sure that you understand the concepts and, um, can practice it. And there are other cells which are partially done. So, for example, the one we have here, which has some code comments and also some to-do calls.

Here, the idea is really to— to help force you not to learn new APIs. So next to the, um, "Generate Content" method, there's also a "Count Token," um, um, API, which we can use to count our tokens. So similar to our "Generate" method, we provide our model ID and then our prompt here.

And basically, what the, uh, API does is now it counts only the tokens for our prompt, since we— we haven't generated something. So we can run it. Um, and we get an input tokens of 11. So the Gemini tokenizer basically converted those 2, 4, 6, 8, 9 words plus a, uh, full stop to 11 tokens, uh, which is then, um, an estimate of roughly $0.000002.

Um, the "Count Tokens" API doesn't expose the pricing. So basically, what I did here is, like, looked up the 2.5 Flash pricing and calculated it. Um, similar to only counting the input tokens, we oftentimes also want to count the output tokens to understand, okay, um, how much does it cost.

So in the next, uh, example, we basically generate content. And each response has a very nice method which is called, or, like, an abstraction which is called a text, which allows us to actually, uh, easily access the generation, but also has a usage meta data object.

And the usage meta data object includes all of our consumed tokens and generated tokens. So we have input tokens. Uh, we have thought tokens. So Gemini, uh, 2.5 is a thinking model. So before generating your response, it first generates thinking tokens, basically an abstraction where it uses more compute to have, like, more room to generate a good answer for you.

And then also the candidate tokens, uh, which are the response tokens at the end. And that's how we can calculate the total cost of a request, where we use the input, uh, token price, and then our candidates' tokens and the thought tokens.

And, um, for this case, it would be, um, less than, like, 0.2 cents. Yep.

Is 2.5 Flash also a thinking model?

Yep.

Okay.

Yep.

I just see different multipliers.

Yep.

Is there a different price for a thought token in your estimate versus?

No. So, um, input and output tokens are calculated different. Um, as.

Input versus output?

Yeah. So we have prompt tokens as basically the input tokens of your prompt. And then we have the candidate tokens, which is the your response, and the thought tokens. And those basically have the same pricing. And the output tokens are much more expensive than the input tokens because that's where the computation mostly happens, and the input is just one encoding.

So that's why you always see, like, the big difference in output, uh, input pricing versus output pricing.

Is that order of magnitude more expensive?

More.

More than order of magnitude?

Yeah. So for Gemini 2.0 Flash, which is our, uh, most cost-effective and cheapest model, the input price for 1 million is 10 cents, and the output price is 40 cents.

Okay. What if someone says, uh, like, 39,000, like, is that really expensive, or is it just a function of, uh, the prompt?

So you mean, like, why we got a 639 thought token? So that's sadly not, like, directly visible to us. So we can, like, look up thought summaries. But basically, the model generates, first of all, like, a lot of, like, reasoning.

Of course, in our case, like, we— we ask it to generate a haiku. It might be not, like, the most, sorry, most difficult, um, prompt. Um, you can control the thought tokens with something called thinking budget, where you can limit how many tokens the model has to think or to reason.

So you have a some sort of a cost control, but it's basically done dynamically based on your prompt. Okay. Yeah, there's also a question.

Yeah. I was just curious about the price of the output tokens.

Yep.

I, I was just looking at the documentation. It says 350 for thinking.

Yep.

60 cents for not thinking.

Yep.

Why? I'm just— it, it seems different from what you've got here.

Yes. So, um, I can— let me open the docs and make it— make it easier. Um, so, um, Gemini 2.5 Flash is a hybrid model. So you can use it with thinking and without thinking. And without thinking, basically, the computation is much more cost-effective.

As, um, you might know, like, the transformers is all based on intention, which isn't, like,

n by nx, like, kind of calculation. So it gets bigger and bigger, which means it gets more and more and more compute intensive. And, um, without thinking, it's for us much easier or inefficient to run. So if you set, um, thinking to, uh, 0, or, like, thinking budget to 0, uh, you're all— you will have 0 thought tokens, but you have, like, your candidate tokens.

And those candidate tokens will then be charged with 0.60 cents. But once you— once you use thinking, meaning a thought thinking budget greater 0, you will, um, pay the price for the thinking tokens and for the output tokens.

And that's where's the, the $3.5.

Is there a particular budget for thinking?

Yes. Like, I will open the documentation, so we'll see it in, in one second.

Okay. And how do you turn a thinking on or off? Do you have to control it with the budget, or is there a flag? I saw you have some config, general config.

Yeah. So, um, in any case, if you have, like, any questions, the Gemini docs are a great way to, like, find the answers, or, like, just ask Google or Gemini directly. And on the model capabilities, we have the thinking section.

And there are, like, the thinking budgets. And if you want to disable thinking, you basically set the thinking budget to 0. Uh, you can control it. It could be, uh, an integer between, like, 0 and 24,000. And then setting thinking budget to 0 disables thinking.

So that's your way to disable thinking.

Is the thinking budget, like, a number of, of reasoning traces to do? So if I say 4, or how does it?

No, it's tokens. So.

Oh, it's tokens.

Yeah. So we, we have.

So it's expanded.

Yeah. So we have seen in our example here, we had 600, uh, a bit more than 600 thought tokens. So if we would set our thinking budget to 512, it would be a maximum of 512 thinking tokens.

So you cut it off. But how, how does it know how many reasoning traces to do? Or is that not controlled? Doesn't it— does it spin out other reasoning traces or behind the scenes?

No.

No. Okay.

Okay.

I see. That's just the intermediate number of tokens for the chain of thought?

I guess.

Okay.

Okay. Continuing, uh, with our, um, notebook. So and please continue yourself. Like, I'm— like, you can do it with your own tempo, even, like, do it fast or slower, just to make sure that we are on, like, the same page.

And I guess, like, the most interesting part about, like, streaming and, like, LLMs in general, we all have seen it with ChatGPT, is that waiting for the whole response is a very bad user experience,right? Like, who wants to wait, like, 60 seconds, 2 minutes for a response?

So that's why, um, everyone now kind of uses streaming. And with the Gemini and the Gemini SDK, it's, like, super easy. So instead of having just generate content, we have generate content stream. Same input parameters, except that we now get, um, an iterator back from our, um, call, which we can loop over.

And we can, like, print our, uh, junk, or, like, stream it back to our user using, uh, an HTTP service.

Okay.

Cool.

And then similar to other models, Gemini is a, uh, chat model,right? And, um, maybe you are familiar with the OpenAI SDK, where you have the concept of messages, where you have, like, a different, like, inputs per user turn, assistant turn, user turn.

And it makes it very, I would say, still complex to manage yourself because you need to keep track of it. To make it easier, we, um, added something which is called a chats API, um, which basically does all of the state management, uh, on the client, but as part of the SDK.

So you can create a chat, uh, with your, uh, model. And then you can basically send messages into the chat session. And the user, um, like, in this case, it's like, we are planning for a trip. Um, we send a message.

We get back the response, but also our chat session includes the user prompt and the assistant message. So instead of needing to create this object of user turns and model turns, we can directly, uh, continue with, like, sending our next message, um, asking, um, for some, um, good food, uh, recommendations.

And since we are in, like, a conversational setting, the model knows that we are, um, or, like, it mentioned for us to go to different European cities. And based on, like, the next request, it uses, like, the whole, like, conversation as history, um, to get our response.

Uh, I can, like, also quickly print the response here.

And it

get some different examples. And then if you, of course, if you need to store it to a database on general, um, you we have a nice get, um, history method available, which allows you to basically retrieve the complete current state.

And you can, like, store it or update it or whatever you want to do. Yeah.

Is there a client abstract on the history? Is it on the client or on the?

That's only a client abstraction. So the backend receives the same request. If you would send it, like, as a single request, uh, with an array, it's only a client abstraction to make it easier for developers, people to quickly build.

Okay. They're on my laptop. But the tablet is better if we go there.

Yep. And then, um, similar to OpenAI or to other models, you can, like, give the model some kind of system instruction to have it behave differently, respond it in a different language, make sure it respects, like, policies or guideline you provide.

Um, this can be done through, uh, generations, uh, config. So we have another argument now in our model call, um, next to the model ID and the content. We have a config that we can provide our systems instructions.

And similar to the systems instruction, we can provide, um, other, um, generation configurations. So temperatures can be used to make the generation more creative or more deterministic. So if you, for example, build a retrieval augmented generation, where you want the model really mostly trying to use what you provide as a context, then you would normally set a temperature to a very low, um, value.

If you work on some content writing, marketing, you would set the temperature to a very high value. We can control the max output tokens to make sure that we are not exceeding some budget, some length. And top P and top K are also ways to make our generation more diverse.

Is this where you set the thinking budget? Budget for thinking?

Yep. Um, so it let me open. So here we have the similar in the config. We have the thinking config. And in the thinking config, you can basically set the thinking budget or also include the thoughts, uh, in your request.

Okay. And then, um, I think what's very more unique about, uh, what we can do with Gemini is that we have direct, direct support for files. So in this case, I download the Adventures of Tom Sawyer, uh, a book completely, store it in a file, and I use the files, uh, API to upload the file to a Google Cloud, uh, storage bucket.

Um, that is, um, free for you. So, like, if you, uh, don't want to use your own corporate bucket or whatever bucket, with each AI Studio account, basically, there's your personal, um, bucket, which stores the file for, I think, one day.

Uh, but you can control the, the time to live. And, um, instead of, um, needing to provide the whole file with your request, which can be very, um, intensive, you can upload it and then instead just provide the reference to the file.

And what, um, the Gemini API does behind the scenes, basically, it downloads the file on the backend and makes it available inside your prompt. And similarly, here we can, um, we uploaded our book. We passed it into our contents, uh, list here.

So we don't know not longer have, like, a single prompt. We have now an array with our file, and we ask it to summarize the book. And it was also done while I'm talking. And then we can also see, okay, the token usage now we had for, uh, was 100,000 tokens.

So much bigger than what we tested before. And using the file API makes it very easy, uh, also to work with, with PDFs, which we'll, um, see in, like, the next chapter. And then, um, as an exercise for you is, um, to combine a bit all of those things.

So how can I use a book to use the chat JSON to chat with our, uh, model to help me better understand it? Yep.

So the file, uh, it will be it's not a ranked system. It's just in the context window.

Yep. So what you do is basically you upload the file from your client to the cloud into a bucket. And when you send the request, part of your I mean, I can, like, show it. Um, part of your request, uh, would be only the reference to where the file is stored.

And what the Gemini API does behind the scenes is it loads the file into where the request runs and then puts it into the context. So you can use now this file pointer for all requests, and you don't need to send it every time.

And here we also have, like, the URI. So that's basically where, um, our file, um, is stored or can be accessed as well. Yeah.

Um, how do you deal with that? Is that that file is that, um, hidden through obscurity? So is it only available to the chat agent with reference to itself, or?

It's only available to your user. So when you send a request, you send an API key. And this API key is basically used to get the file. So nobody else can access the file.

Oh, it's just hidden by the API.

Yeah. Okay, cool. Yep.

Can I ask you one question? The file is too big, um, that it's bigger than the context window. How can I handle that with no rules basically?

So if you would use a PDF which has, like, more than a million tokens, basically, what would happen? You would receive an error most likely, uh, with, like, saying that, uh, the file is, like, too big in terms of, like, uh, context.

What you can do is basically you can use the, the file, uh, the file to count the tokens. So we have the file now here, and we can client, uh, models, count tokens.

Can I ask you? Can you really need the whole file? Um, so what can I do?

You would need to chunk it then. So if the file doesn't fit, it doesn't fit. And you would need to, like, think about, okay, can I chunk it? Can I summarize it? Can I maybe do use other techniques to first extract the important information?

And once I have, um, context, which is, um, smaller than the maximum context of my model, you can provide it again. Yeah.

It doesn't have, like, load context?

No.

Because it's chunked?

No.

Okay. There are any costs associated with the file transfer?

Definitely the files files upload?

Uh, are there any costs associated with the file transfer?

No. No. So it's like but it will be deleted. So don't expect the file you upload now to be there an hour or a day. Um, but you can, like, use the same concept with, uh, Vertex and your own bucket, where you have, like, more control over it, where you can say, okay, maybe I want to upload it using a different API call or already have it available.

That also works.

Okay. Oh, so there's an entry point to saying, use this in the Google storage bucket?

With Vertex AI, yes.

With Vertex AI?

Yes. It's roughly the same, but you need to set up the client differently.

It's a different entry point?

Yep.

I see.

Okay. Yeah.

So for the PDFs, but.

Yeah.

Extract, uh, just the text, or will it also include the proxy or all the storage content?

Yeah. Good question. So, I mean, we can maybe directly jump into, like, the, the, the PDF section. So, um, continue with section one or jump to section two or three directly. What I will do is, like, in the section two, which is all about multimodality, meaning we will cover, uh, visual understanding, audio understanding, videos, and document processing.

And that's where I will, like, jump to. Um, and part of the, um,

okay, being connected. Part of the, uh, working with PDFs, basically. So similar to what we have seen a minute ago, we have a PDF. In this case, it's basically an invoice from a supermarket. Um, I upload it, and I ask the model, like, what's the, the total amount?

Uh, we can run it. And what happens behind the scenes, I can show oh. It's not here. Wait one second.

Okay. We don't have the file here. I will upload it quickly. But what happens, um, behind the scenes is we run OCR on your on your PDF and provide the PDF as image. Um, so you don't need to do it manually.

So there's no, like, hey, it's a PDF. Let's convert it to an image and then run OCR. And then I provide the image. And the OCR, that's not needed. Um, we are doing it for you.

Okay.

Any other questions?

Yep.

Yeah. Yeah. Yeah. Yeah.

Okay. I'm just curious what the what the point is of passing in the OCR text if you're passing the PDF as an image or anything.

Yeah. So OCR isn't or, like, the image understanding is not perfect yet,right? If we reach a point where the model understands it, like, in the same way as without the text, then I guess there's no point. But, like, based on, like, what we have seen and also what the industry does is you receive better results when you provide the OCR plus the image.

Um, yeah. That's, that's basically it.

Yeah.

Maybe you kind of missed that. So the PDF itself, do you actually look at it as an image? So if there's tables and diagrams.

Yeah.

And you extracted an image space, or you just take there's also sometimes text hiding in the PDF for cut and paste.

No. I, I think I I mean, I don't know exactly, but I think it's just basic OCR, nothing special, no magic, and then a screenshot of the PDF.

Oh, good. So it is an image space.

Yeah. So it's both. Okay. We can try again. We have now the PDF available. And in case, um, the workshop has multiple sections with files which are being part of the repository. So if you run into, like, a similar issue, especially for the image understanding part or the audio understanding part, uh, and you use Colab, you might need to download the files manually and then upload it.

But in our case, so we have now our invoice. I think we can, like, uh, quickly show it.

So I was shopping in Germany. We have a co uh, supermarket called Rewe. And I basically bought some butter, um, some bread, like, some sweet potatoes. And, uh, we prompted it and asked, okay, what's the total amount? We can see here the total amount is, like, 2020.

Now, let's see if we got it correctly. And we got it correctly. And it also correctly extracted it in German, even if I prompted it in English, which I think is, like, pretty cool. Okay. Let's start with the image understanding part.

Can I ask one quick question?

Yeah, please.

Sorry. About the thinking budget.

Yeah.

Is so it's not just omitting the thinking process that you see. It's actually changing the change the chain of thought.

I'm not exactly sure what happens. I only know that by defining the thinking budget, you can limit how many how many tokens will be used or generated as a maximum. And very similar to what OpenAI has with, like, low, medium, high effort, we have a bit more granular control.

So we could technically do the same, say, like, low would be, like, 1,000 token thinking, medium would be maybe, like, 12,000 token thinking, and, uh, high would be 24,000 token thinking. Uh, and it would then use, like, at a maximum those tokens to before generating your response.

But what exactly happens, I can't tell you.

You said you could set it to zero, but the model is trained to go through that chain of thought process,right? So that might actually skew the results.

Yeah. So without thinking, what we have seen, especially on more, like, math type of question where the model benefits from, like, the reasoning, uh, the performance is a bit worse. But for, like, general everyday use, especially image understanding or, like, OCR, you can, like, easily run it without it.

The truth, it would be you need to try. And I think the, the real benefit here is that you have those granular controls. So you can run evaluation of thinking budget zero, 1,000, 2,000, 4,000, and see how it impacts your evaluation.

And then you can, like, calculate for yourself, okay, how much am I able or, like, what's my, my maximum cost of it? And, like, what's the, the accuracy I need to reach? Okay. More questions, or should we continue?

There's a question. Yeah.

Do you support any other file types or is it PDF?

No. It's like, uh, there's documentation for it, but it's like, uh, JSON, PDFs, all different image types, all different video types, uh, all different audio types. So all of the multimodal, um, features we support. Um, if it gets a bit more specific with, like, .jsx file and .vu files for, like, web development, we are working on it, but it might I mean, you will see an error.

Uh, and the easiest way is to just replace it with a .txt.

Is it .txt file?

No. I would say, like, there was some like, you would need to, like, use, like, Mark it Down or, like, another library to convert it and then or, like, copy-paste the input and yeah.

Actually, I have two questions. First question. Is it because including the OCR along with the images, uh, have a better performance? Um, and that's why we include the OCR, um, with the, uh, images?

So I'm not exactly sure what our researchers did. The only thing I know is that we get better performance when you provide the image plus the OCR. So I guess there's a benefit of having both.

Awesome. And the second question is the script you just showed us.

Yeah.

Is it it's actually the same thing as we saw on the UI interface of Gemini, like, when we upload the PDF files?

Yes, kind of. So the, the UI or the AI studio, of course, doesn't use Python. Um, but it calls the same APIs behind the scenes. So the API behind the file upload one is the same API we call from AI studio.

We call the same exact model. Both have the same, uh, exact parameters. So you should be easily what you test and experiment in AI studio. Can you convert into code and run it locally? There's also this, um, get code, um, button.

So if you're on AI studio and I mean, we can quickly try uploading our invoice again and

acknowledge it and use our prompt.

So

here.

Now, we run basically the same request, and we can now we also have this, um, it's a bit hidden. It's like this code, um, button at the top, which, uh, where you can get the exact same code. Basically, it takes a few seconds.

Yes, the Wi-Fi is super bad. But here, you get, like, the exact same Python code where you create your client. We have our model. In this case, as we uploaded it manually, we provide the document not as a file URI.

We provide it directly as base64. We have our prompt. Uh, we have the model request already since we generated it. And then you can continue. Okay. Cool.

Uh, question.

Yeah.

Uh, can we still use the code tokens, um, function for when we use the code?

Yeah. So, I mean, we can quickly so try it. So we are here. We have our PDF. I think I guess it's very interesting to know, like, how many, um, tokens will we use. So, um, let's quickly

count tokens. So

if we go here,

we have our count tokens. And now we use the same contents. We have the same model ID. We have our prompt, and we have our PDF. And let's print our token count. And alternatively, what we also have done, so if you use the, uh, response, so you run a request already, you should have access to the res or usage metadata.

Yes. And so we have our count tokens. So our PDF here is converted into, like, um, roughly 500, uh, tokens. Um, and the, the prompt we have is, like, around 20. And if we compare it to, like, the request we run, we see, okay, we have the same exact amount of prompt tokens.

Uh, we have prompt details. We have our output tokens.

Four tokens of 42 and candidate tokens of 78. Okay. More questions or yeah. Okay.

Is the 560 here based on the OCR content of the files?

I think we don't charge for the OCR. I think one image is roughly 500 image uh, like, 500 tokens.

Gotcha. Okay.

Um.

Does it really vary much depending on the.

Not for you. No.

Thank you.

Okay. Yeah.

Um, can you share more about the parsing? Uh, is it just OCR text or does it understand IRDs and images inside PDFs as well?

So

the PDF is converted into an image, and we provide the image. And if the image or, like, the PDF has tables, visuals, mind maps, uh, the model is trained on similar data, so it will definitely understand parts of it.

I mean, we can we can try it. I mean, maybe we can, like, search for some mind map and ask it something. Um, you maybe you can start thinking about a prompt while I'm searching for a mind map

if Wi-Fi allows us. Okay.

Okay. We have our mind map. I guess it's just a mind map on how to do mind maps. Um, any, any idea what you would like to know?

What's the most central concept or idea?

Sorry?

What's the most central concept in the mind map? Yeah.

Like this?

Yeah.

Okay.

I mean, I'm opening it, but I think on the small scale, it looks correct with how to mind map.

Answers?

Thank you.

Perfect.

I think the, the issue that we're looking to happen,right, like, for example, there's a table. What's the fidelity of the table? In fact, because we're going to capture it as an image, will the fidelity of the image able to represent that table well?

If, for example, let's say it's a complicated table. So those are sort of challenges,right, that you can basically see here by the PDF. And given that you're saying it's only good with OCR and only then from OCR and then only through an image, that if anything gets missed out, for example, or even if the mind map came back to be something without images, nothing,right?

But if that's the basic challenge that you run into when you start your learning document.

Yeah. So where we have seen the most success is when you, like, separate a bit. So there are already, like, ex very good existing methods which allow you to extract tables or other visuals from documents. And when you then, like, work with those images and tables directly and not like, you the way how we work changes from, like, previously we provided a table, run OCR, then ask our question.

Now we directly ask the question based on the table as the models get so good in, like, the multimodal understanding that it can combine, like, the different aspects. I mean, we can try it maybe as a good example as well.

Um, maybe we find some nice, I don't know,

invoice image or something. And then we can, like, ask it maybe to add something or to combine it, which would be very hard, I guess, for, uh, a normal model. But in general, like, especially Gemini is so good with, like, the multimodal understanding.

It also videos is, like, that's, like, my most new favorite thing is, like, take a YouTube video, which is, um, below one hour, put, put it into AI studio and, like, have it summarize it. Or if you have, like, any specific question, like, it, it's so much faster than, like, sitting there, even, like, watching it in, like, two speeds.

You get a response in, like, 80 seconds or something. And you can, like, even, like, ask it to extract specific timestamps on when somebody something was said or to help you, like, section it very easily.

Is it the protection against, uh,

sorry, can you speak a little bit louder? Is it the protection against prompt injections from the files used by the researchers?

I don't think so. So AI studio is, like, very developer-centric, and we don't want to do too much black magic. There are safety filters and control which you can configure in the SDK or in AI studio. And in AI studio is under advanced settings.

I have safety settings. I have all of them off. But if you, like, want to filter on very explicit content or hateful content, we run some classifications basically before and after to make sure that you are not creating, uh, for your users harmful content.

Okay. I have our invoice image.

Okay.

Okay. What should we ask?

How much it would cost if we subtract the pedal maybe?

Excuse me. I'm going to quick question.

Yeah.

Uh, so does the PDF need to be really well formatted? Because the kind of problem that we face, generally the PDFs are auto-generated,right? And that's why it's not always, uh, very well formatted. Like, there's errors, filters, what about, you know, talk of each other, images, you know, talk of each other.

Even then, I mean, have you, uh, you guys, like, seen the performance of the new evaluation on if those type of PDFs are uploaded or as a bunch of PDFs? What type of, um, uh, responses? How the responses are?

So I certainly don't have an answer for this. I think we you can always think a bit about, like, would we as humans have struggled to understand those PDFs if they are, like, switched, switched up? If yes, then most likely the model will do as well.

I think one very nice part about Gemini AI studio and the Gemini API is you can get started very quickly. Like, all of us kind of set up within, like, 20 to 30 minutes a free account, uh, with access to Gemini 2.5 Flash via API 2.5 Pro in the UI.

So the best thing always is, like, to test and to explore and evaluate. And even if you need to run, like, 1,000 PDFs, it's not very cost or, like, expensive anymore. So it's really best thing is to run your own evals to get some more than, like, I try five PDFs in the UI, really look into it.

And if you have, like, any questions or problems, like, best way is to reach out to us. We have teams helping and building with customers. And then we can, like, iterate it on it together. Going back to your PDF, any question, any prompt idea?

Yeah. I have this.

Yeah.

I know you explained about security, but there are mandated, uh, security by law, which are LLM firewall or, uh, benchmarking automation or LLM guardrails or SCM, posture management, uh, mandated by CISA. So when I implement those rules, um, for, uh, either, uh, uh, defense or, uh, for financial applications, how do you just a small set of I know that Google is pretty strong in setting up those, um, security measures from a different perspective too.

How do you integrate this GoLab and the other security measures?

So I guess for those type of environments, uh, where you have a lot of, like, compliance regulations, best is to work with Google Cloud. So everything we do in AI studio is also somewhat similar available in Google Cloud in Vertex AI.

And Vertex AI provides more features for those kind of use cases. They I'm not exactly sure, but, um, they definitely have more, like, information they can provide on how to handle all of those things and those guardrails with Gemini.

So that.

So the, the, the suggesting me to, um, go on a go, uh, global Google Google GCP environment. When I go global on GCP environment, then there are certain, uh, performance and, uh, cost being more here, such as, um, um, so there are there's a guardrail for bot-based and there's a guardrail for organization-based, and there's a guardrail based on the industry industry-based.

So we have to, uh, split those platforms into, uh, from grand most granular level to all the way to industry-wide level. So, um, that's becoming when I do it globally at GCP level, then it becomes, uh, more expensive and as well as, uh, there can be a performance issue.

Yeah. So I know that there are regional endpoints for Gemini and Vertex AI as well. And also at Cloud Next, um, they announced a new Gemini on-device kind of thing where big companies can buy basically a huge box where Gemini is pre-installed and it gets delivered to your environment.

I'm not exactly sure about the details. Easiest is, like, to, to do a quick Google search and, and look for it. But those is exactly where, where Vertex provides you more features and support than AI studio.

Yeah. Thanks. One of those ASTAC, uh, 2025 conference, uh, someone from Google, uh, also represented and, uh, that's a good platform, but I didn't know how you can integrate from here to there.

Yeah.

This is the most granular level.

Yeah.

How do you integrate from the most granular level to the global level? That, that's the part I was interested in. But anyway, thanks for.

Okay. Maybe back to your PDF question, uh, regarding, uh, what to ask. So instead of, like, asking what the, the total sum was, I ask it to, to sum up the unit prices, which worked very nicely. Of course, it's a, a very well-formatted PDF in this case, but, uh, the image understanding is, like, very good.

And the new way of how we should think about it is, like, I should ask the question directly based on the image before doing too much of processing we, we have been doing in the past. Cool. Okay. We are now at, like, halftime almost.

I guess it's time we move maybe a bit away from all of the multimodality part into more about the, I guess, agentic, um, parts, which are, I would say, definitely more interesting, at least to me, um, especially if you combine in with, like, the multimodality parts.

So, um,

okay. Um, so part three is, um, all about structured output and function calling. Do you know what structured outputs and function calling is and, like, how it roughly works? Any hand signals? Yes or everyone? Okay. Not bad.

Um,

okay. So the part three, um, goes continues with PDFs, um, as they are kind of very interesting. And also, um, we want to do structured outputs. So structured outputs is for us a way to create more structured data structures from text, so which we can use to work way more easily afterwards,right?

And at the end, we prefer structured output much better or much more because we can integrate it into other APIs. We can connect APIs. And Gemini supports or the SDK supports Pydantic. So Pydantic is a very nice Python library, which lets you create those data structures.

And also we can create nested data structures. So here we have a recipe, uh, with a name, ingredients, which is a list of strings. And then we have a recipe list, which is basically a list of our recipes.

And we can provide it in our configuration. So similar to the generation arguments or our thinking budget, we have, um, a response type and a response schema we can provide. Here we ask it, okay, can it generate two popular cookie recipes for us?

And we basically force it to use our, um, structure. And I already, like, nicely, um, printed it here. But if we look at the, the raw response, um, of our model, we get back a, a JSON, uh, with, um, all of the different input fields.

And there's also a nice dot parse method, um, which allows us to convert it back into our Pydantic, uh, schema. And then we can access all of the, the data points. And as we had our invoice, uh, we can now maybe, like, I complete the exercise with you.

Um, what if we, like so we ask about the total amount,right? But when working with PDFs, normally we want to have structured data as a result,right? Text is not very helpful for us when we want to put it into a database or want to work with it.

We really need those data schemas. And what is very nice about Gemini is that we can basically combine both of it. So we use our, um, structured output method with our multimodal capabilities for files. And we can provide our, um,

file.

Oops.

Okay.

Oh.

We need our invoice there. So maybe it's a good example. So I didn't change the recipe, uh, from our, like, data structure, what which we want to create. And we ask it to extract the information from our PDF.

Our PDF is an invoice from a supermarket. So it doesn't have a recipe name, ingredients. So Gemini did not generate or hallucinated something. So we get back a, an empty recipe list. So if we now change it to our invoice data, um, we should hopefully then see the correct extracted information.

Yes. So we extracted the date, all of the items we bought, and all of the different prices. And with that data now, it makes it much more easier for us to work with,right? If I have some kind of automated system where I need to, like, take in invoices, any kind of, like, PDF document, I can now provide, uh, a structured to what information I want to extract.

And Gemini does basically all of the matching for us, which is, like, super nice. And function calling basically is the same idea, but instead of, um, having, like, a data structure, our output is a name and the argument.

So similar to what we have, um, uh, we, we create, uh, a this case it's like a function declaration. It's a structure of, like, how a function signature is done. So we have a weather function, which has a name, a description, and, like, the properties, uh, which we need to provide.

Um, there's the same function, but just as Python code. And with function calling, we provide the function declaration and our prompt. And then the model generates a struc a structured output, which has the function name it wants to call and the, um, input and environment or the input arguments for how to call it.

So in our case, we have a weather function. We only have one location. Um, we provide it, uh, similar to all of the other configurations in our configuration, uh, argument. This time we have tools. And we want to know what the weather is in Tokyo.

And obviously, what's the weather fits into the description of our function as it helps us retrieve the current weather. So if we run it, the model, uh, instead of generating a nice response, wants to call the get weather method and with the location Tokyo.

If I change the prompt to hello. Oh, makes sense. Uh, one second.

So if I change to hello, we don't have a function call,right? Like, the model correctly understands, hey, that's just a greeting. Let's response and, like, how can I help you? So but we want to call a function. So we have what's the weather is in Tokyo.

And then the next step is for you as a developer, uh, to call the function,right? The models cannot call or invoke directly the function. What you would normally then have in your, um, code or in your applications, a way to identify, okay, which function is called.

Could be a simple switch statement, um, to check, okay, what's the name? And if you get the name, call the method with the provided argument. And then what you do is the output of your function is the next user input.

Uh, so the model generates this name and arguments object, and we provide it as a user the output. And in our case, it's the result. And if we look at the weather method we have, it's basically some dummy data about the temperature, the condition, and where it is.

And then the, the model generates a very nice response. So we have user input, model has a structured output, user provides the structured response, and then the model generates a very nice user-friendly response. So we call our function.

The function returns the weather. And then the model generates a very nice response, which is the weather in Tokyo is sunny with a temperature of 22°C and it feels like 24°C. So you can think about it, okay, that's how I can integrate tools or make it or convert my LLM into an agent more or less or a way to call something.

Um, the get weather method can be anything. It can be a database call. It can be a real API call. It can be, I don't know, reading emails, sending emails, all of the things we currently see with all of the MCP hype basically going on.

And MCP servers have tools as well. And those tools of MCP servers basically expose the same, um, declarations. So MCP servers has the tools def defined get weather, for example, if it is an weather MCP server and it has an endpoint or a method which you can call, which is list tools.

And this list tools, uh, method would then return the schemas of our functions, which look very similar to what we have created here. And then what you do on your LLM side or client side is you take those schemas from the MCP server, provide it into your LLM call, and then the LLM generates, depending on the context, uh, the output as well, which is structured.

And then instead of calling the method on the client as we did with our get weather, you would use the MCP client again and then call the remote tool. So very similar to what we have done here on the client side manually, but more abstracted away and more managed.

And of course, the benefit here is that not everyone of us needs to implement the get weather method. It's way easier to use, like, the weather MCP service from, I don't know, some weather provider. Similar, um, you don't want to use maybe your own personal Google Drive MCP server.

This would be if Google creates those. So that's the whole idea why MCP servers is kind of so cool. So, and, um, yeah, yeah, yeah.

How many tools can we have for the LLM?

So currently, we are working on improving and extending. Um, but currently, the suggestion is probably between 5 to 10. And, um, if you have more tools, you can use embedding models to basically filter. You have the user prompt.

And what you would do is basically run some similarity matching between the descriptions and, like, what makes sense, what doesn't make sense. And then you only, um, put the top tools to speak.

A lot of questions. I think so Anthropic in this year, they introduced a notion of, like, take, like, their model and then you run in a loop for about two hours or two and a half hours. Is there a similar method for even Gemini?

I'm not sure about the claim from Anthropic and, like, how you do it. The only thing I can share is that Gemini 2.5 Pro was the first model to complete Pokémon Blue, uh, which ran, I think, for, like, 200 hours straight.

Um, the only, like, the, the big challenge here is, like, so we have, like, a limited context,right, which is for Gemini 1 million. So if you would continue for two hours, you would definitely run out of, like, those two, like, 1 million tokens.

And Anthropic, I'm not sure what their context currently is, but I'm pretty sure two hours is not enough to run out. So what you most likely do is you summarize, you compress the context and the conversation and what you provide to your model.

So yes, I'm pretty sure Gemini can run for more than two hours, but it depends on, like, what you want to solve and how you are going to solve it. Yeah.

If you call it nation of two, it seems like the background traces are all hidden. Is that correct? And is there any way to access them? Because you can imagine if you use them for managing performance.

Yeah, that's correct. So native tools are the next section. We can go into it, um, in one second. And yes, the currently, the native tools are not, like, being returned in a way we had with, like, user assistant.

That's basically happening behind the scenes. And what the user gets is the final, uh, good assistant response. Um, that's great feedback, um, regarding, like, can we have them or not. Like, very happy to take it to the team.

I can definitely see why it would be helpful for you, for, like, people to directly build with it. Um, but for now, it's not the case. And speaking of native tools, so, um, Gemini can is basically trained to do native things.

So function calling is very generic. We generate the declaration and can try to do everything. Uh, but native tools are much easier to use as you don't need to define or create a declaration. And they work basically on the backend side, so you don't need to execute anything.

And as native tools, we currently have Google Search available. Um, basically, all of the native tools are in AI Studio here. So we have structured output. This is not a native tool, which we used to get back the structures.

Code execution is a native tools and basically means that we or Gemini runs code for us. So, uh, function to sort the top five

cities based on population.

So what it can do is, like, run Python code for us. So if you prompt it to solve a task using Python, um, it should run for us the Python code. So it generates the Python code, and it did not run it.

That's a bad example.

How do you get create a function? Maybe you said run.

Sorry?

Your prompt had create a function.

Oh, did it? Ah, so okay, my bad. Not Gemini's bad.

Use Python. Let's see what it does. Um,

okay. Yeah, now it gets. Thanks. Perfect. So we have some reasoning, and then it generates executable code. Executable code is also provided via the API, um, which runs for us Python. So it writes the Python script. It executes the code, um, and it generates the MAT plot lip chart.

And normally we okay, in the notebook, we have an example available on how you get the chart if we look into the code execution tool. So similar story here. We run it, and it returns the markdown. And you can also, like, it can regenerate and return, um, images.

Next to code execution tool, there is the URL context tool, which basically allows you to provide a URL as part of your prompt, and we'll extract the information from the website behind the scenes and make it available into your, uh, context.

So instead of going to a website, command A, command C, command V, uh, we do it for you. So in this case, I ask it, okay, like, what is the other benefits of Python, um, from, like, the URL?

You can provide up to 20 URLs in one request. Behind the scenes, it goes to the URL, extracts the information, provides it as part of your prompt. And if we look into our prompt, yeah, here, uh, it returned our nice MAT plot plot, um, chart, uh, which is very cool.

And then a final tool is, of course, Google and Google Search kind of makes sense. Um, so let me find it. Yeah. Uh, you can allow or enable Google Search, which then, um, allows Gemini to use or do a Google Search behind the scenes.

And what happens here is that, uh, it takes our prompt. Um, in this case, what are the latest developments in renewable energies? And first, it converts it into one or multiple Google Search queries. Then it executes those Google Search queries, provides it back to the model, and then the model generates your final response from your user input and from all of the search results.

Currently, we don't, as mentioned, don't export or expose those tool calls, but, um, I guess especially helpful or interesting for, um, Google Search is that we have, uh, a grounding metadata object.

Is it ground?

Yes.

So we have grounding support, uh, which basically allows or points exactly to where it got or, like, which information refers to which, um, source. And also grounding meta information, which websites there is, um, which websites were crawled. And in our case, so we have the yes, the.

Is it bot tool with these tools you say, or is it just part of the regular token?

So, um, for both code execution and grounding and URL context, we increase the context size,right? Those contexts will be built. Uh, tokens will be built. Uh, for Google Search, um, there's a free tier from, I think, 1,500 Google searches, which are free.

And then afterwards, I think they have $35 per 1,000 searches. Uh, code execution, the Python running is free. Um, URL context is currently in preview, so I'm not sure what the if and what the pricing will be. Um, yeah.

And I think, like, what's, what's very cool and, um, is part of the exercise if you already have done it or are going to do it later is you can combine those. So you can use the Google Search tool with the URL context tool or the code execution tool with the Google Search tool to basically have it done more agentically, like first search for, like, what's the latest React or Python version, then write a Python script and run it and return it, um, which makes it very nice to use.

Are you guys planning to also expose the Deep Searcher tool, uh, or?

Sorry?

The Deep Research tool that Google has, is that also going to be released?

Yeah, we, we have heard, like, a few people would like to have the Deep Research API. I think the more people ask for it, the more likely it will going to be. Um, maybe something I can, like, plug and share is so I, um, as part of, like, our team, we, um, yesterday we open sourced an example for how you can build your, um, your own deep research using LangCraft with Gemini.

Very similar to, um, wait. Very similar to what basically the, the whole deep research agents do. We have a question, we generate queries, we then run, like, multiple web searches, and then, like, reflect and see, okay, was the user question already asked?

Do I need to do more research? And then you have this kind of loop for, okay, do I need to search other tools? Uh, it's completely open source. It uses all of the things we have currently seen today, Gemini 2.5 Flash, Gemini 2.0.

Um, so if you want your own deep research, that's probably the best way to start.

And, uh, what is the pricing? Are you still getting, uh, charged based on the, uh, tokens, uh, or the LLM calls?

Yeah.

So if there's no additional, uh, price for token calling, uh, uh, of course, if those tools are calling some APIs, then there will be additional pricing for those APIs, but.

So, so there is additional pricing for the Google Search. Uh, we had it, like, two minutes ago. Basically, 1,500, uh, searches are free. If you use the native tool, then it costs money. And all of those tools, in a way, enrich your context.

So if you use the URL context tool, which goes to a blog post, and the blog post has 10,000 tokens, you pay for those 10,000 tokens as it's included into your prompt and then tries to answer your, your question.

For, like, the function calling we had, it's a bit different. Of course, we generate tokens, but, like, the structured output tokens are less, uh, and you do the function calling or, like, the, the Python call on your side.

One, one if there are two tools which have overlapping functions, like, uh, in the same weather, you use one tool which is giving and the temperature one tool is which is giving temperature and humidity, how does the LLM figure out which one to call or, uh, call both of them or call one from them and whatever?

Yeah. So that's definitely a good question. So function calling, of course, is much more than one tool and never going to be calling the weather API because, I mean, we have weather apps,right? Um, but it's a good example to show.

But what's very cool is so there's parallel function calling. So, uh, we are under Gemini documentation again. And let's assume you want to have a party and you have function tool, like, power your disco ball, uh, ball, start the music, and dim the lights,right?

If you, like, want to say, okay, set us into a party mode, you would expect all of them to start at the same time. And, uh, with Gemini, we have parallel tool calling. So basically, what Gemini would do instead of generating one, uh, object as an output, it generates a list with basically free objects for all of the functions you need to call, and then you iterate, uh, over those and call them.

Parallel tool calling works if the inputs and outputs of the tools are not dependent of each other,right? I can start my disco ball and the music at the same time and not, okay, I first need to start my disco ball, and if it runs, I can, like, start the music.

Um, if you have more of those sequential tool calling, um, basically you need the input, uh, sorry, you need the output of the first function for the input of the second function. Basically, if you, I don't know, you have, like, some kind of, uh, smart home system, um, and you want to set the temperature based on, like, the outside weather, you would first need to check what's the weather and then, like, set your temperature inside of your house.

Um, here would you basically provide instructions, uh, using the system prompt to your model to say, okay, um, to change the weather, you first need to look up the weather and then set the temperature. And what Gemini would do, basically, you have your user prompt, it generates the function, uh, the structured function, you call the function, and instead of generating a user-friendly response, it generates another function call, and then you can call it.

So basically, it continues this function calling loop before generating a very nice user-friendly output.

Can you say that again? So it generates function calling in a loop?

Yes, kind of. I mean, I can, like, so there's I put up an example, uh, for, like, the sequential, um, function calling. Uh.

Is there a way to manage multiple function calling timeouts and user responses? Uh, is there a library that helps us manage that? Do we do, or should we have to write if there are multiple parallel function calls and some sequential and some may take longer?

Uh, and is that workflow something we have to code for?

Yeah. So currently, there's not a good way to handle, like, functions who could time out or, like, the best way. So it's a bit like, I would say, parallel calling is, like, you need to wait for all of the results to put in into the next one.

If one takes longer, you can basically provide an error message or not ready yet or something. But you would need to, to explore how it works on the live API, which is basically our way to create real-time agents.

They are working something called asynchronous function calling where, um, the conversation continues. So when you think about a customer support agent which you can which you talk to,right, it would be very weird if the agent stops for, like, three minutes and doesn't say something because it needs to look up your information.

So that, that's what they call asynchronous function calling. So you can continue the conversation with the agent, but the agent runs a tool call and then, like, injects the response, uh, later. But that's only for the live API.

And is that available as a Python library or.

Yeah.

How do we?

So, um.

The live API?

Documentation, live API.

Um, it was launched on, I think, in I/O or on Cloud Next. Um, so yeah, there, there you have, like, uh, tool use with live API, and there's asynchronous function calling, um, with code snippets for, for Python and JavaScript.

Cool.

Yeah.

Is asynchronous function calling a synonym for, uh, parallel?

No, it's, it's more like, I start a function call now. I need to continue my conversation. I get back the response later. So instead of, like, getting the response, what you get back.

You see the back call?

Yes. What you get back from the model could be, for example, interrupted or scheduled so that you, as a developer, know, okay, I started something. The model knows it started something, and then you can inject once the your function is the output is ready, put it back into the conversation, and then the model uses that information to continue.

And what about, uh, logs of, uh, all the API calls?

Yeah.

Like in OpenAI, uh, playground, we can see.

Yeah.

What, what's the equivalent?

Um, so in AI Studio, we don't have a feature like this. Um, but you can, like, use third-party tools like Langsmith or Arise AI Phoenix. Uh, but Vertex AI supports those features.

Okay. So if we have a.

Yeah. That was the question. Yeah.

Um, in, in the live API.

Yeah.

Previously, um, it's based on understanding of bounding boxes and corners was bad.

Yeah.

In the live API, you had to, like, ask for. Does it has it got better?

I don't know. The only thing I know is that so previously, there was only Gemini 2.0 Flash available in the live API, but now we have Gemini 2.5 Flash. So basically, the model we used in, like, the Jupyter notebooks, um, you I guess you have to try or maybe, like, reach out to the people who are working more with it.

Okay. More questions? Yeah.

URL context is different from something? How is URL context tool different from grounding?

So URL context tool basically works based on the link you provide. So if you provide your own personal blog, it goes to that link and tries to extract the information and uses it in the context. And search basically uses Google Search, creates one or multiple search queries based on the prompt, searches, and then provides the outputs into your prompt.

So if you already know where the source is for your prompt to create the best answer, URL context could be a good use case for it.

Does agent tools as a real-time tools, I mean, uh, functionality, does it also access, uh, premium paywall content? Does it have access to paywall content?

I don't know. You paywall content, you mean, like, pages?

Websites?

I don't know. I would not expect to have it ex like, but the live API supports function calling. So if you have a subscription or, like, a way to access this paid content, you can create your own function, which the model can then invoke and use as a context for, for its, uh, information.

Okay. Cool. Then last section. I guess every one of you might have heard about model context protocol by now, and that's, like, either our solution to how we all align on how to call agents or, like, what's theright thing to do.

I'm a big fan of it. Um, I think it makes it way more accessible for people to build agents, especially if we get more first-party remote MCP servers, um, where you can basically focus on building your agent instead of creating all of those functions.

I mean, I'm pretty sure there are, like, now a million get weather functions which can be used. And I hope with, um, MCP, we can, like, fix this. Uh, what we, um, shipped and announced at Google I/O is, um, native integration of MCP servers inside the Google Gemini SDK.

So the SDK, which we have used during the whole workshop, um, allows us to directly use MCP servers and sessions, which makes it even easier for all of us to integrate it. So when we, um, look back at our function calling example, we needed to call a function, create a declaration, and all of those different things.

And now with the MCP, uh, integration, we can basically, uh, only start or initialize our client. Here, I created an MCP weather service, which kind of has the same functionality. And, um, all of the things we now need to do is, like, we use our generate content method, and in our tools argument, we provide our session.

So we start our server, we create our session, and we provide the session to, um, the Gemini SDK. And then what happens behind the scenes is basically the same function call loop we did manually. It's like, okay, what are the tools available?

Get all of the tools from the MCP servers, put them into the LLM call. If the LLM call makes a function call, extract the function call, call the MCP server, get the response from the MCP MCP server, put it back into our conversation, have the model generate final response.

Um, I guess it's, um, probably time to find out if Colab has Node installed and to see if it works.

Okay.

Okay. It doesn't.

Cool.

What I can do is I can quickly set it up locally and show you in one second on how it works.

Okay.

What's the question?

Okay. So we are back. I'm in Cursor, same notebook, same setup. And now I can, like, use it. And now basically, what happens if I let's not ask about London,right? We are in San Francisco. Um, let's ask it about the weather in San Francisco.

It now does all of, like, the different function calls and loops behind the scenes. The MCP server is very simple. It uses the OpenMetal API, which is kind of free to use on a small scale for testing. Um, so we generate the output, run all of the MCP calls.

Might take a bit longer for the API call. And then we get back, okay, the weather in San Francisco today will be 70 degrees Celsius at zero, and then all of the other numbers. And that's basically now all you need to, to combine or connect an MCP server with the, the Gemini SDK, which is, like I mean, it fits on a single screen, like, which is easy enough for you to get started.

Um, we use here, like, a, a local running MCP server, but it works the same way with a remote MCP server. That's, um, also the exercise, uh, for, for this, um, part of the workshop. So basically, uh, DeepViki from, uh, the Cognition AI guys, Hinterdeven, they had a have a very nice remote MCP server, which, um, can talk to GitHub repositories.

So instead of, like, creating a SIDIO client, you can use the streamable HTTP client and connect a remote MCP server to Gemini to talk to it and, um, yeah, basically benefit from one of the advancements currently happening in AI.

Do we have more questions? Yeah.

Uh, do we also have the GPT-115 basic?

Yeah.

And does the Gemini stack out? What's the tool things, if you're playing, uh, and what is the versioning of this tool SDK?

So ADK is, um, for those of you who don't know, is like Agent Development Kit. It's an, uh, agent library which adds a lot of more abstraction on top of, like, the, the client, uh, SDK we used. It makes it a lot easier to do all of the tool calling, um, integrates MCP server as well, makes it much more easier to manage, like, multiple agents or deploy it to cloud.

So it has a lot of, like, I would say, batteries included. Uh, question again, depends on you if you would like to use, uh, frameworks or rather prefer building it yourself. I see a bit benefit of, like, getting started very quickly using agentic frameworks, but the more abstraction you add,right, the more the less you know on the first time.

And then maybe you need you need to dig a bit deeper later, and all of the, the advancements or speed you get in the beginning will be your slowdowns in the future. But ADK is definitely a great way to start.

They also have, like, tons of examples. It has support for Gemini in the Gemini API, so definitely a good way to, to take a look. Yeah.

Do you use any URLs for your tools? Does it turn back on a crowded location and then go to Google Search rather than this?

Um, no. I don't think so. Uh, but what you can do with, uh, YouTube links or YouTube videos is you can ask it to return timestamps for you as the response. And it works very nicely. Also, it is very accurately because so basically, a video is more or less just many images behind each other,right?

And currently, uh, when we process videos, it will be done at one frames per second. And you will always have, like, the timestamp, the image, the timestamp, the image. That's how Gemini exactly knows at what place in the video is, I don't know, a man jumping or dancing or something.

So it's analyzing one frame per second?

Yes.

And then describing it.

It depends on your pro I mean, we can I mean, it's part of the section. Um.

Yeah. I use that all the time.

Okay. Cool.

Summarize videos. Yeah.

Yeah. But, yeah, basically, what happens with videos is I mean, videos are 24 frames per second or even more. That, like, easily lets the, the context explode. So currently, what is done is that we have one frame per second.

So if you have 60 seconds, it would be 60 images, uh, which makes it also easy to count how many tokens it would be. And that's also how we can fit in, um, a one-hour-long video into 1 million tokens, basically.

Is it using the transcript from YouTube, or is it doing, uh.

No, it's.

Speech to text?

It's all multimodal natively.

Ah, okay. It's not using.

No. I mean, you can, like, easily upload normal.mp4 files, and you can ask it, like, what was say or said in, in the video.

Yeah. I tried that for Dutch-speaking stuff, and it was quite good.

Okay. Cool.

Yeah.

It's very interesting. Uh, do you play around with any of the memory MCP servers? I saw you had the mem0.

Yeah.

And then just Gemini seems to struggle a bit with the, the memory stuff, I think.

So I'm also, like maybe for you, if you don't know, it's like memory is basically next to tools, what makes an agent an agent. And for memory, we have short-term memory and long-term memory. Short-term memory is basically your conversation, which is part of your current, like, state when you talk to an agent.

And long-term memory is basically memory or, like, information about a previous interaction or something about the user. And long-term memory is not, like, part of the LLM, which we need to provide externally. And mem0 is, like, something I looked at it because it's very nice.

It does it implicitly. So basically, it takes the conversation and tries to create a nice abstraction for you, we which you can include. Um, I'm not sure, like, what the current state is, like, how or what works well with Gemini.

I'm, like what I always see or where Gemini really shines is the long context. So the better your information extraction is and, like, what you provide to the model based on the memory, I think it works very well.

I'm not sure about those tool-calling memory kind of systems. Yeah.

Internal context window is the harder plan?

Yeah.

Or using it?

Yeah.

Down the road?

So we are working on 2 million. Um, not sure when it is, like, generally available, but.

I heard internally you guys can go to 10 million with that.

Yeah. Like, there, there was research, but, you know, like, the bigger you go, the more expensive you get. And do you want to pay, like, $50 per 1 million tokens?

You're not paying $50 to an agent.

Yeah.

More questions or everyone working?

I have three questions.

Yeah. Then.

Yeah. So before MCP, we need to define the functions.

Yeah.

And to call it ourselves.

Yeah.

And we need to decide the logic, how do you call the tool? When do you call the tool ourselves as well?

No. So the, the difference with MCPs now and, uh, what we had previously is that we don't need to write and define the functions and the function declarations. The LLM sees at the end the same exact exact same thing.

So if I create my function declarations manually with a JSON schema, or if I retrieve them from an MCP server, those are the same things for the LLM. It just makes it much easier to not need to rewrite the same functionalities over and over again,right?

If you work at a company and you want to integrate APIs, which you have internally, there's a pretty big chance that two teams write the same function declaration and the same wrapper to call it. And the idea here is to only have one team needing to write it and everyone to benefit from it.

The same is with, like, public APIs like Google Maps, GDrive. So MCPs gives us a way to collaboratively create, like, the best or, like, the standard as a way to how to call it. And then everyone kind of can in can implement the MCP tools, uh, which you need for your use case.

Yes. Like, use the MCP even the prompt. How does it decide which tool to use under that MCP?

It's so an MCP server, for example, exposes four tools. Um, you provide all of those four input schemas, which are, like, function declarations with a name, a description, and the parameters to your LLM call. And then based on the prompt and those declarations, the LLM, in this case, Gemini, decides if it should call a tool, which tool could to call, or should it call multiple tools.

And that's the same logic we had for normal or have for normal function calling. So there's no, no difference in that way. The only difference we see is that MCP servers being easily available or implementing many tools, people start to be very easy on how many to add,right?

So we see people add, like, 50, 100 tools. So we need to improve our LLMs to be able to have them really know, okay, which is theright tool to use when you have, like, 50 different tools available at once.

Google introduced API, is that, uh, a part of what Gemini is, API or ADK, or is that maybe separate?

So there are examples for ADK to work with A2A. So A2A stands for Agent-to-Agent Protocol, which is done by the Google Cloud team. The idea here is, uh, to allow companies to build agents in different frameworks like LangChain, LangCraft, LlamaIndex, and then have a, uh, an easy way to create multi-agent systems that one agent can call the other agent without needing to implement comple complex logic.

Um, but the Gemini SDK is, like, not directly, but there are great examples for it. Cool. Yeah.

So when you refer to Gemini, do you use it, like, natively to use website search data servers, or do you use it in the cloud in the same way that you use Google Search for browsing?

So we are working on basically browser use or computer use use cases, which is available in preview, uh, which we are currently testing with a few companies. Um, that would allow Gemini to control UI. So it could basically go to whatever website you want it to go to.

And then there's sorry, the URL context tool where you can provide a website, and we would programmatically try to extract information from it. Of course, it's like a if it's a super heavy JavaScript website, there's not much to extract.

That's where, like, the, the browser use agent would then, uh, be useful.

It's like an API for the Mariner project?

Yeah, kind of. Yeah. Hopefully, coming soon.

Just browser use, is there currently, uh, spin-up of, uh, of a cloud, uh, browser and work on that, or does it work more for, like, the internal?

So it will return structured outputs again based on what you provide. So you can control a local environment. But we are also working with the Cloud Run team to make it super easy to you for you to run so that you can, like, run a Chrome instance on Cloud Run, uh, which Gemini can talk to and control, or you can, like, control a local instance.

So both would be.

Yeah. Yeah. Yeah.

How do you manage the authentication for the MCP?

That's a good question. I think the whole industry is currently trying to answer, like, what's the best way to handle agent authentication. So the MCP, um, as a protocol itself supports OAuth. So basically, when you want to connect to an MCP server, which is protected, you would get back a 403 with, like, not authorized, which can then trigger an OAuth flow on your client.

Maybe you have seen it in, like, Claude Desktop, where you get a pop-up, like, to log in with, I don't know, like, your, like, uh, Atlassian account or something. So that's, like, one way of doing it. But it definitely needs more work,right?

Do you want your agent to access all of your emails or all of your GitHub repos repositories, or how can you, like, scope it down to, like, only the one specific repository? But they are currently, like, actively working on it.

I know that Auth0 will be here tomorrow as well. Those guys are very great to talk to. I think I know that they're doing a lot there. So.

Yeah.

I was wondering if you could say something about the citation citations and how that's different from, like, URL context. I think I was struggling on finding documentation on it, and also just sort of curious on how, like, how you trigger it as well.

It seems like I can mess around with the prompting and just sometimes the output sometimes it won't.

You mean, um, citation in when you just send a regular prompt or citation when you, like, use Google Search and that have citations?

I think very specifically citation metadata, like the output field.

Yeah. So that's available in, like, the Google Search tool. Um, and it has information about which websites was used to, to retrieve. You can, um, click on the link directly to see where you land. And then there's this, um, metadata or, like, chunking sup or, like, grounding support, which basically has a start index and an end index from the answer, and then also which sources were used.

So you technically can highlight or put numbers behind it to help users understand, okay, that part of the response is generated based on those two, um, links.

I think the, the part I'm confused about is the it seems like if you use the Google Search tool, that there's grounding to those citations that you get there. But then, like, the citation metadata, it seems like it's maybe, like, guessing from the training data, but.

You mean in, in general, there are sometimes citation metadata or?

Sorry, what?

Which citation data do you mean?

The one literally called citation metadata, not grounding or URL context.

I know or I only know that's part of, like, legal and, like, training that if there's something referred directly, you need to, like, provide it. That's why it is there. That's why it is not always there. Um, it can be there, but it doesn't must be.

And it's not, like, for a user to, hey, that's coming from there, but it's more like, um, compliance, basically, why we need to have it.

So allright. So follow up that, the thing I'm the thing I'm interested in is you're familiar with, like, Anthropic's citations?

I only know about, like, their web search feature.

They have a citation thing where it's just whatever you pass as input, it'll give you direct quotes from that. So it's kind of like in the response.

Ah, okay. So you provide as a context, like a document.

You pass a quote as input.

Okay.

And that's the agent's where it's getting this from. Um, I imagine you can kind of, like, coach that.

Yeah.

Through structured outputs. But I'm curious what, like, good best practices are with that. And if, if you can use, you know, structured outputs versus the, the URL context, seems like another avenue you could go with that.

No, that's, like, a very good question. Currently, we don't have, like, the same experience as Anthropic has. Um, I guess the base is, like, to really try and, like, different prompting strategies and, like, what do you want to achieve.

Um, yeah. No, but it's, like, good, good feedback.

Is that the approach? No LLMs can do that. Right. Yeah. Are there, uh, are there any, uh, that. That you can go with that or that will make a use?

Sadly, not yet. I hope hopefully one day that you can, like, just here's Gmail, here's Drive, and let's chat. Um, but currently, there's no public remote MCP server.

Is it because it's too new and not secure?

I don't know exactly what what are the reasons. I think the more people ask for it, the easier it will be that we get ones. So if you like, that's great feedback. More people, um, if the more people are going to use MCP servers, the higher the chances are going to be.

And I think in general, for MCP to succeed, we need more first-party remote servers,right? Because we cannot build a secure GitHub MCP server. That's something GitHub needs to do because they know how their OAuth system works, how you can scope it.

Um, so we really need those first-party MCP servers in the long time. Yeah.

How should websites prepare themselves for the, the AI mode? Is there some changes required on the schema or anything else?

For, for AI mode?

Yeah.

I have no idea. I think, like, so AI mode, you mean about, like, inside Google Search?

Yeah.

I think in general, overall, same goes for, like, normal Google Search. It's like, have high-quality content, um, try to stick to, like, um, web standards. Um, I guess same works here as well.

But I don't know. Sorry. Yeah.

When making, like, Google calls with the files API, should it trigger the prompt caching for Gemini 2.5, or does it work in the user interface?

It should. So if you uploaded a PDF, you need to make sure that you put a PDF at the first part of your prompt. So you like, the, the automatic, um, caching works from the beginning to the end,right?

If you change the beginning, you can never cache, like, the long document. But if you put the, the PDF in the beginning and change the prompt behind it, it should work. Yes.

Nice.

Yeah.

I had a question, um, Google Search running as well. So sometimes there's a case where, um, the index that it provides, either the start or end index.

Yeah.

Is actually out of range for kind of, like, the length of the.

Really?

Yeah, answers provided. Yeah. I was curious if you had any similar experiences.

No, but if you, like, have an example for us to reproduce and to share, that would be very helpful. The only thing I know is that so sometimes there can be a start index, which can be null, which basically means it's zero, like, it starts at the beginning.

Yeah. Sometimes I get for even start and by end actually kind of outside of the range. Because I assume it's basically just a character length,right?

Yeah.

Um, the outside of the range, the, the end.

No, like, if you have an example, please, like, send it to me on Twitter or somewhere, like.

I can, I can do that.

Okay. That, that would be very helpful because that should not be the case.

And then for the, um, excuse me, the grounding metadata as to where it's the grounding supports and the grounding chunks.

Yeah.

Is there any way to actually, um, get the, the citation? Or maybe that was the question that the gentleman asked, the citation from the website, basically the text that it refers to, or not?

So that.

Because you only get the website actually. So you just get the URL.

Yeah. No, like, so currently, you get, like, the, the start and end of the response, which refers to something, but not the web part. But it's like, like, please put it all together and, like, send it to us.

Like, very happy to, like, talk to the Google Search team who is, like, building the native tool. Um, it's super helpful with, like, to better understand, like, what you need.

Thank you.

Okay.

Cool. Then thanks all for coming. Um, please continue with the workshop. Try it out. If you have, like, any questions, we are very happy to receive any positive, negative feedback, any ideas, any pain points you have. Um, we are available on, like, social channels.

You can, like, find me, Philipp Schmid, basically everywhere. If not, we open a GitHub issue or be very noisy about when, when something doesn't work. Uh, we, we always try to make sure that we fix it. Cool. Thanks.