From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google

Yeah, so while we wait for it to come up, because I know we're short on time, I'm going to talk about

agents on-device. So I know whoever asked the question about skills and AI core, we have an answer to that. We've built a simple skill harness on top of AI core that you can build skills on. I'll be able to show that.

Also going to talk about tiny LLMs, which are what we would call LLMs that are like smaller than a billion parameters, that are small enough to build into your app if you want to have more customization or you want to do something that isn't already available for you in AI core.

So that's the gist. So, quick overview of AI Edge, well, how we think about small language models, tiny LLMs, and system GenAI. Then we're going to take a quick look at agent skills, which is something we can build on top of kind of system GenAI or the new models that are coming down the pipe.

And then we're going to take a quick look at tiny models.

So that's that. Okay, so, yeah. Cool. Yeah. Okay. Yeah, feel free.

Okay, so AI Edge, SLMs, and TLMs. Okay, so I think Ali already covered this. We know it's great to do things on-device: latency, privacy, offline use, reliability, or savings, depending on things. These are all motivations to do things locally.

Me, by way of intro, I didn't really do this. I kind of, I'm a software engineer and kind of tech lead working on the Google AI Edge stack. So that's, we have MediaPipe, which is an asset some people may be familiar with.

We have LiteRT-LM, which is an LLM harness that you can integrate with your app, where you download the model and ship the model with your app. And then we also have kind of LiteRT as a runtime that supports both LiteRT-LM and MediaPipe.

It's kind of formerly known as TensorFlow Lite, which is a kind of cross-framework runtime for running models. And all of that can run on CPU, GPU, or NPU, depending on the platform and depending on what's best. And you as a developer get to choose.

Yeah, it's already trusted at scale. Like the LiteRT runtime, there's a version of that built into Android OS. Lots of Android apps already use it. So it supports over 2.7 billion devices, like lots and lots of daily invocations, and lots and lots of Android apps leverage this.

But it also works far beyond Android as well. So we support all of these platforms. And for example, Gemma is available on many of these platforms. Our team, like, is giving another talk tomorrow. So you can hear more about Gemma performance on all of these types of platforms and how we're able to do really useful things with the latest Gemma 4 models.

But then building on Ali's and Florine's talk, this is kind of key idea. We have system-level GenAI, which is something that will be pre-installed into the system. So there's Gemini Nano via AI Core. This is an example of the summarization API.

Apple also has something going on with their intelligence on iOS that I probably know a lot less about. But as a concept,right, as an app developer, when you go to build a mobile app, this is kind of one choice is there will often be some form of intelligence built into the system that you can leverage, which is, you know, highly optimized, as kind of Ali and Florine covered, that's available for use with your app.

Then, so this is kind of typically like small language models. Like for Nano, it is the Gemma 4 E2B and E4B are the base models for what we ship there. That's really capable, highly optimized, preloaded with device. If you can use it, it's great.

Your app doesn't get any bigger. And if it meets your use case needs, it's a great place to start. If you want, like, more, if you have a more specific task that you want to do that's kind of highly customized or something really boutique, you can use an app GenAI.

So that's with the LiteRT-LM runtime. That can be loaded with your app or even your web page,right? And this offers kind of a higher degree of customization and reach, like definitely more work. But yeah, you kind of access to smaller models that can run on lots of devices and full customization.

So it's clearly a lot more work, but it's the other option that's available. Okay, so rest of the talk, 15 minutes, going to cover two key ideas. One is, hey, how do you do skills on-device? Because this is something new that we can do with Gemma 4 that came out last week.

We have a few examples of that. This is one key idea. The other idea I want to cover is, hey, for tiny models, what can you actually do with those types of models today? Because we've actually made a lot of progress on this in the last 6 to 12 months.

So I kind of just want to share what's the state of the art with tiny LLMs and if you want to use one in your app, how do you go about that? Okay, so this is, wow, there's a lot on this screen.

This is an app that our team have developed that works on both iOS and Android for running LLMs locally. And here we show both really tiny LLMs so you can see what they can do. But also, because Gemma 4 just came out, we're also using this to showcase how Gemma 4 can work on Android and iOS as well.

And this actually builds on AI Core. When AI Core is available on the device, it will use AI Core to kind of provide the Gemma model for the app. So skills is the thing I want to kind of go into deeply today.

But there's a bunch of other things in the app, like you can do AI chat, you can ask image, you can do audio scribe, and there's lots of example models. And the app also supports 3P models, like kind of Qwen or Phi or these types of models.

If you just want to load a model, get a feel for how it performs on-device. And this app is also open source in Android, and it's built using LiteRT-LM. So it's both a neat way for you to try things out, but also if you're keen, you can kind of dive into the code and see, hey, how does it all kind of hang together as an example for LiteRT-LM.

Okay, but we're going to dive into skills because this is kind of a topic du jour. Okay, I'm not going to play this video because I don't have enough time. But yeah, this app is available on Android, iOS, code available on GitHub as well.

There we go. Let's take a picture.

Okay. Okay, and the app is called Google AI Edge Gallery. So this is the video we will watch because it's shorter and meets my time budget. And we don't, could we get sound?

You want an agentic task?

Sorry, I'll go.

Hé Gemma, est-ce que tu peux trouver un restaurant français à San Francisco ? Please reply to me in English.

So this uses a restaurant roulette skill, and we'll see how that's built in a moment. Let's puzzle the restaurant. Select one. Winner,right? So that's an example of something neat that you can build, like, with a simple agent harness on top of Gemma 4 that's like really just a few lines, like pretty easy to do with a few lines of code or a few lines of theright vibe coding prompt, as we'll see in a minute.

Okay.

Okay, I don't know how that, ah, yeah. Okay.

Okay, I kind of got lost a little bit. Okay. Right, sorry. Back, back, back to where we're supposed to be. So what's actually happening under the hood? So like I was saying, this is built on, like, this is built just using a prompt,right?

And here you can provide, we have our own system prompt in our app. Then we also put the skill descriptions into the prompt.

So the model is aware of the types of skills it can use, but it doesn't have to see all of the functions and details of the skill. That's only kind of loaded on demand. So we actually have

a load skill tool call built into the model that then, like, selectively, so if we say, hey, can you show us the location of the Google office, it will then know, wow, I should use the map skill. It then loads the skill for map navigation.

The tool responds, and then it uses the show JS tool to show you the location in the app as well. So one of the things that's neat about being in an app is you can put simple JavaScript into the skill that we then call as part of the skill.

So this is how, like, I don't have the corresponding demo for this, but this would kind of pop up a nice kind of like JavaScript UI of kind of Google Maps to kind of just show you in the appright there.

Similar to the restaurant roulette, that was a custom JavaScript to do the rendering, to do the roulette wheel piece. Okay, so you can create your own skill as well. The app supports this. Sorry, I'll, yeah, instructions on GitHub.

I don't know if I can get past this page too fast. But also, and to create your own skill, there's full instructions there if you want to kind of handwrite it out. This works really well, though. So we can use skills to write skills.

So we have Gemini CLI or Claude Code. Like our team have done, like, maybe 80 skills and had a lot of fun with it. So this is an example of a prompt which works really reliably. In Gemini CLI, we actually have an ADB skill as well that we, our team uses a lot.

So you can even debug and test by saying, hey, you have access to a device via ADB, and you can also ask to test that. So this type of thing actually works really, really well, and it's fun. And you can then create a skill.

And then in the app, there's a dot dot dot button, and you can go to load your own skill from a URL if you kind of publish it to your custom, to your own GitHub. And it's kind of really easy to do from within the app.

You can then also let us know in our discussion on GitHub that you've created a skill, and then other people can check out your skill and kind of use that as well. So these are some things. This has only been out, like, since last Thursday, but these are some example skills that the community have built.

So feel free to do it and tag it up here. Okay, so that's skills. So the last 10 minutes we are going to spend on TLMs, or probably more ideally maybe five or six of the minutes, so there's time for questions.

Okay, so LiteRT-LM, this is the runtime that we have that we use for running models. It runs models in LiteRT-LM format, which is a single file that packages everything we need to know about the model in order to be able to run it.

It's open source, it's fast, and it works on multiple platforms. And there is a Swift API and a JavaScript API coming soon. At the moment, if you go to the GitHub, you can see the C++ and Java version.

And when we publish the Swift version, we will also publish, we will also open source the iOS app at that point in time. So if you go to Gallery for, at the moment, you can only see the code for Android, but hopefully in the next few weeks we can get the Swift work finished, have a really good API, and then we'll be able to open source that as well.

So, yeah, and it supports Gemma 4 as well on all of these devices. Also supports loads of other models, but understandably, Gemma 4 is our favorite.

So then to deploy a tiny model, what do you do? So typically, starting transformers, we then have a package called LiteRT-Torch that can help you export the model. And then LiteRT-LM, there's actually a reference version of that that you can use on your desktop as well if you want to try out a model.

You can either try it out for desktop, or you can load it into the gallery and see it perform there. And then you can deploy with LiteRT-LM. It's worth noting for smaller models, you will either pick a fixed function model, like a visual language model, or a transcription model, or something like this.

So there are some pre-built models available on our transformers page that you can use. But something we also see that's really common is people fine-tuning models. Because certainly once you get down to like 200 or 100 million parameters, for that model to work, it needs to have a very narrow and focused task.

And we've had a lot of success deploying those models internally and in an app, a different app that you're going to see in a minute, by doing kind of fine-tuning using synthetic data. So this is what the export and inference flow looks like.

So

this is showing, okay, on the left-hand side, it's showing us exporting a Qwen 0.6 model and then running that on desktop with LiteRT-LM run so you can just see how that behaves using a GPU, for example. Theright-hand side is showing a different example, which is Apple's FastVLM.

This is a visual language model. It's only 500 million parameters. And this is optimized and running on, like, a, this is running on the Qualcomm NPU that's also available through our stack NPU optimization. So this is an example of that happening end to end.

And this is running really quick because it's using hardware acceleration. And this model is just, that particular model is 500 million parameters, by way of example. Another example that we've spent a bunch of time with the DeepMind team on was publishing Function Gemma, something we published last December.

This was based on Gemma 3 technology. This is only 270 million parameters, but it's robust function calling when fine-tuned, type of there. And this is then, it's small and it's really fast, even on legacy devices. So if you go all the way back to a Pixel 7, this still can process almost 2,000 tokens per second prefill and 140 decode.

So it's really useful for lots of, it's really useful for lots of simple use cases. Like you can do text-to-function calling or voice-to-function calling using this size model. And there is a whole YouTube video on this called Function Gemma if you want to find out lots more details about how to do it.

We also

have a Function Gemma Fine-Tuning Lab. So if you search Function Gemma Fine-Tuning Lab, I don't have it here. This is available as a Hugging Face space. So you can kind of import, you can define functions, upload your own data, and see how to kind of fine-tune Function Gemma.

And this is kind of recommended for really high, for really robust function calling. So we have an example in the app called, like, App Intents where it'll do, like, the thing you saw previously of, like, add calendar or add email.

So when we took Function Gemma out of the box, our success rate in that was, I think, 46% or something like that. Then we put it through this fine-tuning flow where we're like, hey, we have these seven functions.

And instead of providing that via a system prompt, which is what you would do if you're using a larger model or if you're on a device with AI Core, for example, but you instead need to kind of synthetically create a dataset,right?

It's typically the workflow. We use Flash to synthetically create a dataset. Upload it to this type of tool, or we obviously have our own internal tools. But that then got that 46% to over, like, it was over 90% for eight of the 10 functions we were trying, and two of the functions were a bit lower in the kind of 80s.

So you can get really robust and reliable function calling using this fine-tuning workflow. Yes, it's a bit more work than just prompting a larger model, but it does allow you to kind of ship something robust in your app at scale.

Oh, sorry, going the wrong direction. Oh, yeah. So then pre-built tiny models are here.

Yeah, okay. I'm going to pull, stop for questions. Okay, so we have, and I don't want to go into this in too much detail. We have another app, I'll just speed run this for one minute. We also have another app called Eloquent, which is a transcription service.

But what's more interesting than the app is just an example of, like, how we built that. So it also supports things like personalization. So it does transcription with your own favorite keywords. So if you use a lot of, like, tech jargon or a lot of people's names, transcription service don't always get that correct.

Sadly, this is only available on iOS and not available in Europe yet,right? So this will be increasingly available soon. But the more interesting thing for the purpose of this conversation is under the hood, this is something we've built using tiny LLMs ourselves.

So this uses an ASR engine that we have built based on Gemma 3 technology. And then it also has, like, something we call, like, a text polishing engine that we've also built with Gemma 3 technology. And both, each of these models are only a few hundred million parameters, but chained together, they can create a really compelling offline, like, offline transcription service that is able to leverage your personal dictionary,right?

And also, like, the polishing also removes ums and ahas and that sort of stuff,right? Which is also a common gripe with kind of, kind of offline transcription apps. But yeah, so not really available widely yet. It'll be available, but for the purpose of this conversation, it's more just a, like, a proof of life example.

So, like, this does work in production. Once you put in the effort to kind of fine-tune a model, and you can create pretty compelling things.

Okay, so it's not available on iOS in Europe, so that's not true. So yeah, takeaways, system GenAI, in-app GenAI, that's the kind of the overall, yeah, overall kind of message and wrap-up. Happy to take questions. I have three whole minutes.

I think the person there was first.

Yeah, so talking about skills, so do you personally, maybe your team, how many skills can you serve with these tiny models before performance starts to deteriorate?

Yeah, we are still putting models on the clock there. So we've literally been playing with a model for about two to three weeks now. It's been in public for about one week. We see, like, within a single conversation, so we can provide, like, certainly for the 4 billion parameter model, like, if you, by default, we enable about eight skills, and it's able to choose between the eight skills reasonably well,right?

Within a conversation, you're able to say, hey, like, you know, like, find me out a fact on using a Wikipedia skill. Then, oh, wow, show me where that is on Google Maps. So if you have a conversation that uses skill, skill, skill, that works really robustly.

The thing we're still working on that's harder is through a single, like, interaction with the app for the app to know to call multiple skills as part of a single answer. That works sometimes,right? And we're still, like, that's something we're still figuring out how to make that more robust,right?

But yeah, like, it's all in a, just our, like, our agent harness thing is really simple. So I'm sure we'll figure that out, but we're still kind of discovering the limits of how far we can push the model.

Yeah.

Is there a particular migration format of documentation available when moving from Media by Inference API to LiteRT-LM? The reason I'm asking is currently the process which I used to follow.

Yeah.

Is if I have a model that's in Python and I can do a typical CSI, quantize it, bundle it, but the bundling happens in Media 5, Media 5 task file format, I think, or task file. And that can be imported to the AI scaling app.

So LiteRT-LM, where does it fit in these?

Yeah, LiteRT-LM file format for, just for stock LLMs is effectively a replacement for .task file. That's a transition we made last year. .task files are still useful for things like, like, a task file creates more things than just an LLM model,right?

So there is, like, a face mesh task. And obviously that has a lot of other code as well. But for LLMs, we want something dedicated, simpler, that people can use with open developer tooling.

So it does bundle the tokenizer as well?

Yeah. So, yeah, so it bundles things like the tokenizer, but it's just the LLM model. Okay.

What about CPU? How is the performance on TPU or?

On TPU?

CPU.

CPU.

I saw it.

Yeah, so there is another talk tomorrow from some of my colleagues, including Wei who's here in second row. And that has lots of performance data on Gemma and various models on, yeah, you can also check out our model card in the meantime.

If you search Gemma LiteRT-LM model card, we keep it kind of running, like, we update that whenever we have new performance numbers on new platforms. So there's a lot of comprehensive data there as well.

Awesome.

Cool. Allright, I'm at zero seconds and it's flashing at me.

2:30 tomorrow.

Yeah. Yeah, 2:30. Sorry, Shinton's here. Sorry, Shinton didn't see it. Hi, sorry.