TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google

Hey, my name is Cormac Brick. I work

on Google AI Edge, which is a way of kind of bringing models to the edge. This is something we use internally for our own products; it's also something we make available as open-source products. As part of this, we also work really, really close with the Gemma team, because they publish a lot of models that are targeted to edge devices as well.

Yeah, quick bit of background on me. Yeah, I've been focusing on edge AI for the last, like, wow, like maybe 10 years at this point. So I started off in 2016 with, like, running a Google Node on a USB key hardware accelerator plugged into a Raspberry Pi at, like, New Rips 2016.

So yeah, it's been a lot of fun over the years. I then joined Intel, worked on, led the architecture for the NPU that goes into all of their laptops these days. Three years ago, I moved to Google to work as tech lead on edge AI here.

Yeah, and it's been great fun, like, been a crazy few years, as you can imagine. But yeah, happy to share, like, both where we're up to and also give a kind of overview of, like, as part of that, we can also kind of cover the story of, like, where is, where is mobile AI up to today,right?

Like, what's the state of the art? What are the different patterns we see for model deployment? So the two things that I want to focus on in this talk are Tiny LLMs and agent skills. So Tiny LLMs are very, very small models.

Agent skills are now possible on-device, with larger models are needed to make those kind of work fluently on-device. But both are kind of exciting new directions that have only recently been made possible. So as recently as last week when we launched Gemma 4, with the DeepMind team launched that.

We supported them with a kind of Android and iOS app. That's kind of opened new possibilities for things we can do on mobile. So we'll also kind of deep dive into that, as well as then looking at what are the types of things we can do with Tiny models.

So kind of two halves of the presentation. OK, so that's the intro. So yeah, this is kind of what I was saying. So first, give an overview of, like, hey, what's like AI on the edge, like looking at what we look at small language models and Tiny language models.

Then taking a look at Gemma models, just because that's pretty topical, and the types of performance we see for Gemma models on various types of edge devices, because that's kind of what our team does. Then looking at agent skills, which we built on top of the latest generation of Gemma models that can run on both Android and iOS, as well as many other platforms.

And then second half, we're going to look at Tiny models, and how, like, how you can fine-tune and deploy a Tiny model to edge devices. And then finally, I have an example of an app that our team built, like, a real app, using not just an example app, using that's built using Tiny LLMs based on Gemma technology as well.

OK.

OK, so firstly, yeah, I guess folks who are you guys are already in this room, so you're probably interested in edge AI. This probably goes without saying, but yeah, like, there's a lot of benefits to running on the edge.

There's, like, latency or UX improvements for some really sensitive-in-the-loop things, like live voice translation, for example. That's something our team shipped on Pixel last year, where you can do a live voice translation. Very challenging to do that with the required latency using a cloud service.

So being able to do that offline and on the edge is key to be able to meet latency or user experience requirements. Privacy,

that's definitely a thing as well. Like, we do some yeah, there's people use AI within, like, messaging apps, for example. And people like their messages just to stay on their phone, encrypted, fully private. So that's a good kind of example of where we're seeing LLMs getting deployed is assisting in some of those types of use cases.

Whoo. OK, where privacy is really important. Offline use, yeah, that's kind of obvious. And then savings. Yeah, like, we see this being increasingly relevant for laptop users, where you see a trend towards folks at least experimenting with experimenting with small language models to do some types of tokens they may do with a desktop agentic workflow they have going on.

So some folks are interested in exploring savings as well. Yep.

OK, so this is, like, what we do as, like, our team. We do enable people to build apps. We do MediaPipe, LiteRT-LM, which is an LLM runtime that works on mobile and edge. And then LiteRT, which is just a standard inference framework.

This was a long time ago, known as TensorFlow Lite. That is the types of things we've used that over the years in, as, like, it ships in photos in unblur, uses MediaPipe in photos, YouTube Shorts. If you've used any of the funny effects in YouTube Shorts, they're also things that our team have built for these, along with these, with the YouTube team.

Using MediaPipe and various deep learning tracking models under the hood. And that's also built on the same MediaPipe and LiteRT technology. And then, yeah, we run this on, like, all Android phones today. We'll use this as part of system services or, you know, third-party apps building on a version of the stack that ships with Android.

But our stack also, like, runs far beyond Android. So we also run even it being available as a system service in Android, it's also you can take the same TFLite file and ship that to iOS, macOS, Linux, Windows, web, or IoT devices, just from that same file.

So yeah, that's kind of why deployment is helpful. One caveat there: that same file deploys on CPU and GPU. For NPU, we need to do some special compilation, and you end up with a special NPU file. But certainly, for even for, like, the Gemma models we published last week, this is true that one file can work in CPU and GPU across a wide set of devices.

So then, yeah, some of the things we're seeing for LLMs on-device is kind of privacy-centric stuff, voice agents and local agents, with tool calling as a very popular workflow.

And then, yeah, within our stack, we LiteRT-LM is the thing that we'll look at a bit later that helps us run Tiny LLMs on-device, cross-platform, and it's fast because we support all of the different hardware accelerators. Oh, yeah, so this is kind of an important concept is we see two trends happening today.

One is we see kind of system-level Gen AI. So for very, very large models, the way these tend to turn up on mobile phones isn't that, you know, when you launch a single app, it will download a 4 billion-parameter model just to, you know, just to help you find a good restaurant in, you know, in whatever app you're looking in,right?

Instead, the trend is to build larger models into the OS. So we call this, like, system-level Gen AI. These models tend to be in the kind of 2 to 5 billion-parameter range,right? Like, it depends on the OS and whatever.

But, like, this is one choice that we see both the Android team, who we work closely with, and the Apple Intelligence team. We see kind of similar choices being made by both mobile iOS vendors, where there's a central model built in.

And that is, like, an Android that's called AI Core. And there's things like summarization APIs and an increasing set of APIs available for developers, including a prompt API, where you can use that API. And that's available on kind of premium Android devices and premium Apple devices as well.

Or, like, obviously, Apple has their own Apple Intelligence thing. But as a trend, this is kind of worth this is really relevant,right? So if you want a if you want to leverage a built-in model, this is a great way to go.

So then the other trend we see is, like, in-app Gen AI, which is where the kind of Tiny LLMs, or TLMs, as we're calling them in this presentation, are more are more relevant,right? So system Gen AI, generally, you customize that via prompting or via skills, as we're going to see later.

And it's a foundation model preloaded on-device. In-app Gen AI, generally, these are custom to tasks. They're loaded with the app or with the web page. We also deploy some of these on the web. And generally, they're targeted at kind of, like, wider reach.

So this may work not just on premium devices, but all devices, because app's reach is really important for a lot of the app application developer teams that we work with. And these, yeah, surprisingly, you can get LLMs to if you fine-tune for a single task.

We've seen really, really strong performance on, you know, simple tasks like, you know, summarization, transcription, or kind of voice-to-action type things. We can get really reliable performance from models in the

really reliable performance for models in the, like, 100 to 500 million-parameter range, depending on the complexity of the task,right? So a good example here is we launched Function Gemma in December. And that's was a 270 million-parameter model that was dedicated for function calling.

And we're able to show that doing, like, 10 doing voice-to-function calling for 10 different functions that were relevant to kind of Android developers. And our internal evals we did an internal eval set. That kind of reached over, it's like, 85% to 90% kind of reliability, just using that very small model, which was widely deployable to iOS and Android devices.

And that's actually something you can play with in a sample app that we have that we'll look more at a bit later. So that's an example of, like, a tiny of one use of a kind of a Tiny LLM model.

But we're seeing more interest from application developer teams now into kind of fine-tuning models to deploy as in-app Gen AI. So the difference here is, on the left, yeah, you kind of the customization on the left is via prompting or skills, customization on theright.

Like, we would encourage people to do some degree of kind of fine-tuning to make a Tiny LLM work. Like, in practice, like, yeah, certainly below 500 million parameters, that's true. Maybe for 500 million parameters and above, we can do more general-purpose tasks with a model.

But for the really, really tiny models, certainly less than 500, in our experience, you need to fine-tune to get kind of production-level reliability.

OK, so now I'm going to talk about Gemma 4. And Gemma 4, the models that were launched last week kind of fall into that kind of system Gen AI candidate category,right? So we can do lots of powerful things with Gemma 4, as you'll see.

OK, first, we're going to talk about sizes. So there's two small sizes, which are the first two, which are E2B and E4B. So these have E2B is called that because that has it only needs about 2 billion parameters to be present in RAM to run the model, because one of the limiting factors we see with these models is how much RAM you need to have in a device in order to run it reliably.

E4B has been optimized to run it with 4 billion parameters on-device. There are more parameters used by the model. But the other parameters that the model uses are you could, like, DeepMind have talks on this later in the week.

So you can go there for more detail. But the tale de ore is the other parameters in the model are used for kind of Perlier embeddings. So in our runtime, we don't actually need to load all of those parameters.

So we need to maintain them, like, the 2B and 4B, they need to stay resident in RAM. The other ones, we typically, like, memory map them. And we only need to load, like, one line of the embedding table of the Perlier embedding table and the embedding table once in the autoregressive loop.

And we actually only need to load

a few hundred bytes or thousands of bytes of that in order to do the next token inference. So as you go through inference, you don't ever end up requiring to load the whole PLE table into memory. And then, you know, depending on the OS, it'll do a reasonably good job of kind of, you know, evicting memory used by kind of older PLE tokens.

So that's why we have this idea of effective. So the smaller models run on, yeah, you can see on theright, it runs on a variety of platforms, the E2B and E4B models. These models are on the AI Core roadmap.

So these models that are available now for experimentation, at the appropriate point in time in the future, the Android team will integrate this into AI Core, and they'll be available more broadly on a wider set of devices. And there's another talk next week, or there's a talk here this week from

Oli from the AI Core team, who will probably share more details about exactly what that roadmap looks like. And also, there's, yeah, Omar is going to do it from the GDM team is going to do a deeper dive on all of the Gemma world.

The bottom two models, just for reference, these are also relevant for the edge. They're not really the focus of my talk today, because these are relevant for folks to run on laptops. The sizes have been optimized to run really well on consumer-grade laptops,

albeit ones that have maybe, you know, 32 gigs of RAM. But, yeah, that's what these do. So I'm going to focus less on these other two models, even though they're very powerful and useful.

Yeah, so we can see, like, certainly the E4B model, like, both the E2B and E4B model have excellent performance on a wide range of things.

Yeah, on knowledge and reasoning. But one of the big step-ups relative to the, from my perspective as a, like, user of the models, relative to the last generation is they've kind of built in function calling, which is excellent.

And they also have built in thinking. So that combination of kind of thinking plus function calling is what unlocks our ability to now do skills on-device. As to so you can just, as you'll see in a second, we can describe a skill, give it to the model, and the model can just pick it up and use it.

So that allows us to use that kind of pattern that's proven very popular in the last few months to bring that pattern to mobile, which allows for new types of kind of mobile experiences.

Also, for the E2B and E4B, these are multimodal. So they support audio, image, and text. The larger models support just image and text. Oh, also, the other change from a deployment point of view is this is the first time the Gemma models are released as in just a stock standard Apache 2.0 license, which means they're more usable by more people.

Again, you'll hear more about this in Omar's talk. Just that I'd mention it here. OK, so that's Gemma in general. Now, just to deep dive Gemma on our runtime and our platform. So, yeah, that same kind of picture earlier,right?

We have a single LiteRT-LM file, which is a LiteRT file with the things like the tokenizer and the other things we need in one package to run a model. That single model runs across all of these classes of devices,

across mobile, desktop, and embedded. And we're, yeah, we're pretty excited to do more in the embedded space. There's a lot particularly with image input, there's a lot of scope for new

for new IoT use cases. OK, bit of an eye chart now, but just to dig into performance. So I would say this is a snapshot as of today. This is something we are continuing to work on, both in both ourselves and our team, and also with various partners that we have,right, across, like, Intel

with the Raspberry Pi team and the Qualcomm team. We're continuing to optimize all of these numbers. But we can see we can do

certainly on, you know, for the 2 billion-parameter model, we can get really compelling performance on a, like, a high-end Android phone can do thousands of tokens per second on a GPU, also thousands of tokens per second on MacBook or but the bottom two rows then are, like, on a Raspberry Pi, we can get about 133 tokens per second, which is sufficient to do kind of simple image analysis use cases with reasonable latency.

And also, the bottom one is us running on a hardware accelerator. This is a Qualcomm kind of IoT/robotics development, like, development platform that they have available. And there, we can see pretty compelling performance as well, because we've gone and used the NPU, which gives a lot better performance.

And then, yeah, you can imagine then we have corresponding performance on the E4B model, which works on, yeah, a wide set of devices as well with, yeah, kind of proportionately less prefill and decode performance,right, given the size and the numbers of parameters we need to fetch.

But, yeah, broadly, they're available on lots of platforms. OK, so now, what can we do with those models? Lots of things,right? So one of the things I wanted to talk more about, because it's, like, net new, rather than just kind of show you, like, our image analysis or audio transcription or audio translation,right, there's a lot of things we've been able to do with models, from my perspective, for a while that these that the Gemma 4 models are, you know, much better than their predecessors at.

But we the thing we're the thing that's newest, from my perspective, is kind of agent skills on-device. So this is so we have an app. I don't know if you guys have seen it. It's available on both iOS and Android.

It's called Google AI Gallery. And this allows you to do lots of different things, like to just do basic AI chat stuff or ask questions of an image or

do kind of transcription or translation use cases starting from audio or, like, audio to function calling type use cases. But the one in, yeah, the second from the left is agent skills that I'm going to do a deeper dive on today.

OK, hopefully, this will play.

Oh, OK, that is very annoying. Sorry, I will fix this before we post slides. Oh, wait, let me see if this will

ah, OK, now does it sound going to work?

Morning, Gemma. Let's log a new Mood Journal entry with its coordinates.

OK, so this is claim that sound is not working.

I got eight hours of sleep.

Oh.

And I'm looking forward to hanging out with Amy today.

So this is a journal skill app we're looking at here, or a Mood Tracker, where you where it logs mood and sleep, and then.

Perfect. Analyze the trend in my mood over the last seven days.

OK, so what's happening here is, with the Mood Tracker app, it kind of logs it logs your moods or observations to a diary.

Awesome.

And then the LLM is able to go back.

And then you can run.

And summarize the contents of that diary.

Can you check my calendar for the day?

What's interesting here is this is just.

Show me the bullet list.

OK, this is just I won't kind of repeat this more. OK, so what's interesting here, though, is the way this is done, which we're going to look in more detail. This isn't a kind of custom fine-tune. It is just us giving a particular skill.

Any one tool?

With a little bit of JavaScript that the.

Show me where that.

That the model can call to the model. And then, through just a free-text interface, you're able to like, the model is able to use and pick up that skill. So it's also able to call there's two skills actually happening here.

I think.

One is the Mood Tracker skill, and the other one is the.

Can you check Wikipedia for the latest information?

The other one is the was the map skill. And now it's using a query Wikipedia skill here to look for latest information from the Oscars. So versus having a model that's just, like, pinned in time, it now becomes really easy to extend the model with, like, with skills such as kind of map lookup, something like interactive journal that you can both add subtract entries to from voice as well as query, and also kind of adding more modern knowledge or relevant knowledge.

And compare.

So we're showing.

Can you compare this vibe with something?

We're showing Wikipedia in this case. Oh, this is another fun skill that somebody in the team also developed, was the Mood Music skill, which calls a web service to compose music based on a single image and plays it.

So, yeah, it's able to kind of compose some lo-fi music to go with the person's breakfast sort of use case. But it's a new, like, I guess, kind of paradigm in how we're able to, like, really, really, really easily extend the models.

And to do that in a pretty kind of low-code way, as you'll also see in a bit when we see when we dig into how this works under the hood.

OK, so, yeah, so examples here, and I'm not going to play all of these videos, maybe, to save time, because I think you've seen many of them. So one is we can augment the knowledge base. There's one pattern that we find interesting.

We can produce kind of rich interactive content, like flashcards for visualizations. And I yeah,

where we can, instead of, like, if you ask to summarize something in three bullet points, you can have, like, a JavaScript skill to kind of show those bullet points as a card instead. So you can just say, like, summarize and show a card, like, summarize this.

And if it thinks that a card display will be more helpful for the user, that will get used.

Or music synthesis. OK, so I'm going to skip forward a little bit from the demos to

how we built them. OK, so what's actually happening is

the way we've built the skills is they're efficient. So the instructions can be kind of loaded on demand. So this kind of uses a principle that you may have seen

that you may have seen elsewhere of kind of, well, like, kind of progressive disclosure of conditional or conditional depth. So instead of, like, in an MCP workflow, where you need to describe everything about all of the functions that you need, the way we've structured the skills is there's a kind of one-line description.

The one-line descriptions of the skills is what the agent sees. And then, if it thinks that sounds interesting, then it asks for it asks to load the skill. It is a, like, we teach it a skill to load a skill.

And then it goes in and loads the skill, and it finds out all of the details about how to use it, what function calls it can use to do that skill. So this is, like, that pattern is particularly important for kind of token efficiency and, frankly, like, reliability on edge models.

Because if we had to load all of the details for all of the skills into the edge model, that would be a lot of context for the model to reason over. And in a lighter weight model, like, that will, you know, that will hurt kind of performance ultimately,right, because the lighter weight models are, you

know, you know, they're really great to be able to run on-device. But in terms of reasoning ability over very, very long context windows, if you can if you can have a more condensed context window, that will kind of up your batting average in terms of kind of quality metrics you're looking at to kind of ship a particular app.

The second part is tool access. It helps us integrate new tools dynamically. So things like there is a, like, we think of them in terms of there's a set of, like, input tools, like, how do you get more information, which could be, like, Wikipedia or looking up a weather service or something like this.

There's a set of

then there's a set of things that are helpful to kind of present new outputs to the user, like kind of showing something on maps or showing something via cards. So you can have skills both to extend the input and extend the output, you know, the output patterns that a model can do.

And it also kind of helps us to bring in domain-specific knowledge bases. So that skill that's kind of calling Wikipedia, you could easily imagine that calling some, you know, customer CRM internally or some asking data from a local RAG system as well.

OK, so our yeah, so our, like, within our structure of skills, we have skill.md. And then there's optionally kind of scripts or assets, where the skill.md is the metadata that we always process. So, like, yeah, the example here is showing extracting text and tables for PDF, and we trigger on PDF for extraction.

And then the instructions are only loaded when the model thinks that it requires that skill. And, yeah, this pattern is particularly important. Then in within scripts, we do also support

people kind of writing custom JavaScripts that can then be rendered within the app, because that's a really easy way to extend

within within both kind of iOS and Android systems.

OK, so dig a bit deeper then, is we have within our system prompt, we have our own system prompt that we ship with the app. Then there's a set of skill descriptions, which get added to the system prompt.

And then, when the user asks for something, the model decides to trigger the skill by reading its metadata. It then calls the load skill app that we have under the hood. It the tool response from that function call is then the contents of the skill.md file, which is then in the context window.

So it now knows about these functions. And then it calls the run oh, yeah, so then that also contains some JavaScript. And then we call the run JavaScript tool, which runs the JavaScript that we picked up from the from the skill file.

And then we call a response on that as well. One other thing to kind of mention as part of this workflow is the one of the things we did when we were porting function Gemma for reliability is, within the runtime, we have constrained decoding that applies, but it's kind of tuned to only apply to the output when we're generating a when we're calling a tool.

And we can also the constrained decoding can constrain it just to the particular tool that the

that you're supposed to be calling. So, like, in this system, instead of just kind of having generic JSON constraints, we know that there's a finite set of tools that the model is supposed to be able to use. So we can, therefore, have stronger constrained decoding.

This also kind of helps us have a more reliable system overall by using stronger constrained decoding. We find that's less, yeah, for we find this helps a lot for, like, the 2 billion parameter model. As models get more capable, we find the, like, the margin you get from this sort of strict constrained decoding, as we call it, is it's less essential when you're running a very large model, like if you were running a 10 billion or a very large model.

But for really small models on-device, this is a very helpful tool that helps us have stronger guardrails for the model so that we can opt the quality so we have something that is useful in production.

So then we support wonder, wait, I'll see if this will play for us. We then support

within the app, we support you can toggle the skills you want to use. So even the skill descriptions, you can decide how many skill descriptions you want to have live at any given point in time. So this is loading a piano playing scale, virtual piano, which

will do its things. You can actually tap the keys and play sounds. You can, yeah, this is just more JavaScript stuff. But then we can also you can also load custom skills. So you can write a skill yourself and load it from a URL, which is kind of fun if you want to experiment with kind of prototyping a skill with Gemma for an app idea you have.

This is something that you do. Oh, back there also, the skills can have, like, an API key as well, so like a secret key. So if you need to use a web service, that's something that you can prompt the user to put in.

This is showing then the Mood Music example again. But we and we have a on our GitHub page, we have a GitHub discussion where users are posting skills that they have written themselves. And the skills in the community that we like, we are then able to kind of pull up and have as, like, featured skills in the app.

So if something developed so that ones that people develop, we can have a way of kind of showcasing useful community skills to the wider community as well. So this is that third-party scale. I think this is going to show us.

OK, this is adding an animal intro skill. This is, like, a kid's thing.

So what is the to the app?

So, yeah. But, yeah, the key point here is there's a really, really low barrier to extend the model,right, and to extend the model in a way that is, you know, kind of relevant to downstream app users. So this is very, very easy.

And we're going to go a little deeper now and to see just how easy that is.

If the clicker is it's a good thing.

OK, so, yeah, so skill yeah, so skill architecture, when we go one step deeper, this is kind of relevant. So we have our own kind of orchestrator. It has a skill registry, and we call the load skill skill in order to load the skills.

And within skills, then there's, like, a JavaScript skill. There's native intents. So at least within for the Android system, we're able to call, like, kind of Android system intents or native intents. And you can you can just call those in your skill.

So things like if you wanted to turn off on and off the Wi-Fi or something like this. So intents that are exposed to all Android that are available in JavaScript, you can certainly use.

Then there's a kind of the roleplay, the skill.md, which is the persona and scenario data. And then there's the specific skill and resources. And like I was saying before, this can include

this can just be JavaScript that runs entirely locally, which is a fully offline experience. Or, like, in the music composer one, we actually kind of called a web API, and that needed an API key that then the user was prompted.

That also works within Gallery. Oh, so then, yeah, then the predefined tools we have under the hood, just to give a mental model of how it works, is we've got a load skill, which helps us load skills, run JavaScript, or run intent.

And just the orchestrator, just using these three skills, is able to make the overall system work.

So this is another video.

Now, let's test our effective tools.

This is from our Gemini product manager showing.

Hey, Gemma.

Olivier.

Est-ce que tu peux trouver un restaurant français à San Francisco? Please reply to me in English.

This is a restaurant roulette skill.

This, yeah, we actually have about, like, it was so easy to develop skills using

using both, like, anti-gravity,right, or Gemini CLI. We actually had the team do, like, 80 of these skills. So we had, like, lots and lots of things to do with lots of options in terms of kind of what to what to show.

Allright, it was a lot of fun for folks to develop these. So here we can go in and just look at the structure itself of restaurant roulette.

Yeah, this is the under-the-hood part. So this required requires secret true. OK, we wanted to get an API key, and it searches for 10 restaurants and then returns in, yeah, OK, returns locations cuisine.

Ah, and this is the index.js.js file then, which is where we can have a, yeah, simple kind of web view then kind of rendered to have the kind of roulette wheel.

And you can, like, obviously go and code all of this yourself,right? And there's full source code here for the examples. Cloud Code, we, yeah, I never got that actually published. The but it works really well in Cloud Code as well.

And we have both a source code for the example and a skill spec for getting started. But what and there's full instructions on GitHub if you want to try writing your own skill. But this was this was the pattern we actually use most.

It was using, like, skills to write skills. So using something like Gemini CLI or Cloud Code or anti-gravity, this was our kind of favorite pattern where we just say, hey, I want to write a skill for AI Edge Gallery.

This example works in Gemini CLI, where we just say, hey, this is the documentation. Here are some examples of some skills that you can go and read. Then this is the skill I want to build, which was offline archiver.

Oh, yeah, the idea is when you, yeah, you can

well, just based on things, pictures you took around London, for example, it would then be able to give you, like, there's that cool Churchill statue I kind of walked past. It could kind of then it would kind of go and research a bit about Churchill for you.

And then on your flight back, you could go back into this skill, and it would have a bunch of information about the stuff that you saw that you could kind of read on the flight home, for example. I think that was the inspiration for this one.

Yeah, so it kind of it fetches Wikipedia content and then stores it locally and then has an index. Also, here with CLI, we can also even ask the

because we have an ADB skill in Gemini CLI, you can also ask it to test the skill itself by saying, hey, you have access to a phone connected via ADB. And then Gemini CLI uses its Android ADB skill to go in and test that the app actually kind of works and that the skill is doing what it says in the app.

And we can just add ask it then. You can put this in a prompt and ask it to iterate, and it'll do some basic validation itself and return it.

Yeah, so this I think is going to be if this plays, this will be an example of doing that. I could also have done this.

Yeah, so this is this is us using Gemini CLI just to do that. And this worked really robustly. I think of the maybe 80 skills our team did internally, maybe more than half were kind of vibe-coded skills, at least initially, which then really, really reduces the barrier for folks to kind of extend an LLM to do kind of new things in a way that's kind of useful for their audience.

Take over the screen. Yeah, yeah, go for it.

All examples here are running on the smallest model in the in-app, or is it in on-device?

The we all of the examples here are defaulting to the to the good question. So the question for the recording is, are all of the examples running on the smallest model? In this case, the examples are running on the 4B model,right?

The 2B model, the skills will work with the 2B model as well. And you can try that. But it's, yeah, your kind of mileage may vary, like, simpler skills, fewer skills, that sort of thing. But all of the examples you've seen are running on the 4B model.

Can I try to do the Chrome CVP?

Trying doing a version of skills in Chrome?

Yeah, yeah, so like using the Chrome CVP to with the skill to actually use a local model to work in your browser.

We haven't, no. Yeah, it's an interesting thing to try.

OK, oh, yeah, and this then is quick shout-out to this link. So a few things about Gallery that I should also mention are quick check on time. A few things about Gallery that I should also mention are one is Gallery is an open-source project.

And the open-source project itself builds on top of kind of the LiteRT-LM kind of tooling that you saw kind of earlier as part of the intro. So the app itself, you know, as well as being something that's kind of fun to use, so you can, like, I guess, kind of prototype kind of app ideas or to see, wow, is this possible in the model on a phone, or how fast would this be if I ran it on a phone, or is this sort of skill actually viable to do?

As well as being able to kind of do kind of model prototyping, you also have kind of full access to the source code here. And the

and the model the model also enables or, sorry, and it also builds on top of the same infrastructure, like the open-source LiteRT and LiteRT-LM acceleration infrastructure under the hood. So if you see something you like in Gallery, you know, yes, you can deploy the Gemma model some other way as well.

But if you want to get the same experience or the same speed, you can then just take the underlying kind of open-source APIs and pull a model from our Hugging Face page that we'll look at more in a little bit and just run that directly as well.

So then as part of that, we also have a set of discussions on the Gallery on Gallery that shows the some of the community skills that have been uploaded from things from, like, Cat Entertainment to

Blackjack,right, I guess. So these skills you could also you can people have posted these skills. So if you develop a skill, feel free to post it here, and it'll make it more discoverable by other people in the community.

And then the ones here that seem compelling or maybe useful to a lot of people, we can add to our kind of third-party preferred skills list in the app so that then it'll make it easier for folks to discover.

Yeah, so this is yeah, this is something we saw a reasonable amount of action over the weekend on. And it's been something that has only been up since last last Thursday.

OK, quick time check. OK, so next up, I wanted to switch gears. So everything we saw with skills kind of really applies to, from your question, applies to the 2B and 4B model, which mostly on mobile phones will be destined for, like, system gen AIs, where that'll really turn up in production.

Certainly for IoT or for desktop or, you know, for edge applications, you could just load that model yourself and run these sort of skill use cases in production on, like, that kind of Qualcomm IoT platform we saw earlier, for example.

But to deploy to deploy models in-app today and kind of ship them in production apps, like, we're seeing more people are using smaller models to do that. So we would call tiny models, which are less than 1 billion parameter models.

In our in our view, these are the sort of things that we work with teams to deploy LLMs within their app. And these are the types of things we're seeing. So I want to just kind of briefly take a look at what that workflow looks like.

So LiteRT-LM, this is the engine that kind of that powers Gallery,right? And itself is a open-source project that has

C++ and Java APIs. There's Swift APIs coming soon. It also has a Python API as of last week, which is kind of relevant for

relevant for IoT developers who kind of like to work in Python.

And that takes an LLM file and has the required components to kind of fully run an autoregressive loop and expose that for your easy-to-use APIs.

Yeah, so it's cross-platform C++ APIs. And where relevant, the same API can be used with hardware acceleration, like the Qualcomm example.

We also, for other smaller models in the past, we've published

we don't have this yet published for Gemma 4, but in the past, we've also published models that work on MediaTek silicon as well and Intel silicon as well. So yeah, so over the course of the year, and as there are more and more smaller models available, we'll see broader NPU support.

So then the workflow is, you know, you start from transformers. We use a package called LiteRT Torch that can do that has some hydrogenative optimizations that will optimize for LiteRT and also has quantization built into the workflow. Then that gives you a LiteRT-LM file, and you can then deploy that.

You can either prototype it with the Gallery app or just deploy it directly to your kind of production candidate use case using LiteRT-LM and whichever platform you want to work with.

Yeah, this is another view of that flow. I guess, yeah, for really advanced use cases, we have LiteRT Torch generative API. So if you actually want to write your own if you want to write your own tiny model from scratch and train it from scratch, we support that workflow as well as just kind of fine-tuning stock standard workflows.

And there's an API called the Torch generative API that allows you to

basically, it's got kind of building blocks for LLMs that supports many common LLMs that are in native PyTorch that are in a way that give really, really good performance when you run them on-device. And yeah, so that's the flow.

Wow, this is really in the weeds now. So we this is our stack to kind of deploy on NPUs.

The wow, takeaways from this slide are maybe two things. One is under the hood, we actually invest an awful lot of work in optimization libraries. So we've XNNPACK, which is a CPU optimization library, and MLDrift, which is an optimization libraries for GPUs.

So we have teams who work in these to ensure that both of these have excellent performance. Lots of Google's 1P apps rely on both of these libraries. So we're very motivated to ensure they're they have really great performance and work in the widest possible set of devices.

And these are used what we call, like, the JIT workflow, like, where we produce a single artifact called a LiteRT-LM file or a LiteRT file that can work across CPU and GPU and get deployed to lots of types of devices.

For NPU, we need something a little more specialized. For NPU, we need to call a vendor compiler plugin up front, and that uses an ahead-of-time compile workflow. So you need to produce an artifact that's particular to a particular NPU.

And then within our runtime, we call a dispatch to a particular

API that's allowed to dispatch work to the device driver of the NPU. But, like, both of these are available through a consistent API. So even though the past the JIT versus AoT affects the kind of the build workflow, the actual the app development workflow is very similar across NPU or CPU or GPU.

Export and inference is really simple. We have LiteRT Torch to just export. I know I've talked a lot about Gemma models because it's kind of like the Gemma week for us. So we're very excited about Gemma at the moment.

It's worth kind of calling out, we do support third-party models as well. So there's a Qwen 0.6b parameter model is what you're there. And those Qwen models also work in the app,right, which is what you're going to see on theright-hand side.

And the app also supports just loading any LiteRT-LM file and kind of running it and getting benchmark stats, which we'll hopefully see in the video in a second. So you can run it, like, say, on GPU, and then.

Is it the latest pixel that you're simulating here?

Wow, actually, that's a great question. I, to be honest, I don't know this. We do a lot of testing on Pixel and a lot of testing on on S25 as well,right? So I assume it's one of those two devices,right?

But I don't know definitively. And we actually, we didn't show it there because, like, recently, like, in the last release, we've added an optional icon underneath each chat. And if you click in it, it shows you the kind of prefilled decode stats for the model.

So you can just if you want, like, so you can you can find any model we have in Hugging Face and load that in the app and then run to see the benchmark stats for this model on a particular phone if you want to.

So yeah, so you can and also in the command line, like, if you run if you're running on desktop, you can also just use kind of LiteRT-LM run. If you want to kind of do some desktop prototyping to kind of understand how models work in desktop, you can also do that.

And then, yeah, so I guess this is another yeah, this is another third-party model. This is Fast VLM. This is a model from

this is a model from Apple. It's actually really it's a really nice VLM model that's only 500 million parameters. And literally, this is running with hardware acceleration on Qualcomm, which is why it's running so fast. And this is running yeah, this is running on an S25, I think, because it's Qualcomm silicon.

And this is literally, we just have it running in a loop saying, describe the scene, describe the scene with video input. It runs yeah, it runs really, really fast. So this is a good example of what's possible with a model that's feasible to deploy on-device.

Like, we haven't we haven't, in this case, done 4-bit quantization. But I assume, had we been motivated enough, we could have done that. And we would then have, like, something that has, like, just that only requires an extra maybe 250 or 260 megabytes in your app to give this type of experience.

So yeah, they're certainly feasible. And this is earlier when I was saying about models. Like, this is an example of a general-purpose 500 million parameter model. Like, it's been trained on a fairly narrow set of things, which are, like, kind of scene descriptions or just, like, image to description.

So it's like, it's a narrow-ish use case, but still general purpose also,right? So this is a good example of the larger tiny models. The smaller ones, like, that we have with, like, Gemma 3270M, for example, those models typically require fine-tuning to do a particular task.

But yeah, but this is a good example. This is why I included it. It's a good example of a general-purpose tiny model that is very useful. There's also, like, it's worth noting, like, I don't think I've included in the talk.

Other examples of general-purpose tiny models include there's a lot of very small transcription models or some narrow pairwise translation models out there at the moment as well, many of which we support, many of which are on our Hugging Face page as well.

If you want to kind of check them out, there's a yeah, there's a bunch of things available on Hugging. Oh, yeah, so yeah, next slide is this is just showing a handful of models that I wanted to talk about.

So Function Gemma is one that we published with a partnership with GDM last year. This is Function Gemma that was a general-purpose model that you can further fine-tune for function calling. And there is collab notebooks out there if you want to look it up in terms of how to format data sets and how to how to fine-tune how to fine-tune Function Gemma for function calling.

The next two ones above that are Mobile Actions and Tiny Gardens. Mobile Actions is a this is the example I mentioned earlier that has the 10 different mobile actions that are we fine-tuned ourselves, where we achieved, I think it was, like, 86% or 70% reliability on those.

Tiny Garden is another, like, this is a game that is we built inside in the Gallery app that you can play with as well. It's just another example of a voice-to-function calling fine-tune model that you can use.

What's the IT suffix of?

IT is instruction fine-tune. So we in models, when GDM publish models, they sometimes have PT suffixes and sometimes have IT suffixes. PT is when PT is when we publish a model just after pre-training before fine-tuning. That's kind of helpful for kind of expert users because you can do all of your own instruction fine-tuning.

So you can fully control the model's personality, and you're not trying to, like, unlearn some fine-tuning we did. In this case, the way we teach the model how to do function calling is actually via instruction fine-tuning itself. So when we publish a model for further fine-tuning for function calling, it already has a, like, function calling personality.

And that's actually what we wanted to publish. So in this case, this model is for further fine-tuning, but it's an IT model. More typically, if you see for larger models, you like, certainly for the Gemma 3 family, we had both IT and PT checkpoints.

Yeah, we had both IT and PT checkpoints for those, depending on if you had a large volume of data and you want to do full fine-tuning yourself. Like, I think there's there's a separate conversation on, like, sovereign AI use cases that some of the DeepMind people are going to do later in the week.

And I imagine that will be, hey, start with our pre-trained checkpoint and now add a huge corpus of data of sovereign data or enterprise data, and you can fully fine-tune a 27 billion parameter model to do stuff. Yeah, oh, another example here is Embedding Gemma.

I know that's not a technically, it's not an LLM, but Embedding Gemma does that's a text embedding model that that we published with the Gemma team last September. And that does text embeddings for RAG-type use cases. But that's also, like, a very, very strong embedding model that only takes 300 million parameters.

It's another example of a kind of high utility.

Yeah, I guess it's not technically an LLM, even if it is a transformer inside. But it's another example of a tiny model that's very useful for on-device use cases.

OK, yeah, I've kind of already covered this, I think, because we have, yeah, both AoT is AoT compilation is our workflow for hardware acceleration. On-device is best for distributing small models to lots of platforms.

OK, yeah, and then I kind of covered this as well. So LiteRT-LM itself is built on LiteRT, and it supports, like, all of these types of models. And you can find, like, non-LLM models also on Hugging Face. This is kind of relevant because you typically, when you're building a more complex app, you need some things around an LLM, like a voice activity detection model or a denoising model.

So many of these models are also available using just the LiteRT runtime. Like, LiteRT-LM is the runtime that has the full autoregressive loop. It builds in LiteRT. And there's a set of LLM models that are available to use with LiteRT-LM.

But there's lots of supporting

just kind of how do you say non-autoregressive models that are available to use with the LiteRT API that are kind of relevant to deploy that are typically used to deploy a full app.

OK, and this was the earlier thing I was saying about for advanced usage, you can fully customize models. So if you wanted to write your own Llama or Moonshine or Phi or Qwen variant, you can using the coding directory.

OK, I'm going to have probably on track to finish a little early, I would say. OK, so that's it in terms of the tiny models examples and workflows. So there's a lot and there will be yeah, so there's a lot we can do now with very tiny models.

500 m class are available for some standard features. For smaller LLMs, we've had a lot of success fine-tuning them for apps. And what I'm going to show next is an example of a app that we've built using tiny LLMs.

So this is an app that's available on iOS only for some reason called AI Edge Eloquent. What this is, is a kind of transcription model. But instead, you may have noticed that as a speaker, I say lots of ums and ums,right?

If you got the transcript of this presentation, it would it wouldn't be a great transcript. There would be lots of lots of interjections. This Eloquent is built for that type of transcription user story where you

where it does dictation, but then as a separate kind of automatic kind of polish step where it can kind of remove all of the kind of interjection and words. So if you want to dictate a message for use later, it's able to, you know, clean up idioms of speech.

One of the other things that it has as well is a biasing list. So one of the things we found ourselves is, like, if we're talking about LLMs, you would say, oh, have you trained a LoRA for this or got it?

And any standard any standard transcription service would when you say LoRA, it would translate that to the name L-A-U-R-A. That's typically what happens. Whereas you can give it a list of kind of keywords or technical terms as well.

And then the model will kind of bias to those words,right? Because that's how real people spell. So you can so the kind of key features here is, A, it runs entirely offline. B, it's got its own kind of biasing dictionary, if you want to call it that.

And then C, it kind of cleans up the text. If you see in the middle, it has this kind of polish step,right? And overall, yeah, this is the kind of gives a pretty neat offline

yeah, kind of no cost,right? Like, it goes back to that kind of cost motivation at the start as well, because there are some paid services that do this. But it can give you really clean cleaned-up text.

Is that last step done by changing like, setting special tokens in the RL? Or, like, is it done last, like, by overlaying a map or something?

Which is this? The.

The last step where the LoRA example? Is that done by the second model? Or is that done in the model?

Ish is we're going to give me two slides. I'll answer your question. But yeah, good question,right? The question for the for the recording was, is the is the polishing done inside in the main model or outside? And how is biasing applied?

We're going to answer it in two slides, hopefully. Yeah, personalization. Yeah, this is the personalization. So you can add you can add things like LoRA is the example we always use because it's kind of near and dear to our hearts.

So it allows you to if you want to connect it to your Google account, it can import stuff from Gmail or something and look for unusual words and then add them to their biasing list. Or you can add them in yourself.

Those sort of things people usually put in here are names because, like, models will get uncommon names wrong frequently,right? As well as technical terms,right? So, like, Jianing is and Cyril, these are two of the people on the team who developed the app.

Yeah, so no surprise. That's the example we have.

OK, so this is where we yeah, so the text polishing engine is what we're calling it in this stage. So we have two steps here. Important to note that both of these yeah, firstly, I'll describe, then I'll talk about models.

So microphone input goes into a speech recognition engine that delivers a unfiltered transcription. Then in the bottom half, we have the kind of personalization flow where you get a set of kind of uncommon or unique words that then goes into relative terms.

Both of these go into a text polishing engine, which is then a dedicated mini-LLM just for text polishing,right? We could probably have built, like, one LLM to do all of this,right? But it's actually, like, sometimes, like, one of the other realities of mobile development is sometimes it's, like, with tiny models as well, there's this kind of, like, modularity story.

So that same transcription engine in your app, you may have some other use case for that, and you may not want to pay. You know, you can recoup the cost of those weights by using them in multiple places,right?

So this is the pattern we see emerging as we build more of these types of apps is that kind of modularity playbook. So yeah, in theory, those two models could be stitched together. In practice, it's kind of a more pragmatic choice to have separate models.

This is more adaptive as well.

Yeah, yeah, and you can also inspect what's happening in the middle. It's easier to debug,right? We see the same a little bit with voice-to-function calling, like, you know, as well. But that's another story for another day is but it's yeah, so OK, then the other thing to point out, the ASR engine and text generation, these have little I don't know if you recognize the Gemma logo in the corner.

These aren't, like, officially published Gemma models, but these are we've basically done the same workflow that we're advising to other people to do. We have taken the Gemma models, the smaller Gemma model,right? This would be a derivative for, like, this would be something from the Gemma 3 to 70M lineage is the best way I would describe it,right?

And we've basically taken that model and done a fine-tuning just to have

just to have a transcription engine and then just to have a text polishing engine. We've fine-tuned them. Generally, what that workflow looks like to give you a bit more insight is we'll use a workflow of using a much stronger LLM in the cloud to generate lots and lots of synthetic data that corresponds to the type of thing we want.

And then once you have a few, you know, low-digit millions or tens of millions, depending on how ambitious you want to be, of synthetic data, you kind of put that into a fine-tuning workflow, and you fine-tune the base tiny model you're working with to get a derived model.

And, you know, that same workflow we're showing here, you know, you can then use to, like, we've used that internally to kind of ship a stronger to ship a stronger note-taking app,right? But that same workflow is the reason I'm showing it here is, like, a real-life example that, like, of how we're using Gemma-derived tiny models in order to be able to, like, build new production apps.

And we're using this flow in, yeah, in for lots of other use cases internally as well, supporting Google One P products and lots of other ways that those products will probably talk about themselves in time,right? But yeah, like, like, smaller Gemma models are really, really powerful for this type of use case.

And we're seeing good mileage coming from this.

Are you going to, like, combine it with, like, keyboard? Because this is a very useful.

Yeah.

Yeah.

Yeah, so this is that's what the text polishing engine does. Like, as part of that, we wow, like, the text polishing engine was able was trained was instruction-fine-tuned, I guess, to kind of have a system prompt of, like, hey, these are your special words.

Please please correct anything that sounds like these words to these words. And then also remove

interjections or lack of clarity or even things like, I forgot to say this or scratch that. Like, things like this.

Probably a skill.

Yeah, no, this in this case, it's not a skill because this is a tiny model. We've actually just trained the behavior into the model. It's true with a with the 4 billion parameter model, maybe there's a version of this you could do as a skill, which would actually be an interesting side project.

Yeah, maybe there is a version of this you could do as a skill. But for the tiny models, the playbook we're generally seeing is really useful is synthetic data generation with a larger model and then pick an off-the-shelf, like, Gemma 3 to 70M,right, or similar, and run with that with fine-tuning.

And then you can

combined with quantization, you can then ship a pretty compelling narrow feature to a very wide set of users powered by an LLM that works on lots of devices.

Do you have, like, in GitHub before, like, some workflow how to fine-tune it? Because not just fine-tuning,right? You had, like, warm up the training and then start.

Yeah, we do with the Gemma 3 to 70M publication, we do have a collab notebook. And with Gemma 3 to 70M and Function Gemma, both of these, when we ship those models, have collab notebooks that show how to do kind of fully fine-tune models, how to do full fine-tuning.

Yeah.

OK, so regarding the fine-tuning.

Yeah.

What's your experience? Let's take the Function Call example.

Yeah.

And you said you had 80% chance of hitting the 10 functions.

So it was AR we finished that.

That is probably after fine-tuning.

Yeah.

And could you say some numbers before? What's the amount?

Oh, wow.

Oh, wow. Is it 10%? Is it 50?

Like, 40-something percent, 40-something percent to 86%,right? Within that 86% as well,right? Because I don't want to there was one oh, wow. We had 10 functions, and maybe two of those functions dragged our average down a lot. There was, like, eight simple functions that were over 90, like, 93% type thing.

So we on very simple functions, we had, like, really, really high reliability. And then yeah, I just forget the actual details, but there was, like, two that kind of brought the average down to more, like, kind of 86%, which is where we finished.

There's also a blog post you can read about that for more detail. Yeah. But we'll see this in yeah, we see this with smaller models all the time. It's like a like, our experience is on a given eval, like, fine-tuning is between 20 and 40 points on the eval.

So it's like, it's a really, really significant win for tiny models,right? Like, when we're talking 200 million parameters,right, or 270,right, like, we publish with 3M. Yeah, their fine-tuning is is essential for most things,right? Unless you have a unless you have a model that is already published, you know, to do a narrow task,right?

Like, you'll find a transcription you'll find transcription models out there that are in that size that work really well at one task and don't require further fine-tuning. But that's because they've been scoped to the narrow task from the outset,right?

But if you want to do your own narrow task, yeah,right? And ship something to lots and lots of devices, then yeah, fine-tuning is at least for now, it's the workflow of choice. Yeah, it may change.

Do we get a collab that can use web view? Now, these models, of course.

That's a good question. So we have a collab to get as far as the LiteRT file,right? Our support for LiteRT-LM on web is work in progress,right? Like, we

yeah, yeah, I'll just say that. Like, please look at our GitHub. Please look there for latest status,right? You'll see it. Yeah.

So, like, will we get a fine-tuning manual for Gemma 4 by any chance?

There are so Gemma 4, what was announced last week was small models and medium-sized models,right? In the past, for Gemma 3, we also published tiny models,right? So

but this is, like, they were the first models we shipped for Gemma 4 were last week,right? So there will be more Gemma models in future, I imagine,right?

There are some fine-tuning already available for the larger models,right? But for the tiny models we've published at this point in time,right? And if anybody's reading or listening to this talk in a few months' time, please search the web for the latest information.

But as ofright now, the tiny models we have published are Gemma 3 tiny models. And they do have for Function Gemma and for Gemma 3 to 70M, there are fine-tuning workflows available,right? For Gemma 4, as available last week, there are some workflows.

There are fine-tuning recipes in Vertex and elsewhere, I believe, for those. But check out the other Gemma talks later in the week, and you'll see and hear more.

Oh, yeah, so the Eloquent thing is actually available on iOS if anybody wants to give that a go.

Not available.

Not in Europe?

Yeah, I could not download it.

Oh, OK.

Yeah, you can find it on web browser, but not on the App Store.

Oh, wow.

So that needs to be enabled, I guess.

That's really helpful feedback. I'll pass that along. OK, yeah, and then wrap up. Yeah, the kind of key takeaways, System Gen AI, medium-sized models in our or small models in our parlance,right, will be kind of turning up in a mobile device near you.

Those same models are excellent for use in embedded systems, embedded platforms.

At least for now, given the memory we have in mobile phones, which doesn't look to be getting larger anytime soon, given the cost, tiny models are kind of where it's at for deploying things widely,right? We hope to make those easier and easier.

Like, we hope to have stronger and stronger models available through the partnership with GDM. And we want to also make the fine-tuning workflows as easy as we can,right, to enable to make this kind of more accessible. But yeah, happy just to kind of share what we've been doing in both of these fronts.

So I think we have nine minutes if anybody else wants to ask any questions. Happy to yeah.

Is it challenging to handle safety on the edge models? Because you don't have the ability you don't have, like, a hosted server where you can do a holidays check-in. You just have to shove a lot of this into the training.

Yeah.

You don't want to restrict things too much.

OK.

Because of the balance.

Yeah, like, great question. So questions about safety on edge models. So firstly, the Gemma team spend an awful lot of time on this, and I would defer you to them for all the questions on safety for the models that were published last week.

But they do like, it's really top of mind. Like, I like, they spend an awful lot of time on safety for those models. Additionally, for, like, for System Gen AI within AI Core, for example,right, what actually ships there and you'll probably hear more of this from other people is within System Gen AI, when that actually ships as part of the OS, not as a raw model that we published last week, the system vendor will typically have some sort of input and output safety checker on the model,right, as a kind of aftermarket addition because there's particular things for their product they want and they don't want,right?

So that's another layer. For smaller models then, for, like, really tiny models, the, like, safety there is really important. But generally, the way, like, at least for us, the way we've deployed tiny models is to within a very narrow kind of API or task.

Like, you can imagine with Eloquent, like, the app you saw, that kind of the risk profile there, like, technically, that's more like of a regenerative app than a generative app, if you know what I mean. It's not going to fully make things up on the fly.

So we like, yeah, so there's a, you know, you can kind of look at the scope of what the model is trying to do and the API surface and the functional surface. And typically, tiny models have to make a tiny model work, it generally has narrower functional scope, which allows for a more kind of, like, you still need to do safety, but it's a narrower problem you need to solve when you're kind of looking at it,right, is what I would say.

Yeah.

Is there a way, like, a place where I can look for how to deploy smaller, like, a little bit bigger Gemma models on, like, 5090? Because I have a 5090. I want to deploy it on it and then try it out.

Is there a way I can look it up?

Oh, so the OK, the

hmm, so on 5090, so I believe I'm not sure if we have like, so I know we don't have specific documentation for that, but our tool flow does support NVIDIA GPUs. NVIDIA is also, like, a Gemma partner,right? They they have supported some of the Gemma launches in the past.

So it may be it may be if you look on NVIDIA's own web page, you may see some of that through their TensorRT-LM. I know they've supported some of the Gemma launches in the past. But we don't like, I can't point you to specific documentation of the fast answer.

But they're the two places I would check. Yeah.

OK, yeah.

Yeah, sorry.

Yeah, sorry.

So the examples that you showed for, like, the agent skills, they're obviously, like, one skill execution, like, based on the intent to run the you run the skill that's appropriate for it. That's a single skill in the example that you saw.

Would you sort of, like, change the architecture for, like, multi-skill execution? Or is that difficult with smaller models?

It's a question of so we do actually support multi-skill execution,right? It's something so within the app, if you download the app, you can actually define you can define the skills you want loaded,right? And you can just toggle them on or off,right?

And then you can easily say, like, if you're very, very specific in your prompt, which is, like, wow. Like, if you're very specific in your prompt, which is, like, you know, look up this topic in Wikipedia, summarize to three bullets, and then display a display a flashcards.

If you're really, really specific, that's going to work,right?

Frankly, like, we've only had this model a couple of weeks. So we're still putting miles on the clock and seeing what are the boundaries of how, like, how much can we do skill stacking and skill chaining,right? So we're yeah, we're literally still in the mood of putting miles on the clock there.

So most of the examples we were publishing were single skill. But even in the even in the diarization app,right, if you recall there, like, Alice, the person doing that, she asked for

she asked to, oh, summarize my mood or what time I'm meeting up with Amy. And then, oh, Wikipedia is going to come up. Ask for blah, blah, blah. So, like, within in that example, she was able to kind of show within a single conversation individual turns using individual skills.

Yeah, but like, skill stacking within an individual prompt, I believe we've seen that work where you're pretty explicit. But also, we're still like, frankly, we're still learning,right, the boundaries of, you know, like, this yeah, we're still learning the boundaries of what's possible with these classes of models.

Sorry, a follow-up question.

Yeah.

So it's like the decision to identify which skills are important for the prompt. And given that it's a small model, do you, like, do instruction tuning from a larger model to build that intelligence?

No.

Or is it really good at.

This is

like, the model was wow. The model was trained to be really good at agentic workflows and really good at function calling,right? It wasn't trained like, nothing in the Gemma model was trained specifically for our skill pattern,right? That just kind of came afterwards.

Like, when we got the model and we started playing with it, it was like, wow, this works. And then, well, can we do this,right? So it was more that kind of workflow. So there's no specific training for our skill structure,right?

But the, you know, the team spent lots of time doing general purpose thinking and function calling,right? Like, the GDM team did lots of great work there to give us some a general purpose model that was really strong. But there's nothing special for our app,right?

So if you had a if you had a slightly different take on skills and maybe there's a better skill architecture than the one we've shown you today,right, it's entirely possible. Yeah, you could expect to be pretty successful,right? There's no there's no, like, hidden secret sauce in what we've shown.

Yeah.

I'll take class sorry, I think you're maybe next. Yeah.

I'm building, like, an agentic app.

Yeah.

I've been using all the kind of model providers. And I tried to switch to the Gemma models. I've got a few challenges. The first big challenge is context window. How, you know, particularly once you start doing the agentic loop, the context window.

Yeah.

Go up to, like, 100K quarties a week.

Yeah.

Can you talk a little bit about context window on the E2B, particularly the E4?

Ooh, E2B. OK.

Yeah, I would defer you to the Gemma team for official guidance. Like, the medium-sized models have a context window of 128K. And the smaller models, I would actually need to double-check,right?

You said 32K.

32K. Yeah, that's why I was wondering if it was 32K. So, like, our implementation, like, in Gallery, we default to, like, 8K or 12K or something just for performance reasons,right? But the models do support up to like, E2B and 4B support up to then 32K.

And the other models support up to 128K.

With 32K, you need to use a lot of memory,right? Is thatright?

Wow, I wonder if we have stats for that in the model card. It's for the E2B and the E4B model, the memory footprint for larger context is

it's not as bad as you think,right? There's actually put a lot of optimiz like, the team optimized that metric for those models because it was targeted for edge use cases. So the amount of kind of KV cache that's required for each input token, that was something that was optimized.

So the models behave pretty well on that front. I don't have I don't have a number off the top of my head of, like, bytes per input token to give you. But it's it's it's good for its model class is what I would say.

Yeah.

And then I have a second question. Do you have the iOS version of the Edge?

Yeah, the AI Edge Gallery works on both iOS and Android.

But in terms of being open source, because I think the Edge version for.

Ooh, I'm macOS. That's a good point. I will put that in the coming soon bucket,right? It's certainly one of the items on our to-do list because we published yeah, the iOS app, we only published for the first time in January, whereas the Android app has been available since last summer,right?

But we do like, our intention is to have what you see is what you get experience for developers,right? So you can use the app, have fun experiment with the models, then also get the source code, see how it's built, et cetera,right?

So yeah, that's certainly our intention.

Yeah.

Is there a.

Probably last question because we're at.

Is there a trade-off between, like, fine-tuning individual models for a specific task and then, like, actual amount of memory on device they consume? So it's like, you know, you've got your quantized model be like, 3.6 gigabytes or something.

Yeah.

Does that not, like, add up over time when you're, like, chaining the models together?

Ooh, so to clarif so for which for which model is your question? The.

I just downloaded whatever the E2B.

The E2B model?

Yeah. So if you've got multiple models fine-tuned.

Ah, OK.

Presumably.

So yeah, so for E2B, we would kind of recommend customization via skills or via prompting, not via fine-tuning,right? For the smaller so for the small models that are published, we recommend customization through skills and prompting,right? For tiny models, we would recommend we would recommend customization through fine-tuning.

If you were deploying a smaller model,right, I don't like, on the other path that is available to you for the medium for the small models is LoRA fine-tuning,right? So I know Apple supports that in their foundation model framework.

You can check with the AI Core speaker if that's on their roadmap,right? There's an AI Core speaker who has an AMA coming up. You can check with him about their roadmap for this. But certainly, if you were deploying that on an embedded system,right, like, if somebody asked me about deploying the 2B on an on a robotics platform, I would be like, absolutely.

You should fine-tune LoRAs for each of your things for each of your tasks. And then, like, our runtime supports loading the model and, like, hot swapping LoRAs. So you don't even need to kind of load and unload the model to load and unload LoRA adapters that it's built for that particular use case for, like, robotics or IoT platforms.

And that's and those LoRAs then are, like, maybe yeah, it depends on the radix you choose, but they're much smaller. It's like, you know, maybe 16 to 100 megabytes or actually, even smaller, like, 8 to 100 megabytes in that kind of range depending on the radix you use.

Yeah.

Thanks for the question.

Yeah. Cool. Allright, that's a wrap. I'm going to let everybody get lunch. Yeah, thank you.