Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind

Good afternoon, everyone. We'll get started. My name is Chintan Parikh, product manager for LiteRT, which is part of Google AI Edge, and I also have my colleague here, Weiyi, also. He'll be joining us also for the Q&A part of this session.

Quick show of hands. How many of you are working on deploying on edge or are keen on learning more about what it's going to be, the big benefits? OK, and some of you are already deploying. So I'm going to go through the slides.

I'll try to also leave it open to understand if you guys have any use cases or any sort of things you're working on that you'd like to discuss so we can kind of keep it open in that sense.

Great. So here's going to be a quick set of agenda items, and I'm going to go through some of the new models that are coming up, also some edge use cases that would be relevant. I'm going to show you some of the new capabilities in the Gallery app, and then we'll go through our stack for deploying AI on edge devices.

In addition, we also want to emphasize the cross-platform support we offer, because I think a lot of you are looking to deploy on more than just on mobile platforms, but also other platforms. So we'll kind of do some of that.

Google DeepMind recently launched Gemma 4, and this talk is going to be focusing a little bit more on the 2B and the 4B edge models that are focused on deploying on-device. Google certainly has a big suite of models, and even the Gemma 3 family, which are also in the smaller size, all the way down to 270 million parameters.

So if you're looking for extremely small models that you're able to fine-tune, we certainly have a Hugging Face page, which I'll go over, which has more of these models. So yeah, the big evolution with Gemma 4 is going to be really moving from chatbot-type capabilities to more autonomous agents that also support reasoning capabilities and more sophisticated features.

And a lot of it is also it's going to be great how you can deploy these on different devices. So running on edge has many benefits. I think my colleague went over some of these yesterday, and I'll just run through this really quick also.

So certainly, latency is important for those who are keen. Any of these real-time camera use cases, like filters where you're looking to replace backgrounds or video calls, real-time latency is king over there. So on-device can help with that.

Privacy is also critical for any use cases where you have sensitive details. You're doing some kind of summarizing documentation, which is sensitive. Offline use cases where you have poor connectivity and cost. I've seen so many presentations this AI Engineer event where people are complaining about the number of tokens that are getting used and this and that.

But I think on-device always offers this hybrid approach, where if you're looking for how do I really offset running things on the edge versus on cloud, and where can I get the best balance, I think there is something here.

Cool. So in terms of models, there's two models, and I think a lot of the use cases I'll show are going to be built on these. So the Gemma 4E 2B, roughly from an amount of RAM usage, it's roughly anywhere from 1 to 2 GB of RAM usage.

So it could be usable, but it depends on your end use cases again. Certainly good for any voice interfaces or use cases for summarization or any kind of low-latency local processing. 4B is going to be a little more heavy-duty.

If you're looking to run on bigger platforms like laptops or IoT devices, it'll have a higher RAM requirement. Again, this is once it's been quantized to your desired size.

So I want to do a quick deep dive on what's going to be new this time on Gemma 4E 2B and E 4B capabilities in terms of the agentic capabilities. So I'm going to show you what's already there in the next couple of slides in terms of use cases, and here touch a little bit on exactly what other new capabilities are there.

So function calling, built-in support for tool calling, and also essentially models to interact with other local APIs. So you can certainly start the inferencing on edge, but you have the path to essentially call other APIs outside, and I'll show that in some of my examples.

But the core of the inferencing will be on the edge. Structured JSON output. So for native support for any structured JSON output is supported here, which was built into the model architecture rather than achieving through some kind of specific prompt engineering.

You can kind of do this too, and I'll show examples for the same. Chain of thought is new. So there is a thinking mode, which can be demonstrated by an app, where the thinking mode will help you understand the thought process that the model is going through.

Our Gallery app, which I will showcase, will support that. And finally, what's going to be great is these models are optimized for hardware-native support, means you can run seamlessly across multiple platforms and multiple hardwares. And really, we want to give you the flexibility of deploying into various platforms.

So just for starters, if anyone's thinking, OK, so where can I get these models? We have these models already ready to go. So our Hugging Face page has these models. So if you're looking, they're all Apache 2.0 licensed models.

So you are able to download these and then start building with it. And we'll give you some paths of how you may want to build with it. Allright, so yeah, let's talk about some of the use cases now.

So in terms of these are just some of the many possible use cases. These are use cases that are possible today. This is our Gallery app, which we have been demoing down on the third floor. And we also have a demo session after this for anyone who's keen on learning more.

I do have QR codes for this too, so in the next couple of slides. At present, what's new is the Gallery app allows you to demonstrate the agent skill capabilities. And also, it has the audio scribe capabilities or Ask Image, and also other features that were related to chat experiences.

All of this is happening on-device. The purpose of this app that is created by Google is to help you get a playground to get a feel for what these models are capable of. Each of these capabilities has sample code as well that anyone can go and grab, and then you are able to also fork this app to build your own experiences.

But this is essentially to inspire and motivate you to build your own experiences.

So some of the next couple of slides I'll be focusing on is going to be related to some of the Gemma 4 edge use cases. These are going to emphasize on the voice agent capabilities or the local agent capabilities, and also a lot of these are going to be privacy-focused as well.

So there are three key pillars that are going to be new here versus what was already supported in the previous slides.

So let's dive into the Gemma 4 use cases. So let's see this place. So here's one use case where you're augmenting knowledge. And for example, you can build a skill to query Wikipedia, allowing the agent to query and respond to any encyclopedia question.

So this is a skill that's already available in our app that's there. So if you're looking to build something like this, this is possible. These are new skills in addition to basic on-device skills like summarization or Ask Image and things of that nature.

This is another category where essentially, let's see if you can.

Remove journal entry with score 9 and comment.

So Amy.

I've only got eight hours of sleep, and I'm looking forward to hanging out with Amy today.

So here's what you're going to see creating and summarizes and display trends of hours of sleep. So you're going to build a quick agent that helps you track your mood.

And analyze the trend in my mood over the last seven days.

So you can kind of see that it's able to create this all on-device. It's taking your input, it's feeding it, it's able to understand. So the reasoning and the thinking capability are new in the new model, and the on-device capabilities have gotten a lot more powerful.

Another similar example is going to be on expanding the core capabilities. So for instance, you can also pair photos, and it's going to read the photo and have image understanding and generate music also all on-device. So let's try this one.

And for breakfast, I'm sending a photo. Can you pair this vibe with some music?

It's taking a few seconds, and then.

So a lot of these skills can be written by yourself on the app itself. So you don't even need to leave the app. The app has instructions on how to do this. Again, the whole focus of this is to help you understand the capabilities of a model and build something by yourself that you really like.

So that's useful. One last one is to really want to do sort of a working app that describes, let's say, vocal calls of animals. Now here we are trying to navigate multiple apps and users, and you can manage a more complex workflow here in this case.

So this is another example.

And you can also change the CPU, GPU, and hopefully have NPU support soon, so you can decide the accelerator you want to use when you decide. So this was sound generation here from purely the prompt by the person.

Again, the skill was set up and created and loaded, and the skill is running on-device. And this is for something similar. So there's a lot you can do with it. There is a whole GitHub repo that we have where actually users are posting their own skills on the GitHub website, and they're able to basically share that with others, with the community.

So you're also welcome to explore that, or you're welcome to download one of these apps. Another option is the sample app is actually open source as well on GitHub. So you're welcome to take it and fork it, and you can also make changes to that app if you like.

So the GitHub link is also here, and it's an opportunity to do that. And creating your own skill is also here. So these are just examples of what folks have done. This QR code is instructions on how to build a skill.

So if you're looking to essentially get guidance on, OK, how do I do this, you can kind of get started. Allright, the next part of the talk is I'm going to focus a little bit on deploying this now on edge devices.

So we've talked about what's possible with the Gemma models. The framework that is used here,right? So Google has an offering with LiteRT for bring your own models, essentially. So LiteRT essentially is Google's on-device framework. It's built on the TensorFlow framework, TensorFlow Lite.

If you guys are familiar with that, is anyone familiar with TensorFlow Lite? Some of you guys? OK, awesome. OK, so that's what this is built on. It's really meant to also be built on using the same TensorFlow Lite model format, and that's what we are focusing on at this point.

So here, the point is that the underlying framework for the app that was running all these experiences is LiteRT. And then we are going to show you that this is basically it's been one of the most widely deployed frameworks so far.

It has 100,000-plus apps, billions of active users, and also lots of daily interpreter invocations. This is essentially the number of inferences that are happening every day. Why this is interesting is to show that we are building on a trusted foundation.

And when you do this, we also have our TFLite file format. And this is going to be very important because if you were a developer that was building with TensorFlow Lite, your models are still going to run on LiteRT.

And these models, the same model format is cross-platform. So it's not just Android, but you can run it on iOS, macOS, Linux, Windows, web, and even IoT devices. And I'll also show some examples on IoT that we were actually just building this morning, actually.

So I'll show you that.

Cool. So from a development flow standpoint, essentially you have your model files. So we are able to accept the reason. It's got branded to LiteRT. One of the bigger motivations for the rebranding was to also demonstrate that it's not just TensorFlow Lite models that we accept, but also PyTorch models and Jax models.

So if you are working with PyTorch models, you can take one of those models, convert it to the TFLite file format, and then go through this journey and deploy. So we have a lot of sample apps and things like that, but I will not go through the whole journey here.

But simply put, you have portability of the model, and you have multi-framework support for the models.

Allright, so this is a complete solution that provides one unified cross-platform architecture. Maybe a quick question. How many of you are looking to deploy on multiple Android or iOS devices? You want something that you build that you can test easily on many devices.

Because if you are trying to build apps that you're like, OK, I made it, but how do I know if it's going to work on five-year-old phones and six-year-old phones and all of that stuff? So we have options for that too.

So I'll walk you through the stack. So LiteRT Torch is going to be your conversion path. Basically, it's your bring your own model. If you found a model, you like it, you want to run it, you convert it to TFLite format.

You can quantize it if needed. If it's LLM, you go through the LiteRT LM path, essentially. And then if it's not, you can go through LiteRT. What's interesting here is that we also have the Model Explorer tool, which can help you explore the graph and decide which aspects of the graph you want to change and quantize.

So you can actually study the graph and decide how to best do mixed precision or basically convert the quantize the model as you like. The AI Edge portal is a benchmarking tool, which could be of interest for those who are looking to deploy broadly on Android.

So this is a cloud-based benchmarking service called AI Edge portal. This is available basically to help a lot of our third-party app developers and even internal developments use this tool essentially to get a good pulse check that, hey, if I have a model that has so many parameters, do I need to use ahead-of-time compilation or just-in-time compilation?

What is going to be myright recipe to ensure it's actually deployable across a broad fleet of devices and be reliable in that manner?

Yeah, sorry. One more thing on this slide is the next part is acceleration. So CPU and GPU are pretty universalright now. So these are essentially libraries that will help you run CPU and GPU. I also want to emphasize back, with this sort of framework, you are able to deploy on multiple platforms with the CPU and GPU running.

NPU acceleration is also another focus. So we have completed integration with Qualcomm MediaTek, and we are also focusing on additional integrations with other partners on different platforms. We also offer, if you are looking, I think this is going to be a game changer for a lot of your apps or a lot of your products that you are trying to run a lot of ASR, TTS, or any of these applications that are going to be requiring real-time capability or you're setting up some kind of AR/VR application where you want to be able to have real-time improvements or updates to the camera feeds.

I think the NPU is going to give you at least 3 to 10x improvement in performance, and it's going to be a game changer in terms of the amount of energy you're going to use and the amount of performance you're looking for to unlock those use cases.

So the flexibility part is we have the ahead-of-time or on-device. And then for ease of use, also there are many options here in terms of how we simplify, and there's a lot more documentation and support on that. Allright, so the next part is I'm going to go with some performance numbers on how we are performing.

So far, we've given you an example of what you can do at the app level, what's possible at the framework level, and how you can scale it. In terms of our coverage, so just with the Gemma models, this is the coverageright now.

So a lot of times they come to the booth or demo booth and ask, hey, OK, so can you just do this on Android? But no, we've tested this on all these platformsright now. So the models that you have on Hugging Face, you can actually test it on many of these platforms, so on Android, iOS, Linux, Raspberry Pi as well.

And here's just a quick demo we just did this morning. It's going to take a while, but this is our little robot that's sitting down in our demo booth. So we just made it this morning, so it's not super performant.

But what we did here is we showed a robot assigned to say, move your antenna. So now the two lights are blinking, so it's doing inferencing. And then you'll see it's running a Raspberry Pi, it's running on a CPU running on our LiteRT LM.

And you'll see in a few seconds that it's going to wiggle the Sharpies that are its antennas. And so there it is. So there are other questions like, will you marry me and things like that? If you're interested, you can try it out downstairs.

It's there.

Cool. So yeah, that was amusing. But also, yeah, we do need to work on the performance because we just made it this morning. We also have a CLI tool. So for those who are looking to deploy and you want to have an easier way of doing it, so there is a new CLI tool that's been developed.

So that is also available on our website in terms of Python binding support and many of that. So it's on our website, so if anyone's interested, you'll find it. In terms of performance, so I have some quick performance numbers.

I just want to emphasize running on some of these new accelerators and NPUs can't give you a big benefit advantage. You can get up to 30x boost in some cases. And also, you have iOS performance as well. So we are supporting all of this here.

So roughly 56 tokens per second and so on and so forth.

Desktop is here too. So a lot of these numbers can be found on our Hugging Face page. So the quant is already listed there. So I'm just pulling from there. So you will find it there. So when you download our models, you will get performance details on all these platforms and IoT.

Our runtime is also very performant versus Llama. At least on mobile, we've seen up to 35 times faster performance. On desktop, it's at par. And then IoT also, we have 3x performance. Allright, so I have a minute left.

So yeah, in summary, LiteRT supports all these frameworks. If you're looking to bring models from different frameworks, you can. We have the models on Hugging Face, and then we also have a gallery app, which is a nice playground for you to try this out.

And these are there. Yeah, so if there's any questions, happy to take them.

Any questions?

Yeah, thank you.

Any questions about prices?

Yes.

Speaking of all that, OK, the security camera at my home, recognizing that my son came home.

Yes.

Because pushing this to the cloud would cost enormous amounts of money. And since this can be done locally with a local computer, maybe this is a security feature of my home camera.

Yeah, so on your phones, like on, let's say, an iPhone or Pixel, when you do face unlock, it's typically running locally already. And that is using this same framework in Google devices as well. Apple's face unlock is also on-device on Core ML.

So that is possible, yes.

And then you would screen it constantly, like every two seconds, for example, to recognize the face? Or how do you do that?

Typically, yeah, you have the camera on and you can stream it. But for your home camera, I think it would be a little different story, I think. Yeah, so you would have to maybe do some sort of an algorithm to check the authenticity and kind of do that.

So it'll be a little different from how it's deployed on the phone. But yes, you can totally run that on-device.

I want to ask you about that. Would be better to apply to the camera and only when that individual is recognized?

Then message your phone,right?

Yeah, I'm thinking about the local device connected with my camera. The local device is actually checking the frames on the camera.

Yeah, that would be a Raspberry Pi. Right. Raspberry Pi could do it,right?

Yeah.

Yeah. Do you have a question?

You mentioned Orin. Is it a target device? How does LiteRT compare to TensorRT for executing on Orin?

Do you want to answer that?

Oh, you mean for LiteRT? We actually use either web GPU or OpenStable DGL. Give me all the best features. That will actually go through the deployment. Do we have the numbers?

Let me see. I don't know if I have Orin here. Not here, but I think we have it on our Hugging Face. I don't think I added it here. Yeah.

We have all the numbers here. Have you tried any exercise where you have multiple nodes running a model? And if any of them trigger something, they connect to a higher agent, another agent that processes that input?

I think we have some groups that are working on, let's say, speaker and a thinking agent type of architectures. I don't have any examples to share, but that is one way of distributing what. And there's orchestration to decide what should run locally and should run elsewhere.

So I don't have examples here, but yes, there are

different types of you can think of a health agent, a coaching agent, of sorts.

Is it very common practice? You have a classifier or training a very small model just to clarify whether the complexity of the requirement. So if you're not, go to yeah, actually, it's pretty common practice already.

Yeah.

Thank you very much.

Thank you. Yeah.

I have a mobile app which is kind of using Gemma Live APIs for audio to audio models.

Right.

Is there any recommendation on this? This seems like audio to text models.

Right,right,right.

Is there any recommendation around that?

So if you have open weight models that you prefer, we are able to support any of them. I mean, hopefully, we just need to make sure we get them in theright file format. And if the sizes are so we should be able to, if you know which models you're thinking about

for application.

Not these, at least Gemma E4b.

Here we have just focused on the Gemma because of the DeepMind track. But essentially, our Hugging Face page has other open weight models. And so some of them we provide for ease of use. Others, you are welcome to convert and use yourself as well.

But yeah, if there is a model that you've identified, yeah, then we can certainly discuss.

Thank you.

Yeah, thanks.