Running LLMs on your iPhone: 40 tok/s Gemma 4 with MLX — Adrien Grondin, Locally AI

Okay, hello everybody. I'm going to show you today how to run Gemma 4 on iPhone with MLX. So first, let's introduce myself: I'm Adrien, you can find my Twitter if you want to learn more about all on-device things.

I'm the developer of Locally AI, so maybe you've already seen the app. Locally AI is a chatbot that allows you to run on-device models on your iPhone with MLX. I will just go through what MLX is in a few seconds.

Basically, as I said, it's a chatbot, it's fully native. You can also chat with Apple Foundation with it, and many models that are compatible with MLX. One of these models is Gemma 4. So basically, Gemma 4, by Golden Mine, has a lot of models, and some of them can run on iPhone, like the smaller ones, and they're pretty great.

Maybe you've seen on Twitter one of the posts I've made that I demo it running in the app on iPhone. It's really fast, it runs really well on MLX, and behind the app it's using MLX. MLX is a framework made by Apple that is optimized for Apple silicon, so mainly the chip in iPhone, but also the chips on the Mac.

The app locally is available on iPad, it works also very well on this device, and macOS, and everything is built to be as optimized as possible on these devices. So if you want to run on iPhone a language model, Gemma works well, but you have a lot of models that you can run also: the Qwen model and the small LM model from Hugging Face.

The place you'll want to go is GitHub, and go to the repo mlx-swift-lm. I won't go into detail on how you implement the repo, I think I'll let your agent implement that for you, but it's one repo that you need to install if you're developing an iOS or macOS or iPadOS app.

And you can use that to simply download the model and then run it. The API is very straightforward, very simple to implement. In less than 10 minutes you can have an iOS app with a model that is running on your device.

That's very simple to do, as mentioned. MLX, so this is mlx-swift-lm, but if you're more into Python apps or macOS apps, you can also run mlx-vlm from Prince. Maybe you've seen him, he's doing on-device for audio with MLX Audio and visual model with mlx-vlm, and also mlx-video

to run image generation model or video generation model. MLX, the ecosystem is getting bigger, it's getting bigger, it's really greatright now. You can do pretty much everything, like omni-models, or as I said, text-to-speech, speech-to-speech. There's a lot of things that you can do with a model.

And basically, let's say you integrate

this model, you can't just run any model on it. You need to get some model, and there's a good place to get the model is Hugging Face. I'm pretty sure everybody heard of Hugging Face. And on Hugging Face, you will want to look for the MLX community.

This is where all the weight of a model, the quantized weight, the full size, will be uploaded. So you will just be able to go to this community, look for the models. I thinkright now there's almost 4,000 or 5,000 models uploaded, so the community is really active on it.

When a model is released by your lab, you will directly have it almost 30 minutes after release, quantized in 4-bit, 6-bit, and everything that you can imagine. Here you have an example for Gemma 4 HUB, which is the one I run on iPhone.

There's a lot of variants of it, from BF16, MXFP4, 5-bit, 6-bit, there's everything. So you download mlx-swift-lm, install it with your agent or anything, then you go to the MLX community and you just choose the model you want to run.

And then with the ID you can just pass it to the framework and it will be integrated with Hugging Face to download the model with mlx-swift-lm directly. So you just need to grab the ID and pass it to the framework.

Usually when you're running the model on iPhone, what you want to do is select some quantized version of the model, because the full size will be way too large. What I recommend is trying, depending on the size, between 3-bit and 8-bit.

Usually between 4-bit and 8-bit. Usually under 4-bit it's starting to get a lot of impact on the output and the models are not that great usually. 4-bit is the lower I would go, and 8-bit is the higher I would go if you're using really small models.

In my app, for example, I have some bigger models like Gemma 4, but I also have some liquid models that are 350 parameters, and this can run in shortcuts. So you can do a lot of kind of automation, because the models are really fast and really efficient at that size to do some text processing and things like this.

And on the latest iPhone, if you take Gemma 4 HUB, quantized in 4-bit, it's extremely fast. It can run easily at 40 tokens per second. I will just do an update in the slide, because I removed a slide where there's the video, but maybe I can add it back because I have a little bit more time.

I will just go like this, so just to show you in demo what the 40 tokens per second means. And that's running live, offline. And as you can see, it's really fast. 40 tokens per second is more than acceptable for a lot of use cases.

This is, of course, streaming. You can also not do streaming and do a UI that just will wait for 4 seconds. And the output is quite long, so it's generating a lot of tokens. So on-device with MLX, it's working, it's here.

It's really not easy and really not hard to integrate. As I said, if you go to the repo mlx-swift-lm, it's a breeze to install. On top of that, as I said again, the latest iPhone is really great, but it works also with older iPhones.

You will not get 40 tokens per second, which is quite fast, but even if you get 20 tokens per second, that's already great and useful

for a lot of applications, a lot of use cases that you would want to do

with your app. You can go and scan this QR code if you want to try it by yourself. If you have an iPhone, the app is on the App Store, it's free to use. The only thing is that you will have to download the model.

That's usually 1 GB or 3 GB, really depends on which model, but that's the biggest barrierright now, it's the size of the model. But this also is getting better. Models are getting smaller or getting smarter, and also the iPhone is getting better.

So next iPhone and the next next iPhone, everything is reaching really great usability from what I can see. And also on top of that, maybe you've heard the news yesterday, Locally has been acquired by LM Studio. If you don't know LM Studio, it's basically kind of an AI studio for all your local models.

So you can download any model with LM Studio directly from Hugging Face. You can run them, and you can open the server, you can run them with Llama CPP, but also MLX, so you can really compare the different engines how they work.

As I said, you can open a server locally and connect your application to this hosted server with

various response types. For example, OpenAPI response type, or Anthropic response type for streaming and anything, and you can just get any model running really easily with that. So I want to thank you, that was a really short introduction, how you can do the same and run any model like in Gemma 4 if you want on your iPhone.

If you have any questions, also.

Does this support tool calling?

Sorry?

Does this support tool calling?

Yes, yeah, I forgot about that. So it supports tool calling.

Not yet structured generation. There are some packages on top of mlx-swift-lm that are trying to make this working.

Hugging Face is doing it, but you can easily find them online. But mlx-swift-lm, yes, it supports tool calling, so that's really useful if you want to do tool calling and call other systems. And the models are getting also better at tool calling.

They were not so great a year ago, now it's getting much better.

I see you mentioned two things. First was the GitHub repo, and then the second thing was the app.

So you have the GitHub repo for mlx-swift-lm, that's the package that you will install in your app, and then you need to go to Hugging Face to get the weight of a model.

But for a normal user, they can just download the app from the app?

Oh, and yes, if you want to try my app, you can. If you want to try itright now without having anything to install, you can do that.

And you can choose any open-source model inside the app?

If there's a selection, but it's not any. I'm not sure that all the models run correctly on the iPhone, because not all of them work well.

Thank you very much.