# Running LLMs on your iPhone: 40 tok/s Gemma 4 with MLX — Adrien Grondin, Locally AI AI Engineer · 2026-04-20 Adrien Grondin, developer of Locally AI, demonstrates how to run Gemma 4 and other LLMs on iPhone using Apple's MLX framework, achieving 40 tokens per second on the latest devices. He explains that MLX is optimized for Apple Silicon and that the open-source mlx-swift-lm GitHub repo enables easy integration into iOS, macOS, and iPadOS apps in under 10 minutes. Grondin recommends quantized models from the Hugging Face MLX community—typically 4-bit to 8-bit—and shows a live demo of Gemma 4 generating text offline. He confirms that mlx-swift-lm supports tool calling, though structured generation is not yet available. He also notes that Locally AI has been acquired by LM Studio, which now lets users run models via MLX or llama.cpp and connect them through OpenAI or Anthropic-compatible APIs. ## Questions this episode answers ### What performance can I expect when running LLMs like Gemma 4 on the latest iPhone? Adrien Grondin shows Gemma 4 4-bit quantized running via MLX on iPhone at 40 tokens per second streaming. He notes it's extremely fast and more than acceptable for many use cases, even older iPhones get 20 tok/s which is still useful. The demo runs live, offline. [5:40](https://aie.addtry.com/44b83589-d7bd-4181-a7c0-e4f3f9fc509f?t=340000) ### What are the simplest steps to integrate an on-device LLM into my iOS application? According to Adrien Grondin, you use the mlx-swift-lm package from GitHub, install it, then grab a model ID from the Hugging Face MLX community. Pass the ID to the framework, and it handles download and inference. The API is straightforward; he says you can have a running iOS app with a model in less than 10 minutes. [2:03](https://aie.addtry.com/44b83589-d7bd-4181-a7c0-e4f3f9fc509f?t=123000) ### What quantization level should I choose for running models on iPhone, and why? Adrien recommends between 4-bit and 8-bit quantization for iPhone models. He notes the full size is way too large, and going below 4-bit significantly impacts output quality, while 8-bit is best for very small models. Gemma 4 runs at 4-bit achieving 40 tok/s. [4:42](https://aie.addtry.com/44b83589-d7bd-4181-a7c0-e4f3f9fc509f?t=282000) ## Key moments - **[0:00] Intro** - [1:13] Locally AI app runs Gemma 4 by Google DeepMind on-device on iPhone with MLX - **[2:03] mlx-swift-lm** - [2:20] mlx-swift-lm framework: build an on-device iOS app in under 10 minutes - **[2:33] MLX Ecosystem** - **[3:21] Hugging Face** - [3:35] Hugging Face MLX community uploads nearly 5,000 quantized models for Apple silicon - **[4:42] Quantization** - [4:42] iPhone model quantization: 4-bit to 8-bit recommended to balance quality and size - **[5:40] 40 tok/s Demo** - [5:40] Gemma 4 HUB quantized at 4-bit achieves 40 tokens per second on latest iPhone - **[7:10] Try Locally** - **[7:39] LM Studio** - [7:56] Locally AI, on-device chatbot app, acquired by LM Studio - **[9:03] Q&A** - [9:04] Q: Does mlx-swift-lm support tool calling? Yes, tool calling is supported, structured generation is not yet available ## Speakers - **Adrien Grondin** (guest) ## Topics Inference Optimization, Quantization ## Mentioned Apple (company), GitHub (company), Hugging Face (company), Gemma (product), LM Studio (product), Llama CPP (product), Locally AI (product), MLX (product), MLX Audio (product), Qwen (product), iPad (product), iPhone (product), macOS (product), mlx-swift-lm (product), mlx-video (product), mlx-vlm (product) ## Transcript ### Intro **Adrien Grondin** [0:15] Okay, hello everybody. I'm going to show you today how to run Gemma 4 on iPhone with MLX. So first, let's introduce myself: I'm Adrien, you can find my Twitter if you want to learn more about all on-device things. I'm the developer of Locally AI, so maybe you've already seen the app. Locally AI is a chatbot that allows you to run on-device models on your iPhone with MLX. I will just go through what MLX is in a few seconds. Basically, as I said, it's a chatbot, it's fully native. You can also chat with Apple Foundation with it, and many models that are compatible with MLX. One of these models is Gemma 4. So basically, Gemma 4, by Golden Mine, has a lot of models, and some of them can run on iPhone, like the smaller ones, and they're pretty great. Maybe you've seen on Twitter one of the posts I've made that I demo it running in the app on iPhone. It's really fast, it runs really well on MLX, and behind the app it's using MLX. MLX is a framework made by Apple that is optimized for Apple silicon, so mainly the chip in iPhone, but also the chips on the Mac. The app locally is available on iPad, it works also very well on this device, and macOS, and everything is built to be as optimized as possible on these devices. So if you want to run on iPhone a language model, Gemma works well, but you have a lot of models that you can run also: the Qwen model and the small LM model from Hugging Face. ### mlx-swift-lm **Adrien Grondin** [2:03] The place you'll want to go is GitHub, and go to the repo mlx-swift-lm. I won't go into detail on how you implement the repo, I think I'll let your agent implement that for you, but it's one repo that you need to install if you're developing an iOS or macOS or iPadOS app. And you can use that to simply download the model and then run it. The API is very straightforward, very simple to implement. In less than 10 minutes you can have an iOS app with a model that is running on your device. That's very simple to do, as mentioned. MLX, so this is mlx-swift-lm, but if you're more into Python apps or macOS apps, you can also run mlx-vlm from Prince. Maybe you've seen him, he's doing on-device for audio with MLX Audio and visual model with mlx-vlm, and also mlx-video ### MLX Ecosystem **Adrien Grondin** [2:55] to run image generation model or video generation model. MLX, the ecosystem is getting bigger, it's getting bigger, it's really greatright now. You can do pretty much everything, like omni-models, or as I said, text-to-speech, speech-to-speech. There's a lot of things that you can do with a model. And basically, let's say you integrate this model, you can't just run any model on it. You need to get some model, and there's a good place to get the model is Hugging Face. I'm pretty sure everybody heard of Hugging Face. And on Hugging Face, you will want to look for the MLX community. ### Hugging Face **Adrien Grondin** [3:35] This is where all the weight of a model, the quantized weight, the full size, will be uploaded. So you will just be able to go to this community, look for the models. I thinkright now there's almost 4,000 or 5,000 models uploaded, so the community is really active on it. When a model is released by your lab, you will directly have it almost 30 minutes after release, quantized in 4-bit, 6-bit, and everything that you can imagine. Here you have an example for Gemma 4 HUB, which is the one I run on iPhone. There's a lot of variants of it, from BF16, MXFP4, 5-bit, 6-bit, there's everything. So you download mlx-swift-lm, install it with your agent or anything, then you go to the MLX community and you just choose the model you want to run. And then with the ID you can just pass it to the framework and it will be integrated with Hugging Face to download the model with mlx-swift-lm directly. So you just need to grab the ID and pass it to the framework. Usually when you're running the model on iPhone, what you want to do is select some quantized version of the model, because the full size will be way too large. What I recommend is trying, depending on the size, between 3-bit and 8-bit. ### Quantization **Adrien Grondin** [5:01] Usually between 4-bit and 8-bit. Usually under 4-bit it's starting to get a lot of impact on the output and the models are not that great usually. 4-bit is the lower I would go, and 8-bit is the higher I would go if you're using really small models. In my app, for example, I have some bigger models like Gemma 4, but I also have some liquid models that are 350 parameters, and this can run in shortcuts. So you can do a lot of kind of automation, because the models are really fast and really efficient at that size to do some text processing and things like this. And on the latest iPhone, if you take Gemma 4 HUB, quantized in 4-bit, it's extremely fast. It can run easily at 40 tokens per second. I will just do an update in the slide, because I removed a slide where there's the video, but maybe I can add it back because I have a little bit more time. ### 40 tok/s Demo **Adrien Grondin** [6:02] I will just go like this, so just to show you in demo what the 40 tokens per second means. And that's running live, offline. And as you can see, it's really fast. 40 tokens per second is more than acceptable for a lot of use cases. This is, of course, streaming. You can also not do streaming and do a UI that just will wait for 4 seconds. And the output is quite long, so it's generating a lot of tokens. So on-device with MLX, it's working, it's here. It's really not easy and really not hard to integrate. As I said, if you go to the repo mlx-swift-lm, it's a breeze to install. On top of that, as I said again, the latest iPhone is really great, but it works also with older iPhones. You will not get 40 tokens per second, which is quite fast, but even if you get 20 tokens per second, that's already great and useful for a lot of applications, a lot of use cases that you would want to do with your app. You can go and scan this QR code if you want to try it by yourself. If you have an iPhone, the app is on the App Store, it's free to use. The only thing is that you will have to download the model. ### Try Locally **Adrien Grondin** [7:21] That's usually 1 GB or 3 GB, really depends on which model, but that's the biggest barrierright now, it's the size of the model. But this also is getting better. Models are getting smaller or getting smarter, and also the iPhone is getting better. So next iPhone and the next next iPhone, everything is reaching really great usability from what I can see. And also on top of that, maybe you've heard the news yesterday, Locally has been acquired by LM Studio. If you don't know LM Studio, it's basically kind of an AI studio for all your local models. ### LM Studio **Adrien Grondin** [8:05] So you can download any model with LM Studio directly from Hugging Face. You can run them, and you can open the server, you can run them with Llama CPP, but also MLX, so you can really compare the different engines how they work. As I said, you can open a server locally and connect your application to this hosted server with various response types. For example, OpenAPI response type, or Anthropic response type for streaming and anything, and you can just get any model running really easily with that. So I want to thank you, that was a really short introduction, how you can do the same and run any model like in Gemma 4 if you want on your iPhone. If you have any questions, also. **Guest** [9:03] Does this support tool calling? ### Q&A **Adrien Grondin** [9:04] Sorry? **Guest** [9:05] Does this support tool calling? **Adrien Grondin** [9:06] Yes, yeah, I forgot about that. So it supports tool calling. Not yet structured generation. There are some packages on top of mlx-swift-lm that are trying to make this working. Hugging Face is doing it, but you can easily find them online. But mlx-swift-lm, yes, it supports tool calling, so that's really useful if you want to do tool calling and call other systems. And the models are getting also better at tool calling. They were not so great a year ago, now it's getting much better. **Guest** [9:43] I see you mentioned two things. First was the GitHub repo, and then the second thing was the app. **Adrien Grondin** [9:50] So you have the GitHub repo for mlx-swift-lm, that's the package that you will install in your app, and then you need to go to Hugging Face to get the weight of a model. **Guest** [10:03] But for a normal user, they can just download the app from the app? **Adrien Grondin** [10:07] Oh, and yes, if you want to try my app, you can. If you want to try itright now without having anything to install, you can do that. **Guest** [10:13] And you can choose any open-source model inside the app? **Adrien Grondin** [10:17] If there's a selection, but it's not any. I'm not sure that all the models run correctly on the iPhone, because not all of them work well. Thank you very much. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.