# Foundry Local: Cutting-Edge AI experiences on device with ONNX Runtime/Olive — Emma Ning, Microsoft AI Engineer · 2025-06-27 Emma Ning, Principal PM at Microsoft, presents Foundry Local, a platform for building cross-platform on-device AI applications using ONNX Runtime and Olive. She argues local AI is essential for low-bandwidth, privacy-sensitive, cost-efficient, and real-time latency scenarios, noting that modern hardware and optimized models like Phi-4-mini and DeepSeek make it viable. Ning demonstrates Foundry Local's CLI, model benchmarking (Qwen 1.5B at 90 tok/s vs. Phi-4-mini with richer output), and a cross-platform document summarization app built with JavaScript SDK, running identically on Windows and macOS. She also previews a local agent with MCP servers (file system and OCR) that extracts receipt totals using Phi-4-mini. Customer testimonial from Sava (CEO of Pieces) highlights improved memory management and tokens-per-second, while another partner underscores ease of install and hybrid cloud-local solutions. ## Questions this episode answers ### What is Microsoft Foundry Local and how does it help run AI on devices? Foundry Local is an end-to-end solution from Microsoft for running AI models directly on devices. It uses ONNX Runtime to accelerate inference across NVIDIA, Intel, AMD, and Qualcomm hardware. It includes a model management service that hosts models locally and downloads open-source models from Azure AI Foundry on demand. Developers can interact via a CLI to explore models or use Python and JavaScript SDKs to integrate local inference into their applications. It is available on Windows and macOS. [4:28](https://aie.addtry.com/95e18b01-d996-40fb-a05b-9d6c99cc67df?t=268000) ### Why would a developer choose local AI over cloud AI? Emma Ning gave four key reasons: offline or low-bandwidth operation is critical when internet is unreliable; privacy and security let sensitive data like legal documents be processed without leaving the device; cost efficiency matters for high-volume inference, such as in games with millions of daily calls; and real-time latency needs cannot tolerate cloud round-trips. These factors make local AI essential for many scenarios. [0:46](https://aie.addtry.com/95e18b01-d996-40fb-a05b-9d6c99cc67df?t=46000) ### How can I build a local AI agent using Foundry Local? Foundry Local allows creating local agents by combining a model from its catalog with one or more MCP servers. Emma Ning demonstrated an OCR agent in private preview: it used the Phi 4 mini model and file system and OCR MCP servers. She asked it to find a receipt image on her device and extract the total amount. The agent automatically selected the right tools—search then OCR—and returned the total, all running entirely locally. [18:23](https://aie.addtry.com/95e18b01-d996-40fb-a05b-9d6c99cc67df?t=1103000) ### What inference performance does Foundry Local deliver? In live demos, Emma Ning ran Qwen 2.5 1.5B on Foundry Local and achieved 90 tokens per second in verbose mode. The larger Phi 4 mini provided more detailed responses at a slightly lower token rate. Sava, CEO of Pieces, reported significant improvements after adopting Foundry Local: better memory management, faster time-to-first-token, and higher tokens per second for their offline-first AI applications. [11:45](https://aie.addtry.com/95e18b01-d996-40fb-a05b-9d6c99cc67df?t=705000) ## Key moments - **[0:00] Why Local AI?** - [0:46] Emma Ning lists four reasons for local AI: offline access, privacy, cost efficiency, and real-time latency. - [1:57] Game apps with millions of daily inference calls are not cost-sustainable on cloud AI, says Emma Ning. - **[2:25] Is It Ready?** - [2:50] Smaller optimized models like Phi 4 mini and DeepSeek make local AI ready now. - **[3:16] Foundry Local** - [3:45] ONNX Runtime reaches over 10 million downloads per month, accelerating inference significantly over PyTorch. - **[6:22] Customer Feedback** - [6:47] "If you're looking to deploy on-device models, you can't go wrong with Foundry Local." - [8:07] Foundry Local demo processes Maria Rodriguez's bank account number locally using speech-to-text. - **[8:29] CLI Demo** - [11:45] Qwen 2.5 1.5B model runs at 90 tokens per second on Foundry Local. - **[13:07] App Demo** - [13:07] Emma Ning's cross-platform app summarizes internal documents on device with Foundry Local. - [16:53] The same Foundry Local summarization code runs unchanged on a Mac, demonstrating cross-platform support. - **[17:55] Agent Preview** - [18:23] Foundry Local's private preview agent feature uses local models and MCP servers to automate tasks like OCR. - [21:12] OCR agent demo finds a receipt on device, extracts text, and successfully reports the total amount. ## Speakers - **Emma Ning** (guest) - **Maria Rodriguez** (guest) - **Sava** (guest) ## Topics Model Context Protocol (MCP) ## Mentioned AMD (company), Intel (company), Microsoft (company), NVIDIA (company), Pieces (company), Qualcomm (company), Azure AI Foundry (product), DeepSeek (product), Foundry Local (product), Llama C++ (product), ONNX Runtime (product), Ollama (product), Phi 4 mini (product), PyTorch (product), Qwen (product), Windows (product), macOS (product) ## Transcript ### Why Local AI? **Emma Ning** [0:16] Hello everybody. Uh, my name is Emma, I'm a program manage- uh, product manager at Microsoft. It's a pleasure to talk to you today about Foundry Local, which enables developers to easily build up cross-platform applications powered by local AI. So let's get started. Uh, the first question is: if the cloud AI is so powerful, why do we need local AI? So here are four key reasons, based on our conversations and observations with our customers. So first of all, how does cloud AI work in environments with low network bandwidth or even offline access? Many of you have experienced the bad internet during this conference,right? And a common sentence we have heard from many speakers with live demos is, "Oh, fingers crossed, hopefully the Wi-Fi connection is good." So it's not their fault, uh, it's required by cloud AI. But it's not a concern at all for my session because my live demo runs entirely locally. So reason number two: privacy and security. Many companies work with very sensitive data, uh, such as legal documents and patient information. They need to process that da- data, uh, entirely locally without anything ever leaving the device,right? And reason number three: cost efficiency. Um, think about, uh, game applications, uh, which are deployed to millions of devices. With hundreds of millions of inference calls every day, it's not just sustainable. And reason number three: real-time latency. So many AI applications need to respond in real time, and it's not just, uh, not possible if we wait on cloud. So that's why we need local AI. Then the next question: whether local AI is ready now. So thanks, uh, thanks to the decades of progress in computing, hardware has become more and more powerful. So many current devices are equipped with modern GPUs, NPUs, capable of running advanced AI models. ### Is It Ready? **Emma Ning** [2:50] Meanwhile, model companies are keeping publishing more and more models, which are, like, leaner, faster, more optimized for local inference, such as, uh, Phi 4 mini and, uh, DeepSeek, uh, small variants. And we are also seeing more and more advanced, uh, state-of-the-art optimization techniques introduced at runtime level. ### Foundry Local **Emma Ning** [3:16] So this convergence now makes the local AI a reality. Then how do we build up the solution? So Microsoft already, uh, has many, uh, great assets. Uh, Azure AI Foundry, uh, introduced, um, last year at, uh, Microsoft Ignite, uh, has been, uh, trusted by over 70,000 organizations with over 1,900 models. And, um, ONNX Runtime, our cross-platform high-performance on-device inference engine, has now seen over 10 million downloads per month. And our customers are pretty happy with the significant, uh, performance acceleration provided by ONNX Runtime compared to PyTorch. And lastly, let's not forget Windows. The scale and the reach of Windows on client devices are massive. So when we think about, uh, democratizing AI, um, the millions of, um, you know, devices, uh, and the millions of customers using Windows devices really matter to us. So we are not starting from scratch. We are bringing all those advanced assets into Foundry Local, an optimized end-to-end solution for seamless on-device AI. So at the bottom, as you can see, it uses ONNX Runtime to accelerate performance across various kinds of hardware. On the top, uh, we are introducing a new Foundry Local management service, which hosts and manages models on your, uh, dev uh, client device. It also connects to Azure AI Foundry to download open-source models on demand. And for user experience, we provide Foundry Local CLI, which allows you to easily explore models on device. And we also provide SDK so that developers can easily, uh, integrate Foundry Local into your own applications, um, from cloud to local from different hardware. So Foundry Local was officially announced just one month ago at a Microsoft Build conference. It's available on both Windows and macOS. On Windows, it is integrated into the platform, which makes the experience even simpler for Windows AI developers. As I just mentioned, uh, Foundry Local accelerates performance across different kinds of scenarios. We have been, uh, closely working with hardware vendors, including NVIDIA, Intel, AMD, Qualcomm, to integrate their hardware accelerators into Foundry Local, ensuring the best-in-class performance that you can get on their hardware. ### Customer Feedback **Emma Ning** [6:22] So before our official announcement, uh, over 100 customers joined our private pre-preview. They have shared valuable feedback on how easy Foundry Local is to use and how good the performance it is. So let's hear some of their feedback. **Sava** [6:47] Hey there. Sava here, CEO and co-founder at Pieces, where we've been on an ambitious journey to give developers artificial long-term memory across the OS. Now, offline-first AI is core to this vision. And in late 2022, we began to explore small language models running at the edge on all major platforms. But between rolling our own Llama C++ and then to Ollama, frustrations around versioning, performance, and end-user experience still remained. That was until our recent partnership with Microsoft on their new Foundry Local platform, an end-to-end AI inference solution that offers ease of use and high performance across different hardware. In no time, our team went from documentation access to a production-ready build with noticeable improvements in memory management, time-to-first-token, and tokens per second. If you're looking to deploy on-device models, you can't go wrong with Foundry Local. **Guest** [7:43] We have been working on AI projects for our customers for several years now. **Guest 2** [7:48] Some clients want to use the latest AI technologies but are restricted from using external services when the information they want to process contains sensitive data. Foundry Local is a perfect solution for these scenarios, as it allows us to easily run Gen AI models locally. Here, we can see a solution that combines Foundry Local with a speech-to-text service. **Maria Rodriguez** [8:07] Hello, my name is Maria Rodriguez. My savings account number is 123456. **Guest 2** [8:15] Which also runs locally. One aspect we were really impressed by was the simplicity of the installation and the ease of using the models. With Foundry Local, we can now create hybrid solutions where part of the solution can be run locally. ### CLI Demo **Emma Ning** [8:29] It's been our privilege to work with these customers to improve Foundry Local together. Allright, I've talked enough. Who wants to see live demos? Okay, uh, let's do that. Um, so first of all, we can, uh, let's see our CLI experience. So on Windows platform, you can, uh, install Foundry Local package using winget. And on macOS, uh, you can use Homebrew commands. Um, so I have already installed Foundry Local, so. So first, uh, we want to see what models supported by Foundry Local. So we can type Foundry, um, model list. So as you can see, it supports many popular, uh, generative AI models. And for each model, you can get, uh, you can get different variants optimized for different hardware. So, uh, you can see we have, uh, optimization version for CPU, for CUDA, for integrated GPU. Uh, we also provide NPU variants because, um, my device doesn't, uh, contain Qualcomm NPU, so that variant doesn't show up. Okay, so we want to run some models,right? And, um, if you haven't downloaded the model, uh, before, the Foundry Local, we'll download the model from the cloud and then run the model. Uh, it requires internet. But, uh, I have already pre-installed, uh, pre-downloaded the model, so we don't need that. Uh, so we're gonna see what model I have already downloaded using Foundry Cache Nest. So as you can see, I have downloaded, uh, four models here. So I want to, like, during our experiments, I want we might want to explore different models to see, um, the quality, to see the performance, then decide which model we want to use to build up our application,right? So firstly, um, I want to try, uh, Foundry Model Run Qwen 2.5, one 1.5 billion model. Since I have already downloaded this model, so the model loading is pretty quick. Uh, should be, uh, pretty quick. Okay, uh, the model is set up. You can talk to the model directly. So let's, uh, ask a simple question. What was ONNX Runtime? Oh, it's pretty quick,right? Um, so I think, uh, we might want to see the, um, latency level. So let's, uh, exit here and rerun it with, sorry, robust mode. And same question. Okay, so here we get around 90 tokens per second. Um, we also want to try a different model. So let's do that. Um, we want to try Foundry Model Run, uh, this Phi 4 mini. Also with robust mode. So it's loading the model. Okay, model is set up. Same question. What's ONNX Runtime? So Phi 4 mini is, um, more advanced than, like, Qwen model. The model size is, um, bigger than, uh, the Qwen model. So, uh, I would say in terms of the performance, it is a bit smaller than Qwen model. But, uh, in terms of the quality, as you can see, uh, Phi 4 mini, uh, can provide more detailed information. Uh, allright, so, uh, personally, uh, I vote, um, Phi 4 mini. So I want to use this model to build up, um, application. So what application do we want to build? Uh, I guess many of you have such experience. Your team moved to a new organization, needed to ramp up the existing project very quickly. And, uh, there are many long, detailed document, uh, documents, uh, you needed to read. ### App Demo **Emma Ning** [13:25] And it's very time-consuming to read every word,right? So you might want some high-level, summarized version of, uh, about this project so you can quickly ramp up. And, but this project is an internal project. You cannot upload all those documents to cloud. And meanwhile, you have, you know, some of your team members are using Windows, some of your team members are using Mac. So you want to build an application cross-platform powered by local AI. So let's do that. So I have this application setting up. Um, so what it does, um, so we can run it first. Oh, um, and let's quick the existing conversation. Allright, um, so the app is setting up. Basically, it is used to summarize content. Uh, you can give it, uh, your URL or you give the local file, so it can do summarization. And it also has a setting tab. You can choose the model you want to run. Uh, as mentioned before, I prefer Phi 4 mini, so because I want to get some, uh, more detailed information. So, uh, we can put, uh, this model ID here. Then, uh, we'll pass it with our project document. And let's. It to give me some high-level information. So I will hit summarization. So as you can see, the summarized version is coming out. And, uh, it says Foundry Local is useful to build up cross-platform AI applications that run directly on device. That's pretty cool. And, uh, then let's take a look at the code quickly. So as you can see, uh, uh, we in terms of SDK, we, um, provide, uh, Python SDK and a JavaScript, uh, JavaScript, uh, SDK. So here we use a JavaScript one. So we create the Foundry Local manager. And, uh, we and we initial this manager. Oh, uh, just a second. Where is it? Yeah. So we initial the manager with the model name. Uh, so as you can see, I passed Phi 4 mini here. And then, um, it used, uh, Foundry Local endpoint to create a client. And then you just wait, uh, for the check to, uh, be complete to be completed and output the result. So, um, this is the application, uh, running on Windows. Uh, I some of, uh, one team member is using Mac, so I want, um, he to use on my app as well. So I packaged the whole project and shared to him. And let's see what his experience is. He take the project and record a demo for me. So as you can see, it's exactly the same code. And, uh, he just, uh, used the same command, npm run start, to start this, um, application. And exactly UI, exactly application. And, oh, he choose Qwen model. Um, maybe he like this model more. And he also used the same document, uh, documents I used, uh, in my previous demo and hit summarize button. So, uh, as we, uh, see in the previous demo, uh, Qwen model is kind of, uh, provide more brief information, um, than Phi 4 mini model. So as you can see here, it also shows the summarization is more, like, shorter than what Phi 4 mini provides. ### Agent Preview **Emma Ning** [17:55] Okay, so we build up cross-platform applications. Um, is that all my demo? Of course not. We forgot one important thing, agent,right? So everybody talks about agent. So we so do I. So Foundry Local enables you to easily create and build and run a local agent using local model and MCP servers. Um, this feature is still in private preview, but I want to give you a sneak, uh, give you a quick, uh, look so you know, uh, how it works. So let's back to our, uh, CMD. So, uh, we can use, uh, use Foundry Agent Nest to show all the available agents, uh, in Foundry Local. As you can see, we have built up, uh, three sample agents here. So in terms of the concept, an agent in Foundry Local, uh, consists of one model and one more MCP servers based on your need. So you can use, um, one model from the list and pick up, uh, whatever MCP server you like to create your own agent. But here, uh, we want to run the existing one. So I'm interested in this OCR agent. So, uh, let's see, uh, Foundry Agent Info, uh, to know what it can do. Okay, so it can extract text from images in your local device. And this agent, uh, contains one, uh, model, which is Phi 4 mini, my favorite one, and two MCP servers. One is file system MCP server, one OCR mini MCP server. So let's run this, uh, agent, Foundry Agent Run. So this command, we'll, uh, check the dependencies of this agent first. If, um, the dependency hasn't been installed before, it will, uh, installed with your permission. So I have already installed all those, uh, dependencies. So, uh, it just run. Okay, the agent is setting up. Uh, it asks for permission to use this MCP server. So let's say yes. And he asks directory you want it to access. Uh, I give it, um, the demo photo. And he also asks permission for the OCR MCP server. I will say yes. So, allright, so it from here, you can get all the tools supported by this agent. So literally, the tools, um, provided in the MCP servers. So you can get some tools related to file system, tools related to OCR. So, uh, what we want it to do, so here is the use requirement. I want it to find my receipt, um, process it, and get the total amount from it. So let's see, uh, whether it can complete this task or not. So it starts to thinking because it needs to figure out which tool to use. Okay, uh, the first tool to use is search file because he need he it needs to find the receipt. And then, uh, he figured out to use the, um, so after search, it used the OCR one to, uh, extract the text and then get the output, get the total amount. Okay, that's cool. So that's all my demo. Um, so finally, I know it's a little bit over time, but I just quickly ramp up. So Foundry Local enables you to build up, um, applications powered by local AI. And, uh, one best practice. Um, so local model generally are not, uh, ca uh, not that capable, uh, as cloud model. So, um, you cannot expect it to do the fancy work that cloud model or cloud agent can do, but it has unlocked a lot of potential. So I leave that to you guys to explore. Uh, if you want to get more information, uh, here is the link. And you want to, uh, try out our agent, uh, feature, you can sign up our, you know, our private preview form. Allright, uh, thanks, everyone. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.