Foundry Local: Cutting-Edge AI experiences on device with ONNX Runtime/Olive — Emma Ning, Microsoft

Hello everybody. Uh, my name is Emma, I'm a program manage- uh, product manager at Microsoft. It's a pleasure to talk to you today about Foundry Local, which enables developers to easily build up cross-platform applications powered by local AI.

So let's get started. Uh, the first question is: if the cloud AI is so powerful, why do we need local AI?

So here are four key reasons, based on our conversations and observations with our customers. So first of all, how does cloud AI work in environments with low network bandwidth or even offline access? Many of you have experienced the bad internet during this conference,right?

And a common sentence we have heard from many speakers with live demos is, "Oh, fingers crossed, hopefully the Wi-Fi connection is good." So it's not their fault, uh, it's required by cloud AI. But it's not a concern at all for my session because my live demo runs entirely locally.

So reason number two: privacy and security. Many companies work with very sensitive data, uh, such as legal documents and patient information. They need to process that da- data, uh, entirely locally without anything ever leaving the device,right? And reason number three: cost efficiency.

Um, think about, uh, game applications, uh, which are deployed to millions of devices. With hundreds of millions of inference calls every day, it's not just sustainable. And reason number three: real-time latency. So many AI applications need to respond in real time, and it's not just, uh, not possible if we wait on cloud.

So that's why we need local AI. Then the next question: whether local AI is ready now. So thanks, uh, thanks to the decades of progress in computing, hardware has become more and more powerful. So many current devices are equipped with modern GPUs, NPUs, capable of running advanced AI models.

Meanwhile, model companies are keeping publishing more and more models, which are, like, leaner, faster, more optimized for local inference, such as, uh, Phi 4 mini and, uh, DeepSeek, uh, small variants. And we are also seeing more and more advanced, uh, state-of-the-art optimization techniques introduced at runtime level.

So this convergence now makes the local AI a reality. Then how do we build up the solution? So Microsoft already, uh, has many, uh, great assets. Uh, Azure AI Foundry, uh, introduced, um, last year at, uh, Microsoft Ignite, uh, has been, uh, trusted by over 70,000 organizations with over 1,900 models.

And, um, ONNX Runtime, our cross-platform high-performance on-device inference engine, has now seen over 10 million downloads per month. And our customers are pretty happy with the significant, uh, performance acceleration provided by ONNX Runtime compared to PyTorch. And lastly, let's not forget Windows.

The scale and the reach of Windows on client devices are massive. So when we think about, uh, democratizing AI, um, the millions of, um, you know, devices, uh, and the millions of customers using Windows devices really matter to us.

So we are not starting from scratch. We are bringing all those advanced assets into Foundry Local, an optimized end-to-end solution for seamless on-device AI. So at the bottom, as you can see, it uses ONNX Runtime to accelerate performance across various kinds of hardware.

On the top, uh, we are introducing a new Foundry Local management service, which hosts and manages models on your, uh, dev uh, client device. It also connects to Azure AI Foundry to download open-source models on demand. And for user experience, we provide Foundry Local CLI, which allows you to easily explore models on device.

And we also provide SDK so that developers can easily, uh, integrate Foundry Local into your own applications, um, from cloud to local from different hardware. So Foundry Local was officially announced just one month ago at a Microsoft Build conference.

It's available on both Windows and macOS. On Windows, it is integrated into the platform, which makes the experience even simpler for Windows AI developers.

As I just mentioned, uh, Foundry Local accelerates performance across different kinds of scenarios. We have been, uh, closely working with hardware vendors, including NVIDIA, Intel, AMD, Qualcomm, to integrate their hardware accelerators into Foundry Local, ensuring the best-in-class performance that you can get on their hardware.

So before our official announcement, uh, over 100 customers joined our private pre-preview. They have shared valuable feedback on how easy Foundry Local is to use and how good the performance it is. So let's hear some of their feedback.

Hey there. Sava here, CEO and co-founder at Pieces, where we've been on an ambitious journey to give developers artificial long-term memory across the OS. Now, offline-first AI is core to this vision. And in late 2022, we began to explore small language models running at the edge on all major platforms.

But between rolling our own Llama C++ and then to Ollama, frustrations around versioning, performance, and end-user experience still remained. That was until our recent partnership with Microsoft on their new Foundry Local platform, an end-to-end AI inference solution that offers ease of use and high performance across different hardware.

In no time, our team went from documentation access to a production-ready build with noticeable improvements in memory management, time-to-first-token, and tokens per second. If you're looking to deploy on-device models, you can't go wrong with Foundry Local.

We have been working on AI projects for our customers for several years now.

Some clients want to use the latest AI technologies but are restricted from using external services when the information they want to process contains sensitive data. Foundry Local is a perfect solution for these scenarios, as it allows us to easily run Gen AI models locally.

Here, we can see a solution that combines Foundry Local with a speech-to-text service.

Hello, my name is Maria Rodriguez. My savings account number is 123456.

Which also runs locally. One aspect we were really impressed by was the simplicity of the installation and the ease of using the models. With Foundry Local, we can now create hybrid solutions where part of the solution can be run locally.

It's been our privilege to work with these customers to improve Foundry Local together. Allright, I've talked enough. Who wants to see live demos? Okay, uh, let's do that. Um,

so first of all, we can, uh, let's see our CLI experience. So on Windows platform, you can, uh, install Foundry Local package using winget. And on macOS, uh, you can use Homebrew commands. Um, so I have already installed Foundry Local, so.

So first, uh, we want to see what models supported by Foundry Local. So we can type Foundry, um, model list.

So as you can see, it supports many popular, uh, generative AI models. And for each model, you can get, uh, you can get different variants optimized for different hardware. So, uh, you can see we have, uh, optimization version for CPU, for CUDA, for integrated GPU.

Uh, we also provide NPU variants because, um, my device doesn't, uh, contain Qualcomm NPU, so that variant doesn't show up. Okay, so we want to run some models,right? And, um, if you haven't downloaded the model, uh, before, the Foundry Local, we'll download the model from the cloud and then run the model.

Uh, it requires internet. But, uh, I have already pre-installed, uh, pre-downloaded the model, so we don't need that. Uh, so we're gonna see what model I have already downloaded using Foundry Cache Nest.

So as you can see, I have downloaded, uh, four models here. So I want to, like, during our experiments, I want we might want to explore different models to see, um, the quality, to see the performance, then decide which model we want to use to build up our application,right?

So firstly, um, I want to try, uh, Foundry

Model Run

Qwen 2.5, one 1.5 billion model.

Since I have already downloaded this model, so the model loading is pretty quick. Uh, should be, uh, pretty quick. Okay, uh, the model is set up. You can talk to the model directly. So let's, uh, ask a simple question.

What was ONNX Runtime?

Oh, it's pretty quick,right? Um, so I think, uh, we might want to see the, um, latency level. So let's, uh, exit here and rerun it with, sorry, robust mode. And same question.

Okay, so here we get around 90 tokens per second. Um, we also want to try a different model. So let's do that. Um, we want to try

Foundry Model Run, uh, this Phi 4 mini.

Also with robust mode.

So it's loading the model.

Okay, model is set up. Same question. What's ONNX Runtime? So Phi 4 mini is, um, more advanced than, like, Qwen model. The model size is, um, bigger than, uh, the Qwen model. So, uh, I would say in terms of the performance, it is a bit smaller than Qwen model.

But, uh, in terms of the quality, as you can see, uh, Phi 4 mini, uh, can provide more detailed information. Uh, allright, so, uh, personally, uh, I vote, um, Phi 4 mini. So I want to use this model to build up, um, application.

So what application do we want to build? Uh, I guess many of you have such experience. Your team moved to a new organization, needed to ramp up the existing project very quickly. And, uh, there are many long, detailed document, uh, documents, uh, you needed to read.

And it's very time-consuming to read every word,right? So you might want some high-level, summarized version of, uh, about this project so you can quickly ramp up. And, but this project is an internal project. You cannot upload all those documents to cloud.

And meanwhile, you have, you know, some of your team members are using Windows, some of your team members are using Mac. So you want to build an application cross-platform powered by local AI. So let's do that.

So I have this application setting up. Um, so what it does, um, so we can run it first.

Oh, um, and let's quick the existing conversation.

Allright, um, so the app is setting up. Basically, it is used to summarize content. Uh, you can give it, uh, your URL or you give the local file, so it can do summarization. And it also has a setting tab.

You can choose the model you want to run. Uh, as mentioned before, I prefer Phi 4 mini, so because I want to get some, uh, more detailed information. So, uh, we can put, uh, this model ID

here.

Then, uh, we'll pass it with our project document. And let's. It to give me some high-level information.

So I will hit summarization. So as you can see, the summarized version is coming out. And, uh, it says Foundry Local is useful to build up cross-platform AI applications that run directly on device. That's pretty cool. And, uh, then let's take a look at the code quickly.

So as you can see, uh, uh, we in terms of SDK, we, um, provide, uh, Python SDK and a JavaScript, uh, JavaScript, uh, SDK. So here we use a JavaScript one. So we create the Foundry Local manager. And, uh, we

and we initial this manager. Oh, uh, just a second. Where is it?

Yeah. So we initial the manager with the model name. Uh, so as you can see, I passed Phi 4 mini here. And then, um, it used, uh, Foundry Local endpoint to create a client. And then you just wait, uh, for the check to, uh, be complete to be completed and output the result.

So, um, this is the application, uh, running on Windows. Uh, I some of, uh, one team member is using Mac, so I want, um, he to use on my app as well. So I packaged the whole project and shared to him.

And let's see what his experience is.

He take the project and record a demo for me. So as you can see, it's exactly the same code. And, uh, he just, uh, used the same command, npm run start, to start this, um, application. And exactly UI, exactly application.

And, oh, he choose Qwen model. Um, maybe he like this model more. And he also used the same document, uh, documents I used, uh, in my previous demo and hit summarize button.

So, uh, as we, uh, see in the previous demo, uh, Qwen model is kind of, uh, provide more brief information, um, than Phi 4 mini model. So as you can see here, it also shows the summarization is more, like, shorter than what Phi 4 mini provides.

Okay, so we build up cross-platform applications. Um, is that all my demo? Of course not. We forgot one important thing, agent,right? So everybody talks about agent. So we so do I. So Foundry Local enables you to easily create and build and run a local agent using local model and MCP servers.

Um, this feature is still in private preview, but I want to give you a sneak, uh, give you a quick, uh, look so you know, uh, how it works. So let's back to our, uh, CMD. So, uh, we can use, uh, use Foundry Agent Nest

to show all the available agents, uh, in Foundry Local. As you can see, we have built up, uh, three sample agents here. So in terms of the concept, an agent in Foundry Local, uh, consists of one model and one more MCP servers based on your need.

So you can use, um, one model from the list and pick up, uh, whatever MCP server you like to create your own agent. But here, uh, we want to run the existing one. So I'm interested in this OCR agent.

So, uh, let's see, uh, Foundry Agent Info, uh, to know what it can do.

Okay, so it can extract text from images in your local device. And this agent, uh, contains one, uh, model, which is Phi 4 mini, my favorite one, and two MCP servers. One is file system MCP server, one OCR mini MCP server.

So let's run this, uh, agent, Foundry Agent Run.

So this command, we'll, uh, check the dependencies of this agent first. If, um, the dependency hasn't been installed before, it will, uh, installed with your permission. So I have already installed all those, uh, dependencies. So, uh, it just run.

Okay, the agent is setting up. Uh, it asks for permission to use this MCP server. So let's say yes. And he asks directory you want it to access. Uh, I give it, um, the demo photo. And he also asks permission for the OCR MCP server.

I will say yes.

So, allright, so it from here, you can get all the tools supported by this agent. So literally, the tools, um, provided in the MCP servers. So you can get some tools related to file system, tools related to OCR.

So, uh, what we want it to do, so here

is the use requirement. I want it to find my receipt, um, process it, and get the total amount from it. So let's see, uh, whether it can complete this task or not. So it starts to thinking because it needs to figure out which tool to use.

Okay, uh, the first tool to use is search file because he need he it needs to find the receipt. And then, uh, he figured out to use the, um, so after search, it used the OCR one to, uh, extract the text and then get the output, get the total amount.

Okay, that's cool. So that's all my demo. Um, so finally, I know it's a little bit over time, but I just quickly ramp up. So Foundry Local enables you to build up, um, applications powered by local AI. And, uh, one best practice.

Um, so local model generally are not, uh, ca uh, not that capable, uh, as cloud model. So, um, you cannot expect it to do the fancy work that cloud model or cloud agent can do, but it has unlocked a lot of potential.

So I leave that to you guys to explore. Uh, if you want to get more information, uh, here is the link. And you want to, uh, try out our agent, uh, feature, you can sign up our, you know, our private preview form.

Allright, uh, thanks, everyone.