Llamafile: bringing AI to the masses with fast CPU inference: Stephen Hood and Justine Tunney

So we are here to talk to you about Llamafile today and what we've been doing on this project.

So I'll get us started. I'm going to tell you what Llamafile is, how it works, and I'm going to spend a little time talking about why we're building it, why Mozilla specifically is involved, and then I'm going to hand it over for the fun part to Justine.

Justine is going to talk about the actual work that she and the open source community have been doing on this project. Lots of insights and tricks and hacks that have made CPU inference go faster than ever before, so that'll be fun.

When we're done, we want you to share the feeling that we have, which is kind of a sense of excitement and empowerment, from the knowledge that there are lots of really interesting, juicy, impactful problems still left to be solved in AI.

A lot of them. And the key thing is, it's not just the big folks who can solve these problems. It's individuals and small groups working together in open source. So anyone in this room, or anyone listening to this talk, can potentially make a big impact in this space.

So what's Llamafile? Llamafile is an open source project from Mozilla that has the goal of democratizing access to AI. So we do that in a few different ways. The first is probably how, if you've heard of Llamafile, the reason you heard of it.

It's the original magic trick of the project that Justine figured out, which is how to turn weights into programs. So Llamafile is a single file, executable, that runs without any installation on pretty much every operating system, every CPU architecture, and every GPU architecture.

And that's all. Thank you very much.

That was easy. Yeah, so by the way, this isn't just one file, like for Windows,right, and a different one for Linux and Mac. It's actually a single file. You can download a Llamafile, run it on any computer in the world, and it'll just work, and it'll use the hardware you have, whether that be fancy GPUs or your CPU.

So we'll talk a little more later about how Justine made that work. But we're here to talk about another topic too. Most of the talk is actually about this, which is CPU inference speed. Now you might ask, why do we need to worry about CPU inference speed?

We've got these fancy GPUs,right? Well, no disrespect, almighty Jensen, first of his name, master of market cap. Don't strike me down. But I would posit that it is not a universally good thing that we are so dependent in this room on GPUs.

They are expensive. They're difficult to source. Let's face it, they consume a lot of electricity, which we might want to think about. But bigger picture, we have an entire planet of CPUs out there, literally all over the world, great hardware, often affordable hardware.

And we are at risk of just kind of throwing that all away with this new era of AI. And we don't need to do that. So who here knows Llama CPP? This is an easy question,right? So we all know and love this project.

We build on top of that project with Llamafile, and we contribute our performance enhancements back to it. Many have been merged in. That project proved that CPUs could do inference perfectly well. And so we have been basically trying to take that performance to the next level.

And as a result of Justine and the community's work, depending on what CPU you're using, what model you're running, what weights, you will see between 30% and 500% speed increases with Llamafile, which kind of still blows my mind.

And by the way, I don't think we're anywhere near done.

So these things also run locally, by the way. This runs totally on your machine. There's no network access. You could take a pair of scissors and cut the Ethernet cord, and it'll work, which is what I asked DALL·E 3 to draw.

OK. I don't think it understood the assignment, but that's allright. But seriously, we're not calling cloud LLMs. There's no monitoring or analytics. No bits leave your machine. It's totally private and local. And everything you need comes in the box.

So whether you want to just play with a model that you just found on Hugging Face, or you want to start building locally running LLM applications on your machine, you've got everything you need in the box. And they're readily available.

So Hugging Face now supports Llamafile as a file type. So you can search and filter by Llamafile. You can also just search Mozilla on Hugging Face. You'll find we have a bunch of Llamafiles that we've already published. And with a single command, you can create your own.

So really, this project is collapsing all the complexity of the open source AI stack down into a single action and a single file.

So why are we involved? Why is Mozilla involved in this? You might be saying, don't you folks make browsers? In fact, we do. We make a damn fine browser, and you should try it out if you haven't lately.

But we exist also for a bigger purpose, which is to fight for the web. So I'm going to ask a question here. Who here remembers using the original Netscape Navigator? Don't be shy. No one can see how old you are.

They can only see how old I am. A lot of hands,right? So you are my people. You remember the '90s. Must see TV. Terrible haircuts. Nilly, vanilly. I don't know. Whatever. My point is, you remember the early days of the web.

And you remember how close we came to one company and one product kind of controlling the whole thing. And we kind of see that maybe happening again today with AI. No matter what we may think of these companies, the reality is there are some very influential big tech companies that are in a position to maybe control the future of machine intelligence.

And that's itself not a great thing. It's not great for equity. It's not great, especially for users' sense of privacy and safety and agency and control. And we've had an answer to this for many years. It's called open source.

And the answer isright in the name,right? Open source. Transparency is the solution here. And it's important for us to have viable open source alternatives in AI. And that's why Mozilla is getting involved. That's why we made Llamafile and more projects to follow.

And I know many of you in this room are already working on open source AI. We want to help support what you're doing. So I'm going to hand it over to Justine, who's going to tell you actually the cool part, which is all the things that she and the community have been doing on this project.

Justine.

Thank you, Stephen. So I'm Justine Tunney. I'm the lead developer on Llamafile. And as Stephen mentioned, I'm going to talk about some of the cool work we've been doing in the community to help you run the fastest local M experience possible.

And in order to do this, we started by first getting it to run on the systems at all. And with Cosmopolitan, what it enables us to do is take your weights in a single file and run it on six OSes.

And there's a really cool hack that makes that possible, which is we basically take a Unix 6th edition shell script, put it in the MS-DOS stub of a portable executable, and that enables it to run on Mac, Windows, and BSDs, and Linux, et cetera.

Really cool stuff. And once we conquered the portability issue with CPUs, I had the opportunity to work with Mozilla on bringing this to AI. And with AI, GPUs are indispensable. As much as we focus on CPUs, we care very much about GPUs too.

But GPUs have always had the problem of distributability. Many people have needed to ship Kubelet binaries with their project, 500 megs in size. Can we really call our software open source if it spends the majority of its time in a proprietary blob?

So I never felt comfortable with that. And one of the ways we're solving that is by distributing a library called Tinyblast that enables you to ship your LLMs to platforms like Windows without depending on SDKs. It'll run with only the driver installed.

But more importantly, performance. Now, LLMs spend the majority of their time doing matrix multiplication. Probably the most important algorithm in the world has a really simple definition. We've been making it go faster for prompt processing. And the way we did it is with a very simple trick we figured out.

And this is something all programmers can adopt in their code. And it entails unrolling the outer loop. So let's talk about what not to do first. And that would be unrolling the inner one. We've all seen fun roll loops, Gen 2.

It's a bad idea. Computers can generally do that on their own. If you unroll the outer loops, then your algorithm with matrix multiplication can sort of unfold like a flower and focus on pure flops like a blast kernel.

And that's really all there is to it, to getting the majority of the benefits of blast, to make prompt processing go really fast. So what's the impact of this really simple solution?

This generalizes to a wide variety of hardware. We've seen everything from a scrappy hobbyist Raspberry Pi to much bigger computers going significantly faster. You need algorithms like this to exploit the latest capabilities of hardware. Token generation race, I wouldn't believe.

If you use a gaming computer like Intel, you're going to see better performance with Llamafile on those two. Really exciting stuff, like particularly with Alder Lake, we were able to get a 4x improvement. But Threadripper, most of all, for the first time, AVX512 is available to consumers.

And we've been able to help you prepare for that future. So if you have a Threadripper, you're going to see better performance than ever, almost like a GPU. Now, prompt eval speed, what makes it important is

it's really cool to be able to generate text and use a chatbot. But the way I want you to think about Llamafile is it's more of a word-crunching machine that can help you understand our world. And I love to use it personally for tasks like summarization.

I love that it can help me read a blog post. And we've used other performance tricks too.

With NVIDIA, part of what makes them so successful, it's not just great hardware, but they built a great framework too. And their framework helps developers think about programming in a different way that helps them be successful. I mean, who here thinks that software with CPUs just gets slower each year?

Can I see some hands?

Well,

one of the things that's great about NVIDIA is they showed us a better alternative to getting performance. And when I learned how to program in CUDA, I found one of the most important functions was sync threads. This is how you can implement it for CPU in like 10 lines of code.

And if you use the lockset programming model, use your CPU as though it were a GPU, you can get really good performance.

Now, this is going to be a demo showing the impact of this work before and after for summarization. And here we're going to be processing an essay by Dijkstra. Really cool, worth reading. But I want you to watch as it processes it in terms of speed.

Here we see it going. And on theright, we have the new version. It's like bam, bam, bam, bam. Huge night and day difference. It's already summarizing it. And the old version is like nowhere close.

So that is the kind of new performance you can expect. And it's the kind of performance that's actually possible, which I wouldn't have imagined beforehand. It's really great.

Thank you. CPUs can do so much. And people in the community have loved this work. We've managed to attract some really amazing contributors, like Eyewen, the inventor of KQuant, very popular. I'm sure many of you have used them.

He got them going 2x, 4x faster too on both x86 and ARM. So if you use quantized formats, those are going to be better than ever with Llamafile now too. And it's worth mentioning that we've seen really interesting things about this.

Like once we put this out into the world, people have come back and given us feedback and reported their own experiences. We found out that someone was running Mixtral 8x22B on a $350 CPU. And to me, that's just wonderful because performance matters, but it's not really the thing we care about.

What we care about is intelligence. And to have the intelligence, you need to run bigger models. And RAM is cheap with CPUs. For the price of a graphics card, I put 512 gigs in my workstation. And that means I can run all the frontier models coming out.

And I just have to wait a little longer, but I get a much more intelligent answer. And the fact that that went from impossible to possible for most consumers is a story I want you all to tell.

Individuals are making a big difference. And you can be a part of that too. And I'm going to hand it back to Stephen, who can explain what Mozilla can do to support you getting involved in that effort.

Thanks, Justine.

Thanks a lot for all your efforts. So yeah, that's a key message of this talk is anyone in this audience, you don't have to work for these big, giant, largest in the history of humanity companies necessarily to have a big impact.

There's lots of headroom here. There's lots of unsolved, interesting problems in this space. And we want to get involved in helping. So we recently launched a program called Mozilla Builders. And this is a program by which we either sponsor or, in some cases, co-develop impactful open source AI projects.

Llamafile is actually the first in this program. I'm happy to now say the second, which is SQLite Vec. This is from a developer named Alex Garcia. Alex is adding vector search capability to SQLite. And so for some folks in this audience, that'll have some obvious implications that are kind of cool.

But just imagine, remember that little modest Raspberry Pi 5? So imagine now a local LLM, open LLM, running privately on that machine with no network connection, connected to your personal private data, which you can use with confidence that it's safe to do RAG and other interesting applications.

That's the kind of stuff we're talking about here. We also just launched our own accelerator. It's called the Mozilla Builders Accelerator. So we are offering $100,000 US in non-dilutive funding for open source projects that advance the promise and potential of local AI.

So that's AI applications running at the edge on user devices. These are some of the bullet points and areas we're particularly interested in, but it's not an exclusive list. And you don't have to necessarily be building a company to apply for this accelerator.

So if you want to learn more about the accelerator, this QR code will take you there. Take a picture of that. Or just go to future.mozilla.org/builders. And Justine and I and a lot of Mozillans are here this week.

If you have something you're working on or something you think we should know about or you want to collaborate with us, please find us, reach out, or reach out to me via email. So thanks again. Thanks to Justine and the community for all their work on Llamafile.

Thank you, Stephen.

Thank you, Justine.