From Mixture of Experts to Mixture of Agents with Super Fast Inference - Daniel Kim & Daria Soboleva

This is the API key, so if you haven't scanned this already, please scan it. It will take you to our cloud where you can sign up for a free API key. This will be the only thing you kind of need to do the workshop today.

Everyone got a picture? Everyone good? Cool.

OK, hello everyone. We are very excited to see you here. Today we organized a very fun workshop for all of you. We're going to first start with an explanation of what Mixture of Experts is and why this architecture matters.

And then we're going to use agents to create something very similar to it, but it's called Mixture of Agents, so you replace your experts with real agents. There will be a code alone session, so everyone will build their own up at the end.

And for the agenda, first of all, we're going to introduce the concept. What is the Mixture of Experts? Why does this architecture exist? And what kind of models are using it? And this is basically like the way for us to continue improving the models that we build.

So like ChatGPT, think about that model. How do we scale them further so they become smarter, more intelligent? And there are two ways to do it. So one is to pre-train from scratch with Mixture of Experts architecture. Another way is building a Mixture of Agents, so when you combine already pre-trained models together in one complex architecture.

So we're going to build exactly approach number two and a hands-on workshop. And at the end we will have Q&A. So if you guys are interested about these models, what we do with these models at Cerebras, or you have any other questions, we're going to answer that.

Yeah, a little bit about me. So I'm a Head Research Scientist at Cerebras. I worked there for almost 4.5 years. I am specifically focused at researching Mixture of Experts architectures and just other architectures that help us improve the LLMs and make the training more hardware efficient.

In the past, I spent some time working on data scaling. So I created a data set called SlimPajama. That data set was the largest, the best in quality data set when it was released. And prior to Cerebras, I was at Google working on research and engineering projects.

Perfect.

Hi, my name is Daniel. I'm the Head of Growth here at Cerebras. I do both developer activations, developer marketing, but I also do startup sales. So if you need tokens and you're trying to run your startup and you want to use Cerebras in production after the workshop, come talk to me.

I am the token arbiter for Cerebras for everyone that is not enterprise. And if you have a cool research project you want to use Cerebras for, I am also the person to talk to. So if you want tokens for your project, talk to me.

I also really like Hot Pot. I had Hot Pot three times in the last week. So yeah, that's a little bit about me. I'm based in San Francisco and I do a lot of these types of workshops and events and things like that.

And here's my Twitter if you guys want to follow me. And this is our intern, Kevin. He's not here with us because he's taking his last high school final. He's in high school. But I wanted to put him here because he's the person that built 99% of the workshop that you're going to be doing here today.

And I want to make sure we shouted him out. Wait, I want to take a selfie with him. I want to show him because he doesn't know I'm doing this. OK. And yeah, he's going to school in UCLA coming in the fall, but he's currently in high school.

So yeah, and that's his Twitter if you guys want to follow him. Great. Before we get started, who here has heard of Cerebras? OK, pretty good. Last time I asked this question like eight months ago, there was like two hands in a room full of like this.

So it's so much progress that we've made in the last couple of months. So I will dive a little bit deeper into what Cerebras and why our hardware is so much superior than our competitors in the later part of our presentation.

But quickly, Cerebras is a hardware company that makes custom silicon that runs AI models super duper fast. And here's a side-by-side of the chips comparing us to an NVIDIA H100. So there's just a sizable difference here. And things that make our hardware architecture super dominant, kind of like all boils down to the innovations we've made in the hardware itself and how we're able to linearly scale with larger models, which I'll get into later.

But currently, we hold the world record in every single model we host publicly. And it's not even close. So for Llama 3.3-70B, we're around 15.5 times faster than the fastest inference provider on a GPU. So if you want kind of like something that doesn't compare to anything that's currently in the market, Cerebras is kind of your only option for fast inference.

So that's what our company does. And this is what I'm bringing to startups. So if you're a startup that wants this inference, you should come talk to me after the talk. So what are we doing today?

Yeah, so actually, spoiler alert, we're going to use Cerebras hardware today for our workshop. So you guys can actually try it. But before, I want to explain what we're actually building. So we're going to build an application with a mixture of agents.

Each agent will be a separate LLM. And kind of likeright now, I want to explain why this is beneficial, why we want to build that, and why this architecture is better compared to like a monolithic one LLM that we useright now.

Thank you. Yeah, so sort of like from the pre-training perspective, how do we make larger models more intelligent, better? How do we scale them faster? So all of these types of questions, we ask at Cerebras. When we have our hardware, we can scale models pretty fast.

But like how do we make them more efficient? What kind of architectures do we need to invest in? So I kind of wanted to give sort of like the evolution that happened in the LLM space for you guys.

So we started with the GPT-3 that was released a few years ago. And the model there was quite small. And basically what GPT-3 paper showed is that if you continue scaling the model size, you're going to improve the performance.

Your models will have better skill sets. That's how you're going to scale it. The next thing that we saw in the LLM evolution is you actually have to spend a lot of time improving your data that you pre-train on.

So Llama model became bigger, but it also spent a lot of time on curating the data set and scaling the number of tokens that you train for as well. And now you guys probably heard about DeepSeek V3 that was released a few months ago.

That model took some additional innovations into place. So if you want to go even larger, you see that GPT-3 is 13 billion, Llama 3 is 400 billion, and DeepSeek V3 is 600 billion. So if you want to continue scaling the model size, which is what gives us better models, we need to come up with not just data set improvements, but also architecture improvements.

So how do we actually improve the models and thus like serve the large models? Because as you increase the number of parameters, you have to come up with a way to scale it, to scale your inference infrastructure and make it more efficient.

The answer here is Mixture of Experts. And this type of models, like to just give you like an overview how it works, imagine that you have a transformer architecture, which is what we use as a backbone for large language models.

It has different types of layers. So here I highlighted that there are more layers there, but like one important layer, some important layers are embedding attention at feedforward layers. They all have different types of purposes in the network.

And now we're going to see like how we change the standard transformer architecture into something called Mixture of Experts. So if you look at different layers and you do some interpretability work, you will figure out that feedforward network has a specific bottleneck.

It has a challenge because feedforward network sort of like has to disentangle all the information that previous layers process, like attention layer. So you can think about it this way. Feedforward network has to decide which neuron in the network to activate when it sees a golden gate bridge in a text.

So it's really hard because the tasks that we have for LLMs, sometimes they have different languages. Sometimes they require different specialization. They could be like math domain, biology, et cetera. So feedforward network has the hardest job in the whole.

So how does Mixture of Experts solve this bottleneck? Instead of having one monolithic feedforward network, we will create separate feedforward networks and we will call them experts, as you can see on theright side of the screen. Each expert will be specialized in a specific task.

So you can think about it this way. One expert can be solving math problems. Another expert can be a biology teacher,right? And the other thing that is crucial here, this type of architecture allows us to increase parameters of the model.

You create multiple copies of the feedforward network, but you don't have to activate all of them for every token you route through the network. You can see that there is an additional network called router within our network, which basically decides which expert to select for a particular token.

So you can click next. Yeah. And that allows us to actually increase the width of the model, increase the capacity of the model, and scale the parameters because we know that from parameters you are getting better skills without increasing the inference time.

So you can actually activate the same number of parameters as for the monolithic model. So you will might map like in terms of the time, but you will be better in quality because you trained a larger model.

Yeah, so sort of like to close on this, this is the approach that other companies are using. Like this is the industry standardright now. OpenAI, GPT-4 models, Anthropic, Claude, all of them are using this way, this approach to scale the models and to gain better skills for their models.

So yeah, MoE models are really, really cool. And they are kind of like becoming the industry standard for being able to run large, large parameter models in an efficient way. So you don't have to continuously just add hardware to be able to run better and better quality models.

But there are some other approaches that also work. So something I want to talk a little bit about is inference time compute. Last year, I believe, Ilya Siskover gave a talk at NURPS around how we're in the age of inference time compute, where we have just thrown as much data as possible when we pre-train these really, really large models.

And eventually, we're going to hit a data wall,right, where we have no more additional unique data to train our models with. So now what we can do is do more compute after the model has been trained to be able to get better and better results and more and more intelligent models.

So an example of problems that benchmarks test for are math problems. So I want to first take a math problem from the AME Math Competition and see how certain models kind of tackle this kind of problem. And the thing with these types of math problems is that it's harder for like a single non-reasoning model to be able to solve these because they require multiple steps and sequential thought, where it's really hard to do things like this without reasoning.

And when I ran this exact problem through GPT-4o, which is not a reasoning model, it took 45 seconds to come to the wrong answer. And this is like the frontier model. But ChatGPT-03, sorry, GPT-03, which is a reasoning model, took 293 seconds to come up with theright answer.

So this is it on theright, doing everything correctly. But it just took 293 seconds. So if you want something in a reasonable, within three business days kind of timeline, this is like probably not the solution for you. Like imagine you're like scrolling through an app and it just is loading for six minutes or three minutes or four minutes and 53 seconds.

That's just like an unreasonable amount of time for a lot of tasks. So I want to introduce something called Mixture of Agents, which is leveraging the collective intelligence of multiple LLMs to come to theright answer. And I think like because Cerebras is a super fast inference provider, I'm sure everyone can see where this is going.

So a Mixture of Agents is basically the ability to take advantage of these earth-shattering speeds from our hardware and apply them into harder problems like this. So it's not just about speed. It's about getting higher intelligence with less smart models.

And Mixture of Agents is a lot of it is inspired by Mixture of Experts architecture because essentially you're trying to do the same thing. You're trying to squeeze out as much intelligence in an efficient way from a lot of kind of tokens, whether it's within the model or outside of the model.

So basically how it works is that you send inputs to multiple LLMs with custom system prompts like agents. And then each model gives its own response. And then basically a final model combines all of the answers from all the individual models into a single answer.

And this has shown that it outperforms even frontier models on certain benchmarks as benchmarked by Together AI, who's the ones that kind of came up with this idea and this term. So I want to show you an example of a startup that's actually building in production with Cerebras in this Mixture of Agents model.

There's a YouTube video. Let's see. Worst case, I can just show the Google Drive link. Sorry, the internet, as y'all know, is not great. So I hope this loads. If not, wait, I think I have it locally saved.

One second. Oh yeah, I do have it locally saved. Perfect. So this is ninja-tech.ai. And you can try this product in productionright now. And they're basically building a smarter chatbot. And this is Ninja Tech solving the same exact question in 7.4 seconds and getting the answer correct.

So people are using this type of technique and our inference together in production to solve really hard questions like this math problem. Cool.

OK, hopefully.

Please, please tech gods. OK, cool. So basically what this startup did was take a bunch of models and a bunch of LLM calls and get theright answer that a frontier model, reasoning model, took 293 seconds all the way down to seven seconds.

And here's how their whole application works. So they have a planning agent that spits out eight potential proposals for theright answer. And then another token, a critique agent, comes in and be like, hey, are these any of them feasible answers to this particular problem that I'm trying to solve?

In this case, all of them were bad. So then what happens is that the planning agent goes back to the drawing board and then spits out 16K context worth of thinking tokens of eight answer proposals. And then the same virtuous cycle happens where the critique agents like, are any of them good?

In this case, in the example I showed before, two of them were potential answer candidates that could have been the correct answer. And then another agent, a summarization agent, takes those two top answers and then turns them into the final answer, which is eventually theright answer.

So this whole process, even though it took seven seconds, took over 500,000 tokens to be generated and 32 LLM calls, some of them in parallel, some of them sequential. So this type of system allows you to take advantage of non-frontier models, even open source models that may not perform as well on benchmarks, and turn them into performing better than frontier models.

So yeah, like I said again, why don't people use O3 in production? It's because it's very slow. Like I can't even think about like use cases where you can wait five minutes to come up with an answer unless it's like very asynchronous.

And Cerebras solves the speed bottleneck. And obviously, Cerebras is the leader in fast inference. So I want to show you why we are so fast. So this is an architecture diagram of a GPU. And highlighted in red is the core or the thing that does all the mathematical computations that allow LLMs to predict the next token.

In this particular GPU, which is the H100, there are around 17,000 cores on this chip. The problem is that the memory where all the weights and all the intermediate calculations and all the other information needed to produce the next token is stored largely off the chip.

And these memory channels that communicate between the cores and the external memory become bottlenecks as you run larger and larger models because, of course, you have to transfer in more weights and you have to transfer in more intermediate calculations in the KV cache while you're trying to calculate the next token.

Cerebras tackles this by having a radically different memory management system. We have 900,000 individual cores on one chip. And then with those 900,000 cores, we have 900,000 individual memory stores that are distributed all across the chip one to one with our compute cores.

And each core has direct access to memory. The core-specific memory holds the same set of weights regardless of what you're putting into the system. So basically, you don't need to wait for the external weights or the intermediate calculations to load to be able to do the computations.

We can just do it in real time because there's no kind of memory transfer time that you need to wait for. Everything is just on the chip. And Cerebras scales linearly across larger models. The thing that makes Cerebras really fast with super large models is that the only piece of data that's being transferred from chip to chip is activations.

That's the only piece of data that travels. It can even be like transferred using a single Ethernet cord, that amount of data. It's very small. With a DGX cluster with multiple GPUs, they have to transfer so many activations and cache computations in between layers because single GPUs cannot do multiple layers of computations via hundreds of NVLink connectors, switches, et cetera.

And that networking piece is the reason why we're so dominant compared to NVIDIA GPUs. So Daria, how does this all translate?

Sure. Yeah, I probably want to start with like a problem that I have and see how many people have the same problem. So when I interact with a monolithic model with just one model, I usually ask to solve a particular problem.

And then it doesn't get theright solutionright away usually if the problem is complex,right? So I have to continue prompting it and refining it and at some point heating the number of tokens that the model can process or something like that and start over the chat from scratch.

Does anyone else have the same problem? OK, cool. So I'm not alone. Here, what we do with Mixture of Agents, we're going to specialize each agent to solve a particular portion of the problem. So imagine you have a very complex problem.

I don't know, you need to do a surgery, let's say,right? Imagine that. So we need different types of people to help with the surgery. And we're going to ask each expert to specialize in one specific part of the surgery so they all together can work and produce like a better result compared to just one person who can do a surgery,right?

So we're going to do it through prompt engineering. And the one nice thing about it is instead of doing multiple rounds, multiple iterations to find the best solution at the end, you're going to find a solution at the zero shot, at the one shot.

You're going to ask one question. And because each kind of like agent is already specialized in solving a particular portion of the task, they will combine the result together. And it's going to be the final solution without continuous prompting.

So this is what we're going to build today.

So this is the time for the hands-on workshop. So before we move on, this is like the second time I'm showing this. Everyone needs an API key. This is like the one thing you need to do for the workshop.

So everyone, get an API key. OK, cool. And then please go to this GitHub link and then star and fork the repo. That's the next step.

Kevin created it, Kevin. Not Daniel, Kevin.

But I don't have HD shots of Kevin going like this. So that's me.

Yeah.

This is also in our Slack channel if you don't want a QR code scan. So if you are in the AI Engineer Slack channel, feel free to go to.

Yeah, we also dropped the slides there. And you guys can interact there if you want, help each other out.

It's called MoA?

MoA workshop.

That works.

Does anyone have issues finding that? Let us know.

Yeah. And once you guys star and fork the repo, we're going to deploy this app via Streamlit. Or you can run it locally, whatever you prefer. I just don't want to deal with Python installation issues. So that's why I suggest if you have the internet bandwidth to go through Streamlit.

But if not, doing it locally via Python is fine.

And just quickly, Daniel is going through the slides. But if you want to come back to some of the steps, we shared it in a Slack channel. So you can open slides 50 to 54. They will have the same instructions.

What's the Slack channel?

MoA, Mixture of Agents.

Hash.

Dash workshop.

Yeah, so once you star and fork the repo, you basically are going to deploy it via Streamlit or locally. And basically how you deploy it on Streamlit is you go to streamlit.io. I would suggest logging in with GitHub.

That way, everything is set up for you.

And then deploy the cerebras.moa workshop that you have forked into your GitHub repo, changing the main file path, and then clicking Advanced Settings. And once you click Advanced Settings, you just plug in your Cerebras API key here. And that's how you run the app.

So I'll wait like two, let's say, three minutes for everyone to go and clone and spin up their app. And then we can go from there. Raise your hand if you need help from me or Daria. We can help you get set up if you have any questions.

OK, let me set a timer.

If you guys want to have access to the presentation, here is the QR code to our Slack channel called MoA workshop. And if you want steps on how to set up everything with a Streamlit, you can go to the slide 50 to 54.

I already shared it in a chat there.

Yeah?

I think Kevin committed the API key. Do we just use that one?

Kevin committed the API key. God damn it.

Thank you so much for bringing that up.

It's OK. His API key is rate limited. It's not like he has like the god API key. But it's OK. He's an intern,right? Yeah, feel free to use your own though, please. That would be great because that will get rate limited.

If everyone uses it, it's going to like obviously get rate limited.

Daniel, I think.

I'm glad that my boss is not here.

Yeah.

The secret just stays between you, us, and the internet, you know, because apparently this getting in.

It's being recorded.

No, I know. I know. I'm going to rotate itright now.

Perfect.

Daniel, I think we are getting the prize for the funniest workshop ever,right?

Wait, really?

Yeah, probably,right? API key is committed.

I kind of want to win though. So like if everyone wants to vote for us as like the most popular slash funnest workshop, you know, I wouldn't say no.

Huh?

Oh, it's a bribery thing. OK, I see. I see. There are strings attached to this vote. I see. I see.

Yeah, let us know if you have any trouble spinning up the workshop and then using definitely your own API key.

OK.

Do we have a slide how the workshop page, front page should look like?

Yeah.

I think I can use some help.

Wait, what?

Sorry?

I can use some help.

Oh, OK.

You need some help? OK.

So I clone the repo. What do I do next?

Oh, no. It's in the README. It's basically you have to install like all the requirements file. And then you run one command. It's very simple. We tried to make it as simple as possible.

Thank you, timer.

So, oh yeah, when you have it all running, it should look like something like this. I would say very sexy UI, you know? So we wanted you to really experience what MoA systems look like because I feel like this is a very new concept.

And it's a great way this kind of app that Kevin built is a great way for you to get started with what the possibilities are with MoA systems maybe in production. So basically how this UI works is that in the top, you configure your summarization agent, basically the thing that summarizes all the things and actually like aggregates all the individual results from your individual agents into one.

We have a very basic system prompt here. You can change it if you want. And then under agent management, we have individual agents that you can create with custom prompts. And here you can adjust things like the temperature, the model, as well as the specific prompt that you want to put in the agent and also rename it to something fun.

And here you can delete agents or you can create new ones. So you can have more than three. And you can also have different layers. You can have multiple iterations of how many times you want them to go through the solution.

And basically, the first part is not competitive. It's just going to be us all having fun,right? So I can ask a question here like plan a trip to San Francisco for me and my friend Daria

from 3:00 p.m. to 9:00 p.m. and then see what happens. And then what it's doing is behind the scenes, it's basically spawning all of the individual agents and then going through multiple layers of calculations for it to return the final answer.

Here we have three layers with multiple LLM calls. So it might take a little bit of time.

It's just the Wi-Fi.

Oh, it's probably the Wi-Fi actually.

Can you zoom in?

Oh, yeah.

And if anyone doesn't see a screen like this, could you guys let me know? And I can come and help you with any steps.

Well, the Wi-Fi is really bad.

Does anyone not have the screen?

And then the other one is going to.

You don't?

Yeah. So I'm setting the cloud and in regards to setting the secrets. So Python will hold on. So that's where I'm stuck at in regards to confusing here.

OK, maybe I can just like walk you through what we seeright now on the screen. There are multiple layers that we created. Each layer is basically a set of models. And they are connected together. Layers are sequential. But at each layer, models are processed in parallel.

So once we get the output from layer one, it will be combined together as an input for layer two. And then after the layer three, there is this final model that we call summarization model that will process the final output combining the results from previous layers.

So all of these tokens were generated, sacrificed for the final answer. So all of these tokens basically allow you to cover more surface area and have a more comprehensive answer. So all of this was generated through all the other models.

Yes?

So each layer is an agent?

So each layer has a set of agents that operate together. Their output is finalized after that layer. So you can think about it. It's concatenated and used as an input to the next layer.

Was everyone able to ask a very fun personal question to the MoA chat? Anyone struggling with that? Happy to help.

I'll be close.

OK, great. So now comes the actual competitive fun part of the workshop. This is all fun and games until comes the competition. So then go and select the AI configuration challenge. And basically, your job is to come up with the perfect MoA system to generate OK, perfect code is a little extra.

Maybe it's like really good code, really good code.

We believe in you.

So instead of writing code yourself, you'll become an AI prompt engineer and system architect. And your goal is to configure your AI agent, like Mixture of Agents system, to automatically generate code that scores the maximum 120 points in our automated grader.

And basically, to participate, you just have to go into Configure AI and Generate. So here you can select. So I can actually just do it with you for the first one. So here it has some very awesome preset responses.

So let me see. You can like set some awesome presets that are in turn created. And then we can generate the code.

Hopefully, the Wi-Fi is good. OK, the code has been generated. So here the code has been generated. And we can submit the solution. And here the baseline is already designed with the prompt, the preset prompts to be like C.

And your job is to be an A student because we're all overachievers in this room. So basically, you can now change things like the prompt itself or the model. So let's like set it to the Qwen model and then see what happens.

So we can generate the code again

and then submit the solution for grading. And here it scored slightly better. So we went from a C to a B. It's a solid B with literally just by changing the model, like the model that it uses for the main agent.

So your job for the rest of this workshop is to figure out first, what is theright combination of the main model, the number of cycles, like how many layers you want, the temperature, as well as the system prompt for the main prompt, and as well as the individual layers

that will spawn for every single iteration. So here you can select different models, different particular prompts, and then use that to come up with theright answer. And I believe Daria and our intern both got perfect scores. So you're shooting for perfect scores for your Mixture of Agents system.

Do we have a function somewhere that we have given them?

Wait.

Like the actual Python function that has bugs in it?

Oh, no. That gets inputted through here. So you can just like change this.

But how do they where do they find the function?

Oh, they don't. They kind of optimize it. Oh, it'sright here. Default challenge.

A default challenge. Where is the function itself, like the Python function?

No, it's the function. This is the function requirement.

Where is the implementation of the function that we give them as a baseline?

Oh, we don't. We changed it. So the challenge is like generates the Python function from scratch.

OK, OK, gotcha. OK, so this is important to say. Like let's come back to that and set up the challenge,right? Because we have engineers who know how to write Python. So the challenge is here we want to create a function that's called calculateUserMatrix.

And the purpose is basically to calculate the metrics,right? We have some details here what the function is supposed to output given an input.

Yeah. So this is the test that is used when you click grading. This is like the input to your AI agent system. This is the input that will go in. And then basically.

OK, I'm going to send them the baseline function in the chat.

OK, yeah, feel free.

So imagine this. Like you have an interview, let's say a tech company,right? And at the interview, they can ask you to optimize a function. For us, today's Python function. This Python function has some bugs. And it's also not optimized.

So what we want from you is using LLMs basically find the solution that fixes all the bugs and optimizes it to a level that you get the maximum score. We created the grader that basically will use like LLM generated function as an input.

And it will give you a score. And an idea is to get 120 out of 120. You can see we already created some specific agents to help you solve this task in one shot. One agent is working with bugs.

It basically tries to find all the bugs and edge cases. One agent is working with.

Performance.

Yeah, performance. So it's going to optimize it. You don't like recompute some stuff. And the last one is what is it?

Just like overall.

Oh, just overall, yeah. So like the final model, the way it can work, you can ask it to look at the outputs from like three or whatever number of agents you created at the previous layer and kind of create a final function using the inputs from different agents.

And I'm going to send you the Python function in the Slack channel. So if you guys see some errors yourself, like engineers,right, you can actually configure the prompts this way. You can ask one agent to like, oh, I see that there is an empty list that we try to like access by index,right?

Could you please fix that? And that agent will work on that specific problem. Or you can also see what the default config outputs and figure out like what are the remaining issues in the Python function. So it's sort of like vibe coding but with your brains,right?

Isn't that vibe coding? Don't you vibe code with your brain?

I've seen multiple of these.

Do you turn off your brain when you vibe code? It's just like you really let the computer take over, huh?

Oh, this should be default. Like this is literally all I did was spun up the stream limit. Like.

And there will be a prize for whoever gets the highest score, actually. Have I thought about what the prize will be? Not yet. So I'll take suggestions from the winner.

Sorry, I don't know how to respond to that. I'm like erroring outright now.

Not the lowest score. That's so funny.

It's actually quite good. If they can get zero, it means they can get maximum two.

Oh, really?

Yeah. If you get like zero, let us know.

Wait, to get zero, can't you just edit the thing to be like just return nothing? Wouldn't that be zero? Or it'd be like write a Rust function.

Try it.

And the grader will be like, it's Rust.

Don't give them ideas.

Oh, OK, OK, sorry.

But yeah, if someone is really interested in getting maximum score and you need hints, try it yourself though. But like if you need hints, we are happy to give you some.

Oh, yeah, we can do hints. I guess.

Hits the max score or zero, please let us know.

Not the zero, Daria.

No, I want to see the person who gets zero.

Not the zero, Daria. Not the zero.

Sorry, what did you say?

Like is it top score? What is the existing top score?

No, it's the perfect score. There was like three people that got it.

It's on.

OK, everyone. OK, can you raise your hand if you got the 120 perfect score?

OK, everyone, congratulations. Wow, we got a hella winners. OK,

after the Q&A, please come up to us so I can get your email. And then I can get you something fun. I don't know what it will be. So please, when you come up to me, come with suggestions of what you want.

But like let's keep it realistic. No Teslas, no like large gifts like that, OK? So something fun.

Unlimited input.

Huh?

Oh, I don't know about that one. OK, before we keep you all keep going because I feel like a lot of people got the challenge. So great job, you guys. I want to do a quick Q&A with me and Daria.

So if you don't mind, if you can come up with one of those, if you have questions for us about anything Cerebras related or Mixture of Agents or Mixture of Experts, come ask us questionsright now, please. Wait, can you come up to the mic?

I'm so sorry for you to do this. But apparently, like that's how we get content.

Do you guys got like some benchmark?

Can you come to the mic? I'm so sorry. You have to do cardio.

Yeah, if you guys can like hit the mic.

How do you auto ML this stuff?

Auto ML this stuff?

Yeah, OK, so I mean it's all fun and all to go manually and figure out the prompts. What if the prompt's slightly harder where it would take you 100 hours or 1,000 hours to figure out the prompts? And you said, I want to throw a solution to solve this.

That's my question.

I mean, I feel like in the.

It's just going to happen,right?

I mean, it's already happening. If you look at like on Devin, for example,right, like the code gen startup, you ask it to do something. And like a couple hours later, it comes back to you with like a proposed solution,right?

And then tackling beyond just like fix a snippet of code. It's just building whole new systems or whole new applications. So that kind of thing is already happening. It's all about like how can Cerebras as a company that builds custom hardware, how can we enable those people?

So instead of taking hours, it takes minutes. So I don't think it's the technology that's not there yet. It's about like how do we make it usable in the current landscape that doesn't take that's not so painful. So I think it's like actually, it's already happening.

Like to apply AI towards these really hard multi-hour problems, this is kind of a small scale simulation of what you could do with a lot faster inference to speed that up.

What regions is your hardware currently running? And do you have plans for global distribution?

Yeah, soright now we opened, I think, six data centers in the US in the last year. And we are opening one in France this year. And then we currently have plans for one in Canada as well. And we only expect that to go more global as the time goes on.

And one more. How long does it take you to onboard a new model when a new model is released or a new version of a model?

That's a great question. I think it depends on the model. So the blocker for us to onboard a new model is if we have all the kernels that are written to make sure that it supports the new model.

And in some cases, like Qwen32B that you all have access to, that took very little time because that architecture was very similar to Llama architecture in terms of the kernels needed to run it. And we had all the available kernels.

So all it took was all a bunch of QAing and like implementing API level features to get that to work on our system. On the other hand, there are other models that are extremely hard because we don't have theright kernels ready yet to support that model in an efficient way.

So that takes more time because the kernel engineering team needs to write the custom kernels needed for that model. So it really depends on the model architecture.

Thanks.

I've got a question about power consumption. So this sort of new architecture is going to take a shorter amount of time. But what about power consumption? Like are we talking about similar to what Nvidia is doing? Or is it more?

Is it less?

Yeah, that's a great question. And I think it's not a one-to-one,right? Because we have a just completely different chip architecture. So it really depends on the workload. But we've observed and this is what we put in our website that it's around a third of the power consumption of Nvidia GPUs for the equivalent workload.

It's just that our chips are a lot more massive. So it has a lot more throughput and a lot more like it can just take in a lot and generate more. And it takes a significant number of chips, Nvidia chips, to match like what one system can do.

So it's not an apples to apples. But like I think we are a lot more power efficient in most use cases.

That's a great question.

It is. I do get that question also. So it's a very commonly asked question, I think. Everyone's like, how does this big chip take in energy? OK, what's up?

My question is about mixture of agents when I compare with, I don't know, like the SOTA models, you know, like benchmarks wise, where I should put it?

I think it's like about the configuration of the MoE agent. And I think that's where things get a little bit tricky. It's like if you have really shitty prompts for your MoE system, it's going to perform shittily,right? So it's about like it's like tuning all of the prompts in your MoE system.

And then it will perform better in the benchmark. So you actually do need to put in it's not like an out of the box thing,right? Because it's not training. It's like optimizing the system for your use case. So you need to make sure that you put in the engineering work needed to optimize all of the whole system, whether it's like using theright models or is it writing all of the combination of the things needs to work out for it to be better.

The whole point, though, is that it can be better. It's just that you have to like actually like engineer it to be better.

One thing like from the theory perspective, if you have like if you have an idea of like what's the Ansible learning case where you create like multiple models that communicate between each other and like together, Ansible basically provides a more robust, a better solution than just one model.

So this is inspired by that, by like decision trees. Or like when we created like standard ML models, we always get like more less, I guess, like memorized solution, more generalized from multiple models. So in theory, even if you configure each model the same way, create the same prompt, the final answer can be better than just one model.

Yeah, yeah, like actually boots gotch my mind when I saw that kind of approach,right? Like you have multiple like a swarm of decisions. And then you can get like

I guess what I'm trying to get at is maybe there is a trade off when you have like too many trees or agents. Like maybe like if the question that you need is too off.

Can it like create worse solution if you create too many agents?

I guess if there is a danger of being more like a homogeneous answer if you have too many agents because maybe you have like a single agent that gets itright. But you have like 2,000 agents that get it wrong.

Yeah, this is a very good question. And so for like Mixture of Experts, when we create different experts, we also think about it this way. Like how many experts do you want to have? So then they're all kind of like used in the network.

So here, what's likely going to happen, if you create too many agents, not all of them are going to be used. So you will create like redundancy in the network. And you will just spend more time like getting the output from like agents here.

And they're not going to be used in the final solution, if that makes sense.

Yeah, yeah, I feel like feature importance in actually boots,right?

Yes, yes, yes.

OK, cool. I got it. Thank you.

Yeah, hi. Do you all support like bringing your own fine-tuned model? Like let's say if it's fine-tuned on like Qwen32B itself, which you already support?

We do for enterprise clients. I know that's not the answer anyone wants to hear. Butright now, we are working on supporting LoRA fine-tuned models. That's in the roadmap. But it's not currently supported. But if you're an enterprise customer that's looking to onboard custom models, we do have a number of customers running fine-tuned models in our cloud.

Hello, yes, a short question. Did you already tried diffusion text generation models? Like they're 10 times faster than simple LLMs. And they may have super different architecture and how it fits your approaches and course.

I'm curious, are you working on diffusion models yourself?

No, not now. We're mostly trying to use it, onboard it from pragmatic perspective. And they're already really fast. And probably with your approach, it will be much faster, like 10 times.

Yeah, so from like diffusion models, they're definitely like one of the architecture to consider after Mixture of Experts and transformer-based architectures. Their field like is exploring different types of models. There are state-based models. You know, all of them have different like improvements on top of the existing transformer decoder.

I would sayright now, they're still in sort of like research. So in our like inference API, we kind of try to put models that are proven to be the best. And they're like robust. For diffusion models, I think we still try to scale like from the research perspective, trying to figure out what's the best recipe to train them.

But I know that this is a very interesting research direction that labs are looking at.

I did see an internal demo, though, that was insane around diffusion models. So like diffusion models on Cerebras hardware is going to be insane when it comes out. That's not helpful to anyone. But you know, I thought I'd throw that in there.

I can't even talk about it because the guy was like, don't talk about it now. But I'm like but I'm being very vague about it, you know.

So I have a question related to what the previous gentleman asked about fine-tuning model. So say, for example, I have a new architecture, which has got certain different layers which are not there on not supported by Cerebras or does not have those kernels defined.

What would happen for those kind of situations?

Yeah, that's a great question. I think one of our customers, Mistral, is a great example of this where they brought in a custom architecture model. And they were like, hey, we want to run this on Cerebras. And basically, what happens is like a partnership and a collaboration between their engineers and our engineers to make sure that all of the kernels are in place to support their new architecture.

So it's very simple. It's just like all our hardware is, if you think about it, is a bunch of memory and a bunch of compute that are organized in a very efficient way in a very large surface area.

That's all it is. So we can technically, in theory, support like a very diverse set of models and architectures that may not even exist yet. So it's all about like creating that partnership and figuring out like how do we support the models that we don't yet support.

So that partnership is more about defining those kernels so that the architecture is better?

There's also other quirks about models. Like if there is a custom like RL or something, you know, like it's all about like making sure everything is supported that you want to run in your model.

Thank you.

OK, OK, last call for questions.

Yeah. So I've got a little light. But do you think you'll ever do real-time APIs and sort of those other multimodal models as well?

So we actually support we released our first multimodal API, not available publicly, but through the Mistral app. So now, if you use Mistral, the chat, some of the image-based queries are running on hardware. So next comes our cloud.

So once we have it running in one place, we assume we're going to scale it into our public cloud. So that will be coming pretty soon, actually, to our Cerebras cloud for multimodal. Around real-time, we are actually thinking about this.

So like I would love to learn more about your use case. So yeah, real-time is definitely a very interesting, I think, idea for the company because of the speed of our inference. I think it'll be great in like real-time use cases.

I think we have time for more questions.

Yeah, we do. We have like so much time.

40, 40 minutes left.

Yeah, maybe not all 45 minutes, though. But you know. What's up?

So I'm curious. Has there been models engineered especially for Cerebras hardware? As in like, you know, you have new types of capabilities. And I'm curious, how good are people at exploiting those capabilities,right? Because people are taking existing models.

And yes, it's easy to port them if you have the kernels. But what about stuff that you really would encourage people to try?

Wow, this is a very good question. So it really depends on what your use case is. If you are a researcher who wants to try a new architecture and train it, I would say we have specific advantages for unstructured sparsity algorithms.

So you can try that. I don't think it's at the speed at any other hardware. So if you try that on GPU, it's going to be hard. That's for pre-training side. For inference side.

I think for inference side, not yet because we released inference like nine months ago. So it's a very, very new product. And as we are working with frontier model companies like Mistral, we're planning on working with them even more closely to design models specifically to take advantage of not only our current generation of chips, but our future generations of chips.

We'll have even more optimizations.

But maybe from the inference perspective, if you have a very large model you want to serve, then Cerebras is the best position to do that. It's going to scale multiple chips. You're going to use multiple chips. And you won't have to like distribute the model weights and wait for all this orchestration to work together.

So I would say like the best use case here is like if you have a very large model, then use Cerebras inference.

So I have two questions. First is, you know, like what kind of model sizes can be used simultaneously on a Cerebras instance?

On one instance or like in a?

Like when I'm using it now, I don't know how many

VMs are currently.

So it's not a VM. It's like the number of systems. But basically, there's no limit to the size of the system. Because what we can do is we can just infinitely add more chips. Like so we can go from like as small as like an 8 billion parameter model.

And we can go all the way up to like Maverick is a recent model that we were onboarded. We're planning on supporting the bigger meta models, you know. There's no limit because we can just scale linearly. Our networking architecture is actually very simple.

But there is a cost associated with it.

Of course.

So I wouldn't be able to partition a single SoC or one Cerebras instance into.

Oh, we can't. We don't offer thatright now in our cloud.

No partition.

No, we basically handle it. We handle the load. It's a name. We currently only offer the service unless it's like an on-prem client with just an API. So we give you rate limits that you can hit. And we in the back end provision the number of systems needed to match your workload.

For the public API that everyone used today, that's a shared pool where we set rate limits for each user. And then they can consume until that rate limit.

Great question.

Awesome. If there are no more questions, me and Daria will be up here. If you have more questions, come up to us. Or if you're a startup looking for inference, I'm the guy to talk to. And if you also got 120, please bring you getting 120.

Like get the proof. And then I'll get your email. And give me ideas for prizes. And then we can go from there. Thanks so much for coming, everyone.

Thank you.