Decoding Mistral AI's Large Language Models: Devendra Chaplot

Uh, hi everyone. I'm very excited to be here. I'm very, um, happy that there is an open models track. So I'm going to talk about the open models of Mistral AI and go a little bit deeper into why we do open source and how we do open source.

So, first of all, Mistral AI, we started last June, about 1 year ago. We released our first open model, Mistral 7B, in September '23. And then after that, in December, we released our first mixture-of-experts open model, 8x7B. And along with that, we released our platform with model APIs and also commercial models, Mistral Medium and Mistral Embed.

And then earlier this year, in February, we released Mistral Large, which is our flagship model, which has the best in-class reasoning and

math ability. And also, in April, we released a new open model, 8x22B. And then very recently, in June, we released a code-specific model called Coastral 22B. And it's also available in the chat interface that we built along with Mistral Large, and it's free to use.

So our mission is to bring frontier AI in everyone's hands, and we specifically focus on building cutting-edge AI for developers. And we have certain principles behind how we go about training models and releasing them. So the first is openness.

We want to train best-in-class open models and release it for the open source community. We want our models to be portable. All our models are available on Azure, AWS, GCP, Virtual Private Cloud, and also they can be deployed on-prem, which means you can license the model weights and use it on your own servers with full control over security and privacy of your data.

We try to optimize for the performance-to-speed ratio. Our models are particularly good at getting the best performance out of a particular size. And we want our models to be customizable. We are building our platform with all the libraries and tools to customize our models depending on your application.

We recently released the Mistral Fine-Tune open source library, which can be used to fine-tune any of our open source models. And also, we have a fine-tuning API on our platform. And before that, we also released Mistral Inference, which is the inference library, again, open source.

So I talked about these three models that we have open sourced in the last 1 year. The first model is a dense transformer model. It was the first model, first 7B model, to achieve 60 on MMLU. And we saw that the 60 MMLU is like a bare minimum where the models become useful.

And this was the first 7B model to achieve this. And people

have been using this model for many, many different applications. And particularly, we have seen that this model can be deployed on laptops and phones and still get reasonable speed on-device. We released our first sparse mixture-of-experts model in December, 8x7B.

It's based on the mixture-of-experts architecture, which basically allows us to push the performance of a model while keeping the inference budget in check. The idea here is we have a higher number of total parameters in the model, which allows the model to still have the knowledge stored in the model weights, but at the same time, we use only a small subset of the parameters for every token, which makes it really fast and cost-efficient at inference time.

And then we released a bigger version of this sparse mixture-of-experts architecture, 8x22B, in April. It has even better performance, higher context window, and also it's multilingual. It supports English, French, Italian, German, Spanish, and also many other languages.

So a lot of people ask me, if you open source your models, how do you make money? And I think this is a common misconception that people have, that open source is somewhat competitive with profit. That's actually not the case.

We see open source as

something that goes hand in hand with profit. It doesn't necessarily have to be competitive; it can be complementary. And we want to be in this quadrant where we can open source our models and still have long-term business value with the models.

So why do we open source? So the first reason is it serves as a very good branding and marketing tool for us. So we believe in open source and open science, and we want to contribute to the community.

But it's not a one-way thing. We are also benefiting from open source, just as the community is benefiting from our models. So it helps us doing a lot of branding and marketing. A lot of people like our models.

They tell other people that our models are good. The model performance speaks for itself. We do not have a marketing team in-house. And just the open sourcing the models allows us to create awareness about our products. It also helps us in customer acquisition.

If people try out our open source models and they really like it, they come to us for an upgrade to proprietary models, and they pay for the upgrade. And it also helps in customization and portability.

Whenever, for example, the 7B model, people can try

it, to try to run it on laptops and phones. And this is the kind of stuff we benefit from because we don't necessarily have to do this out of the box, but the community works around our models, and we learn from the community how our models can be customized or deployed in new settings.

So how are these open source models trained? So I'll give you a very high-level overview of the different stages of LLM training. And typically, LLMs are trained in three stages: pre-training, instruction tuning, and learning from human feedback. So the idea behind pre-training is very simple.

You take a piece of text and you pass word by word or token by token through the large language model and ask the model to predict the next token.

So the idea itself is very simple. The task is the next token prediction. Each token is roughly 0.75 word. The vocabulary size is roughly tens of thousands of tokens, or sometimes hundreds of thousands. And each token is basically represented as an integer, and it has an embedding associated with it.

And so the task of the model is to take in a sequence of embeddings or tokens and predict the next token.

Although the concept is very simple, in practice, it's actually very hard. Why is it hard? Because it requires a lot of effort in building the data sets. The data sets are huge. They are order of trillions of tokens, tens of trillions of tokens.

It requires pre-processing, cleaning, deduplication, curation. And this is, again, a common belief that more data leads to better performance, but that's not necessarily the case. If you have noise in your data, that can actually hurt the model performance.

It also requires a lot of investment. These models are huge. It can go up to hundreds of billions or even trillions of parameters. Each model takes tens to hundreds of millions of dollars to train. And the hardest part is you don't get multiple chances to train the model

because it's so expensive. If something goes wrong in your training, it's very difficult to get the investment to do another training run. Because typically, for small companies, you don't get that kind of budget. If you do a model run and it's not successful, it becomes harder to get the funding for the next run.

And this is hard because the best hyperparameters for a smaller model might not be the best for a larger model.

Here I'm showing you some hyperparameters for LLaMA 1 model family sizes. And you might ask,

why are the number of layers 80 and not 82 in LLaMA 65B? And the answer is we don't know.

There's a lot of things that have been decided by intuition, and it's not exact science. So you need a lot of experience and intuition working with these models to come up with things that are very likely to work.

But we

still are not very mature with the science of what is the best way to train the model or what's the best architecture, what's the best data set mixture. So can we use this pre-trained model? So let's say if you want to use this pre-trained model and ask it to write a Python function to find whether the input number is prime or not.

And the model might give you a response like this, continues the text, gives an example, and describes the approach, but it might not give you the code. And this is because the model is trained to do this. It's trained to predict the next token.

So it predicts the most likely token from the text data it's been trained on. But there is a way to trick the model if you give this input, like as a Python function definition and a docstring, to get the same function.

The model actually produces the code. And so this shows you that the model actually knows the answer, but it is not aligned with human preferences. It's not trained to interact with humans in the way humans want to. And this is why we need the next two stages.

So in the instruction tuning stage, instead of just a string of text, we have prompt-response pairs. So here we are giving the prompt, but in the way humans want to interact with the model. So for example, this prompt to write a Python function.

And the response is directly the code because that's what humans want as the response. And the technique is very simple. Again, we are doing next token prediction, but the only difference is we are going to mask the prompt itself.

We are going to do prediction only for the response.

So the data set is paired prompt-response pairs. We typically use hundreds of thousands of instructions. The task is next word prediction, but just we mask the input instruction. It requires way less compute. Order of 100 GPUs for a few hours or days is typically sufficient to do instruction fine-tuning.

And then the last step is learning from human feedback. And here the idea is

that human preferences are cheaper or easier to obtain than full human annotation. If I give you a prompt like this and two responses, it's much easier for a human to decide which response is better than to write the whole response from scratch.

And so this allows us to scale data faster. And there are two main techniques: reinforcement learning from human feedback and direct preference optimization, where we use this kind of preference data to fine-tune the model further. So just to summarize, these are the three stages.

They have different orders of data set and compute requirement, and the task is slightly different. And all the open source models we have have been trained using these techniques. And so I won't go into the details of

the model architecture itself, but I'll show you this nice graph of performance to cost ratio. This kind of shows that we really try to optimize this metric. We try to get the best performance out of our models of a particular size.

So here on the x-axis, we have the active parameters, which is directly proportional to the cost of running to the model. And on the y-axis, we have a popular benchmark, MMLU. So we try to be in the top left corner to get more performance with a lower cost.

We recently released the CodeStrel model, CodeStrel 2020 Rearrangement Habitat Challenges. It's a dense transformer model trained specifically for code. And again, we are trying to optimize performance and speed. It's fluent in 80-plus programming languages, and it has both instruct and fill-in-the-middle mode, which means that you can use it for code completion in your code editor, just like GitHub Copilot.

But also, you can use it to ask questions about the bugs or errors you are facing, just like you would put it in ChatGPT.

So it outperforms Code LLaMA 70B, DeepSeek Code 33B, LLaMA 370B, while being a significantly smaller model. So again, we are getting more performance out of a model of a particular size. And it also has a longer context window than the other open source code models.

It is multilingual. We trained it with more than 80 programming languages. And

across all these different languages, it tends to perform better than the other models. So it's free to use on our chat interface, chat.mistral.ai. We also have the API access available on La Plateforme, which is our

platform API endpoint. And here, it's also free to use till, I believe, end of July. We also have integration with VS Code and JetBrains. So you can download a plug-in in VS Code or JetBrains and use it as a coding assistant for code completion.

So

in the end, I would just discuss some practical tips because these are some commonly asked questions about how to use open source models and when to use open source versus when to use commercial models. So if you have a particular application in mind and you want to try out commercial models, you could do things like prompt engineering, few-shot prompting, chain of thought.

And you could also do retrieval augmented generation because commercial models typically don't allow you to do fine-tuning. But for open models, you can do task-specific fine-tuning as well. You need a little bit of data and compute for this.

But in the end, the choice is between how do you balance performance versus cost. Commercial models have a higher general purpose performance. So they are much easier to get started with if you are trying to build a new application.

But once you get into production or once you have high volume, open models can beat commercial models on specific tasks with fine-tuning. And

typically, what we have seen is people prototype with the highest-end models. And then once they figure out that this is the task they want to solve, they take an open source model like Mistral 7B or 8x7B and then fine-tune it for their task.

And this optimizes the performance to cost ratio.

We have offices in Paris, London, and in Marriott. We are always looking for talented researchers, engineers, business, marketing people. So please do apply. And thank you. I don't know if you're taking questions, but happy no. OK. Thank you so much.