20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna

Hello. So today we're going to try to ask— to, to ask us the question: what AI model is state of the art? And I guess this is a super important question for everyone, because of course for our applications in research or when we deploy a product, we want always to have the best performance out of our models.

But the problem is that state of the art is a bit a confusing concept, and people maybe have different vision on this. So we'll just try to see a bit, like, first what— what— how people approach this question, and how they try to answer this question.

And usually there are two main methods that people try to, to use to know what model is state of the— state of the art. The first one is simply, you know, they go on the internet and check some public leaderboards, see what is the best model on the public leaderboard.

And another method is actually just perform some internal evaluation, and again see, based on their internal evaluation, like, what model is the best. The problem with these methods is, like, in most cases, if you apply them naively, you will always find, like, a kind of lazy solution, which is just to, to use a large foundation— foundation model.

So we're going to just try to see a bit with these methods, like, what people try to— tend to do, like, a bit quick and can— can be done better. So the first method, again, is, like, just simply checking a public leaderboard.

So for example, if we take the use case of, like, image editing, let's try to find the best image editing model in this case. So usually the first step is just find some leaderboard. In this case you can use, like, Design Arena, which is a very famous one.

And then you just pick the top one, which is ChatGPT image, and then you feel that you are happy, this is the best model for your use case. In general it's, like, kind of a good solution, like, you get, like, a first reasonable— reasonable model at low effort.

But the problem is that you don't know exactly a lot of things about how the users will interact with your model and so on, so you can actually make a much better choice. So the first problem is that if you look at many leaderboards, but not a single one, you will see that each public leaderboard will have a different ranking.

So here maybe it's a bit small, but you can trust me, there are three leaderboards, LM Arena, now called Arena, Design Arena, Artificial Analysis, and they try to rank image editing models. And if you try to draw a bit the difference between these leaderboards, you will see that it's not the same ranking.

Like, the top model is not the same. Also, like, relatively, if you compare model two between each other, they will be different. For example, there is one model, Hunyuan, that goes from rank 10 on Artificial Analysis and is ranked 5 on, like, Arena.

So the idea is that each leaderboard had a different perspective, and sometimes there are also some models that have duplicate entries, so it's a bit noise, a bit what is the information you want to get out— get out of it.

There are even some models that appear in the leaderboards and are not in some others, so it's hard to get the main information. And when you check, actually, details like the ELO scores, which is supposed to be the quality scores you use to know what is the best model, you will see that even these ELO scores, they are very different, meaning that for some area— for some leaderboards it will be between 1100 to 1300, but for some it will be a completely different range.

So relatively, models, we don't know how strong they are between each other. And usually the main solution for this is not to trust a single one, but really to look at multiple ones. And when you see that there is a lot of difference between, like, different rankings, it means that probably there are, like, some models which are approximately equivalent.

It's not because ChatGPT image is ranked top 1 on one— on one leaderboard that it means that it's the best overall. Another problem is, in most cases, you will have applic— you will release and use your model for specific application.

What we've seen before is, like, some aggregated score over a lot of different tasks. I don't know, for example, we can see, like, removing objects, changing background, editing text. But we can actually build some leaderboards for each of these specific use cases.

And this is also some leaderboards from, like, Design Arena, I guess. And I think there is a problem with the— okay, it comes back. And you can see that actually, if you draw the difference for each specific use case, you will see that again the ranking, they are completely different.

And ChatGPT image is never top 1 in this— in this ranking. There are always some new model and some models which are super good at removing objects, some models which are super good at doing some other things. The idea is there is no model constantly outperforming the others, and there are very different models working well for different target use cases.

And this is normal because this is just due to the fact that, you know, some models, they have been, you know, for example, trained more on some specific tasks than others. And the solution for this is when you check, like, public leaderboards, you should always try to target what your use case will do in the end.

Like, if you focus on removing objects, look at this leaderboard and not the others. Another problem is that usually leaderboards, they are not really statistically significant for your specific use case. So here I try to show two— two different things.

So the first thing is on how many samples these leaderboards are built. And if you check on the left, for example, Artificial Analysis, this— all these— these rankings, they are built on, you know, a few thousand samples for each of them.

So it's not much. If you compare to the load of inference you have for many applications, it's probably super low. For some of our models that we have, we have millions of runs per day, so probably we'll get more information by just, like, just training the model on our API rather than just looking at this— at this leaderboard.

Another thing is ELO scores. Usually you can also compute what is the win rate of each model. So when you build these rankings, what you— what you do is you actually make models battle against each other and ask people, okay, what is the best— what is the best model between the two to a lot of users.

And what you can see is actually the win rate, usually there are no models which are close to 100% win rate. It means that most of the models, they lose at least 40% of their— of their battles. And if your use case is in this 40% of the battles, it means that you will just— if you take the best models, you will just take the wrong model.

So again, here it's important really to evaluate on more samples and always, like, have, like, evaluation which is close to the final setup, the final use case conditions. Now we can check also the second solution to— to try to know what is the state of the art for AI model.

And the second solution is to do just internal benchmark.

One way to do it is what I see the most, actually in image and video generation research and so on, people just do manual inspection. They try a couple of prompts, a couple of models, and they get a feeling intuitively of it, what is the best model.

Another thing that sometimes people do is they just, like, run some benchmark, automated benchmark out there, and then try to see, okay, based on this benchmark, what is the one that has the best performance. So there, yes, basically then you just select the preferred model.

So it can be, I don't know, for example, the third model or the one with the highest score. The problem is that— so there are a couple of problems with this, and we can start this with a little game.

So here I'm just going to show, like, three images. And maybe one question for you is, how many people in the room prefer the first— prefers the first image among these three?

What's the requirement?

So this is a question, like, in general, you can ask a lot of questions. So does it adhere to the prompt? Is it— what image do you prefer? And so on. But the prompt was, I think, a little guy and a parrot or something like this.

And yeah.

I like the first one.

Okay, first. Who prefers the second image? Okay, a couple of people. Who prefers the third image? Okay. So what is great here that we have seen that people have different preference. So it's important to see that if you do, like, manual inspection, you will be super biased to your own preference.

So it's very important to not trust only your preference because then you have big surprise that actually it's not the models that are preferred by everyone. Now we can do it again. Same question. Who prefers the first image in this case?

I think the prompt was, like, probably a man eating some soup with pasta or something like this. Okay.

Who prefers the second image? Okay, great. And who prefers the third?

Mango.

Okay. So that's also super interesting because I've seen some people changing their mind. So always on the left it was the Seedree model, middle FluxOne, and on theright, like, some models we developed one image. And the idea is, like, also you are super biased toward the few samples that you look at.

So when you do manual inspections, you are two times biased by you and by also the number of samples, the specific samples you look at. So in general, the idea is, like, you should never only trust the

manual inspection. It's good to get a feeling, but it's not enough. You should always ask many people to do it. And human evaluation is usually great, but you have to scale it properly.

Another problem is that when you do not human evaluation, but more, like, proper, like, automated evaluation with metrics, sometimes you have, like, non-consistent results. So for example, this is a bit small, but you can trust me, we ranked, like, eight models regarding some metrics, like, a very standard metric, which is called CLIP score.

And sometimes people, when they try to evaluate image models, they check this metric first. And you can see, actually, that if you check, like, the rankings for the three metrics we looked, like CLIP score on different datasets, it changed all the time.

And these metrics are supposed to be between 0 and 1 or 0 and 100. And actually the variations between models, they are super small. So it means— it means that it's hard to know from this metric what is the best model.

What you should do is actually first having some clear understanding of what the metric does and also use multiple— multiple of them. So here, for example, this is another type of metric. When you know your— you know your use case, for example, you know to— you know you want to be the best at text rendering, there are a lot of text rendering metrics that would be better to evaluate your models.

So here you can see again, like, the ranking is way more consistent. You have always Z-image being the first and P-image being the second model. And also the variations, they are way more significant. So the models are supposed to be 0, the metrics are supposed to be between 0 and 1, and there are, like, clear difference between, like, every model.

So yes, in general, very important. Understand your metrics. People usually tend to just use some metrics and see, okay, I did my benchmark and then I stop here. But it's important to understand what you actually measured with this.

And now a last problem, which is actually common to the first and second method to that we've seen before, is that usually quality is driven by compute. So here this is ChatGPT image, and for their evaluation, Design Arena, I think, or maybe it's LM Arena, they did, like, 27— 26k battles.

So it means they generated 26k images. And each of these images takes one minute to generate. So here I summarize all this information. 62 seconds, 26k evaluations. And in total, to do these 26k evaluations, it takes 20 days of compute.

In terms of cost, it's 5k just— 5k just to run this evaluation. And in terms of energy, it's approximately, you know, 556 kilowatt— kilowatt hour. So I know that people might not have the order of magnitude of what it represents, this amount of energy.

So just to give, like, some idea, I checked my Strava and checked how much energy I was consuming by running a marathon. And actually it represents 400 marathon just to generate all these images. So it's a lot. I'm tired after one marathon, so I don't want to do 400 for sure.

Now there are some alternatives. You can use some different models. So of course this is a model that we've done that does, like, time generation— editing of images in less than one second. And for the same amount of evaluation, it takes only 7 hours.

It takes also, like, way less— it uses also way less money, so $265. And instead of running 500 marathons, I just need to run four marathons. So if three of you want to run a marathon with me, it will be enough to do this.

So again, the idea is, like, people tend to just look at quality, but it's important not to look only at quality but also at efficiency because sometimes the additional gain you get with quality is not worth the efficiency, the compute cost.

So to the question, what model is state of the art? The answer is there are multiple state-of-the-art models. And the tool I prefer for this is usually the Pareto plots, where basically on the x-axis you have one efficiency metric.

For example, on the left it's latency for the generation of one image. On theright it's the price for the generation of one image. On the y-axis you have some quality score, let's say the ELO score. And here you can draw the Pareto front in red.

And you can see that there is not one single state-of-the-art model, but there are actually multiple of them, and there are, like, three or four. And you can see that even though the quality score is not— there is no big variation, it's always between 1,100 and 1,200, there is a big difference in terms of efficiency.

So you can be, like, really, like, times— times, I don't know, 20 times faster just by using the different model.

Even better, if you know the specific task you want to do, you can do, like, the Pareto front not with a quality metric which focuses on general capability, but really based on quality metrics which is for the target use case.

So this is some Pareto front focusing on text rendering. And here, for example, we optimized a lot, like, the Flux2 model, Flux2 Flex models. We worked with BFL for this. And you can see that you can get way faster, and you can still be on the Pareto front for the specific use case of text rendering.

So is benchmarking dead? The idea is it's not dead. We can do it properly and get a lot of useful information out of this. And if you use it in a better, like, by taking all these, you know, rules when using the evaluation, you will usually not find a large foundational model, but more, like, a lot of small preference models that will be very good for your use case.

So I just listed a couple of takeaways which are, like, evaluate on many samples. You look at the use case conditions, use multiple benchmarks or efficiency, which are key things to keep in mind when evaluating models. And how to reach, like, state-of-the-art models in general.

Like, this is what we are doing at Pruna. We are actually building a lot of what we call preference models that are served behind endpoints. We have the fastest, for example, image models, video models that can run between 1 second to 5 seconds.

And but we also try to give a lot to the open source with a lot of open source contributions with a package to show you how to compress your models on your own. Also a lot of materials on all the best research papers for efficiency or even, like, some efficiency course.

So thanks for your attention.

I think we are out of time, but if there are any questions, happy to take them.

Okay, perfect. Then you have a question?

Yeah, yeah. So about the compression that you guys do, could you, like, elaborate, like, on how you do compression on video models and image models?

Sure. So actually there are multiple, like, I mean, you know it as well, there are a lot of family of compression methods. So of course you can guess, like, quantization things we do a lot, and we do it a different quantization for every specific module in the model, which is super important.

We can do also some pruning where we just remove some components which are not important. And for all these image and video models, something that works quite well is working on the step that the denoiser, like, when you generate a video or an image, you usually use, like, 20 to 50 steps to generate, like, the content.

And you can actually reduce it a lot either via distillation or caching methods.

Okay.

So you instead of doing, like, 50 times the computations using the same backbone, you can do it way less, I don't know, 20 times or even, like, four times depending on how aggressive you want to be.

No, I want to get on my own envelope's video, we are, like, doing some caching.

Yeah.

And I wanted to understand if you guys know something different that I could, like, use to make it even faster.

So we have, like, in our package, we have a lot of open source algorithms for good caching. But we have also some internal, you know, algorithms that we have for the models we serve behind the endpoints. But yes, there are really advanced caching methods and so on, but yeah.

Thank you.

Sure.

Awesome.

Thanks. Okay, then.