Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor

Hi everyone. So I'll be talking about, like, some work on evaluations, particularly evaluations across, like, I guess I've done in the last 4 years. So let's get started. So, uh, I'll be talking about co- coding evaluations across varying time horizons.

So I've been working on, like, in the code space for about 4 years now. Like, it wasright before, like, early Copilot came out, and my first project was actually working on generating, like, single-line Pandas snippets. And my last project was generating an entire codebase, so the field has, like, really progressed very quickly.

So I'll be talking about, like, uh, different stages of evaluations we have considered and some learnings across each of the projects, and how I see evaluations going forward. So the first work I did was on, like, evaluating coding models in, like, second work doing in seconds of time, like generating single-line snippets, your Copilot code completions.

Then I wo- did some work on, like, evaluating on, like, interview-style competition programming problems, which, uh, where models can work up to minutes. Uh, then we worked on some work on, like, repository question answering, which required, like, maybe more, uh, multiple minutes, tens of minutes.

Uh, and finally, like, pushing the frontier forward, we are thinking about evaluating models on very complex tasks which can take hours or, like, multiple hours of work, like code optimization and, like, even further. So let's get started.

Uh, so first work I'll be talking about is, like, code bench, which is, like, evaluation work on models for, like, competition coding. So here, like, this is what a problem would look like. This is, like, very standard leetcode problem, and don't worry, you don't need to solve something like this.

So, uh, like, here, as you can see, there's a problem statement. And the nice thing about these interview-style problems is that these problems are very well defined. You have, like, good natural language specification, some example, input-output examples, so you can very reliably evaluate whether the models are doing a good job or not.

So what was the motivation behind this and how we improved the frontier here? So the first challenge in evaluating language models these days is, like, data contamination. These models are trained on, like, the entire internet. And, like, on Stack Overflow, you'll find, like, very similar programming problems, puzzles.

Similarly, like, you'll find, like, very similar programming problem sources on GitHub or on the internet. So, like, contamination is a big deal. Uh, another very challenging factor which I struggled with the field is, like, insufficient test suites. So you'll see that, like, in this program, like, the goal was to return a sorted unique common elements between the two lists.

But, like, even a solution which does not do the sorting and just returns the set actually works because the tests were brittle and were not catching this mistake. So, like, test suites is another, like, very challenging factor and how do we generate good and diverse tests.

And finally, difficulty distributions, which is something which people do not really reliably, like, calibrate. Uh, like, when I first was working in this space, like, there were two benchmarks available. On one benchmark, the performance was 80% or 90%, and on the other one, it was 1%.

And there was nothing in between. And, like, as, like, benchmark users, what we care about is having some signal from the benchmark to, like, basically hill climb to make progress, to measure progress. And in either of these regimes, when if the problems are too easy or too hard, you don't get a lot of signal.

So it is very important when you're designing benchmarks to think about, like, the kinds of problems you are taking and will it provide enough signal for the users of your benchmark. So, like, in Lightcode Bench, we pioneered, like, dynamic evaluations.

Particularly, like, we can periodically update the evaluation sets. And this gives you two very nice factors. First is you can combat contamination. So you can evaluate the models on problems that were released after the model was trained. So it has likely not seen the problem, something like that.

Uh, and, uh, then you can also modify the problem difficulty distributions over time. So as we have talked about, models are, like, improving very rapidly. Uh, so what was difficult for the model six months back might not be now.

So you can, uh, if you're updating your evaluation sets constantly, you can actually keep calibrate the difficulty distributions calibrated so you still get more signal out of your benchmarks. So how we did that here, like, we had, like, an automated approach for curation of these problems.

And, uh, similarly, we could automatically construct these test cases in an automated manner. And, uh, this allows a very nice thing. When since we are, like, collecting problems over time, we have time as a control knob. So, like, we have these problem release months on leetcode.

And if you evaluate the model performances, like, the pass at the rate one metric, like, on problems released over different months, you will see that after, like, these model release dates, you would see stark drop in model performance.

So, like, after DeepSeek was released in, like, September 2023, the performance starkly drops from, like, maybe 50% average to, like, over, like, 20% or 15% average. So, like, based on these sliding windows, you can evaluate performance, measure contamination, and even combat contamination.

Um, we have the running leaderboard, which is, like, very well maintained. And on this leaderboard, you can actually, like, view performances by scrolling this horizontal time bar. And you'll see that as you're scrolling, the contaminated models, which are the red bars, actually go down, which does highlight that, like, problem does, like, model performance does change on these newer kind of problems.

Um, finally, for test generation, we maintain, like, these test generators. So if you worked on fuzzing, you would have, like, input generators where you generate diverse inputs. And each of the problems are supported by, like, 30s or 50 inputs.

So you can reliably find mistakes and bugs in incorrect code. And these are all automatically generated using an LLM-driven approaches. And these problems have been, like, continuously being released and updated. So we have released, like, six different versions of Live Code Bench.

And these one of the nice things or one of the worrying things for me at the start was that, like, if you're constantly updating the eval sets, will, like, people be able to keep track of them? Will people be using them or will they just restrict to a single version?

Uh, it turned out that these newer eval sets were constantly, like, adopted by different foundation model labs. And, like, since we updated the problem difficulty over time, the evaluation sets continue to provide strong signal to compare different models.

Um, so this was, like, Live Code Bench. Let's talk about, like, something which is more on coding agents, like, more real-world programs. And this is work on, like, software optimization. So this is a problem I'm very excited about.

And I'll talk about a few factors why you should maybe be excited about this. So, uh, here we are trying to measure model capabilities in generating high-performance software. And I feel that this, like, problem domain, like, mixes two factors, like, the algorithmic coding field I talked about, which is, like, Live Code Bench setting, but also, like, global software editing, like, SWE-bench and other, like, software general software engineering benchmarks.

Uh, in high-performance software, you will have to do algorithmic work. You have to do deep analysis and find, uh, generate software with, like,right runtime. So, uh, one of the key principles when we were trying to build this benchmark was, like, ensuring construct validity.

Because when you see a lot of benchmarks today, uh, we get very high benchmark scores, but a lot of the times, they don't really translate to real-world performance gains. So construct validity refers to how close a measurement reflects the underlying concept it's meant to measure.

So, like, here we are measuring code optimization, and we want something which is, like, reliably evaluates real-world takes. So this usually requires, like, two aspects. First is, like, the task distribution. Your task should be natural and sourced from the real world.

And then you should be able to reliably grade them. So let me talk about, like, what steps we take to make this happen and how we construct this benchmark. So let's say we take a codebase like Llama CVP.

Uh, we take we crawl over all the commits of the codebase, and we find the commits which are, like, doing something related to performance optimization. So here there was this commit which is optimizing the quantized performance of, like, certain kinds of models.

For all of these performance-optimizing commits, we would, like, generate performance test cases. Um, and these performance test cases would look like some workloads. And once we have these workloads, we have a very nice and precise way to specify the problem statement that, uh, given this workload of, let's say, running Qwen 7B model, can we give this problem to SWE agent and ask the model to optimize the code Llama CVP repository so this code runs faster.

So as you can imagine, this task is, like, fairly challenging. You need to understand, like, low-level implementation details and, like, how quantized models behave, how we can improve the runtime. And so models can generate a patch, and the evaluation is done on whether the patch is correct.

So does it pass the equivalence check with the human patch? And is there a valid optimization over the reference human patch? That is whether you can generate a better runtime than what a human could do. So, like, this is a very challenging task.

We have, like, 100-plus optimization tasks sourced in this manner. And this is, like, fairly, like, important in, like, high-performance settings. So think about, like, data science, like, ML visualization scenarios. Our benchmark, like, comprises of, like, various low-level code, like, C, C++, Rust.

And the very nice thing is, like, these are precise problem statements. You can easily specify to the model what is the goal in the form of a performance test which the model has access to, and it can continuously iterate over it for a long time.

So here we can scale the test time, compute, and pick the best solution based on the test cases that we have. And this can happen, like, synchronously or asynchronously. So, like, we generate these performance test cases, and that worked reasonably well.

But we found that there were, like, cases of reward hacking here. So what do I mean by reward hacking? Like, frontier models would write non-idiomatic code to, like, actively exploit the evaluation infrastructure or overfit the test distributions. So one funny example we saw was, like, models would add, like, LRU cache to, like, arbitrary pandas methods when we were trying to optimize pandas.

And the official solution should have required changing something in the internals. So we tried to pass this by changing our evaluation infrastructure so it's, like, more robust to this kind of hacking approaches. But then we saw something, like, even more drastic.

Models would sometimes completely hijack the infra where they would add a, like, sidecar customize.py file which runs at the start of Python runtime. And it would basically change the numpy library, like, which was installed in the codebase to something it crawled from source.

And there is, like, I think you can do some ways to, like, take some measures to make your evaluation infra which is robust to these kind of, like, adversarial, like, attacks. But here, like, there could be myriad ways in which models can hack these kind of scenarios.

And here we propose, like, Hack Detector, which is a detection system that leverages GPT-5's, like, code analysis capabilities and test time compute to, like, basically identify these kind of hacking behaviors at runtime. So you don't have to imagine all the possible failure scenarios at the start.

So what it would take is, like, a model patch, the expert patch, and test cases. And we'll ask GPT-5 to give, like, verdicts on, like, whether it's reward hacking with some kind of explanation. We'll do this a few times and take the consensus.

And based on this consensus, we'll determine if this is doing some, like, non-idiomatic coding patterns or not. And we did some failure analysis based on this. So now you can detect mistakes using test cases, whether the code is correct or not, whether it is optimizing or not.

But you can also detect reward hacks using this, like, LLM as a judge factor. And what you see is kind of surprising. Like, models make a lot of, like, correctness mistakes that you can catch by tests. But even if the code passes the test cases, like, O3 attempted reward hacking patterns in, like, 30% of the problems it tried.

And this fraction is, like, going down for the newer models to some degree, but it is still existing. And as we go to more and more real-world tasks, this is going to get more challenging. And we need to figure, like, ways to combat these kind of reward hacking patterns by using LLM judge and other ways to make just evaluation infra more reliable.

So next, I'll talk about, like, Syzygy, some of our new work on, like, pushing the boundary of code evals even further and taking a look at more challenging tasks. So here we were thinking about, like, can, like, these language models translate, like, an entire codebase?

Specifically, given a specification as a C program, can you generate a safe Rust implementation for the C? And we took a fairly complex codebase. So Xoffle is a, like, highly efficient compression library from Google. Like, it has about, like, 4,000 lines of code, hundreds of functions, and complex data structures.

And we want, like, very precise and correct code. So we generated, like, a million compression inputs. And your test case was to generate a Rust implementation that maintains correctness over those million test cases. And when I did this work back in, like, last year, it took us 12 hours to actually do this translation.

Now, perhaps with better models, this can be done in two hours. But still, I think this is pushing the frontier of, like, what the models can do currently. So what was one of the key findings when we were trying to make progress in something like this?

Like, end-to-end correctness is important, but it only gives you, like, one bit of feedback. But for these very long-horizon tasks, one thing which will become more important going forward is, like, having some measures of intermediate correctness. So, like, for our case, we could measure, like, fraction of code translated, fraction of code refactored.

And based on these kind of settings, you can understand, like, if you're making progress or not and how you can scale systems better.

So, like, as we're closing, I'll talk about, like, quickly talk about some of the work I did on, like, in the wild evals. So this work was done in collaboration with LMArena folks. And, like, I'll talk about two settings here.

First is Copilot Arena. So this is, like, evaluating in-IDE code completion assistance. So what we would do here is we'll have an IDE plugin where, like, similar to GitHub Copilot setting, we'll generate a completion for you. But instead of just a single completion, you'll have two completions appearing, like, top and down.

And you can pick either one of them via shortcuts, like Tab or Shift-Tab. And based on the, like, acceptance rates, we can pairwise compare what the code completion assistants are doing.

We also did some work on RepoChat where, like, to evaluate, like, code question answering capabilities of models, we built a system where you can provide a GitHub URL. And you can ask a natural language query about the codebase, which could be something about explain the codebase to as complex as, let's try to solve this issue.

Let's give me a model patch that could solve this issue. And we integrated a very basic and simple, like, SWE agent system that fetches the codebase, resolves user queries, and, like, multi-turn code assistant conversations. So one thing that stood out to me in these kind of things is, like, how human-centric experiment design needs to be.

So, like, for Copilot Arena in particular, we realized that, like, latency is a big concern for acceptance rates. So if you look at, like, acceptance, like, latency below and acceptance rates, like, if it is, like, anything more than one second, like, the acceptance rates drop very starkly.

So people care a lot about latency. So you have to so we had to design an experiment so that it's robust to these kind of, like, latency differences between models, balanced latency across different models. So, like, if you're doing, like, anything in the wild, having this human-centric component, understanding human behaviors is very important to do anything meaningful.

So at the end, I think just to recap, like, I think I talked about a bunch of folks. Like, what are some big takeaways? So I think dynamically updating evaluation sets to, like, prevent contamination, like, modify the problem distributions, like, in terms of difficulty, in terms of distribution of tasks we care about.

As we, like, improve as the language model capabilities will improve over time, the types of tasks we'll start to do with model change. You can even think of this, like, we were doing, like, code completion where you were generating, like, few tokens, few lines.

And now we are generating, like, tens of lines, hundreds of lines. And to some degree, this will continuously change. And we have to update our evaluation sets so that it reflects the real-world usage and kinds of things people need.

The second very important thing is, like, ensuring reliable grading in this domain. And, like, tests are very good for ensuring correctness and provide a lot of reliable feedback. But once we go to real-world settings, like, models can start doing, like, a lot of non-idiomatic coding patterns.

They would add try-catches everywhere to just prevent any kind of bug from occurring. So having these kind of LLM judges to detect non-idiomatic coding patterns, code quality, and just any, like, arbitrary hacks will be very important. And finally, like, as I talked about in the last work, like, intermediate grading signals so that you can measure, like, incremental progress is, like, another key factor here.

So I think that's the end of my talk. Thank you.