An AI Agent Became the #1 Contributor in OpenAI's Hiring Challenge — Zhengyao Jiang, Weco

This April, OpenAI ran a hiring challenge, a competition called Parameter Golf. The top contributor was one candidate that they couldn't hire. It wasn't a person; it's an agent we built called Aiden. In Parameter Golf, the goal is to train the best language model you can under size and computation constraints.

About 1,000 machine learning engineers researchers participated. They filed 2,000 submissions. Only 47 passed OpenAI's review and made it into the leaderboard. Seven of those are actually Aiden's. More than twice what any human contributed. You've seen a lot of auto-research today.

Agents are hill climbing benchmarks. Those are really impressive results. The question I want to ask is a bit different here. Can the auto-research agent produce work that a human community actually recognizes? Beyond a good score, agents are optimizing for something that other engineers can merge, fork, and build on.

So instead of having an agent just hill climbing locally, we built one that publishes its own work, and that's Aiden. Quick context on us: Weco is an auto-research company that founded about 2 and 1/2 years ago. I'm Cofounder and CEO, Zhengyao.

I got my PhD at UCL on reinforcement learning. About 2 years ago, we built Aide, the top auto-research agent independently evaluated by OpenAI in their MLE bench paper. Even though back then there was no such name called auto-research, people call it machine learning engineering agent.

Aiden is the next step in an experimental prototype. It's a multi-agent self-improving system that can read public information like research papers and other PRs, run its own experiments, and the submitted PR wants the findings plus a quality gauge.

We sent Aiden to Parameter Golf competition, and it ran for about 22 days. By the end, Aide has set 7 leaderboard records. Each one is a new best for the competition, stamped by OpenAI, and the best a human only made 3.

Passing the host's review is one signal for the quality. A second, maybe more important one is whether other participants would build on your work. And it turns out Aiden's work had the highest impact within the whole community. Here we are using an inference measure that is used widely in academia.

It's called the H-index. Roughly, if you have X papers get cited X times, then your H-index is X. Computed over PRs, Aiden was 10, and the next human was 7. The whole community was building on an AI system's work, including many of other leaderboard entries.

To break it down a little bit, why can an autonomous AI system be so powerful? One obvious reason is that it's an AI that can run tirelessly. Over 22 days, it ran about 1,300 experiments on a single H100 node.

But throughput isn't the whole picture. A well-tuned AI system can also keep its output quality high. On the compute side, it uses at most 4% of the competition's total compute,

and it made about 15% of the records. Also, 28% of its submissions made the leaderboard, roughly 6 times higher hit rate than the community average. So Aiden actually lifted the signal-noise ratio within the whole community's public communication channel, which is PRs.

It didn't wing through massive parallelization, even though auto-research has tons of potential for parallelization.

By those numbers, it might feel like auto-research already dominates human experts on ML engineering and research, but that's not the full story I want to tell. Humans and AI actually contribute in very different ways. When we trace the ideas, Aiden's record PRs, almost all of them come from humans: research papers, other participants in Parameter Golf, or in similar communities like NanoGPT.

Those ideas are not necessarily a merged PR. Sometimes it's a note a human researcher said, "Oh, I give up this idea because of some implementation difficulty." And the agent is good at finding them and actually implementing them. There are also a very small fraction of original ideas Aiden came up with by itself, which emerged from its efforts to navigate the file size constraints.

Here's a concrete example that traces the patterns I just talked about. So Aiden picked up an idea from a quandary paper called Gated Attention, and it worked, but it introduced more parameters and it broke the 16 megabytes file size limit.

So it figured out a quantization mechanism to bring the file size down. But with those two primitives combined, the score barely moved. Then another contributor posted a tokenizer improvement. Aiden recognized the idea, combined it with architectural work, and it just worked for 5 days or so.

And after this combination, the three ideas turned out to have a huge synergy that led to a big jump in performance, and they became one of Aiden's leaderboard records.

So to sum up how I'd interpret Aiden and, in general, auto-research systems' effectiveness, it's very strong at finding and implementing ideas. In the case we just saw, it brought an idea from a recent paper into actual implementation in the competition.

And it's good at dugging promising ingredients out of the Parameter Golf community, even though the public channel is actually very noisy information-wise. It can also come up with logically straightforward ideas. For example, in this case, once you add parameters and it breaks the file size limit, one obvious next move is just quantization.

And it's really fast and really efficient at finding theright combinations across a huge search space.

OK, maybe none of those sounds very sexy. Most of them are just good execution. But in reality, execution is mostly the bottleneck. What moves the frontier is usually exactly some belief in existing ideas and tons of good executions.

OK, to step back, the state of human-AI collaboration is that humans collectively provide a lot of creative ideas, and the agent does the execution to solve a concrete challenge. What we are looking at is a large group of humans and one AI system.

Does this mean a single human engineer's contribution marginally gets smaller? I'd say even for that, not really. In Parameter Golf competition, it's easy to only focus on engineers that are actually doing hill climbing. But the design behind the competition itself is tremendously important.

A bad design can make the whole community's efforts useless. And their eval design work will have huge leverage in the auto-research era. I really like one tweet from Andrej Karpathy about 10 years ago, where he said, "Gradient descent can write code better than you.

I'm sorry." For the context, about 10 years ago, deep learning was starting to eat up a lot of software engineering, like conventional coding work. And his tweet was arguing against those people who thought they can handwrite better code than a trained model.

OK, now obviously no one is seriously trying to handwrite code to beat a model. However, software engineering as a job still exists. And so many people's jobs are just training those models, and those are one of the most well-paid jobs today.

I think how gradient descent changed coding is a great metaphor for how auto-research will change research and ML engineering. It accommodates certain execution skills. At the same time, it makes some higher-level skills far more valuable. So actually doing auto-research is a lot like training a model.

Your code-based abstraction is essentially the architecture. It sets the constraints and the priorities for what the agent can explore. Your eval is the loss function and the data. It sets what the agent optimizes for. Take the eval first.

The eval is the signal you use to train a model. In this case, it's training your code. It plays the same role as data and the loss function in model training. Or in a reinforcement learning setting, it's like an environment that the agent is training in.

Nowadays, no one would argue data or environments don't matter. And this is where a vertical mode can also be built. You might have proprietary data for evaluation, or a unique understanding of, in a particular field, what matters and how to measure it.

And a good evaluation would be amplified more and more as auto-research is getting stronger.

The other one I think is really underrated is code-based abstraction. The abstraction provides the framework that auto-research can iterate on. And that

starting point hugely biases the whole search direction. This is a lot like architecture design in neural networks. Different architecture, in theory, can represent the same function. But the architecture systematically makes some of the functions easier to be learned.

And a good architecture biases the optimization towards solutions that generalize better, perform better, even when the training loss might look the same. That's exactly the same for auto-research. Here's an example. We run auto-research for a fraud detection pipeline, and we're trying to optimize the data preprocessing.

And first, we give it a loose API where the same function processes both the training and the testing data. And the score looks great. But the solution was polluted because there is a certain test set information that got leaked to the training information.

We then tightened the abstraction to a more strict API where the test data couldn't reach the training. And the data leakage rate just dropped to 0. In this case, a good abstraction leads to better solutions, even though if the agent really wants, they can steal a reward hack.

So my point is, using auto-research is a new craft. It's about designing a hill for an agent to climb. And we are still very early on it. I think that makes this an extremely exciting time to be an AI engineer.

Auto-research will change what skills matter most: creativity, the judgment to design a good eval, or an abstraction. Those will soon get exponentially more important. Driving those systems itself is where it will be a new skill. And that one barely existed one or two years ago.

So the search is automated. The human would just move up the stack, not out of it.

Again, Weco is an auto-research product research lab. We will keep sharing what we are learning as we build on our blog, and I will also post some of my thinking too on X. If you think some of this is useful to you, feel free to follow me on X.

Thank you.