# Hard Won Lessons from Building Effective AI Coding Agents – Nik Pash, Cline AI Engineer · 2025-12-12 Nik Pash, head of AI at Cline, argues that frontier models have made clever agent scaffolds obsolete—capability now beats engineering tricks. He insists that real model improvement comes from benchmarks and RL environments, not from RAG or indexing systems. Pash details Cline's RL environment factory, which converts real-world coding tasks into training data by qualifying tasks, reconstructing environments, and defining pure outcome verifiers. He announces Clinebench, an open-source benchmark built from actual software development captured via Cline's provider, designed to measure and improve models on real tasks rather than synthetic puzzles. Pash urges the community to contribute by using Cline on open-source projects, turning model struggles into benchmark candidates. ## Questions this episode answers ### Why does Nik Pash argue that building clever scaffolds around AI coding models is no longer effective? Nik Pash explains that for years, developers built clever scaffolds like RAG and tool-calling to compensate for weak models, but now frontier models like Gemini 3.0 'bulldoze those abstractions.' He states that capability beats scaffolding, citing Gemini 3.0's performance on the Terminal Bench leaderboard without any agentic harness. The takeaway: 'get out of the model's way' and stop over-engineering with marginal tweaks, as the real gains come from stronger base models. [0:35](https://aie.addtry.com/dc7e59fd-0ada-4f8c-9049-c3c5b2c7a22a?t=35000) ### How does Cline create RL training environments from real-world coding tasks? Cline's 'RL Environments Factory' pipeline converts real-world coding data into training environments. First, sub-agents qualify tasks by checking repository existence, user intent, and ground truth commits, filtering out trivial tasks. Then, 'archaeology' reconstructs start and end states locally, containerizes them with Docker, and defines outcome-based verifiers that avoid overprescribing solutions. Originally taking 16 hours per task, the process now takes under 20 minutes, aiming for full automation where models learn to build their own RL environments. [6:04](https://aie.addtry.com/dc7e59fd-0ada-4f8c-9049-c3c5b2c7a22a?t=364000) ### What is Clinebench and how does it aim to improve AI coding models? Clinebench is Cline's open-source benchmark providing standardized RL and eval environments from real software development tasks, not contrived puzzles. Nik Pash argues that agent labs possess valuable real-world engineering data but keep it closed, slowing frontier research. By making Clinebench free and inspectable, anyone can measure and improve models on actual engineering work. Users contribute by opting in while using Cline, allowing stuck tasks to become benchmark candidates, creating a community-driven substrate for progress. [12:30](https://aie.addtry.com/dc7e59fd-0ada-4f8c-9049-c3c5b2c7a22a?t=750000) ## Key moments - **[0:00] Bitter Lesson** - [0:35] "For years we compensated for weak models by building clever scaffolds around them," says Nik Pash. - [1:55] "Capability beats scaffolding," says Nik Pash. - **[2:12] Trivial Tweaks** - **[3:43] Real Bottleneck** - [3:43] The real bottleneck is not agent cleverness but benchmarks that determine what frontier models learn to do next, says Nik Pash. - **[4:40] What's a Benchmark?** - **[5:44] RL Factory** - [5:44] Benchmarks measure models; RL environments improve them by using the score to update policy weights, says Nik Pash. - **[7:19] Environment Build** - **[7:57] Verifier Art** - [9:05] "The kettle doesn't care how you achieved it — a pure outcome-driven verifier just signals the result," says Nik Pash. - **[10:01] Automation** - [10:24] Automating RL environment creation: reduced from 16 hours to less than 20 minutes per task, says Nik Pash. - **[11:11] Truth Nuke** - [11:20] Every major agent lab captures real engineering interaction data but keeps it closed, slowing frontier model improvement, says Nik Pash. - **[12:30] Clinebench** - [12:30] Clinebench launched: open-source benchmark for real software development tasks, not toy puzzles, says Nik Pash. - [13:32] Clinebench invites community contributions; when a model gets stuck on a real task, that task becomes a benchmark candidate, says Nik Pash. - **[13:51] Outro** ## Speakers - **Nik Pash** (guest) ## Topics Agent Evaluation ## Mentioned Cline (company), Clinebench (product), Docker (product), GPT (product), Gemini (product), Git (product), Sonnet (product), Terminus (product) ## Transcript ### Bitter Lesson **Nik Pash** [0:21] Wow. It's wild to be on the same stage as so many people I've drawn inspiration from. Let's dive into it. My name is Nick, I'm the head of AI at Cline, and today I'm going to share some lessons we learned along the way. So let's start with the bitter truth: for years we compensated for weak models by building clever scaffolds around them. All kinds of clever ideas like RAG, indexing systems, search trees, tool calling scaffolds, all of this was invented to cope with weaker models. And frontier models simply bulldoze those abstractions now. You don't really need your scaffolding anymore; your scaffolding just gets in the way of these models. And the question really isn't how fancy is your agent stack; increasingly, it's how strong is the model driving it. And the lesson here is relentless. A perfect example of what I'm talking about is Gemini 3.0 released this week, and it immediately dominated terminal bench leaderboards with no agentic harness supporting it at all. In this chart you can see Gemini 3.0 on terminus scored better than the vast majority of model-agent combinations in the world, all out of the box. And what's remarkable is that terminus is designed to be an unopinionated, generic, stripped-down harness, and it has no graph search, no RAG, no indexing, just "here's a terminal, go figure it out," and it crushes. The whole point of terminus is that it has no clever tool calling, no context engineering features. So the takeaway here is that capability beats scaffolding. If you get out of the model's way, it will perform just fine. So really what I'm driving at, and the key takeaway from this whole talk, is: if you're building agents, just relax. ### Trivial Tweaks **Nik Pash** [2:12] Cool it with all your clever engineering tricks. Stop overthinking it. That's it. That's the lesson. And another point on this, kind of like an aside, is: I don't know about you guys, but we're all on Twitter, I'm on Twitter, and at this point I just think talking about these clever little context tricks and hacks is a little played out. Like, at this point I'm straight-up tired of seeing some of this stuff. And like, I get it, it's free engagement and we all indulge in it a little bit, but personally I think there's not really much signal there. So if you want the full playbook for building an effective coding agent, like, the playbook'sright here, it's up on the screen. There was really some novelty talking about it, like, months ago, but at this point, in my opinion, it's been done to death. And we've been in this, you know, we're model agnostic at Cline, we support all the models. Every two weeks there's a new big model release going out, and we've basically come down to the same playbook of supporting each model as it comes out. So I'm sure everyone here knows how to tune an agent from Sonnet 4 to Sonnet 4.5, from Gemini 2.5 to Gemini 3, and GPT-5 to GPT-5.1. I feel like this entire conversation is a little played out. So I'm not really even going to cover this in depth because the tweaks here are trivial and the gains are marginal. ### Real Bottleneck **Nik Pash** [3:43] So what I really want to talk about is something that's not actually given a lot of attention, and it's the real bottleneck. And the real bottleneck is that you can build the cleanest agent in the world, but that doesn't improve model capability by even 1%. Models only get better when labs train on something hard. And benchmarks, not agent cleverness, not all your clever engineering techniques, not your clever RAG pipelines, it's benchmarks that determine what frontier models learn to do next. And models didn't magically get better at tool use. They got better because people built RL environments that forced them to practice certain actions: handling failure modes, retrying, and, for example, like, agents improve only when the model learns inside theright environment. Every jump in reasoning we've seen came from a benchmark. ### What's a Benchmark? **Nik Pash** [4:40] Every jump in agent reliability came from an RL environment. So the real questions become: what is a benchmark? How do you turn real-world agentic coding data into an RL environment? And what makes a good verifier? How do you detect real difficulty? And how do you train these models to work on the problems that we actually care about as engineers? These are the questions that matter for the next frontier. So what is a benchmark? A benchmark, put simply, it's an environment. So in our case, it's like a Docker container where you let the agent run wild. It's a starting state, which is the snapshot of the code when you started working on a real-world coding task, as well as a starting prompt. And the last thing is a verifier at the end that checks whether an end state is correct or acceptable. So how are RL environments different? Well, here's the thing: they're not really different at all. And you might notice, this chart is basically the same thing as the previous slide. ### RL Factory **Nik Pash** [5:44] The only real difference, the only distinction here, is how the reward is used. Benchmarks measure models. RL environments improve models. The score doesn't just stop in a leaderboard where you publish the results; the score is actually used to update the weights of the policy model. So how do you transform real-world coding data into useful RL environments for training? At Cline, we created the system called an RL Environments Factory, looking for a better name there, but that's what we got so far. And the first phase in this pipeline is you get sub-agents and you have them qualify tasks. And these sub-agents, they work in parallel to decide whether or not given tasks are suitable to be turned into RL environments for the purpose of training. And the qualification process goes as follows. So you have, you start with origins. So you have to validate: does the repository actually exist? Is the starting commit accessible? Is it open source? The journey, where you look at the starting prompt, the other follow-on prompts that the user might have followed up with, with the agent, you have to try to understand what was the user actually trying to accomplish. What was the spirit of their task? And lastly, it's the outcome. So can we find the actual commits or PRs that fixed the problem in real life? Like, did they actually commit the solution to their problem later on in the timeline? And we're actively looking for easy disqualifiers as part of this. So things like vibe-coded slop. We don't need another benchmark that tests for, you know, build the next JS app from scratch. We're looking to disqualify trivial tasks that are too easy and tasks that have no reliable start or end states. ### Environment Build **Nik Pash** [7:39] And lastly, what makes a good RL environment good? How do we actually make an RL environment? And what makes a good test or verifier? So phase two of this pipeline is building the actual RL environment. So you start out with archaeology, where you actually reconstruct both states locally. ### Verifier Art **Nik Pash** [7:57] You pull down the code. You see if you can implement it yourself, reconstruct it, build it, and verify that the bug that the user was referencing and the solution actually exist. You document every obstacle and dependency. You containerize it with Docker, removing Git, obviously, so agents can't reward hack. And lastly, you define the verifier at the end. And this is where it gets into, like, a little bit of the art of building a good verifier. And I want to talk about this because the analogy that I typically give is a tea kettle. So let's say the user's goal is, "I want to boil water." A really good example of a verifier to test whether or not the water is boiling is a little whistle attachment that goes inside your tea kettle. And the whistle is a pure outcome verification. It's an example of a pure outcome-driven verifier where the water either reached the boiling point or it didn't. Either it's whistling or it's not. The kettle doesn't care how you achieved it, whether you used a gas stove, an electric induction stove, or a campfire. It just signals the result. And in the process of doing this, all these weird bad tests can emerge. So you might have noticed, like, that the sub-agent might have noticed, like, "Oh, in the ground truth solution, like, in a previous run, the burner was set to high. So maybe we should be checking for that." But we all know that water can boil at a low setting on the burner. Or, "Was it on the front left burner? It has five minutes left." Like, all kinds of weird bad tests. And the key point here is: don't overprescribe based on the ground truth. Test for the spirit of the task. Test for the outcome of the task. And the outcome at the end of all of this is a containerized benchmark or RL environment for that task. Agent work is recorded so you can see the traces, the trajectory that the agent took to complete the task, and you can reliably score it and verify it. When it's fully portable, you can run it on any device. ### Automation **Nik Pash** [10:01] So the path to automation that we've been undertaking as part of this is: can we fully automate the process of converting real-world coding data into RL environments for the purpose of training models? And this work largely started out manual, but then the first time in the RL environment was, like, about 16 hours of my time. And what used to take 16 hours now takes less than 20 minutes per task. And we're building towards a fully automated RL environment factory where the bottleneck shifts from engineering to collecting high-quality tasks. And an interesting kind of point here: the natural endpoint of all this is, what if we actually built RL environments? And this is, like, a question for everyone in the audience: what if we built RL environments to test how well agents can actually make RL environments? Kind of like a meta-benchmark. What would hill climbing on that look like? And you can kind of start imagining that as models get really, really good at making their own RL environments to train on, based on real-world user data, you kind of complete that loop. It's something to think about. So, okay. ### Truth Nuke **Nik Pash** [11:11] This next part is the truth nuke, also known as TRUCK. An unspoken fact is that we're not alone at Cline building this kind of system. Every major agent lab captures this data. They all do some version of this behind the scenes, but no one really talks about it. And I don't even need to name them. If you know, you know, and realistically, you all know. These same companies cite internal benchmarks to justify legacy systems that they spent months maintaining, but curiously, you'll never be able to study or inspect them because they don't publish them openly. And this data is so valuable, yet no one shares it. It's the only thing that actually moves the needle. And here's the heart of my argument: by standing between real-world engineers working on real-world tasks and the models, agent labs have a unique role in history. We can build better prompts, we can build better tools, but none of that improves the underlying models. We possess the single richest dataset of real engineering work anywhere in the world. Models don't improve without this data. And keeping them closed is slowing down frontier research. So today we're announcing Clinebench. This is our attempt to finally create a benchmark that isn't cosplay engineering. It's not, "Write me a server that generates Fibonacci sequences." This is real software development captured and packaged into standardized RL and eval environments. ### Clinebench **Nik Pash** [12:47] And this is the benchmark that we always wanted someone else to build. No one did, so we're doing it. And anyone can participate. So here's how it works. The whole thing is open source. There's no secret sauce, no locked-away datasets. You can openly run it yourself and inspect it to see how it works. Anyone can use these environments for SFT, RL, evals, whatever. The point is to just give the entire ecosystem a real substrate to measure and improve models on, not just leak code puzzles. And this only works if the community contributes. And the good news is, you don't actually need to do anything special. Just work on your open source project with the Cline provider turned on and opt into the Clinebench initiative. If a frontier model gets stuck and you step in to fix it, that's actually an ideal task to be a candidate for a benchmark. And that's it. Just use the Cline provider, see where the model struggles, and we'll pick it up and introduce it into this open source benchmark. ### Outro **Nik Pash** [13:51] So Clinebench will always remain free, fully open source, and freely accessible. Thank you all. If you want to contribute. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.