Cline

Evals Are Broken, Use Them Anyway — Ara Khan, Cline

Jun 6, 2026 · 19:04

Ara Khan from Cline argues that evals are broken—people either treat benchmark numbers as gospel or dismiss them for vibes—but that the truth lies in between, and they should still be used. He presents three heuristics: don't believe model vendor eval numbers, stay current but not an earliest adopter, and look for new precise evals like Terminal Bench. Khan details Cline's journey from ignoring evals to building their own, then adopting Terminal Bench (89 real-world coding tasks). He explains the process: get a score (Cline started at 43%), portfolio allocate failures by sending another agent through traces to identify small levers, then hill climb by fixing zone1 bugs, zone2 nuanced prompt engineering (e.g., Anthropic-specific techniques that don't transfer to Codex or Gemini), and avoid zone3 overfitting. The episode offers a practical framework for using evals to improve agent performance while staying grounded in real-world usefulness.

Hard Won Lessons from Building Effective AI Coding Agents – Nik Pash, Cline

Dec 12, 2025 · 14:18

Nik Pash, head of AI at Cline, argues that frontier models have made clever agent scaffolds obsolete—capability now beats engineering tricks. He insists that real model improvement comes from benchmarks and RL environments, not from RAG or indexing systems. Pash details Cline's RL environment factory, which converts real-world coding tasks into training data by qualifying tasks, reconstructing environments, and defining pure outcome verifiers. He announces Clinebench, an open-source benchmark built from actual software development captured via Cline's provider, designed to measure and improve models on real tasks rather than synthetic puzzles. Pash urges the community to contribute by using Cline on open-source projects, turning model struggles into benchmark candidates.