Malleable Evals: Why Are We Evaluating Adaptive Systems with Static Tests? — Vincent Koc, OpenClaw
May 12, 2026 · 15:05
Vincent Koc argues that AI applications are adaptive systems, yet evaluations remain static datasets—a problem he calls 'eval calcification.' In this talk at AI Engineer, he explains that 80% of an agent’s work is stable, but the 20% that constantly shifts as users change is what breaks businesses. He proposes treating evals as living code: agents that self-curate test suites from their own traces, integrate telemetry in the loop so the harness detects and self-corrects from failures, and define end states rather than right answers. Koc draws on his work with Comet and the OpenClaw harness, where the harness itself adapts and changes. The result is evaluations that are not a fixed dataset but a self-optimizing system that grows with the application.
