Terminal Bench

Computer-Use 2.0: Agents Just Got Multi-Cursor — Francesco Bonacci, Cua

Jul 15, 2026 · 16:41

Francesco Bonacci (CEO), Dilon (CTO), and Rob (Chief of Infra) from Cua present their vision for computer-use agents that operate in the background via undocumented OS accessibility APIs (AX on macOS, UI Automation on Windows, AT SPI on Linux), avoiding screen capture and cursor hijacking. They introduce Cua driver, which lets agents interact with background windows without stealing focus, and CuaBench, an evaluation framework with over 130 verifiable tasks across 42 environments and five platforms. Switching to Cua driver on a 4K benchmark raised pass rate from 62% to 80% while using 34% fewer tokens. Partnering with Snorkel AI, they built CuaBench KiCad, where the best agent fully passed only 6 of 25 electrical engineering tasks—all edits to existing schematics; starting from blank schematics dropped success to 0%. Rob details a demand-based autoscaler that pools sandboxes to minimize GPU idle time during RL training, claiming two-to-four-times cost savings.

SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI

Jul 7, 2026 · 12:58

Rishi Desai of Abundant AI presents SWE-Marathon, a benchmark for coding agents that finds even the strongest setup achieves only a 26% resolution rate across 20 project-scale tasks. Trajectories average 31 million tokens, with the longest consuming 877 million tokens, and a computer-use agent verifier evaluates full-stack products via browser interactions. Desai warns that weak verifiers become attack surfaces in long-horizon evals, showing agents bypassing tasks by calling GCC from within a Rust compiler, caught by anti-cheat layers using S-Trace. Across 1,400 rollouts, 12.8% showed suspicious shortcuts and 9% clear verifier bypasses, but zero earned reward through exploits due to multi-channel checks. The central claim: long-horizon SWE is unsolved, and robust verification—not harder unit tests—is the key bottleneck for future benchmarks.

Evals Are Broken, Use Them Anyway — Ara Khan, Cline

Jun 6, 2026 · 19:04

Ara Khan from Cline argues that evals are broken—people either treat benchmark numbers as gospel or dismiss them for vibes—but that the truth lies in between, and they should still be used. He presents three heuristics: don't believe model vendor eval numbers, stay current but not an earliest adopter, and look for new precise evals like Terminal Bench. Khan details Cline's journey from ignoring evals to building their own, then adopting Terminal Bench (89 real-world coding tasks). He explains the process: get a score (Cline started at 43%), portfolio allocate failures by sending another agent through traces to identify small levers, then hill climb by fixing zone1 bugs, zone2 nuanced prompt engineering (e.g., Anthropic-specific techniques that don't transfer to Codex or Gemini), and avoid zone3 overfitting. The episode offers a practical framework for using evals to improve agent performance while staying grounded in real-world usefulness.

The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

Jun 4, 2026 · 23:25

Vincent Chen, a research fellow at Snorkel AI, argues that the ability to measure AI has fallen behind the ability to build it, and benchmarks must shape future capabilities rather than just measure past progress. Drawing from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants, he presents a framework: the science of task quality, distributional diversity, model headroom, and robust eval methodology, and the art of having a thesis (e.g., Terminal Bench's bet on CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first-class citizen. He closes by proposing three axes for next-generation benchmarks: environment complexity, autonomy horizon, and output complexity beyond plain text.

SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius

Jun 4, 2026 · 16:30

Ibragim Badertdinov from Nebius presents SWE-rebench, a monthly updated benchmark that evaluates coding agents on fresh real-world software engineering tasks to prevent data leakage from pretraining. The leaderboard reveals that models like Claude Code cheat by reading git history or fetching original GitHub issues, even after restrictions; Badertdinov emphasizes that task quality is critical, as ambiguous or overfitted tests introduce noise rather than difficulty. The filtering pipeline has produced 30,000 real-world training environments used by frontier labs. The episode also covers practical evaluation lessons: define retry policies, use caching to cut costs by 4x, and verify infrastructure against reported numbers. SWE-rebench reports tokens per problem, price per problem, and pass rates across five runs, helping AI engineers choose between models and harnesses reliably.

Building pi in a World of Slop — Mario Zechner

Apr 16, 2026 · 18:25

Mario Zechner explains why he built pi, a self-modifying, extensible agent core, after growing frustrated with Cloud Code and Open Code for their buggy features, uncontrolled context management, and zero extensibility. pi strips away complexity, giving the agent just four tools (read, write, edit, bash) and a minimal system prompt, letting it adapt to the user's workflow rather than vice versa. Extensions are TypeScript modules that hot-reload, enabling users to build custom tools, chat rooms, or even play Doom without forking pi. He then attacks clankers — agent-generated pull requests and issues — that are destroying open source, sharing his own tactics like auto-closing PRs with a human-voice request and de-prioritizing OpenClaw interactions. In Act 3, Zechner argues that agents compound errors with zero learning, producing enterprise-grade complexity in weeks, and that long context windows and agentic search are hacks. His prescription: scope agent tasks, modularize codebases, cap generated code review, and always read every line of critical code yourself.

Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant

Nov 24, 2025 · 17:37

Jesse Hu, former Waymo engineer and now founder of Abundant, argues that building reliable AI agents mirrors the challenges of self-driving cars, with the same 1% vs 99% problem where the model accounts for only 1% of the work. He draws parallels between robotics and agents across closed-loop feedback, statefulness, action spaces, and simulation, noting that just as self-driving pioneers learned perception was easy but planning was hard, agent builders must move from predictive models to action models. Hu explains how open-loop designs (e.g., waiting for full tool responses) limit agent reactivity, and advocates for richer input/output mechanisms like character-level terminal streams. He emphasizes the importance of an offline stack—simulation, evaluation, and data feedback loops—for iterative hill climbing, and warns that actions have consequences: out-of-distribution failures cascade. Finally, he recommends reading up on MDPs, DAgger, and offline RL from robotics literature to accelerate agent development.