Fighting AI with AI — Lawrence Jones, Incident
May 17, 2026 · 17:29
Lawrence Jones, founding engineer at Incident.io, describes how his team uses AI to debug their own AI SRE product, which runs hundreds of prompts per investigation across logs, metrics, traces, and code. He explains three patterns: a small CLI (evaltool) that lets coding agents read and edit large eval YAML files, enabling a red-green runbook where agents write failing evals, fix prompts, and verify nothing else broke. Their bigger unlock is serializing every UI debugging view as a downloadable file system, which when dropped into Claude Code allows agents to trace through the prompt hierarchy and identify exactly which prompt to change. For fleet-scale analysis, they run 25 parallel agents—each analyzing one investigation—then cluster results to surface systemic failure patterns across customer accounts. Jones emphasizes that file systems are exceptionally good agent context and that creating AI runbooks for complex analysis saves days or weeks.
