CI in the Era of AI: From Unit Tests to Stochastic Evals — Nathan Sobo, Zed
Jun 27, 2025 · 14:50
Nathan Sobo, co-founder of Zed, explains how his team adapted continuous integration for AI-powered features in their Rust-based code editor. Zed's traditional CI eliminates non-determinism with simulated schedulers and deterministic tests, but LLMs forced a shift to stochastic evaluation. Their process starts with broad data-driven evals, then drills into focused stochastic unit tests run 100–200 times with pass thresholds, and finally refines into deterministic tests. Specific challenges like parsing streaming edits, XML tag mismatches, indentation normalization, and strange escaping were solved by robust algorithms—e.g., fuzzy matching and indent delta detection—rather than complex ML. Sobo emphasizes that rigorous empirical testing and traditional software engineering skills remain essential, even when embracing probability over binary pass/fail.
