The ROI of AI: Why you need Eval Framework - Beyang Liu
Dec 31, 2024 · 25:28
Beyang Liu, CTO of Sourcegraph, presents six evaluation frameworks for measuring the ROI of AI coding tools, arguing that no single metric suffices because AI ROI reduces to the intractable problem of developer productivity. He dismisses the 'roles eliminated' framework as irrelevant for engineering, noting that backlogs are never fully addressed. Instead, he highlights customer examples: A/B testing velocity at Palo Alto Networks showed 20-30% acceleration with Cody; time saved as a function of engagement lower-bounds gains (e.g., each code search saves ~2 minutes); and LITOS tracked value-add zones (building features, writing tests, reviewing code) instead of lines of code. For key initiatives like code migrations, ROI is measured by how many months the migration is pulled forward. Liu also advocates developer surveys bounded by a 5-25% productivity tool budget, and warns against the Mythical Man Month fallacy: preferring one 100x human developer over 100 mediocre AI agents.
