How to run Evals at Scale: Thinking beyond Accuracy or Similarity — Muktesh Mishra, Adobe
Jul 22, 2025 · 9:25
Muktesh Mishra, lead engineer for Applied AI at Adobe, argues that evaluations (Evals) are the most critical aspect of AI application development, advocating for Eval-driven development. He explains that Evals must be tailored to the use case: RAG applications require accuracy or similarity metrics, code generation needs functional correctness tests, and agent systems demand trajectory evaluation and multi-turn simulation. Mishra emphasizes starting with data — synthetic data, continuous refinement, and multiple labeled datasets for different flows. To scale, he recommends caching intermediate results, orchestrating parallel runs, and establishing a measure-monitor-analyze-iterate loop, while balancing high-fidelity human-in-the-loop reviews with automated speed. He stresses that no universal Eval exists and process matters more than tools.
