Your Evals Are Meaningless (And Here’s How to Fix Them)
Feb 22, 2025 · 18:50
In this AI Engineering Summit talk, HoneyHive co-founder exposes why most LLM evaluations are meaningless due to criteria drift—where evaluator criteria misalign with user needs—and dataset drift, where test cases don't reflect real-world queries. He argues that static evaluation frameworks from tools like LangChain or Ffragas fail because they measure generic metrics rather than business-specific relevance, citing an e-commerce recommendation system that looked perfect in testing but broke in production. The fix is a three-step iterative alignment process: align evaluators with domain experts via continuous critique and few-shot examples; keep datasets alive by logging production underperformance and flowing those cases back into the test bank; and track alignment over time using F1 scores for binary judgments. Practical advice includes customizing the LLM evaluator prompt, starting with 20 domain expert examples in spreadsheets, and avoiding templated metrics. The speaker emphasizes that evals must evolve continuously, just like the LLM application itself, or they become meaningless.
