Mechanical Turk

Your Evals Are Meaningless (And Here’s How to Fix Them)

Feb 22, 2025 · 18:50

In this AI Engineering Summit talk, HoneyHive co-founder exposes why most LLM evaluations are meaningless due to criteria drift—where evaluator criteria misalign with user needs—and dataset drift, where test cases don't reflect real-world queries. He argues that static evaluation frameworks from tools like LangChain or Ffragas fail because they measure generic metrics rather than business-specific relevance, citing an e-commerce recommendation system that looked perfect in testing but broke in production. The fix is a three-step iterative alignment process: align evaluators with domain experts via continuous critique and few-shot examples; keep datasets alive by logging production underperformance and flowing those cases back into the test bank; and track alignment over time using F1 scores for binary judgments. Practical advice includes customizing the LLM evaluator prompt, starting with 20 domain expert examples in spreadsheets, and avoiding templated metrics. The speaker emphasizes that evals must evolve continuously, just like the LLM application itself, or they become meaningless.

How to Improve Your Agents: Academic Lit Review

Feb 22, 2025 · 39:02

Joe from Columbia University and founder of Arklex AI explains research on AI agents, focusing on improving their reasoning and planning through self-reflection, test-time compute, and tree search methods, without relying on human supervision. He details the TriPath method, which uses larger models to edit smaller models' feedback for better self-improvement, achieving up to 48% accuracy on math benchmarks. He shows how multi-color tree search (MCTS) with contrastive reflection and multi-agent debate (RMCTS) outperforms other search methods on Visual Web Arena and OS World, achieving top non-trained results. He proposes exploratory learning, where models learn from search trajectories rather than optimal actions, improving performance under compute budgets. Finally, he presents the Arklex open-source agent framework, combining machine learning, systems, and security for practical multi-agent orchestration.