How to evaluate a model for your use case: Emmanuel Turlay
Feb 5, 2025 · 7:32
Emmanuel Turlay, CEO of Sematic, explains why evaluating large language models for specific use cases is more difficult than traditional ML evaluation, as metrics like BLEU and ROUGE and benchmarks like GLUE do not measure task-specific performance. He advocates using another LLM as a grader, describing a workflow where a scoring prompt with grading criteria is fed to a model like GPT-4 (best but costly) or FLAN-T5 (good speed–accuracy trade-off) to numerically score outputs. Turlay demonstrates with a politeness evaluation for email closings and introduces AirTrain, a platform that lets users upload datasets, compare models, and visualize metric distributions to make data-driven LLM selections. The episode argues that teams must build custom evaluation procedures rather than relying on generic benchmarks, treating evaluation as a test suite in the ML development pipeline.
