Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop) — Taylor Jordan Smith
Jul 27, 2025 · 32:28
Taylor Jordan Smith from Red Hat presents a hands-on workshop on evaluating large language models (LLMs) for production, using three open-source tools: GuideLLM for system performance benchmarks (latency, throughput), lm-eval-harness for factual accuracy via MMLU Pro, and promptfoo for safety and bias custom evaluations. He argues that traditional benchmarks are insufficient and advocates for a layered evaluation pyramid—starting with system performance, then factual accuracy, then safety/bias—taking an incremental approach similar to software testing (unit, integration, end-to-end). He demonstrates deploying an IBM Granite 2B model with vLLM on an L4 GPU, showing how to adjust input/output tokens for use cases like chatbots or RAG. The episode emphasizes that evaluations must be tailored to the specific system (RAG, agents, etc.) and continuously integrated into CI/CD pipelines to manage risk, cost, and reliability in production. Attendees gain actionable strategies for custom eval suites and human-in-the-loop feedback.
