How to Construct Domain Specific LLM Evaluation Systems: Hamel Husain and Emil Sedgh
Sep 19, 2024 · 18:45
Emil Sedgh (CTO at Rechat) and Hamel Husain (independent consultant) describe how they built a domain-specific evaluation system for Lucy, an AI assistant for real estate agents, to move beyond vibe checks and achieve production reliability. They argue that systematic evaluation starts with simple unit tests and assertions based on observed failure modes, logging traces for human review with custom tools to remove friction, and using LLMs to synthetically generate test inputs. The presenters emphasize that focusing on process over tools, avoiding generic off-the-shelf evals, and not jumping too early to LLM-as-a-judge are critical to success. This eval framework enabled Rechat to rapidly increase Lucy's success rate and made possible fine-tuning for complex tasks like mixing structured and unstructured outputs, handling multi-step commands that invoke five or six tools, and incorporating user feedback loops—capabilities they could not achieve with few-shot prompting alone.
