Hamel Husain

How to Fail at AI Strategy: Hamel Husain & Greg Ceccarelli

Apr 13, 2025 · 17:03

Greg Ceccarelli and Hamel Husain argue that the most reliable path to AI failure is to follow a set of inverted worst practices, including cultivating disconnect between executives and builders, promising unrealistic AI capabilities, drowning communication in jargon, and avoiding data analysis. They describe how to divide your company by incentivizing secrecy and using jargon like 'agents' to exclude domain experts, ensuring that AI projects are disconnected from real needs. The speakers advocate faking strategy by highlighting random paragraphs from last year's report, announcing vague goals like 'become the global AI leader in everything,' and creating a massive backlog with no timeline. They recommend throwing tools at problems—buying expensive vector databases or switching frameworks—without understanding root causes, and blindly trusting off-the-shelf evaluation metrics like BLEU and ROUGE. Crucially, they insist on never looking at data, using complex systems inaccessible to domain experts, and trusting gut feelings over evidence. This inverted guide guarantees wasted resources, alienated teams, and spectacular failure.

How to Construct Domain Specific LLM Evaluation Systems: Hamel Husain and Emil Sedgh

Sep 19, 2024 · 18:45

Emil Sedgh (CTO at Rechat) and Hamel Husain (independent consultant) describe how they built a domain-specific evaluation system for Lucy, an AI assistant for real estate agents, to move beyond vibe checks and achieve production reliability. They argue that systematic evaluation starts with simple unit tests and assertions based on observed failure modes, logging traces for human review with custom tools to remove friction, and using LLMs to synthetically generate test inputs. The presenters emphasize that focusing on process over tools, avoiding generic off-the-shelf evals, and not jumping too early to LLM-as-a-judge are critical to success. This eval framework enabled Rechat to rapidly increase Lucy's success rate and made possible fine-tuning for complex tasks like mixing structured and unstructured outputs, handling multi-step commands that invoke five or six tools, and incorporating user feedback loops—capabilities they could not achieve with few-shot prompting alone.

Lessons From A Year Building With LLMs

Jul 19, 2024 · 35:21

The six authors of the O'Reilly article "Lessons From A Year Building With LLMs" — Bryan Bischof, Jason Liu, Hamel Husain, Eugene Yan, Shreya Shankar, and Swix — argue that the model itself is not a moat and that success comes from continuous improvement centered on evals and data. They stress that AI engineers should treat models as SaaS, quickly swapping for better ones, and focus on product and user interactions. The talk warns against toxic practices like prematurely hiring ML engineers without data or blindly adopting tools, advocating instead for deliberate eval practice and data literacy. On tactics, they compare LLM-as-judge (quick to prototype) vs fine-tuned evaluators (more precise and faster), and emphasize looking at real user data regularly with automated guardrails. Ultimately, they conclude that going from demo to production requires sustained investment in infrastructure and evaluation, echoing MLOps lessons from a decade ago.