Iterating on LLM apps at scale Learnings from Discord: Ian Webster
Nov 22, 2024 · 18:26
Ian Webster, Senior Staff Engineer at Discord and maintainer of Promptfoo, shares how Discord built and scaled Klyde AI, a chatbot for 200 million users, focusing on evaluation and safety. He argues that evals should be treated as simple, deterministic unit tests that run locally, avoiding complex metrics, and that breaking the system into small, testable pieces (e.g., checking for lowercase output to enforce casual tone) achieves 80% of the goal with 1% of the work. Webster details how Discord mitigated risks like the 'grandma jailbreak' (which originated on Discord) by using an attacker LLM to generate adversarial inputs and a judge to refine them, exposing cracks in safeguards. He advocates for pre-deployment red teaming over live filtering, and describes using Promptfoo for risk assessment across brand, legal, and safety categories. The episode also covers prompt management via Git and Retool, routing with occasional GPT-4 responses to correct model drift, and the challenge of closing the feedback loop due to privacy constraints, relying instead on dogfooding and public examples.
