Duolingo

Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

Jul 7, 2026 · 25:53

Angel Ortmann Lee, a Software Engineer at the Duolingo English Test, argues that human-in-the-loop AI systems often fail because humans cognitively surrender—adopting AI output with minimal scrutiny, as seen in a Wharton study where 80% accepted wrong AI answers. At Duolingo, an experiment with fake AI cheating alerts showed skilled proctors flagged legitimate sessions 50% of the time due to automation bias. A simple guideline change emphasizing independent evidence boosted accurate rejections by 21%, proving the fix lies in engineering the interaction, not better models or oversight. Lee outlines design principles: engineer reasoning patterns (e.g., frame humans as investigators), match friction to stakes (add review gates for high-stakes decisions), treat every interaction as a label (capture diffs when humans override AI), and proactively define success metrics to structure interactions that yield high-quality data. The flywheel of intentional design yields a virtuous cycle of better data, models, and human judgment.

Shipping AI That Works: An Evaluation Framework for PMs – Aman Khan, Arize

Dec 26, 2025 · 1:26:16

Aman Khan, AI PM at Arize, presents a framework for product managers to evaluate LLM-powered products beyond gut-feel 'vibe checks.' He demonstrates building an AI trip planner with multi-agent LangGraph, then using Arize's tracing and prompt playground to iterate on prompts. Khan shows how to create datasets from production traces, run A/B experiments on prompts, and use LLM-as-a-judge evals for friendliness and discount offers, comparing against human labels to refine evaluators. He argues evals are the new requirements docs, enabling PMs to own the product experience by writing acceptance criteria as eval datasets. The talk covers building eval teams, handling variance with temperature settings, and continuously improving golden datasets with hard examples, citing real-world analogies from self-driving cars at Cruise.

Structuring a modern AI team — Denys Linkov, Wisedocs

Jul 24, 2025 · 17:40

Denys Linkov, who leads ML at Wisedocs, argues that building a modern AI team hinges on identifying your company's bottleneck—shipping features, acquiring users, or scalability—rather than reflexively hiring AI researchers. He introduces Ampere's Wager: trading your entire domain-savvy team for five top-lab researchers is usually a losing bet. For early-stage AI strategy, generalists who blend model training, serving, and business acumen outperform specialists; Linkov lived this in 2021 building a custom MLOps platform for a conversational AI startup and again in 2024 using advanced open-source tools for medical record processing. He stresses reskilling existing teams through weekly learning cadences and moving domain experts from giving feedback to writing evaluations. Hiring should hold context and act on it, verifying trends like 'don't hire juniors' against YC's AI school drawing 2,000 young people.

Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work — Dat Ngo, Aman Khan, Arize

Jun 27, 2025 · 24:46

Dat Ngo, AI architect at Arize AI, presents advanced LLM evaluation strategies for production systems, arguing that effective evals go beyond out-of-the-box LLM-as-a-judge to include code-based heuristics, human feedback, and golden datasets. He explains how to build a virtuous cycle of collecting observability data, running evals, and tuning them over time, and demonstrates agent evaluation techniques like trajectory evals to identify failure modes across complex workflows. Ngo covers trade-offs between offline evals and inline guardrails, the use of log probabilities for confidence scoring, and automated prompt optimization through meta-prompting, all illustrated with customer examples from Reddit, Duolingo, and Booking.com.

Real ROI: Lessons from Enterprises that have already succeeded with LLMs at Scale: Raza Habib

Dec 31, 2024 · 20:01

Raza Habib, CEO of Humanloop, shares lessons from enterprises like Duolingo, Filevine, and Ironclad that have achieved real ROI with LLMs at scale. He argues that success hinges on centering domain experts (e.g., Duolingo's linguists do all prompt engineering), needing less ML expertise than expected, and breaking down evaluation into small, testable components rather than chasing a single metric. Habib emphasizes collecting end-user feedback (acceptance rates, edits, thumbs up/down) and building tooling for logging, regression testing, and team collaboration. Filevine doubled revenue by launching six LLM-powered products; Ironclad's open-source Rivet logging tool enabled agents to auto-negotiate 50% of contracts. The talk covers how to optimize the four-component chain (base model, prompt, data selection, function calling) and why simpler systems will win as models improve.