David Karam

Layering every technique in RAG, one query at a time - David Karam, Pi Labs (fmr. Google Search)

Jul 29, 2025 · 20:22

David Karam of Pi Labs (formerly Google Search) walks through layering every technique in RAG, from in-memory retrieval to planet-scale search with 70+ corpus mix of token, embeddings, and knowledge graphs, jointly retrieved and re-ranked at 160,000 queries per second in under 200msec. He advocates a Quality Engineering Loop: baseline with simplest methods, analyze losses, then apply incremental techniques based on complexity-adjusted impact. Karam explains why queries like "falafel" are notoriously hard due to ambiguous intent, highlights failures of chunking documents, and shows when BM25 suffices, when relevance embeddings are needed, and when custom embeddings or domain-specific signals (price, popularity, user clicks) become essential. He covers query orchestration via fan-out, supplementary retrieval across back ends, and distillation for cost optimization, concluding that at sufficient complexity, problems must be punted to LLM or UX for graceful degradation.

[Full Workshop] Building Metrics that actually work — David Karam, Pi Labs (fmr Google Search)

Jul 29, 2025 · 40:28

Achim and David Karam, former Google Search leaders now at Pi Labs, argue that building reliable AI evaluations requires a multi-signal scoring system rather than single LLM-as-judge metrics. They walk through designing custom metrics starting with simple correlated signals, using a copilot to generate and iterate on dimensions like 'includes insights' or 'title length,' then calibrating them against ground truth thumbs up/down data from users. The workshop demonstrates applying this to a meeting summarizer, integrating the scoring system into Google Sheets to score 120 examples and produce a confusion matrix showing alignment with human feedback. They also show how to use the same scoring system for model comparison (e.g., GPT-1.5 vs 2.5) and online best-of-N sampling, which boosts response quality by generating multiple responses and selecting the highest-scored one.