The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks
Jun 18, 2026 · 37:06
Sandipan Bhaumik, a technical lead for Data and AI at Databricks, presents a five‑pillar playbook for taking AI agents to production: evaluation (define numerical success before touching code), observability (trace every decision for regulators and debugging), data foundation (agents do not forgive bad data), multi‑agent orchestration patterns (orchestrator‑worker, choreography, human‑in‑the‑loop), and governance (PII pre‑validation, prompt versioning as change management). He recounts a retail bank that spent £85,000 over six months on a chatbot PoC that failed because no one could measure or trace it. His team reversed the order: they built the evaluation dataset and tracing infrastructure first, selected the model in week 7 of an 8‑week engagement, and launched successfully. Six weeks post‑launch, when the bank updated interest rate policies, the tracing system caught that the new document had not been re‑embedded, so the agent served stale answers—a production incident the five pillars were designed to handle. The evaluation dataset is a living system that grows from 200 test cases; a production incident playbook connects all pillars: detect via eval dashboard, diagnose with…
