Shipping complex AI applications — Braintrust & Trainline
May 1, 2026 · 1:38:34
Giran Moodley of Braintrust, joined by Trainline's Oussama Hafferssas and Mayank Soni, demonstrate how to ship production-grade multi-step AI agents by combining rigorous tracing, evaluation with golden datasets, and automated online scoring. They walk through building a support triage agent that progresses from a single prompt to a five-stage tool-calling pipeline, where tracing captures latency, tokens, and costs per step for deep debugging. Trainline shares using Braintrust to run offline evaluations before switching LLM models and to enable cross-functional self-service. The workshop covers identifying failure modes via production logs, tightening prompts, and re-evaluating to complete the feedback loop, emphasizing that observability and iterative evaluation are essential for moving from prototype to reliable production systems.
