Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft
Jun 29, 2026 · 14:10
Tisha Chawla and Susheem Koul of Microsoft argue that debugging production agent failures requires replayability, not bitwise determinism. They illustrate with a stock-selling agent that sold 1,000 shares instead of $1,000 worth due to a reasoning error, a problem temperature-zero decoding cannot fix because GPU non-determinism and batch invariance cause varied outputs. Instead of freezing model output, they propose recording every state transition at method boundaries using their open-source tool Chronicle. Chronicle annotates nodes to capture inputs, outputs, and metadata like model version, then enables deterministic replay by stubbing LLM calls during testing, letting engineers fix guardrails and verify with the same trace. They distinguish deterministic testing (for tools and guardrails) from behavioral testing (e.g., LLM-as-a-judge), advocating for replays over chasing API determinism to make on-call cycles bearable.
