Azure Monitor

Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft

May 14, 2026 · 1:20:07

Amy Boyd and Nitya Narasimhan of Microsoft explain how to close the gap between agent behavior and requirements using Microsoft Foundry's observability stack. They demonstrate tracing via OpenTelemetry, built-in evaluators for quality, safety, and agentic metrics (e.g., intent resolution, task adherence), and red teaming where a second AI attacks the agent to reveal vulnerabilities. The showcase is the observe skill: pointed at an agent with no eval data, it generates a dataset, runs batch evaluations, optimizes the prompt, compares versions, and rolls back to the best one—all from a single prompt. The skill surfaces failures developers didn't know existed, accelerating the optimize loop with human-in-the-loop guidance.

Creating and scaling your own custom copilots with Azure AI Studio: Hanchi Wang

Feb 6, 2025 · 24:21

Hanchi Wang, Software Engineer Lead at Azure AI, introduces Azure AI Studio and Promptflow for creating and scaling custom copilots, focusing on tracing, evaluation, and monitoring. He demonstrates a chatbot app that uses the Assistant API with a sales data insight tool (natural language to SQL) and a code interpreter. With Promptflow's trace decorator, developers capture inputs, outputs, and LLM interactions, viewable in a local UI and shareable via Azure AI Studio. For evaluation, he shows synthetic test data sets, content safety evaluators, and custom evaluators like execution time, error rate, and SQL similarity, comparing models such as GPT-4 Turbo, Mistral large, and Phi-3. In production, monitoring dashboards in Application Insights track model duration, token usage (prompt vs. completion), and failure rates, enabling engineers to optimize performance and cost.