Azure

Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft

May 14, 2026 · 1:20:07

Amy Boyd and Nitya Narasimhan of Microsoft explain how to close the gap between agent behavior and requirements using Microsoft Foundry's observability stack. They demonstrate tracing via OpenTelemetry, built-in evaluators for quality, safety, and agentic metrics (e.g., intent resolution, task adherence), and red teaming where a second AI attacks the agent to reveal vulnerabilities. The showcase is the observe skill: pointed at an agent with no eval data, it generates a dataset, runs batch evaluations, optimizes the prompt, compares versions, and rolls back to the best one—all from a single prompt. The skill surfaces failures developers didn't know existed, accelerating the optimize loop with human-in-the-loop guidance.

One Login to Rule Them All: Cross-App Access for MCP — Garrett Galow, WorkOS

Apr 28, 2026 · 23:24

Garrett Galow from WorkOS introduces Cross-App Access (XAA) for MCP, solving the problem of repeated OAuth consent screens when connecting agents to multiple services. The flow leverages a three-way trust between the MCP client, server, and an Identity Provider like Okta: a single SSO login issues an IDJag token that is exchanged for short-lived access tokens across all MCP servers without manual intervention. A demo shows Figma automatically connecting after an Okta login. The approach improves security—if the IdP session is revoked, tokens cannot refresh—and requires minimal IT setup: granting permission for the client to request server access. Currently only Okta supports XAA; Azure/Entra does not yet. The session also notes that authorization scopes are not handled by default but are a future consideration.

Why your product needs an AI product manager, and why it should be you — James Lowe, i.AI

Jul 28, 2025 · 18:37

James Lowe, Head of AI Engineering at the UK Government's Incubator for AI, argues that technical expertise makes AI engineers ideal candidates for the critical role of AI Product Manager, which balances business viability, technological feasibility, user desirability, and the core question of possibility. He shares three hard-won lessons from building products like CONSULT, Minute, and Redbox: resolve AI uncertainties early with evaluations and real-user tests (achieving 1,000x faster and 400x cheaper consultation analysis), experiment widely with features then cut back (streamlining a transcription tool from overwhelming options to a focused Justice Transcribe used for Prime Ministerial meetings), and pivot harder than ever as the landscape shifts (Redbox evolved from digitizing ministerial red boxes to a secure LLM chat client, then to an MCP-based tool provider when Microsoft Copilot Chat went free). The episode emphasizes that AI product management is a mindset, not just a title, and that engineers should step into this leadership gap to build impactful AI products.

The Rise of Open Models in the Enterprise — Amir Haghighat, Baseten

Jul 24, 2025 · 16:50

Amir Haghighat, CTO of Baseten, argues that enterprises are increasingly moving from closed frontier models like OpenAI and Anthropic toward open source models, driven by four specific cracks in the assumption that closed models will work indefinitely: quality for specialized tasks (e.g., medical document extraction), latency requirements (especially for voice), unit economics ballooning from agentic use cases where a single user action triggers 50 inference calls, and the desire for competitive differentiation. Drawing on conversations with over 100 enterprises, he explains that while most started with dedicated deployments on Azure/AWS for toying around in 2023, by 2024 about 40-50 had production use cases, and in 2025 the shift accelerated. However, adopting open models forces enterprises to build inference infrastructure, facing challenges like speculative decoding, prefix caching, guaranteeing four-nines reliability with hardware failures and VLM crashes, and scaling replicas—with one Fortune 50 soft drink company reporting an eight-minute spin-up time. Haghighat concludes by contrasting the simple API-call world with the complexities of mission-critical inference, where…

Forget RAG Pipelines—Build Production Ready Agents in 15 Mins: Nina Lopatina, Rajiv Shah, Contextual

Jun 27, 2025 · 1:15:43

Rajiv Shah, Nina Lopatina, and Matthew from Contextual AI demonstrate how developers can build production-ready RAG agents in minutes using Contextual AI's managed RAG platform, emphasizing that RAG should be treated as a managed service to avoid reinventing infrastructure. They walk through ingesting documents like NVIDIA financials and spurious correlation reports, then querying the agent with questions requiring quantitative reasoning across tables. The platform handles extraction, layout analysis, image captioning, hybrid retrieval, and a state-of-the-art reranker, capped by a grounded language model that avoids hallucinations and provides attribution. Evaluation is done via LM Unit, a model-as-judge that scores responses on criteria like accuracy and causation. The episode also shows integrating the agent with Claude Desktop via MCP and answers audience questions on pricing (consumption-based with a $25 credit), scalability, entitlements, HIPAA, and domain-specific language.

Collaborating with Agents in your Software Dev Workflow - Jon Peck & Christopher Harrison, Microsoft

Jun 27, 2025 · 1:04:06

GitHub's Christopher Harrison and Jon Peck explain how to collaborate with Copilot Coding Agent as a peer programmer, emphasizing that context—through clear code, comments, instructions files, and MCP servers—is key to success. They walk through a lab where Copilot works on assigned issues in ephemeral GitHub Actions environments without internet or write access to the main repo, creating draft PRs that require human review. Harrison advises against being passive-aggressive, advocating specific prompts and instructions files to guide the agent. The episode covers MCP's role in providing external tools, the need for code review and security checks even with AI, and pricing: $39.99/month for Enterprise tier with coding agent. They also highlight .instructions files for reusable patterns and note that coding agent currently supports only tools, not resources.

Agentic Excellence: Mastering AI Agent Evals w/ Azure AI Evaluation SDK — Cedric Vidal, Microsoft

Jun 27, 2025 · 20:01

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates how to systematically evaluate AI agents using the Azure AI Evaluation SDK and AI Toolkit. He argues that evaluation must start early, covering four layers: model/safety system, system message, grounding, and user experience. He shows spot-checking models side-by-side in VS Code, building a web-scraping agent with Playwright MCP, and running batch evaluations with built-in quality and safety metrics like groundedness and fluency. Vidal also covers multi-turn and multimodal evaluation, using a violent image example where the SDK returned a score of 4 out of 5, allowing teams to set custom thresholds based on application context.

Building Code First AI Agents with Azure AI Agent Service — Cedric Vidal, Microsoft

Jun 27, 2025 · 1:54:06

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates building code-first AI agents with Azure AI Agent Service, using function calling, code interpreter, file search, and Bing grounding for a sales analysis use case. He creates a conversational agent that queries SQLite, generates pie charts from Python code, and blends product PDF data with relational data. The workshop explains stateful agents, tool routing (LLM generates JSON to call functions), and the limits of single-step agents versus multi-agent orchestration with AutoGen. Cedric addresses when to use Agent Service (managed persistence and tools) versus raw LLM endpoints, and covers MCP servers as a tool lifecycle manager. Key insights include using instructions to ground agent behavior and the need for eval frameworks for agent quality.

Building Protected MCP Servers — Den Delimarsky and Julia Kasper, MCP Steering Committee & Microsoft

Jun 20, 2025 · 20:13

Den Delimarsky and Julia Kasper, Microsoft engineers on the MCP Steering Committee, explain how the new MCP authorization spec simplifies securing remote MCP servers by separating the resource server from the authorization server, eliminating the need for developers to build custom token factories. They argue that developers shouldn't need to be security experts, so the spec offloads token minting and lifecycle management to standard identity providers like Entra ID, Okta, or Auth0. The client now handles the full OAuth 2.0 dance, while the server only validates tokens via a Protected Resource Metadata (PRM) document. A live C# SDK demo shows a server configured with minimal code, and an Azure API Management deployment demonstrates production-ready remote MCP servers with OAuth proxy and consent flows, including integration with VS Code and GitHub Copilot. The session also previews transforming REST APIs into MCP tools via Azure API Management.

Privacy First Enterprise AI: Building AI Agents that Never Leave Your Security Boundary

Feb 22, 2025 · 7:10

Steven Moon, founder of Aech AI, argues that enterprise AI agents should be deployed within existing security boundaries by treating them like human employees—using existing identity management, compliance frameworks, and audit tools rather than building parallel systems. He explains that IT departments will evolve into HR departments for AI agents, provisioning them through active directory and applying standard security policies. Moon highlights email as a powerful medium for agent-to-agent communication, where every interaction is logged and auditable through existing systems. He advocates for enhancing current enterprise platforms like Microsoft 365 and Azure with AI agents instead of creating new interfaces, noting that the era of mandatory translation layers between humans and machines is ending.

Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley

Feb 8, 2025 · 20:46

Humza Iqbal of Snorkel AI explains how the company uses Azure AI infrastructure powered by NVIDIA GPUs to fine-tune foundation models for enterprise customers, achieving better performance per dollar by switching from A100s to H100s. He details their distributed training stack (PyTorch, Horovod, NFS) and lessons learned such as balancing node count for batch size and monitoring GPU utilization to avoid networking or data-loading bottlenecks. A cost comparison found two H100s outperformed four A100s on both training and inference, enabling faster iteration through more synthetic data. Azure's dedicated VMs, reliable NFS throughput, and flexible capacity allowed Snorkel to scale experiments from single-node to dozens of GPUs. Future work includes programmatic preference signals and multimodal retrieval algorithms, all planned on Azure.

Build, Evaluate and Deploy a RAG-Based Retail Copilot with Azure AI: Cedric Vidal and David Smith

Feb 6, 2025 · 1:57:58

David Smith, Cedric Vidal, and Miguel Martinez lead a hands-on workshop on building a production-level RAG-based retail copilot using Azure AI. They demonstrate how to build a chatbot backend that retrieves product information from Azure AI Search via vector embeddings and customer history from Cosmos DB, then augments the LLM prompt to generate grounded answers. The session covers using Azure AI Studio and Prompt flow to orchestrate the RAG workflow, deploying the flow as a managed endpoint, and evaluating quality with GPT-4 as a judge on metrics like relevance and groundedness. The speakers also explain the LLM Ops lifecycle for iterative improvement and compare Prompt flow with Semantic Kernel and AutoGen.

AI Templates: Gabriela and Aishwarya

Feb 6, 2025 · 1:03:20

Gabriela de Queiroz, Aishwarya, and Pamela from Microsoft present AI templates for rapidly prototyping and deploying generative AI applications, demonstrating how startups can leverage Microsoft for Startups Founders Hub, including up to $150,000 in Azure credits and expert guidance. The workshop covers three templates: a simple chat app using GPT-3.5 Turbo, a Retrieval Augmented Generation (RAG) app on Postgres with hybrid vector and text search, and a RAG app on unstructured documents using Azure AI Search and Document Intelligence. Key insights include the importance of query rewriting, hybrid search over pure vector search, and streaming responses for better user experience. The templates are open source on GitHub and can be deployed via Codespaces, with a proxy provided to bypass Azure OpenAI approval delays.

Scaling AI in Education: A Khanmigo case study: Shawn Jansepar

Feb 5, 2025 · 22:39

Shawn Jansepar, Khan Academy's Director of Engineering, details building Khanmigo, an AI tutor and teacher assistant on GPT-4 with OpenAI, arguing generative AI can democratize one-on-one tutoring. He describes a rapid prototyping culture that launched Khanmigo in three months via a company-wide hackathon, replacing traditional agile with a prototype-to-beta-to-launch framework. Technical challenges include math accuracy through a math agent and chain-of-thought prompting, refactoring prompts into a component architecture for testing, and managing scale using multiple models and dedicated Azure compute. Jansepar highlights ethical design: a Socratic tutor that avoids answers, teacher moderation, and a writing coach with revision history. Khanmigo now has 200,000 paid users, half in school districts, with teacher tools sponsored by Microsoft free to US teachers. Plans include releasing a math tutoring benchmark and evaluating smaller models like Phi-3.

E-Values Evaluating the Values of AI: Sheila Gulati and Nischal Nadhamuni

Dec 31, 2024 · 28:13

Sheila Gulati (Tola Capital) and Nischal Nadhamuni (Klarity CTO) argue that evaluating AI systems is at a seminal moment, requiring a shift from simplistic benchmarks to multi-faceted, user-centric approaches. Gulati highlights the inadequacy of current leaderboards, citing benchmark hacking and the need for dynamic datasets and value alignment. Nadhamuni shares how Klarity, which raised a $70M Series B, tackles eval challenges in its document-processing product—processing over 500K documents for customers with 15+ LLM use cases. He details practical strategies: front-loading UX risk, thinking backwards from user experience, reducing degrees of freedom by standardizing prompt engineering across customers, and building scrappy future-facing evals. The episode emphasizes that the AI engineer community must innovate benchmarks, bring depth and community sharing to evaluations, and introspect the values instilled in AI systems.

Decoding Mistral AI's Large Language Models: Devendra Chaplot

Nov 21, 2024 · 18:16

Devendra Singh Chaplot of Mistral AI details the company's open-source large language models, including Mistral 7B, Mixtral 8x7B, Mixtral 8x22B, and CodeStral 22B, arguing that open models complement rather than compete with profit by serving as branding tools and driving customer acquisition for proprietary upgrades. He explains the three-stage LLM training process—pre-training on trillions of tokens, instruction tuning with prompt-response pairs, and learning from human feedback via preference optimization—emphasizing that more data does not guarantee better performance due to noise. The episode highlights Mistral's focus on optimizing the performance-to-cost ratio, with CodeStral 22B outperforming larger models like Code LLaMA 70B while being smaller and multilingual across 80+ programming languages. Practical guidance is given: prototype with high-end commercial models, then fine-tune open models for specific tasks to balance performance and cost.

Building and Scaling an AI Agent Swarm of low latency real time voice bots: Damien Murphy

Oct 8, 2024 · 1:07:23

Damien Murphy, Senior Applied Engineer at Deepgram, demonstrates building and scaling low-latency real-time voice bots using Deepgram's new voice agent API, which wraps speech-to-text, LLM, and text-to-speech into a single streaming endpoint. He shows a drive-thru ordering demo with function calling (add/remove items) using GPT-4o, achieving sub-second latency by co-locating components. Murphy explains scaling to millions of concurrent calls through regional Kubernetes clusters and multi-agent swarms (routing, booking, support agents) to reduce complexity and cost. He addresses endpointing challenges, VAD-based barge-in, and cost/quality trade-offs with hosted vs. self-hosted models, noting Deepgram offers 20x cheaper TTS than ElevenLabs and 50ms STT latency self-hosted. The talk emphasizes keeping agents simple, using smallest capable LLMs, and composability for reuse.

Running AI Application in Minutes w/ AI Templates: Gabriela de Queiroz, Pamela Fox, Harald Kirschner

Aug 14, 2024 · 1:29:02

In this workshop, Microsoft's Gabriela de Queiroz, Pamela Fox, and Harald Kirschner show how to deploy AI applications in minutes using AI templates, Azure OpenAI proxy, and GitHub Codespaces. They walk through deploying a simple chat app, then two RAG applications: one that queries a Postgres database with SQL filtering, and another that performs RAG on unstructured documents using Azure AI Search. Key decisions include using async frameworks like Quart and FastAPI, token-based chunking, and hybrid retrieval with semantic reranking for best results. They stress the importance of running evaluations with hundreds of samples and share production insights from Copilot Chat, where TFIDF sparse indexing and LLM reranking are used. The session includes free Azure credits and a proxy to bypass Azure OpenAI approval, allowing attendees to deploy everything without spending their own money.