Microsoft

Build Evals That Actually Matter - Nick Ung, Lyft

Jul 19, 2026 · 37:45

Nick Ung and Akshay from Lyft explain how to build evals that actually matter for customer-support AI agents, sharing their end-to-end pipeline. They argue offline evals must use a fine-tuned user simulator trained on real rider and driver transcripts to produce messy, frustrated behavior—not an off-the-shelf LLM that sounds too nice. Their LLM judge is treated as a binary classifier, calibrated against human labels using precision and recall, with metrics tied to actionable business outcomes like escalation decisions. They detail a continuous error-analysis loop that feeds production failures back into the offline test set and closes the loop through context, harness, or model learning. The talk also previews their config-driven eval harness with YAML primitives for tasks, datasets, and personas, enabling repeatable runs across development stages.

On AI and Knowledge — Pablo Castro, Distinguished Engineer & CVP for AI Knowledge, Microsoft

Jul 17, 2026 · 17:35

Pablo Castro, Distinguished Engineer and CVP for AI Knowledge at Microsoft, argues that AI agents require three knowledge types—intrinsic (model training), extrinsic (RAG and grounding), and learned (optimization loops)—and demonstrates how Microsoft's Foundry IQ, Azure AI Search, and Agent Optimizer deliver them. He shows that combining retrieval methods outperforms individual ones, citing Azure AI Search evaluations where combined methods improve evidence recall and answer completeness. He introduces Foundry IQ as a layered system with agentic retrieval that reflects on data to satisfy information needs before returning results. Castro demonstrates creating a knowledge base that connects unstructured data, structured Parquet tables, and the web, then linking it to an agent via MCP server. He also showcases the Agent Optimizer, which uses hill climbing on baseline evaluations to automatically refine agent instructions, tool definitions, and skills based on runtime traces, enabling continuous learning loops that capture organizational differentiation.

The AI bugpocalypse is here. Now what? - Jack Cable, Corridor

Jul 12, 2026 · 19:44

Jack Cable, Co-Founder and CEO of Corridor, argues that frontier AI models can now find and exploit software vulnerabilities at scale, creating an 'AI bugpocalypse' that demands fundamental security changes. He cites 84% of developers using AI coding tools, with models like Mythos outperforming humans in finding vulnerabilities. He notes that basic vulnerability classes like buffer overflows remain prevalent despite decades of known fixes, and that memory-safe languages like Rust can eliminate 60-70% of such bugs, as shown by Android's drop from 75% memory safety vulnerabilities in 2019 to 30% in 2022. He warns that even advanced models introduce vulnerabilities 20-40% of the time (backsbench.com) and points to an Opus 4.6 incident that lost $2M. His recommendations to Congress include preventing vulnerabilities in new code, hardening open-source software, and fostering American open-weight models, arguing that defenders need access to these models to stay ahead.

From Writing Code to Designing Systems: How the Developer Role is Changing — Chris Noring, Microsoft

Jul 11, 2026 · 23:05

Chris Noring from Microsoft argues that the developer role is shifting from writing code to designing systems and orchestrating AI agents, using tools like GitHub Copilot and Claude. He proposes a workflow starting with the CLI rather than the editor, employing agents.md for high-level guidance, skills for repeatable tasks, and custom agents for orchestration. Noring demonstrates scaling by delegating tasks via /delegate in the CLI or assigning issues to agents in the GitHub UI, allowing developers to become 20x more productive. He stresses that guardrails are essential to prevent agents from producing 'slop', and that human-in-the-loop oversight remains critical. The episode emphasizes that developers must encode standards and constraints into their workflows to maintain consistency and quality at scale.

Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

Jun 8, 2026 · 15:50

Max Ryabinin from Together AI presents their research on extending transformer context length to 5 million tokens using Untied Ulysses, which cuts activation memory by reusing buffers across attention head iterations. The talk walks through a stack of techniques including fully sharded data parallelism, DeepSpeed Ulysses context parallelism for an 8x activation reduction, activation checkpointing for another 8x, CPU offloading of transformer block inputs, and chunked sequence training. Even with these, training a LLaMA 3B model with 3 million tokens fits on an 8xH100 node, but 5 million requires Untied Ulysses. Instead of allocating one large buffer per attention head group, it chunks heads further and reuses buffers across iterations, cutting activation memory with negligible throughput impact. At both 8B and 32B scale, results match the most memory-optimized transformer training baselines while pushing sequence length 25% further than prior Ulysses implementations.

Building Interactive UIs in VS Code with MCP Apps — Marlene Mhangami & Liam Hampton, GitHub

Jun 6, 2026 · 16:06

Marlene Mhangami and Liam Hampton from GitHub explain MCP Apps, which let server tools return rich interactive UI components inside VS Code Chat via sandboxed iframes, replacing plain text responses. They demonstrate a flame graph profiler that profiles a Go app running bubble sort and Fibonacci, rendering an interactive flame graph directly in the chat window. The MCP server returns data and a resource reference to bundled HTML; VS Code fetches and renders it, enabling live interaction like clicking and querying without leaving chat. Use cases include data exploration, e-commerce checkouts (Shopify), and interactive diagrams (Excalidraw). The iframe is sandboxed for security—like putting a hamster in a cage to prevent it from modifying VS Code settings or calling external APIs. Marlene and Liam walk through building an MCP App from scratch using a skill from the model context protocol repository.

Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft

May 16, 2026 · 19:45

Marlene Mhangami of Microsoft and GitHub argues that AI boosts developer productivity only when paired with clean code practices, citing a Stanford study of 120,000 developers and GitHub Octoverse data showing 275 million weekly commits in 2026 with a growing share co-authored by AI. She advocates test-driven development (TDD) with Playwright for functional testing over unit tests, as AI-generated tests often affirm code behavior rather than user experience. In a live demo, she uses GitHub Copilot CLI and Playwright MCP server to automatically write failing Playwright tests for a toy store's search and filter features, then generate code to pass them. She emphasizes testing one feature per test, capturing screenshots for PRs, and committing code before fixes to preserve context. She closes with advice on using Playwright agents for complex state management and confirming cross-screen-size support.

How AI is changing Software Engineering: A Conversation with Gergely Orosz, @pragmaticengineer

Apr 21, 2026 · 26:42

Gergely Orosz explains how AI is reshaping software engineering, from token maxing at Meta, Microsoft, and Salesforce—where engineers use AI tools to inflate token counts out of fear of layoffs and performance reviews—to the broader shift in the engineer's role toward orchestrating agents rather than managing people. He notes that big tech companies like Uber, Airbnb, and Shopify are building custom internal AI infra (MCP gateways, coding agents) to stay ahead, even tolerating high churn for a six-month competitive edge. Shopify secured GitHub Copilot a year early by offering feedback from 3,000 engineers. Gergely also shares how The Pragmatic Engineer hit product-market fit: 100 paid subscribers before publishing, reaching 1,000 in six weeks, and becoming the #1 paid tech newsletter by focusing on two deep-dive articles per week for two years.

State of the Claw — Peter Steinberger

Apr 17, 2026 · 44:12

Peter Steinberger, creator of OpenClaw, presents a five-month update on the world's fastest-growing open-source project. He details the project's staggering growth—30,000 commits, nearly 2,000 contributors—and the immense security burden: 1,142 advisories (16.6 per day) with 99 critical, often AI-generated slop that demands human vetting. Steinberger refutes media fearmongering, citing how researchers ignore security docs to fabricate scary scenarios. He clarifies OpenAI did not buy OpenClaw; he joined the company while establishing the OpenClaw Foundation to remain vendor-neutral. He emphasizes the importance of local models for data sovereignty and describes his coding workflow of running 5-6 agent sessions simultaneously, iterating on taste and personality. Future visions include ubiquitous agents, 'Dreaming' for memory reconciliation, and modular plugins. For engineers, he champions taste, system design, and learning to say no.

$1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs – Diego Carpentero

Apr 16, 2026 · 43:53

Diego Carpentero argues that LLM-based attacks—Prompt Injection, Indirect Injection, Model Internals (gibberish suffix), RAG Poisoning, MCP Exploits, and Agentic Escalation—are now the baseline, not the exception, and that model alignment and human review alone are insufficient. He identifies the core problem as a Zero Trust Gap: LLMs natively lack separation between system controls and data, allowing adversaries to override decisions via malicious instructions in inputs or external content. To build a protective layer, Carpentero fine-tunes ModernBERT—a state-of-the-art encoder with Alternating Attention, Unpadding & Sequence Packing, RoPE, and FlashAttention—into a safety discriminator that classifies prompts as safe or unsafe in ~35 milliseconds with 85% accuracy, all for under a dollar. He walks through the fine-tuning pipeline using the IngetGuard dataset and demonstrates live detection of real attack examples from each vector.

AI Didn’t Kill the Web, It Moved in! — Olivier Leplus (AWS) & Yohan Lasorsa (Microsoft)

Apr 10, 2026 · 52:55

Yohan Lasorsa (Microsoft) and Olivier Leplus (AWS) argue that AI isn't replacing the web—it's becoming its native layer, embedded in every stage of development. They demo coding agents that use lightweight 'skills' to automate workflows like fetching GitHub issues and testing via Playwright, then show Chrome's MCP server letting agents debug performance across different network conditions. New on-device Web AI APIs (summarizer, proofreader, prompt API) run models locally in the browser, enabling features like auto-writing reviews from an uploaded product image without cloud calls. Finally, they introduce LLMs.txt for guiding AI agents to relevant documentation and Web MCP—a proposal to register native tools on websites (e.g., 'add to cart') so agentic browsers can execute actions without mimicking human clicks. The episode delivers concrete implementations, from coding to deploying AI-ready web apps.

Shipping AI That Works: An Evaluation Framework for PMs – Aman Khan, Arize

Dec 26, 2025 · 1:26:16

Aman Khan, AI PM at Arize, presents a framework for product managers to evaluate LLM-powered products beyond gut-feel 'vibe checks.' He demonstrates building an AI trip planner with multi-agent LangGraph, then using Arize's tracing and prompt playground to iterate on prompts. Khan shows how to create datasets from production traces, run A/B experiments on prompts, and use LLM-as-a-judge evals for friendliness and discount offers, comparing against human labels to refine evaluators. He argues evals are the new requirements docs, enabling PMs to own the product experience by writing acceptance criteria as eval datasets. The talk covers building eval teams, handling variance with temperature settings, and continuously improving golden datasets with hard examples, citing real-world analogies from self-driving cars at Cruise.

Leadership in AI Assisted Engineering – Justin Reock, DX (acq. Atlassian)

Dec 19, 2025 · 18:11

Justin Reock, Deputy CTO at DX (acquired by Atlassian), argues that AI's impact on engineering productivity varies wildly and that leaders must move beyond top-down mandates to focus on psychological safety, measurement of actual outcomes, and targeted integration across the SDLC. He presents data showing a 2.6% average increase in change confidence but extreme variability across companies, with some seeing 20% drops. Emphasizes that writing code is rarely the bottleneck; instead, leaders should identify and fix bottlenecks like context switching, citing Morgan Stanley's DevGenAI saving 300,000 hours annually by converting legacy code specs and Zapier reducing engineer onboarding to two weeks via AI agents. Introduces DX's AI Measurement Framework covering utilization, impact, and cost, and stresses trust-building through system prompt feedback loops and temperature settings. The episode delivers actionable guidance on measuring AI's true impact and enabling engineers through education, time to learn, and creative unblocking of usage.

2025 is the Year of Evals! Just like 2024, and 2023, and … — John Dickerson, CEO Mozilla AI

Aug 6, 2025 · 19:14

John Dickerson, CEO of Mozilla AI and former co-founder of Arthur AI, argues that 2025 is finally the year for AI evaluation (evals) due to three converging forces: ChatGPT making AI tangible to the C-suite, enterprise budget freezes that funnelled funding into GenAI pet projects, and the rise of agentic systems acting autonomously. He traces how ML monitoring existed pre-2022 but lacked C-suite attention, with Jamie Dimon’s JPMC spending only $100 million on AI from 2017 to 2021. The ChatGPT launch on November 30, 2022, unlocked CEO discretionary budgets for GenAI, leading to 2023 science projects, 2024 production deployments, and 2025 scaling. Now, as agents reason and act, connecting evals to downstream business KPIs—risk mitigation, revenue gains—is a first-class discussion for CEOs, CFOs, CISOs, and CTOs. Dickerson notes that evaluation companies have shifted to multi-agent system monitoring, and while LLM-as-judge is popular, biases require human validation, with companies like Mercor hiring experts at $50–200/hour to lockstep multi-agent outputs.

Shipping something to someone always wins — Kenneth Auchenberg (ex. Stripe, VSCode)

Jul 28, 2025 · 16:17

Kenneth Auchenberg, former VSCode and Stripe developer platform lead, argues that great AI products come from rapid iterative loops — shipping continuously viable 'skateboards' to real users rather than aiming for a perfect final product. He advocates for writing launch blog posts before building, working intimately with a few real customers (even texting them), and ignoring constraints like legal initially to design the best product. APIs are harder to change than UI, so early user feedback is critical. In AI, the fundamentals haven't changed: customer knowledge and iteration velocity matter more than ever, even as AI tools accelerate building. The goal should be to run a product feedback loop in under a day.

Why your product needs an AI product manager, and why it should be you — James Lowe, i.AI

Jul 28, 2025 · 18:37

James Lowe, Head of AI Engineering at the UK Government's Incubator for AI, argues that technical expertise makes AI engineers ideal candidates for the critical role of AI Product Manager, which balances business viability, technological feasibility, user desirability, and the core question of possibility. He shares three hard-won lessons from building products like CONSULT, Minute, and Redbox: resolve AI uncertainties early with evaluations and real-user tests (achieving 1,000x faster and 400x cheaper consultation analysis), experiment widely with features then cut back (streamlining a transcription tool from overwhelming options to a focused Justice Transcribe used for Prime Ministerial meetings), and pivot harder than ever as the landscape shifts (Redbox evolved from digitizing ministerial red boxes to a secure LLM chat client, then to an MCP-based tool provider when Microsoft Copilot Chat went free). The episode emphasizes that AI product management is a mindset, not just a title, and that engineers should step into this leadership gap to build impactful AI products.

Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek R1/V3) — VIbhu Sapra

Jul 25, 2025 · 53:54

VIbhu Sapra presents DeepSeek's latest models and announces a new curriculum-based Test of Time Paper Club. DeepSeek's May 28th R1 update doubles reasoning tokens (12k→25k) and matches O3/Gemini 2.5 Pro, improving AIME 2024 from 70% to 87.5%. A new distillation of R1 traces into Qwen 3 8b achieves performance on par with Qwen's 235b thinking model, proving reasoning models distill efficiently. The Test of Time Paper Club will cover 50–100 foundational papers over six months in San Francisco and remote, targeting core AI engineering topics from attention and scaling laws to inference techniques. Sapra emphasizes that pure RL (GRPO) on verifiable math and code unlocks emergent reflection and aha moments, shifting scaling from pre-training compute to inference-time reasoning.

Building AI Products That Actually Work — Ben Hylak (Raindrop), Sid Bendre (Oleve)

Jul 24, 2025 · 18:42

Ben Hylak (Raindrop) and Sid Bendre (Oleve) argue that building reliable AI products requires iterative real-world signals over traditional evals. Ben debunks eval myths—evals don't measure product quality, LLM-as-judge fails, and production evals are costly—and stresses tracking explicit signals (thumbs up/down, copy rate) and implicit signals (refusals, frustration) to identify issues. Sid introduces Trellis, a framework for scaling viral AI apps that uses discretization—breaking infinite output into intent buckets—prioritization by volume times negative sentiment times achievable delta, and recursive refinement. Starting with an MVP, teams classify user intents, convert them into semi-deterministic workflows, then repeatedly drill into sub-intents to engineer repeatable, attributable magic. Oleve's approach, powered by Raindrop, has scaled six viral products to $6M ARR profitably with four people.

Building Applications with AI Agents — Michael Albada, Microsoft

Jul 24, 2025 · 15:50

Michael Albada, Principal Applied Scientist at Microsoft, explains how to build effective AI agent systems, defining agents as entities that reason, act, communicate, and adapt. He emphasizes that tool use requires exposing grouped, human-facing functions rather than one-to-one API mappings, noting a 254% increase in agentic Y Combinator startups. Orchestration should start simple with chains and trees before moving to fully agentic patterns, using deterministic logic for safety and external state management. For multi-agent systems, he recommends splitting tools into semantically similar groups to avoid overwhelming a single agent. Albada stresses investing in evaluation via open-source tools like Intel Agent, Pirate, Label Studio, and automatic optimization with Trace, TextGrad, and DSPY. He warns against common pitfalls: insufficient evals, poorly designed tools, excessive complexity, and lack of observability, while urging design for security with tripwires and fallback to human review.

Practical GraphRAG: Making LLMs smarter with Knowledge Graphs — Michael, Jesus, and Stephen, Neo4j

Jul 22, 2025 · 19:46

Michael Hunger and Stephen Chin of Neo4j present GraphRAG as a method to enhance LLMs by integrating knowledge graphs, achieving more accurate, contextual, and explainable answers than standard RAG. They highlight limitations of vector search, showing GraphRAG improves relevance and reduces hallucinations. They detail a three-step construction process: lexical graphs, entity extraction, graph enrichment with algorithms. They demonstrate open-source tools like Neo4j Knowledge Graph Builder and GraphRAG Python package, and show an agentic approach using domain-specific retrievals. They cite studies showing three times improvement in accuracy.

UX Design Principles for Semi Autonomous Multi Agent Systems — Victor Dibia, Microsoft

Jul 21, 2025 · 20:28

Victor Dibia, Principal Research Software Engineer at Microsoft Research, presents four UX design principles for semi-autonomous multi-agent systems: capability discovery, observability and provenance, interruptibility, and cost-aware action delegation. He demonstrates these through Blender LM, a multi-agent system built from scratch that translates natural language into 3D Blender scenes, featuring a planner and verify agent. Dibia stresses eval-driven design—starting with a non-AI baseline, then iterating with agents—and warns that only a small subset of tasks truly benefit from multi-agent autonomy, advocating for careful ROI assessment. He also showcases AutoGen Studio, a low-code tool for composing multi-agent workflows, and shares code and resources for further learning.

CIAM for AI: Authn/Authz for Agents — Michael Grinich, CEO of WorkOS

Jul 21, 2025 · 20:13

Michael Grinich, CEO of WorkOS, argues that AI agents need first-class identity and access management, distinct from human or machine-to-machine auth. He identifies key challenges: headless login, least privilege for non-deterministic systems, and compliance tracking. Grinich presents four emerging patterns—persona shadowing, delegation chains, capability tokens, and human-in-the-loop escalation—and references standards like OAuth, UMA, GNAP, OIDC for agents, and verifiable credentials. He predicts a shift from 95% human traffic to 95% agent traffic, calling for middleware trust boundaries and urgent collaboration on agent identity standards.

Full Spec MCP: Hidden Capabilities of the MCP spec — Harald Kirschner, Microsoft/VSCode

Jul 18, 2025 · 14:53

Harald Kirschner from Microsoft/VSCode argues that MCP's full specification unlocks powerful stateful interactions beyond the common 'tools-only' implementations, transforming AI assistants into more contextual and efficient agents. He highlights underused primitives like resources for rich data context and sampling for server-requested LLM completions, demonstrated via a dungeon game where dynamic tool discovery adapts to game state. VS Code's upcoming full spec support includes dynamic tool discovery, user-defined tool sets, a debug mode for server development, and support for streamable HTTP to reduce stateful server churn. Upcoming features like elicitations will allow tools to request user input directly. Kirschner calls on developers to build progressive, full-spec servers and contribute feedback to the open ecosystem, emphasizing that client and SDK support will follow as usage grows.

Foundry Local: Cutting-Edge AI experiences on device with ONNX Runtime/Olive — Emma Ning, Microsoft

Jun 27, 2025 · 22:52

Emma Ning, Principal PM at Microsoft, presents Foundry Local, a platform for building cross-platform on-device AI applications using ONNX Runtime and Olive. She argues local AI is essential for low-bandwidth, privacy-sensitive, cost-efficient, and real-time latency scenarios, noting that modern hardware and optimized models like Phi-4-mini and DeepSeek make it viable. Ning demonstrates Foundry Local's CLI, model benchmarking (Qwen 1.5B at 90 tok/s vs. Phi-4-mini with richer output), and a cross-platform document summarization app built with JavaScript SDK, running identically on Windows and macOS. She also previews a local agent with MCP servers (file system and OCR) that extracts receipt totals using Phi-4-mini. Customer testimonial from Sava (CEO of Pieces) highlights improved memory management and tokens-per-second, while another partner underscores ease of install and hybrid cloud-local solutions.

AI Red Teaming Agent: Azure AI Foundry — Nagkumar Arkalgud & Keiji Kanazawa, Microsoft

Jun 27, 2025 · 19:31

Keiji Kanazawa and Nagkumar Arkalgud of Microsoft present the AI Red Teaming Agent in Azure AI Foundry, arguing that adversarial testing is essential for building trustworthy AI agents. Nagkumar demonstrates the tool: it runs scans against RAG apps or models directly, using attack strategies like Caesar encoding and Base64 to simulate adversarial prompts across four risk categories (violence, hate and fairness, etc.). In a demo with GPT-4.0 with full guardrails, no attacks succeeded; switching to Phi-3 without guardrails yielded 5 out of 40 successful attacks in hate and fairness. The tool integrates with Azure AI Foundry's content filters, which apply both input and output guardrails, allowing engineers to iteratively test and mitigate vulnerabilities. The talk emphasizes that red teaming should be part of a broader risk mapping and evaluation lifecycle, and that trust is a team sport requiring collaboration between engineers and security experts.

Agentic Excellence: Mastering AI Agent Evals w/ Azure AI Evaluation SDK — Cedric Vidal, Microsoft

Jun 27, 2025 · 20:01

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates how to systematically evaluate AI agents using the Azure AI Evaluation SDK and AI Toolkit. He argues that evaluation must start early, covering four layers: model/safety system, system message, grounding, and user experience. He shows spot-checking models side-by-side in VS Code, building a web-scraping agent with Playwright MCP, and running batch evaluations with built-in quality and safety metrics like groundedness and fluency. Vidal also covers multi-turn and multimodal evaluation, using a violent image example where the SDK returned a score of 4 out of 5, allowing teams to set custom thresholds based on application context.

Building Code First AI Agents with Azure AI Agent Service — Cedric Vidal, Microsoft

Jun 27, 2025 · 1:54:06

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates building code-first AI agents with Azure AI Agent Service, using function calling, code interpreter, file search, and Bing grounding for a sales analysis use case. He creates a conversational agent that queries SQLite, generates pie charts from Python code, and blends product PDF data with relational data. The workshop explains stateful agents, tool routing (LLM generates JSON to call functions), and the limits of single-step agents versus multi-agent orchestration with AutoGen. Cedric addresses when to use Agent Service (managed persistence and tools) versus raw LLM endpoints, and covers MCP servers as a tool lifecycle manager. Key insights include using instructions to ground agent behavior and the need for eval frameworks for agent quality.

GraphRAG methods to create optimized LLM context windows for Retrieval — Jonathan Larson, Microsoft

Jun 27, 2025 · 15:09

Jonathan Larson of Microsoft Research presents GraphRAG methods, arguing that LLM memory with structure is a key enabler for effective AI applications, and that agents paired with these structures amplify their power. He demonstrates GraphRAG for Code, which uses graph-based understanding to outperform regular RAG on code comprehension, translation from Python to Rust, and even multi-file feature addition like adding jumping to the original DOOM codebase. Larson announces Benchmark QED, an open-source evaluation framework for measuring local and global query quality across comprehensiveness, diversity, empowerment, and relevance. He introduces Lazy GraphRAG, which beats vector RAG on 92% of local and 90% of global queries at one-tenth the cost of million-token context windows, and is being integrated into Azure Local and Microsoft Discovery.

Building Protected MCP Servers — Den Delimarsky and Julia Kasper, MCP Steering Committee & Microsoft

Jun 20, 2025 · 20:13

Den Delimarsky and Julia Kasper, Microsoft engineers on the MCP Steering Committee, explain how the new MCP authorization spec simplifies securing remote MCP servers by separating the resource server from the authorization server, eliminating the need for developers to build custom token factories. They argue that developers shouldn't need to be security experts, so the spec offloads token minting and lifecycle management to standard identity providers like Entra ID, Okta, or Auth0. The client now handles the full OAuth 2.0 dance, while the server only validates tokens via a Protected Resource Metadata (PRM) document. A live C# SDK demo shows a server configured with minimal code, and an Azure API Management deployment demonstrates production-ready remote MCP servers with OAuth proxy and consent flows, including integration with VS Code and GitHub Copilot. The session also previews transforming REST APIs into MCP tools via Azure API Management.

The Geopolitics of AI Infrastructure - Dylan Patel, SemiAnalysis

Jun 19, 2025 · 18:29

Dylan Patel of SemiAnalysis argues that despite US sanctions, Huawei has engineered a 384-chip cluster (Cloud Matrix 384) that Nvidia failed to deploy, while accessing TSMC via Softgo and HBM from Samsung via shell companies — all legally. China's SMIC will soon produce 7nm AI chips in high volumes, debunking the notion that China lacks compute. Meanwhile, Middle East players like G42 (UAE) and Datavolt (Saudi Arabia) are building multi-gigawatt data centers, with G42's deal letting it keep 20% of 500,000 GPUs yearly for itself while 80% goes to US companies like OpenAI. Patel highlights the US's 63-gigawatt power shortfall vs. 100 GW of planned data centers, explaining why US companies rely on Middle East capacity and why China's superior power buildout gives it a geopolitical edge.

MCP: Origins and Requests For Startups — Theodora Chu, Model Context Protocol PM, Anthropic

Jun 18, 2025 · 17:45

Theodora Chu, product manager at Anthropic, explains the origin of the Model Context Protocol (MCP) as an open-source standard for giving LLMs agency to interact with external tools and data, born from two engineers copying context manually. She details MCP's evolution from an internal hack week project to wide adoption by Google, Microsoft, OpenAI, and coding tools like Cursor and VS Code. Key protocol decisions include optimizing for server simplicity over client complexity, adding streamable HTTP for bidirectionality, and fixing authentication via community contributions. Chu outlines three startup opportunities: building high-quality MCP servers for verticals beyond dev tools (80% weight), simplifying server building with hosting/testing/deployment tooling, and creating AI security and observability tools. She emphasizes that models are a third user of servers, so tool design must consider end users, client developers, and the model itself.

Exposing Agents as MCP servers with mcp-agent: Sarmad Qadri

Jun 11, 2025 · 18:05

Sarmad Qadri, CEO of Lastmile AI, argues that agents should be exposed as MCP servers, enabling multi-agent composition and async workflows. He introduces mcp-agent, a framework implementing Anthropic's effective agent patterns—augmented LLM, evaluator-optimizer, orchestrator—on top of the Model Context Protocol. Qadri demonstrates an agent that reads a student story, fetches an APA style guide via a fetch MCP server, and writes a graded report, all orchestrated as a durable Temporal workflow. He explains that MCP standardizes tool connections for LLMs, shifting agentic behavior from client-side to server-side, allowing agents to be invoked from any MCP client like Claude or Cursor. The talk covers how this paradigm enables scalable, platform-agnostic agents that can be paused, retried, and scheduled.

Real AI Agents Need Planning, Not Just Prompting - Yuval Belfer

Jun 3, 2025 · 7:58

Yuval Belfer of AI21 Labs argues that LLMs alone still fail at instruction following, as shown by GPT-4.1's struggles in 2025, and that true AI agents require dynamic planning, not just prompting. He critiques ReAct for lacking look-ahead, contrasting it with AI21 Maestro's planner and smart execution engine, which uses best-of-n sampling, candidate pruning, and replanning. On AIfeval, Maestro pushes GPT-4.0, Claude Sonnet 3.5, and R3 Mini to near-perfect scores; on internal requirement satisfaction benchmarks, it improves over single LLM calls despite higher runtime and cost. Belfer advises starting simple (SLMs, ReAct) and escalating to planning only for complex tasks, inviting listeners to join the Maestro waitlist.

The Devops Engineer Who Never Sleeps — Diamond Bishop, Datadog

Apr 22, 2025 · 16:18

Diamond Bishop, Director of AI Engineering at Datadog, explains how his team builds AI agents—the On-Call Engineer and Software Engineer—that automate DevOps tasks like incident investigation, remediation, and postmortem writing. He details how the On-Call Engineer wakes up for alerts, reads runbooks, queries logs and metrics, and suggests fixes to let human engineers sleep. The Software Engineer proactively fixes errors by generating code diffs and pull requests. Bishop shares four key lessons: scope tasks and evaluate rigorously, assemble teams of optimistic generalists and UX experts, adapt UX for human-agent collaboration, and treat observability as critical for debugging multi-step agent workflows. He predicts that within five years, AI agents will surpass humans as primary users of SaaS platforms like Datadog, urging builders to design for agent consumers.

Anchoring Enterprise GenAI with Knowledge Graphs: Jonathan Lowe (Pfizer), Stephen Chin (Neo4j)

Apr 7, 2025 · 20:59

Stephen Chin (Neo4j) and Jonathan Lowe (Pfizer) explain how Pfizer uses knowledge graphs with GraphRAG to accelerate drug manufacturing technology transfer, cutting time from years to weeks. They argue graph databases provide superior accuracy and explainability over vector-only RAG, crucial for life-saving drugs. Jonathan details navigating organizational silos in a 100,000-person company, from C-suite taglines to client partners demanding cost savings. Gartner's prediction of 30% GenAI project failure is addressed with a concrete business case: manufacturing worker tenure plunged from 20 to 3 years, making AI essential to capture lost expertise. The architecture combines vector and graph retrieval to deliver contextually relevant, auditable answers, reducing data consolidation from three months to three weeks.

The Model Isn’t Wrong—You’re Just Bad at Prompting

Feb 22, 2025 · 8:54

Dan from PromptHub argues that prompt engineering remains critical for improving LLM outputs, covering Chain of Thought, few-shot, and meta prompting techniques. Chain of Thought breaks problems into sub-problems and is built into reasoning models; few-shot prompting works best with just one or two diverse examples, but can degrade performance on reasoning models like O1 and R1. Meta prompting uses LLMs to write or refine prompts, with PromptHub offering model-specific enhancers. For reasoning models, Dan advises minimal prompting, encouraging more reasoning instead of few-shot, and avoiding instructing the model on how to reason. Free resources include PromptHub's templates, the AutoReason prompt, and the Prompt Engineering Substack.

RAG at scale: production ready GenAI apps with Azure AI Search

Feb 13, 2025 · 21:53

Pablo, a Distinguished Engineer at Microsoft, explains how Azure AI Search enables production-grade RAG applications at scale by addressing key scaling challenges: data volume, query load, workflow complexity, and diverse data types. He demonstrates the platform's hybrid retrieval combining vector search (HNSW, exhaustive), keyword search (BM25), and filters, with a two-stage system using cross-encoder re-ranking for improved quality, achieving 100ms re-ranking latency. Azure AI Search supports multi-billion vector indexes through 10–12x storage limit increases and quantization (int8, single-bit) preserving 90–95% precision, with oversampling for full-precision re-ranking. Integrated vectorization automates ingestion from Azure sources (blob, Cosmos DB, OneLake) with change tracking and format handling (PDFs, Office documents, images). OpenAI uses Azure AI Search to back ChatGPT file uploads and the Assistants API, scaling user limits 500x after Microsoft's infrastructure upgrades.

Giving a Voice to AI Agents: Scott Stephenson, CEO, Deepgram

Feb 10, 2025 · 13:08

Scott Stephenson, CEO of Deepgram, outlines the evolution of voice AI from slow, domain-specific systems to fast, open-ended conversational agents powered by LLMs, arguing that speed and accuracy are now solved and the key differentiator is contextual understanding. He explains that state-of-the-art speech-to-text and text-to-speech can achieve round-trip latencies under 500 milliseconds—matching human turn-taking—but notes that current systems fail to pass context between components, causing 10-20% of interactions to feel unnatural. Stephenson introduces Deepgram's 'contextual AI' vision, where models are prompted with the full conversation history, including tone, pace, and background audio, enabling the LLM to generate responses and instruct the TTS on delivery. He cautions against monolithic speech-to-speech models for enterprise use due to controllability and cost concerns, advocating for a modular stack that lets businesses optimize each component (e.g., small LLMs for simple tasks like password resets). Deepgram’s upcoming voice AI agent API integrates all components for low latency and offers $250 in free credits for experimentation.

Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley

Feb 8, 2025 · 20:46

Humza Iqbal of Snorkel AI explains how the company uses Azure AI infrastructure powered by NVIDIA GPUs to fine-tune foundation models for enterprise customers, achieving better performance per dollar by switching from A100s to H100s. He details their distributed training stack (PyTorch, Horovod, NFS) and lessons learned such as balancing node count for batch size and monitoring GPU utilization to avoid networking or data-loading bottlenecks. A cost comparison found two H100s outperformed four A100s on both training and inference, enabling faster iteration through more synthetic data. Azure's dedicated VMs, reliable NFS throughput, and flexible capacity allowed Snorkel to scale experiments from single-node to dozens of GPUs. Future work includes programmatic preference signals and multimodal retrieval algorithms, all planned on Azure.

GitHub's AI Powered Security Platform: Sarah Khalife

Feb 8, 2025 · 23:45

Sarah Khalife, Principal Solutions Engineer at GitHub, details how the company is weaving generative AI into GitHub Advanced Security (GHAS), focusing on code scanning auto fix (in public beta, with a 70% success rate on vulnerability fixes within pull requests), secret scanning enhancements such as AI-generated custom regex patterns and unstructured password detection (reducing false positives), and supply chain security via Dependabot. She demonstrates how AI enables faster remediation—detecting cross-site scripting and suggesting fixes before merge—and shows the security dashboard’s coverage view, which reveals that secret scanning covers 99% of repositories while code scanning only 57%. The talk emphasizes that AI shifts security from reactive to proactive, bridging the gap between AppSec teams and developers through shared responsibility and community driven improvements.

[Full Workshop from Microsoft] Github Copilot - The World's Most Widely Adopted AI Developer Tool

Feb 7, 2025 · 1:19:45

This workshop from the AI Engineer World's Fair features GitHub's Christina Warren, Dave, Alex, and Harald presenting GitHub Copilot, the world's most widely adopted AI developer tool. They demonstrate Copilot's three interaction modes: ghost-text completions using GPT-3.5 for speed, inline chat (also GPT-3.5) for quick code edits, and the chat panel powered by GPT-4 Turbo for deeper conversations. Speakers emphasize prompt crafting—being specific, providing examples, and keeping relevant files open—to improve results. They show how to use slash commands (/fix, /explain, /test) and the new attach button for explicit context. The workshop uses GitHub Codespaces preconfigured with Python 3.11 and Copilot extensions; a coupon provides a 7-day free trial. Harald explains the trade-off between model quality and latency, and notes upcoming features like automatic agent delegation and workspace integration for cross-file edits.

Build, Evaluate and Deploy a RAG-Based Retail Copilot with Azure AI: Cedric Vidal and David Smith

Feb 6, 2025 · 1:57:58

David Smith, Cedric Vidal, and Miguel Martinez lead a hands-on workshop on building a production-level RAG-based retail copilot using Azure AI. They demonstrate how to build a chatbot backend that retrieves product information from Azure AI Search via vector embeddings and customer history from Cosmos DB, then augments the LLM prompt to generate grounded answers. The session covers using Azure AI Studio and Prompt flow to orchestrate the RAG workflow, deploying the flow as a managed endpoint, and evaluating quality with GPT-4 as a judge on metrics like relevance and groundedness. The speakers also explain the LLM Ops lifecycle for iterative improvement and compare Prompt flow with Semantic Kernel and AutoGen.

Multi model multimodal and multi agent innovations in Azure AI: Cedric Vidal

Feb 6, 2025 · 28:56

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates Azure AI's multi-model, multimodal, and multi-agent capabilities in a session packed with live demos. He shows how GPT-4 Omni mixes text and vision to read handwritten French menus and translate them, and diagnoses infrastructure damage from photos for energy and insurance industries. A new video translation service lets him speak German, Spanish, Italian, and Japanese in his own voice while preserving tone, such as whispering or yelling. The Azure AI model catalog now offers 1,600 models, including serverless deployment options, and Phi-3 Vision, a small 3.8B parameter model running locally in a browser via WebGPU. He also demonstrates code interpreter analyzing a GPX file from a kitesurfing session, plotting a map with turn markers, and GitHub Workspaces preview generating a Java GUI from Python code.

AI Templates: Gabriela and Aishwarya

Feb 6, 2025 · 1:03:20

Gabriela de Queiroz, Aishwarya, and Pamela from Microsoft present AI templates for rapidly prototyping and deploying generative AI applications, demonstrating how startups can leverage Microsoft for Startups Founders Hub, including up to $150,000 in Azure credits and expert guidance. The workshop covers three templates: a simple chat app using GPT-3.5 Turbo, a Retrieval Augmented Generation (RAG) app on Postgres with hybrid vector and text search, and a RAG app on unstructured documents using Azure AI Search and Document Intelligence. Key insights include the importance of query rewriting, hybrid search over pure vector search, and streaming responses for better user experience. The templates are open source on GitHub and can be deployed via Codespaces, with a proxy provided to bypass Azure OpenAI approval delays.

Creating and scaling your own custom copilots with Azure AI Studio: Hanchi Wang

Feb 6, 2025 · 24:21

Hanchi Wang, Software Engineer Lead at Azure AI, introduces Azure AI Studio and Promptflow for creating and scaling custom copilots, focusing on tracing, evaluation, and monitoring. He demonstrates a chatbot app that uses the Assistant API with a sales data insight tool (natural language to SQL) and a code interpreter. With Promptflow's trace decorator, developers capture inputs, outputs, and LLM interactions, viewable in a local UI and shareable via Azure AI Studio. For evaluation, he shows synthetic test data sets, content safety evaluators, and custom evaluators like execution time, error rate, and SQL similarity, comparing models such as GPT-4 Turbo, Mistral large, and Phi-3. In production, monitoring dashboards in Application Insights track model duration, token usage (prompt vs. completion), and failure rates, enabling engineers to optimize performance and cost.

Scaling AI in Education: A Khanmigo case study: Shawn Jansepar

Feb 5, 2025 · 22:39

Shawn Jansepar, Khan Academy's Director of Engineering, details building Khanmigo, an AI tutor and teacher assistant on GPT-4 with OpenAI, arguing generative AI can democratize one-on-one tutoring. He describes a rapid prototyping culture that launched Khanmigo in three months via a company-wide hackathon, replacing traditional agile with a prototype-to-beta-to-launch framework. Technical challenges include math accuracy through a math agent and chain-of-thought prompting, refactoring prompts into a component architecture for testing, and managing scale using multiple models and dedicated Azure compute. Jansepar highlights ethical design: a Socratic tutor that avoids answers, teacher moderation, and a writing coach with revision history. Khanmigo now has 200,000 paid users, half in school districts, with teacher tools sponsored by Microsoft free to US teachers. Plans include releasing a math tutoring benchmark and evaluating smaller models like Phi-3.

E-Values Evaluating the Values of AI: Sheila Gulati and Nischal Nadhamuni

Dec 31, 2024 · 28:13

Sheila Gulati (Tola Capital) and Nischal Nadhamuni (Klarity CTO) argue that evaluating AI systems is at a seminal moment, requiring a shift from simplistic benchmarks to multi-faceted, user-centric approaches. Gulati highlights the inadequacy of current leaderboards, citing benchmark hacking and the need for dynamic datasets and value alignment. Nadhamuni shares how Klarity, which raised a $70M Series B, tackles eval challenges in its document-processing product—processing over 500K documents for customers with 15+ LLM use cases. He details practical strategies: front-loading UX risk, thinking backwards from user experience, reducing degrees of freedom by standardizing prompt engineering across customers, and building scrappy future-facing evals. The episode emphasizes that the AI engineer community must innovate benchmarks, bring depth and community sharing to evaluations, and introspect the values instilled in AI systems.

The AI emperor has no DAUs why most devs still don't use code AI: Quinn Slack

Nov 20, 2024 · 18:45

Quinn Slack, CEO and cofounder of Sourcegraph, argues that despite massive hype, only about 5% of professional developers actually use Code AI tools, with total recurring revenue from Code AI sitting at roughly $300 million ARR—a fraction of Salesforce's $36 billion. He cites GitHub's 1.3 million paid Copilot subscribers and just 935,000 yearly active users receiving suggestions, revealing the gap between perception and reality. Slack warns that the entire AI ecosystem—foundation models, infra, and applications—risks collapse if usage doesn't grow, and most revenue in AI flows to NVIDIA and chip makers, not software. From building Cody, the number two Code AI product, he shares lessons: hype fools everyone, autocomplete is a freakishly good feature that spoils expectations, while chat and agents are harder to vet and adopt. He advises builders to use their own product daily, ignore customer demands for buzzwords like fine-tuning, and manually build explicit interactions before adding magic. Slack concludes that the industry must collectively dehype and focus on real daily active users to turn the potential into sustained enterprise revenue.

GraphRAG: The Marriage of Knowledge Graphs and RAG: Emil Eifrem

Aug 28, 2024 · 19:15

Emil Eifrem, Neo4j co-founder and CEO, argues that GraphRAG—combining knowledge graphs with vector search—significantly improves RAG application accuracy, ease of development, and explainability. Citing studies, he reports accuracy gains of 3x (Data.org), 75-77% (LinkedIn), and Microsoft's finding that GraphRAG enables answering entirely new question types. He demonstrates Neo4j's Knowledge Graph Builder, which auto-generates graphs from PDFs, Wikipedia, and YouTube, making graph creation accessible. Eifrem frames this as the next evolution in search after PageRank and Google's knowledge graph, urging developers to adopt GraphRAG for richer context and better AI outcomes.

Running AI Application in Minutes w/ AI Templates: Gabriela de Queiroz, Pamela Fox, Harald Kirschner

Aug 14, 2024 · 1:29:02

In this workshop, Microsoft's Gabriela de Queiroz, Pamela Fox, and Harald Kirschner show how to deploy AI applications in minutes using AI templates, Azure OpenAI proxy, and GitHub Codespaces. They walk through deploying a simple chat app, then two RAG applications: one that queries a Postgres database with SQL filtering, and another that performs RAG on unstructured documents using Azure AI Search. Key decisions include using async frameworks like Quart and FastAPI, token-based chunking, and hybrid retrieval with semantic reranking for best results. They stress the importance of running evaluations with hundreds of samples and share production insights from Copilot Chat, where TFIDF sparse indexing and LLM reranking are used. The session includes free Azure credits and a proxy to bypass Azure OpenAI approval, allowing attendees to deploy everything without spending their own money.

Open Questions for AI Engineering: Simon Willison

Nov 25, 2023 · 24:33

Simon Willison recaps the AI industry's past year—from ChatGPT's breakthrough to open-source local models—and poses key open questions for AI engineering. He argues that ChatGPT's chat interface, while popular, is a poor fit for advanced use, urging better UIs like his command-line tool LLM. He celebrates Meta's Llama release as a 'stable diffusion moment' for language models and highlights the rise of small, locally-run models such as Replit's 3B model, asking how small models can remain useful. On security, he warns that prompt injection remains unsolved after 13 months, limiting what can safely be built. He champions ChatGPT's Code Interpreter (which he dubs 'Coding Intern') as the most exciting tool, able to write and compile C code on a phone, and argues that LLMs flatten the learning curve, making programming accessible to more people. He concludes by urging the community to build tools that enable anyone to automate tedious tasks.

Pragmatic AI with TypeChat: Daniel Rosenwasser

Nov 14, 2023 · 18:34

Daniel Rosenwasser, TypeScript program manager, introduces TypeChat, an experimental library that uses TypeScript type definitions to guide and validate unstructured LLM output into structured JSON for traditional apps. He demonstrates a coffee shop ordering system where types define the schema, showing how the library can handle ambiguous inputs like 'a purple gorilla' by including unknown text for recovery. TypeChat also generates programs as JSON using a fake language to safely script multi-step operations, avoiding sandboxing issues from real code. A Python prototype extends the same approach, with examples like CSV data manipulation via a class-based API. The library aims to make AI tools accessible to all engineers by leveraging types they already use.

Supabase Vector: The Postgres Vector database: Paul Copplestone

Nov 3, 2023 · 16:05

Paul Copplestone, CEO of Supabase, makes the case for pgvector as an embedded vector database within Postgres, arguing it offers production-grade performance and unique advantages for AI applications. He recounts how pgvector was contributed by a single developer, Andrew Kane, and how Supabase integrated it, leading to 12,000 new databases launched weekly with 10-15% using pgvector. Addressing benchmarks that claimed pgvector was 20x slower, Copplestone shows that adding HNSW indexing brought accuracy to 0.99, matching specialized vector databases, and a cost comparison vs. Pinecone ($410 vs. $480 for comparable queries/sec) demonstrates Postgres competitiveness. He demonstrates Postgres's extensibility with a cat image filtering example using partitions and triggers to separate good cats (similarity >0.8) from bad, all in 14 lines of SQL. Finally, he outlines future work on sharding with Citus to handle billions of vectors, and invites design partners for enterprise use cases.

Announcing the AI Engineer Network: Benjamin Dunphy

Oct 23, 2023 · 7:44

Benjamin Dunphy launches the AI Engineer Summit 2023 by welcoming attendees and recognizing sponsors Auto-GPT, Supabase, Fixie, Microsoft, and Cloudflare, as well as VC Alessio Finelli. He then unveils two major updates for the conference's mobile app, Network: AI-Enhanced Matching, a generative algorithm using LLMs to connect attendees based on stated problems, and the open-sourcing of both the event app and matching algorithm on GitHub, enabling any event to use them. Dunphy credits lead engineer Simon Sturmer for the mobile app, 'Sweezett's teller' for the matching algorithm, and infrastructure partners Dscope (auth) and Supabase (database and PG vector). The announcements aim to improve high-signal networking among the 500 selected engineers, founders, and sponsors at the summit.