On AI and Knowledge — Pablo Castro, Distinguished Engineer & CVP for AI Knowledge, Microsoft

Now taking the stage is CVP and Distinguished Engineer at Microsoft, Pablo Castro.

Hello everyone. Hello everyone, good morning. It's great to be back here at the AI Engineer World's Fair. Now, my job at Microsoft is to connect the dots between AI and knowledge. As an information retrieval nerd, like that's great for me.

Like, I spend a lot of time on looking at knowledge representation, extraction, search, and whatnot. And thinking about agents and knowledge really invites to reflect on, you know, what it means to know something. And, uh, you know, the nature of how do we get things done based on what we know.

Next slide.

Allright.

There. So this morning, what I thought we would do is spend a little bit of time talking about the nature of knowledge and split it into these three categories of intrinsic, extrinsic, and learned.

Intrinsic knowledge is just the knowledge that comes with the models. You know, it's what we, uh, train the models on, the training data, and what, um, uh, it's stored in the model's kind of parametric memory. And while it's kind of the obvious thing, I would argue this is the knowledge that actually threw us into the exponential we are in today.

It's what started many of the scenarios that then grew on all the things we're doing with agents today.

Let me give you an example with code. So I wrote these two pieces of code about 25 years apart. And yet, the process to put this thing together was surprisingly similar. Like, I had to sit, sit down with what I knew or what I had to go look up and, um, and then just write it up.

And, uh, while, you know, I'm illustrating this with knowledge, you could say the same thing about, you know, writing an email or creating a summary of a document. Now, you can see this exponential at play in tasks like these where, you know, I'm sure you can go further back, but an interesting point in time to start looking at this would be when Microsoft introduced IntelliSense.

That was in '96. And, you know, it was great. You didn't have to remember function signatures anymore and whatnot. It takes 22 years from there to go, uh, for the next step where machine learning helps us actually rank the options we give you in IntelliSense, so it's quicker to pick theright choice.

Just three years after that, GitHub Copilot launches. And that was one key inflection point. This was even before ChatGPT, um, was announced. And, you know, I would argue that GitHub Copilot, ChatGPT, that sort of experiences were heavily grounded on this intrinsic memory, what the models already knew.

From there, of course, things shifted. You know, a couple of years later, Cursor launches, GitHub Copilot X launches, and how we do things kind of evolve really quick, which takes us to kind of late last year. Opus 4.5 ships, and then in rapid succession, you know, GPT, Opus, and other models keep getting better and better at coding, which takes us to early this year where incredibly successful, uh, software like, like OpenClaw, uh, comes out to existence with not a single line of code written by hand.

So this is the shape of the exponential we're in. And a lot of this was powered by the, by the, uh, intrinsic knowledge in models and, of course, the ability to reason.

Now, in the context of Microsoft, we want to make available all these models and make it easy for you to integrate them into the agents you're building. We do this from, uh, our agent platform that starts in GitHub where we all go and build.

It has a contextualization system so you can ground your agents. And when it comes to agent hosting, observability, and management, we all do all of this in Foundry. Microsoft Foundry is also where we, uh, offer, uh, thousands of models in our model catalog so you can pick whatever is theright, uh, model for theright task.

And we keep adding more every day. In fact, just yesterday, we announced, uh, that Claude in Microsoft Foundry is generally available. So you can use all the capabilities of Claude, uh, in the context of the kind of unified experience in Foundry.

So you get the best of both worlds.

Now, intrinsic model got us here, but it only gets you so far if you're building, uh, a system that or an agent that needs to participate in what's happening in an organization or a company. Uh, and, you know, as an industry, we realized this early and we, you know, we saw the, the RAG pattern emerge.

That started as a pretty low-tech, uh, technique, but quickly evolved. And what we do today with context engineering, uh, and, uh, you know, it, it became a pretty sophisticated system for connecting agents and the knowledge they need to get their job done.

Of the many dimensions of, of which, uh, this got kind of complicated, I'm going to pick on two. One is kind of the evolution from simple and isolated data sets to whole company-wide grounding. And the other one is how we started with simple vector search and whatnot, and we really saw this evolve into fairly complicated retrieval systems.

So let's start with company grounding. Like at Microsoft, you know, spending time with customers, one of the things we saw early was that whenever you build an agent, you, you always have the knowledge you care about for that agent and you'll manage that yourself.

But you also need to ground the agent often on the kind of ambient data of your organization, you know, whenever the agent leaves. This includes maybe your documents, your emails, your chat, uh, threads, or, uh, the, the information in your data warehouse and whatnot.

So we built Microsoft IQ as a way to give you a single entry point into all this kind of ambient data that agents need to get their job done in addition to the specific information that you build into the agent.

Microsoft IQ is not one feature. It's more like a set of capabilities that goes from work IQ that connects your agents to all the documents in, say, SharePoint, all the emails, calendar, your chats, and the connections between people, uh, to Fabric IQ that gives you access to all the, uh, all your analytics assets, you know, from data warehouses and data lakes to Power BI reports.

And Foundry IQ, which is what you use for your own agents where you can push your own data and then use it for grounding. And of course, sometimes you have, uh, your agents need to go out to the web to, uh, ground on data that maybe not yours is public information, but, uh, but you need to, uh, use it to comp-complete the picture of what the agent worldview is.

And for that, we have, uh, Web IQ. Now, this first part allows agents to ground on the kind of this ambient data. Now, the second dimension I mentioned before is the evolution of the actual retrieval systems. You know, when RAG first emerged, I think, you know, what we saw is like an initial adoption for, uh, vector databases that really unblocked us from, uh, getting a lot of the systems off the ground.

And that was great. Um, I think, you know, for a hot second, as an industry, we thought that if we could get really, really good at computing cosine similarity between vectors, we were all set for retrieval. It turns out, you know, things never are never that easy.

Uh, so, you know, what evaluations show over and over again is how, you know, if you combine methods, you just get better results. Like in this case, this is an evaluation from Azure AI Search, the search technology behind, uh, Foundry IQ.

And, uh, and you can see how individual methods don't do as well as combined methods, particularly when you apply them to, uh, real-world customer scenarios. Now, the trick is how you build a platform that allows you to combine all these building blocks without putting the complexityright in front of you.

It lets you opt into it when you need control. But when you have a scenario that is clear, then you can have an easy system. So in Foundry IQ, that was one of our core design goals. And, uh, the way we do this is we actually layer the system.

So you can start at the top. You can go to Foundry and say, "Hey, I have a bunch of, I don't know, PDFs or pictures over there. Just deal with them." And then we'll do everything under the covers.

Like, I don't know, chunking, vectorization, deal with relevance and ranking, deal with, uh, agentic retrieval and whatnot. Now, if you're an expert and you want control, you can also do that. You can go to the bottom of the stack.

You want to build vector indexes and tell us how to quantize the vectors or control lexical retrieval and whatnot. You can do all of that. And you can do it in the same stack, which means you can go up and down as you as your needs change.

Now, on, on top of the core retrieval system, we also introduced an agentic retrieval stack because we see that for easy cases, like, you know, quick single-shot retrieval is great, but for more sophisticated cases, you do want, uh, a system that, uh, can reflect on, on what's in the data set and decide whether or not we've satisfied the information need stated in the input before we come back with results.

Of course, we see a lot of, uh, patterns like this emerge. And the obvious the question is, is this actually useful? Like, are the results better? Our experience in our own evaluations is, uh, for di for difficult cases, agentic retrieval can make a difference.

Across the many metrics that we, we track, you know, things like, um, the actual, actual evidence recall or answer completeness, we see like the agentic retrieval approach continuously does better than, than simple that's, uh, individual simple parts. Now, let me show you some of this in action if we can go to the laptop.

Can we switch to the laptop?

There you go. So here, I'm in I'm in Foundry. And, uh, Foundry is where you, you manage your agents, manage models, but also the place where you can manage all the knowledge that you give your agents in order to, uh, do their jobs.

Um, when here, you can you can create knowledge bases as the kind of the entry point of agents into the knowledge you care about. In this case, I'll create a knowledge base. I have a data set about movies.

These are agentic retrieval systems. So I'll give it a model to power the retrieval workflow. And I can say how much effort you want the model to, uh, to make or the system to make. And this is effectively a trade-off between latency and, and quality.

I can configure a number of other things, but critically, I want to say where the data I want to ground is coming from. And, uh, I can start from scratch. Or in this case, I have a bunch of unstructured data like PDFs and whatnot in, in blob storage.

I have a structure, you know, Parquet tables with statistics, and I also want to ground on the web. So if I take these three steps and then I save this knowledge base, now I have this asset, this knowledge base that I can connect to an, uh, Foundry agentright here, and it'll take a second.

Uh, but also, it's a standalone asset that if I have already a harness that I'm using in other in other places, every knowledge base is an MCP server. So you can just connect to it, uh, without having to write any glue code in the middle.

Now, a knowledge base like this has, uh, you know, has a bunch of parts. Some of them, like, for example, this, uh, storage, uh, content, you usually build indexes, uh, and you, you know, vectorize these things and whatnot.

Uh, and if you want control over that, like if I if you don't, you can just use it here. But if you do, let me just switch to Azure and show you the service behind that particular instance where if I go to knowledge bases, this is the knowledge base we just created a second ago.

And I can go pick inside. For example, I can go fish out the indexes that back this particular, uh, piece of content. And in that index, I can see what is the structure of the index. Uh, if I'm opinionated about, I don't know, maybe the quantization, uh, approach I want to use or which indexing algorithm I want for my vectors, I can say all of that.

And of course, I can actually go and explore the data and, you know, see what's inside, how chunks were organized, and, and whatnot. So the goal of this is to, again, give you high produc a highly productive environment when you need uh, when you don't need, uh, the sophistication and when you need it to make sure you have it to get your job done.

We go back to slides.

And of course, the other aspect of this is, you know, top of mind these days for all of us is token uh, is token efficiency. And, uh, so, uh, we carefully evaluate this system to make sure that we give you the most information-dense answer that has the fewest tokens, uh, so that you, you know, the, the your consumption of tokens has a high value when it comes to all retrieval tasks.

The last category of knowledge I wanted to talk about is learned knowledge. Now, learned knowledge is the result of us doing the work we do as individuals and as organizations every day. And the, the idea that we can actually observe the processes and get better at them by reflecting and improving every step of it is something that has really, uh, changed now that we have agents doing the work and we can go tune the agents automatically.

Satya wrote about this recently and reflected on the fact that people and agents can really compound in, in how they do the work and how they can create this learning loop that, uh, effectively captures what's unique about the company or, or the organization you're working on, uh, and inputs that to work to differentiate the work that you do.

Now, in Foundry, we wanted to offer like a material a materialized version of this that you can use today. So we built a, a component called the Agent Optimizer that effectively goes through this process and allows you to evaluate a baseline, generate candidates, and then, you know, evaluate the new candidates.

And we have a strong result, then deploy that to production. Let me give you a kind of a quick flavor of what this looks like if we can switch back to the laptop.

Allright. So here, I'm in VS Code. I have the Foundry, uh, toolkit installed. And, uh, I have a simple agent. It doesn't matter how you write your agent as long as you externalize configuration like, you know, your instructions, tool definitions, skills, and whatnot.

So once you have one of those, it takes two key steps to do this. So first, whoops.

Um, I can actually so usually you have an evaluation already, but if you don't, you can actually say, "Eval generate." And what we'll do is we'll look at what we know about the agent, like traces and instructions and whatnot, and we'll produce a task adherence-focused evaluation for you.

In this case, I ran this, uh, a little bit earlier. So just to give you a flavor of what this looks like, you, you have a bunch of tasks and then, you know, the questions and the criteria and whatnot.

Once you have a data set you can evaluate, then what the next step is you can say, uh, optimize. And, uh, I could just run optimize on its own, and that will run in this case, this run for maybe 45 minutes or so, and you get an optimized version by effectively hill climbing the metric that's established from the evaluation.

Um, so I ran this earlier. And so let me show you the output for this particular one where you can see that, you know, we established a baseline first, and then we kept iterating on candidates using different combinations, using a JPA style kind of loop, uh, and, uh, looking for options that perform better given, uh, the rubric that we have.

And, uh, the interesting thing is that once you found one that is that is better, then you can simply just say, "Optimize, apply." And what this does is since you externalize the configuration, it allows you to swap one configuration for the other.

Um, if I if we look here, you can see that, for example, I have a baseline and the one we just applied. And just to pick on instructions, these are just the kind of trivial instructions for this example agent.

But if I look at the optimized one, then you can see like a bunch of instructions that are not handwritten, but are that they emerged out of the hill climbing process to get to make this particular, um, agent better given what we have in terms of instructions and skills and tools, but also based on reflecting on the actual, uh, traces from the agent as users are using it.

So this is a real learning loop materialized in practice. We can go back to slides.

So this was like a very quick overview about how do we think about knowledge in the context of AI and how do how we think we can enable these learning loops that will capture, you know, this differentiated capability that lives in each one of the companies and organizations we work on.

If you want to try anything of what I talked about or showed today, you can head to ai.azure.com and get going. And with that, thank you all for listening this morning. I hope you have a great rest of the event.

Thanks.