Building Code First AI Agents with Azure AI Agent Service — Cedric Vidal, Microsoft

Well, um, this is very exciting because, I mean, 2025 is clearly the year of agents. Compared to the past 2 years, things have moved so fast. We went from very simple prompts, which were already incredible, but now we move to the next step where we have agents that can autonomously achieve goals without us knowing exactly how they do that, which is quite incredible.

But the question is, how do you make them? So that's what we're going to see today. I am Cedric Vidal. I am a Principal AI Advocate at Microsoft. You can find my information down below. And today I'm going to be your host, and I'm going to be helped today by proctors, by Mark.

And I'm sorry, I forgot your name. I feel terrible.

Mark.

Not Mark. Did I say it correctly?

Yep.

Okay, thank you. So a big thank you to you two to help me today. So during the workshop, if you have any questions, please raise your hand. And Mark or Ar- Ar- Argmar?

Not Armar.

Not Armar. Sorry. We'll help we'll come and help answer questions. I feel super bad. Anyway, so

today, like I said, I'm going to set the scene first. So what are we going to do today? So in order to put our hands on the keyboard and create and show to you how to create an agent, we're going to use that use case.

Imagine that you are working for an outdoor and hiking equipment company that sells equipment online. We are going to so what you want to do is you need to build a system that allows to analyze your sales data mixed with product information, generate ad hoc diagrams, basically a UX that your salespeople can use very easily, where we move away from the old paradigm where we had to hard code every single use case, every single view, every single query, where now those the database, the queries are going to be generated automatically.

The UX is going to be generated automatically to accommodate the type of information that you are displaying. And we are going to see how to create such an application.

An agent. Okay. What is it? Because, like, the definition of an agent has changed so often. Let's be honest, even the specialists in the industry don't agree exactly on what they are. And even the definition of what an agent is has evolved over time over the past 3 years, as people have got more acquainted and were discovering what we could do with it.

You could you would imagine that a definition should be set in stone, but in that case, it's been difficult to agree. But what we're going to the definition we're going to use today is that it's semi-autonomous software

that to which you give a goal and will work to achieve that goal relentlessly using tools and information that you can pull from databases and data stores at large and iterate until it achieves that goal. So until the system stabilizes and the goals are met.

That's the general definition. But we're going to see that depending on the context, they can be more complex.

In order to do that, an agent should be able to do three things: do reasoning over a provided context, to provide a cognitive function such as deduction, correlation, understanding cause and effect, all that domain of cognition the LLM now has proven that it was able to do a lot of those.

Not perfectly, but it's getting better every day. The second one is integrate with data sources for context. And the last one is act on the world. Because in order to stabilize the system, in order to be useful, like before the first generations of LLM powered systems were just about pulling information and displaying it.

But now we are moving a step forward where we are acting on the world and modifying the environment until it stabilizes and reaches the expected goal.

Ta-da. So what kind of application are we going to build today? It will look like this. So this is a screenshot of a slightly different application. What you're going to build today does not look exactly like this. But the idea is that you can ask a question in plain English, such as, "Show the sales of backpacking tents by region and include a brief description in the table about each tent."

And it's going to pull information from the database as well as the product information and mix all that information together, reason about it, and display the content. And the shape and form of the display of the UX will be will depend on the type of information which is requested.

Here it's a table. Here it's a pie chart. Because the question here, yeah, create a pie chart of sales by region. So

the system is going to understand that we want to create a visualization.

Okay. What technology are we going to use today to build our system? We are going to use Azure AI Agent Service. There isn't so before I dig more into the details of what this is, you have so many ways to build an agent today.

So many frameworks, long chain, long graph, semantic kernel, and so many others.

The Azure AI Agent Service has the advantage that it's stateful and quite easy to put together. Because usually when you build any kind of LLM application, I don't know if you're aware, but it's stateless. You need to manage the state client side.

So it's the responsibility of the application developer to store the conversations and to handle all the logic of pulling information from various systems as well as executing the functions, the tools. Agent Service moves all their responsibility to the cloud on the Azure platform and everything is managed.

So it comes with pros and cons. We're going to see what they are. But when it's about the pros, the big advantage is that it provides a very simple development workflow because all the state and the context and the agent configuration is managed in the cloud.

The integration with backend data and data sources is also managed in the cloud. Even if you may also mix and match. You can mix things that are managed by Agent Service in the cloud with things that are managed locally.

It's possible.

And it supports all the model families that are supported on the Azure AI Foundry Model Catalog, plus the Microsoft Enterprise Security, which is very well known to be very robust. It's sometimes a bit difficult to set up, but that's the price to pay for security.

So the application that I just showed was using Chainlit. The one we're going to build today is going to be common line. It's going to be slightly easier. But so basically at the top, you have the application layer with the framework.

So here, Chainlit. In our case, it's going to be common line, very simple, very basic, with a query function, which is going to use Azure AI Agent Service with instructions and models and actions. So for function calling, and I'm going to explain what it is, code interpreter, and I'm also going to explain what it is, file search and grounding with being search for web information grounding.

And we're going to go through each one of those during the workshop. So like I said, Azure AI Agent Service comes with pros and cons. The pros is that it manages everything for you. That's a big pro,right? The con is that you need to understand that diagram,

more or less. You don't need to understand all the details. But one of the so I'm going to change slide. So the first thing, what is you're going to have to follow a sequence of steps. And you have quite a few steps to follow in order for an agent to work.

First, you need to create an agent. Once you have created your and configured your agent, your agent exists in the cloud. It's kind of weird at first because when you're used to a stateless way of doing things, that stateful programming model is not so common those days anymore.

But it becomes relevant again in the age of agents. So once you have your which means that if you're you want you have an application that wants to reuse an agent, if you have created the agent before, you need to reconnect to an existing agent.

It's kind of like in SQL, like the create or update table. You only create the schema if it does not exist yet. So it's kind of a create or update agent for most applications. So you create the agent.

Then you create a thread or you reuse a thread. Then you run the agent on the thread. Then you check the run status. And then you display the agent's response. Those are the big steps that you need to get familiar with when building with Azure AI Agent Service.

Then you are going to configure instructions. Those instructions are going to be attached to the agent. And they are stateful. So they are uploaded in the cloud. And once they are there, you can reuse your agent and you don't have to send the instructions every single time, which in terms of bandwidth and network is interesting.

Then you configure your model. You add data sources. So one of the pros of using Azure AI Agent Service is that you can attach data sources directly to the agent. And you can do that either graphically through Azure AI Foundry or you can do that programmatically through the SDK.

Then you can attach tools. And we're going to see how to do that, how we attach each one of those tools. Some of those tools can be client side, so you are managed by the application code. Some of them can be managed by Azure.

Today we're going to see file search, code interpreter, function calling, and Bing search.

Here is an example of a thread, of what a thread looks like. So the user's message is going to be, "Tell me the total sales by region." So what's going to happen is that in order to get the total sales by region, we need first to get the sales.

I mean, we need to query sorry, we need to query the sales data store. And it happens that in this case, the sales data store is a SQL relational database. And as you know it, the way to interact is using SQL queries.

So we are going to generate a SQL query dynamically depending on the user's request. And then we're going to send that SQL query, execute it on the database, get the list of records back, re-inject those records into the LLM, which is going to generate a message in plain text from those list of records.

Excuse me.

Yes?

The SQLite is just for an example. It can be.

Yeah, of course. Yeah, yeah, it's an example. In the agent we're going to build today, just because it's convenient, but you can, of course, connect it to any database you want, relational, the document, in an API, like, really doesn't matter.

For the ones which are managed locally, for the ones which are managed by Agent Service on the backend, the list is more restrictive. And to be honest, I don't have it on the top of my mind.

We're not going to see any RBAC or anything.

Sorry?

We're not going to see any role-based access control or anything.

Not today. Not today.

It's a more advanced topic, obviously. And it depends a lot on your use case. So then show as a pie chart, which is the second question we asked in the previous screen I showed you. This one is going to be quite interesting because we're going to use a tool called code interpreter.

What it does is that it's going to take the query, generate Python code. And the Python code is going to be executed in a sandbox, in a secure, safe sandbox. And it can be whatever is supported, whatever Python packages are available in the environment.

Usually, you use it to generate diagrams. And it's going to generate the Python code, which is going to generate that visual representation. It's going to execute the code. The code is going to save the image somewhere on the file system inside the sandbox.

Then the agent is going to pull that image out of the sandbox and send it back to the client application. Okay. So like what I said, this is quite a lot to digest,right? But the thing is, you just have to go through and get a mental model of how that works.

And once you understand that, you don't have to manage it yourself, which is quite interesting.

Does it use any vector databases behind the scenes?

Today? Or

do we have a vector database today? I'm blinking. I think we do. I think we do. But I'm going to double-check. Because we ingest the documents. And I think we ingest the documents inside an AI search instance.

One more very important thing, function calling.

So to be honest, function calling is not new when it comes to LLMs. I was doing function calling, like, literally when the first version of ChatGPT was announced by asking the LLM to give me answers separated by commas and say, "Generate a function and an argument separated by commas."

You can still do it, by the way. But nowadays, it's much more efficient to have structured output. It generates well-formed JSON. Even the LLMs are optimized under the hood inside the data center to generate JSON optimally. But so the principle of function calling, actually, the name is bad.

I've hated that name ever since it was coined because it's not function calling. It's function routing. The LLM does not call anything. An LLM runs on a GPU. It's not going to call any code. So what it does rather is that it generates a JSON representation telling you what function to call with what parameter values.

And it's going to map it's going to decide which function to call. And it's going to map the natural language sentence and extract values that it's going to pass as parameter to the function to be called. And then it's the responsibility of the application code to take that function call specification and actually execute the code.

So that's what it does.

How is that function called?

Yes?

How is function called? How does LLM know which functions to use? Is it using your simpler classification model? Like, do you know under the hood? How does it know?

Oh, that is an excellent that is a very good question.

Well, it's exactly the same way when you ask a question. For example, when you ask, "What is the color of the sky?" Like, very one of the first use cases. Obviously, the answer is going to be blue most of the time.

The way LLMs work is that it's a statistical distribution. You have the probability that the word that comes after when you ask the question, "What is the color of the sky?" the most probable answer is going to be, "The color of the sky is."

And after "is," the most probable color is going to be blue because that's how the model has been pre-trained,right? But if you say, "What is the color of the sky?" And in order to get me the answer, you can only use, for example, color equal and the value of the color.

Then you give an instruction. You tell the LLM, "Hey, here's the output that I want." And then it's going to generate color equals blue. Just because you constraint, like, statistically in the world of all the possibilities that you can answer, you're narrowing down the type of output that you want to be generated.

So color equals blue. Imagine that

and so you're going to parse that answer. And you're going to interpret color equals blue and do whatever you want with it. Except in the world of function calling, it's not color equals blue. It's set you have a tool, say, execute color, which takes a parameter with the value of the color.

And instead of having just one, you have many possible functions. And same statistical probabilities. When you imagine you have one question to set the color and another function to order a pizza, if you ask what the color of the sky is, it's not going to generate order pizza call for the same reason.

It's exactly how it works.

So how do you know when to use function calling versus creating your own classification model?

Okay. Well, because a classification model cannot extract values, entities out of a context and, like, answer many answers across multiple dimensions at the same time. Classification is just one out of n possibilities. So it's an entirely different type of task.

Okay. Let's move on. Okay. So I want you to, on your laptops, to open the following URL. So I'm sorry it's not very big. I'm going to spell it. So Microsoft events.learnondemand.net. So Microsoft events, all attached, dot events sorry, my God, dot learnondemand.net.

Is everybody on that page? Okay. Then you're going to log in. Be very careful. This is very important. You need to choose

Microsoft account. Well, usually, you're going to use Microsoft account. Oh, I thought we had other types of okay. So if it's a personal account, you need Microsoft account. If you have a corporate account, like, if you have an account as part of your company which is managed by Entra, you need to select Entra ID.

So it's kind of misleading. Do not select Microsoft account if you use a corporate account. Use Entra ID.

Then

you're going to

land on that page. That page, you're going to have a redeem training key link. Click on it. And then you're going to be asked to enter a training key. Enter the following training key.

So hopefully, this is big enough and everybody can see. But let me know.

Remember, raise your hand if you have any questions.

Admin editing and admin approval.

Sorry, what?

It says it needs an admin approval for the.

What?

The organization.

So it may not be possible.

You can use also a personal Microsoft account. Like, you can use an Outlook.com. You can use if you have an Xbox, you can use an Xbox account.

Any Microsoft account or any of the consumer domains out there. So in your case, you might want to try a personal account.

And to be honest, in that case today, it's usually easier. Like, you do not need to use a corporate account today. There is absolutely no need.

I just have a question about that first slide that you had up. So you were saying an agent could go to a SQL query, come back, use the code interpreter, and make a graph. Is there any reason you're using agents for that as opposed to just training a lot of workflows and injecting the tools?

You could totally do that.

Yeah?

Yeah.

I'm just wondering if there's, like, a reason why you're using agents for that sort of stuff.

Well, like I said, so we need to go back to the definition of what an agent is. And like I said, the so it's a bit of an overloaded term these days. It encapsulates a wide range of definitions, including the simplest.

Today, we are not in the simplest use cases. We are not in the more complex use cases. For example, at the beginning, I said one of the things that an agent can do is be goal-driven and iterate until it achieves the goal.

Today, we are not going to see that specific thing. We are going to stop one step we are going to be in the middle in terms of complexity. Sorry. We are going to be above the simple completion.

And we are not going to see the looping. We are going to be we're going to be seeing a mix of using tools and code interpreter with multiple data sources where the information from all those are mixed together to do reasoning and act on the world.

We are going to see that. And yes, you could do that using just an LLM locally. Nothing would prevent you from doing that, except today, we're using app service, which manages everything server-side. Also, I'm going to show the limits.

That today, I'm going to give examples so that you get a sense and you touch exactly when

the current architecture hits its limits and when you need to go further and use a more orchestrated

planning and orchestration that iterates which loops until it reaches the goal. And I'm going to give an example where you see where it breaks.

Where are we?

Where are we? Training key. Okay. After you click on Launch.

Building?

Building.

Oh, my God. It's not pre-built?

It's about eight minutes. Once you go to Launch, it takes eight minutes. It launched the lab.

Okay.

Oh, my. Okay. I thought there would be pre-built.

Did everybody click on the Build button?

Yes.

Okay. Please do. So that means we have eight minutes to answer questions.

So we just had one. Thank you.

Yes, one more.

So if an agent is just an LLM running in a loop until it thinks that it's done what you've asked it to do.

Yes.

How does it know.

When to stop?

Yeah. How does it know when, like, when to kick back out of that recursion?

Yeah. So today, we are not going to we are going to see the limits of not okay. You have two types of agents, two levels of complexity.

An agent does not have to do the looping to be called an agent in the simpler

area of the spectrum.

In order to go one step further and do that looping, you need to use something like AutoGen, for example. And those types of agents, you need to define a criteria of done, a definition of done, basically. And you're going to have and the criteria can be implemented programmatically, deterministically, or using an LLM, which decides, is it done?

Like, is the task finished? Have we accomplished the goal? And the workflow engine is going to loop until the goal is reached.

But it is the most tricky thing, in my opinion, clearly. That's when you're going to develop an agent. Figuring out when to stop is tricky. For example, I often I've done a couple of prototypes with BrowserUse, which is a famous open-source agentic system for browser navigation.

When you ask it to complete a task on the web, it's going to navigate from website to website and read the pages and do actions. And the evaluation of when the task is done is clearly not always perfect.

Yes?

I have a question. One is related, one might be related.

Sorry?

One might be related.

Yeah.

The other one is so my first question is, are we going to learn today, like, how to send feedback to the agent, like, in case the agent gives, like, an incomplete answer or an incomplete answer?

You mean the looping?

Yeah, yeah.

So no.

Yeah.

So not today.

Not today.

Not today.

The other question is, what's, like, in general, what's the difference between, like, an AI agent and an MCP server?

Oh. So it's very easy. It's a very easy explanation. So basically, MCP is just a tool, a function. You remember I explained what a function is? So an LLM is able to do function routing, decide which function to call when you have a question.

MCP is just that plus

management of the lifecycle of the program which completes the which executes the function.

Because normal function tooling, if you just take, like, GPT or LLaMA and you do function calling, what the LLM is going to return is just a JSON telling you the name of the function and the list of the values for each parameter.

Nothing more. It's your responsibility as an application developer to execute the function.

So that might be the MCP for that?

No. It's not that you need it. It's one of the existing technologies out there that exists that you can use as a tool. And the advantage of MCP, at least as a client-side AI application developer, is that the MCP protocol takes care of downloading the executable, the binary, whether it's a Node, Python, or whatever, or a Docker image.

Like, pull the executable on your machine and execute it automatically. And it's an overall protocol which comes with and also, it encapsulates the possibility for the tool to declare its functions.

So from the standpoint of the user, you just have to declare, oh, I want to use file system MCP server, or I want to use a Blender MCP server, whatnot. And that's all you have to do. You can select it from a catalog, and it's going to auto-declare what tools it has and automatically

start it and stop it. Also, when you're done with your MCP client, it's going to stop all the MCP servers and clean up everything.

Is it where are we with the?

I got a question.

Okay.

Where are we?

Give me a moment.

Oh, my God. Does it give a time? Seven minutes? Oh, my God.

I've got a question.

Did we forget to do something?

I don't think so.

Yeah. Anyway.

I had a question.

Yes.

I was going to wait and see kind of how the demo played out later. But one of the

questions I often find with the agents is the balance between making a fairly general agent that can do lots of things and just kind of giving it the tools, not giving it much direction, versus having to be fairly controlled with it.

And you're kind of trading off the, like, autonomy to, you know, a bit of reliability. The one that you set up for, I couldn't quite work out if you literally just given it the tools and then it can do everything with those tools or if you've been quite controlled with it.

So how do you think about the trade-off? And when people are using your tools, do they tend to fall more on one side than the other?

It's a more complex answer that it looks like. More complex question than it looks like because of two things. So when you give instructions, you know, it's like I was talking about the space of probabilities. The more vague you are, the more you leave options open.

And so the more your agent is going to be able to do wide area, wide range of things, but it might get it wrong. The more specific you are, the more you're restricting the things that it's going to do well, but it's going to do well more often.

And then there is you didn't really ask this, but I'm assuming that's on your mind, is how many tools can I give to my agent? And also, another common question that we often get is, should I give all my tools to one agent, or should I split tools on multiple agents?

Or should I create should I only give one tool per agent? And the answer is same. It's complex, but it's kind of imagine same. You need to think in terms of statistical probabilities. Like, it's stochastic. And so when imagine you have one agent and you give all the tools.

And at every single instant, for any question you ask, the LLM has to decide which one of all those tools to call. The probability that it gets it wrong is higher,right?

So the solution to that is to create agents which are more specialized by, like, areas of expertise and do some kind of routing and multi-step selection where instead of having one agent and giving all the tools, you, for example, you have a first round of agents that is going to determine classify we were talking about classification earlier classify the question and say, oh, this is a sales question, or this is a product question, and then route to an agent which is more specialized for to answer things about sales or things about products.

And then when the answer comes back, the first agent can say, oh, do I need to use another agent? So it's like an agent is like a multi-tool. So you can imagine, like, a tree of tools, and each tool can be composite or leaf.

You can see that way. And AutoGen allows us to build such topologies. Yes?

It means specialized in which one could be working with a different database or a different track or.

Yeah. Yes.

And then the coordinator or the maestro that.

Yes.

This.

Yes. But you can also have topologies where the different agents share memory. Because the thing is, sometimes you also have the situation where you have, like, a team of agents, and they are specialized. And it's good that they are specialized because you don't want them to pick the wrong tool.

For the reason I just sorry?

Or hallucinated.

Yeah. Well, hallucinated is.

Because each one has its own track or.

Yeah.

Its own specific.

But what's going to prevent the hallucination is the grounding. So each one of those agents is going to be grounded in something. But something which is can some kind of a grounding is memory. Like, for example, if you have multiple tools that answer questions about a consumer, and you want to memorize or remember the preferences of your consumer, of your user.

Like, for example, what's the name of his or her pet? But you want each one of your agents potentially to be able to use that information. So what you want to do is take that memory, connect that memory to each one of those agents.

Even if you have multiple agents, they can each one of them have access to that shared memory because you want all of them to be able to access the name of the pet.

Yeah.

So to be honest, it's complex. Like, you have many topologies which make sense depending on your use case.

Yeah.

Where are we?

How many?

Ah, finally. So where are we? Is every oh, one more. Still building. One more. Can you please raise your hand when it's still building?

I can remind everyone in a second. It came out too much.

Mine came out too, but I tried again. I tried a few times.

And how did you try? Did you refresh or something else?

I think it went back to the trigger.

OK. So you can.

Because I already tried twice. But I tried multiple times.

OK.

The time was way time was less. And then it didn't work.

OK. Hopefully, it's good. So is it still no. Is it still building? No. You're good? So sorry. Can you raise your hand again for those for which it's still building? For whom it's still building? So just one. OK.

So I'm going to move.

20 seconds.

OK. So hopefully, it is actually 20 seconds.

Yeah.

I'm going to move ahead.

I think I'm good now.

Sorry.

It's back.

Oh, cool. OK. So everybody is there. Awesome. So let me move on.

And

let me see. Because I want to reconnect.

I heard 404. That's not good.

OK. So I have the same environment as you. So I'm going to show you how to make the most of it.

Log in as admin?

Yes. No. Sorry. Yeah, I went too fast. I forgot. So when you have the black screen, I should have OK. Let me

when you have the when you have the black screen,

the username should be preselected and should be admin. So there, you can click on Password Here, which is going to fill the password box. And then you hit Enter. Do not type every time you see a T like this, you do not need to type.

You just click on it, and it's going to auto-type.

Is everyone on that screen?

Yeah.

Yeah? OK. So we can start.

OK. Says we're going to fast on this. I'm going to move on to

so what I'm going to do here what you can do, it's not mandatory, but you can click here and go to Split Windows to get more real estate.

So that way here, I'm moving the instructions away. And here, that allows me to have more space. Then I'm going to go to

Getting Started. Do we need Getting Started? I don't think we do. No, I think we can move on. No, we do, actually. So we're going to move on to Getting Started. What we are going to need to do is AZ Login here.

So I'm going to open it from here, Terminal.

Type AZ Login.

Type Select Work or School Account. But this is very important. Continue.

Even if you connected the first time, even if you connected with your Microsoft account, in this instance now, we are not going to use your account. We are going to use a temporarily generated account, which is a work account.

So you need to select Work Account. And there, you're going to

select the so in the Resources tab here, in the instructions here that I in the pop-up, you click User. It's going to auto-type. Then Next. Then the password. Sign in.

Yes.

You're all set. There we go.

OK. So now I'm logged into AZ. So I can go back to the instructions here. And we are going to have to type that command because in Azure, you have some roles that need to be assigned. And that's something we could not automate as part of the lab provisioning.

So that's something you need to execute manually. So you just type that command, pass it in the terminal here. You're going to have to say to agree to that warning and say Pass Anyway. Enter. And you're going to see a bunch of logs.

And at the end, normally, everything should be fine.

OK. Here we go. So you have a JSON. Is everybody there? Yeah? OK.

Select Subscription.

OK. So now, finally, we're going to be able to open the workshop. So you type that command, which starts with git clone. So what we're going to do is we're going to check out the Git repository from GitHub.

And we're going to build the project and open it and install some code extension and open it in VS Code. So I'm going to go back here, type the command, Pass Anyway. Enter. So like I said, it's cloning the repository.

Creating the Python virtual environment. So today, we're going to use Python for this workshop.

Where do you access the instructions?

Excuse me?

Where do you access the instructions?

Where do you oh. So when you get into the VM, you just open Edge. You open the browser. And the instructions should be displayedright away.

Soright here?

Yeah.

OK.

Yes.

You must have clicked on a link. No. Maybe not. I don't know.

I don't know how you got here.

Can you go back?

Thank you. I don't know how you got here, and I'm not sure what the

can I see?

Yeah. I could start over.

Oh, here we go.

So I was at this step where I cloned the repository and I installed a PDF extension. Then we're going to move on, and we're going to open

VS Code.

There we go.

OK.

So now we are with VS Code here.

So now, we still have a few setup steps before we can get to coding. So you're going to have to go to the Azure AI Foundry here.

Sign in.

For the sign in, use that user from the instructions pane.

Password. OK. Stay signed in. Yes.

OK. So Azure AI Foundry. So now we are in AI Foundry. Then what we're going to do is we are going to search for

go down. Go down. We're going to oh my god. Come on.

So we're going to open the project here. So be careful. You need to select the project, not the hub.

Once you are on the project, you can ignore those pop-ups.

OK. And we're going to need that project connection string here. So you can copy it. And we're going to paste it

in VS Code. So once you have copied the project connection string,

you can go back to you can go back to VS Code. Search for the env.env.sample file. Rename it. So we're going to remove the sample.

And we're going to paste the connection string here. So that connection string is what allows us to connect to the AI Foundry project in which we're going to deploy the AI agent.

Be careful when you paste it to not forget characters, to paste between the double quotes. Otherwise, it's not going to work. So once we've done that, we can move on. So as you can see, we're going to use GPT-4.0.

We can save the file. OK. Is everybody at this point? This is important because now, finally, we have configured all the I'll get a note over there. Can you guys go help?

Mark

is is everybody else at this point? OK. So do we have everyone at that step?

Basically, just to sign in.

Is everyone set up? Yes? So we can look at the code?

OK. So the first thing that we are going to look at first is that I'm going to explain quickly what the files are. So the main file that we are going to look at today is the main.py file, which is the entry point for the application.

And it's where everything is configured. The goal of this workshop is not to give you what code should look like in production. The goal of this workshop is for you to understand how it works, to understand all the pieces.

Because once you understand, you're going to be able to use any of those other frameworks. I mean, this one or another one, it's going to be the same. What matters is to understand how an LLM works, how a function calling works, how grounding works.

That's what matters.

So and the sales data, yeah, that's where the SQL query generation logic is. You have some stream handling. I mean, you have a bunch of things. Another directory which is very important is the shared/instructions directory.

OK. Let's move on. So the first step, the most important thing, is to understand function calling. So the first example

is it's going to show you what it does.

So

in sales data, let's look at

where am I? Yes. Sales data.

So that's where all the SQLite logic is. So you have a bunch of functions.

So as you can see as you can see here, that function takes a SQL query. I mean, let me scroll so that you can see it. It's a SQL query. So the LLM is going to be the one generating the SQL query compliant with SQLite syntax and pass it to the function.

And the function is actually pretty dumb. The function does not do much. It just uses the SQLite driver and executes the query, nothing more. That's because all the smartness of generating a SQL query is done by the LLM.

When you think about it, an LLM is very good at language. And SQL query is a kind of language, like coding, like Python. And so and it happens that when the large language models are pre-trained, they are pre-trained on a massive amount of GitHub repository.

And a lot of them contain SQLite queries. So if you use a common database, it's going to work, like PostgreSQL, MySQL, SQLite, MongoDB. If you use an exotic database that nobody has ever heard of, it's not going to work because the model needs to have been pre-trained on it.

Here we go.

OK. So the exercise is in the main.py file. So I'm going to open the main.py file here. So we're importing a bunch of packages. So those are the packages which are required to connect to AI Foundry to get the models, to authenticate with Azure Identity.

.env is the package that will load the environment variable from a .env file that we edited previously. You have some logging. OK. Let me move on. So here, that's how you connect to the AI Foundry workspace

or project, I should say.

OK. And so line 59, what you're going to do is comment the first instruction files.

And then we're going to look at its content. So if you go to instructions here, you open function calling here.

Those are the instructions that are going to be passed to the agent. You are a sales analysis agent for Contoso, Return of Outdoor, Camping Gear. So it explains what personality the agent should have, what the mission of the agent should be, help users by answering sales-related questions, at least what tools are available.

So here, sales data tool, use only the Contoso sales database via the provided tool with the name of the function. So to be honest, those instructions are optional. They are made to actually, it echoes to the question you were asking earlier.

I didn't get your name, but.

Demps.

Demps, where you were asking how specific you must be. The tool comes with its own schema. And the LLM is going to understand the JSON schema of the tool, which explains the function name, the parameters. And you have an explanation of what each parameter is.

Here in the system instructions, you can add some more information that are more specific to your application or to the agent using the tool.

Then you have information regarding formatting and localization and examples, et cetera. Well, it's quite thorough.

And

here, so we are specifying the instruction files. And we are also specifying which tools to use. So here, we are going to use the functions, which contains only one function tool, which is async, which is the function I showed to you earlier.

Here we go. And also, the documentation here, the Python doc, is passed to the LLM. So what you write here is important because the LLM is going to interpret that documentation as well as the name of the parameters plus the documentation of the parameters to understand how to call your function.

And then I'm going to

open a terminal,

command palette. And to be honest, I never remember where the terminal is. So what I do is that I cheat. And I create a terminal here.

There we go.

You need to remember we're on Windows. Usually, I have a Mac. OK. OK.

OK. We have looked at this. I have explained to you all of this. Run.

So now we're going to hit F5. No, here. F5. Or I could do that too. I could click on the play button here. OK. Yeah, you don't even need to open the terminal. You don't even need to do what I just did before.

It's just it's a habit I have. But you can just click run here. It's going to work. I mean, hopefully. We'll see. So it's doing what I mentioned earlier, which is that first, you need to create the agent.

Once you created an agent, you're going to be assigned with an ID because it's stateful. But the agent is actually an entity which lives in the AI Foundry project. Then we are going to enable the auto function calls.

We're creating a thread.

So the thread is basically the conversation, which is stateful too because you don't have to save the messages yourself. And then we can finally enter a query. So I'm going to go back

because we have a list of questions we can ask here. What was the sales by region? And go back to the terminal and paste it. No, sorry. Wrong terminal. This one.

And if it works correctly, we'll see.

Yes, it does work. So to be honest, every time I see that, it still amazes me, even if I've done I've been doing that for a long time because basically, from the simple question that we ask in plain English, based on the knowledge of the schema of the database, which is somewhere in the code, it can automatically generate such a query.

Let me go back to the question I asked.

What were the sales by region? So we are asking what the sales are. So

where is the from? From sales data because we are pulling from the sales data table. And we want to group

by region. So we want to sum the revenue, group by region. And automatically, it limits to three. And I believe it's because in the instructions, there is an instruction that says that by default, you should limit to three.

And so no, sorry. Sorry. 30. So that's how we get that great table. And we can see that for Africa, we had 5 million, 5.2 million Asia Pacific, 5.3 million, et cetera. So we have our table. And now we can ask follow-up questions.

What was last quarter's revenue? It gets more tricky.

To be honest, I don't know the scheme of the database very well. So even for me, I would have to look at if I had to write a SQL query, I'm sure you've all done that. Remember, the goal of such an agent is to enable non-technical people to use technical tools.

And in this case, the technical tool is a SQL database. And so here, using

oh,

I don't remember that one. What year?

Let's try 2024.

Yeah.

OK. So this is interesting.

OK. So we did find so that question was, what was last quarter's revenue? So I'm assuming that it's the yeah, select the sum of the revenues from sales data where year equals 2024. And we ask for the last quarter sorry, did we say last?

Last quarter.

So it selected 7, 8, 9.

So I'm not sure why it selected those three months as opposed to, for example, 10, 11, 12. So let me correct it, actually. It's an interesting experiment. Every time we do that, obviously, we get different responses,right? So actually.

I think it's because 4 of those last training date was November of '24. So it's thinking last quarter from November is 7, 8, 9 before November.

It's possible. But actually, let's say we're

in.

You got 1, 2, 3 when I did the.

April.

Yeah, April.

So it works. So I just said, actually, we're in April, which means that the last quarter is January, February, March. And as you can see, it changed the request with 1, 2, 3. So one thing we could do to improve this example is add a date tool.

We could add a tool that allows the LLM to ask for what is the current date so that we would not have to enter it manually. Well, I'm going to leave it as an exercise.

Which products sell best in Europe?

OK. So this time

so this time, it is interesting because we are generating a query against a different dimension, which is the product type. So as you can see here, we are grouping by product type. OK. I'm going to pass on because you understand the concept.

I'm going to ask the last one because I want to move on to the next examples and see how what happens when you mix multiple data sources.

OK. Same one by region. Let's move on.

So yes.

Yes.

So now we have a function that knows how to execute the sequence,right?

Yes.

How did the LLM learn about the table essentially?

Yes, good question.

So let's go in here.

OK. So this is how. The way it works is that at initialization of the agent, so we are loading the instructions from the instructions files. So if I go back to

download, replace, if I search for this in the function calling instruction file,

here we go. So sales data tool. So here and I passed too fast on this when I was going through the file and explaining it to you. But so what's going to happen here is that the instructions of the agents is going to contain what the schema of the database is.

That's how the LLM knows what query to generate and what tables and columns are in the database.

So without actually passing this instruction.

It would not work at all.

OK.

Like nowhere.

OK.

Because it would have it would completely hallucinate the schema. Your grounding, basically, the LLM with what the schema of the database is.

Can I be more specific? Like what if I'm just saying, what if one column has some confidential data? And I don't really want to get data.

Yes, you could.

I can.

Yeah. Yeah. You could. And actually, you don't even have to be too fancy for that. You can literally add it to the instructions file. You can say, hey, the column X of table Y is confidential. Never return it.

Disclaimer, it is not bulletproof. It is not guaranteed that the LLM is going to follow those instructions. If you have confidentiality I mean, matters, problems in that space where you want to restrict access, you should somebody mentioned IAM or I think it was you.

You should implement this. You should add column level. And the best is to do what you would do with normal code because even if normal code is non-deterministic, the odds of a user getting access to something he's not supposed to get access to is still possible.

You can still use SQL injection, like all sorts of hacking to exploit normal code. Obviously, with LLMs, it's a whole different area. But so you should implement the exact same safety measures you would for a normal program.

Where were we? OK.

Yes.

If we were supposing that the schema is kind of perfect, this perfect world, we have a schema that we point, I mean, our agents to theright, I mean, table to theright.

I'm not sure what you mean by perfect because the schema of a SQL database is I mean, has to be perfect, I hope.

Yeah, yeah, yeah.

At least matching the data.

Yeah. But I mean, it depends on the one that read it. I mean, it depends on the name of the columns. It depends on the name of.

OK. So you mean clear. You mean interpretable. OK. Yes.

Yes.

Yes.

So in this case, when I mean, a real world.

Same answer as regarding security. It's the exact same answer. He asked, how can I make sure that confidential information is not returned if I want to? You add an instruction saying, do not use that column. It is confidential.

To answer your question, what you would do is document your schema. You would say, that table, which has a cryptic name that nobody understands, is actually the list of orders.

Yeah. So it's kind of a semantic layer.

Yeah.

Something like that.

Yes. That's exactly how it would work.

OK. Let's

where am I? I'm lost.

We did that. OK. OK. OK. Basically explaining what I explained. OK. Blah, blah, blah. Breakpoint no.

OK. So this is interesting.

Let me try that one. And I'm going to increase

here. So what regions have the highest sales? OK. So that's very interesting.

Why is it interesting? Because to answer the previous question, which was

total shipping costs by region, remember that the LLM has access to the context of the discussion of the past messages. It's stateful. It has all that context of the answers and questions that were previously asked. And it uses it when you ask a new question.

So for example, when I asked total shipping costs by region, it did not have the information required to answer within the previous answers. So it executed the SQL query. But now, when I asked what regions have the highest sales, it can actually use the table that was generated previously.

There is no need to execute one more SQL query because it has everything it needs. Exactly like you I mean, without doing anthropomorphic, but exactly as you, if you were reasoning about the problem and you were like, do I need to go write a SQL query in order to get that information?

No, you don't. You have it already. So the LLM is going to do that reasoning.

That's why here you don't see any SQL query executed. OK. Yes.

How do we know it's doing math theright way? Because like.

It's not doing math theright way. But here, we did not ask it to do math. We asked it to oh, you mean because there is a top. OK.

If you're adding two numbers.

No.

So highest OK. That's a very interesting question. So highest sales is not about doing math. It's about comparing. Like what's bigger or smaller?

11 or 1.

Yeah. And that's easy because that's something that does not I mean, easy. What I mean is that LLMs are bad at math. They are bad at calculus, bad at addition, multiplication, division. It's going to get it wrong. And it's funny because I have an anecdote that's from this morning.

We had a team meeting. And we were wondering what the compound value of a monthly and you were there. And what would be, after a year, the compound effect of a daily rate of increase of 1%? And in my mind, I remember the formula for this, which is you take the percentage, you add 1, and you put it to the power of the number of periods,right?

And I executed it. And I got a number. Somebody else was like, oh, that's big. And he typed it in GPT. And he got a different answer. And the LLM got it completely wrong. It was erroneous. It was very convincing.

It looked like it was doing wrong. But it was doingright. But it was not. It was the original answer, which I typed in Google Sheet with like a formula, was correct. So to answer your question, when you need to do math, you need to use a tool.

Here, we're using code interpreter, which can do math. But to be honest, I've tried. It's not the best way to do math, to use code interpreter. It's good at generating diagrams, at reading files, extracting information from a CSV, from other types of structured documents.

It's very good at that. If you try to do math with code interpreter for whatever reason, which I cannot explain, it does not work very well. When you want to do math, it's better to create your own tool, like calculate.

And it takes like a LaTeX expression or some kind of mathematical formalism to represent an expression. You have plenty of mathematical calculators out there that you can use to do the actual and accurate math calculation. And so that the LLM can delegate to a tool the responsibility of doing calculationsright, exactly as you, as a human, would do it.

Because me, I don't know. But 1% to the power of 365, I don't know how to do that myself. I use a calculator for that. And I'm going to have to reconnect again. Oh my god. This is annoying.

OK. So where was I?

OK. What were the sales of tens?

And here, we get a query.

We need to ask again to do a query on the database because in the past information, we do not have that information. We don't know the drill down of tens in the United States in April 2022. So it needs to execute a SQL query.

OK. I mean, you get the point. Let's move on. I'm going to stop the tool.

Exit. OK.

Oh my god. What's happening now?

No, I don't want to meet and chat with friends and familyright now.

OK.

Sorry, mom. OK. Let's move on.

Where am I? OK. Rounding with documents, very important. So we're going to do RAG. Vector store. You were asking if we're using a vector database. For whatever reason, I was blanking. But yes, we are.

And so we're going to increment those lines in main.py. So I'm going to go here, go back up. I think it's back up. Let me search for yeah, it's near the instructions.

So file search. I'm going to comment that one. I'm going to uncomment the file store here. Can I do that? Yes. Cool. No. OK. It works just no. OK.

Oh my god. Can I do that? No.

Sorry. I was trying to select multiple lines at once. But I'm going to have to do it

over the way.

OK. So now, we're defining we're creating a vector store. I'm not going to go into the details of the creation. There is a utility which does all the heavy lifting. But basically, what it does is that it creates an AI search vector store.

It reads all the documents which are here.

Oh my god. The Wi-Fi today.

I'd like to click on this. But

OK. I'm going to try to reconnect.

OK. Password.

OK. So like I was saying, I was going to show you the files.

OK.

Where are my files? Oh, here we go. So it's a PDF, which contains all the product information for my products, for tens, for whatever. And so what this function does is that it reads the PDF, chunks it, cuts it in small pieces, and uploads all those pieces into the vector database.

I'm not going to go into the details. I can, if we have time at the end, I can explain to you in excruciating details how that works. But notright now.

So tool sets, yes. I think I wanted to show you something else. But for whatever reason, I'm blanking.

Forget.

Yes, I remember. I wanted to show you the difference between the instructions file we were using before and this one. So function calling and file search. So

compare selected. So I'm using the diff tool here, included in VS Code,

to show you the difference. And so here, what's interesting is that you can see the difference between the instructions. So

OK. I'm going to yeah, that's better. So in the file search instruction, we have new instructions. So now, we have a context of product information vector store. We have a search tool, which allows us to search into the vector database.

And we have a few different things when it comes to content and clarification guidelines, such as the kind of questions that you can ask with new questions, like what brand of tens do we sell, which you could not ask before because you didn't have the brand information into the sales database.

That kind of thing. Also, what's interesting with the use case here, and really the colleague that created that content, it was very, very it created something very interesting because it allows to show many things. Like when you have a source of information, a source of data, think about how before, when we had to mix different data stores, different databases, and we had to inject them into some kind of big cube system to OLAP yeah, OLAP or whatever.

I even forgot. To do some very complex aggregated queries to link things together. Now, you can do that with AI. Like here, when I want to link to join information from a database with information in a PDF, which is not structured, I can use the LLM for that, which is quite extraordinary.

So when I hear all the skeptics about AI, I just don't understand because I mean, that's just insane what you can do. Anyway, I'm just going to skip for the rest of the difference between those instructions. But I just wanted to show you that.

Now, we're going to execute it. And I can type run.

OK. So as I said, it's uploading the PDF.

And actually, the chunking I said it was chunking. Actually, the chunking is done by AI Search, I think. I forgot if it's client-side or server-side. But I think it's server-side. It's one of the features of Agent Services that you do not have to take care of chunking yourself.

Creating blah, blah, blah. OK. OK. OK. Same. So now, we can type our query.

I explained this.

Run, run. OK. Oh, here are our questions. So what brands of tens do we sell? Now, we can ask this.

And so this time so what's interesting is that this is not using the SQL database. This is only reading from the PDF, nothing else, because there is no sales information required to answer that question. Also, remember that I killed the previous instance of the agent.

I created a new one. So all the past conversation was lost on purpose. This is I could have reused the same conversation if I wanted to keep the previous context. But I don't want.

OK. So outdoor living and alpine gear, plus some information about it. OK. Cool. Simple enough. Now, that's interesting. We're asking, which one do we sell? So are we going to query the database?

No, not yet.

Oh, you know. OK. That's because we do not sell hiking shoes. So it's not using the SQL database. It's just using the PDF. But what's interesting here is that there is no information about hiking shoes in the PDF.

OK. Let's move on.

So now, what's interesting is that it's referring to the previous brands we talked about, outdoor living and alpine gear. So what product type and categories are these brands associated with? Same, only PDF. Outdoor living, the tens, camping and hiking, both, actually.

Both do the same. To be honest, it could have summarized it since both do the same. But it did not. What were the sales of tens? Oh, now it's interesting because now, we are asking for specific sales information about a specific year.

So now, it's going to need to query the SQL database. So we are asking for the sales of tens in 2024 by product type.

Include the brands associated with each. So what's very interesting here is that the product type and the total sales, I think, come yes, the product type and the revenue come from the SQL database. But the brand does not.

The brand comes from the PDF. So because in the questions before, we read the mapping between the product type and the brand, that's how it's capable of adding the brand into the table here.

And that's also I'm not going to do the demonstrationright now. I'm going to do it at the end. At the end, I'm going to kill the program. And I'm going to restart it. And I'm going to re-ask the exact same question.

And you're going to see the difference in the question. Hint, because I'm going to ask that question without the previous question. And because what I said earlier, remember when I said that an agent can be goal-oriented and relentlessly work until it achieves the goal?

In this instance, we have not implemented the loop. So it does not have the planning capability. So it's going to fall short. It's not going to be able to say, oh, but in order to answer that question, I need to look into the PDF and into the well, you might get lucky, to be honest.

It might do it out of luck because sometimes, it does. Because sometimes, the planning is simple enough that it doesn't need multiple steps. But if it gets slightly complicated and you need multiple-step planning, it's going to fall short.

OK. Next one.

What were the sales of alpine gear in 2024 by region?

So very interesting too because the database does not contain the brand. So here, what it does is say, where product type like it automatically generates a like matching criteria based on the brand because it knows that the brand does family camping tens.

OK. So now, we're going to generate charts. And we're going to use a code interpreter for this. Oh, and like I told you I told you before I do that, I'm going to make an experiment. And I'm going to ask again

the last question, actually, the most yeah, this one, the last question. And this one most likely is going to fall short.

So now, I'm executing the agent again.

Yes.

OK. And I'm going to ask the last question. Right. So here, in theory, it's not going to work. Yes. What the heck?

Well, it worked. I guess it figured out that it needed to use the product database. Or maybe there is an instruction in the file search that says that when you ask a sales question requiring product information to go read the PDF, it's possible.

It's possible. So yeah, I think that's what happened here.

Question number four, actually.

Oh, you tried?

I might have to quit and launch again.

I know my response was different than yours. I got it gave me alpine and it doubled up on the second result.

It what?

It doubled up on the when you did it, it said it was just it was outdoor living for backpacking and camping tens and alpine gear. But for mine, it said outdoor living for both.

OK.

It was just one for one and the other for the other.

OK. Interesting. Yeah, it's not deterministic. So it's not perfect. It's going to make mistakes for sure.

Which is is that the whole eval thing? Like I guess that's where I struggle sometimes because it is that whole.

Yes. So I give a breakout session, by the way, on evals tomorrow at 12:30, I think, or 2:00 p.m. I forgot. But I'm going to give a breakout session tomorrow specifically on how to evaluate agents because yes, it's.

It's a little crazy because it's like, well, the answer might beright.

It's a problem. It's hard. OK. Let's move on.

Code interpreter. So now,

we're going to go back to our main.py. And I'm going to comment that one. OK. We're going to and every time, we keep the previous tools. We just add new tools.

OK.

So we can also just have all the tools ready,right?

Yes. At the end, when I'm going to uncomment everything, all the tools will be working at the same time. But what's very interesting is to uncomment it's a workshop.

Yes.

It's so that you get an understanding of exactly the subtlety of how everything works.

OK. So run.

So we're going to look at the instruction. And we're going to compare code interpreter with

file search. Compare.

So here, with code interpreter, we add visualizations. We add a chapter, a section in the instruction, which explains how to do visualization. So when you have questions involving visualization, we basically say, hey, go use the code interpreter to generate charts.

And at the end, yeah, PNG. OK. Let's move on.

Main.

There we go.

OK. So show sales by region as a pie chart.

Remember, in the previous in the slide I showed before, we were first asking for the sales by region and then for a pie chart. Here, we're asking for both at the same time. And so it's tricky because sometimes, when you need because basically, what this is doing is that it's calling two tools.

And sometimes, it's simple enough. The reasoning capabilities of the LLM is capable in one question and answer to

it has enough reasoning power to say, oh, I need to call that tool and then this one. And sometimes, it falls short. And it does not. And it does not have enough reasoning power. And you need to use a multi-step

system. So here, we saved the results here. So I'm going to OK. So here is the diagram, the pie chart diagram, with our answer. I can try to zoom to show. OK. So we have the driven of revenue by region.

OK. Cool.

I go fast on this. But this is incredible. I mean, under the hood, it's literally generating Python code to generate the pie charts. So basically, what that means is that you can generate a pretty crazy diagrams idea. You can be pretty imaginative in the type of information that you want to display.

For the anecdote, last year, at this conference, I gave a talk on code interpreter because at the time, it was a hot topic. It was brand new. And what I had done, I do kite surfing. And I had extracted one of my sessions where you're on the water.

And you do attacks. You do turns. You do jumps. And I had the XML file of my session recorded from my watch, exported. I imported it into code interpreter. And I asked to calculate how many times I turned or how many times I jumped.

And from it, it was able to read the XML because XML is a very rich, strict format and self-sufficient. And it extracted all the information, understood all the structure, and generated the Python code to go through all the data points in the file and calculate how many turns and jumps and height I had during my session.

It's a pretty incredible tool when usedright. Anyway, end of the anecdote. Show blah, blah, blah. OK. Next.

Why

download as JSON?

I forget what this one does, to be honest.

Yeah. OK. I mean, it can be useful. But let's move on. Continue asking questions about yeah, because there was no tools involved. Yeah, it just interpreted whatever was in the context before and downloaded it as JSON.

Blah, blah, blah. What would be huh, interesting.

I mean, it's really crazy what you can do. You can really, really be imaginative.

By the way, you hear a lot about one of the latest things in AI is deep research. Under the hood, deep research is an agent that has those tools, just that it's been developed by an army of engineers.

And they've made sure that everything works well. But it's basically what it's doing. I don't know how it's implemented. But I'm assuming it's a mix of AutoGen or something kernel or some kind of orchestration system that mixes tools together.

And it loops until it achieves a goal. And it can run for hours. Anyway, so what would be the impact of a shock event, 20% sales drop in one region?

Oh, yeah.

Oh.

Can you elaborate again the difference between what this can do, how it's not looping? But if you use AutoGen, then you could loop through the different tools that it needs and how it's planning orchestration. I see that it has planning and orchestration.

It's able to use several tools at the same time.

Two.

Yeah, two. Fine.

And for example, let me try to find an example. And

for example, if you ask a question, you have two data sources,right? One is product. One is sales. Imagine you ask a question for which it is obvious that you need to query one. But it's not obvious that you need to query the other.

The information is in the other. But you have no hint in the phrasing of the question that you should be querying the second in order to answer the question as a whole. It would fall short.

This system would fall short.

Yes.

Not to AutoGen?

Depending on how you configure it. And this one will the falling short would also depend on what instructions you gave it because you can hack your way through those kinds of problems because you can just, in your instruction, add some instruction saying, hey, if the user asks for that kind of question, you should also go look into the data source.

But the difference sorry, to cut. The difference with a multi-agent, multi-step planning system with a more complex topology is that you have an LLM, which every time you get an answer, looks at the answer and asks itself the question, is the answer correct?

Is this answer answering the question or not? And it's capable of extrapolating. It can do something such as, huh, this is not answering the question. What do I have at my disposal? What could I do more to try to answer the question?

And then maybe the first step, it did not query the second data store. But it's going to say, actually, maybe we should go look into the data store because actually, maybe the answer is in there. Or we should query the internet.

See what I mean?

And AutoGen can do that.

Yes.

They can configure it,right?

Yes. Automatic kernel or other more collaborative multi-agent system. Yes. But this is today, it's an introductory. So we don't go into those more complex it's a workshop of its own, just looking at multi-agent systems. Yes.

So you said that sometimes, I mean, the answer is not obvious. Or the question is the prompt itself. It doesn't point to theright maybe set of data or the tool.

Yes.

So here, what about having on top of those specialized agents, a coordinator that works with reasoning?

That's exactly what we were saying. That's a multi-agent system. And a coordinator, when you use the word coordinator, it's actually one of the patterns. We call it a topology. It's one of the topologies of multi-agent systems.

And you can build such a topology with AutoGen.

Yeah.

Heads up, it's not trivial.

It's not?

Not trivial.

The hardest question in multi-agent systems is, in my opinion, how to specify the definition of done.

So we did not open, did we? No, we did not. OK. So this is a simulation,

impact of a 20% sales drop in North America on global sales distribution.

Percentage of global sales.

Oh, yeah. Because in North America, here, it's dropping. So the percentage, obviously, of North America goes down, while the percentage of others goes up. That's a pretty cool example. What is the shock event was 50%. I mean, I'm going to skip because it's pretty obvious what it's going to do.

Which regions have sales above or below their range?

Oh.

Interesting. So maybe the previous question was

no, I don't think so. I think it's

huh, do we have a problem? Connectivity problem maybe?

Let me launch it again.

OK. Let me just so we are

OK. I want to show you real quick the Bing grounding with Bing search. So I'm going to skip over that one because this is important. Very often, you want to add internet search capabilities. So I'm going to comment uncomment this, use the other instruction file.

Ah, did it do?

OK. Here we go.

And the last one, code interpreter multilingual, we will skip it. It's important because that's how to configure code interpreter once you want to use you want to work on non-English languages. So with specific encoding, specific

fonts, that kind of thing. But we're going to skip it for today. Resource not found. Huh.

Well, that's interesting.

Did you have that problem? Did you try to that point? Did you have that issue?

No.

Is it working for you?

I got the error.

You have the same?

Yeah.

Huh. OK.

Is it because of the Bing grounding API?

Maybe we've had a change. OK. I'm not sure what happened. It may be because build was recently. And a lot of APIs have changed. And maybe that one changed too. I'm not sure. Anyway, what would have happened is that let's look at the question.

OK. So here, what beginner tends to resell queries are products. PDF. What beginner tends to our competitors sell, include prices, needs to go query the internet. You need to go query information about what competitors are out there and what they sell and for how much.

So that would use the Bing grounding tool, et cetera. So same logic as before. We showed and I'm going to wrap up and go back to my slide.

But just to finish my sentence. So with Bing one, two, three, Bing grounding, it's just one more tool that same as the database query tool, the vector database file indexing tool is one more source of data that the agent can use

to make informed decisions or rather reports.

So we've seen how to do more with LLMs, with function calling, with a bunch of very powerful tools. And yeah, Azure AI Agent Service. So yeah, the UI today was very simple. It's always a question when you do a workshop like this.

Do we go with a web UI, which is complicated and adds some React and web stuff that some people might not be familiar with? That one is like bare bone. It's just CLI, the bare minimum of code, focusing explicitly on making sure you understand how to build such an application and how an LLM works because it's a whole new paradigm.

And building those is not trivial. And as you mentioned, there is a question of evaluation. And so when it comes to evaluating no more just an LLM question answer, there has been many frameworks out there for some time now.

But when it comes to evaluating agents, it's a whole new world again because not only you are evaluating one answer, but you need to evaluate a whole conversation, plus because you need to call tools. So you need to evaluate whether the goods or the accuracy of which tools were selected.

Anyway, we're going to see all of this tomorrow during the breakout session, where I'm going to introduce the Azure AI evaluation SDK with the agent evaluation capabilities, which is fascinating. You have on that QR code a bunch of additional resources.

And I'm going to point back to

my contact if you want to.

One more thing.

Oh, sorry.

There we go.

Do you have questions? Because we have three minutes. Oh, no. One minute, sorry.

I don't know.

Oh, sorry.

Tools which are used by agents, should they be part of the Azure ecosystem?

No. So you can mix. When you create an AI Foundry Agent, you have a bunch of pre-made tools that are readily available. You just have to click on it and configure it. Like I said, I can't remember from the top of my mind.

You would have to go look, to be honest. But the tools we define today, they are client-side. We define them using code. Well, except Bing grounding because Bing grounding is you declare it client-side. But

the actual Bing grounding call is happening on the server side. But the SQL query is client-side. So we had a mix, actually, today.

So I know that the agent service in AI Foundry isn't crazyright now. So why should we use the agent service versus just the serving service for like GPT-4.0, 4.1? Because I know the agent service is limited to 4.0, 4.0 mini, 3.5.

So I guess if we can instantiate the tools in the application itself, do we need to use the agent service then inside of Azure? Or can we just stick with the regular old LLM endpoints?

You do not need to.

OK.

Like I said earlier, everything I just showed today with Azure AI Agent Service is just

a managed

AI system. Managed in the sense that it takes most developers, every time you build an AI application these days, you need a vector database. You need conversations. You need to remember those. You need to store those. You need to search on the web.

I mean, those are the basic features that every single AI application out there needs. I mean, most of them. So Azure AI Agent Service makes it super easy, manages everything for you. But no, you do not have to use it.

You can use the bare bone LLM completion from two years ago, and you can do whatever you want with it.

It's just it's easier. You have a full-blown API. It manages persistence. It's just easier.