Build, Evaluate and Deploy a RAG-Based Retail Copilot with Azure AI: Cedric Vidal and David Smith

This is the workshop on developing a production-level RAG workflow, so you're in theright place if you want to learn how to build the backend for a chat application that works off of OpenAI and builds its answers based on information that we draw from databases and vector databases.

We'll see all about that in this presentation today. My name is David Smith. I'm a Principal AI Advocate at Microsoft. I've been with Microsoft for about 8 years now, after my startup was acquired back in the big data space, and I've been at Microsoft ever since.

My background is in Data Science. I also did a lot of work as a statistician, and these days I'm a specialist in AI Engineering. And I have with me today two other members from Microsoft that are also specialists in AI Engineering.

First of all, I'd like to introduce Cedric Vidal, who's on my team in AI Advocacy. If you would mind introducing yourself, Cedric.

Hello everyone. So, like David said, I'm Cedric Vidal. I'm a Principal AI Advocate at Microsoft, and I have a background in AI, self-driving cars,

software design architectures, and everything in between. I've been working in this space for 20 years, and today I'm going to help David with the workshop. Welcome, everyone.

Thank you, Cedric. And we also got Miguel Martinez, who came all the way from Houston. He's a Technical Specialist at Microsoft. Miguel, tell the crowd a little bit about yourself.

Absolutely. Hello everyone, welcome today. My name is Miguel Martinez. I am a Senior Technical Specialist for Data and AI at Microsoft. So a lot of our clients, you know, this can be startups, businesses, they hear about OpenAI and ChatGPT and all of those things.

So they think about, well, how can I actually use it for my business? How can I use those tools to drive business value? And that's where me and my team come in, and we help all of our clients develop some new solutions to drive that value.

Allright. Well, Cedric and Vidal will be here for the next few hours helping you as you go through this workshop. So they'll be wandering around if you guys want to head out. And if you need any help during the workshop, raise your hand and one of the three of us will come up and help you.

So thanks, guys. Allright, let's jumpright in. If you would like to get started, you can use that URL on your screenright there. Just pop up a browser on your laptop. I'll give you more information about what's going on when you get there, but you should be able to follow along and get started if you like.

Now, to participate in this workshop, it is going to be hands-on, so you will need to have your own laptop. And I can see it looks like everybody does have their own laptop. That's great. This is not something you'll be able to do on your phone or on a tablet because there's lots of work that we'll be going through as we work through a GitHub repository.

And that is the second thing that you'll need to have to participate in this workshop is a GitHub account. If you don't yet have a GitHub account, please go ahead now to github.com/signup and create yourself a brand new account.

We'll be using the GitHub Code Spaces feature to provide our development environments, but the free GitHub Code Spaces is more than sufficient for the work that we'll be doing here today. We are going to be building an application in the Azure Cloud, but you do not need to have an Azure account for this workshop.

We're going to provide you with a login to an Azure account, and we will have already set up all the resources that you need to work with this application. And that specifically is things like Azure AI Search, the vector database, Cosmos DB, the database which we're going to be using for our customer information, connections to OpenAI which we're going to be using for our LLM, and various other resources and tools that we'll be using in Azure.

You can do all of this on your own. If you do have your own Azure account and are willing to spend a few dollars in credits to run the resources for a few hours for this resource, by the end of this workshop, you will have

all of the information, code, data, and everything you need to recreate everything that I show you here today. One of the first things we will do, in fact, is to clone, to fork, in fact, a repository into your own GitHub account, and everything you need will beright there in there.

Allright. But if you would like to run through this at home, you will need an Azure account, and you can create one at the link you see on the screenright there. So before we jump in and I start giving you some demos about how this works, let me orient you a little bit.

That link that I gave you on the last slide will have launched you into a virtual machine. It is a Windows virtual machine, but we're not going to be using Windows at all. In fact, the only thing we're going to be using this virtual machine for is the instructions for the workshop, which you can see on theright-hand side of the screen there in front of you.

You'll be going through that page after page. This instruction workshop feature has some nice tools that you can use. For example, wherever you see the green text, you can click on the green text and it will paste that directly into your browser.

That'll save you some time when you're entering in the passwords and URLs and things like that and so forth. I will mention that we are in a virtual machine environment. It looks like the Wi-Fi is pretty good here.

But if you find that the virtual machine is slow or there's lots of lagginess because of the Wi-Fi, or if you just prefer to use your own browser on your own laptop, you can totally do that. You can just open up a browser and follow all the same set of instructions.

The only difference is you'll have to open up that virtual machine every now and again to look at those instructions, and you'll have to manually cut and paste the green parts from the guide into your own browser. So it's your choice about which direction you want to go there.

When we actually come to running shell commands and things like that, the dev environment that we're going to have you use is GitHub Code Spaces. You could totally run all this on your local desktop, but we've got to make sure you've got VS Code installed, you've got to have all theright Python libraries installed, et cetera, et cetera.

So to make things easy here, we're just asking everybody to go straight into GitHub Code Spaces where everything is set up and you can do exactly the same process for yourself at home. If you're not familiar with GitHub or not familiar with GitHub Code Spaces, just pop your hand up when we get started.

We'll come and chat to you and talk to you about how that works and what's going on there. But if you have used GitHub Code Spaces before, it should be pretty straightforwardright there.

Allright. So this is what we're going to do today. We are going to build this appright here. Or at least we're going to build the backend to this app and have you interact with that backend through a little UI that you'll be working through.

If you do want to build the frontend to this as well, we do provide the code for the frontend and all the data. That's also linked from the GitHub repository, so you'll have access to that. But I'm going to give you a little demo about this website just to set the scene.

So let me go over to my browser here. I think I have the website open. So

sort of the idea here is that we are engineers working for a retail company, something like REI that sells camping equipment and backpacks and trail hiking shoes, things like that. They already have a website which you can scroll through and you can see the products that they have available.

It's a pretty limited selection just to keep things simple. And we can click through to see product information for each of the products that this company sells. So if it is these Trailblaze hiking pants, we've got lots of details about the features.

There are some room for use, FAQs, there's a return policy, some cautions, technical specifications, user guide, care and maintenance, lots and lots of information available to the customer about all the products that are available at this store. In addition, this storefront has a customer login feature.

In this particular example, the customer Sarah Lee is already logged into this system. What we're going to be building here is a chatbot that operates on this website. You'll be able to access that, click on the chatbot button, and ask the question, for example, "What can you do?"

And this is going to connect to the system that we're going to build to get an answer to that question. This is a pretty simple question. It's coming straight from the LLM and its context to tell it that as an AI agent, I can provide you with information about our products to help you with your purchases, so on and so forth.

Now, we can also ask questions about specific products. For example, I'm going to pop up my paste here. Let's try one like. Here we go. What is a good tent that goes with

the trail walker shoes? Okay. So in this case, it's actually consulting all the product information that's available in that website to formulate that answer with the LLM, and it comes up with, "For your upcoming trip to Andalusia, I recommend pairing your trail walker hiring shoes with the Trailmaster X4 tent."

We're also going to provide the LLM with information about the customer themselves, their name, their status in the loyalty program, where they live, and their order history. So it's able to answer questions like this, "What have I already purchased from you?"

So in this case, the LLM is able to consult the customer's purchase history and give back the answer to say that Sarah Lee, a valued customer, has purchased the Kozi Night Sleeping Bag, the Trailblazer hiking tent, and so on and so forth.

So this is the system that you'll be building on the backend so that that chatbot can use an LLM like OpenAI to answer those kinds of questions based on the customer's purchase history and the products available from this particular retail store.

Any questions on that so far?

Very straightforward. Okay, great.

Allright. We are going to be building this on Azure and using various Azure resources, which we will provide to you to do that. We're going to be using the Azure AI Studio platform to manage the large language models and the various resources that we're working with.

We're going to be using Azure AI Services, which are going to provide various tools to us, but in particular, the OpenAI models that the chatbot is going to use to generate its responses. We'll be building an Azure AI project, which is going to manage all the flows that we have to have the chatbot gather its information and generate its responses.

The product database is going to be stored in Azure AI Search. And Azure AI Search is a vector database, or at least it has a vector database feature, which we're going to be using to match the customer's questions to the nearest or the most relevant products that are related to those customer questions and using that to provide context to the chatbot.

And lastly, the customer information is going to be stored in a database, a regular Rosen Column type of database, relational database, where we will extract the customer's information and order history.

Yeah.

Okay. So let's have a look at the architecture we're building for the backend in a little bit more detail.

Customer types a question into the chatbot on the website, and that gets extracted out and sent to an endpoint just as a simple string along with the customer's chat history as well. That question gets sent to Azure OpenAI, where it will get embedded into a vector format.

If you're not familiar with embedding, it's basically the idea of converting a piece of text, a piece of string, into a point in multidimensional space in such a way that the other points in multidimensional space we have already defined by embedding the product information pages.

The ones that are closest to the question are the products that are most relevant to that. And we'll use that to extract out the most relevant products and push them into the context of the large language model. At the same time, we'll also extract out information about the customer and their purchase history from Cosmos DB.

So we do the embedding, do the search in Azure Search to get the product information. We extract out the customer information from Cosmos DB. We feed all that information into a prompt. So we create a large prompt with information about the problem to solve.

You have the customer information about our products, information about the relevant products from the Azure AI vector search, information about the product's customer history, and then generating a response based on the customer's question. Any questions so far?

Allright. We'll getright into it just shortly.

I'll come into all this a little bit later on, but we're going to be using a tool within Azure AI Search called Prompt flow. I'll come back into the details a little bit later on. But what we're going to be doing is taking that same retrieval augmented generation process that I just outlined to you.

The steps are to take the question, retrieve related data, which in this case is our product data, augment the prompt with that information that's come from the knowledge base, generate a response with the large language model, and then send the results back to the user.

We'll be creating a version of that retrieval automated generation flow using Prompt flow, which is a development tool that's in Azure AI Studio to streamline this process of putting all that data together. We take the inputs, we embed it, we retrieve information from AI Search, we look up customer information from the database, put all that together with the customer prompt to generate the response, and that's what the customer sees in the website.

So that's what you will be doing. So with that, let me go back to the start here. Has anybody not yet been able to log in to that website? Anybody having trouble? If so, Cedric and Miguel will help you guys out.

The password will come up in just a second. Allright. So let me actually show you that.

Okay. So I've gone to that website, and you should have come to a page just like this where you can go ahead and launch the virtual machine.

I just get an admin password.

Yep. I'll show you how to get to that in just a second. Okay. So on theright-hand side of the virtual machine is the instructions. At the bottomright-hand corner is the Next button. That'll take you to the next page of the instructions.

And that is the page where you'll find the password for the Windows machine, which is, curiously, password.

And from then, you should be able to log in to the Windows virtual machineright there.

Okay. The next step after that, and I can actually go to the next step here, is to open up the browser. I'm doing this directly within the virtual machine.

We're going to browse to a particular GitHub repository that we provided to you for this workshop. You can just click on the link there and have it typed into the browser in the virtual machine, or you can cut and paste it into your own browser as you prefer.

Okay. You'll need to log in to GitHub at this point. Just a slight warning is that if you use single sign-on, this happens to us at Microsoft. We use single sign-on to access GitHub. That does not work through the virtual machine.

So I'm going to show you actually doing that directly from the browser, where I've already logged in to GitHub there.

And we're going to go to Code Spaces. It looks like I've actually already started a code space here, so let me just go ahead. The instructions will tell you to launch a code space. Oh, I forgot one very important step.

Before you launch any code spaces, we're going to switch to a different branch. Rather than doing the main branch, which is the one that you'll use if you want to do this at home, we've got a special branch here for this lab.

It's called msbuild-lab322. The only difference is this version of the lab skips all the deployment instructions because we've done that for you already.

And then from that branch of the repository, we're going to launch a code space on that specific

branch. I'll show you that in just a moment. But to launch the one that I already have here,

this is what happens once code spaces set up. Takes a couple of minutes for code spaces to warm up. Warm up. This one should already be warm.

And what we have here, in case you've never seen this before, is basically an instance of Visual Studio Code running in the GitHub Cloud directly in your browser. So it's the same user interface as Visual Studio Code, but it's just running in your browser.

If you're experienced using code spaces, you probably know that you can also run this directly from within Visual Studio Code on your desktop and connect to that instance. If you've got questions about that, happy to show you.

One of the things that we are going to be doing with code spaces, and it looks like I've already

closed the terminal, is logging in to your Azure account at the terminal. And the instructions to that, if you've been following along, you might be ahead of me already,

is to log in to the portal and actually do this directly within the local browser in this case.

Okay. This is the username we provided to you for this temporary Azure account. It'll only exist for the duration of this workshop. Everything will be deleted once we're finished.

One thing you'll probably find is because we're in a brand new Windows virtual machine, it thinks you're a brand new user to Windows. It thinks you're a brand new user to Azure. It keeps on popping up all these hints.

If you're familiar with Azure, you can just delete them. If you'd like to do an introduction to Azure, you can do it here, or you could do it at home. And the trick is to get back to the starting place is to go to the Home button there.

And then I won't go through these steps on the screen for you, but you'll be following through the steps on here to have a look at the resource groups we have available. Also, the resource group we provided to you in Azure, and you'll be able to inspect all the resources that we've already deployed for you that you'll be working with.

Lastly, I'll just show you this last part

is when we log into Visual Studio Code. Let me show an example of copying.

Sorry, I don't normally do it this way. There we go.

Where is my copy button? I'm not using my usual laptop here.

Allright. So what I would do, this might be a neat trick, is just paste it directly up here

and copy that back into my other Visual Studio window.

Allow. There we go.

Allright. It's usually easier than that just to show you that process of how you're going to connect your Visual Studio Code instance in GitHub Code Spaces with your Azure account is with this AZ login. After a moment, it will give me an eight-digit code, which I'll copy, and I'll go to this device login page to actually log into Azure.

And now all the commands that I run within the Visual Studio Code command line terminal will run against the Azure account that we've set up for you. So that's kind of the main things to watch out for as you get started.

Just continue to work through the instructions as you go. Cedric, Miguel, and I'll be wandering around to help you and put your hand up if you get stuck. And in about 20 minutes or so, I'll jump in and give you a bit more context about the Prompt flow that we're building.

Thanks.

Okay. And then do I need to open

the terminal?

It takes a little bit to deploy the code.

Yeah.

Okay. So it's still building. And then trying to say, like, oh, but you got to verify with two-factor on your side to the work account.

Oh, okay. You figured it out now?

Yeah. Yeah, I think so.

Okay.

So I was just about to try and launch the code spaces. I don't know if.

Yeah.

And so now this is building. I'll do thatright at the same time.

Yeah.

So it's like having the Visual Studio.

So if you're already logged into GitHub, it'll just continue to use that login.

Yeah. It already has everything. All the packages.

Yeah. Yeah. Before you do that, though.

Yeah. Yeah. Okay. So I'm logged in.

It looks like you're logged in. Yeah. Yeah. As long as you're logged in, you're good.

Yeah.

And when you forked it, did you uncheck this copy of the main branch only?

Oh, I haven't forked it yet.

Okay. Oh, good.

So I'm glad you mentioned that.

Yeah.

Oh, wait. This is what I got to fork then? The Contoso chat?

Yep. Yep. So click the fork button first of all. Yeah. Which is up here. Yeah.

And then.

And be sure to uncheck that box.

So which step are you doing? And then I can't remember.

There are two ways to do it like this. The easiest is to just delete. And the nice thing is now whatever GitHub account you're using, you'll have this in that account to work from when you get home.

Oh, yeah. Yeah. Yeah. I love that.

Yep.

I mean, the setup's awesome. I love that you have the VM and everything already to go. Okay. So fork the repo. We just did that. I can check it.

Yep.

And then you fork again on the cross branch.

Just telling you that you just skipped ahead. Yeah.

Oh, I did,right?

Yeah. It doesn't matter.

Just launch good cases.

Yep.

Oh, I got to go to that branch.

Yep.

It's the MS Lab 4.0.

This one?

Yeah. That one.

Go to code and then code spaces.

So I didn't try code spaces.

You have your toy driver. Yeah. So click on the code spaces tab up there.

You can see that I haven't.

Yeah. And then click create code space.

So that's available just to my private account?

Yep. Yeah. Yeah. It's limited in how much you can use. It's 20 hours worth of computer stuff, but those kinds of things. But it's a nice consistent environment. And what we can do on our end, which we do for all of our samples, is we make sure that when you launch code spaces, you get a command line that has everything installed that you need to go.

So there's no.

Right. Yeah. And then let's see.

Yep.

Oh, wait. Oh, it isn't.

It's doing it. Yep. So that's going to take probably 60 seconds or so to do the thing. So what we actually recommend is you go to the next step and then come back to it later. Yeah. Yep. Allright.

I'll let you go to it.

So I mean, it looks like you gave me an Azure account.

Yep. Yep. And there's the username and password.

I was able to log in. Oh, wait. It'sright here. Yeah. So I had all the resources.

Exactly. All deployed for you.

Yeah. Because some of them take a little while to provision. So for these workshop environments, it's easy for people to be able to jump straight in.

But none of this is AI Studio, though, okay?

One of them will be. You should have 11 resources. So you got an Azure AI project. So that's an AI Studio project.

Oh, okay.

Yep. And then the next step, we're going to actually log into AI Studio.

Okay. Allright.

No, no. That one, I'm not familiar. I think you will have to register something. It should be simple once you find out. But the good news is that all of this should be available on a GitHub. So today.

It's good to go to. Yeah. So today, the VM I closed, but you still have the GitHub. So later on, you can do everything. Yep. Perfect. Thanks, man.

Allright. Great.

And we have the code space.

How much IP rate for the code space?

Nope. There you go. Yeah.

Sorry, what? I don't understand what I'm supposed to look for. The image or?

Everything going allright so far?

The documentation.

Go.

Which part is in KeyArm? This look at type column for which? Oh, because you see, it says.

It should be the second tab across, I think.

Listed.

After.

Space.

Yep.

Yep. That one is code spaces. The one before. It's an area of the type. So basically it's saying.

This tabright here where it says preview reading, that's code spacesright now. So it's showing you the contents of the repository with a command line connected to that file system.

I'm so used to using sorry. Oh, I don't do that. Okay. I'm sorry. I'm so used to just using VS Code. I don't even get this other stuff. Sorry. So.

I don't want to use. It doesn't seem like you are inside the VM.

I'm not. I'm logged in.

Yeah. I won't support because I kept getting blocked on break.

No, but you should. It's weird because you should see icons here. You should be in Windows.

Oh, I had it earlier, actually. It showed up. I have it. Yeah. It showed up earlier. I don't know what happened. It used to have all the icons. And it showed iExplore and everything else,right? I guess it just stopped.

But yeah, I did it earlier. So I guess it just wiped itself or something. That's really odd because I did that earlier.

Okay. It's okay.

Yeah. No, I had it earlier.

That's incorrect.

Yeah. It did that the first time. You have to actually manually type it.

Apple. But I already logged into this ages ago. That's the first thing I did. So I mean, that's the reason why it's not loading, potentially.

Oh, no. Here we go. Now you can open the browser.

Oh, I did that earlier. I don't know why it wiped everything. Yeah.

Yeah. Because you know the tab that you opened? Oh, no. That's your thing.

Oh, that's my other GitHub. So that's my other thing. I'm just saying, I did this earlier already. Yeah. So that's why I'm confused because I already have this, which is the.

But the thing is, you should do that inside the virtual machine, Windows virtual machine.

Oh, I thought I did.

Because the whole environment is set up for you.

Yeah. I thought I did that already. That's what my friend was telling me. I did that like how many steps ago?

Okay.

Yeah. So I mean, I can just do it again.

Yeah. I guess you need to do it again. I'm sorry.

It's okay.

So you should be able to just click on it, and it should turn up for you. You don't have to copy-paste yourself.

Oh, okay. Well, I already clicked everything all over again. I just did this. Is it just because I'm in a different window? I want to just skip this part.

For whatever reason, it's just not loading.

It's just pointless. It's so weird. Well, no. The reason I'm saying this is because I had this load. I had everything else loaded. It's pointless to have to do all of it again. The only thing I stopped at was after I loaded this because I just needed to.

Yeah. When I got to Azure.

What you're doingright now, you're inside of the Azure.

No, I understand that. I'm just saying your stuff is buggy because it shouldn't not work across Windows because it literally just did this. I shouldn't have to go back five times to.

Say that again. Oh, no, no. I'm just catching up to you guys. Yeah.

Yeah. You're about to do?

I'm just logging into GitHubright now.

Yeah. That's where I was earlier.

There we go.

I'm sorry. I have to redo everything because it basically, yeah, it didn't refresh the data. It just didn't.

It fixed it. It fixed it for me.

Yes. There you go. Just to avoid confusion, you might want to close the other tabs. Otherwise, it's confusing.

No, that's not what's confusing it. These are other tabs from Anthropic that have nothing to do with that. That's not an issue. The issue I'm having is you're saying this thing should be loading from.

Literally, I'm just trying to understand what you said after this because I already have this person. Yeah. The MS build, I already have that, which I have in a different tab. So after that, what I was stuck with on.

Okay. What I'm saying is you open it inside your macOS browser instead of inside the VM file.

Oh, okay. I got it.

That's what I was trying to explain.

Got it. So copy the main branch only. I did this already.

So that's why I was saying that it would be better to close those tabs.

Oh, okay. Got it. Got it. Got it. Got it. Got it.

That's what's confusing.

Oh, I was just thinking about how to load the code space. And I did it in a different browser.

Yes.

So the recreate.

No, you could just open the one you had already created. Here. Yes.

Okay. So then, yeah, that works. That's fine. The issue I have is when I write out.

Takes a while, or? Takes about 60 seconds sometimes for the code to pop up.

I keep wondering if I can edit.

Yeah.

Yeah. And you do need to hit enter in the terminal afterwards as well to confirm. Yeah. Yeah. I'll take a look.

Yes. In this browser.

And then open the portal. Basically, redo what you've done, but do it inside the VM instead of your macOS browser. That's what I'm saying.

Yeah. I just took an AI architect role at 8451. It's like the Data Science Arm of the program. So yeah, we've had Claude and Azure as our main place for doing things. Now, we haven't used Azure AI Studio because we've been like a big Databricks shop.

Can you open?

Great idea, by the way.

Thank you.

So

let's see what we've done in the raw client.

Yeah.

So here.

Yes.

Oh, I literally just copied this.

Last command. Where did it say? Right. It's a word here. Provision what?

That'sright. You connected and you can close that tab now. You won't need that one again.

It might work. It's time to say refresh the credential or wait.

Maybe.

Do you know if this laps?

There. And show again. And just press enter on that to confirm. Yep.

Soright now, it's going to create the Azure DB table and the data.

Yeah. Now you should be good.

Where am I entering in any of this information from the?

So now you should. Oh.

No, that one looks.

Can you?

Where?

It said something about.

Yeah, but except you are in your macOS.

So now you should be able to go to Azure and see the.

I mean, yes, but inside the VM browser.

Oh, okay. That's what I was getting confused by.

Yes.

Okay. And that's just for that

is for GitHub. So it's to basically clone VS Code within the browser.

Yeah, but you have already the code space open here. Let me show you where you'll get to once you get to about step seven.

Okay. Perfect. There it is.

I've gone through the steps of logging into GitHub, cloning the repository, launching code spaces, logging into the Azure portal and AI Studio from the browser, and also logging into Azure from the Visual Studio command line. So that can run some commands.

And I've run this post-provision command here. You're welcome to have a look at what that script does. But let me actually show you.

Tool because.

What we have done.

Otherwise, you don't have much real estate.

So if you go to the homepage of the Azure portal, you can also do this from the Azure CLI if you prefer. You can have a look at the lists of resources we've deployed into this single resource group for you called Contoso Chat RG.

And you can see that there's 11 resources that we have launched for you. What that script did was to populate some data into Cosmos DB and to Azure AI Search. Let's have a look first at Cosmoso DB, which is the Azure Cosmos DB account.

And here in the portal,

I can go into and have a look at the Data Explorer.

Okay. There's a video to watch if you're interested. I'm not. Okay. And what you can see here is now within our Contoso Outdoor Cosmos database, we have a customer's database. And you can actually drill in there if you're familiar with using databases and have a look at the tables.

There's about 12 customers in the table and all of their purchase history. Likewise, if I go back to the resource group and have a look at the Azure AI Search research, which is called Search Service,

I can go to the indexes in the sidebar

under Search Management. By the way, if you can't see this sidebar, that happens if you're using a very small laptop screen or if you've got a large font. It might be hidden behind this hamburger menu here in the corner.

Go to indexes.

Type again.

Oh, that's a problem. I've got no indexes found. So I must have missed a step. Hopefully, you've got indexes there. And if not, we'll come back and check. I don't think I went through all the steps to actually pre-provision things there.

So we'll go back and have a look at that again. Okay. I just want to give you a preview of where you're coming to. Again, always any questions, just pop your hands up.

No, that one is not necessary because that one has previews.

Did I? Yeah, probably. I skipped ahead a bunch of stuff. Oh, there we go. A bunch of errors. Did you get errors too?

Yeah, I got errors as well. Permission errors.

Let's see if I can figure out what's going on here.

Yeah, yeah.

Okay.

That would be unfortunate.

One is that I can't basically

find.

When did that happen?

Oh, you know what? I know I skipped a step. I know which step I skipped.

We have one good question from the audience.

Yes. What's the question?

So I was just looking at the prompt flow example on step, sort of end of step seven, where we're looking at the graph of the pre-existing prompt flow that's been created. One of the questions that I had is, can the graph also be cyclical?

Can the graph be cyclical?

No. It's a directed acyclical graph. And that is because there isn't any symbol for any kind of looping like that. So yeah, is there a reason why you'd want it to be cyclical?

For example, React agents?

Yeah. Well, within a single component of that prompt flow is anything you want. It could be any Python code. So if you need to do that kind of interactivity within one of those nodes, you can do that. But the flow itself is acyclic.

Thank you very much.

Yeah.

It worked on the second time.

Yeah. I realized the step that I skipped because I was hitting ahead was I didn't do the bit where we create the .env file. So let me check. Yeah.

I'm trying to remember which step that was. Done that one.

Allright. This is the one I didn't do. Yeah. Yeah. I just skipped this step.

Oops. Clicked too late.

We had to run the post-processor script twice.

Okay.

Yeah. Did I log into this one? Let's make sure I did.

We have another good question here.

Sure. Go ahead.

So again about the prompt flow DAG. So this time, there's a lot of power in the DAG description that we can build. So one thing that we'd like to explore that we're currently exploring with a couple of our clients is how to sort of hit these kinds of DAGs from Enterprise ChatGPT to use the actions framework and then go to a middleware layer where a DAG resides.

So what I'd like to understand is what are our options to expose this as an endpoint to external sources other than like our web app?

So that's actually going to be about step 11 or 12. We're actually going to deploy it as an endpoint and then connect it to the website.

So that's the reason I cannot see that.

Say that again.

Let me take a look.

Allright. If you just press enter at this point.

Did you get anythingright?

Yeah. That's just to confirm you want to log into that particular subscription. Yep.

Okay. I figured out what I did wrong. I completely skipped step six, which is an important step. You can confirm if you've done step six by having a look in your explorer for Visual Studio Code. Once you've done that, you should have a file called .env.

And that is where we've set up all of the endpoints and keys that you will need to access the resources that we provided for you. And you'll also have a config.json file, which does similar kinds of things for AI Studio.

And then once that is all set up, I should be able to go back and run this pre-provision script.

Is it a classification error or something else?

I don't know. Let me have a look.

It looks like it's something else. Where's the config?

I'm under.

Yeah, it's correct.

Are you in theright directory?

It looks allright. I think it's because I ran the script already once. I'll just have to do it again.

You said the search service is VectorDB?

Yep, that'sright.

So one of the things we did in that pre-provision script was to take each of the mockdown files, which are in the repository. There's one mockdown file per product that the company sells. And then we just script indexing that into the AI search database, which essentially converts that entire mockdown file into one point in multidimensional space.

It is actually chunking it, yeah.

Actually, no. Actually, in this example, we don't chunk it, just for simplicity. If you actually do it through AI Studio on the search on your own data, it does do the chunking there. Yeah.

Yes, question.

Just while we're getting going, how does Prompt Flow, AutoGen, and Semantic Kernel all come together? Are they all competing projects at Microsoft or are they?

Not so much competing. And they have slightly different versions. Let's start with Semantic Kernel and AutoGen. Both of those are orchestrators. They come from different parts of Microsoft. Semantic Kernel, we kind of view as the enterprise product. That's the one that is designed for using enterprise settings, has strict versioning, API changes, all those kinds of stuff.

But like LangChain and other orchestrators, you can use it to connect different tools together in different environments. AutoGen serves a similar kind of purpose, but it comes out of Microsoft Research. So it's a little bit more cutting edge.

It's a little bit more flexible based on a slightly different paradigm. But that's not the one that we recommend for enterprise applications today. Prompt Flow is a different beast again. Prompt Flow is directly within the AI Studio product and is purely for orchestrating within an endpoint that you deploy through the AI Studio product.

So one endpoint.

One endpoint. The whole purpose is to create one endpoint that goes through, in this case, a RAG process. But it's designed to be more flexible than just for RAG.

Can I use Semantic Kernel to have multiple prompt flows?

You could use Semantic Kernel to manage multiple prompt flows as endpoints. So yes.

And also, you can embed Prompt Flow as a library. Deploying it as an endpoint is one possibility. It can be used to write integration tests, evaluation tests, embed in your application. Like if you have a Python native application and you want to embed a DAG in it, it's also kind of interesting.

Is there any live prompt flow that sits on top of Semantic Kernel so I can have multiple endpoints managed in one graphic?

Is there anything like Prompt Flow that sits on top of Semantic Kernel to manage multiple endpoints? Oh, just the graphic side of it.

I don't think so. And honestly, when you work with Prompt Flow for any length of time, you'll be working with the YAML files. It's nice to have that picture. It's great for these workshops because I can point to things and show how things connect.

And it's great for debugging because you can actually see how the data is flowing through the thing. But in terms of an editing environment, that's not really what it's for. Yeah.

We also have the Python DSL version now.

Tell us about that.

In addition to the graphical YAML-based DAG, now you can also use Prompt Flow in a more programmatic, code-first way, where you write the Prompt Flow configuration in Python instead of graphically, as an alternative.

That's in Prompt Flow documentation?

Yeah.

Thank you.

Yeah.

To be honest, I'm not sure how

generally available it is.

I'm sorry, to be verified.

But ultimately, this is the representation of the Prompt Flow. It's a YAML file, which just defines each node with a bunch of tags associated with its inputs and outputs and how it connects to the various endpoints and the types of nodes that we provide, excuse me, in AI Studio.

If you're running that as like a Python library, then at that point, it's fairly similar to using LangChain in Python?

Exactly. Yeah. Yeah. So there's a command line way to, which is what we're doing in this thing, is to run that Prompt Flow with a given set of inputs to generate the outputs.

So are you saying that there's a Python alternative to the YAML that's possible?

Correct, which provides a more LangChain-like experience.

And

AutoGen could, AutoGen could, in theory, do similar stuff, but it's more of a labs thing, not ready for a prime time.

Yeah. Like David was saying, the team at Microsoft that works on both projects are very different, and they have very different goals.

Prompt Flow and, and I'm blinking, Semantic Kernel are very product-oriented, so they follow a very strict software release lifecycle, whereas

the other team, the AutoGen team, is really research. So they try the latest cutting-edge AI things. They also use that for papers. So

if you use AutoGen, you are literally on the bleeding edge, and things might break. And things are experimental. So you might want to do that. It might work for your application, but you need to know where you're going into.

Yeah. We had used it for a hackathon, and we were able to put together like an agentic type flow really quickly with it, so we liked it. But it's good to know that we need to be a little careful with it.

Yeah, definitely.

Okay.

Yes.

I'll project and maybe David can hear it. So if I'm just like cursive, then if I can play back and make sure that we're understanding.

Thank you. So Semantic Kernel is more for the orchestration, and this could be more at like a closer to the app layer where I'm building a backend where I'd be using Semantic Kernel to orchestrate the LLM and the calls on that side.

Prompt Flow is more on, I want to build a DAG, manage its lifecycle, potentially evaluation. Hopefully, we'll get to that in the workshop, and then deploy it as one endpoint.

Exactly.

Is that theright way to look at things?

And like I was saying, you can also embed Prompt Flow as a framework, as a library. You don't have to deploy it as an endpoint.

But that's the use case we'll be using in this one, is it specifically as an endpoint?

Yep.

And to make things even more confusing.

Sorry. Semantic Kernel, you can also build agentic applications with it. It also has

some of AutoGen's use cases, simpler ones, but you can build agentic applications with it.

Can you give me an example of why I'd want to use

Prompt Flow?

Personally, I like to use it that way because during development, it makes things easier to write tests, to write snippets of code that you can reuse. So this is very convenient for that. So it's more like a development convenience rather than anything else.

But if you wanted to build, I don't know, like a rich Python desktop app where you happen to be wanting to orchestrate LLMs, that would also be a relevant use case.

Are you able to see the data in the database?

I would say I can jump in there, Cedric. Prompt Flow you'll find useful when you're managing all your resource within Azure AI Studio. I haven't actually got to show you that yet. We'll do that in a minute. But within Azure AI Studio, you can launch, you can create endpoints for OpenAI models, Mistral models, Llama models, anything from the model catalog.

You can create connections into other Azure resources like Cosmos DB and AI Studio, AI Search, rather. You can hook it up to our evaluation framework. We'll be seeing that a little bit later on. All the features in this for managing the entire lifecycle of the endpoint itself and all the resources that are required to make that endpoint search.

That's a situation where you'll mainly be using Prompt Flow, is in basically managing that connection between all those resources with the goal of creating an endpoint. You'll be, in our example here, calling from a web app, just regular React, you know, API endpoint.

But likewise, you wanted to call that same endpoint from within Semantic Kernel or any other orchestrator, you could do that same thing there as well.

And if I may add, what I like also about Prompt Flow is that when you go to AI Studio, because it's very, very well integrated in AI Studio, so you have the visual aspect of it. But if you go to the playground of AI Studio, you can configure a RAG application and also, I believe, a code interpreter application and export it to Prompt Flow.

So instead of having to come up with a whole design of a Prompt Flow-based application, you can just use the playground, configure it as you want, and export it and have a ready-to-use Prompt Flow application that you can reuse and embed in your project.

So you also have a workflow that goes from the UI to the code that way, which you don't with Semantic Kernel.

Okay. That wasn't very clear. No, that's okay. Google is worse.

If you'd like to play around with that side of it, it's not part of the workshop, but feel free because you have an instance of Azure AI Studio running already in your virtual machine.

You go through the steps of selecting the one project we have here, which is called Contoso Chat SF AI Pro. One of the things you might want to play around is the playground. We've provided you with GPT-4 and GPT-3.5 Turbo endpoints, I believe.

And this is the place where you can interface through a playground to test out the endpoints you've created. Same place as when you get into the Prompt Flow thing a little bit later on, you can then test out the connections between those endpoints and the databases and everything else that you've built your RAG application around.

From there, you have the Prompt Flow button where you can export to Prompt Flow. That's what I was talking about.

Yeah, absolutely. That's what we're here for.

So OpenAI Studio seems to be getting upgraded in the same three months or so.

Where do you start and stop in Copilot Studio? When do you start and stop?

Yeah. So Azure AI Studio is for mainly the purpose of creating an endpoint that works against an LLM and evaluating those endpoints. They're the main two use cases of AI Studio. Copilot Studio by hand is about building applications, not endpoints, either building a complete application like a chatbot application or any kind of user interface that has an LLM element, or integrating those types of applications into applications like Teams.

So Copilot Studio is for building entire apps. AI Studio is just for building endpoints.

And as you might have guessed, Copilot Studio was built with AI Studio.

In terms of the integration pattern, we're getting quite a bit of the same questions exploring that. Will you see it then that we build an application in Copilot Studio and then call endpoints that we build in

Azure AI Studio?

If you want that level of customization, you don't need to with Copilot Studio. It's designed in such a way you can build complete apps without having to customize your own endpoints. But if that's the position you're in for your particular use case, then yes, absolutely.

You can call any endpoint from Copilot Studio, including ones created with AI Studio.

As of yet, one of the reasons is that you can't evaluate stuff in Copilot Studio.

I don't believe I'm not so familiar with that product, but I don't believe they have evaluation features.

So if you want to do that, you'd have to start in Copilot and go to an endpoint.

Yeah, you would evaluate the LLM through its endpoint here in AI Studio. We'll come to that in probably about 20 minutes or so.

Thank you.

Yeah, welcome.

It's all a little confusing when you start.

Yeah. But here is the very simplest Prompt Flow I just generated from the chat playground. All it does is take an input, passes it through to the LLM called chat there, and then generates the output directly back into the playground.

So that's a completely unfiltered AI endpoint. Whereas if you use an application like ChatGPT, there's a whole much more going on with your prompts and the context and everything else in a more RAG style than just passing it directly to an endpoint.

I'll keep on going to catch up to where you all are so I can demo the good bits when we get to the end.

That's where I would just look to the documentation to make it a little bit clearer. Okay, cool. And then let's see.

Yeah, so you're just missing that in the documentation that makes it clearer. Oh, it didn't do it.

So you're already there.

It says no file.

Oh, because you're already seeding permissions.

Oh, no. Oh, I already seeded it. So just set up.

There we go. And then it's going to execute to set up the database and the index.

So it's going to set up the database, all the data, and then the index.

Sure.

Let me know if you run into any other.

What was it you changed?

What did you say?

Yeah, just what's up?

The question was, are we able to use another vector database besides Cosmos DB?

Yeah, absolutely, you can. When we get the Prompt Flow up, I'll show you that. But if we go into the Prompt Flow node, which I think is the Retrieve Product Information node, you can see it's set up directly as a connection to AI Search, but you can set it up as a connection to other vector databases as well, or even just an endpoint.

Okay.

What do you think is the biggest difference? Oh. What do you think is the biggest differentiator between AWS and Microsoft?

What do you think is the what do I think is the biggest differentiator between AWS and Microsoft?

Like why should I go with using Microsoft for startups versus AWS for startups?

That's a different question because I was going to go like the main differentiator between Azure and AWS is the enterprise scale. Azure is a database that's used by big companies like Microsoft and so forth, and is designed with a lot of features in that around authentication, security, scaling, monitoring that big companies that run production apps really need.

That actually makes it a little more difficult for startups, honestly, because the first thing you do when you start working with Azure is start dealing with things like resource groups and security. What's the you know more about this than I do.

We have to deal with that a lot. That's why we set up Microsoft for startups, is to help startups get into the process of understanding the different kind of process of working with resources in Azure, which is quite different from AWS in the sense of you can just spin up a single VM in AWS and you're done.

Microsoft, when you spin up a VM, there's actually six different resources that get created because it's there to support the enterprise use case as opposed to just I want a single VM.

And I will say Microsoft for startups, they have a pretty cool program, the Microsoft for Startups Hub. They can give you up to $1,000, $100,000 in credits for Azure. It's like different tiers. So you know, level one, they give you this many credits.

Level two, they give you this many credits. So you really get a lot of credits for developing your first startup application. And they also have a pretty good ecosystem. They have like different mentors that you can get pair up with.

They have like different sessions that you can attend to learn like how do I use this tool? How do I like implement this technology? So they give you the credits and a lot of the guidance to make that happen.

And the whole Microsoft for Startups team is here at the booth in Salon 9. So chat to them about the Startups Hub program and they can get you into theright level.

Do you have a question?

So that script loaded the search service with all the vectors. That's correct, yeah. Yeah, I had to start over because I ran that script before I did all the configuration.

There we go.

Yeah, so no, don't do that actually. Can you click on text embedding?

Yeah, it is connected. So yeah, I'm not sure. Let me talk to David and ask him.

But.

But it's going to get embedded. And then from those embeddings, you can create new embeddings for the embedding to get updates. So you have the embeddings in the database of all of your documents.

Can you click the deployment?

Yeah. And then from those embeddings, the embeddings you can find embeddings.

If

you wanted to store ChatGPT, could you store it in.

You could, absolutely. Yeah, the way Prompt Flow is set up is it actually manages the chat history for you as it comes through the flow. But if you want to restore it beyond that interaction, then yeah, you can store it in the database.

There might be a bug in the documentation that we missed.

What's it?

Some of them areright.

In case you might ask about like products or like a specific item, so it's going to search for those and then it's going to put those as a part of the workflow for the LLM.

Connection default as 0.8. Actually, it's AOAI.

Oh, okay.

Oh, okay. And by default, it is correctly set.

Okay.

I missed it when I went through it.

Okay.

But a few of them were. And what's confusing is that in addition to the Azure OpenAI AOAI connection, you do have a default.

Right, yeah, there's two sets of resources.

And so when you switch to default, actually you break things.

Oh, okay.

Allright, when folks get to that, I'll have you show them. Yep.

For those of you who haven't done step eight yet, the custom connection for Cosmos DB, I'm going to go through that now.

Go to the connected resources and view them all.

And then create a new connection.

So this is one of the things that the Prompt Flow does for you is it has standardized connections to all these tools. But if you want to connect to any other service, you can use a custom connection. And that, in fact, is what we're going to be doing here.

It looks like we've a question over the back row too.

We're going to add the key value pairs for our connection to Cosmos DB.

I'm going to grab that value from our .env file.

So

it's just any.

We have another good question here.

Yeah, what is it?

How does the vector search perform at scale? So if you have a million vectors and you want to perform a nearest neighbor search against a query vector, what are some of the strategies to prune that? And then how does that impact recall as well?

Yeah, well, first of all, that's the entire reason why vector databases exist is to exactly do that search quickly. They're indexed in such a way that they can do that nearest neighbor connection at scale and at speed. So you can certainly do that.

It's not very difficult to do yourself, to do an embedding for a bunch of documents or a bunch of chunks of documents and then do a nearest neighbor algorithm to find what is closest to the embedding for the customer's question.

But the advantage of doing it within a vector database is you can do that quickly and at scale. And what was the second part of your question?

Yeah. There's different nearest neighbor algorithms,right?

Yeah. There's like ANN as well. And then you have to, I think there's parameters that you have to tune. That affects recall as well, depending on how far you want the tree search to occur.

Yep.

And so what are some strategies to balancing that and getting the highest recall instead of just doing brute force? Because brute force, I'm assuming, takes a while at that scale.

So interestingly, and this might not be the answer you expect, is that at least in our experience working with real-world applications, vector search by itself actually isn't enough, despite playing around with the algorithms and choosing the parameters so you can expand out the search in non-smith documents.

What we've actually found is actually a combination of keyword search and vector search together actually outperforms either. And that's a feature that's built straight into Azure AI Search. I think by default these days, it actually defaults to a hybrid search.

In this particular example, just to keep things simple, we're just doing a vector search. But for practical applications, we actually recommend a combo of keyword and vector search.

And I'm assuming that metadata that you can search, like for example, keywords can be updated for a product in this case at any point in time for any vector,right?

Yeah, that'sright.

And then is there like the last question, is there an upper limit on the dimension size of the embeddings?

I'm sure there is an upper limit somewhere, but we haven't come across it for AI search at least. Yeah, I don't think there's one sort of in there by design.

Because this is supposed to be your. So the copy and paste didn't copy. I have the same issue.

Yeah.

So if you verify. What about using Entra ID with this, certainly external Entra ID? If you're trying to develop this for an externally facing application, multi-tenanted, and everybody having their own little space, shall we say, for every. Is there any sort of frameworks or

things like this that are set up to show how to sort of set that up across all levels? Because you're sort of across, like if you are, if you're going from what I'm talking about, you're doing a Copilot Studio into this, into AI search, into Cosmos DB, is there any sort of, is there anything written down anywhere?

I haven't been able to find it, basically.

Okay, that's an area I'm not an expert in, but Cedric or Miguel might be either of you too. Entra ID.

So using external Entra IDs, I know it's only come out a couple of months ago, but being able to create a scope across all of those systems so that the scope that you can see is for that external Entra ID, is there anything written down to show how to do that across each of the individual systems in theright way?

Or is there anybody working on that? Or is, you know.

You said Entra ID? So from an authentication standpoint?

So say again?

You're talking about the Entra authentication mechanism?

Yeah, Entra. Specifically the external ID, so using external IDs, so social logins and things like that. So if you're facing this to the outside world, enabling external users, external IDs to actually get scope across the whole thing.

Yeah, I know. To be honest, it's not a domain where I've spent much time. Yeah, so I'm not going to be able to talk much about it.

I can add a little bit of color. You wouldn't have external users log into AI Studio. That's for the developers and for the IT managers. But what you are exposing to the app, which the end users are then accessing, is those endpoints.

And you can manage those endpoints either by tokens or by managed identity. So whatever way that you want the app to talk to the endpoint based on that identity controls, obviously, what the endpoint is then able to do.

And then all the way through.

Exactly, yeah.

Okay. And then this at the moment on the, if I'm reading this correctly, the retrieval document, retrieve documentation from AI search is retrieving the vector and then it's handing it off to the actual document itself. So it's only the vector that's getting passed, not the document itself.

No, the vector is then used to retrieve the document, in this case, a Markdown file. And then the Markdown files actually get inserted into the prompt so that the LLM can see it, so that OpenAI can see it and form its answers based on that information.

Okay, and so that chunk is being sent.

Exactly, yeah. And I'll show you how that gets put together in a sec.

Okay, cool. There is one file there that has the prompt, and then they have the variable for the document. So that variable simply gets replaced by the text that it grabs from the search.

So you click on the open project? And

then you expand the sidebar, and then you're going to see Prompt Flow at the top.

So just to continue building on the Entra ID piece, because one thing that we're looking into is, for example, if we want to do granular access control and make sure that we don't pass to our Prompt Flow, you know, the ability for it to search any database and retrieve any data from a user, just to make sure that a nefarious actor might not be able to get data from other users through something like prompt injection.

I noticed that in the authentication type for Prompt Flow, we're able to use token-based, so Azure AD tokens. Let's say that we do have a user log into our app, pass through that token to call the Prompt Flow endpoint.

Can we then use that token authentication of the user to call subsequent components of our Azure stack? So to pass through that user's delegated token and just make sure that we're retrieving data for that user.

I don't think so. Go ahead, Cedric.

I was going to say yes, but.

I'm not a security expert. These guys know a lot more than I did.

So the only thing is that I don't know much about Intra. So any Intra-specific things, I don't know much.

Because I joined Microsoft not so long ago, but when it comes to normal security, you could use any kind of token, such as JWT token. And then, like you would encode a JWT token for any kind of REST API, you can pass the JWT token to the Prompt Flow endpoint and use it inside the Prompt Flow definition to pass on to whatever internal service you would like to.

So I think a very concrete example of that would be, let's say we have the Cosmos DB endpoint, and I want to ensure that I can only access the specifics user data in that Cosmos DB. So I want to actually use row-based access control where that user is only allowed to see certain rows in that.

Oh, yeah. I can actually show you something to do with thatright here. Now that I understand. So what we have here is the simple Prompt Flow for this particular application where I do the input. There's the embedding to retrieve the product documentation.

There's nothing secret in that. What's relevant here to your question is, let me make this a bit bigger so we can see. Here we go. And the customer lookup, which is using information that is provided by the customer through them having logged into the website.

So for example, when I go to this website, do I have it open still? Did I close it?

https://ai-tour.contosoweb. This link is given in one of the last steps of your

AI tool tea. That's notright. Contoso chat.

So in this case, when the LLM gets the prompt, it already knows who you are. It already knows you are David, and it gives to the LLM only David's information. It will be different if the LLM asks, you know, what's your name?

You're like, David. Never mind, I'm Miguel. Because then it could be jailbreak. But in this case, all of that is set up so that when the LLM gets there, it already has your name and it's authenticated and it has only your information and only your information.

And I think that's just about what you were going to show.

Yeah, this is what I was trying to get to. Allright, so when we're at this chat application, we can see Sarah Lee is logged in already.

So as we go through the Prompt Flow, one of the inputs there is the customer ID. And that's come from this app through the token that's being provided to the endpoint. Actually, that's a parameter to the endpoint in this particular case.

It's not set up exactly that way. But when we ask the question, what did I order last time?

What's important to understand there is that there's nothing in this app that is searching any database. All it's doing is passing the user ID and that question, what did I order last time, into that whole Prompt Flow. And then as part of that Prompt Flow process, which is a privileged account, it is then queried the database with that user ID to get back her list of product purchases.

Then the LLM is operating on that information with that question to generate that bit of text that you see. And that bit of text is the only thing that actually goes back to the app. So the app doesn't have any direct access to databases at all.

Is that what you're getting at there?

I think that is what we're getting at. And I think you hit the nail on the head when you're saying that Prompt Flow has privileged access to the database. What I'm trying to avoid is for Prompt Flow to have privileged access.

What I want it to do is to inherit the access of the calling user through, for example, his AD token.

You'reright. So I'm speculating this is not my domain, but I think the way that would work is through the features of the database where you give that authentication information with your query that prevents you otherwise, then would that prevent what you're trying to do?

Yeah, absolutely. I'm just wondering whether or not that's already something that you're looking into with, for example, the Prompt Flow connection to Cosmos DB.

That I'm not sure of, but I'm sure they are. Because that's the whole purpose for this thing existing in the first place. This is what Copilot Chat was built on, for example. And sort of that's all based on enterprise logins and things like that.

Questions there? Yeah.

Is there any support for dynamic control in the Prompt Flow? Like if you want to have variable loops in the control DAG, is that possible?

Yeah, the question about do you have variable control points in the loop? Yeah, there's absolutely. If you run this within Visual Studio code, you can set breakpoints in the Python code that runs within each of the nodes. Is that the question you're asking?

I guess depending on the results, like of one node, for example.

Oh, conditional.

You might want to route to different nodes.

Yeah, absolutely. Because within what's actually being run within each of those nodes, let's actually even take a look at that. If I go over to the Prompt Flow itself, let's have a look, for example, at the

LLM response node, I think. Actually, that one's not very exciting. Let's have a look at the customer prompt node.

So what's actually happening at this point is it's got,

this is actually just the prompt that actually gets built together. But you can see it's got this meta-programming language for item in documentation and so forth. What that's doing through at that point is looping through all of the matched products that are related to the user query, extracting them out from Azure AI search as vectors, then extracting out the Markdown files that relate to those vector indices, putting that directly into this prompt.

So when

I ask the question of the app, you know, what's a good pair of shoes?

That's not the only bit of text that is going to OpenAI at that point. What is in fact going to OpenAI is a whole bunch of text defined by this customer prompt here. And including telling OpenAI, you're an AI agent for the Contoso Outdoor product retailer.

You should always reference factual statements. The following documentation should be used in the response. And this is where the individual relevant products are inserted into the prompt. And to our question earlier on for this particular customer, this is where their previous orders are inserted into the prompt.

And then finally, the question, you know, what's a good pair of shoes, is sent to OpenAI. So it has all that context from that RAG process to formulate a meaningful response based on that particular customer, their purchase history, the question, and the products that are related to their question.

But I guess in this case, I guess the flow is kind of static.

Yeah, I wanted to give you another example. Here's a better example of that kind of thing. In this case, this particular node is just running Python code. So you could put conditionals into that Python code, for example, based on the inputs to do different kinds of things.

Anything you like, in fact.

May I add something? Because I think I know the specifics that you have in mind. What David showed, he showed two things. He showed in the templates, in the template nodes, he showed looping logic and conditional, but it's looping and conditional string rendering constructs.

And in the Python code, you can have any Python. Like you can have conditions, loops, whatever. But to your point, all the nodes in the DAG are going to be executed. You cannot have conditional node execution. But what you can have is inside a node, in the Python code, you can conditionally execute something.

But all the nodes are systematically going to be executed. It is not a business process orchestration system. It is really tailored towards building LLM applications. So it's simplified. It's not generic.

Does that make sense?

Yeah, it looked like in that Python node that you had, that's where it was doing the customer lookup.

So how does that tie together? I mean, I see the line connecting it, and then I see the Jinja template for the prompt. And the Jinja template was iterating over customers and that, you know, for, or sorry, iterating over the orders.

So how does that tie together? So where are you atright now?

This one?

Yeah, there was like the customer lookup Python that we were just looking at?

Trying to make it bigger so I can see it.

Oh, yeah. So one on theright, yeah, the customer lookup.

You have one node which queries the database, fetches all the information from the database, stores it into a variable into the context. And then the Jinja template uses the previously set collection of results for the rendering.

Right, that makes sense. So when you say it stores it into it, is that where the response, the orders on line 13, is that doing it?

Correct.

OK. And then if we click on the next one down, the customer prompt, and we go to that loop again, there it is. OK, customer.orders. So that's how it ties then,right?

Correct.

Yeah. OK, thank you.

Using input and output bindings on each node.

Yeah. And the arrows that are coming into the top of the representation in this graph, those are the inputs. And the arrow coming out of the back is the outputs. And there could be multiple of those.

So I can't actually edit it?

No, you need to click on the node and edit in theright edge page. Yes.

Correct. It's widely.

Yes.

The callback was it's called a visual editor, but it's really more of a visual reader, and that is absolutely true. I want to highlight a little subtlety too when you get to step 10. When you first run your Prompt Flow in Visual Studio code, you're going to be clicking on the run button once you've viewed the Prompt Flow itself in the Visual Studio code environment.

You can see the commands it's running. It's just running a little Python command to launch in the YAML file. But what I want to emphasize here is that in reality, everything here is running locally. And in fact, in the usual developer environment, it would be running directly on your laptop or a shared machine.

In this case, it's running on the GitHub Code Spaces environment, which is subbing in for your local environment here. And the whole idea behind this is you have a very fast, responsive place to try out different prompts, to make sure your connections are working, perhaps testing different types of LLMs, replacing them in the LLM steps.

So you can actually figure out, you know, what are the bits of the puzzle that go together to give you a good experience for the endpoint that you're trying to create just in a local environment. Now, I say local because, of course, the database is still in the cloud and the OpenAI endpoint is still in the cloud, but all the orchestration is happening directly on your local machine.

Our next step after this is going to be then publish that Prompt Flow into Azure inside its own container app, as it turns out. And then that's going to be a hosted cloud version of that same Prompt Flow, which is going to support the production use of that endpoint in your application.

Yeah, side effect of what David just said is that because it's building a Docker container, you can actually customize the environment and add packages. So earlier, we were talking about the differences between Semantic Kernel and Prompt Flow. One of the nice things with Prompt Flow is that

it's very interesting for web developers because they don't have to care about creating an environment, deploying a Docker environment, scaling it. The whole scaling is done automatically by the platform. So you just need to add packages. And so you can combine an LLM with some packages for some specific processing.

And the whole deployment is done automatically. So you can focus on the UI and the user experience.

Allright. And then let me get to the evaluation.

Quick question. So when we say, like, what else did I purchase? Does it also touch, like, does it also do the embeddings? Or does it skip that because, like, you technically don't need that,right?

Like, you just need the purchase history for that question?

So in that specific DAG, we will systematically query both the vector database and the customer information. So

there are two ways to answer that question. We will still be embedding the question because we still need to query the vector database. But

yes, when it comes to answering the question of, can you repeat the question you...

Yeah, it was, what else did I purchase?

Yeah, what else did I purchase? Then, because we will query the order history from the relational database, the LLM is going to pay more attention to that part of the context than the product documentation side of things. But it's really, we are really relying at that point on the feature of the LLM to be able to pay attention to what matters.

One more question. So like, if I asked it, like, what else did I purchase? Let's say, what did I purchase in the last month? So can it, like, form a relational query to...

Not in that specific Prompt Flow DAG. Because that specific Prompt Flow DAG just returns the, I believe, the last 10 orders from the history is what we do, I think.

And that's it. That's what we put in the context. Because that RAG application, you know, is for workshops and demos. What you're talking about is to do something else, which is text to SQL, where you take a query in natural language, you transform it into a SQL query that you execute against a relational database where you have a filter where date is one month ago or whatever.

That's a similar use case, but a different implementation.

And that's also an area to be wary of too, because that's an area where prompt injection could come into the fact. If you're forming a SQL query on the basis of user input, you've got to recognize that there might be malicious input in that process, which might generate SQL.

There's still an intermediate step. It's not directly pasting the string into a SQL query, but there still is an opportunity there for bad actors to control what happens at that SQL generation step.

And I believe we have a template for that. I believe Pamela has created a template called RAG on PostgreSQL, which is in the same Azure Samples GitHub account. That does exactly what you're saying. It takes a natural language query, transforms it into a SQL query, and executes it on a PostgreSQL database.

But you could do the same with Cosmos DB.

So that actually leads me into another topic, which I wanted to get to before we run out of time here today, which is about evaluation. So this is an important step. Anytime that you put any kind of an LM-based application into production where users are going to be providing input to that.

And in this context of a chatbot, the kind of questions you want to ask are, did my chatbot give a relevant answer to my user's question? Was the chatbot's answer grounded in the information that is available in my databases that is part of my RAG flow?

Was the chatbot's answer coherent? Was it good English? Was it understandable? And the other metric that is in that list, which I'm trying to rememberright now, is I'll get back to it in a minute. But when you get to step number

13, we're going to take you through a Python notebook, which shows you a process for answering these questions

manually, essentially. And then I'm going to show you how that's built into the Azure AI Studio platform itself. But we think of debugging in just regular apps, CES, and tests that we write for applications. It's very simple. It's a yes/no.

Like,

did the application return a positive value? And it should be a positive value. Very easy thing to test for in programming style. Much more difficult test to answer the question was, is the answer generated by my chatbot relevant?

How would you even program such a thing? And the answer is, you get an LLM to answer that question. Now, this particular chatbot application we have running on GPT-3.5 Turbo, very cheap, very fast, very reliable LLM.

GPT-4.0 Turbo just came out recently. I haven't played around with it a lot myself, but I imagine that will probably take the place of GPT-3.5 Turbo in a lot of these applications pretty soon. Next time we run this workshop, we're going to switch it over to using GPT-4.0.

But you've also heard of GPT-4. There are very large, very powerful LLMs that have reasoning capabilities in some sense. Now, you wouldn't want to use GPT-4 in a production application like this because every time the user types in a chat, not only are they going to have to wait quite a long time for a response, but it's going to cost you a lot of money on the endpoint.

In this RAG architecture, GPT-3.5 works great. As long as you give it the context, it needs to answer that question. But for this testing paradigm, for asking the question, is the answer

"Trail Master jackets are good" to the question, "What jackets should I buy?" is that relevant? That is the kind of question a powerful LLM like GPT-4 can answer quite readily. So think about how you might automate that process.

You might use the prompt to GPT-4, "Given this question and this context, the stuff that we put into the RAG and this answer, ask GPT-4, on a scale of zero to five, how relevant is this answer? How grounded is this answer in the data that I've also provided here?

Is that answer coherent?" And these are all things that GPT-4 can do quite readily. And we can use the scores in this case that GPT-4 provides as a ranking of how well GPT-3.5 is doing in our endpoint for generating its answers based on the RAG process.

And that's exactly what goes on here. In this notebook, at the top of it, you can put in a question. I just ran it on, "Can you tell me about your jackets?" You can have a look at all the code.

You can even see the prompts that it's using to GPT-4 to answer these questions. And you can see the actual answers that came back are in the next node up here. Here we go. "Hey, Sarah Lee, let me tell you about our jackets.

We have two awesome things, awesome options that will go with your previous purchase. Summer Breeze jacket, et cetera, et cetera." So that's the answer that the LLM came back. This is the context that the RAG process was provided by that it used to generate that answer.

And then with all the information, we can ask those questions we just asked. Was that answer about jackets grounded in Contoso's product database? And GPT-4 ranked that at a scale of five a rank of five on a scale of zero to five.

And likewise, we can ask questions about coherence, fluency, groundedness, and relevance. And we get the answers, "This particular question is doing really well." You probably also want to test out your LLM on some adversarial types of responses. For example, you might ask it the question, "I want to buy a toothbrush."

Now, remember, this is a camping store. They don't have toothbrushes. Nothing is going to come up in the database when we do the RAG search. Well, actually, something will come back because we always get back some responses that are somewhat close to our earlier.

But let's see how our LLM actually does here. When I run this notebook, it's going to run through those scripts. It's going to pass that question to our RAG flow, generate the response for GPT-3.5, and then ask GPT-4 to rank it on those four scripts using the prompts that are linked to in this script.

And when we come back to it, we can see the answer it came back with was, "Hi, Sarah. Since you're into outdoor adventures, I recommend the Fresh Breeze " and where's my scroll bar? Here we go. "Fresh Breeze travel something, something there."

There it is. There's my scroll bar. "Fresh Breeze travel toothbrush." Allright. This is interesting. Contoso does not sell a Fresh Breeze travel toothbrush. GPT-3.5 just made that up out of whole cloth. But this is what LLMs do. And we have to test to see whether or not they're doing these kinds of things for the type of use cases that we anticipate.

And we can detect that particular test is not going well. Groundedness score of one out of five. It really wasn't grounded in our data because there was nothing about toothbrushes in our context data that we provided through RAG.

And similarly, coherence, well, it was in nice English. So I've got a score of four for coherence, a score of five for fluency, but one for groundedness and one for relevance. And so now you can think about automating this process.

You can think about what are the types of questions that we want our application to do well at? What are the types of questions that we might want to, say, not give any responses to at all and score accordingly?

And I won't go through all the details of this, but when we get into AI Studio, there is a whole section here on evaluation. Let me just discard that.

And this is the process where you can actually load into it a bunch of tests, which in this case are not Python code or C# code or whatever. It's responses, so questions, responses, and context, and then automate the process of evaluating how your endpoint, how your RAG process does on all those questions.

So that next time when you add new products or next time when you decide to upgrade from GPT-3.5 to GPT-4.0, you've got a series of tests ready to go to evaluate how your application does in face of those changes.

Allright. Got questions here?

Good question on that.

Yeah.

So after you've evaluated the model and you sort of understand the performance of it, what typically are your next steps and what actions do you take to drive improvement on the measures that you see there?

That is an excellent question. I have a slide just for that. Honestly, this was not planned. But this is essentially the LLM Ops process, which is essentially the same as DevOps, but with a fancier name that gets you lots of funding.

Here's the life cycle. So exactly the same idea as when we build applications. We go through the ideation process. We figure out our use case. We do some exploration testing out to our data. We build our basic prompt flow in the LLM case.

And we develop our first version of that flow. And then we actually run it against sample data. This is the evaluation step. We're still very early on the process here. If the evaluations don't give the scores that we're looking for before we put it into production, the next step then is to modify our prompts.

You saw that Ninja template with a bunch of prompts around, "Do this, don't do that." You would modify those until you get the behavior that you're looking for. Maybe you change the process in RAG. Maybe you chunk the data differently.

Maybe you present it differently to the RAG process. Then once you get satisfied in that process, you would keep on testing that against perhaps a live user cohort or bring in some testers, bring in a red team to try and break it, and again, go through that same evaluation process until you're satisfied.

And then finally, you'll be ready to actually deploy that to production. You'd put monitoring in. You'd actually monitor live, probably a sample of actual user questions, responses, and have real-time charts, not real-time charts, actually, probably daily charts, of how your model is doing in scores, like groundedness for the types of questions that you can ask.

And that might be detecting maybe things are drifting because your product set has changed and there are trigger words in your products that are making the GPT model do a strange thing. Maybe you've got some adversaries that are coming in to try and hack into your system.

That might come up in some of your monitoring scores. And then you go back through that iteration process to go back and build and augment the model for its next deployment. Is that the kind of question you're looking for?

Yes.

Great.

Other questions?

I have one question.

Yeah.

Right now, when we look at the input from the user, you put text, you put what you have purchased or something like that. Can this be improved to take

graph or PDF file? I'm not sure my question is clear.

Yeah. I don't understand how graph fits in there.

Let's say a PDF file, yeah.

PDF file. Let's say I want to input my PO rather than I type something. Can this accommodate that kind of?

Absolutely. Yeah. Azure Search, for example, can index PDF files. Then you can do a search to find the PDF file that is most relevant to that user's question. You can then extract out from that PDF file context, which is put into the prompt, which is then used to generate the response.

And then you can put references back to those source files if it's a trusted user kind of a situation, so they can come back to see them. That's how Copilot works, for example.

OK.

Yeah.

But still, from the prompt, you can only input text,right?

From the prompts, well, yes and no. This particular example, everything is converted into text. But today, we have what we call multimodal models. GPT-4.0, for example, as the prompt, you can input not just text, but also images, even audio, not video yet.

Not video yet.

But you could set up that RAG application to insert into the prompt the images or the audio or whatever it is that you want the LLM to be able to reference.

Great. Great.

That's still relatively new because 4.0 doesn't have all of its multimodal capabilities out yet, but the principle exists.

OK.

Yeah.

Good.

And we have a Florence that we can use. And Florence version 2 was actually released earlier this week, which is a model which allows you to do image-to-text. So you can analyze an image, generate text out of it, and then take the text and give it to GPT-3.5 or something else.

Are you giving a talk about that tomorrow?

Yes. Well, I mean, briefly. It's one of the things I'm talking about, yes.

OK. And what time is your talk tomorrow?

That's a very good question. I forgot.

It's in my calendar. I'll take a look.

I'm going to join that session because that's one kind of primary request from our team. Andright now, I have built this customized prompt window, but it can only take text. And now they want to say, "OK, I want to use PDF file or even JPEG file."

So, well, like we said, for JPEG file, when GPT-4.0 when we make available on Azure the multimodality capabilities, then you will be able to use it directly. For now, you can use another image-to-text model, such as Florence 2 or something else.

For PDFs, so it really depends exactly what your use case is. Like, is it a transient use case? Are you storing the PDF long-term? Because if it's the latter OK. OK. So if it's transient, then an alternative approach is instead of taking the PDF and storing it into the vector database and indexing it long-term, which you can use one, two, three.

What you can use is do that. You upload the PDF, you chunk it, and you can actually the algorithms for embedding the chunks, you can actually run them in Python in memory. You don't have to do it in a long-term vector database.

So you can do the chunking and the embedding in memory. And actually, the vector similarity search function to find what's relevant, you can execute those functions in memory to provide your users with a transient ephemeral experience where they upload the PDF, and you query the PDF just for the sake of the current conversation.

OK.

I think I need to take this offline with you because my use case is a little bit different. I'll explain a little bit in detail, and then you can.

Sure. And you can do it in memory or in a PostgreSQL database or a Customer SDB that works too and delete the data after once you are over with it.

The last thing I wanted to say regarding because that's a good question is the last thing I wanted to say isright now, today, you can go to Azure OpenAI. You can deploy it at GPT-4 model. And in the chat, they have a chat section where you can chat with your model.

There, you can enter images. There, today,right now, you can go there. There, you can enter a picture, and it can do things with it. Like, hey, here is a picture of a website. Can you write the code for that website?

And it will write code according to the picture.

I know this one. I found out already. And this is more related to RAG, the prompt window. And so far, what I have developed can only take text. It cannot take picture or any PDF file.

And not to cut you off because we love these detailed questions, but I've been told that I'm going to get cut off up here in just a minute. And before I do that, I just want to let you all know that we'll be here for a few minutes for in-person questions around, but also come to the Microsoft booth in Salon 9.

Lots of people there to have you can ask exactly these kinds of questions of. So please go ahead and do that. Cedric is also giving a talk tomorrow about multimodal models at 12:30 p.m. in Salon 10. So please come on and check that.

But thank you, everybody, to coming today. If you'd like to do this at home, the repository is already in your GitHub account. And if you happen to miss that step, there's a QR code where you can get to it there as well.

But thank you, everybody, for coming today. And enjoy the rest of the conference.