Creating and scaling your own custom copilots with Azure AI Studio: Hanchi Wang

Hello everyone, my name is Hanchi Wang. I'm a Software Engineer Lead at Azure AI of Microsoft, and welcome to this talk. Um, if 2023 is the year when the world discovered generative AI, then 2024 is the year organizations truly began using and deriving business values from this new technology.

And, um, I can actually go quick on some of the slides because Cedric showed some of them already. But, uh, Azure at Micro at Microsoft Azure AI is, um, helping a lot of customers, and we definitely saw the AI adoption accelerating.

Um, and our customers, enterprise customers, are landing their AI solutions. At the same time, many other companies are just starting to explore what AI can do for them. And, um, but like, as AI engineers, you and I know that AI is not just a plug-and-play solution.

Uh, it cannot be just tugged onto existing applications. Large language models need to integrate with domain-specific knowledge and tools. Outputs of these models must be carefully filtered for content safety and thoroughly evaluated. Moreover, once an AI application is deployed, engineers must continuously monitor its performance to ensure it operates effectively.

Um, and Microsoft, my team, has developed a suite of comprehensive tools that enable engineers to build their AI applications efficiently. And today, I'll introduce to you two of our products, Azure AI Studio and Promptflow, and describe how they help engineers to build their own AI-powered applications.

Um, Cedric showed this a little bit already, um, and AI Azure AI's portfolio is hosted on Azure AI Studio, the development hub for building AI applications. Within Studio, you'll find AI services like models, cognitive capabilities, and search, giving you pre-trained APIs so that you can get into production faster.

We offer Azure Machine Learning for advanced data science, model training, and full LLM ops management. We also have an end-to-end suite of tools to make AI systems trustworthy, safe, and secure. And, obviously, last month at Microsoft Build, Satya announced the general availability of Azure AI Studio, which is a big milestone.

Uh, we now have more than 20,000 monthly enterprise customers using the platform. And Promptflow, on the other hand, uh, complementing the Azure AI Studio provides SDK, CLI, VS Code extension, and other developer tools to streamline the end-to-end development cycle of AI applications.

It helps developers to build, test, trace, and evaluate their applications with ease. Promptflow is 100% open source and free. You can use it purely in your local setup without the need of an Azure account, but it's more powerful when used together with Azure AI Studio, where you can track your assets, traces service connections, and run results in the cloud.

And, uh, Azure AI Studio and Promptflow have so many amazing features. I'd like to highlight three of their superpowers in this talk. Tracing and instrumentation allow developers to capture events from their applications and gain insights into what might have went wrong.

Evaluation capabilities give you ways to improve your application for quality and safety, and monitoring enables developers to be proactive and have visibility into production workloads. To show you a real-world example, here's an application my colleagues and I have built using Azure AI Studio and Promptflow.

Okay. Okay. Um, it's basically a chatbot app that helps to answer sales data questions for an outdoor equipment company. I have some questions prepared here already. So that you don't need to see my awkward typing. Um, but this app is using the Assistant API, and the Assistant API has access to two tools.

The first tool knows how to translate a natural language question into SQL query and fetch data. Uh, and the second tool is the Code Interpreter tool, which knows how to make use of that data and draw charts. Right?

And so this is the result of the first question, and you can see it helps to, like, it draws the chart and also explains what it is doing. And we can also ask a follow-up question, like.

No, I'm scrolling.

Oh. Oh, okay. Sorry about that. What's going on?

Um.

Do I need to duplicate?

Um, it's either on mirror or on extended mode. If it's on extended mode, you would have to move your item over.

Oh, okay.

Yeah.

Maybe that will be easier. Where is that? Okay.

Yeah. You could do duplicate as well.

Okay. I'll do this. Um,

yeah.

If you, um, name display here.

Here's your, there's your mouse. If you can, um, name display.

Mirror?

Mirror. Yeah.

Okay. Okay. Cool. And do I switch back once I go back to my slides?

No, because it's just mirroring from here to here.

Okay. Okay. Cool. Sounds good. Thank you. Cool. Um, so, yeah, you can see that the question I asked is the show the sales data in 2024 by months and ask it to draw a line chart. Right? It just did that.

And I can ask a follow-up question like, uh, overlay that bar chart with a line showing the percentage difference compared to 2023. And what's cool about the Assistant API is that it always has the chat history in the context, so that I don't need to restate the previous question.

It has, like, everything, um, in the context already, so that it hopefully will be able to, uh, like, answer that in a little bit.

Okay.

Thinking about this.

Okay. Um, this, um, it did draw a line chart, but, like, it's incorrect. And, uh, it does happen from in my preparation, so I had a backup plan, which is, like, um, this is actually the line chart. And that's why, actually, it is important to do evaluation and monitoring, which is something I'll show next.

Right? But this is a correct line chart. Um, unfortunately, the numbers are all negative, so it looks like this company is not very doing not doing very hot currently, unfortunately. Um, okay.

Now going back to my slides. Okay. Cool.

And here's the sequence diagram of the app. With the Assistant API, it has access to two tools. The first one knows how to fetch data using the query generated from a natural language question, and the second tool is the Code Interpreter.

And by adding Promptflow as an out layer of the application, it enables superpowers like tracing, monitoring, as we will see.

Okay. Uh, before I show you the code, uh, can you guys see this link?

We can. Right.

A little bit. Yeah. So all the code I will show, you can find it from this GitHub repository, and you can, like, uh, clone it and try it out yourselves. Tracing and instrumentation are top priorities for developers. Let's face it, the code we developers write doesn't always run the way we expect.

Right? Um, so that's why it's critical for developers to understand how the code really runs. Let me show you the code of the chatbot app now. If I.

Okay.

Oh. Um.

Okay. You don't see it. Right? Hmm. It's still in. Okay. Um.

Okay. So this is the public interface of the chatbot, and it's just a typical function. It takes a question from the user and returns a chat output here. The only thing special here is the trace decorator. Oh, sorry about that.

Provided by, um, Promptflow. And the trace decorator helps, um, to emite, uh, events, uh, emite traces to capture inputs, outputs, and events of a Python function, and it also automatically captures any interactions with LLMs like Azure OpenAI. And I can also quickly show you the code for the sales data insight tool.

Again, it's nothing but a typical callable Python class with the trace decorator here. And with the, um, and for the second tool is the Code Interpreter, which is the built-in tool from the Assistant API, so there's no code for that.

And, uh, as an engineer, you might be wondering what is the best way to test this application and look at traces. Promptflow provides a local UI just for that. Okay. And to.

Okay. To start the UI, this is a command you would run, and it pf flow test, and you point at point it at the chat completion function we were looking at earlier. And I have already run the command earlier, so, um, here is the UI.

You can see it's running purely on my local machine, and we can ask it a question.

Like, oops.

Like, the same question we were asking the chatbot app. While this, while this is running, I can look at the traces already that is coming up. It's still thinking about it, so I can click refresh to see the upcoming traces.

Um, now it looks like it has completed. Let's dig into the trace a little bit. From the high level, this is the out layer of the chatbot app. You can see the input and output from the chat. And, uh, we can look, dig into the sales data insight tool a little bit and see that the Assistant API actually modified the question a little bit, and the, the tool was able to reply theright output.

And if we really want to understand how the LLM was able to do that, we can drill down even one layer deeper to see, oh, this is the question from the customer, and this is the auto-generated SQL query.

And lastly, this is the Code Interpreter tool, which writes some Python function and eventually draws the chart we were looking at.

Here. Okay. And what, what I should point out is that all the instrumentation emitted by Promptflow are open telemetry traces and events. Uh, and open telemetry is obviously, uh, industry standard. That means you can configure your collectors to send the traces to your preferred destination, like Azure Monitor.

And it also means that if you have other part of your application already sending open telemetry traces, you can look at that and the traces sent by Promptflow at the single place. And, um, you may also be wondering, like, if I want to, like, collaborate with a colleague, how do I do that on a local UI?

Right? And Azure AI Studio has that covered. Uh, in my local environment, I have configured my traces to go to the, um, Azure AI Studio already, and that's how I got to this page. And you can see it's the same trace view on Azure AI Studio, and I can simply copy and paste this URL and share with, share it with someone.

Okay.

Coming back to this.

Yeah. That wraps up the first superpower tracing. Uh, the trace decorator can be added to any apps from single LLM calls to RAG, function calling, or single multi-work, uh, agent workflows, and it's framework agnostic. Promptflow provides a local UI for testing with trace views.

To keep the traces for a longer time or share with a colleague, I can view the traces in Azure AI Studio. Now let's talk about evaluation, focusing on application quality. LLM-based application can be unpredictable, uh, it might fabricate data or give incorrect responses, impacting the application quality.

Um, and o-often, like, folks may want to, like, try out different models, compare the pros and cons, and see which one is the best for their application in terms of accuracy, cost, or performance. Um, and if you recall, the a lot of the magic in the chatbot app happened in a single tool called Sales Data Insight Tool, and that was able to kind of translate natural language questions into SQL queries using LLMs.

Let's see how we can evaluate that.

Okay.

Okay.

Cool.

So the. Okay.

Okay. Um, due to the non-deterministic nature of LLMs, it's running evaluation regularly is definitely critical, and what's equally important is to have high-quality test data set. If you don't, um, Promptflow has a suite of tools to help you to generate a synthetic test data set.

In my case here, I have already prepared a test data set, and the question column is what a user would ask the chatbot app, and the ground truth column is what the Sales Data Insight Tool would generate based on that question.

Okay. Let's also take a look at the evaluation code. I'm using the evaluate function provided by Promptflow for the evaluation. Uh, sorry. I'm using the evaluate function, um, provided by Promptflow from this namespace for the, uh, evaluation evaluation logic here.

And the evaluate function, you can consider that as an execution engine that links the test data set and the target evaluation target and evaluators together. I like to focus on the evaluators here. The first evaluator here is a content safety evaluator that comes with the Promptflow SDK.

It helps to act as a safeguard to make sure the query and the generated output are free of any harmful, hateful, or, um, uh, violent content. And the evaluate the three evaluators following are customer evaluators, which gave me execution time, error rate, and SQL similarity scores.

The SQL similarity scores are calculated based on how similar the generated SQL query is to the ground truth. Okay. I have pre uh, I have run an evaluation previously, uh, before this talk, and I can look at the evaluation results briefly from the terminal.

But what is even better is that I can look at the evaluation results on Azure AI Studio for a better view.

Okay.

I can see the aggregated metrics at the top. And okay. If I refresh. Okay. Uh, I can see the aggregated results at the top. I can see my content safety metrics here. It looks like I'm golden for that.

And per row level result below. What's really cool about this page is that I can also look at traces for the evaluation in case I wonder how some of the metrics might have been calculated.

Okay. Uh, Cedric, in his talk, showed the model catalog, so I don't need to, like, talk too much about that. But another scenario that, um, evaluation really shines is to compare across different models. Right? Obviously, like, you there are, like, more than 1,600 models on Azure, um, AI Studio for the model catalog, and I can look at, like, benchmarks.

Right? But what is even better, like, in my case, is that I have a custom data set, and I want to evaluate the same data set across different models. I did just that earlier this week, and here is the result.

I found a couple of popular models on the model catalog. Uh, you can see the code here. Here, three Phi-3 Mistral, uh, Llama, and GPT, and I, uh, in a list view.

I can pick a couple of them to compare. For example, I can pick the GPT-3.5 Turbo, GPT-4 Turbo, and maybe Mistral large.

Okay. In this comparison view, you can see that if I if we use the GPT-3.5 Turbo as the baseline, um, Mistral large has higher execution time, which means it's slower, but it has better error rate and higher SQL similarity score.

If we pull GPT-4 Turbo into picture, GPT-4 Turbo has the highest execution time, which means it's the slowest among the three. It has the same error rate compared to Mistral large, and it has the highest SQL sim SQL similarity score.

So this is telling me that if speed is not a concern, then GPT-4 Turbo is still the best choice for my situation.

Okay.

With the second, uh, superpower, I just showed you the evaluate function and, uh, the built-in evaluators like the content safety evaluator, and also you can write your own custom evaluators using Python function.

For the final superpower, I like to concentrate on moving to production. Uh, application devel the deployment does not mean the job is done for an engineer. On the contrary, it's the start of a new journey. It's a developer's responsibility to continuously monitor the application to ensure the application always runs as expected, and the developer needs to look at, uh, both the details for each individual request and also the overall telemetry for the whole system.

Um, Promptflow, uh, tracing sends open telemetry spans and events, which can be collected by multiple tools, including Azure Application Insights. We deployed the chatbot app and started to collect traces in an Application Insight resource and built a dashboard.

Here is the dashboard. Oops, sorry.

Okay. Um, you can see there are performance metrics, usage metrics, and failure metrics on this dashboard. And there are so many interesting insights we can draw from here. I'll just highlight a couple. Uh, for example, the model duration is, like, basically the speed of the model.

Right? And GPT-4 Turbo has the highest number compared to other models, which aligns with the evaluation result we were looking at earlier. And for the token usage, we can see the majority of the tokens are actually used for the prompt, uh, and only a few are for the completion.

So if we are concerned about the cost, then maybe we need to, uh, short the prompt templates a little bit. And the failure tells me, like, how many of the costs failed, and some of the spikes may come from, um, some of the rehearsal I did earlier this week.

Uh, and if something really went wrong, we can also, like, dig into a specific trace to look at the same trace view and the stack trace. Okay. And

final slide. Uh, this is just a glimpse of what's possible with Azure AI Studio and the Promptflow. My team at Azure AI is dedicated to provide an end-to-end suite of tools to make AI application building easy and efficient.

At the same time, we also always have responsible AI in mind and provide enterprise-grade, uh, scalability and security. I can't wait to see what you do with Azure AI Studio and Promptflow. Thank you all.