Ship it! Building Production Ready Agents — Mike Chambers, AWS

Um, yeah, so my name is Mike Chambers. I'm going to pick you up a little bit there; I'm from Queensland in the eastern part of Australia, but that's okay. Yeah, very happy to be here. So I'm a developer advocate for Amazon Web Services, and I completely and utterly and only and totally specialize in generative AI.

Used to be machine learning, now it's generative AI. I'll be talking about why this slide is up here in a moment. Any tabletop RPG players in the room? There's got to be at least one or two. There are some.

Randall, you were saying you were before. Okay, it doesn't matter if you don't get that. But I will be showing some code, and what I'm going to be showing is how to get the code and how to get into production.

So if you're a developer, this is how you can get your code and get it to cloud scale. If you're a leader, then this is how you can get your developers to get their code and get it into cloud scale.

Just a little bit about the kinds of things that I've done in the past. So I was incredibly fortunate a couple of years ago to work with Dr. Andrew Ng and some colleagues from AWS on this course. This is the fundamentals of LLMs.

Over 370,000 people have taken this course so far. It was the first of its course of its kind. I also got to play Transformers with Andrew, which is cool. And if you get the reference there, I'm playing with Transformers.

Okay, and if you don't know who this is, that's Optimus Prime. Okay, I think I'm holding a card. What's the other one called? Anybody?

Megatron.

Megatron, thank you, yes. Yes, that'sright. So let me jump into it. Let me jump straight into it. Some of you saw that code was coming, and code is coming. Now, everything I'm going to show is prerecorded because Wi-Fi,right?

We know that. But everything that I'm showing also is in real time. So what I want to show, first of all, is just something super, super simple. This is a single Python function, a single Python file, which contains an agent,right?

And it's the agent I'm going to use for the demonstration moving forward. This is where we get to dice rolling and TRPG. The idea here is that I want to do a simple agent with a simple tool that's not getting the weather, because we know how to do that.

So in this code here, it doesn't use any frameworks or anything at all. It's actually kind of inefficient. But I have this one, what we'd call a tool here. This is my tool. It's how to roll a dice.

Essentially, it's a random number generator, but I'm calling it a dice roller. And you can see here, if you're familiar with this, it's using a Llama, and it's using the Llama 3.1 8 billion parameter model. Super tiny, just running on my laptop.

This is literally an agent running on my machine. So if we skip through here, I've got some things which I've sort of collapsed down for brevity. This is a simple agent. It's an agentic flow. We're going to talk more in detail about what agents are in a moment.

I have my system prompt where I tell it what tools it's got available. I then also actually do include some examples as well. The reason I include examples, again, it's a tiny little model, and it just needs a little bit of a helping hand to understand how to use these tools.

For example, to roll a d20. A d20, this is going to come up again in this session, so I'll tell you now. It's a 20-sided dice. If you want to get one, we have them on the AWS stand.

Come and see us afterwards. You can have an IRL d20. But we're going to have an agent which will roll it for us. So let's just open up the terminal. I'm just going to run this locally so you can see the kinds of things that my agent is going to do.

So I'm just running my Python code here, and you can see it's waiting for my prompt. So I'm going to, or waiting for my input. I'm going to tell it, okay, this is some terminology here from gameplay. Roll for initiative and add a dexterity modifier of 5.

That means something to the few people that put their hands up in the room. But it basically means that what I've done is I passed in my natural language into the agent. It's used its large language model and its language understanding to figure out what I meant by that.

It's then looked at the tools that it's got available. The tool it has available is to roll a dice. And then it's performed that action, and then we've got an outcome here. So it's rolled a d20. It's come back with a random number of 10.

It's added 5 to it. I've got my output of 15. So this is a simple, really simple example of what an agent and how an agent is structured and what an agent can do. I know it's just a random number generator, but I think it's pretty exciting.

So I want to take this code, though, which is all running on my machine at really low scale, and I want to make this cloud scale. I want to get this production ready. So that's what I'm going to look at next.

So let's, yes, let's get back into slides just for one moment. And I want to talk about the anatomy of an agent. We've just looked at it running there, but let's look at the anatomy of an agent. And as we're doing this, each one of these things, we actually have to get into the cloud, running at cloud scale, so that this is the components that we need to scale.

And so the first thing, I think it's probably the obvious thing, is the model. This is actually super simple and super easy for us to do, as in they exist already, so we can just go ahead and use those.

But this is our natural language understanding. Of course, we need this to be able to run our agent. The next thing that I've got on my list, this is my list. There's probably other things you could add to this, but I think this is a minimum list that you need for an agent.

You need a prompt. So you need something to explain to the agent why it exists in the world and what kinds of things it can do. Give it a personality, for example. Next, a loop. So this is the agentic loop.

So this is the ability for the agent to be able to think, essentially. So it looks at the input, and it can then process that input. It then needs to go and use a tool. But then it needs to evaluate whether that tool actually answered the question or not.

It needs to figure out if it needs to go and run another tool. It needs to loop around. It really is nothing much more than a while statement, if while, whatever, with some strings flowing around. A lot of this stuff is just string manipulation.

But we need that to be hosted and cloud scale. Next up, history. This is a really important one, actually, and I've had a couple of deep conversations about this recently. When I say history and I talk about conversational history, people think of that as like, okay, well, I'm asking my agent to do something today, and tomorrow I want it to remember what I asked it yesterday.

It's actually a little bit more deeper than that. The actual conversational history inside of an agent is crucial to the running of the agent. So when we ask it that question, such as rolling for initiative or whatever it might be, that goes in, and then it does some reasoning steps,right?

It decides what it wants to do, and then it goes and calls a tool. And the results, the stuff that we were talking about before in the loop. But it needs to remember those things that it's done so that at each next step, it can do that within the context of what it's done before.

So history is, and conversational history at a low level is actually really important. And then finally, I'm going to have tools here. So these are, of course, how our agent has agency to be able to go and perform something in the outside world.

So another important part. Are there other components of agents? Probably yes. Yes, there absolutely are. But this, I think, is the base fundamentals. You need these things in order to be able to have the minimal viable product, I guess, of an agent.

So I'm here from AWS. Let me tell you about what AWS can do to host these agents at cloud scale. The first one, I mentioned it before, is being easy. Obviously, anybody who works for a model or a laboratory would say actually making agents, making models is actually kind of tricky.

But they've done it for us,right? So the models exist. We've got Anthropic. We've got the Amazon Nova model. And this icon here represents Amazon Bedrock. So Bedrock is a suite of capabilities that allow us to be able to build generative components into the applications that we're building.

So we can just take these components, slot them in, and build something. We have models from a number of different leading providers: Amazon, like I said, Anthropic, Meta, Mistral, AI21 Labs. We have a number of different models available.

You can plug them into your system. So that's kind of the easy bit. Let's now build up the actual agent itself. And so for that, I have Amazon Bedrock Agents. And Amazon Bedrock Agents is fully managed. There's no infrastructure to manage with this.

You can just put your configuration in, and it will be cloud scale. So the answer to the question I'm here to answer is actually really straightforward. We're just going to go and use this service. Now, inside of that, we have the configuration that we're going to have for our agent.

Now, I have put here instruction. Instruction is the terminology we use. This is, again, the personality of the agent. It's a bit like the prompt. It's not quite the prompt because it gets combined with a prompt template to be the actual prompt.

But this essentially, for all intents and purposes, is our prompt. That prompt template, by the way, if you're interested, you can absolutely go and edit that if you want. But you get delivered a default prompt template with the model that you've selected.

Then we need to have, well, all of those things I talked about with the loop and the history, conversational history, it's all taken care of. So that's all just inside of the service. But then from our configuration standpoint, the next thing is the action group.

So this is where we connect into tools. So an action group is a collection of tools. That's what you can see that as. I've got a couple of different action groups here just to show that you can have more than one.

We only need one because I only have one dice rolling tool. And you see in the middle here, we've got Lambda. So hopefully you're familiar with Lambda. It's a function as a service. It scales super well. It handles that scaling for you.

It's just the way the service works. And it's a perfect place to host these kinds of tools. And therefore, your tools can do anything that Lambda can do, which is, I mean, I'd be interested if you can find a use case that Lambda can't do.

There are some. But anything that your code can do in there, your tool can do, including reaching out to the outside world, reaching to other AWS services, sending an email, launching a rocket, whatever it is that you want to do.

Actually, I'd really like to see launching a rocket. So I say that from time to time. We need to go do that. We need to build that. So I've whistlestopped toured through those components. The important things to remember there are the action groups, really, because that's maybe terminology you haven't come across before.

So now I'm going to dive into the console, and we're going to build this. Again, this is all real time. I'm advancing it forward with my clicker,right? But in between those things, it's all real time. There are some people in the room that are now saying, well, hang on a minute.

You were saying ship this to production. I'm not going to ship to production with ClickOps. That's awful. And you'reright, it is. And so all the things that I'm going to show you are all possible as well with Terraform or Pulumi or CloudFormation or a SDK or a SAM or whatever infrastructure as code framework that you want to use.

So bear that in mind. But I'm showing you in the console because it's easier to look at. Allright, so what I need to do is just, I'm at the Amazon Bedrock part of the AWS console page. I've gone down to Agents on the bottom of the left, or the down on the left-hand side.

And this next little part's going to be very bouncy ball kind of easy stuff. I'm going to click Create an Agent. Okay, so then I get to name it, and I get to give it a description so that six months from now, when I go into my console, I'm like, what was that agent for?

I say, oh, that's what that agent's for. So it's always a good idea to use descriptions in anything you do inside of AWS. So I'm saying it's an agent to help me play tabletop RPG. If those are the questions you have, by the way, we definitely go to the booth afterwards.

We can talk a lot about this kind of stuff. So I've got that set up. I scroll down. And you remember, one of the pieces of configuration that we needed was to connect it to our model. And so I get to choose models here.

I've got the Amazon Nova models. I'm going to select the Anthropic Haiku 3.5, 3.5 Haiku. It's a small, it's a capable model. It's fast. It's probably more than we need for this very simplistic agent, but it's a good one.

And so here, I'm adding in those instructions. If you remember before, the instructions get combined with the template to become the actual prompt. So you're a games master who can help me play tabletop RPG games. That's probably what you're going to take away from this,right?

Everybody's going to go away and start playing these games. So I've defined what I want my agent to be. So I click Save so that I've got it ready to go. And the next thing I have to do is Action Groups.

So once we've done that, yes, so we scroll down to Action Groups. And this is where I'm going to connect in my logic with the code execution that we were talking about before. So let's click Add. Guess what?

It wants a name and a description. We're familiar territory here. And so we can give it this description. Now, these descriptions are read by the large language model so that it can figure out what this tool is for.

And this is sort of this new paradigm of programming,right? So in terms of definitions and schemas for how these things work, we actually write in natural language in here so that LLM can read it and figure out if that's what they want to do or that's what they want to use.

So we've got our action group as a whole. Now let's go and add in the actual code. We have a few different options here. And this option here is a recommended one, which is to use Lambda, which we saw in that diagram before.

Also, with this quick start here, it'll set up the Lambda function for you. It'll do all the permissions for you, and it will hook it up to the agent for you. So it's a really quick way to get started.

Definitely recommend using that. And now we come down to define the actual tool itself that we want. This is my dice roll or my roll dice. I flip between the two. This one is going to be called Roll Dice.

And again, we're giving it a description. This description is important because it tells the LLM what this does and whether it wants to use it or not. And so it's got Roll the Dice with a certain number of sides.

So we need a way to pass in those certain number of sides. This was all in the code before as well. And so if I scroll down here, I get the parameters. And I get to, again, just describe these parameters that I want.

Just expanding this out because I'm zoomed in for the screen's sake. And I get to add in my parameter. And guess what? I give it a name, and I give it a description. And again, the description is something which is read by the large language model.

We're following the pattern here,right? So it knows what this is for. And I'm going to set this up as an integer value because I only have an integer number of sides on any dice. And I'm going to say it's required.

I need this to be there. I need to tell the model that if you're using this tool, you must provide me with an integer for that. Okay, and now I'm going fast, but I'm respectful of your time, and I want to show you this working.

So let's click Create. Again, this is real time. It's going to go ahead and build that Lambda function for me, integrate the permissions for me, and have everything set up. And so once I save this and scroll down, I can get back to my action group, dive into my action group again.

And what I have inside of there is now a link to the Lambda function. So code's coming back, and we just need to put this logic into our Lambda function so it can work for us. Now, I'm going to do a little bit of quick and dirty here.

I'm going to take the boilerplate that it provided for me, and I'm just going to whack in some function to allow us to do this dice roll. So here's my Python inside my Lambda function. So this is just inside the console.

You'll notice that as I'm coding here, Amazon Q Developer is actually built into the console, into this code editor. And it'll be making suggestions for me in terms of what code I might want to put in. So I'm going to import random, which, of course, is necessary.

If you're familiar with Lambda already, then you'll know that you get this event that comes in, and that event sort of triggers how this thing is going to work. And we have here a function which is passed in.

So it's telling us which function we want to use. And it sends in some parameters as well. So let's rattle through putting some code in. I'm going to say, okay, well, if the function that we're calling is Roll Dice, then I want to basically go and grab the number of sides, generate my random number, and off I go.

So with it being real time, you have to wait for me to actually type, which is the way it is. And you also have to wait for me to paste this one because I'm less good at doing this one by hand.

But in it comes. And we're nearly there. So the next thing I need to do is just format my response. So my response body here is going to have my random number generated. In this particular case, Q Developer inside of this IDE knows exactly what I want to do by this point because I know you.

You want to play RPG. So it's written the code for me. So I can just tab, select, tab, complete on that. And then with a little bit of tidy up just so that it works with the boilerplate that was there, just going to indent that.

Then I can go ahead and click the Deploy button. And that's all I need to do. So now I can go and click the Deploy button. Excellent. And that's all I need to do. So that Lambda function now is being saved into Lambda environment.

That's all you need to do. Everything should be ready to go. So if I go back to my agent, I'm going to have to prepare my agent. And I just want to talk through this just for one moment.

This is a production-ready environment,right? So we have the agent, and then we have alias IDs in here. We've got the whole software development life cycle thing going on. So you can have different aliases as you're publishing out your agents.

But once we've prepared that, we should be able to test it over on theright-hand side. And so let's ask it the question. I'm going to use a little bit more flowery language in this case because we want to stretch the model a tiny bit.

We're not really stretching anything at all. And as soon as we've got this and I figured out how to spell initiative, it will roll the dice for us. And as you might expect, it's going to work. But what's happening here is that this agent is now fully hosted in a fully managed environment.

It's going to work for us at cloud scale. And our answer comes back with a quick 15. I just want to go and wrap this up then. If you're interested in learning more about this, so I mentioned before about the courses we have on Deep Learning AI.

These courses are totally free. You get a free AWS environment to be able to play around with Amazon Bedrock agents completely risk-free. I'll put this QR code back up in a moment as well. Thank you so much. So please come and talk to me following my prompt template.

If you want to talk about anything more than just this at cloud scale, MCP servers at cloud scale, I definitely want to talk to you. If you want to talk about our new open-source SDK for developing model-first agents, this is the stuff I hadn't got time to fit into this presentation.

If you want an in-real-life D20, then come and join me on the AWS stand in the expo hall. If you want to talk about anything else, then thank you so much for your time. And I will let you get to lunch.