Building Agentic Applications w/ Heroku Managed Inference and Agents — Julián Duque & Anush Dsouza

Welcome, welcome you all to this launch and learn. I hope you are enjoying launchright now, that the conference brought you. We are going to be talking about our recent product that we announced at Heroku, which is the Heroku Managed Inference and Agents, or part of the Heroku AI offering.

And we will be talking about the fundamentals of building agentic applications with this service. This is not a session, this is a workshop, so if you want to follow along, just bring your laptop. You only need a browser.

You are not going to install anything in your computer. I swear that is going to be easy to follow along. Also, this content is going to be available for you if you want to continue learning at home and having access to the Heroku platform during the event and the weekend.

So initially, we are going to do the setup while you get ready by signing up to an Heroku account, or if you already have an Heroku account, you can just get access into the platform. I will give you a link for you to follow.

Then my friend here is going to give you an overview of what we release, and then we will go and do the hands-on. My name is Julián Duque, I'm a Principal Developer Advocate for Heroku. I'm here, I am with Anush.

Yeah, hi. My name is Anush. I work as a Product Manager for Heroku AI. Thank you, everyone, for joining us. Really excited to talk to you guys as well as walk you through how simple Heroku is.

Of course. And let's get started. You can join the workshop Heroku AI channel in the AI Engineers Slack workspace. There I share the slides and some links. Also, this is the QR code that is going to take you to the workshop site.

So in the workshop site, you are going to find a link that has the content and a form when you enter your Heroku account email. If you already have an Heroku account, just enter your email that you used to sign up to Heroku.

If you don't have an Heroku account, go ahead, sign up for one. You don't need to put any credit card information. And with that form, you are going to get access to Heroku. You will be able to deploy applications, use our services during the duration of this week.

If you have any problems with the sign-up process, let us know. We are here to help. And this is a follow-along workshop. What are we going to do is deploy Jupyter Notebook to Heroku, and then we are going to load all of the workshop contents into this Jupyter Notebook, and from there I will take you with the workshop material.

So let's take a picture of this. Also, make sure you join the Slack channel. There is the information in there. And while you are doing the setup, my friend Anush here is going to give you an overview of Heroku AI and what we release.

Okay. Thank you for joining us, friends. How many of you are familiar with Heroku? Have you used Heroku before? Show of hands. Oh, nice. So to set the stage, this is the most exciting time to be buildingright now, and especially building with AI.

And this point has come before,right? There have been inflection points with technology that have fundamentally changed the way we build things. There was the internet, there was the cloud, there were web apps, and now there's AI. Previously, when Heroku started, we took a similar challenge.

People wanted to build and deploy their apps, especially with Ruby on Rails, but it was hard. It was hard to operate it, it was hard to deploy it, it was hard to scale it. Heroku made that super simple with Git push Heroku main, and a whole new host of developers could easily build, push, and scale their apps.

We are doing a similar thingright now, where we have taken on the challenge with AI. We have seen that people want to build with AI. They want to build agentic applications. They want to build agents that scale and operate in a way that is very simple.

We want to make sure that every software engineerright now is an AI engineer, and it is simple as attaching agents and AI to your apps. So how are you doing that? Currently, a lot of solutions are only for the day one problems.

But what happens on day two? How do you operate it? How do you scale it? There are so many models out there. How do you know this is theright model for your problem? How do you know your tools are running safely?

Heroku has taken a very opinionated and curated set of models that we believe work best for our customers and developers will enjoy. We have expanded that folder by deeply entrenching these models in an agentic control loop that runs on Heroku that has access to tools like code execution, access to your data, all under the trust layer of Heroku.

And for these extension of agents, we are using the Model Context Protocol. You might have seen online that people keep asking that, who's going to build the Heroku of AI? Or, this is the Heroku of X. So the question for who's building the Heroku of AI?

It's Heroku, of course. Why wouldn't we? Okay. So what are the challengesright now people are facing? One of the things that I can see is that, how do you figure out that this model works best for you? How do you know that it's evaluated and traced and has theright technologies to make sure that it is performing the way you want it to perform?

So these are the challenges that we're taking on. We are curating these things such that whenever you work with the agentic applications, you don't feel like it's a bunch of knobs and bells and whistles like a plane, but it's actually pretty simple.

By taking on that opinionated approach of having those defaults for you, you can get started with as simple as one CLI command. You just do Heroku AI models create, and that attaches AI to your app as a resource.

Moving on to the next slide.

Of course.

Okay. So we have three major things that work great for an AI application or building agentic applications. One is we offer primitives like inference, where you can curate and take a set of models and access this in your apps.

We also have Model Context Protocol to extend your apps. You can build remote MCP servers on Heroku in a simplified way, and you can also build standard I/O MCP servers that run in Heroku's trusted compute. And they can also scale to zero so that you do not pay for things that you're not using.

And we also have had PG Vector, which is a great vector database for embeddings. So all these together give you all the primitives that you need to build agentic applications. So how do we do this? Heroku has a trusted compute layer called Dynos, and this compute is the one that runs tools for you.

So example, we provide first-party tools like code execution that run on Heroku's compute, and these can run on the compute and stream the data back to you and solve your problems. We plan to offer our additional first-party tools such as web search for grounding.

Memory is really importantright now, so in the future we'll probably offer memory and other compute tools that can run on Heroku's compute and provide to you. You can bring your own tools as well using MCP that can run on our compute and stream things back to your agents.

Okay. With that, I'll hand over to Julián for a more hands-on workshop, and you can build all the things I spoke about within the next 30 minutes.

Okay. Let's go back to this last picture for the folks that just joined, so you get access to the workshop content. And remember, we have the Slack channel in the AI.engineer workspace, where I already shared these slides and the link, so you can follow along.

You can follow along with meright now, or you can follow along at home with more time. So what are we going to do is the following. So let me show you. That link is going to take you to this page.

In this page, you have access to the workshop resources. This is a website that has the instructions step by step of what you are going to be doing today. I'm going to be following these same stepsright hereright now.

And second, we have a way for you to get access to Heroku. If you already have an Heroku account, just put your email, the email that you use to log into Heroku. If you don't have one, go and sign up first.

You can sign up for Heroku. You don't need to enter your credit card information. With just this invitation, you are going to get access to the workshop. These services are going to be enabled until

the 7th, like this weekend, and I can extend it if you like the content we have for you. This is going to send you an invitation to a team. The team is called AI Engineer Worlds Fair. Go accept that invitation, and then you have access to this dashboard.

So from the Heroku dashboard, you go to dashboard.heroku.com, you log in, and you should get access to this view. We already have some pre-deployed applications here. The workshop that I will be showing you, the Jupyter Notebook, the Brave Search MCP, we are going to use a pre-existing MCP to give you an example of how to call that MCP from the Managed Inference and Agents endpoint.

And I also have another application here, which is the AI Engineer data that has an Heroku PostgreSQL database, and we are going to see how can we build agents that has access to this database as well. But we start the workshop first by deploying an Heroku application.

So we have this Heroku Jupyter template that you can deploy to Heroku. Go to the repository. After you have the account logged into Heroku, just click to deploy. Click to deploy. You are getting this page that will ask you for an application name.

Make sure that this application name is unique. And the internet situation is a little bit slow. Let's hope it gets better to show you the things work in life. Otherwise, I need to start getting access somewhere.

If you joined us late, please go to the Slack channel.

It's called Workshop Heroku.

Heroku.

Yeah, AI.

Heroku AI Workshop. And in the AI Engineer Slack channel, you can find the how to get started instructions. If you guys are stuck somewhere or you don't know how to get started, please raise your hands and I can come help you out.

Beautiful. This is the deployment page. We are going to create a new application. Remember to have a unique name. Let's do this as Jduque Jupyter. This is going to be my Jupyter. It is not unique. So if it is not unique, you are going to get an error.

So let's do workshop to get a unique. And for the app owner, if you want this workshop to work without you paying anything, make sure to select the AI Engineer Worlds Fair, which is the team we have invited you to.

These Jupyter Notebooks are password protected. So there is an environment variable here for you to define the password to this workshop. I'm going to use a super secure password, lab, and then click to deploy. So this is going to deploy the application with a Dyno, where the Jupyter Notebook is going to run.

The Dyno is pretty much the container unit where your application runs, like the virtual machine on Heroku. And it also has Heroku Postgres, so all the work that you are going to do in the Jupyter Notebook is going to get persisted into this database.

And this is going to take a little bit of time to install. This is pretty much fetching all the source code of this Jupyter Notebook template to Heroku, building that application and all of the dependencies, and then you have this available for you to use.

So basically, you are going to get something like this. I already have this deployed, but you don't have any workshop here. So we are going to be loading that Notebook. And to load that Notebook, you go and keep following the instructions.

Oh, the third step that I miss is, how can we provision Managed Inference, mainly how can we provision an AI model to an Heroku application? So Managed Inference is a service that lets you run AI models within the same infrastructure where your application is running.

So the data is not going to third parties. If you are using, let's say, OpenAI or Anthropic APIs, the data is going outside your application. This is keeping everything running inside the same network that your application is running.

And to provision these, you can use the Heroku CLI, but we are going to make things easy. Once the application is deployed, this is taking time to deploy, so let's go to something that is already deployed. Let's say this one that I already created before.

You go to your application in the dashboard, you click to the management page, then you go to resources. And in resources is where you can provision add-ons. Add-ons like Heroku Postgres or the Key Value Store or any other third-party add-on that we support on our marketplace.

The one that we are going to provision is called Heroku Managed Inference and Agents. This is our AI offering. So you are going to click on this add-on, and then you will select the model that you want to use in your application.

We support text-to-text models from Anthropic, Cloud 3.5, 3.7, and 4. We support Cohere Embed for embeddings, and we support Stable Image Ultra for image generation. We are going to be working mostly with text-to-text with inference models. So let's do Cloud 4.

We submit, and that's it. Now my application can access AI services. That's the only thing that you need to do. And how this works, this is going to enable your application to have access to environment variables that contain the API URL, the API key, and the model ID.

And with those three things, you can use an existing SDK that supports the OpenAI specification or the OpenAI API, or you can perform an HTTP request or build your own solution to run these applications. So now we provision AI into our app.

Now another app is deployed, so we can do the same with that. So I'm going to keep following. Then let's go and load the workshop. So I'm going to be copying this URL. That's the Jupyter Notebook we are going to load on our Jupyter template that we deploy.

And here you go to File, Open from URL, you paste that URL, click on Open, and voilà, we have our Notebook ready to start. So remember that we provision the Managed Inference and Agents service into our application. That is going to give us three environment variables.

If I go here to Settings, Reveal Config Bars, we see the environment variables there: the inference key, the model ID, and the inference URL, the API key that we will use to call the service. So obviously, the first step that we will do is to load those environment variables on our Jupyter environment.

So we will do this workshop in four parts. The first one is set up the environment. We are going to just load those environment variables so we can continue doing our activities. Second, we are performing Managed Inference. This is your basic chat completions endpoint, and we will see two examples, one doing basic chat completions and the other one doing a streaming version of those chat completions endpoint.

The third, which is one of my favorite ones, is the Heroku tools or the agents part. We have another endpoint that lets you run agents that we support natively. We have code execution agents, we have database access agents, document conversion, execute commands on a Dyno, and we are going to continue adding more to those.

And this is just an endpoint that performs the execution of the agent in the Heroku infrastructure in our Dynos, and it is just an one-off Dyno. What is a one-off Dyno? It just spins up, runs the code, and it scales back to zero.

So you are not paying for an application that is constantly running. You are just paying for the amount of compute that that tool took to execute. And that endpoint also supports MCPs. MCPs that you can deploy to Heroku and then attach them to your inference agent endpoint.

And last but not least, we are going to take a look at that MCP support that we have. How can you deploy an MCP to Heroku, attach that MCP to the agents endpoint, and also use those MCPs externally, remotely, through the MCP gateway or MCP toolkits that we have?

I know it sounds like a lot, but it's going to be easy. I have a couple of exercises for you if you want to write some code. But basically, what I will be doing here is just running the code that I already have implemented here.

My good friend, thank you for coming. OK, so first thing I told you, we need to load these three environment variables: inference URL, inference key, and inference model ID. And there is a fourth one that we will use for certain tool execution on Heroku, which is the target application name.

So what I'm going to put you here in target application name, the name of the application you deploy, the name of the application, the Heroku Jupyter application that you deploy. In this case, my Jupyter Notebook is called AI Engineer Workshop.

This is the one that I'm going to use here. I am giving permission to my tool to run commands and perform compute operations on this Dyno, on this application. So here you just replace the name of your application so you can do those examples.

And later, I will show you also how can you give access to the application that has the database to run the other examples that we have for you. So let's load environment variables. Now the environment variables are ready, so I can continue with the examples.

Then let's go with Managed Inference. I mentioned that for Managed Inference, we have the basic chat completions endpoint. This is an endpoint that you will find on services like OpenAI or Cloud Anthropic. Basically, you ask a question, you send a message or an array of messages, and you are going to get a response.

This endpoint also supports custom tool execution, but this tool execution is something that is happening on your code. So you will need to specify the function and capture the information to execute those. Or if you are using an agent framework like ADK or LangChain, it also has custom tool execution internally.

But for the example, we are going to perform a basic inference. To perform the basic inference, we are going to do a basic HTTP request. So we have the inference key, the model ID on the payload. So we are going to use Cloud 4, which is the model that I provision here on my app.

And the message that I'm going to send as a user is to explain the concept of Managed Inference in one sentence. You can adjust the parameters that the OpenAI API supports, like temperature, maximum tokens, top P, etc. You can take a look at the documentation by clicking on the path endpoint, and it will take you to the documentation of that endpoint and all of the parameters we support.

And then after performing the HTTP request, we get the response as JSON, and let's run this. And let's see how this works. So we get a response like any other AI endpoint works. So we have a chat completions object with the different choices, like the amount of messages that it returns.

Basically, we have just one message here as an assistant, and the response is, "Managed Inference is a cloud service that handles the deployment, scaling, and maintenance of machine learning models for real-time predictions," etc. So that's a basic example.

So this is something that you can do everywhere, getting inference. It became a commodity. And also to show that we also support the streaming, of course, we want to have a better experience while building our applications. We need to support streaming, meaning that it's not going to return the whole answer as just one JSON object.

It's going to start giving you chunks so you can get that real-time feedback when getting the response from the inference service. So here I'm just consuming chunks and printing out those chunks

on the screen. The first example is going just to show you the raw version of this request. So this is using a server-send event approach. It's sending you message events. Every message event has data, and each data is a piece of that completion object.

But in this case, you are getting like a delta, like a version of that message. So you see here in the content, we get nothing. Then streaming is crucial for this, and then it continues giving me the response.

We get a bunch of different responses, but let's take a look how that looks like on an application when you are rendering the chunks. This is just the pure objects to understand how the API works. Here I have another example where I am doing some parsing of that information, extracting the delta content of each response, and then at the end, rendering everything as a beautiful markdown.

When I execute this, the inference service is going to start responding in real time, chunk by chunk, all the messages that I'm getting from the service. As you can see, it is a markdown response. So at the end, when we finish, we use the markdown display object here in Jupyter to get the beautiful answer here.

Sometimes it will give you a code example depending on what the inference service is giving you. And that's how this works and this is how you can build an application to consume these real-time streams. But now let's talk about the good part, which is the agents.

Who has built an agent before, like an application that executes a tool or runs some code? Perfect. We have a couple hands there. Amazing. So the tools that we support today, and Anush here as Product Manager can tell us what we are going to be supporting in the future, are Dyno Run Command.

Dyno Run Command allows you to execute a command on a Dyno, like a Unix command or a script that you already have pre-deployed on your application. It will execute that and give you the response. So this is pretty good to run trusted code, code that you have written, that you know that it works, that it has a predictive

result, and you can just have that code running on your Dyno and execute it through this agent, this tool. We have query databases. We have two tools here. One is Postgres Get a Schema, and the other one is Postgres Run Query.

Postgres Get a Schema means the LLMs don't know the shape of your database. If you ask for data, it will hallucinate an SQL query. Or if you have some sort of retrieval augmented generation, or if you are grounding the prompt with the shape of your database, it will generate something that is close to the shape of the data of your database.

But with this tool, it will get exactly what you have on the database. And then we can pass that schema to the next tool, which is Run Query. It will generate a query that will run on your database, get the result, and make inference over those data.

Then we have a couple other tools for document transformation: HTML to Markdown, PDF to Markdown. You pass a URL, like a website or a URL of a PDF that is hosted somewhere, and it will give you the Markdown response so the inference can work over that text content.

It's going to perform that text extraction for you. And my favorites are the code execution one. We support for today Python, Node, Ruby, and Go. So the LLM will generate code that will run on Heroku on a one-off Dyno, and it will return that response back to the inference service.

And these also support dependency installation. So the input parameter has packages array parameter where the LLM automatically says, "OK, my Python script is going to use Pandas or NumPy. Let's install it before I attempt to execute the tool."

And all of these are just MCPs that we natively support. But you can also extend this agent endpoint by deploying your own MCPs. So now let's take a look. The agents endpoint is different from the chat completions. It works similarly with some minor differences.

For example, all of the responses are streamed. So it doesn't support the synchronous call where it waits to give you the full response because code execution, access to database, and this tool execution take time. So we prefer to have it like a stream.

And depending on the tool you are using, you might need to give access to the tool to your database or to an application, for example, to execute a command. So let's try that first one. Let's run a demo to run a command on an Heroku Dyno.

So here we are setting up the payload. Similar to chat completions, we have the model ID, we have the message. We are asking,

"What is the current date and time on the server?" LLMs don't know anything about the real time. They need to use tools to be able to have context about what's going onright now. I have asked the LLM about information, and it gives me information from 2024 because that's the date when they stopped training that model.

But with this, it knows exactly what time is because it's running this command on the server. So the tool is an Heroku tool. So I'm specifying the type to run this on Heroku. The tool that I'm executing is the Dyno Run Command.

And then the parameters for this tool are, "OK, the target app name," remember I told you, "use the name of the application you deploy." And then the command to run is date and a description of pretty much what's going on with the output of this command.

And I'm calling it. I'm just calling the agents Heroku endpoint. I'm passing that payload that has the tool. This is also a streamed endpoint, so I'm having the stream output here. And I'm just extracting the tool calls. And this is a standard also shape, like API shape that other APIs support for tool calling.

But this is specifically running the tools on Heroku. So let's execute and see the example. And I'm hoping you can execute this tool in your Jupyter Notebook. So it is running this on my AI Engineer Workshop application. It's going to execute the Dyno Run Command.

So now it is waiting until it runs the command on my app. We get the date and time and then the inference response. Cool. I mean, it's a basic example, but powerful. Instead of running date, imagine if you deploy your own script that does your data extraction or connects to external services, you can just call it from here, from this endpoint as well.

The other one, code execution. With code execution, I am asking the inference service to perform an operation, and I am passing that code execution tool. So now that it has that tool enabled, it will try to generate code in that language, run it on Heroku, and give you the response back.

For this example, I will invite you to change Node to Python, Ruby, or Go to see different responses. I'm a Node developer, even though I have a bunch of Python here. Some of you might recognize this is LLM-generated.

So sorry, not sorry. That's what tools are for. And I'm going to be executing that code exec node to perform this operation. What is the third Fibonacci number? This is a basic algorithm. I just wanted to do something easy.

I will invite you to change this to perform a different operation and see what you can do, like break this thing. And then for the execution, I am just parsing the response. So I get beautiful markdown, code highlighting, and everything.

So let's perform the operation.

And

it will generate the algorithm. Let's see. OK, I will execute code exec node. This is the code that I'm passing as the input. This is the beautiful markdown syntax highlight code that it will execute. That's JavaScript. Yes, that's JavaScript.

And at the end, it's going to execute that code on Heroku, and then I am getting the response back. The Fibonacci number is 233. But I want to do that in Go. So just change from Node to Go.

Execute this. And it will do the same, but now the code that we are going to see here is a Go code. And it will do the same. Get this Go code on an Heroku Dyno, compile it, run it, and get you the response back.

That looks like a Go code to me. I am not a Go developer, but that definitely looks like Go. And it executed the response, and it gave me the same thing and an explanation, like the inference operation over that tool execution.

But that's cool. So far, we have just called one tool. The good thing about agents is that we can chain calls together, mix and match. We have different agents acting together. So now let's complicate things a little bit more, and we are going to use two tools.

One is the HTML to markdown. So go to a website, do something, and then get that result and use it on this other agent. For that, we are going to use two tools: HTML to markdown and code execution Python.

So the prompt here is, "Use the Python snippet from the Wikipedia page for Euclidean algorithm to calculate the common divisor of 252 and 105." So it doesn't know the algorithm. I am telling directly, "Go fetch the exact one that is on Wikipedia, then run this code on Heroku and give me the response."

I'm enabling those two tools: HTML to markdown and code execution Python. And let's run this. And now we are going to see multiple tool calling. Getting the page in markdown. So now it recognizes that the Euclidean algorithm page is this one.

Now it is reading the whole content of that page in markdown, attempting to get the algorithm from there. Then it's going to generate the code. Then it's going to run the code. And at the end, I'm going to get the response.

This execution takes a little bit more time because it is performing multiple tool runs at the same time.

Any questions so far? Is anybody doing it? If not, you have access to the workshop. You can do it at home with more time, modify the code, analyze it, and make sure the concept is understood. So it got the algorithm here, and it executed the thing on Heroku.

Now I'm just waiting for the last inference step, and that execution is done. There you go. I found the Python implementation from Wikipedia. I calculated that number that you asked me. This is the Python snippet from Wikipedia. And then it implemented in this code.

And this is how I did it. These are the explanations. Beautiful. It works. Now, before I get into Postgres execution, we need to have access to the database. Andright now, the application that you have doesn't have access to that database.

If you want to give it a try, you are going to go to the Heroku dashboard. Where is the Heroku dashboard? Here.

To the AI Engineer Worlds Fair.

Click on AI Engineer Data. Go to Resources. And we have one thing here. The Postgres database tools only work on followers. Followers on Heroku are read-only. For security reasons, we don't want to give an LLM tool write access to your database because, of course, LLMs are not trusted for that.

If you want to give write access to your production database to an agent, I will invite you to deploy an MCP that does that, like the PostgreSQL MCP. Give the access under your own risk. The ones we maintain, we want to make sure we don't break production.

So we have two different databases here. The master one, or the main one, is database. Don't touch that one. The other one that has attachments is the follower. So you are going to expand this, manage attachments, and add your application, like Julian Jupyter.

And that's it. Now I gave my application access to the database. Second, you will need to get the name of that database. So go to your application.

In this case, it's Julian Jupyter.

Go to Resources. And this database here,

AI Engineer Data, in my application is called Database. So that's the name I'm going to use in code.

Since I am working on AI

Engineer Workshop, the name is different. So I'm going to show you so you can recognize the differences in names. This is just the name of the environment variable on Heroku that has the connection string. So here I have access to that database.

On this application, my database is called Heroku PostgreSQL Aqua. So that's the name you are going to get and change it in your code: Heroku PostgreSQL Aqua. So now I have access to that specific database on my app.

I cannot access databases from other applications. This is for security reasons. So this is why I have to do the attachment first to be able to give permissions to my database. If the database lives in your application, you don't need to do this because your application already has access to it.

That's the only thing that we need to do here to set up. Then we are going to enable two tools: Postgres get a schema and Postgres run query. These are going to run on my application. Remember the target app name.

And these are going to access that specific database. So I'm giving permission to go to those two places. So the database that we have here is a database for a solar energy company. It contains a table full of metrics with energy consumed and energy produced every hour in kilowatts per hour.

And we have metrics for maybe two months. So now, what type of applications can we build with this? Like asking questions like, how much energy has been saved in the last 30 days? It's going to understand the shape of the database.

It will see that it has a metrics table with these columns. Then it will go and generate an SQL query to be able to get this information and then run that query and give you the response. And let's execute that to see how it works.

So first step is, I don't understand your database. Let me get the schema. So to get the schema, it executes the tool. And then you are going to see the full schema being printed here on the screen in a moment.

So now we have the schema of my database. It sees the different tables, metrics with the fields, products, systems, users, etc. So I have the information that I need here. Beautiful. Now it will run the query. So it already generated the query that it is 100% sure will run on my database because it knows the shape.

This is the response. It got the data. And now that after it got the data, it's going to perform the inference. It's going to extract the answer from that data. And at the end, it's going to tell me the information.

It wanted to run more queries to get a breakdown of the energy by system. I have three different systems. One system that performs well. The other one that doesn't perform pretty good. The other one that it's totally horrible.

So it is also trying to get more information. Just from one question, it is doing inference on my data. And I just enabled two tools. I didn't need to do anything else. And there we go, the report. These are the key metrics.

This is how much you have saved. And this is the breakdown per system: the best performer, good performer, and the energy deficit. And this is getting access to your data and acting over your data. We can add a third tool.

Let's say code execution Python. I want to generate a graphic. So it will now do a Matplotlib graphic with that data. And you can keep adding more things. But also, the more agents and tools that you add, it will take more time to perform the whole operation.

But that's an example of how can you mix and match these tools that we have here and how can you give access to agents to your data. I have a couple of exercises. Just try to come up with an example to use a PDF and extract something from that PDF and run code.

And you can do this on your own time. We are running low on code. I want to show you now how can we deploy and run MCPs with this same endpoint. So those are the tools we maintain. But what about the tools you are building and the tools that already exist on the MCP ecosystem?

So with managed inference and agents, let's say you go to the dashboard, to the configuration page. And here, you get access to the model configuration. And here, it has the toolkit integration and the MCP server list. So basically, this MCP Brave that I have here, the Brave Search MCP, and you might have already have access to this.

So you can attach it to your own Jupyter notebook. Pretty much to attach an MCP, you click on Manage MCP Servers. You attach it as an application, similar to what we did with the database. It's just another Heroku application.

And there you go. We have an MCP server here that is exposing two tools: web search and local search. So now we can use these tools on the agent's endpoint. So let me go and take a look at that.

So these are the instructions step by step, how can you enable an MCP. Then it's just another tool. This time, it is not a Heroku tool. It is just an MCP. And this is the name of the tool that I will execute: MCP Brave.

It's the name of the namespace. You can have multiple MCPs here. So this is kind of like a namespace for your MCPs. And I am going to execute Brave web search. This requires an API key. And I already have the API key on my application.

If you want, you can go ahead and take it. I'm going to remove that API key in a moment. Don't perform like 2,000 queries. Otherwise, I'm going to get charged. But now I have access to it. And the security tool also enables those keys to your MCPs.

And I will execute it as another tool. And the prompt that I'm sending is, what is the most recent news about AI agents? Let's execute this tool.

And this MCP is running on Heroku, the same thing as code execution or the Postgres tools. It will spin off a Dyno, run the MCP as a standard input and output, and scale back to zero. So you are not paying for something that is constantly running.

And there we go. That's the search from the Brave web search. So this is the response from the tool. And now it's going to render that after the inference operation. So I'm running MCPs on Heroku. A quick example of how can you deploy an MCP.

I have an exercise here: deploy an MCP to Heroku. I have a Perplexity Ask MCP that I just forked. This is the official Perplexity MCP. I forked it to my repository to just do one thing. To make it Heroku compatible, you use the Procfile, which is the file on Heroku that defines how an application is executed.

And I added the entry point as an MCP. So this is a new process type we support. Everything that starts with MCP is going to be recognized as an MCP on Heroku. It will execute this standard input and output code.

It also requires an API key that I don't have. But you can deploy it. To deploy to Heroku, I added a button to make it super easy. Click to deploy. I deploy this MCP to my space. And then I attach it to my application.

This is my Perplexity MCP. Let's add it available to the space. I don't have one. I deploy. There are certain MCPs, if you don't have an API key, it fails when you run them. So this is why I'm specifying something.

It doesn't need to be valid. And this is now a Node.js application. It will just deploy this, build this, and keep it available on Heroku. And you will be able to attach it to your app. So this is an exercise you can do and write the code to run that MCP that you just deployed.

Now we got the response from the previous Brave search. And this is pretty much the most recent news about AI agents. But remember that I told you that you can also use those MCPs outside of Heroku, not only for Heroku agents.

Let's say you are using Cursor or Cloud Desktop, or you are writing your own agent in a different platform, but you want to have those MCPs available remotely. You can use those two. So here on the management dashboard, you see the toolkit integration page.

So all of the MCPs that I deploy to my app are going to be available through this endpoint. So this is a server-sent event endpoint. And it is authenticated behind a bearer token. And we are working on OAuth support.

We're getting remote auth as well.

So that you can run it securely. You can build remote MCP servers on Heroku that are accessible without a bearer tokens. We don't like bearer tokens. Perfect. So now that the MCP is there, let's add my Perplexity really quick.

Let's refresh.

I need it. Perfect. I have it. It's available. I go to my toolkit, copy the token. It is the same token that you use for inference. So I don't need to do this. You just need the URL. And in my Jupyter code, I have a basic MCP client.

I'm using. So I need to install the dependency here. I'm using the MCP package from

Anthropic.

And I'm creating an MCP client. That MCP client is going to my Heroku endpoint, passing that API key as an authorization header. And let's run it to list the tools that are available and then execute Brave web search.

So I'm executing the MCP that I deploy on Heroku outside of Heroku. Let's run this demo.

And it connected. We have the following tools: Brave local search, Brave web search, and Perplexity Ask, the one that we just deployed and enabled with just two clicks, mostly. And it executed the web search. I am not processing the result here.

This is just a very basic example. But that's how can you deploy an MCP and use it externally. And I think I have also a Cursor MCP gateway that I use for MCPs that I deploy. I just use the same approach to use my MCPs on Cursor.

And last but not least, I told you that at least the chat completions endpoint is compatible with OpenAI API. 95%. We are working to bring it to 99%. There are certain parameters that are not supported. But you can just use the SDK.

So let's use the SDK to perform a basic operation. I am just using the OpenAI SDK with the API key and URL from Heroku. I perform an inference. And we should get a response

in a moment.

And there you go. We have the response. And that's pretty much what we had for you today. You get access to this notebook. Keep playing with it. I extended access to the Heroku platform, that team, until the weekend.

Becauseright now, unfortunately, we don't have a free tier. We are working hard to bring it back. But you can go deploy, try it out. And if you have any questions, please connect with us on social. We have the dev center site for documentation, the Heroku AI website.

We created a Heroku community on Twitter or X. These slides are on the Slack. So you can get them from the Slack. And thank you very much. I hope you enjoyed this workshop.