Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod

Audry, um, I am from RunPod. This is an intro to RunPod. Can I just get a quick hands to see how many people have already heard of RunPod, or maybe even used RunPod before? Okay, newbies for everybody. Great.

Um, so RunPod, we are a cloud AI infrastructure company, so we have the hardware, we have the GPUs, and we make it easy for developers to deploy, um, models. And that can be your own private model, it can be an open-source model from Hugging Face, doesn't matter to us.

You bring your code and we'll bring the rest.

Um, just really quickly, what problems does RunPod solve? Like, why are we even here today? Um, infrastructure can be hard managing it. Um,

think about back in the day before we had AWS, um, Google Cloud, when everybody would have to have on-prem servers and manage those, maintain those. That is something that we don't want to have to do as developers. Those are things that we happily, um, have moved away from and given off to DevOps, and now it's even, it's even more abstracted for us.

GPU access is slow and opaque, so, um, I don't know if you, if anybody has tried to buy a GPU recently. We're in a global supply crunch. Um, it's a bit like in COVID when everybody went to the store and bought all the toilet paper because we didn't know how, how long they would need to be at home for.

We're a little bit in thatright now, um, but we expect the market will recover, um, as customers, companies, people figure out a little bit better, uh, get a little bit better at, um, estimating what kind of compute they need.

Um, and then last, build your primary focus should be building. So again, um, we want to build app. We as software developers, we bring, bring the value through the applications that we build, um, not from managing the infrastructure.

And I think RunPod has a pretty unique story. These are, these are our founders, Sen and Pradeep. Um, so they had a couple of GPU rigs in their basement in 2022, um, failed crypto mining, and then so they were like, what are we going to do with our GPUs now?

Um, so they prototyped what is now the foundations of RunPod. They posted on Reddit and said, hey, anyone want to use these GPUs for free? Just give us feedback on it. And that is literally how our company has started, and we have been, um, revenue generating ever since.

Um, and the reason why I want to tell this story is, um, not because it's, it's very like bootstrappy, but because, um, the origin, origin story of RunPod has always started with, uh, builders and getting feedback from the community, and that is still true today.

So I won't promise that we'll be perfect, but, um, we are definitely very engaged with, um, our users on, on Reddit, um, on Reddit, on Discord. So, um, we're always trying to stay engaged with y'all. Um, just at a glance, to give you an idea of RunPod, we have over 500,000 developers on our platforms, 30 plus data centers across the world, including, um, Europe and the EU.

Um, and we've just passed a significant revenue milestone for us, 120 million in annual recurring revenue.

Uh, these are just a few of our customers. Um, you might be surprised to see some of the AI cloud-native companies on here too, but, um, they come to us for the same reasons that most of our customers come to us.

It's, um, because they need flexible and reliable GPU infrastructure.

This is a really high over, high-level overview of, um, different ways you can build on RunPod. So I would say our, at our core, um, pods, it's our sandbox virtual environment. We spin up a container for you, um, allocate GPUs to it, and we manage the rest.

You just bring your, um, Docker files, you bring your code. Serverless, um, it's our autoscaling product. So when you're thinking more about like bursty workloads or batch workloads, um, serverless is really great because, um, instead of being always on like a container is, serverless, um, your workers spin down, and when they're idle, you don't pay for anything.

Clusters, um, if you're doing some heavy-duty training, there's a place for you as well on RunPod, um, multi-node clusters with high-speed networking. And then the hub, which I'll, I'll, I'll switch to in a second, um, it's kind of like our central repository for AI repos.

Um, these are already pre-configured, pre-vetted. Um, we have a couple of examples of listings by RunPod for popular models, but also our community, um, contributes to them as well. So they're just repos that you can fork, you can watch, and then, um, you can star and deploy on RunPod.

Um, so today we're going to be talking mostly about serverless. Um, so serverless is best for real-time inference. I talked about the autoscaling that comes with it. Um, why teams use it is mostly because they don't need to, um, preempt and figure out how much compute they need ahead of time.

Um, you can set, you can configure the number of max workers that you want to scale up to. You can set limits for caps, for spending caps, and you can also configure workers that are always on. So they're, um, already have your models downloaded, and they can respond to requests, um, immediately.

For a lot of teams, serverless is the fastest way, um, if you want to start deploying a production-ready API.

And now I'm going to switch over and just show y'all really quick how easy it is to get started and deploy something.

Okay.

Where are we?

Okay. Soright now I'm, um, going to do everything via the console so that it's nice and pretty for you guys to see, but we also have, um, CLI support. We have skills, um, to help work with RunPod, everything that's ready for your agent so you don't have to read our documents.

But since we're all humans here today, I'm going to show you via the console.

Um, we'll start in the hub, which is if you're just trying to explore and see what's out there, what is something that you can get up and runningright now, the hub is a great place to start. So like I mentioned, these are already vetted open-source listings for, um, AI repos, and I am going to pick the LLM, um, and I'll just open the underlying repository as well so you guys can see what it is literally just a GitHub, um, repo.

It tells you how to get set up for it. Um, we can see there's already the Docker file here. It's already pre-configured for you. It's got some defaults for you. Um, depending on the listing, you can, um, pass in different environmental variables, um, to configure it how you wish.

But I'm just going to go ahead and click deploy. And

I have a model that I wanted.

Let me see.

I was going to just pick Gwen. Works well. This is going to download it from Hugging Face and just expand the advanced options and look for the max model length. And I'm going to bump this up for the context window and leave everything else as the defaults, but they're settings for MaxLauras.

Um, all of these configuration options get passed as, um, as flags to the, uh, VLLM server.

And

I'm going to spin it up as an endpoint here.

This might take a minute, since, or two, since this is the very I just created it. We've got to initialize my workers. Let's check out. So the default configuration here is it's going to deploy on some H100s, and A100s are the backup here.

I have my pricing. This is a fraction of a cent per second. Um, like I mentioned before, that this is only going to be charged for while the worker is actually running and handling a request. Max workers is where I can bump this up if I want to have my workloads scale up, up to 15 workers at a time.

And I can set, um, some active workers ones that I want always to be on that I don't want, um, the container to ever spin down.

And I can save that.

Okay. So how do I, how does one interact with the serverless endpoint? Um, this is just an API, uh, HTTP endpointright here. Um, we provision this endpoint for you. You can send requests to this. Your customers can send requests to this.

If I just hit run and I'm going to add a few. Let's, what, what should we ask the LLM today? Is there anyone?

Okay. How did, I'm, I'm American, so how did Bitben get its name? I don't know.

Um, okay. While these requests are queued, let me check on our workers. Okay. We have a handful that are initializing. Um, this is the containers being created. That's the model being downloaded. Um, getting ready. And the ones that are running, they've already finished.

These are probably going to be the ones who are going to pick up those requests that we just added. I've got telemetry about, um,

it's blankright now, but the number of requests, execution time, delay time. So you have observability into how your endpoints are operating. And let's see. Okay. It's already done.

Got a request back in. It sat in the queue for about 41 seconds. Um, that's going to be a little bit longer than all of the subsequent requests because of some of the cold start time that I talked about, like downloading the model, um, initializing the first container, but, um, execution time only about one and a half seconds.

So yeah, that was probably less than five minutes to get started and get something deployed, um, on serverless from a hub listing.

Does anyone have any questions? This is, this is a very short and sweet intro. Um, we have another session later today at 4 o'clock. Um, and that one is going to be focused on our Python Flash, um, SDK.

And that one is going to be completely via the terminal. Um, and I'm going to walk you through how I can spin up, uh, and deploy my code on my code as a remote, remote function onto a GPU, um, and, uh, deploy in the end and make it like a production-ready endpoint here as well.

Okay. But that's all I got for today. So yeah, thanks, thanks for coming.