From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

I am here with, uh, with Pankaj Gupta. He's the co-founder of Baseten. Actually, so, today I was— I was checking Slack, and in the random Slack channel, one of the people in the company was saying, like, "Hey, you know, I heard someone say— call someone 'cracked.'

What does— what does 'cracked' mean?" And, you know, those— those of you who are, you know, Gen Z like me, or know someone like that, it's like, laughingright now because 'cracked' just means an exceptional engineer. And so, Pankaj is the most cracked software engineer I've ever had the pleasure of working with.

He's from San Francisco. His favorite model is LLaMA 3.8b. We're going to be working with a smaller version of that today. I'm Philip. I do development relations here at Baseten. I've been here for about 2 and a half years.

And I'm based in Chicago, but I'm very happy to be here in San Francisco with you all today. And my favorite model is Playground 2. It's a— it's a text-to-image model that's kind of like SDXL, but it's trained on Midjourney images.

You're going to see a ton of Playground 2 images in the slideshow today. So, what are we doing here today? What is our agenda? So, we're going to cover what is TensorRT-LLM and why use it. Model selection and TensorRT-LLM support, because it supports a lot of stuff, but not everything.

We're going to talk about building a TensorRT engine, configuring a TensorRT engine automatically, benchmarking it so you can know if you actually did something worthwhile, and then deploying it to production. And, you know, as much as I love the sound of my own voice and I want to just stand here and grasp this microphone for 2 hours and say things, this is not just going to be Philip reads off a slideshow.

We're going to do tons of coding, debugging, live Q&A. So, the way this presentation is kind of broken up is we've got some sections, we've got some live coding. It's going to be, you know, a very interactive workshop.

I'm going to be taking questions all the time. So, please don't hesitate to let us know if anything's confusing. We really want everyone to come away from this with a strong working understanding of how you can actually use this technology in production.

So, let's get started.

Let me—

Yes?

Let me interject for a second and ask, raise of hands, how many of you know about TensorRT? Great. How about—

Oh, this is so exciting. I'm so glad that we get to teach you all this today.

And how about TensorRT-LLM?

Okay, few. So, we'll cover the basics. I think I'm pretty sure that you'll get a sense of what it is. If you know PyTorch, this shouldn't be too hard.

And if you don't know PyTorch, like me, it's still not that hard.

So, we're going to start with the story of TensorRT-LLM. What, who, why. You know, once upon a time, there was a company called NVIDIA. And they— they noticed that there are these things called large language models that people love running.

But what do you want when you run a large language model? You want a lot of tokens per second, you want a really short time to first token, and you want high throughput. You know, GPUs are expensive, so you want to get the maximum value out of your GPU.

And TensorRT and TensorRT-LLM are technologies that are going to help you do that. So, if we get into it here, what is TensorRT? Here's one of my Playground 2 images. Very proud of these. If the words on the slides are dumb, just look at the images, because I worked hard on those.

Anyway, so, TensorRT is an SDK for high-performance, deep learning inference on NVIDIA GPUs. Basically, what that means is it's just a great set of tools for building high-performance models. It's a, you know, toolkit that supports both C++ and Python.

Our interface today is going to be entirely Python. So, if, like me, you skip the class that teaches C++, don't worry, you're covered. I know Pankaj reads C++ textbooks for fun, but I do not. So, we're going to do it in Python today.

And so, how does this work? You know, do you want to— do you want to kind of jump in here and—

Yeah.

And talk about this a little bit? Because, you know, it's a— it's a really cool process, how you go from a neural network to an engine.

Yeah. Yeah, exactly. So, ultimately, what are machine learning models? They're the graphs. They're computational graphs. You flow data through them, you transform them. And ultimately, whatever executes a model does that, they execute a graph. Your neural network is a graph.

TensorRT works on a graph representation. You take your model and you express that using an API, that graph in TensorRT. And then TensorRT is able to take that graph, discover patterns, optimize it, and then be able to execute it.

That's what TensorRT is, ultimately. When you write a PyTorch model, you're ultimately creating a graph. It's graph followers,right? There is data flowing through this graph. And that's what it is. TensorRT additionally provides a plugin mechanism. So, it says that, you know what, I know this graph, I can do a lot of stuff, but I can't do very fancy things like FlashAttention.

It's just too complex. I can't infer automatically from this graph that this is even possible. Like, I'm not a research scientist. So, it gives a plugin mechanism using which you can inspect the graph and say that, okay, I recognize this thing and I can do it better than you, TensorRT.

So, I'm going to do it through this plugin. And that is what TensorRT-LLM does. It has a bunch of plugins for optimizing this graph execution for large language models. So, for example, for attention, for FlashAttention, it has its own plugin where it says that, okay, now we are in TensorRT-LLM land, take this graph and let me execute it using my optimized CUDA kernels.

And that's what ultimately TensorRT-LLM is, a very, very optimized way of executing these graphs using GPU resources, not only to get more efficiency, better cost for your money, but also better latency, better time to first token, all the things that we care about when we are running these models.

In addition to that, it provides a few more things. Like, when you're executing a model, you're not just executing a request at a time, you're executing a bunch of requests at a time. And in-flight batching is a key optimization that is very, very key.

Like, in this day and age, if you're executing a large language model, you have to have in-flight batching. There's just no way. Like, it's like a 10x or 20x improvement, like, and you have to have that. And TensorRT-LLM provides that.

TensorRT wouldn't. TensorRT is a graph executor. It doesn't know about that. But TensorRT-LLM has an engine that does that. It also has a language to express graphs, just like PyTorch, and it requires that there is a conversion. But it makes it pretty easy to do that conversion, and there are tons of examples in the repo.

Exactly. So, TensorRT is this great sort of engine builder, and then TensorRT-LLM is a mechanism on top of that that's going to give us a ton of plugins and a ton of optimization specifically for large language models. So, TensorRT-LLM, like Pankaj said, defines the set of plugins for your LLMs if you want to, you know, compute attention, do LoRAs, Medusa, other fine-tunes.

And it lets you define optimization profiles. So, when you're running a large language model, you generally have a batch of requests that you're running at the same time. You also have an input sequence and an output sequence. And this input sequence could be really long.

You know, maybe you're summarizing a book. It could be really short. Maybe you're just doing some LLM chat, like, "Hi, how are you? I'm Fred from the bank." Depending on what your input sequence and output sequence lengths are, you're going to want to build a different engine that is going to be optimized for that, to process that number of tokens.

So, yeah, so TensorRT-LLM is this toolbox for taking TensorRT and building large language model engines in TensorRT.

Yeah. I want to say just one thing at this point. Like, why I care about input and output sizes. Like, how does TensorRT-LLM optimize for that? It actually has specific kernels for different sizes of inputs, different sizes of matrices.

It's optimized for that level. And it sometimes becomes a pain when I'm compiling TensorRT-LLM. It takes hours because it optimizes for so many sizes. But it also means that giving it that size guidance is useful. It can use better kernels to do things faster.

And that's why. A lot of the models you'll run, you don't have to care about it. But there is always a trade-off. Here, it does care about that, and you can benefit using that trade-off.

Yeah. And TensorRT-LLM is a great tool for a number of reasons. It's, you know, it's got those built-in optimized kernels for different sequence lengths. And that level of detail is really across the entire tool. And what that means is that with TensorRT-LLM, you can get some of the highest performance possible on GPUs for a wide range of models.

And it's really a production-ready system. We are using TensorRT-LLM today for tons of different client projects, and it's, you know, running in production and powering things. TensorRT-LLM has support for a ton of different GPUs. Basically, anything like Volta or Nua.

Volta support's kind of experimental. But yeah, like your A10s, your A100s, H100s, all that stuff's supported. And yeah, TensorRT-LLM, it's developed by NVIDIA. So, you know, they know their graphics cards better than anyone. So, we just kind of use it to run models quickly on that.

That said, everything does come with a trade-off. Is anyone from NVIDIA here in the room? It's okay, you don't have to wait. Okay. So, I'm going to be nice. No, we really are big fans of this technology, but it does come with trade-offs.

You know, some of the underlying stuff is not fully open source. So, sometimes if you're diving super deep, you need to go get more information without just, like, looking at the source code. And it does sometimes have a pretty steep learning curve when you're building these optimizations.

So, that's what we're here to help flatten out for you guys today. Hopefully, we're still friends. What makes it hard? So, there's a couple of things that make building with TensorRT-LLM really hard. And when we enumerate the things that make it hard, that's how we know what we need to do to make it easy.

So, the number one thing in my mind that makes it hard to build a general model, to optimize a model with TensorRT-LLM, is you need a ton of specific information about the production environment you're going to run it.

Allright. So, I do a lot of sales enablement trainings, and I love a good metaphor. So, I'm going to walk you guys through a metaphor here. Apologies if metaphors aren't your thing. So, imagine you go into a clothing store, and it only sells one size of shirt.

You know, it's just, like, a medium. You know, for some people, that's going to fit great. For some people, it's going to be too small. For some people, it's going to be too big. And on the other hand, you can go to, like, a tailor.

I don't know, in, like, Italy or something. And you go there, and they've got, you know, some super fancy guy with a, you know, cool mustache and stuff. And he, you know, he measures you, like, every single detail, and then builds a suit exactly for you that's perfect for your body measurements.

Like a made-to-measure suit. So, optimizing a model is kind of like making that suit. You know, everything has to be measured for exactly the use case that you're building for. And so, when people come in and expect that they can just walk in and grab off the shelf a model that's going to work perfectly for their use case, that's like expecting you're going to go into a store and buy a piece of clothing that fits you just as well as that custom-made, made-to-measure suit from the tailor.

So, in, you know, to relate that more concretely to TensorRT-LLM, you need information. You need, like we talked about, you need to understand the sequence lengths that you're going to be working at, the batch sizes that you want to run at.

You also need to know ahead of time what GPUs you're going to be using in production. These engines that we're building are not portable. They are built for a specific GPU. You build them. So, if you build it on an A10, you run it on an A10.

If you build it on an H100, you run it on an H100. You want to switch to H100 mig? Okay, you build it again for H100 mig. So, you need to know all of this information about your production environment.

And then also, as we'll talk about kind of toward the end, there are some infrastructure challenges as well. These engines that we're going to build are quite large. So, if you're, for example, doing autoscaling, you have to deal with slow cold starts or, you know, work around the size of the engines.

Otherwise, your cold starts are going to be slow. And overall, also, just model optimization means we're living on the cutting edge of new research. You know, when I'm writing blog posts about this stuff, I'm oftentimes looking at papers that have been published in the last six months.

So, you know, just combining all these new approaches and tools, there can be some rough edges, but the performance gains are worth it.

Yeah.

So, yeah. Oh, please, go ahead.

I want to add one thing is that there are modes in TensorRT-LLM where you can build for a certain, on a certain GPU, and it will run on other GPUs. But then it's not optimized for other GPUs. So, why would you do that?

We never do that. We always build it for the GPU. But there is that option.

Exactly. That would be like if I went to that fancy tailor shop, got a made-to-measure suit, and then was like, "Hey, Pankaj, happy birthday. I got you a new suit." That's what it would be like. So, you know, what makes TensorRT-LLM worth it?

Well, it's the performance. So, these numbers are from a Mistral 7B that we ran on artificial analysis, which is a third-party benchmarking site. And we were able to get, with TensorRT-LLM and a few other optimizations as well on top of it, 216 tokens per second, perceived tokens per second, and 180 milliseconds time to first token.

So, unless any of you are maybe like some super high-quality athletes, like a UFC fighter or something, your reaction time is probably about 200 milliseconds. So, you know, 180 milliseconds time to first token, counting network latency, by the way, counting the round-trip time to the server is great because that to a user feels instant when it's under 200 milliseconds.

And actually, most of it is network latency. The time on the GPU is less than 50 milliseconds.

Less than 50 milliseconds. So, we've got another one of these green slides here. I like to talk really fast. So, these slides I put in this presentation to give us all a chance to take a breath and ask any questions.

So, you know, we're going to cover a lot more technical detail moving forward. But if there's anything kind of foundational that you're struggling with, like what's TensorRT, what's TensorRT-LLM, anything I can explain more clearly, I would love to hear about it.

Going once. Going twice. It's okay. We're all friends here. You can raise your hand. Allright. Well, it sounds like I'm amazing at my job. I explained everything perfectly, and we get to move on to the next section.

So, what models can you use with TensorRT-LLM? Lots of them. There's a list of, like, 50 foundation models in the TensorRT-LLM documentation that you can use. And you can also use, you know, fine-tunes of those models, anything you've built on top of them.

It supports open-source large vision models. So, if you're, you know, building your own GPT-4o, you can do that with TensorRT-LLM. And it also supports models like Whisper. And then TensorRT itself, you can do anything with TensorRT. So, any model, custom, open-source, fine-tuned, you can run it with TensorRT.

But TensorRT-LLM is what we're focusing on today because it's a much more convenient way of building these models. And, you know, on this list of models that it supports, there's one that maybe stands out. Does anyone know, like, what model kind of doesn't belong in this list of supported models?

Like, what up here isn't an LLM? Whisper. Exactly. Why is Whisper on here? Well, TensorRT-LLM, it's called -LLM, but it really is a little more flexible than that because you can run, you know, a lot of different auto-aggressive transformers models with it, like Whisper.

So, if anyone doesn't know what Whisper is, it is an audio transcription model. You give it, you know, an MP3 file with someone talking. It gives you back a transcript of what they said. It's one of our favorite models to work with.

We've spent a ton of time optimizing Whisper, building pipelines for it, and all that sort of stuff. And what's really cool about Whisper is structurally, like, it's basically an LLM. You know, that's a massively reductive statement for me to make, but it's an auto-aggressive transformers model.

It has the same bottlenecks in terms of inference performance. So, even though this is not an LLM, it's an audio transcription model, we're actually still able to optimize it with TensorRT-LLM because of its architecture.

Let me say one more thing.

Of course.

So, the whole, I think the recent ML revolution started with the Transformers paper, "Attention is All You Need," and that describes an encoder-decoder architecture. And in a way, Whisper is machine translation. That paper was about machine translation. You're translating audio into text,right?

And it's basically that. It's an encoder-decoder model, exactly like the Transformer architecture. And TensorRT-LLM is about that. It's about that Transformer architecture. So, it actually matches pretty well.

Exactly. So, moving on, I want to run through a few things, just some things in terms of what TensorRT-LLM supports. So, I assume it's going to support Blackwell when that comes out, like 99.999% certain. But anyway, in terms of what we have today, we've got the Hopper with the H100s, the L4s, RTX 4090s.

If anyone has a super sweet gaming desktop at home, number one, I'm jealous. Number two, you can run TensorRT-LLM on that. Ampere GPUs, Turing GPUs, V100s are, you know, somewhat supported. And what's cool about TensorRT and hardware support is that, like, it works better with newer GPUs.

When you move from an A100 to an H100 and you're using TensorRT or TensorRT-LLM, you're not just getting the sort of, like, linear increase in performance that you'd expect from, you know, oh, I've got more flops now. I've got more gigabytes per second of GPU bandwidth.

You're actually getting more of a performance gain going from one GPU to the next than you would expect off raw stats alone. And that's because, you know, H100s, for example, have all these great architectural features. And TensorRT, because it actually optimizes the model by compiling it to CUDA instructions, is able to take advantage of those architectural features, not just kind of run the model, you know, raw.

And so, for that, you know, that's why we do a lot with H100 migs. This bullet point here is a whole different 45-minute talk that I tried to pitch to do here. But basically, you know, H100 migs are especially good for TensorRT-LLM if you're trying to run smaller models, like a 7B, you know, LLaMA 8B, for example, because you don't need the massive amount of VRAM, but you get the increased performance from the architectural features.

And, you know, just my own speculation down here that I'm sure whatever the next generation is, is going to have even more architectural features for TensorRT to take advantage of. And so, you know, adopting it now is a good move, you know, looking to the future.

Here, we've got a graph showing, you know, with SDXL. Now, this is TensorRT, not TensorRT-LLM, but the underlying technology is the same. You know, when you're working on an A10G, we were looking at, you know, maybe like a 25 to 30 percent increase in throughput for SDXL.

And with an H100, it's a 70 percent. And that's not, you know, just because the H100 is bigger. It's 70 percent more on an H100 with TensorRT versus an H100 without. So, yeah, great.

Yeah, please go ahead. One thing I want to add here is that H100 supports FP8 and A100 does not. FP8 is a game changer. I think it's very easy to understate that fact. FP8 is really, really good. Post-training quantization, you don't need to train anything.

Post-training quantization, it takes like five minutes, and the results are so close. We've done perplexity tests on it. Whenever you quantize, you have to check the accuracy. And we've done that. It's hard to tell. And FP8 is about 40 percent better in most scenarios.

So, if you're using mig H100, if you can, then it can be way better if you use FP8.

And FP8 is also supported by Lovelace. So, that's going to be your L4 GPUs, which are also a great option for FP8. So, yeah, a bunch of different precisions are supported. Again, FP8 is kind of the highlight. FP4 could be coming.

And, you know, traditionally, though, we're going to run in FP16, which is sort of like a full precision. Oh, sorry, half precision. FP32 is technically full precision.

But nobody does FP32.

Yeah, yeah. So, for inference generally, you start at FP16. By the way, FP16 means a 16-bit floating point number. And from there, you know, you can quantize to int8, FP8, if, you know, you want your model to run faster, if you want to run on fewer or smaller GPUs.

We'll cover quantization in a bit more detail later on in the actual workshop. I don't want to spend too long on the slides here. I know you guys want to get your laptops out and start coding. So, the other thing just to talk about is, like I said, TensorRT-LLM, it's a, you know, it's a set of optimizations that you can, you know, build into your TensorRT engines.

So, some of the features that are supported, again, each one of these could be its own talk here, but we've got quantization, LoRA swapping, speculative decoding, and Meduza heads, which is where you basically, like, fine-tune additional heads onto your model.

And then at each forward pass, you're generating, like, four tokens instead of one token. Great for when you're, you know, when you have memory bandwidth restrictions. Yeah, inflight batching, like you mentioned, page attention. There's just a ton of different optimizations supported by TensorRT-LLM for you to dive into once you have the basic engine built.

So, we're about to switch into more of, like, a live coding workshop segment. So, if there's any of this sort of groundwork information that didn't make sense, or any more details that you want on anything, let us know.

We'll cover it now. Otherwise, it's about to be laptop time. Looks like everyone wants laptop time. So, yes, please go ahead.

Can you do, like, some high-level prefers on this, like, to the VLLM?

Yeah. Do you want to handle that one, like a high-level comparison to VLLM?

I can do a very high-level comparison. First of all, I respect both tools. VLLM is great. TensorRT-LLM is also great. We found in our comparisons that for most of the scenarios we compared, we found TensorRT-LLM to be better.

There are a few things there. One thing is that whenever a new GPU lands or a new technique lands, that tends to work better on TensorRT-LLM. VLLM, it takes a bit of time for it to catch up, for the kernels to be optimized.

TensorRT-LLM is generally ahead of that. For example, when H100 landed, TensorRT-LLM was very, very fast out of the box because they've been working on it for a long time. Second thing is that TensorRT-LLM is optimized from bottom to the top.

These CUDA kernels at the very bottom are very, very well optimized. On top of that, there is the inflight batching engine, all written in C++. And I've seen that code. It's very, very optimized C++ code with, you know, STD moves and whatnot.

And on top of that is Triton, which is a web server, again, written in C++. So, the whole thing is very, very optimized. Whereas in some of the frameworks, they also try to optimize in the sense like, you know, Java versus C++, where Java is like, you know, we optimize everything that matters.

But there are always cases where it might not be as good. But TensorRT-LLM is that, let's optimize every single thing. So, it generally tends to perform better in our experience. That said, VLLM is a great, great product. We use it a lot as well.

For example, LoRA swapping, it became available in VLLM first. So, we used that there for a while. What we found is that when something lands in TensorRT-LLM, and it's usually after a delay, it works like bonkers. It just works like so well that performance is just amazing.

So, when something is working very stably in TensorRT-LLM, we tend to use that. But VLLM and other frameworks, they provide a lot of flexibility, which is great. We love all the products.

And, like, does that small performance gains kind of, like, how is it, like, the speaker learning curve? Because it's like much easier to use.

Yeah, yeah. I think we should definitely question that. That is a clear trade-off. If you're working with two GPUs, for example, two A10Gs, it's not worth it probably. But if you're spending hundreds of thousands of dollars a year, it's your call.

But if you're spending

many hundreds of thousands of dollars a year, it can be material. Like, your profit margin might be 20 percent. This 20 percent or 50 percent improvement might make all the difference that you need. So, it depends upon the use case.

So, do you have, like, any intuition, like, how to come up with a unique, like, data tool in support of, like, TensorRT? Because you said it's important, like, just, like, on a few GPUs. So, VLLM is probably easier, faster, and the trade-offs.

It could be, yeah. I think if you're working with one or two GPUs, A10Gs or T4s, I mean, I'm not an expert in that, but it probably doesn't matter which framework you use, whichever works best for you. But if you end up using A100s or H100s, you should definitely look into TensorRT-LLM.

And regarding the learning curve, stick around a little bit. We're going to flatten it out a lot for you because we've built some great tooling on top of TensorRT-LLM that's going to make it just as easy to use.

Little marketing spinright there. So, yeah. So, we're going to be doing an engine-building live coding exercise. And Pankaj is going to lead us through that. I'm just going to kind of roam around. So, if people have, you know, questions, need help, kind of on a one-on-one basis, I'll be able to help out with that during this portion of the workshop.

Great. So,

yeah.

Let's just, like, go through a couple. There's a little bit of setup material,right?

Yes.

Yeah.

You want to run through that?

Okay, sure. I'll run through the setup material. And then,

yeah. So, anyway, what we're going to do, to be clear, is we're going to build an engine for a model called Tiny LLaMA 1.1b. I want to be really clear about this. TensorRT-LLM is a production-ready technology that works great with big models on big GPUs.

That takes time to run. The dev loop can be a little bit slow. And we only have a two-hour workshop here. And, you know, I don't want us to all just be sitting there watching a model build. It's basically as fun as watching paint dry or watching glass glow.

So, we're going to be using this super tiny 1.1 billion parameter model. We're going to be using 4090s and A10Gs just to kind of keep the dev loop fast. But this stuff does scale. So, at this point, we're going to walk you through the manual process of doing it all from scratch.

You're going to procure and configure a GPU. You're going to install dependencies for TensorRT-LLM, configure the engine, run the engine build job, and test the results. And we should be able to get through this in about half an hour or maybe a little less because these models are quite small.

And there's a few important settings that we're going to look at when building the engine. We're going to look at the quantization, again, the post-training quantization, like we talked about. We're going to be on A10s. Oh, sorry, no.

First, we're going to be on 4090s. So, we will actually have access to FP8 so that you can test that out. We're going to look at sequence shapes and batch sizes, how to set that. And we're going to look at TensorParallelism.

You want to give them a quick preview on TensorParallelism?

Oh, yeah. TensorParallelism is very important in certain scenarios. I wish it were more useful, but it is critical in many scenarios. So, what is TensorParallelism? Ultimately, machine learning running these GPUs is about matrix multiplications. We take this model architecture, whatever it is, it ultimately boils down to matrices that we multiply.

And a lot of the wrangling is around that. How do we shove these all batches into matrices? So, ultimately, it is matrix multiplication,right? What you can do is you can split these matrices and you can multiply them separately on different GPUs and then combine the results.

And that's what TensorParallelism is. It's one of the TensorParallelism techniques. There are many techniques. It's one of the most commonly used ones because you need that. Why do you need TensorParallelism versus other parallelisms like PipelineParallelism? Is that it saves on latency.

You can do things in parallel. You can use two GPUs at the same time for doing something, even though there is some overhead of crosstalk between them. With PipelineParallelism, you take the model architecture and you can divide these layers into separate things.

So, your thing goes through one GPU, like half the layers, and then half the layers on the second GPU. But you're not saving on latency. It still has to go through each layer and it's going sequentially. And that's why PipelineParallelism is not very popular for inference.

It is still popular for training. There are scenarios. And there's a lot of theory about that. But for inference, I don't think I've ever seen it used. And nobody pays much attention to optimizing it because of this thing, that TensorParallelism is just better.

There's also expert-level parallelism. If your model has a mixture of experts, then you can parallelize those experts. And that tends to be very advanced. And LLaMA doesn't have a mixture of experts. So, it's an esoteric thing that we haven't covered here.

TensorParallelism is pretty helpful and useful. One downside is that your throughput is not as great. If you can fit something in a bigger GPU, that's generally better. But there are bigger models like LLaMA 70B. They just can't fit on one GPU.

So, you have to use TensorParallelism.

Awesome. So, for everyone to get started, we made a GitHub repository for you all to work off of in this workshop. So, you can scan the QR code. It'll take youright there. Otherwise, you know, this is not too long to type out.

So, I'm just going to leave this up on screen for 30 seconds. Everyone can pull it up. You're going to want to, you know, fork and clone this repository to your local development environment. We're just, you know, we're just using Python, Python 3.10, Python 3.11, 3.9.

So, yeah, just, like, however your normal way of writing code is, this should be compatible. There isn't a lot of what?

Required to.

No. So, yeah, to be clear, in this repository, you're going to find instructions and we're going to walk through all this. We're going to be using entirely remote GPUs. So, you know, I personally have an H100 under my podiumright here that I'm going to be using.

No, I'm just kidding. I don't. But, yeah, yeah. So, we're just, we just all have laptops here. So, we're going to be using cloud GPUs.

Yeah, actually, if you want to follow along, you might need a RunPod account.

Yeah, yeah. We'll talk them through the setup steps there. Does anyone want me to leave this information on the screen any longer? Going once, going twice. Okay. If you, for whatever reason, lose the repository, just let me know.

I'll get it back for you. Yes. Okay. So, this slide means we are transitioning to live coding. So, yes, let's go over to the, yeah, the live coding experience.

So, I'm basically going to follow this repository. All the instructions are here. And I'm going to follow exactly what is here. So, you can see how to follow along.

And if you ever get lost or need help, just raise your hand and I'll come over and catch you up, like, one-on-one.

Yeah, yeah. I'm going to go really slow. I'm going to actually do all these steps here. I know it takes time, but, you know, there's a lot of information here. It's easy to lose track of things and get lost.

So, if you feel lost, like, ask and we'll break. I want to make sure. This is not a long process, a 10-minute process. We can take it slow for everybody here. So, first thing is that we'll do it, like, really, really from scratch.

So, we're going to spin up a new container on RunPod with a GPU to run our setup in. So, if you.

Let's give everyone a minute to, like, make an account,right?

Okay, okay. Yeah, yeah. Please, please. If you want to follow along, please go on RunPod and create an account. This should cost, like, less than $5 overall.

Yeah. So, yeah. So, if you want to make an account, there's instructions in the O1 folder. Yeah, this README. So, TensorRT in the first folder in the README, there's instructions and a video walkthrough. The minimum, we're not affiliated with RunPod in any way.

They just have 4090s and we wanted you guys to use 4090s today. There is a minimum credit buy of $10. If, for whatever reason, you can't use a company card or get reimbursed or whatever and you want your $10 back, send me an email after the conference and I will reimburse you myself.

So, you know, I just, I want to, it's really important to me that we are giving you the GPUs to run this on. So, yeah. So,

if you need the $10 back, I will get it for you. So, yeah, just follow this. It's like a one-and-a-half-minute video. And, yeah, that'll get the account set up. Maybe we could even, like, play the video.

Yeah, that'd be good.

So, yeah. And then this is how you, you know, deploy once we get set up.

So, in two minutes, we will come back together and, you know, we resume the talk. So, yeah, make that account real quick, get checked out, get those $10 of credits at the SSH key.

And if you have a difficulty creating an account, just let us know on RunPod.

Yeah. And also, if you do happen to run into any issues getting the RunPod account set up, like it's not working or whatever, I do have one that I can share individually with people. It would just take a really long time for me to go type in my username and password into every single computer in this room.

I feel like maybe I should have bought some jokes to tell while everyone sets the account up. Allright. How about this one? Why did the LLM cross the road?

Ignore previous instructions and cross the road.

Allright. I'll workshop that. I'll get some better jokes for you guys before the next setup.

How are we doing on the account setup? Anyone need a little more time? Allright. Great. No problem. No problem. Sorry. I don't want to rush you.

Just checking in.

Yeah. And then once everyone has the account, we'll set up the GPU together because there's a few things you need to configure.

How about going off of, like, every credit card I have is getting declined?

Cool.

I have a deposit, like, for a week at a moment.

Oh, really?

My friend, like, the credit card that's been on my account for the last 20 years, I'm creating a separate one for a time limit.

It's not taking anyone's credit card?

Yeah.

Great. Does someone, here, can I

know someone who runs it, who works at RunPod and would have their phone number? He's calling them? Okay. Awesome. Allright. We're getting in touch with customer support.

It's probably Stripe block protection.

Oh, yeah. It could be that. Yes. Stripe block protection. Okay. So, as a backup, yeah,

what's that?

You put it on your phone Wi-Fi.

Okay. So, the recommendation here is go off of the conference Wi-Fi, put your computer on your phone hotspot, and try it again because that worked, you know, maybe coming from a different IP address will help.

How about we do this. I run through this and we can do it again once everybody has their account.

Yeah, that sounds good. So, what we're going to do in the interest of time here is we're going to, Pankaj is going to run through end-to-end sort of the demo as we get the stuff set up and everyone's credit cards get unblocked.

Yeah, you know, who would have thought, you know, we were talking this big game of, oh, TensorRT-LLM, it's so hard, it's so technical, there's going to be so many bugs, and then it's the payment processing. So, yeah, you know, that's live demos for you.

So, anyway, yeah, go ahead and work through it and then we'll do it kind of again together once everyone has their account.

Allright. Yeah, let me run through this. I'll follow all the steps. I already have an account on RunPod. So, let me spin up a new instance here. And I'm picking up the 4090 here, which, this one. And it has high availability, so that should be fine.

And I'm going to edit this template and get more space here. This doesn't cost anything extra.

Yeah, we need more space because the engine, everything that we're installing and the engine we're building takes up a lot of gigabytes. So, otherwise, we'll run out of memory.

Yeah, better to be safer, yeah. Even though these engines are small, engines in general can be very, very big. They can be hundreds of gigs. And I'm going to pick on-demand because I'm doing this demo. I don't want the instance to go away, but feel free to use Spot for your use case.

So, I'm going to do that.

Do you need container disk or volume disk?

You want to set the container disk to 200 gigabytes so that you have enough room to install everything.

And then I'm going to deploy Spot.

It's going to be a bit slow, but, you know, feel free to ask any questions. And I feel like this way, we'll take it slow, but we'll make sure everything is understood by everybody. So, what's happening now is that this pod is spinning up.

One thing to note here is that it has a specific image of Torch with a specific CUDA version. It's very important that the node has GPUs. And the first thing we're going to do is that once this pod comes up, we're going to check that it has everything related to GPUs running fine.

So, this is starting up.

Now I'm going to connect. It gives you nothing sensitive here. It uses your SSH keys, but the names are not sensitive. So, I'm going to just do that. Log into that box.

Could you slow down?

Sorry?

Could you just make it bigger?

Oh, okay. Sorry. Yeah, yeah. This is much smaller.

I think the pod is still spinning up, so it's taking a bit of time.

Okay, it's running.

Okay. Now it's up. So, to test that everything is set up properly.

Just, is it possible to scroll it to the top of the screen?

Oh, yeah.

Okay, great. So, we are on this machine that we spin up. We're going to run NVIDIA SMI to make sure that the GPU is available. And this is what you should see. One thing to note here is this portion, which shows that the GPU has more than 24 gigs of memory, the RTX 4090 has.

Andright now, it's using one memory. I think it does some stuff, like, by default. So, one MAG is already taken.

So, now we're going to go back

to workshop and then just follow these instructions.

Manual engine build. We are at this point. And now we're going to install TensorRT-LLM. This is going to take a bit of time. TensorRT-LLM comes as a Python library that you just pip install. And that's all we're doing.

We're setting up the dependencies. This APD update is setting up the Python environment, OpenAPI, and other things. And then we just install TensorRT-LLM, not from PyPy, but from NVIDIA's own PyPy. That's where we find theright versions. If you focus on this line, let me kick this off.

Then I can come back here and show you that we're using a specific version of TensorRT-LLM. And we need to tell it to get it from the NVIDIA PyPy using these instructions. And all these are on the GitHub repo, if you want to follow from there.

I saw a guy with a camera, so I started posing.

Which place you're working from?

I think 310, I think.

310?

310, yes.

This is like MPI for Python?

This command should have instructions to install it.

I also, I want to check in with the room. Has anyone else had success getting RunPod up and going using your phone Wi-Fi? It's working? Okay. Awesome. Crisis avoided. Thank you so much to whoever from over there suggested the idea to begin with.

Really saved the day.

Great. So, we're just waiting for it to build. It takes some time.

This is the best part of the job. You know, you wait for it to build, you can go get a snack, you can go, like, change your laundry. It's very convenient that it takes this time sometimes.

It used to be compilation takes time. Now, engine build takes time.

Yeah, exactly.

I think we're very close.

And I promise, like, then the fun part begins.

Are you saying that pip install isn't the fun part? I think this is pretty fun. You know, look at all this, look at all this lines. You know, this is real codingright here.

And if you want pip to feel more fun, try poetry.

Oh, that's true. Poetry is really fun.

Yeah. NVIDIA does publish these images on their container history called NGC. And there are Triton registries available for these things. Maybe we should have used that rather than RunPod, but it's all good. So, now let's check that TensorRT-LLM is installed.

And this will just tell us that everything is well and good. So, it printed the version. You should see that if everything is working fine. And then we're going to do the real thing. Now, we're going to clone the TensorRT-LLM Git repository where a lot of those examples are.

And I'll show you those examples while this cloning happens. It shouldn't take that much long. Maybe a minute or so. But TensorRT-LLM has a lot of examples. If you go to the TensorRT-LLM repository, there are these examples folders and there are a ton of examples.

Like Philip mentioned, there are about 50 examples. And we're going to go through the Llama example here. So, if you search for Llama,

that's the one we are going to look into. And so, the cloning is complete. And we go back to these instructions. And now we're going to actually build the engine. Actually, one more thing. How many of you know about HF Transfer?

Have you used their Transformers library from Hugging Face? So, HF Transfer is a fast way of downloading and uploading your engines. It does sliced download. It takes the URL and patches them up into slices, downloads them all in parallel.

And it works really, really fast. It goes up to, like, one gigasecond. So, we should definitely do that, which is what I did just now. Now, we're going to follow this step by step. First thing we're going to do is download from Hugging Face.

And let's see, like, how fast the Wi-Fi here is, how fast this downloads. So, not bad. It's going at one gigs a second. So, HF, yeah.

It's from the Wi-Fi firmware.

Oh, from, oh, you'reright. You'reright. You'reright. See, but this is what I call good software. Downloads at one gigasecond. Now, first thing to build with TensorRT-LLM is that we have to convert the Hugging Face checkpoint into a checkpoint format that TensorRT-LLM works with.

And checkpointing also covers tensor parallelism and quantization. Sometimes you need a different kind of checkpoint for doing those things. So, I'm going to run this command to convert the checkpoint. And this should be pretty fast. It's just converting weights to weights.

That's pretty fast, like three seconds. And now we do the actual build. And I'm going to do this basic build here. There are a ton of options that this command takes, the TRT-LLM build command. In here, we are just saying that take this checkpoint and build me an engine with most of the default settings.

And that should build the engine.

Now, it will print a lot of stuff about what it's doing, what it's finding, and how it's optimizing and all that. It won't make much senseright now, but later on, this could be very useful. So, the engine was built.

That's pretty fast,right? It's a small model, only a billion parameters. So, that was pretty fast. And now, let's try to see how big the engine is. I'm going to do that. And the engine is two gigs in size.

This is about how big that model is on Hugging Face. So, the engine itself adds very little storage or memory needs, maybe like hundreds of megabytes, but very tiny compared to the overall. And those weights are bundled into the engine.

And what is this engine? This engine is something that the TensorRT-LLM runtime can take and it can execute it. You can think of it like a shared library. It's kind of like a binary in the standard format that the binaries are in.

But it's something that TensorRT-LLM can take and interpret. Ultimately, it's a TensorRT engine because that's what TensorRT-LLM works with. It creates TensorRT engine and then TensorRT is the one that loads it. But TensorRT-LLM gives it these plugins that TensorRT understands and then is able to make sense of it.

And now, let's execute this. So, these examples also come up with a run script that we can run. And we're going to run that. So, what this is going to do is start up the engine and give it a very tiny request.

And we should expect a response. And that's what happened here. Our engine was launched. We gave it an input text of born in Northeast France. So, we are trained as A. And the model printed out the response beyond that.

Painter in Paris moving before moving to London in 1929. And this is a standard example that comes with TensorRT-LLM. So, if you follow along these instructions, you should see that.

Any questions at this point?

What is the conversion? What happens during conversion? What is necessary?

The question is what happens during the convert checkpoint. I think there are three things that happen, potentially three things. First thing is that TensorRT-LLM needs the tensors to be in a specific format to work with. So, think of it as a pre-processing.

There are many ways of specifying a model. It can be on Hugging Face. It can be exported from PyTorch. It can be ONNX. There are many, many different ways of specifying these models. So, the first thing it does is that it converts that into a format that it understands.

So, it does some kind of translation into a standard structure. Second thing is quantization. For quantization, it needs to

quantize the weights. It needs to take the weights and quantize them into the quantized versions of them. And that happens at convert checkpoint too. Not necessarily, though. They also have a quantize script. Some of those quantizations happen, some types of quantizations happen in convert checkpoint, but they also have a different way of quantizing.

They call it, I think, AMO. There is a library called AMO which does that. And that can also be used for doing it. But I think AWQ and SmoothQuant, they happen in convert checkpoint. And third thing is tensor parallelism.

For tensor parallelism, you need to divide up weights into different categories for the different GPUs that they will run on. So, it does that during convert checkpoint as well.

Yes.

Me too.

Do you have any notion of the max output of the light? So, for instance, if you do 50, it'll cut off. If you do 250, it gives you like 2 or 300 something. If you do 500, it gives you the same as 250.

So, I just don't know whether or not that actually is a passage to the model and then it tries to approximately get that.

Yeah. So, there's two places that the max output is set. So, the first place is when you're actually building the engine, you give it an argument for the expected output sequence length. And then that's more just so, like, for the optimization side, you know, so that you're selecting the correct codec kernels and so that you're batching everything up correctly.

And then once the engine is built, it just uses a standard, I think it's max tokens,right, is the parameter. And yeah, you just pass max tokens and that'll limit how long it runs for.

Yeah. I guess what I'm asking is, does it influence the generation before generation happens? Or is it strictly just being cut off? Like, I guess.

Right. Like, are you asking if you make the engine with a shorter output?

No, not making the engine.

Oh, okay.

Just a prediction.

No, as far as I know, all the max token does is it just cuts off influence after a certain number,right?

Yeah. So, the way I would put it is that normally, if you give it a large number of max tokens, it would emit an end-of-sequence token. Most models have a different end-of-sequence token. And it's up to you. You can stop there.

You can configure at runtime. Like, I don't want more than that. But you can also tell it, ignore end-of-sequence. I just want the whole thing. And we do need it for performance benchmarking. For example, when we are comparing performance across different GPU types or whatnot, we want all of those tokens to be generated.

So, we can tell it, like, give me all of them.

Thank you.

Welcome.

It's in the back there.

Oh, great. Great question, actually. So, in build checkpoint, a lot of stuff is happening. You're taking these weights and

you're generating this thing called a network in TensorRT. TensorRT has this notion of a network. And what you need to do is populate that network with your weights and architectures. So, it actually does that. It creates that network and feeds it these weights.

It also does inference during building the engine for doing optimizations. So, it generates for every model type, it has a mechanism of generating sample input. And it passes that into the TensorRT engine that it's generating. And then it optimizes it that way.

And as a result, this TensorRT engine is generated in memory, which is then serialized. So, all of this is happening in that.

And there's a lot of nuance to it. If you get a chance, you can look at the source code for that TRT-LLM build. I'll post references in that GitHub repo. And you can follow on. There's lots of options.

But let me try if I can find a help here.

Ah, the VR person.

What's going on? Sorry.

Oh, yeah. This is a lot of stuff here, which we can go through.

Yeah. Maybe I should go through some of them, which are very important. Yeah. I think a lot of stuff is important here. Like the max beam width, if you're using beams for generating the graph, you can generate logits, not just the output tokens.

We can also generate logits if you want to process them. There are a lot of optimizations that you can use. Like, there's an optimization called fused MLP. There is contrast chunking. There is a lot of stuff. And I think you should play around with those at your time.

LoRA is very good to play around with. Yeah. I'll try to leave some more examples in the GitHub repo to try. Okay. So, let me go to the next one. Just one more thing I want to do is FP8 quantization.

RTX 4090 is actually an amazing GPU. It's pretty cheap, but supports FP8. So, we're going to do an FP8 engine build now. So, in this case, like I said, some of these optimizations, these quantizations are not in convert checkpoint, but quantize.py, which uses a library called AMO in NVIDIA.

So, I'm going to run that now. And

let me spend some time here. We're saying is we're telling it that the quantization format is FP8. But also note that we are saying KV cache D type is FP8. So, FP8 quantization actually can happen at two levels.

You can weight quantize to FP8, but you can also quantize the KV cache with FP8. And doing both is very critical because these GPUs, you might have heard of things called tensor cores,right? Tensor cores are very, very, very important because they can do quantized calculations very fast.

For example, if you look at a spec of the H100 GPU, you can see that the teraflops that you can get, number of computations that you can get with lower quantization options are much more than higher. For example, FP16 teraflops will be much lower than FP8 because you can use these special tensor cores for doing more FP8 computations in the same time that you would do FP16.

But for that to happen, both sides of the matrix have to be the same quantization type. Mixed precision doesn't exist. At least now, it's not very common or popular. So, you want both sides to be quantized. And when you quantize both the KV cache and the weights to FP8, you get that extra unlock that your computation is also faster, which can be critical for scenarios which are compute-bound.

And as you would know, in LLMs, there is a context phase and generation phase. Generation phase is memory bandwidth-bound. But the context phase is compute-bound. So, that can benefit greatly from both sides being quantized. So, in this case, we are saying that quantize both weights and KV cache.

And it's actually not a trivial decision to do that because weights quantize very easily. You hardly lose anything when you quantize weights. The dynamic range of weights is generally much smaller. You can use int8 or when you do FP8, there is hardly any loss.

KV cache doesn't quantize as well. And that's why FP8 is a game changer. Because what we found is that when you quantize the KV cache with int8, even using SmoothQuant, there is still degradation of quality. And practically, we've never seen anybody use it, even though there are a lot of papers about it.

And it's great, great technology. But practically, it was not there until FP8. FP8, even KV cache quantization works extremely well.

Let me show something with that, actually, if you don't mind. Back on the

if we go to

I'll just show, like, a little visualization for FP8 that shows off the dynamic range. So, oh, hey, look. It's us. Yeah. So, when you look at the FP8 data format, it has a sign. And then rather than so, there's two different FP8 data formats.

But we're using the E4M3 format. So, basically, you have four bits dedicated to an exponent. And that's what gives your FP8 data format a lot of dynamic range versus int8, which is just, you know, like, what, like 256 to.

256.

Yeah. So, you still have the same number of possible values, but they're spread apart further. That's dynamic range. And it's that which allows you to quantize this much more sensitive KV cache.

Yeah. Yeah. Exactly. Basically, you have the multi-channel exponents. You basically are able to quantize smaller values better. You give more bits to smaller scale than larger scale. You don't have to fit into a linear scale. And that's where FP8 excels.

So, going back to the presentation, the FP8 quantization is done. And I forgot to show you this, but there is calibration involved here. If you look at this stack here, we actually give it some data. We feed it some data and let it calibrate.

Because as you would know, int8 and FP8, you have a start and end range. And they differ in how you divide up that range into data points. But you have to find min and max. And for that, you need calibration.

So, we give it a standard data set. And you can change the data set. But we give it a specific data set. And it does multiple runs. And we try to calibrate, like, what are the dynamic ranges of each of the layers of this transformer architecture.

And based upon that, we specify that min and max for each layer separately. There is more detail there, but at the high level, that's what is happening.

Yes. In the back.

Yeah. Yeah. It's possible that the ranges can vary a lot with data set. And this used to be more critical with int8. With FP8, we found that you get to a good state pretty fast. But it's worth thinking about trying different data sets, especially if you know what data set you are going to be calling it with.

It could be worth it. It just works very well out of the box, but it's not perfect.

So, going back to the workshop. So, we were following along here. And yeah. So, after you quantize it, the steps are very similar as before.

Now, we are building an engine with FP8. And internally, all the CUDA kernels that are being used are now FP8 specific. They are different kernels which use the tensor cores in theright fashion.

And this should be pretty quick as well.

And there is a lot of depth here as you learn more about it. You don't need to. But there are things like timing cache. There are optimization profiles in TensorRT through which you tell what sizes we expect and it does optimizations.

But this is a good thing. So, now we have the engine. Let me do a deu on that and to see the size of that engine now. And the size is 1.2 gigs, which is about half of previous, which is what we expect because we quantized.

And now, let's run this engine and see the output. And it should be pretty similar to what we saw before. So, using this run script, now it's going to load the engine. And then we'll do an inference on top.

That should be pretty quick. So, yeah. Here's the output. The same input as before and about the same output as before. And that's what we generally observe with FP8. FP8 quality is really, really good. It's very, very hard to tell the difference.

That's it for this workshop, this part of the workshop.

Yeah. Awesome. Thank you. So, you know, I definitely welcome you to keep playing around with this RunPod setup and trying different things, try to build different engines and stuff. But we're going to move on to the next step, which is an automated version of basically exactly what we just did.

So, we're going to show a few things to make this easier.

So, we're going to be using for this next step something called Truss. Truss is an open-source model serving framework developed by us here at Baseten. Pankaj is the one who actually wrote a lot of the code. All I did was name it Truss because I was riding on a train and I was like, huh, what should I call the framework?

And then we went over Bridge. And I was like, I know. I'll call it Bridge. But that was already taken. So, I called it Truss. So, it lets you deploy models with Python instead of building a Docker image yourself.

It gives you a nice live wheel of dev loop. And what we really wanted to focus on when we were building this, because it's kind of the technology that sits onto our entire model serving platform, is we really wanted a lot of flexibility so that we could work with things like TensorRT, TensorRT-LLM.

You can run it with VLM, Triton. You can run a Transformers model, a Diffusers model. You can put an XGBoost model in there if you're still doing ML. Like, you can do basically whatever you want with it. It's just Python code.

Let me interject. Like, Truss is actually a very simple system. It's a way of running Python code, specifying an environment for running the Python code and your Python code. So, it's sort of like a very simple packaging mechanism, but built for machine learning models.

It takes account of the typical things you would need with the machine learning models, like getting access to data, passing secure tokens and such. But it's fundamentally a very, very simple system, just a config file and some Python code.

Exactly. And so, looking at that config file, we're not even actually going to write any Python code today for the model server. We're just going to write a quick config. So, actually, this morning, I was eating breakfast here.

And I sat down with a group of engineers and we were talking about stuff. And everyone was complaining about YAML and how they're always getting, like, type errors when they write YAML. So, unfortunately, this is going to be a YAML system.

So, apologies to my new friends from breakfast. But what we're going to do is use this as basically an abstraction on top of TLT-LLM. Why do you need

abstraction on top of TLT-LLM?

Pankaj, quick question for you. What's the name of that C++ textbook you were reading before bed every night the other month?

Modern C++.

What?

Modern C++.

Yeah. C++. So, you know, before bed every night, I was watching Survivor. And so, for those of us who are not cracked software engineers, and even for those who want to get things done quickly, we want to have a great abstraction.

What does that abstraction need to be able to do? It needs to be able to build an engine. And that engine needs to take into account what model we're going to run, what GPU we're going to run it on, the input and output sequence lengths, the batch size, quantization, any of the other optimizations we want to do on top of that.

And then we also want to not just grab that and run it in the GPU pod somewhere. We actually want to deploy it behind an API endpoint so that we can integrate it into our product and stuff. So, I'm going to show how to do that.

Let's see here. This is yours now. I'm stealing ooh, this is a good mic. I might not give this back, Pankaj. This is a good mic. Allright. So, we're going to go over let's see. This is in the RunPod thing still?

Yeah. I think you go to the first one.

Great.

The second one.

Perfect.

Let me do the workshop.

OK. Yeah. You do that. This is his computer, not my computer. So, I don't know where anything is. It's like walking into someone else's house.

There you go.

Allright. Thank you. Thank you so much. OK. So, what we're going to do in this second step is we are going to do basically exactly the same thing we just did, just automated. So, for this step, we're going to use Baseten.

We're going to give you all some GPUs to play with here. So, if you want to follow along, I really encourage you to do so. You're going to go sign up at Baseten. Your account will automatically get free credits if our fraud system is a little freaked out by everyone signing up at the same time.

Well, fortunately, we have some admin panel access ourselves over here. So, we'll just unplug this, approve you all, and plug it back in. So, yeah. So, everyone go ahead and sign up for Baseten. We're also going to want you to make an API key real quick and save that.

And then once that's all done, we're going to jump into

this part of the project.

Yeah. We use your account on this.

OK, everyone. So, I know there's a few errors going on. We have pinged the team about that. Let me let you in on a little secret. We shipped this on Thursday as an internal beta. And this is the very first time anyone who doesn't have an@baseten.co email address is using our new TensorRT-LLM build system.

So, if there

yeah. So, sorry for tricking you all into beta testing our software. But hey, that's what demos are for,right? So, we'll get that sorted out. In the meantime, we have an image cached locally, which means we can keep going with the demo as if nothing ever happens.

So, let's see. So, what you would see in the logs as you deploy this.

Deployed again this time.

Yeah. Oh, well, I mean, I can just kind of look through the logsright here. Let me actually just wake it up. Sorry, what was that?

You take it apart.

OK. Yep, yep, yep. I got you.

One plus.

Yep. Allright. Big logs.

Let's see. Allright. So,

what you're seeing here as we oh, sorry. We got a lot of logs here,right? Because we tested a bunch with

the scale up. Anyway, what you see is you see the engine getting built and then deployed. And to walk through the YAML code really quick yes, here. So, we talked about that there are a bunch of different settings that you need to do when you are working with TLT-LLM.

And you can set all these settingsright here in build. So,right now, we're doing something with an input sequence and output sequence of 2,000 tokens each and a batch size of 64 concurrent requests. We're using the int8 quantization because we're running on an A10.

And that does not support FP8 because it's an Ampere GPU, which is one generation before FP8 support. And then, of course, you pull in your model and stuff. And then if we want to call the model to test that it is working, we can come over here to the call model.

We can just test this out really quick.

Is it a caption?

Yes.

So, we do not. On Baseten, we have T4s, A10s, A100s, H100s, and H100 Migs, and L4s as well. We generally stick with the more like data center-type GPUs rather than the consumer GPUs.

It's a good GPU though.

Yeah. I want one for, well, Pankaj here. So, I'm going to say that I want it for legitimate business purposes. And it should be an approved expense. I don't want it for playing video games, definitely not. So.

One more question.

Yes.

No, but I bet he can.

Yeah. All of that source code is open source. And typically, when new models come up, those companies provide convert checkpoint scripts. But if you can follow those scripts, it's not terribly difficult. It's mostly like if you're familiar with the Transformers library, it's about reading weights from a Hugging Face Transformer model and converting that into something else.

It's a simple transformation. So, it should be possible to do it yourself if you want.

Awesome. So, once your model is deployed, again, you can just test it really quick. You can call it with an API endpoint. But yeah, we're coming up on 2:30 here. So, I'm not going to spend too long on this example.

Let's see. But we've been talking a big game up here about performance,right? And performance is not just, OK, I'm testing it by myself. Performance is in production for my actual users. Is this meeting my needs at a cost that is reasonable to me?

And in order to validate your performance before you go to production, you need to do benchmarking. And you need to do a lot more rigorous benchmarking than just saying like, hey, you know, I called it. It seemed pretty fast.

So, what do you want to measure when you're benchmarking? Say it with me, everyone. It depends. That's what our software engineers are always saying. So, depending on your use case, you might have different things that you're optimizing for.

If you say like a live chat service, you probably really care about time to first token for your streaming output because you're trying to give people instantaneous responses. You might also care a lot about tokens per second. So, that's how many tokens are generated.

Generally, some good numbers to keep in mind is somewhere depending on the tokenizer and the data and everything and the wheater, somewhere between 30 to 50 tokens per second is going to be about as fast as anyone can read.

So, if you got 50 tokens per second, generally, it's going to feel pretty fast. People aren't going to be waiting for your output. However, if you're doing something like code, code takes more tokens per word than, say, natural language.

So, you're going to need even more tokens per second for that nice, smooth output. And then from there, getting into 100, 200 tokens, that's when it just feels kind of magically fast. But again, our influence is all about trade-offs,right, when we're optimizing.

So, sometimes you might want to trade off a few maybe you're going to go at 100, not 120 tokens per second because that gets you a bigger batch size, which is going to lower your cost per million tokens.

Another thing you're going to want to look at when you're running your benchmarks is your total tokens per second. So, there's the tokens per second per user,right, like per request, how many tokens is your end user seeing. And then there's tokens per second in terms of how many tokens is your GPU actually producing.

And that's a really important metric for throughput, for cost, especially if you're going to be doing anything that's a little less than real time. You want to look at this not just once. You want to look at the 50th, 90th, 95th, 99th percentile.

Make sure you're good with all those. And you want to look at the effects of different batch sizes on this. And so, something is that benchmarking actually reveals really important information. It's not linear, and it's not obvious. The sort of performance space of your model is not this nice, flat piece of paper that goes linearly from batch size to batch size.

So, this is a graph of time to first token for like a Mistral model that I ran a long time ago. I just happened to have a pretty graph of it. So, that's how it ended up in the presentation.

So, if you look at the batch sizes as it's increasing, doubling 32 to 64, the time to first token barely budges. But as it goes from 64 to 128, doubling again, the time to first token increases massively. And in this case, the reason behind that is we're in the compute-bound prefill step when we're talking about computing the first token.

And there's these different sort of slots that this computation can happen in. And as you increase the batch size, you're saturating these slots until eventually you have an increased chance of a slot collision. And that's what's going to lock at your time to first token.

I'm glad you're nodding. I'm glad I got thatright. But yeah, all of this to say the performance that you get out of your model once it's actually built and deployed is not necessarily just going to be linear. It's not going to be something super predictable.

You have to actually benchmark your deployment before you put it into production. Otherwise, these sort of surprises can happen quite often. So yeah. So, Pankaj, do you want to take over the benchmarking script?

So, just for this workshop, we wrote a benchmarking script. It's not the script we use ourselves, but it's a simpler version so that you can follow along that if you wanted to modify it, you can play around with it and understand it easily.

If you go into that repository, it's a very simple script where we send requests in parallel just using Python, using async libraries. And all you give it is the URL of the endpoint of the model where your model is deployed.

And you can give it different concurrencies and input lengths and output lengths and give it a number of runs. You want to run these benchmarks a number of times to get an idea of values. One run might be off.

So, I'm just going to run that script. And it's all in the benchmark repository. It's all structured using Makefiles. And there is a Makefile target for benchmark that we're going to use. And I think the README should also have instructions on that.

So, we're basically going to run this. And we're going to need the base URL. We need two things. We need to export the API key. And then we need to supply the URL. So, once you deploy your model on Baseten, you would see deployed.

And like Philip said, there is a call model button. There are various ways you can deploy it. Ultimately, it's an HTTP API. And you can just copy this URL for that model for our benchmarking script. But if you want to play around, there are examples in all kinds of languages.

And you can also click streaming. So, it'll give you streaming code. Streaming is very important with large language models because you want to see the output as soon as possible. So, you're going to take this output. And I don't know if I exported the API key.

So, give me one second to export the API key.

Come on, Pankaj. Scroll up so you can show your API key to us. This is a good time to mention oh, yes. What's your question?

Yeah. So, when well, actually, as we're looking at it, it's slightly different. But yeah, we're basically talking again about

setting all these parameters.

Right. And if you lose the API key, you can always revoke it. So, that's good.

Yes. Well, this is a good time to mention that Baseten is SOC 2 Type II certified and federally compliant. And that is why we cannot show you our API keys. That is one of the reasons.

And now, we're just going to give it the URL here. Let me go back. This

call model. Allright. So, I'm going to do this first run with a concurrency of 32 and input and output lengths of 1,000. And let's see it work. So, first, it does a warm-up run just to make sure that there is some traffic on the GPU.

You always want to have a warm-up run before you get the real numbers. Now, as this is running, and you can see the TPS here, the total TPS is 5,000. And this is on 8NG. 8NG is not the most powerful GPU, but this TPS is still very, very high.

5,000 is very, very high. It's because this is a tiny Llama model. Tiny Llama is a tiny model, just like a billion parameters. And that's why we see this very high. But yeah, on bigger GPUs with Llama 8G, you should also see very, very high values because H100s are very, very powerful.

And TensorRT-LLM is very, very optimized. I think we see up to like 11,000 tokens per second. And you should do a comparison. It's really, really high. In this case, we have two runs. And you see these values. Let me try a different run.

Now, I'm going to do concurrency of one. And one is good to know how best of a time to first token you can get. So, you're just sending one request at a time. And many requests, but one at a time with the same input and output length.

And this should run OK. I think I ran into an error. Let's try again.

So, you see time to first token of 180 milliseconds here. And this is from this laptop. I'm running itright from this laptop on this Wi-Fi. And my model is deployed somewhere in US Central. And this is 180 milliseconds for that.

Yeah. So, the vast majority of that time to first token is going to be network latency,right, not model.

Not the model itself. It should be pretty fast.

Yes, sir, over there. What is that error message on PyTorch TensorFlow Flex?

Oh, not that one. This one. None of.

Is this a message from Baseten or is it a message from.

This is from this script that I'm running. I didn't do a thorough job of cleaning up everything. We're saying that we're just making an RPC call. We don't need PyTorch or whatever.

Also, this is from the local runtime.

Yeah, local runtime. Something from the machine. This script, if you look at that, all it's doing is RPC. Let me go through that real quick. Benchmark script. It's simple Python using async. And it's amazing how good Python has got.

With this async API, you're able to load this model with thousands of tokens per second, all of them coming in streaming. Python has actually gotten really, really well. There was a case where I was loading with K6. And K6 client became a bottleneck because H100s are so fast.

But Python could keep up. Python was able to load it very well. But I'm getting distracted. So, we tried concurrency one, which is like the best-case scenario. Latencies are very, very good. And TTFT should be very low. Now, let's go to the other extreme.

If you look at this model that we deployed, we created it with the batch size of 64 maximum. So, now we'll do 64. And I'm hoping we see throughput improvements. So, ignore the warm-up run. And this is going to take a bit longer because now we're going to send 64 requests at the same time.

64 is not that high. We can go even higher. So, in this case, you see total TPS of 7,000, which is even higher than before. We saw 5,000 before. This goes up to 7,000. But maybe this is a fluke.

Let's wait for the second run. And you can run this is also 7,000. So, this is much better than what we saw before. So, if you increase batch size, you would find that your latencies become higher. Latencies in the sense that for every request that a user is sending, now tokens are coming slower and slower.

And then you have to make a trade-off at some point. Like, is it still good enough? Is it still more than, say, 30 or 50 tokens per second that users won't perceive? And at some point, it will become unusable.

And as you increase batch size, the pressure on your GPU memory also increases because all these extra batches, they require KV cache to be kept in GPU memory. So, you might hit that bottleneck. So, depending on all these scenarios, you want to experiment with different batch sizes and find the sweet spot.

And that takes a bit of time. But it's not terribly complex.

Yeah. So, this script is there for you to modify and play around with. It's pretty simple. It's pretty much a single Python file. Not much in there. You can modify it. You can run it.

Yes.

Yes. The question was, did I have a max batch size in mind when I was running it? Yes. Because I deployed the model with the config in this workshop. Let me show you that. I built the model with a max batch size.

And you can increase that batch size. So, let me show you that.

So, in this tiny Llama model that I deployed, I specified a max batch size of 64. So, if I go beyond 64, it's not going to help me because all those requests will just wait. And yeah, actually, there's one interesting thing I want to show you.

So, this is a good question. You can look at the logs here. And in the logs, we put these metrics for what's going on. I think maybe I'm not.

Yeah. And if you wanted to increase your batch size past 64, you just change the YAML and say, like, oh, match batch size to actually be like 128 and build a new engine by deploying it.

Yeah. So, if you look at these logs here, it shows how many requests are running in parallel, active request count. You can actually observe how many requests are being executed in parallelright on the GPU because there are chances that you haven't configured somethingright and that for whatever reason, the requests are not all getting executed in parallel.

For example, a common mistake one could make is that when you deploy on Baseten, and this is Baseten specific, but just to take an example, there are scaling settings in Baseten. You can specify the scale. And you can specify what is the max concurrency the model will receive.

In this case, I've set it to a very, very high value. So, it won't become a bottleneck. But there are chances that you forget. You make a mistake there. You can check these logs. And they will actually tell you what's happening on the GPU.

And I think I lost that again. Let me go here.

So, yeah. These are actual metrics from the TensorRT-LLM batch manager, which tells you what's going on. It also tells you about the KV cache blocks that are being used. And that helps tune you helps you tune the KV cache size.

For example, in this case, it says it's using 832 KV cache blocks. And 4,500 are empty, which means there is way more KV cache than is needed for this use case. So, just to mention that as an aside.

Yeah. I think that's it for that presentation.

I'm going to talk about that next slide. Thank you. You're reading my mind.

Yes.

It does, yes. TensorRT-LLM does come with a benchmarking toolkit. It's very, very good. The only downside is that you have to build it from source. It's not bundled with the TensorRT-LLM Python library, which is my gripe. I'm going to ask NVIDIA to fix that.

They have benchmarking tools. There are two benchmarking tools. One that just sends a single batch and

measures the raw throughput you can get without serving it through in-flight batching. And there is a second tool called the GPD Manager Benchmark, which actually starts up a server and does in-flight batching on top. So, there are two tools.

And they're very, very good quality. But they're not available easily. Building TensorRT-LLM is like with 96 CPUs, it takes us 1 and 1/2 hours to build it. It's not for the week of art.

Or for the short of workshop. So, we just have a few minutes left. So, I want to run through a few slides and then leave time for last-minute questions. So, I was asked, how do we actually run this in production?

What does the auto scaling look like? How does that all work? So, how do you run a TensorRT engine? So, you use something called Triton, the Triton Influence server. And that's what helps you take the engine and actually serve requests to it.

We're actually working on our own server that uses the same spec but supports C++ tokenization, decode tokenization, custom features for even more performance. But

as we've talked about, the engine is specific to versions, GPUs, batch sizes, sequence links, all that sort of stuff. So, that causes some challenges when you're running it in production. We've talked this whole time about vertical scale,right? Like, how do I get more scale off a single GPU?

There's also horizontal scale. How do I just get more GPUs? How do I automatically scale my platform up to meet my traffic demands? So, some challenges in scaling out in general. You have to automatically respond to traffic. You have to manage your cold start times.

You have to manage the availability and reliability of your GPU nodes. You have to route requests, do batching, all that kind of stuff. And then TensorRT-LLM adds a few more challenges. You've got these large image sizes. So, that's going to make your cold starts even slower.

You've got these specific batching requirements. So, you can't just send whatever traffic however you want. And you have these specific GPU requirements. So, when you spin up a new node, it's got to be exactly the same as your old node.

Or your model's not going to work. And unfortunately, our workshop is almost over. Otherwise, I would love to give you an in-depth answer of how to solve all these problems. But the quick answer to how to solve all these problems is you run your code on Baseten because we solved it all for you.

So, Baseten is a model influence platform. It's the company that we both work at. With Baseten, you can deploy models on GPUs. You can use TensorRT-LLM. But you don't have to. You can use any other service, VLM, TGI, just like a vanilla model deployment.

You get access to auto scaling, fast cold starts, scale to zero, tons of other great infrastructure features. You get access to all of our model optimizations. We have a bunch of pre-packaged and pre-optimized models for you to work with.

So, yeah. The last thing before we go is we are co-hosting a happy hour tomorrow on Shelby's Rooftop Bar. I was not really involved in organizing it. But the sales guys who organized it told me that it's a super sweet spot and that we're going to have a great time.

So, yeah. So, I'm going to be there. And a bunch of other great AI engineers are going to be there. So, please feel free to sign up, come on through. We'd love to have you. It's going to be super sweet, cool party for the cool kids.

And you've been really great. Thank you for listening to us. And we open it up for questions. I feel one question we didn't answer about auto scaling,right? Maybe I should take that one now.

Yeah. Yeah. Go ahead.

So, how does auto scaling work? Yeah, yeah. Please. Yeah, please feel free to hold. I'll just finish that question because you asked that. I wanted to answer it. And how auto scaling works is that we use a system called K-Native.

But we forked it to make it work for machine learning use cases. K-Native, if I understand correctly, was built for microservices where your requests are very, very quick, like one second or two seconds. It doesn't exactly apply to machine learning model use cases where your request is long-lasting and you're streaming.

And you need to still scale. But some of the considerations are different. So, we had to actually fork it to be able to apply settings dynamically. For example, a lot of settings that you see on Baseten, they apply immediately, like within a second.

Whereas in K-Native, you would need to deploy a new revision for those settings to apply. And it's not even practical because the way you deploy a new model, it creates so much hassle and requires extra capacity that it's not good.

So, we made changes to K-Native to cater to the machine learning use case. But fundamentally, the idea is very simple. As your requests come in, you specify the capacity of your pod that it can take in. And if it reaches near there, we spin up a new pod.

And then the traffic spreads. If your traffic goes down, it goes your GPUs are your pods are reduced. Your GPUs are freed up all the way up to zero. And when the traffic arrives, it's kept in a queue.

And then the model is spinned up. And the requests are sent there. A lot of the machinery at Baseten is around improving cold starts. How do we start up these giant models, 50-gig models in under a minute? A minute sounds like a long time.

But when you're talking about 50 gigs, 50 gigs is also a lot. And for 10-gig models, we aim for less than 10 seconds. For 50-gig models, we aim for less than a minute because that's really important. Unless you can scale up in a minute, your request is going to time out.

So, you really can't scale to zero. So, it seems like a detail. But it's very critical. You can't have scale to zero without very, very fast cold starts. So, we have the whole machinery built out for that. Even before LLMs became popular, we've had this machinery in place.

So, what would I expect in terms of performance if I were like if I took a LLaMA 8 billion model and I used TensorRT versus I deployed it on Baseten, would I expect any difference in performance or approximately the same?

I think if you optimize it, can I? Yeah.

Are you saying deploy it with TensorRT versus deploy it on Baseten?

Yeah.

So, you would deploy it on Baseten using TensorRT under the hood to run it. So, Baseten is just going to facilitate that TensorRT deployment for you.

Right. But I could deploy it, I guess, on hardware.

Yes.

As well, like on hardware, for example.

Yeah. No difference. Baseten runs TensorRT-LLM. We just make it very easy to run TensorRT-LLM. So, it's easier and faster for you to get at the optimum point. But if you could do it yourself, yeah, it's the same thing under the hood.

And then I'm compelled by the fact that I'm in the marketing department to say things like, we also provide a lot of infrastructure value on top of that so that you're not managing your own.

Yeah. Yeah. Actually, that is true. Yeah. Because we have a large fleet. We get good cost. So, you actually won't pay higher on Baseten. It's not that it's going to cost you more on Baseten.

Yes. Over there.

Yeah. I was wondering if you had any kind of feedback off of that. Is the open source trust package that you guys have I see that that's got a Python inference client. And it seems to have just some features for being able to push models to Baseten.

Have you open sourced any of the serving infrastructure? If I just wanted to go off of our run pod example and do a local host, is any of that open source?

Yeah. So, only the packaging part is open source. The serving parts, I mean, that's kind of the platform. So.

I do want to mention is that from trust, you can create a Docker image. And that Docker image can be run anywhere. So, you get pretty close. You don't get auto scaling and all of those that nice dev loop.

But you do get a Docker image. And you can do a lot with the Docker image.

Oh, so it builds the Docker image locally?

It builds it locally. Yeah. You can build there is a trust image build command. You point it to a trust. It will build the Docker image locally.

Oh. OK. Allright. So, I guess then the serving code is it's got to be open source.

Serving on a single pod or container. But there is also spreading across multiple containers, the auto scaling and all of the dev loop. That is not. Yeah. But the serving, yeah. I mean, single model serving is in trust with that image.

Could I ask like a I'm just kind of curious. I don't know how long it's been in the case that you could figure it out. What is the stack that you have for serving models?

Yes. It's FastAPI at the trust server level. Then internally, we have our own server layer that we wrote to interact with TensorRT-LLM. That part is not open source. It's very new. So, we're still figuring out when to or where to open source it.

But there is also a version that uses Triton. So, there is the FastAPI. Then there's Triton. And then there is the TensorRT-LLM library. And then it runs the engine. Yeah. Yeah. Yeah. Exactly. Exactly. We have we work on multiple cloud providers.

We are spread across. I don't want to say globe, like mostly US, but also Australia and a few other places. So, we have access to many different kinds of hardware. We find theright hardware to build the engines. And then we deploy it.

Do you use on-premise clusters? Or do you use public cloud?

Yes. You can yeah. You can use self-hosted clusters. Our stack is built that way. That's one of our selling points. We're not giving you an API. You can run the entire stack in a self-hosted way.

Awesome. Well, look. The conference organizers were very clear with us. All sessions and workshops are to end on time. So, I'm going to wrap it up here. But we're going to beright outside if you have any questions. If you have any easy questions, come see me.

If you have any hard questions, please go talk to Pankaj instead. Thank you all so much for being here. It was so much fun doing this workshop with all of you. Again, I'm Philip. This is Pankaj. We're from Baseten.

And thank you so much for being here. Have a great conference, everyone.

Thank you.