Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

Hey everyone. So we're going to go ahead and get started here. We've got a nice close group here today, and that's, I think, to everyone's benefit. This workshop is really for you. You know, I love the sound of my own voice.

I love talking. That's why I'm a developer advocate. But the purpose of this workshop is to help you get comfortable with SGLang. So if you have questions, if you have ideas, if you have bugs, ask Yineng or me.

And we're definitely going to be able to tailor this workshop to you and your interests and what you're working on. So the title of this workshop is an introduction to LLM serving with SGLang. We're going to be talking about SGLang and a little quick introduction.

So my co-speaker here, Yineng, is a core maintainer of SGLang, has been involved with LMSys org for quite a while now, is the sort of influence lead on the project, previously worked at Baidu and some other places, and also is an author of a few papers, including Flash Info.

And I'm Philip, and I got a B+ in linear algebra. So whether you're coming in here and you're super cracked or you're brand new at SGLang, we're going to have something for you. Whatever your skill level, this is the place to be.

So what are we going to do today? We're going to introduce SGLang, get set up a little bit. We're going to talk about the history of SGLang, talk about deploying your first model, a bunch of things you can do to optimize performance after that.

And then we're also going to talk a little bit about the SGLang community and how you can get involved and even do a little bit of a tour of the code base in case you want to start making open-source contributions.

So by way of introduction, let's see. What is SGLang? So SGLang is an open-source fast serving framework for large language models and large vision models. Generally, you use SGLang in a sentence along with either VLLM or TensorRT-LLM. It's one of the multiple options for serving models in production.

So the question is, why SGLang? Why should we invest in learning and building this library? And first off, it's very performant. SGLang offers excellent performance on a wide variety of GPUs. It's production-ready out of the box. It's got day-zero support for new model releases from labs like Qwen and DeepSeek.

And it's got a great community, strong open-source ethos, which means that if something is broken in SGLang, if you don't like something, you can fix it, which is a pretty huge advantage. So who uses SGLang? Well, we do at Baseten.

We use it as part of our inference stack for a variety of different models that we run. We also see SGLang being used very heavily by xAI for their Grok models, as well as a wide variety of inference providers and cloud providers and research labs, universities, and even product companies like Kosa.

So quick history of SGLang. It's honestly really impressive to me how quickly this project has come up and gotten big. If you look at the archive paper that was released in December 2023, that's 18 months ago. So in just 18 months, this project has gone from a paper to 15,000 GitHub stars almost.

You should all go star it so that we can get a little closer. And it's supporting all of those logos, all those companies we saw on the last slide. It's got a growing and vibrant community. It's got international adoption.

So yeah, incredibly impressive what the team has done in that time. And I'm going to turn over to Yineng now to talk a little bit more about that history and also how you got involved in the project.

OK. Hello. I'm Yineng. I'm the co-developer of the SGLang project, and I'm also the Software Engineer at Baseten. And before I joined Baseten, I worked at Meituan. And at that time, I worked for the internal click-through rate ranking model optimization and the LLM inference optimization.

And at that time, the creator of SGLang, named Liaming, just reached out, and then we have Google Meet. So at that time, when I left Meituan, I joined the SGLang project. So I worked closely with Liaming and Yineng on SGLang.

Also, SGLang used Flash Infer heavily because we used Flash Infer as the attention kernel library and the sampling kernel library. So I also worked with Zihao on the Flash Infer project. And currently, I'm the co-maintainer of the SGLang project, and I'm also the team member at LMSys org.

And that's the little point of trivia. That's the same LMSys org that just got $100 million to build Chatbot Arena from A16Z. I learned that while I was putting together the slides for this talk. So if you were here early, you were able to scan this QR code and get everything set up for the workshop.

If not, definitely grab thatright now. You've got the QR code. You've got the URL that takes you to the same place. Does anyone still need the QR code? OK. I've got a couple people still. Allright.

Anyone still need the QR code? Going once. Going twice. Yep.

To folks watching at home, you've got this great button on YouTube. It's called the fast forward button, so you can just skip this part.

Allright. Looking good. If you need this again, just let me know. I'll throw it back up there. So we're going to talk about how to deploy your first model on SGLang. So if you go over to the GitHub

yes.

So in this step, we're just going to get familiar with the basic mechanics of SGLang. SGLang is basically just like a servo command that you're going to run in your Docker container. There's a little bit of difference with using it the way we're going to use it in the workshopright now versus how you might use it if you're working directly on a GPU.

The difference is you're using something called trust to package it. Basically, you're putting in your SGLang dependencies and your command into this YAML file. You're bundling it, and you're shipping it up to a GPU. The reason we are using trust is because that is the way that you can get on Baseten.

And the reason you are using Baseten is because that is the only company on earth that will give me free GPUs because I work there. So we're going to be working on all these examples on L4 GPUs because they are cheap and abundant, and they also support FP8.

But this same product works on H100, H200, and Blackwell is coming soon.

Yeah, yeah.

Yeah, coming soon. So yeah, it's going to be basically like the same principles. If you go through here the configuration, you can actually, in your trust config, you can change the hardware type to H100 if you want. And

in the accelerator lineright there. But yeah, so what is the actual SGLang launch servo command that we're running here? So it's basically just like a bunch of flags. That's the thing to understand about using SGLang. It's all about knowing what flags are available, knowing what configuration options are available, knowing the support matrix that exists for them, and knowing how they interact with each other.

If you turn on a major speculation algorithm and then you also jack your batch size way up, well, that's probably not going to go so well for you. But if you want to do, say, your quantization along with some of these other optimizations, those play nice.

So yeah, what we're going to do this is the fun part of leading a workshop is the part where we just stand around watching you type. What we're going to do is give everyone about five minutes to work through this first example.

We're going to circulate the room if you have any questions. And then we're going to come back together after running the first example. Sound good?

Yeah.

Allright. Let's do it. Can you cut the mics for five minutes?

Pause, skip. It's great. These buttons, they're magical.

Has issues? Is anyone having issues where you're stuck trying to get into Baseten? You're in a waiting room, and it won't let you out. If you are, flag me. If anyone is having issues where you're getting an error in your code, please don't show me.

Show him.

A check on progress. Has anyone managed to get the first model deployed and running?

It's deploying. Awesome. Let's hope it's deploying really fast.

Let me take a look here. Allright. Sounds good. Can you take a look at the logs for me real quick?

Wow. Our Wi-Fi is just amazing here. I promise Baseten is usually faster than this.

Oh, OK. Well, it looks like it came up. So

you can use the sample code in call.py or call.ipynb. Or you can just use an ordinary OpenAI client.

What you need to call it, if you go back to your Baseten workspace with the model, what you need is scroll back up a little bit for me. You need that model ID. That's what's going to unlock your calling code.

Got it.

Yeah, paste it inright there. You'll need to run an actual Jupyter Notebook to run that. Allright. We've had our first successful deploy.

If you want to call it using the OpenAI SDK, using the call.ipynb Notebook, this thing up here, it's going to be different for everyone. This within the UI is your model ID that you use to set up the URL.

Hey, everyone. We're going to come back together here. It's about 9:45. So we're going to move on to the next stage of the workshop where Yineng is going to do some really awesome demos. If you are still getting everything set up, no worries.

All this stuff is going to be live on GitHub. Oh, sorry. Yeah, on GitHub, the repository with the workshop information is going to stay up. So you can keep following along. This is also all going to be published.

So it's going to be easy to go back if you have any issues. Anyway, so the next thing that we're going to look at, now that we have a sort of basic idea of, OK, SGLang is just like running a model servo, how are we going to actually make it fast?

So Yineng is going to show one demo, which is the CUDA what is it?

CUDA Graph.

Yes, CUDA Graph Max BS flag and how to set that to improve performance. And then we're also going to take a look at Eagle 3, which is a new speculative decoding algorithm, which also can improve performance. So take it away, Yineng.

Yeah.

Can I see my screen?

Yes. Yeah, good call. Zoom it in a little bit.

We're using OnePod because on Baseten, you don't get SSH access into your GPUs because security or something, I guess. I don't know.

OK. So yeah, I will use the L4 GPU. Yeah, this is the L4 GPU. And I have already installed the SGLang. Yeah, we can just use the pip install or install from source. And here is

this command line. Sorry. We launch the server, and we use the LLaMA 3.18b instructor model. And the attention backend uses FA3. This is the default. And when we

OK. It just started to load the weights.

So just to give everyone a little bit of context, the top window you're seeing here is the L4 that's actually running the SGLang servo. The bottom window here, LLM eval is a sort of industry standard benchmarking tool that we're just going to use to throw a bunch of traffic at the running servo.

Yeah, yeah. For sure. And yeah, we can see the log from the server. It shows that we capture CUDA Graph batch size. I think CUDA Graph is turned on by default. But the CUDA Graph Max batch size for L4 for this model is 8.

So it only captures 1, 2, 4, 8. And OK, the server is ready to run. And we can use the LLM eval to send a request.

Yeah, we can see that from the log. Here is the prefill batch. And here is the decoder batch. And we can see

under the decoder batch, when the running request is 10, it means that there are 10 running requests. And the CUDA Graph is false because the running request time is larger than the Max CUDA Graph batch size 8. That's why this flag is false.

And when this is false, we get 155 generation tokens per second. And we can use this one, divide 10. So I think per user, nearly 15 tokens per second. OK. We can kill the client. And we can also kill the server.

So yeah, we can use this command line as a base and set the CUDA Graph Max BS CUDA Graph.

For example, we can just set 32.

You've got to tape one there.

Oh, sorry.

Yep.

The network is not good.

Here, everyone is learning a very important lesson in the value of latency.

OK. Yeah.

Yeah, it's loading weights.

Yeah, and we can see that after we set the Max CUDA Graph batch size, the capture CUDA Graph BS, I think that the Max is 32. It's larger than the 8. And the server is ready to run. We also use the LLM eval to send a request.

OK. So first is the prefill batch. And then we can here is the decoder batch. OK. And yeah, here is the decoder batch. We can

wait for a moment. Decoder batch.

Yeah, for example, here is the decoder batch. And there are 13 running requests. And the CUDA Graph is 2. And here is the generation throughput. And I think per user should be

12. And we can compare with before.

It's not easy to compare. Yeah, yeah. I think

we have recording this video. And we can also see here, CUDA Graph. And we upload this one, CUDA Graph Max batch size demo. We want CUDA Graph to be true during decode because I think this is very important for the decoding performance.

But the default Max batch size is 8 on L4. And when we used LLM eval to send a request, we found out that, oh, the Max batch size is larger than 8. That's why we want to set or adjust the parameter.

Here, when we set it to 32, we can handle the realistic batch during benchmark.

Do you have any questions?

Why are we like mans to run this?

Oh, OK. The LLM eval?

Mhm.

Yeah, yeah, yeah. I think LLM eval is an evaluation tool. And we need to specify the model. And here is the model name. Here is the URL because I just used RunPod to run this. And it used the same node.

So that's why the URL is the localhost. And we specify the port, this one, 8000. That's why we use 8000. And we used the OpenAI compatible server. And here, the number concurrent or the batch size is 128. We set the max generation tokens.

We just used GSM Medicaid. I think it's a classical evaluation data set. And because we used the chat completion API interface, that's why we need to apply chat complete. And I just used few-shot 8. The limit means that because the GSM Medicaid has 1,319 promotes.

And when we use the limit 0.15, I think it's nearly 200 promotes. I can also share this command line in the repo. Yeah, yeah. Maybe I can add it.

Oh, sorry.

Yeah. So just to be clear, this command is running on the actual GPU itself.

Yeah.

So this is for when you have SSH access into the GPU you're running. On the service world, using the Baseten GPUs, you can't SSH in. But if you do have the access to a GPU where you can get SSH access, then you would use this LLM eval tool in order to simulate that traffic.

If you're using a more standard HTTP connection to a remote GPU, then you would use a different benchmarking tool that's request-based.

Yeah. OK. And do you have any other questions for CUDA Graph?

Why is default 8?

Yeah, yeah. I think the default 8 is because the L4 GPU feed memory. And we have some default configuration. We will set the when you didn't set the CUDA Graph Max batch size, the default value is none. And when the default value is none, we will set internally for specific hardware or for specific model.

Yeah. For example, it's TP1, and it's on L4. So the default is just 8. Yeah.

So what if someone by mistake adds a higher one for a large model or something?

Yeah, you can just try that. Because when you launch the server, you can see the startup parameters. And then you have a workload,right? And you use the LLM eval to benchmark, for example. And then you can analyze the server log.

And you find out that, oh, during the decoding, the CUDA Graph is disabled. And actually, we want to enable CUDA Graph. That's why we increase the Max CUDA Graph batch size. Yeah. OK.

Awesome. So let's see. Do you want to show the Eagle stuff? Or do you want to show the code-based stuff?

Yeah, yeah, yeah, yeah. OK. I think the next very important is about the Eagle stuff.

Yeah. So Eagle 3 is a speculative decoding framework. It came out very recently,right?

Yeah.

The paper was released a few months ago.

And so SGLang supports Eagle 3. And with it, you can configure a wide variety of different parameters around how many tokens you're speculating, how deep you're speculating, that kind of stuff. And Eagle 3 can have much higher token acceptance rate.

So obviously, when you're speculating, the higher your token acceptance rate, the better performance you're going to get. So we can take a quick look at some of those parameters that you showed. And then maybe the benchmark script you were showing me the other day.

Yeah, yeah, yeah. I think for the Eagle 3, yeah, we also provided the example. We can just change directory to this directory and then use just push. It's very easy. I just want to explain some details. For example, we need to specify the speculative decoding algorithm.

Here is the Eagle, like this one. Yeah. We need to specify speculative decoding algorithm Eagle. And we also need to specify the draft model path. Because

this one, the model path, this is the target model. And here is the draft model. Sorry. Here is the draft model for the Eagle 3 LLAMA 3 8B.

So one thing that's different about Eagle, all the different Eagle algorithms, is instead of a standard draft target where you're, say, maybe using LLAMA 1B and LLAMA 8B together, Eagle works by pulling in multiple layers of the target model, using that to build a draft model.

So the draft model is kind of derived directly from the target model versus being just a smaller model that you're also running.

Yeah, yeah. And you also need to specify this parameter, the number steps, the Eagle top K and draft verified tokens. For example, the depth of the drafting, if it's 3 and the top K is 1, I think the most number draft tokens should not be more than 4.

That's why we said 4 here. And yeah, you can see more details about this configuration at the SGLang official documentation. And I also will show something about how to turn in these parameters. We have these parameters. I think the model path is fixed.

And how about this one, the number steps, the top K, and the number draft tokens? We can turn in these parameters. And I will show you how to turn in that. So in the SGLang main repo, we have a script.

And

we have a playground. Yeah, we have a batch speculative decoding. OK. So we can just use this script to turn in these three parameters. For example, on a single GPU, when we want to this is the target model, LLAMA 2 7B.

And this is the draft model. Here are some default parameters. The batch size is from 1, 2, 4, 8, 16. And the steps is here, 0, 1, 3, 5, 7. And the top K is here. This is the number draft tokens.

What does that mean? I think it's very easy to understand. For example, we have different combinations of these different parameters. And this script will run all of these combinations. And you will get the result. And from the result, you will get to know that, oh, for example, this combination is best.

For example, at the batch size 8, the three steps, maybe, and the top K is 1, and the number draft tokens is 4. You will get some result about the speed and about the accept rate. Then you can use this parameter for your online servering, for your production servering.

Yeah.

Yeah. And when you're running this benchmark, do we show to set the prompts to things that are representative of your actual workload?

Yeah, yeah, yeah.

Because speculation in any format, including Eagle, is all about guessing future tokens. If you are benchmarking on data that is not representative of your actual inputs and outputs that you're seeing live in production, then you're probably going to end up with the wrong parameters.

Speculation is a very topic and content-dependent optimization.

Yeah, yeah. I think so. So you can also update these promotes. Here, in this batch speculative decoding

Python script, we have some promotes. And I think you can update this just according to your needs.

Yeah.

OK.

So let's take a look at some of the stuff around the community and getting involved.

Yeah, yeah, yeah. Also, I think SGLang currently has become very popular. And if you want to participate in this community and contribute some code, I think.

Yeah. I'll show the slides real quick.

OK.

Yeah. So SGLang does have a really great community. And some quick ways to get involved. You can start out on GitHub, file issues and bug reports as you build. They have a great tagging system of first issues to get involved with, which Yineng is going to show in a second.

But the number one thing you can do is follow SGLang LMSys org on Twitter and then join the Slack to keep an eye out for online and in-person meetups. So this is a link to the community Slack. You can scan that real quick if you want to get involved with SGLang.

These slides are also all in the these slides are all in the repo that you got from the workshop. So you can access this link and stuff later. It's also just Slack.sglang.ai, pretty simple link. So if you are going to get involved and you do want to start contributing to the code base, we can kind of show you some of the stuff.

So at a high level, the code base has the SGLang runtime. It's got a domain-specific front-end language. And it has a set of optimized kernels. You can go actually on this DeepWiki page and get a really good tour of the code base, as well as a tour from this other repository that we have linked, which is also by one of the SGLang people, with some diagrams about exactly how this stuff works.

And then, yeah, Yineng is just going to show a quick overview of the code base on GitHub in case you're interested in getting involved and contributing.

Yeah. I think that the best way to get involved in this project, first, we need to use that. And then you will find some issue or you will find some feature missing in this repo. And then the first thing that is, you can raise a new issue here.

It's loading. Yeah, you can just create a new issue, feature request, something like this. And also, I think, yeah, we have labeled something like good first issue or help wanted.

Yeah, you can see that there are nearly

26. So I think, yeah, if you are interested in this issue, for example, if you are interested in support or servering VLM, VLLM model, you can just start with this. I think good first issue and help wanted issue, yeah, we are welcome the contributions.

And here is the development roadmap. So yeah, if some feature is missing or if some feature you care about, you can find it in the roadmap. I think you can join us for this feature development. Or you can also raise a new issue about this.

And the last one is about the overall work through. OK. So yeah.

In the SGLang repo, we have some component. This one is the SGLang kernel. It's an SGLang kernel library. We implement attention, normalization, activation, GIM, all of them in this kernel library. And if you are familiar with CUDA kernels and if you're interested with kernel programming, you can just contribute this part.

And here is the SGLang router. Last year, we published SGLang

3 version. And we supported the cache-aware routing. If you are interested in this part, you can work on the SGLang router.

Currently, we use SGLang as an LLM inference runtime. So I think the Python part, the SRT, is the core part. We support PDD segregation. We support constrained decoding. We support function calling. Yeah, we support OpenAI compatible server. And we also support a lot of models.

I think if you want to support the custom model, you can just take this as a reference. For example, you can take LLAMA as a reference. I think the popular open source model, the architecture is very, very similar.

So if the model you are interested in has not been implemented in the SGLang, you can just take this as a reference and do some modification. And then we welcome contributions. Yeah. That's all.

Awesome. So if we get the slides back up here.

Yeah. So to wrap it up, first off, thank you so much for coming out. Thank you for bearing with us. Thank you for waiting for web pages to load on this wonderful internet connection that we all have. To kind of wrap things up, I do want to issue a couple of invitations to everyone in this room today.

Number one, we're having a really cool happy hour with the folks from Oxen AI. Oxen AI is a fine-tuning company. Their CEO just had a really cool demo that he published a couple of weeks ago where he took GPT-4.1 and made it do a sequel generation benchmark, took the score, said, OK, I think I can do better than this, took Qwen 0.6B, yes, you heard meright, less than a billion parameters, fine-tuned it on some sequel generation data, and actually beat GPT-4.1 with a model that you can run on like three years ago iPhone.

So yeah, we're going to be at this happy hour. We're going to be talking about fine-tuning and stuff. It's going to be a great time. Second invitation I want to extend to everyone in this room is if you think this stuff is cool, if you are seeing all the stuff that Yineng was talking about around contributing to the code base and you're like, yeah, I love CUDA programming, just come work at Baseten.

If you're bored in your job, you won't be bored here. We've got a lot of open roles for both infrastructure and for model performance. If you're at all interested, just come talk to me. I'm going to be here all three days.

So yeah, that's pretty much our workshop today. Thank you so much for coming through. And happy to take any questions in the remaining time we have. Yes.

What are the main reasons why you use SGLang?

Yeah, that's a great question.

I think that

at Baseten, we use all sorts of different runtimes, model to model. Sometimes you just want to use whatever one is best for your use case. But in general, I think that the reason that we've been really attracted to it is because of how configurable and extensible it is.

Out of the box with basic parameters, you're going to get more or less the same performance from anyone. But if you're able to, number one, have a really deeply and well-documented code base like SGLang, where you're able to really deeply understand all the different options that you have, that can get you a long way.

And then, as we were just talking about, it's super easy to contribute. So we're constantly making fixes and contributing them back. And that means that if you're using a different library, you might be blocked waiting for the core developers to implement support for a model or something.

SGLang, you can unblock yourself. Yes.

When there are multiple vendors and different kinds of applications around the endpoint or within the subnet you are defining, how would you define your

cybersecurity or security protocols? How would you enhance your protocols?

Yeah. I mean, that's a great question. I don't really think that your choice of runtime engine affects that too much because you're just packaging it up in a container. Within Baseten, we've thought a lot about this in a sort of runtime agnostic way, where we're thinking about, of course, least privilege.

We're thinking about making sure that there's a good deal of isolation built into the system. But from a runtime perspective, I don't think there's anything special we have to do for security with SGLang compared to VLM or anything else.

Thank you. So

I'm from the Department of Defense and.

Oh, awesome.

Extensive experience in financial applications. So to do some product development in-house,

do you think I can do the entire product development in-house within a subnet? Don't have to go back and forth, open. Right now, for example, I'm just throwing an example.

Yeah.

One of those

CMMC cybersecurity certifications, I have to go through the endpoint controls and defend the endpoint control and then go connect to the OpenAI ChatGPT.

Gotcha.

Kind of.

Yeah, yeah. So in that case, this would actually help you out a lot. Instead of relying on that remote server, you can just spin up a cluster within the same VPC or within the same physical data center as the workload that's relying on the AI model.

You can clone SGLang. You can take a release and fully inspect the code because it's open source and then fix on that release so that there's nothing changing under the hood. And then, yeah, with that, you'd be able to run the models just directly on the GPU.

As you saw in Yineng's demo when he was doing the CUDA Graph stuff, you're able to call it on even a local host basis and run inference. So yeah, it gives you all the tools you need if you're trying to build even a sort of air-gapped type of system with all of these open source runtimes.

You can pull that code in, inspect it, lock it, and then build off of it.

Very impressive. And also, currently, I'm working on I'm also a PhD student. So I'm working on blockchain-based quantum computing and some kind of AI deliverables. So how do you circumvent within your product? Can you so blockchain is completely another community-based code development.

Yeah.

So how do you can you integrate different community-based or a combination of both high-speed community-based protocol? Or is one is because blockchain is kind of

a decentralized network, whereas this one is kind of within a container.

Yeah. To be perfectly honest, I haven't really experienced anything with that. Pretty much all of the use cases that I've run with SGLang are just traditional client-server applications. Any other questions?

Yeah.

Do you have any kind of inference engine anywhere in Baseten? And if so, for what kind of workloads? And how is SGLang different than VLLM? What parts of SGLang should you shed some light on in the configurability aspect?

Yeah. Great. So yeah, so in Baseten, what we do is we call it the Baseten Inference Stack, where we're taking all of these different

providers, the VLLM, the SGLang, and the TensorRT-LLM, which we actually probably use the most heavily of the three, and taking them in, customizing them, doing all that stuff I'm supposed to say for marketing purposes. But we are customizing it quite a bit.

Anyway, where we generally pick VLLM I'm sorry, I'm talking about them doing your SGLang talk. But where we use VLLM is oftentimes for compatibility. For example, I know our Gemma models that we have up in the library are using VLLM because it's what was supported when it dropped.

So yeah, that's in my mind, the best use case for VLLM is super broad compatibility.

Any other questions?

Awesome. Well, like I said, we're going to be around all day. And I'm going to be at the Baseten booth for the next three days. So if you have any questions about SGLang, model serving, model inference in general, or if you want one of them jobs I was talking about, we are hiring very aggressively.

So definitely stop by the booth, hang out, grab one of these shirts. And yeah, thank you so much for coming.