RunPod

Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

Jul 26, 2025 · 43:42

Philip Kiely and Yineng Zhang introduce SGLang, an open-source fast serving framework for LLMs and VLMs, arguing it offers production-ready performance with day-zero support for new models like DeepSeek and Qwen, and a customizable codebase for contributions. Yineng, a core maintainer, traces SGLang's rapid growth from a December 2023 paper to nearly 15,000 GitHub stars and adoption by xAI, AMD, and Meituan. The workshop demonstrates deploying a first model via Baseten's trunk packaging, then tuning the CUDA Graph max batch size flag on an L4 GPU to maintain CUDA Graph-enabled decoding during higher concurrency, boosting generation throughput. They also cover Eagle 3 speculative decoding, where a draft model derived from the target model speculates tokens; users can benchmark different step and top-k configurations on representative prompts to find optimal settings for production. Finally, they invite contributions via GitHub's 'good first issue' tags and highlight Baseten's job openings.

Your Personal Open-Source Humanoid Robot for $8,999 — JX Mo, K-Scale Labs

Jul 2, 2025 · 19:26

Jingxiang Mo, founding engineer at K-Scale Labs, introduces their open-source humanoid robots: the 5-foot K-Bot (pre-order for $8,999, delivered by October) and the 1.5-foot Z-Bot, both fully open-source from hardware to ML models. The K-Bot uses MIT CHiTA actuators, provides up to 250 TOPS compute, runs an RL-based whole-body controller trained with MJX in 1-2 hours, and offers a Python/Rust SDK (pip install kos) with a digital twin simulator for rapid development. Mo emphasizes modularity—swappable end-effectors and upgradable heads—and positions K-Scale as the first US consumer humanoid robotics company, targeting developers and households. The robots are significantly cheaper than competitors (Tesla Optimus at ~60K, Unitree at 40K) and include VR teleoperation, OTA software updates, and a bimonthly hackathon community.