RunPod

GPU Cloud Deployment Without Leaving Your IDE — Audry Hsu, RunPod

Jun 9, 2026 · 20:19

Audry Hsu of RunPod introduces Flash, a Python SDK that deploys GPU cloud functions from a developer's IDE with a single decorator, eliminating the slow iteration cycle of commits, Docker builds, and server allocation. She demonstrates hot reload, swapping Stable Diffusion XL Turbo for DreamShaper instantly, and a pipeline that chains Qwen 3 for prompt generation, DreamShaper for image rendering, and Nano Banana 2 for photo composition. RunPod's serverless H100 pricing is $0.00116 per second, charged only during active inference. Hsu recommends starting with pods for experimentation and switching to serverless when scaling to hundreds of workers across data centers.

Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod

Jun 7, 2026 · 13:26

Audry Hsu of RunPod presents a cloud AI infrastructure platform that lets developers deploy LLM endpoints in under five minutes. RunPod originated from two failed crypto mining rigs in a basement in 2022; the founders offered the GPUs for free on Reddit in exchange for feedback, and the company now has 500,000 developers and $120 million in annual recurring revenue. The demo shows selecting a model from the Hub, configuring the context window, and deploying a serverless endpoint on H100s. The first request queues for 41 seconds due to cold start (container initialization and model download), while subsequent requests execute in about 1.5 seconds. Users pay only while a worker handles a request, making serverless ideal for bursty or batch workloads with autoscaling up to 15 workers.

MCP Agent Fine tuning Workshop - Ronan McGovern

Jun 3, 2025 · 35:30

Ronan McGovern demonstrates how to fine-tune a Qwen3 model on high-quality reasoning traces from an MCP agent with browser tools. Using Playwright's 25 tools via Model Context Protocol, he generates multi-turn traces by running a 30B Qwen agent on RunPod, saving both tool calls and reasoning content. Those traces are then used to supervised fine-tune a 4B Qwen model with Unsloth, applying LoRA adapters to attention and MLP layers. The process includes converting MCP tool schemas to OpenAI format, extracting Hermes-style tool calls, and unrolling conversations to multiply training examples. Even with only nine curated traces, the fine-tuned model shows improved tool-calling behavior on multi-step tasks like navigating trellis.com to extract specific content.

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

Sep 13, 2024 · 1:40:01

Philip Kiely and Pankaj Gupta of Baseten lead a workshop on TensorRT-LLM, NVIDIA's high-performance inference framework for LLMs, arguing its use delivers best-in-class throughput and latency on NVIDIA GPUs. They explain that TensorRT-LLM optimizes computational graphs via plugin kernels and in-flight batching, achieving 216 tokens per second and 180ms time to first token on Mistral 7B. The workshop demonstrates building an engine for TinyLlama 1.1B, including FP8 quantization that reduced engine size from 2 GB to 1.2 GB with minimal quality loss. They benchmark a deployed model, showing 7,000 total tokens per second at batch size 64 on an A10G. The presenters compare TensorRT-LLM favorably to VLLM for high-throughput production use and introduce Truss, Baseten's open-source packaging tool, alongside their managed platform for automatic scaling and fast cold starts.