Continuous Profiling for GPUs — Matthias Loibl, Polar Signals
Jul 22, 2025 · 11:31
Matthias Loibl of Polar Signals explains how continuous profiling for GPUs maximizes GPU efficiency using low-overhead, always-on sampling via eBPF. He contrasts tracing (high cost) with sampled profiling (e.g., 100 Hz, <1% overhead) and details GPU metrics collected from NVIDIA NVMe, including utilization, memory, clock speed, power, temperature, and PCIe throughput. The platform correlates these with CPU stack traces to identify bottlenecks, such as Python and CUDA functions underutilizing the GPU. A new GPU time profiling feature records the duration of CUDA kernel executions, showing actual time spent by functions on the GPU. Deployment runs on Linux with a binary, Docker, or Kubernetes DaemonSet; early adopters like TurboPuffer use it to optimize their vector engine.
