Serving Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing - Neil Dwyer, Gabber
Jul 31, 2025 · 16:09
Neil Dwyer, CTO of Gabber, details how his startup serves real-time voice AI for under $1 per hour using open-source Orpheus TTS, LoRAs for emotive voice cloning, and vLLM with FP8 dynamic quantization to achieve 95-105 tokens per second on L40S GPUs. He explains the critical latency challenge of 'head of line silence' in Orpheus (600ms in default voices) and how fine-tuning LoRAs reduces it to ~100ms, fitting within a 1.5-second budget for real-time conversation. Gabber batches multiple LoRA generations per GPU and uses a consistent hash ring for load balancing across servers, enabling popular clones to be dynamically replicated. The talk argues that with current open-source tools, building affordable consumer voice AI is accessible to small teams.
