Serving Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing - Neil Dwyer, Gabber

I'm Neil. That's— that's Jack in the front there, but it's just me. So yeah, we're really just going to talk about our experience hosting Orpheus inference for our real-time stack. So I'm Neil. I'm the CTO at a company called Gabber, it's a small startup, but I spent a lot of my career doing real-time media stuff, so sending audio and video around the internet.

Started at a company called Bebo, which was ultimately acquired by Amazon, but there I was doing a lot of— we did like a game streaming app, kind of like OBS, built a lot of the streaming infrastructure there, built a ML pipeline to watch people play video games, so they would watch people play Fortnite and put some cool effects on the screen when they got a kill or a victory or something.

So I spent a lot of time in like the G Streamer trenches and with WebRTC and RTMP and all that stuff. Took a detour, worked at Uber for a couple years, then left that, did a multiplayer gaming startup with my brother Jack here.

So doing— basically trying to bring like AAA-style multiplayer to web games, so a lot of real— and with voice and stuff too, so there's a lot of real-time media slash real-time, you know, simulation kind of stuff there. And then, yeah, we didn't do a super good job there and shut that company down.

And we were using LiveKit, I made a LiveKit SDK, and that segue to me working at LiveKit. I think a lot of people have probably heard of LiveKit in this room. And yeah, the second half of my time at LiveKit, I was spent doing the LiveKit Agents platform.

So that's like the platform that was kind of born out of LiveKit's involvement with GPT-voice. So yeah, wrote the first line of code on that and worked on that. And then, yeah, left LiveKit and did another startup with my brother Gabber.

So that's what we're doing now. So Gabber is real-time inference for real-time, basically AI personas. So, you know, we have some core building blocks like voice, memory, video inputs coming soon, tool calling, kind of like the usual suspects, I guess.

But our focus is really on the consumer apps. You know, we see the inter— like the replacing human use cases pretty often, like the call center use cases, customer support, AI, SDR, that kind of stuff. But our interest is really in the consumer space.

We think these kind of like real-time synchronous AI experiences are going to be as ubiquitous as websites and apps in the next kind of like two to five years. So that's our focus, and we— that's how we try and differentiate in terms of opinion into our product and our SDKs and APIs and stuff.

Here are some of the use cases we're seeing. These are also kind of like the usual suspects. AI Girlfriends was the first one. That is like— I'll get to why that's the first one, I guess, but other ones are like AI NPCs, AI therapists, AI personal trainers, AI toys for kids.

I think that you saw that in a couple— a couple sessions ago. These use cases— like we're seeing a lot of different use cases, and I saw it at LiveKit too, and it got me really, really excited about this stuff.

But AI Girlfriends was the first one, mainly because everything's so expensive.

Some of these voice platforms, it's, you know, end-to-end, upwards of $5 an hour, and that doesn't really work for like 90% of the consumer apps. But AI Girlfriends, it works because like the users are paying like— it's like usually like a credit system, like you buy credits and you use the app and it uses credit.

So they're more comfortable with that kind of spend, but most consumer use cases, they need something pretty close to free. So we knew that— and at the time we were not hosting any voice models, but we knew we had to.

We knew that the only way to really get this to execute on our vision of putting these experiences everywhere, we had to start bringing more things in-house and running on our own GPUs. So at the time, open source, there weren't a lot of good open source voice models.

There were a lot of good ones for asynchronous use cases, so generating voice slower than real-time, but there weren't any really good like real-time streaming ones until Orpheus. Orpheus was the first really good one that was kind of like ready to go.

So Orpheus came out and we're like, "Okay, this is our time to shine." We immediately like put it on an H100, hosted it, went viral with Jack's tweet, and got a ton of top of funnel. And yeah, that was kind of like the starting point.

It's like our company— there's like before Orpheus and after Orpheus, our company kind of changed. So a little background on what Orpheus is. It's a voice model, but it started as a LLaMA 3 billion. It was trained on— pre-trained on like 100,000 hours of voice data and text data as well to make sure it kept its understanding of kind of like language.

And it was trained to output audio tokens. They're called SNAC tokens, so that's another open source project, SNAC, which is an audio codec.

So it's trained to output the 24 kilohertz version of SNAC tokens. Those SNAC tokens are then decoded, and then you get audio. You get 24 kilohertz audio. Important thing to note here is it's about 85 SNAC tokens for one second of audio.

So Orpheus, wherever you're hosting it, it has to be a throughput of about 85. I mean, you want like 90 to 100 tokens per second to keep up with real-time. Otherwise, you get gaps, obviously, in the audio, and it sounds bad.

Other things that were important to us because we're going up to the consumer use cases was cloning. So our clones need to be emotive and high fidelity. And one-shot cloning doesn't work that well. That's more true for Orpheus because it only had 100,000 hours of pre-trained data, whereas I think some of the zero-shot emergent behavior comes at like a million plus hours.

So, and we're scrappy, I think you can tell by like our design here, that we're like pretty scrappy,right? We weren't going to fill that gap.

So we went with low-rank fine-tunes for our clones. So here's an example. So this is a low-rank fine-tune. We have some better ones. This isn't like the best example, but they're customers, so I didn't want to put it in the thing here.

So we just cloned Jack's voice like yesterday and used 16-rank, alpha 32, basically all the projections. Here's the source audio. Let's see if it— let me restart it.

We forgot to pick up our child from school. Ugh. Ugh. The school called me in the middle of a meeting. You.

Oops. So that's the source, and then here's the result of a fine-tune. So let me manage expectations here. This was like pretty bad data, like 10 minutes of data. You really want like 30 minutes. So I had to overfit.

So it trained on like five epics. It's pretty overfit, but you'll see it like still sounds okay.

Hey, how are you? I'm kind of sick. This is a longer generation. Let's see if it sounds okay.

Yeah. So it's not bad. You know, I spent my whole life, or most of my— I'm the older brother, so most of my life, so I know his voice very well. So it's jarring to me, but the cool thing is like, yeah, it's trained to do these tokens, which is important for consumer.

And it's pretty emotive. Like when it said "I'm kind of sick," it sounded pretty sad, so it picks up on the language cues as well.

Other thing that's really important, obviously, for all voice use cases, not just consumer, is latency. So there's four things that really affect latency. Time to first token is one of them. Tokens per second is one of them.

I'll get into why that is later. But what we found in network latency is another one. But we found the most biggest cause of latency was what we're calling Head of Line Silence. This is somewhat specific to the Orpheus model, so this isn't going to be true for all models.

But Head of Line Silence is basically that somewhere in the fine-tune of Orpheus, the data had a lot of silence at the beginning because it was voice actors that they hired, and they trained— and they like took those scripts and fine-tuned a model from it.

So this is like the default Orpheus voice, or one of the ones that came with it, called Tara. And it has 600 milliseconds of latency at the beginning. And they probably had other good reasons for like adding silence at the beginning, but this is a lot,right?

So 600 milliseconds of silence. We actually found that, oh, so 600 milliseconds of silence. We're running on L40S machines as of now. They can do about 100 tokens a second. So 600 milliseconds is almost half a second of silence.

So even we are filtering out the silence. Like we're not just playing that audio back to the user, but because it takes a while to generate those tokens, we're adding like basically half a second of latency just on wasted compute, pretty much.

So yeah, even filtering out the silence, you're only like saving 10% there because you're just barely faster than real-time. We're scrappy again, so we're running on L40s. But what we found was interesting is that we could actually just fine-tune the silence away.

So this is an example of a clone that we did, a LoRA fine-tune of a customer's clone, and the latency is basically like 100 milliseconds, like p50. So much better, like half a second basically for free. And that matters because these real-time— you kind of have a latency budget on the real-time application.

So the way these work is, you know, the human talks, and then at some point you decide, is the human done talking? Those models are not perfect, so you typically add like a snooze period at the end of that.

But during that snooze period, you can still do work. So what we do is we kick off the LLM. The way we have our Orpheus stack set up is we start generating audio after two sentences, or if it's done, but two sentences typically, which gives it enough context to like capture the emotions.

So all that to say is if we generate the first audio packet within that snooze period, then we're kind of like in the money on latency, in our latency budget. Now these endpointing models are going to get better, so, you know, that snooze period is going to go down to like half a second to a second is probably like the sweet spot.

But one and a half seconds is kind of the threshold, I think, for anything above that sounds pretty bad, and anything kind of equal to or below that is like acceptable. So yeah, that half a second mattered a lot because it gives our LLM more time to create tokens, and because we're letting customers bring their own LLMs, it's somewhat out of our control.

So the next big category here is infrastructure. Again, we're scrappy, so we really needed something that was robust and not too complicated. And we needed batch inference. So we needed batch inference, obviously, to save money, so we need to run multiple generations in the same batch or on the same GPU concurrently.

And we also needed multiple LoRAs to be running in the same batch on the same GPU. And we wanted one load balancer in front of everything. We're spinning up multiple different models for different languages. So we all wanted this to sort of be like a black box that just sort of worked.

So vLLM to the rescue, it supports all those things. So vLLM can do batch inference with LoRAs, which is really, really awesome. Unfortunately, the FP16 model was slower than real-time on L40s. It worked on an H100, but it was slower than real-time.

But again, vLLM to the rescue, they support FP8 dynamic quantization, which requires basically zero work. It just works automatically. It does all the scaling and everything automatically, so you don't have to like train the calibration data into your own quant.

It just works, and it's amazing. So that brought us up to 105 tokens a second on the non-fine-tuned voices and 95 tokens a second on the LoRA voices with a batch of 10, which were, yeah, well in the money in terms of margins and things like that, so that's nice.

Part of the infrastructure is, of course, load balancing. So, you know, LoRAs are, depending on what your hyperparameters are, they're between 100 and 200 megabytes. So you want to make sure you end up on a server that has a LoRA in memory and things like that.

We also wanted to support, so that's where like sticky session comes in here.

And yeah, latency low, I guess. But we also wanted to support streaming input, mainly because the LLM often, you know, might not be done by the time you want to start producing audio. But we also wanted to support arbitrarily long generation, so like storytelling, things like that.

So we have, so that's another reason why it, I guess, this load balancing problem is interesting because you want to make sure you end up on the same GPU across the whole session. So we went with pretty much like a by-the-book consistent hash ring setup.

So if you've seen hash rings before, this is not that interesting, but basically the way it works is you hash the servers multiple times, so you want it called virtual nodes, so it distributes around this hash ring. And then when a LoRA, you know, generation starts, you hash that with the same hashing algorithm, you pick the nearest server to that, and it just works.

And the reason this is chosen is because you can like remove a server, and it doesn't reload balance like everything. It's just only a few, I guess, migrations that are needed. The other nice thing about this strategy is if a clone gets very popular, it's pretty easy to handle that.

You can just append to the LoRA, so you can just, the more popular a LoRA is, you can just add it to more servers and upscale and downscale that very elegantly without really a ton of engineering work.

So yeah, at the high level, it looks something like this. We have our WebRTC backend that kind of like terminates the client connections. Then we use WebSockets to our GPUs, and then the GPUs are talking to Redis. Redis is not the best choice, but if we scale beyond needing Redis for this kind of thing, we can just solve that with piles of money, I guess.

But yeah, the way it works here is you start a session. The WebRTC backend just connects to any GPU. Then it asks Redis, "Hey, what GPU is this request supposed to be on?" And then it just proxies it with another TCP connection to the correct GPU, which is fine because these GPUs are in the same data center, private networking.

So low latency TCP, that's totally fine within the same network.

So that's pretty much it. I mean, the conclusion here is, you know, we're pretty scrappy, and we were able to host voice models on GPUs and handle that infrastructure, so you can too. Open source is there, and yeah, I think it's going to unlock a ton of cool use cases.

Shout out, shout out Swix. He's a supporter of ours. And obviously, put this on or half of it, I guess. But Swix is awesome. We love him. Canopy Labs, who created Orpheus, haven't met them, would love to if they're here.

And then just free open source software in general, like Canopy Labs is built on Llama and Snack, so this whole ecosystem is greater than some of its parts, I guess. And LiveKit, we're LiveKit alums, so love those guys.

And our WebRTC infra is built on them. And then vLLM, notable open source project. And yeah, that's it.