Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)

My name is Rohit Talluri. I am a member of the Foundation Model Training & Inference team at AWS, and we have an amazing talk—super, super awesome founder here talking about voice AI. So, Arjun, why don't you go ahead?

Awesome. Well, thanks for the intro, Roh. Uh, I'm Arjun, I'm one of the co-founders of Cartesia AI. You know, I'll give you a little bit of a spiel about what we do, which is that, you know, we build real-time multimodal intelligence that runs on any device.

Um, so maybe, you know, the topic of this conversation will be focused a little bit more on how do we build voice AI for enterprise. As many of you know, voice AI is, you know, definitely up and coming, and one of the places where, you know, having interactive models is really critical for being able to get really, uh, good experience for your, you know, agents.

So, just a little bit of a recap,right? When you think about foundation models, you know, we often think about, you know, models that are hosted in the cloud, really large. You know, they're doing things kind of in batch mode,right?

You know, you submit a request and then you might get something back. You know, you're not going to get it back 30 seconds later, but you might get it back, you know, in 500, 600 milliseconds, and, you know, that's okay,right?

Typically, for text, you know, you're not also going to be reading, uh, at speeds of, uh, like 200 tokens per second. Uh, and so it's okay if, you know, there's a little bit of a delay and then you get high throughput later.

When you think about interactive applications, like video and voice especially, this is an incredible, visceral experience that you're giving to your users. Speed is of the utmost importance, and quality is just table stakes. So at Cartesia, we're really trying to change the paradigm of how we think about foundation models.

To not just be, like, in the cloud, what you build for batch operations, but actually how you bring it to real-time, have it cover multiple modalities—and we'll be talking more about voice today—and have it run anywhere in the world, not just on the cloud, but on any device.

So, because we're talking a little bit about voice, I want to maybe give a little bit of a prelude why it's so important to think about speed. Imagine you're having a conversation with the person next to you. Uh, and you probably should do that after this talk, but, uh, I think the main thing is that it would feel really awkward,right, if you were trying to talk to someone and they responded a second later.

So, in voice, you don't have seconds to actually give your response back, you have milliseconds. And when you think about voice agents, where you might be calling, you know, customer support on the line trying to triage your problem, you're going to get pretty annoyed if it's not responding to you and giving you accurate results as soon as possible.

So, when you think about voice, you have to deal with things like interruptions, you have to think about globalization, these things relate to accents,right, background noises that you might be calling from. And at the end of the day, when you actually want to deliver this experience, it's incredibly subjective.

So, there's so many nuances and customizations that you want around voice.

So, what we've done at Cartesia is really thought about building solutions for voice AI on the modeling side from first principles. We care about three things. The first is quality. The naturalness of the voice must be exquisite and something that is just table stakes for the kinds of experiences that you want to be able to deliver.

The second is latency. You want to be able to hear the first bite or the first sound of audio on the other line as soon as possible. This gives your end-to-end agent a lot more time to be able to do things like reason more,right, and have a little bit more slack in the end-to-end system.

And third, arguably most important, is controllability. The experience that you want to exude by having an agent customized for the things that you want to do is a reflection of your brand. You want the agent to be able to talk about what your company is or what you're trying to sell in the way that you would.

So, being able to customize the voice AI to be able to do this is critical and paramount to being able to deliver a great experience to your user. So, what we've done is we've pioneered a new architecture called state-space models.

These are an alternative to transformers, and the main takeaway here is that typically transformers scale quadratically, which means the longer your inputs get, you get quadratic scaling in your memory as well as in your, uh, in your runtime.

So, you know, the longer your inputs, the slower your models get. With state-space models, or SSMs, generation at inference time is O of 1. We maintain a state that you can generate from, and this means that you have perfect, very low latency, things that you can't achieve with traditional transformer architectures.

What we've pioneered is that the state-space models typically have been, you know, lower performing in the recurrent sense compared to transformers, but we've closed that gap. And our models actually perform not just better from a latency perspective, but also from a quality perspective.

So, I can give a little bit of a spiel about, you know, why this matters, but I think we're more interested in having more of a conversation.

Yeah.

Around voice AI. So, Roh, why don't you kick us off?

Definitely. So, just so you all know, this was supposed to be a fireside chat.

Yeah.

We had some chairs and asked some good questions. So, we're going to open it up to the audience in a little bit. I do have a few questions for you, Arjun. Talked a little bit about latency. We hear voice AI challenges like quality and, let's say, the speed of the models, as mentioned, the way that they're hosted on the edge, etc.

What are the challenges that, like, your customers are facing? Why are you building for this?

Yeah. So, uh, I think the main thing is that, you know, when you do voice AI, and, you know, the main model that we have is Sonic 2, which is focused on voice generation. That's one part of the puzzle.

You have to actually try to build voice agents, you have to hook it up with the LM, you have to hook it up with your speech-to-text model. And the biggest issue, honestly, is that there's not enough time. You need your LM to actually have a lot of time.

You know, these models aren't typically built for, you know, high, uh, or low latency workflows, and so you want to actually give it the most slack. So, latency is, you know, of utmost importance there. And I can talk a little bit about, you know, what we've done to create the fastest model in the world for text-to-speech.

Uh, the second, I'd say, is controllability. So, you know, one thing that we've noticed so much is people use our platform because you're able to get amazing quality around things like voice cloning, accents, being able to capture background noises just so natively in the generations that you want.

You know, people think it's like a little bit of an uncanny valley when you're talking to an agent and it just sounds perfect. They like the little phone noises in the background, little beep-boops,right? You know, that's kind of what you like to hear,right, or what you expect,right, when you're on a phone call.

Uh, so I think those are the two main things, and I'd like to say that, you know, we've done a really amazing job of getting nailing those down.

Yeah. And you mentioned something interesting there, a couple of the use cases,right? Voice cloning.

Right.

The control of the voices, etc.

Yeah.

Can you talk a little bit about the customer use cases you're seeing today of voice AI?

Yeah. So, you know, voice AI, honestly, has penetrated so much, so many markets out here. We look at healthcare, customer support. Uh, a lot of this is actually going into real-time gaming,right? When you're dealing with non-player characters, you want them to be dynamic, interact, you know, with the, uh, players of the game.

Um, these are, you know, three of many, many markets that, you know, voice AI has started to grow in. Uh, and, you know, I think what's really exciting is that because it's growing so fast, you know, it's just so great to have partners like AWS who are also, you know, investing in that space and being like, look, you know, these are things that we need to support naturally, um, and, you know, people want to use our platform for.

Yeah. What about human narration,right? I think we hear a lot about, uh, like voice AI taking over different types of industries or these use cases as mentioned.

Yeah.

Do you see a place for human narrators in the future as well?

Definitely, yeah. You know, creators are a huge part of the voice AI platform. I think one thing that I think we do a really good job of is we actually have, like, a voice marketplace for creators,right? You know, why is this so important?

You know, our goal is not to, you know, replace voice actors, actually.

Yeah.

That's not what I want to do at all.

Yeah.

I think my main goal is, like, how do you give them a platform so that their, you know, their, um, uh, I forgot what, like, their essence,right, you know, who they are, their personality can actually be exuded and, like, you know, licensed by other people that want to use the platform.

Um, and so, you know, this actually gives, uh, you know, we've had a lot of voice actors onboard on our platform, you know, great way of amplifying them.

Yeah.

Uh, and yeah, a lot of use cases are honestly focused on narration as well.

Got it. So, I have a few more questions. Maybe we'll end with them. Does anybody have questions in the crowd? Anything about voice AI?

Yeah. Question. I fear this might be a dumb question, but I'm going to ask it.

No, please.

I've just started using Cartesia. It's like it was a game changer.

Awesome.

Part of the, uh, podcast framework.

Yeah, yeah.

Um, because there's an Amazon partnership here, I was just wondering, like, Claude doesn't seem to work so well because of the latency.

Okay.

Is that, is that normal? What are the models that you usually pair with, and is there, is there plans for.

Can you repeat the question on your end, just so that.

Sure, sure.

The logic.

Yeah, yeah. So, uh, the question was, you know, um, were you Cartesia is integrated with PipeCat. You know, PipeCat, Cartesia, we both, you know, work with AWS. Uh, but Claude, you know, still has, you know, pretty high latency when you're trying to, you know, do things end-to-end.

Uh, so, you know, this is, you know, a huge, huge problem,right? This is why, you know, the slack that we give you on the TTS side is something, you know, you can account for on the LM side. Uh, I'll let Roh, you know, take a stab at that question, but I know that, like, you know, Claude and, you know, other, um, other models, like, you know, have different, you know, mechanisms for, you know, you can have, like, a dedicated instance that's running so you can get, like, you know, better latency numbers.

Um, you know, but a lot of this is actually just, you know, it's, it's up to, you know, the LM providers often,right, to make these optimizations. I think one thing that, you know, we're really excited about is our goal is to, you know, make real-time AI pervasive,right?

Uh, and so, you know, we're really excited about if there's, like, certain applications that, you know, you're really after, you know, how can we enable you to do that. So, happy to chat, you know, after this talk. Yeah.

Yeah, of course. And I'll add a couple things to that too. What's really amazing about Cartesia and their background is the development of custom model architectures, specifically for this use case, which is voice AI. And we spoke a little bit about SSM and linear scaling versus quadratic scaling.

One thing from the AWS side and what I think is really amazing about our design philosophy for our generative AI ecosystem is we want customers to have optionality in this, in our platform. And if you look at AWS's model, model gardens, we have SageMaker Jumpstart and Amazon Bedrock.

These model gardens will have a host of different types of models for specific types of use cases. And we're on the lookout for the next foundation model provider, one like Cartesia, that we can bring into our ecosystem and unlock downstream industries that might be underserved by the existing foundation models today.

So, that's part of our strategy, part of our philosophy. And I think with, and I'll say it again, with Cartesia, we're really unlocking the voice AI. We need real-time AI. We need, we need the ability to host on edge devices.

And, uh, we're unlocking a lot of customers with their models. So, hopefully that answers a little bit.

A lot.

A little bit. Yeah, we can chat a little bit more after. Yeah. Yeah.

Okay. So, I have some friends that, uh, that are pretty into video AI research, and something they're kind of telling me is that in video AI, one prevailing theory is that, you know, you don't need so much scale of the actual video data, but rather the density of information per frame of that video is actually more important.

I'm curious if there's something similar in voice data, whether you believe that, you know, to achieve, you know, next-level models, um, you need, like, a, like, the next scale of data, or is it more about, like, the quality and density of that data?

Yeah, that's a good question. So, I think the question, uh, was more around the lines of, like, you know, how does voice AI data, you know, compare to, you know, some of the other multimodal data sets in video?

You know, one of the prevailing theories, uh, was that, um, you know, you don't, um, actually need a lot of data, you just need it to be very rich,right? And, you know, if that's the same in voice AI.

Uh, I think the short answer to that is, uh, yes, but also no. Um, I think, like, when you think about scaling for a lot of models, you know, the generative, uh, you know, in generative AI,right, the general philosophy is like, oh, you know, large-scale pre-training data, and then you have some kind of, like, alignment data or, like, preference data that, you know, you fine-tune on typically.

Um, I think that holds true for, uh, a lot of other modalities as well. It actually holds true for video as well. So, my background, uh, I guess I didn't say, was, like, you know, I did my PhD, uh, in generative AI, uh, at Stanford.

I was working a lot on, like, image video models there as well. Um, but in audio, it's actually quite interesting because, you know, what people want from preference data is actually so diverse that it's like, you know, you can't capture it purely by, like, a one, one-stage fine-tuning step.

So, yeah, you definitely need very rich data. You know, you need high-quality data, of course, but you also need to pair it, uh, with information that, like, you know, many, many different kind of people will want. So, uh, you know, this is how traditional LMs are trained.

I think a lot of the, like, you know, large open-source, you know, uh, image generation, video generation models also are trained this way. Uh, but honestly, it just really depends on what you're trying to, trying to get after.

But, yeah. Sorry, I don't know. I know that's not a satisfying answer, but yeah.

Yeah.

What's your take on the speech-to-speech models? I know Amazon has one. It seems immature, I would say.

Yeah.

Um, but is that the future, or, like, what's happening?

Yeah, that's a, it's a great question. Um, you know, I think there are places where, like, speech-to-speech is valuableright now, but I don't think it's at the point where we can actually use it for, you know, production or enterprise-grade, uh, use cases.

Um, and, you know, it's great. You know, I think, you know, uh, Amazon released, uh, the speech-to-speech model recently. Um, you know, I think there's a few others that are out there as well. Uh, but I think it's still pretty clear that, like, orchestrated solutions are places that you get a lot of controllability around, like, how you want the different pieces of your system to operate.

Um, and, you know, from a latency perspective, I think speech-to-speech models will, of course, you know, dominate over time. But, uh, I think that level of controllability is something that needs to be thought of from first principles when building these systems,right?

The goal of this system is not to build something really cute. It's to build something that actually functions for real-world use cases. And we might not be there yet, but I'm sure, you know, we'll get there over time.

Yeah.

Yeah.

What do you think about local models and the climate of it for the future?

100%. Yeah. Um, so at Cartesia, like I said, you know, we build for any device. So, uh, we have models that run locally on edge devices. Um, and, you know, I think it's actually really important because, you know, I think cloud models will always be there,right?

You know, there's certain size models, certain capabilities, like, you know, that you get at, like, certain sizes that you can't just, you know, get, you know, fit on, like, I don't know, very, very edge devices,right? Like, not even, like, laptops, but, like, if you get to, like, smaller phones and things like that.

Um, but there's a lot of applications where, you know, edge devices are just critical,right? Uh, I think the question becomes, like, where is a network latency plus cloud speed actually slower than edge? And we've broken that. So, running our models on edge is about five times faster than if you were to round trip.

Um, and so you get, like, just, it's like unparalleled latency that you get. Um, and I think that's quite exciting.

So, we're almost up at time. We'll maybe one more question.

Sure. Yeah.

Yeah. Um, so, what tools do you recommend to monitor, basically, the agent,right? Because you're going to be the one that's going to, we're going to call you.

Yeah.

Something's broken. It said, "Pause three seconds."

Yeah.

That's what's say, "Pause three seconds."

Right.

So, it can be, like, speech-to-text, it can be the LLM, it can be the text-to-speech.

Yeah.

It can be, I don't know, something wrong with AWS, but, like, how do you, how do you even tell us, like, "Hey, it's not us, it's not a regression issue"? How do you, how do you, uh, do that?

Yeah, yeah, that's a good question. Evals are an incredibly important part of any system that you build. Um, how do we tell you that? Uh, I guess you can message us and ask us if it's a problem. Uh, but I think what we've found is, you know, typically it's at the LM stage that we see a lot of these issues.

Um, you know, you end up, like, you know, oftentimes there's, like, some level of formatting that you need to do for outputs that come from speech-to-text to make it, like, viable for your LM and the outputs for your LM to make it, like, you know, viable or ergonomic for your TTS to handle.

Uh, but we've actually handled a lot of those edge cases for you. So, over time, you know, with Sonic 2, uh, you know, I'm showing a slide here that's, like, you know, showing most on the latency side, but a lot of the capabilities and quality has also improved substantially.

So, not only did we get our models to be, you know, two and a half times as fast as our initial models, so we can get served at, like, 40 millisecond model latency, but a lot of those, you know, edge case issues that LMs often run into, we know that you don't want to deal with them, so we deal with them, we deal with them for you.

To making our system more robust.

Got it. So, one more from me. Like, it's 2025 now. Think five years in the future, about 2030.

Yeah.

If you look back at what you've done over the, those past five years, where is Cartesia at, where is voice AI at, what do you see?

Yeah. Um, you know, I think voice AI will be just, like, the de facto norm,right? Uh, I think there in every industry that you see, you're going to have some interaction with voice AI, whether it's on the triaging side, the full end-to-end, you know, support side, uh, you know, gaming,right?

All those interactions will, will definitely be, you know, a large part, you know, covered by voice AI. Uh, but I don't think this is the only thing,right? When you think about true interactive models, it's more than just what you can hear,right?

It's about how you experience the world around you,right? So, you know, a lot of these people call it, like, world models,right? But I think actually getting these to work in real time, I think will be really exciting. And I'm, I'm really looking forward to, like, a world where, you know, we can actually have these kinds of systems work with us as, you know, assistants or co-pilots or, you know, whatever, to actually help us understand the world in ways that we couldn't imagine originally.

Amazing.

Yeah.

Well, thank you. I think that's all we had today, and really appreciate it.

Yeah.

And thank you for sharing a little bit about that.

Yeah, appreciate it. Thanks, guys.