Giving a Voice to AI Agents: Scott Stephenson, CEO, Deepgram

Uh, hey everybody. If you don't know about Deepgram, Deepgram is a company that does, uh, audio AI. So we've been around for 9 years now, um, so ancient in AI years. But we brought end-to-end deep learning to speech recognition, and our most recent product is a TTS product that we released back in December.

And so you can use our API on-prem, you can use it in the cloud, you can adapt models to a specific acoustic environment. I just came from a drive-through that has all sorts of crazy things happening in it, like cars running and Amazon trucks backing up and all of that, and we, like, automatically adapt models for that kind of thing, etc.

So anyway, if you don't know about us, that's what we do. Also, we're a research-led company, so what that means is we figure out fundamental things through our research team, and then we bring them to the market as soon as possible through our product team.

So, yeah, it's everything is moving at breakneck speed, but a lot of the things that folks enjoy now with, like, really fast, accurate real-time speech recognition. If you have any devices out there, I was just talking with a founder a couple minutes ago, most of them use Deepgram behind the scenes.

So just, that's who we are. And we help companies build truly conversational experiences. And the bar that we set for ourselves, and we set from the beginning, but it takes a long time to get there, is, what can a human do?

And we ask this all the time in our product meetings, etc., and our research team. And that helps guide us toward what's possible. Because neural networks really do work in a way that is similar to a human. They can learn by example, etc.

And so you have to think about how can we formulate this problem in a way that this machine can actually learn from it. And so we spend all of our time thinking that way. And so arguments about, like, how many neurons does it have, is it equivalent to a mouse or a cat or a person or whatever, we're talking about that kind of thing all the time.

And it's amazing to see the progress that has happened over the last decade, and we're at an amazing timeright now. So one thing to keep in mind is that there was, like, a previous version of voice AI, and it was kind of, you know, voice AI 1.0.

Think Siri, think things like that, where it was slow, it wasn't super accurate, there wasn't really a real-time feel to it. And you could try to ask it anything, but it wouldn't really, it'd only answer in a very specific domain.

So now with the next version of voice AI, there's, it's open-ended, and a lot of what's driving that is LLMs. You can put any text into it, and depending on how the model is trained, you can output any text.

And so if you're having a voice AI conversation, you can take the audio, turn it, using speech-to-text software, turn it into text, inject it into an LLM model, get text out the other side, and then use a text-to-speech model in order to produce audio.

And part of this now, voice AI 2.0, is how fast can you do that, in how smart of a way can you do that so that the agent actually, like, does what it's supposed to, and then how expressive does it sound.

And so we'll talk about that a little bit, but, you know, this is the type of process, speech, like, speech-to-text to an LLM. I would just internally call that text-to-text. Who cares if it's a transformer model or whatever it is, but you transform speech-to-text, then text-to-text, then text-to-speech, and you keep that loop going.

And I would say what we're seeingright now, and we might, you know, the folks out there might say there's a big hype cycle, we're at the peak of it or something. I would say, hang on a second, this is more like 1910, you see the first cars in the street or something like that.

Really, it's just a tiny thing that's happeningright now, and there's going to be a massive 100x explosion in the next, like, decade for voice AI, text, etc. AI is going to be everywhere. The way I would think about this, and, you know, encourage others to think about it this way, is there was, like, an agricultural revolution that took like 1,000, 2,000 years to happen.

Then there was an industrial revolution that took like maybe 250, 300 years to happen. Then there was an information revolution that took like 75 years to happen. See the trend? The intelligence revolution is going to take like 25, maybe 30 years to happen.

And so if you thought tech companies were fast before, AI companies have to move three times faster. So anyway, this is how it works today. There's a speech-to-text system. You can see in there, there's an LLM or a text-to-text system, and then a text-to-speech system.

And each of these systems works fairly independently, but the state-of-the-art works very quickly, and they have high efficacy or accuracy. So speech-to-text now, from even just five or, you know, eight years ago, it used to be maybe 75% accuracy or somewhere around there.

Now it's like over 90, and that over 90 really makes a big difference. Also, speech-to-text used to be maybe two to five-second delay for real-time. Now it's like 100 milliseconds or maybe 200 milliseconds with the high accuracy. And also, you can run it on-prem, and you can co-locate all these services together.

Actually, the founder of Daily, I just saw him roaming around, they just came out with a blog post showing that you can do the entire voice AI round-trip conversation in less than 500 milliseconds using Deepgram for the speech-to-text, Llama for the TTT, and then Deepgram for the text-to-speech.

But nevertheless, that's what a human responds, and it's between like 400 to 600 milliseconds in turn-taking. So you can do all that here, but there is a piece that's missing, which is if you, these are all, like I just said, speech-to-text models, text-to-text models, text-to-speech, they're not passing along any context throughout the conversation.

So what this ends up with is a few spots that you're like, hang on a second, it's not really getting exactly what I'm saying. And maybe that only happens 10% of the interactions or 20% of the interactions. Or what I really mean is like 10 or 20% of the turns.

But the way that you combat that is by adding in context. And so that's what this view isright here. It looks like a really subtle change, but instead of there being speech-to-text models now, where it just takes in audio and it puts out text, it will take in audio and context.

So think promptable, but not necessarily text promptable. It could be promptable with anything. It could be promptable with other audio, it could be promptable with images, it could be promptable with documents, it could be promptable with the previous turn of the conversation.

But what that gives you is, it gives that speech-to-text model context. When you send something to a speech-to-text modelright now, it actually has to, it's kind of amazing what it's able to do. It knows nothing about the conversation, and then it's just thrown into, like, a basketball game and has to transcribe everything, you know, quickly and with just, like, a few seconds of context and then do a really good job.

What happens when you give that model the entire context of the conversation up until that point? It gets way more accurate. But it's not just about the accuracy, because the next step in that is once you can pass that context along, you can pass the original input context along, but you can also have your speech-to-text model output context as well.

And that can output text that is human-readable, it can output audio, it can output images, it can output just embeddings, which people are now familiar with as, like, just a vector embedding. And so that can carry the state of the conversation throughout the entire thing.

I just want to point out, this is not how systems are builtright now, but in the next year, this is how they're going to be built. And this is when things are going to flip into, like, holy shit, this feels like human.

And because once that text-to-text is contextual from the audio, it knows it's hearing an angry person or a happy person, it knows that the conversation is flowing quickly or slowly, it knows that there's, you know, light music playing in the background, it knows all this stuff,right?

And so that text-to-text model can now generate the appropriate response. But it's not just about the text that it generates, it generates its own context,right? So it can say to the text-to-speech model, hey, I need you to say this softly, I need you to say it slowly.

You know, I'm speaking very quicklyright now,right? But I need you to say it slowly, I need you to say it in an authoritative tone. You know, that kind of thing. And then when the text-to-speech model generates that audio, it will say, I tried to generate it this way, I sounded like this, I think I did a good job, etc.

That's context that's going to get passed to the next turn in the conversation. And so all this context is going to be passed around. We call it contextual AI internally at Deepgram. But this is what the next generation of these models is going to look like, and this is actually the innovation that's going to make it feel like a human.

Because the speed part is taken care of, the accuracy part is taken care of, now it's all about context. I know there's, to preempt any follow-up questions about, like, a multimodal model or a speech-to-speech model, sure, absolutely. You may mold or, like, meld some of these together, you may put them all together.

The problem with putting them all together is it's not as controllable. So we are the largest speech-to-text API in the world now, but that's mostly because of businesses using us to power, like, Spotify, to power food ordering, to power call centers, and that kind of thing.

And they need controllability in what they're doing. So if you just give, like, an open-ended prompt to a speech-to-speech model and just say, like, go to town, that's not the kind of experience that, like, a bank wants. You know, they want a little more control.

They want to maybe put a whole bunch of compute power in the speech-to-text to make sure that they get everything preciselyright, and then the text-to-text doesn't actually have to be all that big because they're just doing a few, they're just doing a few things, like helping them reset their password or something like that.

And then they want the text-to-speech to be, like, really expressive, but, like, calm and only a single voice. And so these are all going to be kind of compartmentalized, because that brings us to the COGS conversation, the cost of goods sold.

Right now, a lot of people probably feel that, you know, AI is kind of expensive, but it doesn't have to be if you use theright tools and you use theright services that focus on cost of goods sold. And so, and if you choose theright size for each component in the stack.

So in the future, that's what it's going to look like. I'm trying to hurry through this because I want to leave room for questions. It's already been, like, 14 minutes, or maybe 12 minutes or so. But I'll give you just a flash of, like, what the future will look like.

But one thing that we're doing as, you know, a platform at Deepgram, we think, hey, if you want, if you're doing anything in audio, you should be thinking about Deepgram. Maybe you don't use this for every piece of it, but you should be thinking, hey, if I want low-latency real-time speech-to-text, you should definitely be thinking about Deepgram.

If you want to be, if you want to be using low-latency text-to-speech, definitely at least talk to us. That's a new product for us, but, you know, the next version will be even more expressive. Butright now, it's, I would say, better than, like, Amazon, Microsoft, etc., than their neural models.

Not quite as good as 11 Labs. But anyway, the next product coming out for us, which I want to give everybody the chance to try out or apply to for our preview, is our voice AI agent, which is a full stack where we put everything together.

So you could, if you want, you can use your own API keys and use your own LLMs. But also, you could have it all put together with Deepgram, and this helps with reducing that latency. So you get your turn-taking down to a very short, you know, 300 milliseconds, 500 milliseconds, 600 milliseconds, rather than, like, 800 or 1,500 or something if you tried to piece them together yourself.

And if you want to get access to this voice AI agent API, we have a QR code here. And we have some folks in the back, too. If you saw a workshop, I think, two days ago with Damian, he gave an awesome workshop.

He's in the back, you can talk to him about this. But also, just feel free to screenshot this or go to the link now or whatever it is. Also, at Deepgram, we give out $250 in credit, so anybody can try it out.

Thanks, everyone.