Building and Scaling an AI Agent Swarm of low latency real time voice bots: Damien Murphy

Hey, everybody. Yes, we're going to get started here. Thanks to everybody for coming along, and thanks to AI engineers for having us. My name is Damien Murphy, I'm a Senior Applied Engineer at Deepgram. A lot of people ask me, "What is an Applied Engineer?"

It's a customer-facing engineer,right? So we work directly with customers to basically help them achieve their business cases. Yeah, I've got 20 years of experience, full-stack developer. I've been working in pre-sales, post-sales, really just customer-facing roles for the last 10 years or so.

I've helped hundreds of companies working with Deepgram to actually build and scale low-latency real-time voice bots. A lot has obviously changed in the past few years around how we actually build these voice bots, and I'm going to kind of walk you through that evolution as well.

If you're not familiar with Deepgram, we're a foundational AI company. That means we build our own models, we label our own data, and basically train and deploy at scale. We have a lot of models. We have over 1,000 models running in production, so we run a lot of custom models for different use cases,right?

So you can think of meeting use cases, drive-thru use cases, phone call use cases. The vast majority of audio today is generated in a call center, and call center data is probably the worst audio you've ever heard,right? It's, you know, 8K, Moola, or Linear 16, and it's just very hard to actually understand for an AI model.

So by us building and training models specifically targeted to that low-quality audio, we're able to have a much better performance. We're research-led, so, you know, about 70% of the company is research and engineering. And that means that we really focus on, you know, building that foundational, scalable, and cost-effective AI solutions.

You know, building a model is easy; rolling it out into production at a low price point is hard. Yeah, so what are we going to learn today? We're going to build a voice-to-voice AI Agent. That means you're literally just going to send audio data in, and you're going to get audio data back,right?

So you don't need to hook in, you know, the LLM, the speech-to-text, and the text-to-speech. You're going to be able to basically do audio in, audio out. We're going to build a simple backend API for the AI Agent.

That's going to enable us to, you know, build a frontend that shows what's happening, and also allow the LLM to do function calling. And then we're going to talk a little bit about how you could scale, you know, these AI Agent Swarms.

Yeah, a couple of prerequisites. I'm going to get into those first, just so that people have time to kind of install any tooling that they need. I'll help you understand, you know, the kind of evolution of AI voice bots,right?

So how they were previously, how they're moving today. I'll help you get set up. We'll go over a little bit of application architecture, how it works, and then we'll touch on scaling it as well. Yeah, so you're going to want to go to deepgram.com, sign up for an account.

It's a free account. You'll get $200 in credit. That's about 750 hours of transcription for free. You'll need NodeJS installed on your machine. That will allow you to run a little HTTP server, and if you want to modify the backend and run your own backend, you can also do that.

We recommend Chrome browser. I haven't tested on other browsers, so if you have Chrome, great. It should work in other browsers, but you just never know with browsers these days. You're going to need a microphone and a speaker or headphones.

You're going to be talking to this AI Agent, so it should work fine on a laptop with a speaker on, but yeah, just keep the volume down a little bit so it's not communicating with other people's agents.

Okay. And we need the API key?

I've set it up today that you're not going to need an LLM API key or a Deepgram API key, just to keep it simple. But after this, you'll have the opportunity to sign up to the waitlist where you will need that API key.

Awesome. Yeah, so the current approach, and I've been building these sorts of voice bots at scale for quite some time, and typically they revolve around three key pieces,right? So you've got your speech-to-text, that's going to take your audio, give you back a transcript, and then you've got an LLM that's going to take that text that you've detected, process it, and then generate a text reply.

And then you're going to use text-to-speech to actually speak that back. This has been around for a while,right? It's gotten better and better, and lower and lower latency. We can really bring down the latency on both the speech-to-text and the text-to-speech, especially if you run it self-hosted.

So one of the things I help a lot of customers with is, you know, co-locating all of these pieces together to bring the latency down. The challenge with that is it becomes an infrastructure challenge rather than a software challenge.

So what we've tried to do with our new voice agent API is actually offer all of that as a single API,right? So you can send us audio, we'll co-locate all of those services together, and then we'll be able to send you back the response.

We also handle a lot of the complexity of, you know, endpointing, when does the user finish speaking, and those can be challenging. So being able to have all of that in a single API really just makes it easy for the developer to build the application that they want to actually achieve.

One of the things you'll notice here as well is function calling. So depending on the LLM you use,right, you may want to shift your entire infrastructure to a different provider,right? So if you're using Claude, you'll probably go with something like AWS.

If you're using OpenAI, you might go with Azure,right? You want the LLM and the other pieces of the puzzle to be co-located. If you're running your own local LLM,right, so Llama 3, Phi, things like that, you can just put them really anywhere you want.

And you can see here as well the time difference,right? So going from 500 milliseconds for the speech-to-text to 700 milliseconds for the LLM and TTS, and you can really bring that all down,right? So as you bring those latencies down, you actually start to respond too fast,right?

And that's when we start adding in delays and things like that to make sure that you're not, you know, being rude.

Hey, excuse me.

Yeah?

I see a lot of people taking photos. Do you have the slides uploaded somewhere?

I don't have them uploaded, but after the event, I can talk to the organizers and get them shared. Allrighty, so yeah, let's see if the demo gods are with me today and the audio works.

Whoo!

Hello?

Nope. I'm going to let my mic. Let's try that one more time.

Hello?

Hi there. Welcome to the Krusty Krab drive-thru. What can I get started for you today?

Yeah, could I get a Krabby Patty, please?

Sure thing.

I think it might be picking itself up.

Would you like to go ahead and add anything else to your order?

Yeah, can I get a kelp shake as well, please?

I've added a Krabby Patty and a kelp shake to your order. Anything else?

Yeah, could I get a Krusty combo as well?

We don't have a Crispy Combo on the menu. Would you like a Krusty combo instead?

Yeah.

I've added the Krusty.

And let me just refresh this so it's not picking me up. So you can see,right, we've basically just sent audio to the service. It's gone off and figured out what to do function calling-wise. So the LLM is actually making the order.

I'm not doing any sort of order making within the app. I'm not parsing the LLM's response. Everything has just happened automatically. Cool. So let's jump back into the slides. Right, so if you have NodeJS installed, I have two repositories for you, which is located here.

So github.com/damiendeepgram/deepgram-workshop-client and deepgram-workshop-server. The server itself, I'm actually running it on glitch.me. If you want to make changes to the server, you're going to need it publicly accessible,right? So in order for the LLM to be able to reach out to it, you'll need that publicly accessible.

That's really a stretch goal. You don't need to modify the backend. The backend I have running should be able to handle everything. If you do want to make modifications, and we'll go through that a little bit later, you can basically spin it up yourself, you know, point the LLM to that new API, and it'll be able to then call those functions.

Hands up, anybody already got NodeJS installed? Okay, we've got a good few. Anybody having trouble? All good? Cool. Yeah, so once you have that set up, you can simply run the workshop client. So the workshop client is Vanilla JS,right?

I tried to keep it as simple as possible. Simple HTML page, and then the main.js is where we open that WebSocket connection. We capture the audio, and then we send it. So, you know, this can send audio at a pretty fast rate.

If you were running this with a telephone system, you could probably send audio at like 20 millisecond chunks. That's going to give you the lowest latency. The browser tends to send larger chunks. So yeah, you can definitely bring the latency down when you increase that chunk rate.

We can't process audio faster than you send it to us, unfortunately. Within the config, and this is really where you're telling, you know, the API what it's actually going to do, who it is, how it's going to work.

You can see we have a base URL. So that base URL is just pointing to my WebSocket server, or sorry, my API server. And we have some input parameters,right? So we're sending Linear 16 audio, and down here at the drive-thru speech-to-speech config, this is telling, you know, the system, okay, I want to use OpenAI, I want to use GPT-4.0.

These are the instructions,right? So, you know, simple system prompt. And we have a function call here as well. Let me just bring this down a bit. Yeah, so this function call is telling the system, hey, if you want to add an item to the order, this is the API you're going to call,right?

There's going to be a call ID, so, you know, when the system starts up, it, you know, generates a unique call ID so that all your order items can go into your particular order. And yeah, it's pretty straightforward.

Later on, we'll actually look at how we can add more of these. And I'll just go back to the slides here. Oh yeah, so just kind of talking at a high level, I probably should have done this before jumping into the code, but we have a user, we have a browser, the browser is generating microphone data, and we're sending that microphone data to the service.

So the voice agent API wraps all three pieces together. The LLM can do the function calling, and so can the client browser. So the LLM adds the items automatically, and then the browser is just displaying what items are actually in that order.

Any questions on that before I move on?

GitHub repo?

Oh yes, you need that again? There you go.

Was everybody able to find the GitHub repos?

It's in Slack.

Oh cool, nice.

If you missed it, it's on the Slack.

Thank you.

Awesome.

Yeah, so this is the agent configuration I was showing you earlier. Right now, the app is pretty basic. It only has an add item,right? So you can't take items away, you can't modify items. And the reason we've limited it to that is, you know, that's a challenge for you.

Can you add order modification? We've already got the API there, so we have a remove item API and a few orders. Let me grab, yeah. So there's, yeah, these are all the APIs that are in the server code.

So if you want to add that order modification, you're going to need to look at, you know, basically removing an item from the order. I believe this one here, delete call ID order items. So we've already hooked up the add item, but in order to remove items, your LLMs need to go into understand what's already in the order, and you'll need to use the get order call.

But we'll get into that a little bit later.

Okay, so we support multiple LLMs. So within the API today, you can call, you know, Claude, Llama 3, Mixtral, supported with OpenAI, Anthropic, and Grok. We will be adding to that as well over time, but those are the kind of initial ones.

This API is pre-release as well, so you're basically getting a sneak peek to it.

Yeah, so we have the menu here. So the menu is coming from the menu items API. When I want to create a new call, so on loading of the web page, we basically grab a unique call ID, and we get the menu.

The menu itselfright now is baked into the LLM system prompt. Stretch goal will be, let's turn that into a function call as well,right? Let's allow items to go out of stock,right? The LLM will need to know when you're out of chicken wings and things like that.

And then we have the get order for the call API as well. Yeah, so the WebSocket client itself, if you are familiar with the Chrome DevTools, let me see if I can grab it here.

Yeah, so in the network tab of the Chrome DevTools, there's a cool little WebSocket inspector. So if I start a conversation, you'll be able to see the messages as they happen,right? So you can see here I'm streaming audio at a pretty rapid rate, and, you know, when the API responds, it's going to send us back messages as well.

So there's just quite a few lifecycle messages that we'll send you to, and I can bring those up now as well.

Yeah, so the settings configuration, that's one that we send. So we tell it, and this is being truncated as well. There's a lot of like system stuff and function calling as well in it. But we're basically telling the system what we want to actually send it, what we want it to be, how we want it to work, what functions we want it to call.

And it's going to send us back this session ID. And then the conversation text,right? So that's going to give us a transcript of what the user is saying. That can be useful if you want to display, you know, text on screen while the person is speaking.

Sometimes that feedback is really useful so people know it's hearing you as you talk. And then, yeah, the function calling. Soright now, this is only doing add item, and you can see the argument here is basically the Krabby Patty.

Yeah, so the tool will respond, whether it was a success or failure. So, you know, if it can't add the item, you'll know about it. And then you'll also get the assistant's response back in text. It can be useful to know what the AI is saying if you want to do content moderation.

So you don't want it offering free Krabby Patties, for instance. You might have a mechanism to detect, you know, various things. And you can apply that to a lot of different use cases as well.

Yeah, so the backend API, this is the main one that we're going to be using today. But yeah, I do recommend trying to add a few more. And yeah, on the server log, so, you know, in the server where it's running this API, you're going to see things like this,right?

So there's a new call coming in, you've got that order ID, you're adding an item to that order, and then you're getting the updated order here as well. And this is essentially what, you know, we're consuming in the front end to display the items as they're ordered.

Yeah, so walking through the client code, there's five files. I tried to split them out logically as best I could. So let's go through each of those and just kind of explain what each of them does. The main.js, this is going to be, you know, all the kind of front end hookup code.

config.js, that's going to be how we tell the LLM and the agent API what to do. The services.js, that's just a, like, you know, code kind of interface to the backend API. audio.js does some kind of interesting stuff around, you know, audio manipulation and downsampling.

The browser itself actually sends

higher sample rate audio, but we don't need 48 kilohertz audio at that rate. That's going to be a huge bandwidth usage. So we drop that down to 16 kilohertz, which is, you know, essentially what the API can handle pretty fast anyways.

And then animations.js is really just animating that little bubble that, you know, kind of responds to speech. And then the server code, super simple Express.js API. If you're familiar with Express.js, you know, it doesn't really get much more simple than that.

Okay, so let's take a look at the code.

Yeah, so this is the index.js within the server code. It's just got a few very simple function calls here. So

I'm not sure how much I should go into this, but yeah, it's pretty straightforward. It's just updating the app state and handling some of the code operations. Yeah, so let's go through these top to bottom. The animation stuff, it's just a simple canvas.

It's going to modify the bubble, and I'll show you the bubble again if you forget what it looks like. Yeah, so this bubble here is going to respond to speech. So you can see it kind of getting bigger, smaller, bigger, smaller.

And that's pretty much what that does.

Within audio.js, so we've got a couple of different functions in here. We've got receive audio and capture audio, and then clear scheduled audio and downsample. And this little function is just a conversion function that the downsample uses. The reason we need a clear scheduled audio is because you can interrupt the LLM while you're talking,right?

Or the agent. And we may not know that on the server side because you're handling it on the client side. So if you've got audio already playing, you're going to want to pause that audio as soon as you can.

You could even add a client side voice activity detector. Solero VAD is a really good one that I've used before. And that just allows you to do that barge in. So when you do start speaking, you know, it knows to stop.

You know, and more advanced systems will help the LLM actually understand whereabouts in its speech did it get interrupted,right? Because then it may not know you didn't hear a part, you know, at the end of its prior response.

And then, yeah, receiving audio. So this is basically grabbing the audio from the WebSocket and just sticking it into a buffer. And then the capture audio, this is just grabbing it from the media devices on the browser. And then once we get that, we call the callback, and that callback is what we saw here, which is the WebSocket send data.

Yeah, so walking down through this, on load, we're going to prepare the agent's config and send that over. That'll give us the order ID. And then when I click start conversation on the UI, that's going to call this code here, opens the WebSocket, and begins sending audio data.

The errors, any WebSocket errors will be handled there. And then this is essentially where we're getting back

text-based status messages. So user started speaking, you know, like what the AI said. And then here we're actually receiving the audio. So receiving the audio is what we looked at in the audio.js file. And then we also have the ability to update voices.

So I don't think I showed that actually in the prior example. I have a version running here. Let me just kick it off.

Hello?

Hi there. Welcome to the Krusty Krab Drive-Through. What can I get for you today?

Yeah, can I get a Krabby Patty, please?

Got us a Krabby Patty. Anything else for you today?

Yeah, can I get a Krabby Meal as well?

Sure, a Krabby Meal.

Actually, can I change that to two Krabby Patties?

I've added an additional Krabby Patty to your order. Anything else you'd like?

Yeah, so we have 12 different voices. That voice, that second voice I used there is actually my own voice,

which is pretty handy. Yeah, I had to tell my parents when I trained the voice. I was like, if you ever get a phone call from me looking for money.

Cool.

Excuse me.

I'm still talking to it, am I?

I have a question.

Oh, yeah, sorry, go ahead.

Where in the stack do you categorize the type of function call? So whether that's add item or remove item or, yeah.

Yeah, that's in the config.js.

So you can see here we tell it what to call and we give it a base URL.

So it's not dynamic based on the prompt?

The LLM will dynamically decide what to use.

Okay.

Yeah, so there's no like direct calling of it. The LLM is kind of like, you know, ChatGPT plugins,right? You don't really know if it's going to use it or not. But yeah, they've gotten pretty good, especially GPT-4.0 and I think Mistral as well is pretty good at calling it.

You'll probably have problems with GPT-3.5 and function calling. It's just not really up to the level to do it. But yeah, LLMs are getting better, so they're able to do it now.

Thanks.

Actually, I'm sorry, putting on top of what you just said, I don't know if anybody can hear me. I'm sorry. What I've realized is we have a good system prompt along with a good function definition that really works with most newer models.

You consistently get it to use the tool or request the actual tool instead of just coming up with its own stuff.

Yeah, yeah, definitely. And like, it really is only those newer models,right? So the latest Claude. I don't know if Haiku is going to work super great with function calling, but definitely like Sonnet and Opus work a lot better.

And then on Grok, they host Llama, LlamaTree 70 billion, I believe, and Mistral. You kind of, your mileage may vary with those open source LLMs. I don't think they've caught up to the function calling level just yet. But yeah, like, you know, shoot for where the puck is going to be, and I think a lot of those will catch up pretty soon.

Yeah?

So I'm curious about your take on the UX of the voice. In particular, what do we have in terms of recommendation specifically for interruptability? For these models, for these kinds of interactions, I think one place where people get tripped up is, well, this is cute, but oftentimes people think while they are saying something.

So oftentimes there are like these awkward silences in between where the sentence is not fully formed yet, and midway the thought changes and stuff like that. And how does that affect the usability? Likewise, the model is saying something, and is there an interaction design that you recommend to interrupt the model and think instead of, okay, this is going in the wrong direction?

Sorry, I've got two questions, but I was just curious.

Yeah, yeah, so for anybody who didn't hear, the question is about interruptability and how to handle things like long pauses. And that really comes down to endpointing and contextual kind of semantic endpointing is what we call it. So that's something we're going to build into this voice agent API.

So, you know, you can imagine a scenario where the user says, hang on a minute, let me get that for you,right? You know, say they're going to get their account number or whatever it is. That doesn't necessarily require the LLM to go off on another kind of monologue.

The LLM might say, sure, you know, let me wait for you to get that,right? And that's kind of semantic endpointing. The other type of endpointing, which is kind of, you know, traditionally what people used, was, you know, a span of silence is used to determine when somebody's finished speaking.

But yeah, like if people are calling out credit card numbers, it's pretty common for them to do backchanneling. So when you do backchanneling, you're essentially waiting for like a noise from the other person, like a mm-hmm, you know?

So if I do like, you know, one, two, three, four, mm-hmm, five, six, seven, eight, mm-hmm. And that just kind of gives you that like, I've captured what you said, so that you don't go too fast and then the person like falls behind.

And a lot of the time with these voice agents, what I recommend to customers is, you know, what would a human do,right? And there seems to be this really high expectation that the AI should be able to understand pretty much anything,right?

But the reality is, is that like nobody can understand my email address over the phone. And I have to call it out like, you know, D for Damien, you know, A for Apple. And this is with a human, but you know, with an AI, you need to build in that kind of understanding logic.

And I'll go into it a little bit later about, you know, how you can make that composability with these agents. Because you don't want to create an agent that does everything, you know, for your entire business,right? You want to create an agent that's capable at a particular task and then build them together,right?

So having that kind of multi-agent system where you can offload parts of the conversation to, you know, a slightly different AI agent that's able to collect credit card numbers very accurately and handle all of the edge cases or, you know, verify account information, you know, versus, you know, taking an order.

They're all very different use cases, but yeah, from what I've seen in the market, people tend to want to make it do all the things in one system prompt. And it's just not there yet. You know, even with these large context windows, I don't think it's really good to try to get it to do everything and have, you know, a massive system prompt.

Like as you increase the system prompt length, you also increase your time to first token. Time to first token is really the key metric for an LLM to respond. So you can start responding as soon as you get, you know, let's say five tokens or ten tokens.

You can start the TTS playback at that point. If you wait until like, you know, the 250th token, the latency is going to be much higher,right? Maybe if you're using Grok, you could wait that long because it's so fast, but yeah, most LLMs are outputting, you know, maybe 30 tokens per second.

And it's highly variable,right? Like even GPT-4.0 can give you like 900 millisecond latency on first token. And you know, that's something that's going to improve over time, but yeah, it's definitely something you have to be aware of when you're building these voice bots.

Yeah?

So I suppose that I've got this use case and I'm looking at either using ChatGPT with plugins where I can also give it capabilities for prompting, give it capabilities for calling APIs, versus I'd like to use, you know, more of a spoken solution.

In your experience, what are some of like the things to consider, you know, do I want like a clean and dirty thing? And also like, you know, what are the more nuanced maybe ways to cater to a specific use case like you were mentioning, you know, these like agent folks?

And just to clarify, so is the question about using this approach versus which other approach, sorry?

Yeah, like a ChatGPT plugin where you give it also like functionality of calling APIs and prompting.

Yeah, so I'm not sure I fully understand the question, but let me kind of paraphrase it. So using a ChatGPT today doesn't have ears or a mouth,right? So you just have an LLM text in, text out. So you still have to add the ears and the mouth.

And what we're doing here is real-time low latency streaming. So the audio is being streamed like, you know, straight into the system, and then audio is being streamed straight out of the system. You know, obviously GPT-4.0 had that big fanfare announcement the day before Google's announcement, but neither of them have released anything yet.

And the reason that they haven't released anything yet is that it's hard,right? We've had real-time voice agents, you know, for years, and they've just gotten better and better and better. And one of the key things there is the latency from, you know, end of speech to transcript.

Once you go self-hosted with Deepgram, you can get that down to like 50 milliseconds. In our hosted API, you're going to get closer to half a second. And that's just because we don't like crank up the compute. So as you increase the compute, like say to 5X, you can get down to that 50 milliseconds.

So a lot of these, you know, companies that you see showing real-time voice bots, they're using Deepgram under the hood. You know, today I think we're the only option for low latency real-time speech recognition. That may change in the future, but yeah, today this is kind of state of the art, I think.

That answer your question?

Yeah, I'm wondering like what are your thoughts on, you know, with like ChatGPT,right? You can create all the plugins and enable these plugins to make API calls. Would you say like that is kind of similar to a part of the workflow that you're showing?

Yes, so we're LLM agnostic. So you can use function calling with a lot of LLMs. Like building a GPT assistant kind of follows a very similar API format. It's a pretty standard OpenAI kind of interface, and most LLMs have actually adopted that same interface.

So, you know, you can use that same function calling and system prompt with another LLM. So yeah, I think that's definitely interchangeable, but there's no real difference between a GPT agent and what we're doing here. Yeah?

Suppose the function call that you're making is like a, is going to be long running. Is that blocking to the voice agent? And like, I guess, what are some ways around it if the function call that you want to make is something that's long running?

Yeah, so the question was about long running function calling, and that's definitely a concern. I don't know if you want to do long running function calling with a real-time voice bot. You might hand that off to, you know, a secondary system.

So you say, hey, okay, you know, I'm checking on that for you, is there anything else I can help you with? And then when it comes back, your agent can then offer up the information.

Is this, like, are the voice agent function calls like full only? So like in the context of this demo, there's like a get order function, so like it first adds things to the order and then like looks at the order, but is there a way to proactively push things to the conversation window as part of this API?

Yeah, yeah, so we're pushing the order items from the LLM. So the LLM is actually making the call to the add order or add item to order API. So it's doing the pushing,right? I'm not pushing anything client side.

I'm only reading client side. So client side, I'm just polling the order and it's like, you know, give me the order, give me the order, give me the order. And that's able to allow me to display the order, but the actual pushing, yeah, it's happening all from the LLM.

I see, so with respect to like, like if we need to add information about the order to the LLM, that is in and of itself a function call on the part of the LLM to pull from the API?

Is that correct?

Yeah, so what you would want to do is you would want to give it a new function. And we've already created the functions to give it, and that's kind of the next step in this. So you would want to add a new function here for get order,right?

So this would be like your get order function, and then you can point that to the API to get the order. Another one you'll probably want to do is get menu,right? So, you know, is there an item that's no longer available,right?

Because we're pulling from the, you know, the menu ordering system to see if something's out of stock, because we know all the orders that have gone through. And then another one you'll want is actually remove item. So with remove item, you have the ability to modify the existing order.

I didn't implement these in the function calls because I thought that would be a good kind of learning exercise for people here. But yeah, you could definitely add those and understand a little bit more how to work.

Got it, and

if we did want to have a long running function that ran, it would be a non-blocking function call, it triggers a job, and then at some later point, the assistant attempts to fetch that information. Is there a way for a client to actually push data into the conversation log, like with this API?

Is that a possibility?

Yeah, yeah, absolutely. So as a part of the, you know, the information that the LLM has access to, and I don't know if you can see it here, it might be off screen. You can see here the menu,right?

So the menu there is a part of its system prompt, but you could remove that menu from there and add it as a function call. So now anything that modifies, like you could have a separate service modifying the menu.

It could be a separate LLM,right? And when that menu is modified and it pulls the menu, it's now updated its system context.

Understood.

Cool. Yeah?

Can I ask you a follow-up just to that?

Yeah, sure.

Looking at the workflows, the function calls, have you had any cases where the agent goes, "Oh, by the way, this long running function call is finished in the background. Would you like me to get more information on it?"

I haven't tried it myself, but I'd probably imagine you'd want some sort of webhook callback. So when the function call is complete, it would instantly return, but then have a separate handler that would know that it's completed. And then you can prompt the LLM from that webhook handler to say, "Hey, you know, this thing you asked for earlier," and it would kind of act like a user input as well.

You'd say that as a function call within the company.js?

The webhook handler would probably run in the backend, not in the LLM itself. So the LLM would just say, "Hey, go do this long running task," instantly return and say, "Okay, I've kicked off that long running task." And then when the webhook handler gets fired by the long running task, it would then tell the LLM, "Hey, you know, this long running task has completed."

I think you had a question as well.

On the handling side, how do you differentiate between a noise versus action speech?

Yeah, so we have voice activity detectors, and the voice activity detector will only trigger on audio that's generated by the vocal cords. It will trigger on coughs and humming and things like that, but you may want that to happen as well.

So you can put in things in place to detect, "Okay, did I actually transcribe a word? You know, should I respond to this?" So those are things you can implement as well.

So on that, will it add a latency when I'm speaking on the phone and if I interrupt?

Mm-hmm.

While you have audio going on, you'll have to wait for the response?

Yeah, so we'll send you a very quick user started speaking event using a server side VAD. So the voice activity detector is going to tell you as soon as you get it. We actually have code in there as well to clear, I believe.

Let me see, where is this? In here.

Yeah, so you see if user started speaking happens, we basically stop the audio playback.

Okay. What's the latency on that?

It can vary. I think it should be in the order of like less than 100 milliseconds. Yeah?

Is memory being handled at all, or is that just something separate?

Memory would be kind of like a separate challenge. You can obviously build up the system context. You know, depending on your use case, like if you want to handle hour-long calls, you probably don't want to keep building up the system context.

You'll want to use some sort of memory system. AutoGen have a pretty good teachable agent if you're familiar with it. It has the ability to run a secondary LLM to ask, "Is there anything new or updated, you know, in this new content?"

Update, you know, the existing memory. A good example of that might be like, "Oh, I live at 123rd Street," and then it comes down later on and it's like, "Oh, actually, I live at 456th Street,"right? And you don't want both conflicting in your system prompt.

You want to update the prior memory of it. Yeah?

How do you protect against prompt injection or raw content into the system?

Yeah, and that's really the reason that we have, let me see this one here. This is why we have the conversation text. So, you know, you'll hear stories of people getting, you know, Chevrolet cars for $0 by, you know, modifying the system.

But what you can do is you can actually have a process on the text that, you know, tries to block certain things,right? And that content moderation is very important to prevent things like that. I don't think you're ever going to be able to prevent like prompt modification,right?

Because, you know, if you ask an AI bot five times or even three times to like break its rules, it probably will,right? Like the first time it's like, "No, no, I can't do that." And then the second time it's like, "No, definitely can't do that."

Third time it's like, "Sure, I'll do that for you." And that's just an inherent problem with LLMs. So, you know, you're not going to be able to stop it on the way in, but on the way out, you could be like, "Hey, you know, you've offered something that we've detected is invalid."

But yeah, it's a hard problem. I don't think anybody's really solved that. I don't think we understand LLMs enough to solve it. Yeah?

I'm curious about the TTS and STT side of it as well. What kind of language support is there? What kind of interpolation support is there from a speech-to-text setup? And likewise from text-to-speech, are there any emphasis or different voices that can make it sound more natural?

Yeah, so the question was about the language support on

text-to-speech and speech-to-text. So let me just log in here quickly.

Yeah, so if I jump over to the text-to-speech, these are the different voices we have. So we have 12 voices. Today we've got all of our English voices publicly available. They're super low latency and very low cost,right? About 20x cheaper than 11 Labs.

So we're kind of competing at the, you know, the Google, AWS, Azure voice pricing, but with the quality that's pretty close to 11 Labs. We're working today on promptable TTS. So that's going to give you the ability to say, "Hey, you know, say this in an empathic voice or say it in a pirate's voice,"right?

Like the ability to prompt it. And once we've completed that research, we're then going to roll out other languages. The challenge with building them now would be that we'd have to go back and retrain all of the models once the promptable TTS is out.

I believe our TTS launched about four months ago. So yeah, it's pretty new to the market, but you can play all of them here as well, which is pretty handy.

Deepgram is great for real-time conversations. And also, you can build apps for things.

Deepgram is great for real-time conversations.

Deepgram is great for real-time.

Yeah, and what I've found with customers is the vast majority of customers want female voices.

I don't know what it is, but I guess nobody wants to be mansplained.

Yeah, and then on the language side, we have 36 languages supported today on our Nova 2 model. We have a few more supported as well on our older models, but we're adding languages, you know, every month there as well.

We've actually got an auto training pipeline set up, probably the first in the world, I think, where we have the ability to detect like low confidence words and then like retrain based on low performance. We've also got a ton of other intelligence APIs.

So if you want to do summarization, topic detection, intent recognition, sentiment analysis, you can send all of those off as well. And I think of it like a customer service one here. And those can be pretty useful because like detecting topics in an actual audio file and where they happen is super useful,right?

So if you're in a call center and you want to understand, you know, at a high level, how many calls, you know, of my millions of calls touched on these different things and which ones to automate. And we usually say to people that are, you know, automating the call center is like, you know, step one is analyze all your existing calls,right?

Figure out what you've got. And if you've got 40%, you know, phone issue, look at automating, you know, the phone issue first,right? And a lot of them do agent assist, which is like bubbling up knowledge-based articles to real people.

And then once they have that built, you know, it's very easy to then just, you know, use an AI. We don't necessarily want to replace people in call centers. We just want to take away the work that can be automated.

So like what we're seeing now in call centers is that like one call center agent using a cloned AI voice can hand off a call to the agent. So let's say it's collecting credit card information. They can just, you know, press a button, let the AI collect credit card information, and they can do five calls simultaneously.

And then when a call needs their help, they can jump over to the call that needs their help. So you're kind of 5xing productivity with that approach.

Yeah? Yeah, question at the back there?

How do you go about monitoring? Like if you want to monitor where the order got wrong or where someone tried to jailbreak it or all of those things?

Yeah, so there's multiple places you can do that. You're probably going to have the agent actually running on a server. You know, for this workshop, we just put it in a browser. But, you know, the vast majority of people will have this agent running directly like on a Twilio, you know, phone line.

So when that agent is doing the work, you would handle that in your server code,right? You might have some moderation code that, you know, detects something out of allowment and then, you know, blocks it. You can definitely add it to the system prompt, but it's only going to get you so far.

Yeah?

So what kind of evaluation strategy that you would use in this kind of scenario? Just like end-to-end or like, you know, text-to-speech, text or functional by functional evaluation, and what kind of metrics would you say you use?

Yeah, so the question is about like how to monitor and track metrics, is it?

Evaluate what it is.

Yeah, yeah. Yeah, so there's a lot of open source projects out there at the moment for kind of doing agent ops. One of them is called Agent Ops, which is pretty good. That can give you, you know, temporal debugging.

So you can actually debug what was happening throughout the LLM flow. So there's a lot of stuff in that space that's happeningright now, but yeah, a lot of your typical monitoring tools will work there as well. Yeah?

While I know, of course, that's basically a big part of your

company secret, so to speak. I would be interested in general techniques you use to reach that real-time support speed up compared to just the traditional building your own, just as you showed in the beginning, voice to text, put it in an LLM and then have it then prompt code and speak it to the agent.

Yeah, so a lot of our customers do that today,right? They host all the different pieces. And it becomes like a large infrastructure challenge,right? We'll touch on that a little bit later as well as a part of the agent swarm stuff.

But it's like how do you make sure that you have low latency in different regions,right? So people in the EU don't want to be hitting a server in the US,right? Not just for GDPR reasons, but the latency is going to be higher.

Same with APAC. So now you have to build and scale each of your clusters in multiple regions, and you have to be able to auto scale as well,right? Like one of the major use cases for AI agents is peak traffic,right?

So like you might only need 10 customer service agents, you know, five days a week, but if there is like an outage in PG&E, suddenly they need a million agents,right, for one hour. So the ability to actually scale up, same with, you know, 911 services.

They can't take all the calls when there's a large disaster or something happens. So a lot of people actually just, you know, get a busy tone. So the ability to do that spike up and scale in multiple regions, that's a huge challenge for a lot of startups.

So having somebody that offers that as a service, I think, is pretty useful.

Yeah, sorry to add to that. What I actually meant with the question was more like if you theoretically have it all running locally. So we completely forget about actual hardware or scaling, etc. What kind of techniques do you use there to improve the concurrence time, the time between the models, etc.?

Yeah, so when you run in our hosted API, you're running at a one-second interval,right? So every second you're getting what was spoken,right? Kind of like a metronome. Once you run it yourself, you can crank that up and you can say, you know what, I'm going to run it five times a second.

So you're inferencing like an ever-growing context window, you know, at a much faster pace. Most words are about half a second long. So, you know, you're basically inferencing words, you know, partially as well. So the word something might be so, some, something,right?

So you're getting this increasing context window. We roam with like a three to five-second context window in a real-time streaming, which allows us to solidify, you know, every three to five seconds what was spoken. And then as soon as we detect that end of speech, we'll basically, you know, say, we're not going to get any more words.

Let's, you know, finalize what we have so far. And that's really how you can achieve those low latencies. But it is a lot of compute. Yeah, but with our system, it's very fast, very light on compute. So you can actually run a lot of streams like on a Tesla T4.

Awesome. Oh, one more question.

If we are using our own LLM, how can we co-locate with Deepgram?

Yeah, so we offer self-hosting. So you can basically, you know, grab our Docker images, models, run it on a GPU. Like if you can get like three GPUs on a single motherboard, that's going to give you, you know, lightning fast end-to-end.

Allright, one more question.

So in building a speech-to-text attribution and a using AWS response pipeline, if I were to switch to Deepgram, what three metrics would I see improved?

So speed, quality, and price. So yeah, all three.

That's all?

Yeah, it's easy to sell Deepgram. Okay, I'm going to jump back into slides because I think some of the other stuff might be of interest as well.

So yeah, so some more advanced features that you could play around with. Make a whole new backend,right? Maybe it's, you know, a table booking API. You know, a lot of businesses have to answer the phone. I don't think they want to,right?

They're already busy. Handle multi-agent flows. So have a routing agent with two sub-agents. So you could have a booking agent and a cancellation agent. And that can be the same voice on the same phone line. So, you know, if I say, hey, I want to make a booking, route it to the booking agent.

I want to cancel a booking, route it to the cancellation agent. Those are essentially how you would build out these more complex systems. Use cases, call center AI agents. I think this is already here. We're seeing thisright now.

Like, you know, you have companies that have replaced, you know, a large proportion of their call volume with AI. And, you know, they still employ call center agents because, you know, they always need to hand off difficult calls that they haven't covered yet with the AI agent, you know, to somebody that wants to handle it.

IoT AI devices,right? So wearables, toys, things like that. There's a lot of them out there. And then, yeah, AI worker agents. So, you know, working in a drive-through, taking those orders. You know, the workers that are actually doing that are also, you know, busy preparing food and doing other things.

Yeah, so multi-agent swarms. So reduce complexity. Keep it simple,right? Get something that works really well, really robust, and kind of like box it off,right? Like single responsibility. Reduce the cost. Use the smallest, cheapest model you can to achieve the use case.

Likeright now, it's probably going to be those bigger models, but I think in time, you know, the price point of those is going to come down and the new generation will kind of take its place. So, you know, every six months, we're seeing like a 10x drop in cost.

And then composability. So you can reuse, you know, a sub-agent in multiple different flows. So this is kind of a pretty basic kind of layout. So you have your rooting agent that's able to figure out, you know, which agent to use.

You have a support agent and a booking agent in this example. And maybe you have a technical support agent and an account support agent,right? Two different types of agent, but they can each kind of help depending on the need.

Your tech support agent is probably going to be hooked up to some sort of RAG system. Account support, an existing booking agent, is probably need to are going to need to verify that this person, you know, owns the account that they're calling about.

New booking agent might leverage like a credit card payment agent.

Yeah, so scaling it,right? We talked a little bit about the latency. So distance kills latency,right? Like if you call a server in the US from Asia, you're going to see like, you know, a second extra at least latency.

If you want to do regional scaling,right, you're going to need to have the ability to horizontally scale, you know, within US East, within US West, within EMEA,right? You're going to want to have redundancy as well. As you add redundancy, you increase cost as well.

But, you know, if you want high availability and your agent to always be on, you're going to need that redundancy to do it. And then horizontal scaling within your, you know, your regional clusters. We support Kubernetes. We'll give you all the auto scaling, Helm charts, and everything.

So that can be pretty, pretty powerful. But you can imagine like if you wanted to, you know, build an agent, do you really want to worry about all that infrastructure,right? Or do you want to just build the agent, achieve the, you know, the value, the business value, and roll it out on a large scale?

Yeah, sorry, go on.

So based on this, I was wondering what your outlook is on the embedded models. Do you think that at some point in time, the embedded speech-to-text and text-to-speech models which are on-device, would they ever be good enough that you can have them on-device and not worry about these latencies?

Yeah, I think it's possible. You still have to distribute the models,right? The models tend to be quite large. I think Gemma is like, you know, two gigs. You can put it in the browser, but it's going to take you a while to download.

You know, as you move to mobile devices, you know, it's going to be pretty hard to do it as well. Like I do believe that there's a lot of use cases where on-device makes sense. But those are like single stream,right?

So you have a single stream, you're sending a single request to an LLM, you're getting a single response. What we're building here is like, you know, a million simultaneous calls,right, can come in. That's never really going to work on-device, just from a distribution perspective, I guess.

But there's definitely use cases for it. So like for the wearable use case, there's an open source project called Friend. I helped them integrate Deepgram's real-time speech recognition into that. So, you know, running it on-device doesn't necessarily mean the model has to be on-device.

Yeah, I was curious about the speech recognition models. Because LLMs, of course, they're huge, but the speech recognition models can be effectively turned down to be able to run on-device.

Yeah.

I was curious, like, is the company thinking about getting the embedded model, getting the embedded model space, or the cloud model?

Yeah, it's something we're looking at. You can run our current model on a Raspberry Pi. It's not going to be super fast and it's not going to handle multiple concurrent requests, but it will run. So it can run on CPU and not just GPU.

I have a question.

Yeah?

What services do you recommend for the telephony part of the system? I've used Twilio for various things, but are there other options beyond Twilio?

Yeah, there's a few telephony providers out there. Twilio, Vanash. We're kind of, you know, telephony agnostic. As long as you can get us the audio stream, usually that's achieved either through like, you know, hooking into their API or just doing a SIP trunk.

So SIP trunk basically just hands off the processing of the call to a different server.

Yeah?

When GPT-4o eventually releases their audio-to-audio model, is Deepgram going to be like pivoting around that to build applications around that?

I don't think we would necessarily use it in that regard. So I think it's great for the space that, you know, they're releasing this. I think maybe in the future, this type of multimodal model will make sense. But it's yet to be seen what the price point is going to be and what the latency will be.

Like even their chat completion API for 4o is taking like up to a second. So if you add audio into that as well, that's additional processing. Like it was a really cool demo. And I loved like how the like a single model could have, you know, infinite voices.

And I think that's where we'll see a lot of changes in the future. At Deepgram, our main focus is like scalable, low-cost, efficient inferencing. So, you know, the ability to run at these price points, you know, is probably going to be a barrier to entry.

But yeah, I'm looking forward to see what they release and when. Yeah?

When you implement your solution for a client, how long does it take? Do you need to fine-tune your model?

Yeah, so for model fine-tuning, the question was how long does it take? So we require between like 20 and 50 hours of audio to fine-tune the speech-to-text. And then we actually human-label that. So we have our own team of human labelers.

And what they'll do is they'll actually do three passes. So the first pass usually works out at about like 12% word error rate. So if you give somebody a piece of audio and you ask them to write down exactly what was said, they'll have errors in it as well.

So then we run it through a second pass, they fix the prior errors. And then the third pass is and they're all different people as well. So the third pass goes in and basically gets us to like 99% label accuracy.

And then we train the model. Training the model is pretty quick,right? We could probably do it in under a day. But yeah, getting all that audio and then labeled, and then we kick off the training cycle. For the text-to-speech side, we don't offer cloning, voice cloning today.

I think there's a lot of concerns around, you know, what happens when you clone people's voices. And we do do cloning for certain customers. So if a customer comes to us and says, "Hey, you know, we want to use you, but we need our own voice for our brand," we can do that training.

But that would be a business engagement.

Yeah? Yeah, question over there.

Are you going to use payment API fees or it will be managed by Deepgram?

Yeah, soright now, everything is just on the sandbox API. This API will probably go away after the workshop. But we'll have a way to actually sign up for the API waitlist. So if you do want to get access to this, and then it will require an API key.

And all of the services will be wrapped under a single kind of usage fee. So your speech-to-text, your LLM, and your text-to-speech will all be under a single cost.

Awesome. Any more questions? Yeah, sorry, you go.

Yeah, I was just wondering, are there any plans to allow kind of multiple speaker input within this new API? You know, being able to recognize speaker one, speaker two, speaker three.

Yeah. Yeah, so we have diarization. So if you send multiple speakers on the same channel, we'll be able to determine, you know, speaker A, speaker B. If you're sending us multi-channel audio, that will allow us to inference them separately.

Yeah.

So I was specifically meaning more on the single channel. And I played around with it a bit, but it's not perfect today. I'm just wondering, are there plans to kind of enhance that or expand upon it?

Yeah, we're always improving diarization. It's definitely a challenge because to understand how diarization works is you're building up embeddings of what people say. So like our conversation so far, you've had maybe three or four sentences, you know, maybe 20 seconds of audio.

That may not be enough for the model to say, "Okay, this is a unique speaker,"right? And it's building out these embeddings in like 512-dimensional space. So, you know, as more data comes in, and we typically recommend 30 seconds per speaker to actually generate a solid embedding.

If we were to lower that requirement, we might start like mislabeling people from the same person. But it is a challenge, and I don't think it's ever going to be perfect. You know, one of the hardest parts of diarization is actually when

people actually say like "yeah" or "mm-hmm." So like if you're on a call and somebody, while you're speaking, says "yeah," it's very hard for the AI with that tiny little, you know, segment of audio to know that it's somebody else speaking.

But yeah, we've seen a lot of cases where, you know, if it's a longer call, it works very well. But those first 60 seconds, it's probably not going to determine who's who.

Sure. Yeah, that makes sense. And I imagine behind the scenes, there's maybe some accuracy percentage,right? Like similarity score. Like is that something that might ever get exposed? Or is that exposed today? So we can kind of make a decision ourselves,right?

Where it's we get something back and say, "There's probably a 5% chance that it's this guy speaking." That would be really helpful.

Yeah, and one of the things a lot of people ask for is the ability to, you know, get speaker identification,right? So like a unique identifier for a speaker. So like if you have a call center agent and you know who they are on every call, you know, could you pass that to us and we'll tell you, you know, were they the first speaker or the second speaker?

It's not something we expose today. Obviously, there's, you know, legal challenges around fingerprinting voices and stuff. But yeah, it's something we're thinking about, the ability to at least like identify a speaker and just say like, you know, "This speaker is this person."

Yeah. You're welcome. Awesome. So I think that's everything I had. I'm interested, though, had anybody achieved order update with the remove item?

Anybody get any additional APIs up and running?

No?

Demo running.

You got the demo?

It's a little bit too loud in here.

Yeah.

It picks up you a lot.

Right,right. Yeah.

And it's fast.

Yeah.

I told it to add lots of items back to back and it did.

Nice. Yeah, and this is, again, running on our hosted API. So we haven't even optimized this for low latency yet. But you can see how quick it is even with those hosted APIs that it can respond in that time.

Like we did run a kind of a sandbox environment where, you know, we cranked up that compute and it was just so fast that like it was just like kind of like interrupting you like the moment you stopped speaking, which is a pretty funny challenge.

But yeah, thanks everybody for coming. Feel free to hit me up or chat with me after the workshop. But yeah, hope you all enjoyed it.