Building Effective Voice Agents — Toki Sherbakov + Anoop Kotha, OpenAI

Hello everyone, thanks for coming to our talk here. I'm just briefly going to introduce myself: I'm Toki, I lead the Solution Architecture team in the U.S. for OpenAI, and I have Anoop here as well on the Solution Architecture team.

And we'll talk about building practical audio agents. Before I jump in, maybe a quick raise of hands: who has either, like, built an audio agent or voice agent of some kind, or is thinking about building an audio agent?

Okay, so like half-ish of the room—okay, good, good. It's always good to know what people's interests are. So, yeah, we'll talk a bit about, like, some of the things we've learned, where audio models are today, and then some of the kind of patterns and best practices.

So, just to take a quick step back: GenAI applications, broadly, to date—as many of y'all know—have been very text-based,right? This is models that really deeply understand and generate text with very much human-like fluency. And there's a lot of production applications out there built on text, and that continues to scale.

I think we've really started to enter a bit more of this multimodal era, where we now have models that can understand images extremely well. There's now models that can generate images really well—if you were part of the ImageGen Studio Ghibli craze—or Sora that generates videos, for example.

And then now also audio. These are models—we've had models like transcription models or text-to-speech models for some time, but now this kind of emerging speech-to-speech capability is getting a lot better. And I would say now we will see, I think, a much more of an explosion of use cases around these modalities of images and video.

And today we're really going to focus on the audio modality across this. So, I would say, like, one important thing we're starting to see is that audio is really getting to that point where it's becoming much more usable at scale.

And again, we've had the transcription and the TTS side, but especially that speech-to-speech low-latency experience. The models are getting, I would say, good enough, and are kind of at this tipping point where you can start to really meaningfully build high-quality production applications at scale.

So, I would say before the models were a lot slower, robotic, and brittle, and now they're a lot faster, expressive, and accurate. I'll show an example. The first one will show what the models were like not even 6 months ago.

It's a very painful example, so I hope you feel a bit of the pain in this.

Hello.

Hello, please give me a topic for your haiku.

Uh.

Well, you can see the initial delay. That's actually, honestly, what the—

Could you write me a haiku about whispers?

Silent air, faint words, whispers drift through darkened rooms, secrets linger near.

Dust slides through my hands, sifted grains forget the weight, what is left behind.

So I hope you felt the pain a little bit,right? So these models are—this is actually not just not even 6 months ago,right? The models are incredibly slow. They're very robotic in how they sound, and they're quite brittle. So in that example, we actually tried to interrupt the model and tell it, no, don't write a poem about whispers, write one about stars.

It wrote one about sand. I'm not sure where it got that from. So just showing you that's actually where it was. And then now, if we, like, fast-forward to today, these are the models that, for a very similar use case, kind of how they—how it will behave today.

Hello.

Reading.

Readings. Share with me a topic, and I shall weave a haiku for you, rich with emotion and imagery. What shall it be?

Can you make a haiku about whispers?

Whispers.

Actually, could you make a haiku about stars?

Stars, whispers of light, adorning the velvet night, eternal secrets.

So you can see it's much faster. It's incredibly expressive, emotional. You can steer it, you can get it excited, you can slow it down. And it's accurate. We interrupted it, we had it change course and do something else, and it responded well.

So the models are really—we do believe—at that good-enough tipping point where you can start to build much more reliable experiences around it. In terms of actually building on top of the audio models, there's really two emerging architectures that we see.

One is kind of the one that's been around for a bit, and then I'd say a little bit of the newer one, which I'll talk about. The existing one has been what we call kind of the chained approach.

This is stitching together these three approaches. You have audio come in, you transcribe it, you have some sort of intelligence layer with an LLM that does some sort of intelligence, and then finally text-to-speech on the output. But again, this is stitching together three models.

There's a lot of problems with this. It's slow across time. It takes longer time to actually generate your output audio. It also has some lossiness across this,right? You kind of lose a little bit of the semantic understanding of the conversation.

So really this emerging pattern is more the speech-to-speech architecture, where really it takes these three models into a single model, which for us it's the real-time API. And it does all these layers. It does the transcription, it does that intelligence layer, and then finally it does the speech as the output.

So this simplifies the architecture quite dramatically, really reduces the time it takes to output. This is really good for low-latency experiences. And it doesn't have that lossiness problem. You actually are able to maintain semantic understanding across a conversation.

So this is really kind of the emerging pattern we see. And when building, there's some kind of key trade-offs and considerations to take into account when building on top of these architectures. And we think it's kind of across these five main areas.

You have latency, you have cost, you have accuracy and intelligence, user experience, and integrations and tooling. And depending on what you're building, your trade-offs will be different,right? So if, for example, you're talking about a consumer-facing application, the trade-offs are quite different.

Here, you care a lot about user experience. These are usually a low-latency experience that a lot of end users are interacting with the model. So user experience matters a lot. Latency matters a lot. It can't be slow. It needs to be really fast and interactive.

But things like cost probably don't matter as much. It doesn't need to be accurate. Not really,right? It's more about expressiveness and quick answers. And integrations and tooling, like, you don't really need to integrate with SIP or, like, Twilio for a broader experience.

It's a pretty simple interaction. So things like this would really—the real-time API works really well for a consumer-facing application, depending on these trade-offs. Now, if you go to customer service as another example, the trade-offs are different,right? So here, accuracy and intelligence actually probably matters the most.

You can't get an order number wrong. If it goes to delete an order instead of updating an order, that's a big deal. So accuracy and intelligence matters probably the most. Integrations and tooling start to matter a lot more too, because now you have to integrate with your various internal systems.

You have to integrate with, yeah, SIP and phone providers if needed. So those are probably the most important things. But user experience still matters, but you'll probably take a hit on that a little bit to make sure it's more accurate.

Again, latency still matters, but not as much, because accuracy and intelligence matters more. And cost, you're saving a large cost for this anyway. So I think these are just the other considerations to take into account for customer service.

So for this, real-time API might make sense for a situation where latency might matter the most. But if you care about determinism and accuracy and intelligence, then actually the chained architecture would make a lot of sense here too.

But it's just important to consider what your trade-offs are, depending on what you're trying to build. So hopefully that gives you a paint-you picture, a bit of, like, what to consider. I'll hand it off to Anoop to talk about very specific patterns we've seen when working with customers building on this.

Cool. Awesome, Toki. So maybe just, like, taking a step back. I think we've probably defined agent a bunch of times, but here's just, like, a canonical definition to hopefully keep everyone grounded for this session. Think of an agent as some model, some set of instructions—so your prompts, the tools you give your model, and then the runtime—so, like, the guardrails, like, how you do your execution environment.

And generally, like, when you're building voice agents, that's all the same things you should be worrying about. But there's also a few other things that are maybe a bit different that we're going to focus on today. So the main stuff is, like, your system design is probably a bit different than a traditional LLM agent.

It's probably a bit different on the prompting side and customization of the voice. The tools you're using, probably a bit different. And then also evals and guardrails you might have to think about as well. So these are the things we'll probably focus on that are, like, distinct when you're building specifically voice agents compared to, like, traditional agents.

Cool. So if you've all built maybe, like, a text-based agent—let's use, like, customer service as a canonical example—you have, like, some user question, you have some sort of triage agent that's, like, a small model that goes to other models here that will do, like, the harder tasks.

So you'll have, like, an O3 to do, like, the refund, or maybe, like, an L4 might need to do some, like, cancellation. And typically that's, like, how you build a normal agent today. For voice, one pattern that we're starting to see more and more of is some sort of, like, delegation through tools.

So you would have, like, the real-time API be your, like, frontline agent. So it does a lot of the normal interaction patterns that you're used to. So it can respond to, like, the easy questions that you sort of want your model to answer.

And you know you're confident in your model answering. But then there's also other questions that might be a bit harder. And then you want your model to, like, do a tool call to these other agents that are run by smarter models behind the hood.

So that's a pattern that we'll show a quick video of, of, like, how this would work.

I've authenticated you now. What item would you like to return?

Yeah, I want to return the snowboard I bought.

Got it. I'm reviewing the policy and will get back to you with next steps shortly.

So here's a scenario where we're actually using, like, a much smarter model, O4 mini, and delegating to that, because we don't want a smaller model like the original.

It looks like the return you are making is within the 30-day return policy. We will process the return and send you a confirmation email with next steps shortly.

So that's, like, one pattern we've seen on the architecture side. Another point I touched on is, like, the way you basically customize your brand and what you're sort of building. So the prompts you're using is a huge part of that.

And the way to think about this is, when you're prompting models today in the text-based world, you only can control, like, the instructions you're giving it and sort of what you want the model to do. But in voice-based applications, you can also control the expressiveness of the voice and how you want the model to speak or sound.

So this is, like, an example prompt here of, hey, you can control, like, the demeanor, the tone, the level of enthusiasm, all of those things that don't really get captured in a text agent that you do have much more control over in a voice-based agent.

Here's just a fun website you can use to play around with this: openai.fm, where you can play around with different voices and also click through sample prompts that we've created. This is just a great place to get started on, like, actually playing around with the expressiveness of voices and seeing that there is a ceiling and, like, a lot of fluctuation you can get in performance just by the prompts that you do use.

Another side of this is, when you're prompting models in the text-based approach, you probably have, like, a lot of few-shot examples. Like, hey, I want you to do these few steps and then do these next few steps. And go through, like, a typical conversation flow.

The same thing you can sort of mimic in voice. So if you see in this picture, there's, like, some idea of, hey, I want you to do some greeting, the description of it, and then the instructions for the users.

And you can just go through that. So this is, like, a sample way of you to think through how you should be prompting models, especially for more complex tasks that you want the real-time API to handle.

The next big bucket of things that are sort of important or maybe, like, a bit different for the voice agents you're building is tools. Tool use in general is, like, very important, because you want your model to connect to other things that are sort of important for you.

And the first step is, honestly, just start simple. You don't need to connect 10 tools or, like, 20 tools to a given model. Keep the number of tool calls pretty limited for a given agent. And that maybe goes back to the first diagram that we showed of, you want to delegate to different models or agents to do different tasks.

So here, a given agent, don't give it a ton of tools to start with and then slowly add on more tools as needed. Handoffs. So if you ever use the agents SDK, this is the concept of, if we have a given model, you want to pass it to another.

And here, you can manage context between handoffs as well. So the thing to remember here is, you might have a given agent and you're going to another one, but you want to keep the context the same between them.

So a good pattern to do is actually summarizing the conversation state and then passing it to the other agent so you don't lose any of the context and it's not lossy. The final one is delegation. So the concept we showed a bit earlier of, like, using O3 mini or O4 mini to do the tool call.

That's, like, natively built into the agents SDK if you'd like to try it out.

Evals. So I think this is a pretty big part of the, like, voice experience. And maybe, like, breaking it down into, like, four key buckets. I think the first and most obvious thing is always start with observability. You're not going to get very far if you don't know how your data looks.

So just make sure you have traces of what's going on with your agents, the audio and everything, so you can actually look at it and take actions. And the next thing is, when you're actually starting to do evals, like, the most important thing is actually have SMEs to label these data and actually go and iterate on prompts from there.

It sounds maybe a bit too easy or it's, like, not the most appropriate or maybe the best approach, but it is the most effective that we've seen when working with customers. It lets you go a lot faster. And then from there, the next step is transcription-based evals.

So this is where you would have your traditional LLM as a judge evals or, like, testing your function calling performance on certain business criteria and having a rubric in place. And then there's this next bucket where a lot of people ask us, how do we actually test the audio that's generated?

And for the most part, it's there's not a lot of things that audio evals get you that maybe text evals don't. But the things that maybe it does get you is, like, understanding the tone, understanding the pacing, or other things that are maybe a bit harder to capture just through text.

And this is where you can use, like, a completions model or some other audio model. So, for example, GPT-4 Audio to, like, understand the tone, the pacing, and what other intonations that you care about for your business. And then the final one that we've started to see a bit more is actually synthetic conversations.

So, for example, maybe you have two real-time clients going back and forth across to each other. So maybe one is your real-time API agent or whatever, and then the other is a set of customer personas that you have.

And you can simulate, like, many, many of those conversations and then extract those evals across maybe, like, transcriptions or audio and then have a much better way to understand what you've sort of built.

Cool. And then the final thing here is guardrails. Because, especially for the real-time API, it's pretty fast, you want to have guardrails that make sure that you have the safety that's needed to actually have confidence in the solution you're rolling out to users.

So the main things we have here is, like, run them async, generally because the generated text from the real-time API is much faster than the spoken audio that you get back from the model. So you have room to play with the latency here.

The next thing also is, you have control to set, like, whatever debounce period you have here. So in the code snippet, to theright-hand side of the screen, you'll see that, like,right now it runs every 100 characters to run the guardrails, but you can control that if you want to.

Awesome. And then maybe just some learnings that we've seen from other folks in the field. Lemonade. So AI, like, insurance company. They're, like, building one of the things we, like, noticed from them that made them successful is, they, like, focus really early on evals, guardrails, and, like, feedback mechanisms for their team.

And even if it wasn't scalable, they realized that and it actually made them move a lot faster in the end. And then finally, for Tinder, they realized that customization and brand realism was pretty important for them when they were launching their Riz Chat experience.

So that's what they focused on and it created a much better experience for their users. And now I'll hand it off to Toki to close this out.

Cool. Yeah, just to wrap this all up, so I kind of what I said in the beginning is we're really entering this multimodal era. So away from or in addition to text-based models we have now, obviously video and image models, but now audio.

So very much in that multimodal era. And this real-time speech-to-speech technology is emerging. And I think we had a kind of low-key update yesterday. Our real-time API, we had a new snapshot that we released that's actually a lot better.

So I really feel like now is the time to build. So you kind of have this first movers advantage to build now. I think the technology is getting to that point where it is good enough to build scalable production applications.

So really excited to see what you all build. Please test it out. We'll be around after the talk if you guys have any questions, but thanks for the time.