# Building Effective Voice Agents — Toki Sherbakov + Anoop Kotha, OpenAI AI Engineer · 2025-07-20 Toki Sherbakov and Anoop Kotha from OpenAI argue that speech-to-speech models have reached a 'good enough' tipping point for production voice agents, highlighting the shift from chained architectures (transcription + LLM + TTS) to the real-time API's speech-to-speech approach. They detail trade-offs across latency, cost, accuracy, UX, and integrations depending on use case—consumer apps prioritize low latency and expressiveness, while customer service demands accuracy and tool integration. Key design patterns include delegating complex tasks to smarter models like O4 mini, prompting for voice expressiveness and tone, and starting with few tools. For evals, they recommend starting with observability, using SMEs for labeling, then transcription-based LLM-as-judge evals, audio evals with GPT-4 Audio to assess tone and pacing, and synthetic conversations between two real-time clients. Guardrails should run async with a configurable debounce period (e.g., every 100 characters). Examples from Lemonade (early focus on evals and guardrails) and Tinder (customization for brand realism) illustrate successful approaches. ## Questions this episode answers ### How have voice AI models improved recently, and what are the two main architectures for building them? Toki Sherbakov explains that voice models less than six months ago were slow, robotic, and brittle, as shown in a painful example where it couldn't handle interruptions. Today's models, like OpenAI's real-time API, are much faster, expressive, and accurate. The two architectures are the chained approach—stitching transcription, LLM, and text-to-speech—which is slow and lossy, and the emerging speech-to-speech approach that combines these into a single model for lower latency and better semantic understanding. [2:16](https://aie.addtry.com/f65ede33-3463-4eb1-bfa2-a30bb7cb66f4?t=136000) ### What trade-offs should you consider when building voice AI for consumer apps versus customer service? Toki Sherbakov outlines five trade-offs: latency, cost, accuracy and intelligence, user experience, and integrations and tooling. For consumer apps, user experience and low latency are critical, while cost matters less. For customer service, accuracy and intelligence are paramount—mistakes like processing wrong orders are costly—and integrations with internal systems and phone providers become essential, even if it means accepting some latency. [5:22](https://aie.addtry.com/f65ede33-3463-4eb1-bfa2-a30bb7cb66f4?t=322000) ### What pattern can voice agents use to delegate complex tasks to smarter models? Anoop Kotha describes a pattern where the real-time API serves as the frontline voice agent, handling simple queries. For harder questions that require more intelligence, the agent makes a tool call to delegate to a smarter model like O3 mini or O4 mini behind the scenes. This keeps the voice interface fast and responsive while ensuring accuracy for complex tasks, such as processing a return within a 30-day policy. [8:52](https://aie.addtry.com/f65ede33-3463-4eb1-bfa2-a30bb7cb66f4?t=532000) ## Key moments - **[0:00] Intro** - **[0:50] Multimodal** - [1:07] OpenAI's Toki Sherbakov says GenAI is entering a multimodal era with speech-to-speech models now at a tipping point. - **[2:05] Audio Progress** - [2:16] Toki Sherbakov demonstrates a voice agent from 6 months ago that was slow, robotic, and misheard an interruption, writing about sand instead of stars. - [3:28] Toki Sherbakov shows a current real-time voice agent that is fast, expressive, and accurately pivots from 'whispers' to 'stars' when interrupted. - **[4:22] Architectures** - **[5:22] Trade-offs** - [5:35] Toki Sherbakov explains that consumer voice agent trade-offs prioritize latency and UX, while customer service demands accuracy and integrations. - **[7:28] Voice Agents** - **[8:22] Delegation** - [8:52] Anoop Kotha proposes a delegation pattern using the real-time API as a frontline agent and smarter models like O4 mini for complex task tool calls. - **[9:54] Customization** - [10:21] Anoop Kotha emphasizes that voice agent prompts can control expressiveness, demeanor, and tone, offering a dimension absent in text agents. - **[11:38] Tools** - **[12:57] Evals** - [14:08] Anoop Kotha reveals that audio evals can measure tone and pacing using models like GPT-4 Audio to capture what text evals miss. - **[15:07] Guardrails** - **[15:52] Conclusion** - [16:42] Toki Sherbakov predicts that now is the first-mover advantage window for building voice agents, citing the real-time API's new snapshot. ## Speakers - **Anoop Kotha** (guest) - **Toki Sherbakov** (guest) ## Topics Agent Evaluation ## Mentioned Lemonade (company), OpenAI (company), Tinder (company), Agents SDK (product), GPT-4 Audio (product), O3 mini (product), O4 mini (product), Riz Chat (product), Sora (product), openai.fm (product), real-time API (product) ## Transcript ### Intro **Toki Sherbakov** [0:15] Hello everyone, thanks for coming to our talk here. I'm just briefly going to introduce myself: I'm Toki, I lead the Solution Architecture team in the U.S. for OpenAI, and I have Anoop here as well on the Solution Architecture team. And we'll talk about building practical audio agents. Before I jump in, maybe a quick raise of hands: who has either, like, built an audio agent or voice agent of some kind, or is thinking about building an audio agent? Okay, so like half-ish of the room—okay, good, good. It's always good to know what people's interests are. So, yeah, we'll talk a bit about, like, some of the things we've learned, where audio models are today, and then some of the kind of patterns and best practices. ### Multimodal **Toki Sherbakov** [0:50] So, just to take a quick step back: GenAI applications, broadly, to date—as many of y'all know—have been very text-based,right? This is models that really deeply understand and generate text with very much human-like fluency. And there's a lot of production applications out there built on text, and that continues to scale. I think we've really started to enter a bit more of this multimodal era, where we now have models that can understand images extremely well. There's now models that can generate images really well—if you were part of the ImageGen Studio Ghibli craze—or Sora that generates videos, for example. And then now also audio. These are models—we've had models like transcription models or text-to-speech models for some time, but now this kind of emerging speech-to-speech capability is getting a lot better. And I would say now we will see, I think, a much more of an explosion of use cases around these modalities of images and video. And today we're really going to focus on the audio modality across this. So, I would say, like, one important thing we're starting to see is that audio is really getting to that point where it's becoming much more usable at scale. And again, we've had the transcription and the TTS side, but especially that speech-to-speech low-latency experience. The models are getting, I would say, good enough, and are kind of at this tipping point where you can start to really meaningfully build high-quality production applications at scale. So, I would say before the models were a lot slower, robotic, and brittle, and now they're a lot faster, expressive, and accurate. I'll show an example. The first one will show what the models were like not even 6 months ago. ### Audio Progress **Toki Sherbakov** [2:16] It's a very painful example, so I hope you feel a bit of the pain in this. Hello. **Toki Sherbakov** [2:23] Hello, please give me a topic for your haiku. Uh. **Toki Sherbakov** [2:29] Well, you can see the initial delay. That's actually, honestly, what the— Could you write me a haiku about whispers? **Toki Sherbakov** [2:41] Silent air, faint words, whispers drift through darkened rooms, secrets linger near. Dust slides through my hands, sifted grains forget the weight, what is left behind. So I hope you felt the pain a little bit,right? So these models are—this is actually not just not even 6 months ago,right? The models are incredibly slow. They're very robotic in how they sound, and they're quite brittle. So in that example, we actually tried to interrupt the model and tell it, no, don't write a poem about whispers, write one about stars. It wrote one about sand. I'm not sure where it got that from. So just showing you that's actually where it was. And then now, if we, like, fast-forward to today, these are the models that, for a very similar use case, kind of how they—how it will behave today. Hello. **Toki Sherbakov** [3:30] Reading. Readings. Share with me a topic, and I shall weave a haiku for you, rich with emotion and imagery. What shall it be? Can you make a haiku about whispers? Whispers. Actually, could you make a haiku about stars? Stars, whispers of light, adorning the velvet night, eternal secrets. **Toki Sherbakov** [3:58] So you can see it's much faster. It's incredibly expressive, emotional. You can steer it, you can get it excited, you can slow it down. And it's accurate. We interrupted it, we had it change course and do something else, and it responded well. So the models are really—we do believe—at that good-enough tipping point where you can start to build much more reliable experiences around it. In terms of actually building on top of the audio models, there's really two emerging architectures that we see. ### Architectures **Toki Sherbakov** [4:22] One is kind of the one that's been around for a bit, and then I'd say a little bit of the newer one, which I'll talk about. The existing one has been what we call kind of the chained approach. This is stitching together these three approaches. You have audio come in, you transcribe it, you have some sort of intelligence layer with an LLM that does some sort of intelligence, and then finally text-to-speech on the output. But again, this is stitching together three models. There's a lot of problems with this. It's slow across time. It takes longer time to actually generate your output audio. It also has some lossiness across this,right? You kind of lose a little bit of the semantic understanding of the conversation. So really this emerging pattern is more the speech-to-speech architecture, where really it takes these three models into a single model, which for us it's the real-time API. And it does all these layers. It does the transcription, it does that intelligence layer, and then finally it does the speech as the output. So this simplifies the architecture quite dramatically, really reduces the time it takes to output. This is really good for low-latency experiences. And it doesn't have that lossiness problem. You actually are able to maintain semantic understanding across a conversation. So this is really kind of the emerging pattern we see. And when building, there's some kind of key trade-offs and considerations to take into account when building on top of these architectures. And we think it's kind of across these five main areas. ### Trade-offs **Toki Sherbakov** [5:35] You have latency, you have cost, you have accuracy and intelligence, user experience, and integrations and tooling. And depending on what you're building, your trade-offs will be different,right? So if, for example, you're talking about a consumer-facing application, the trade-offs are quite different. Here, you care a lot about user experience. These are usually a low-latency experience that a lot of end users are interacting with the model. So user experience matters a lot. Latency matters a lot. It can't be slow. It needs to be really fast and interactive. But things like cost probably don't matter as much. It doesn't need to be accurate. Not really,right? It's more about expressiveness and quick answers. And integrations and tooling, like, you don't really need to integrate with SIP or, like, Twilio for a broader experience. It's a pretty simple interaction. So things like this would really—the real-time API works really well for a consumer-facing application, depending on these trade-offs. Now, if you go to customer service as another example, the trade-offs are different,right? So here, accuracy and intelligence actually probably matters the most. You can't get an order number wrong. If it goes to delete an order instead of updating an order, that's a big deal. So accuracy and intelligence matters probably the most. Integrations and tooling start to matter a lot more too, because now you have to integrate with your various internal systems. You have to integrate with, yeah, SIP and phone providers if needed. So those are probably the most important things. But user experience still matters, but you'll probably take a hit on that a little bit to make sure it's more accurate. Again, latency still matters, but not as much, because accuracy and intelligence matters more. And cost, you're saving a large cost for this anyway. So I think these are just the other considerations to take into account for customer service. So for this, real-time API might make sense for a situation where latency might matter the most. But if you care about determinism and accuracy and intelligence, then actually the chained architecture would make a lot of sense here too. But it's just important to consider what your trade-offs are, depending on what you're trying to build. So hopefully that gives you a paint-you picture, a bit of, like, what to consider. I'll hand it off to Anoop to talk about very specific patterns we've seen when working with customers building on this. **Anoop Kotha** [7:28] Cool. Awesome, Toki. So maybe just, like, taking a step back. I think we've probably defined agent a bunch of times, but here's just, like, a canonical definition to hopefully keep everyone grounded for this session. Think of an agent as some model, some set of instructions—so your prompts, the tools you give your model, and then the runtime—so, like, the guardrails, like, how you do your execution environment. ### Voice Agents **Anoop Kotha** [7:49] And generally, like, when you're building voice agents, that's all the same things you should be worrying about. But there's also a few other things that are maybe a bit different that we're going to focus on today. So the main stuff is, like, your system design is probably a bit different than a traditional LLM agent. It's probably a bit different on the prompting side and customization of the voice. The tools you're using, probably a bit different. And then also evals and guardrails you might have to think about as well. So these are the things we'll probably focus on that are, like, distinct when you're building specifically voice agents compared to, like, traditional agents. Cool. So if you've all built maybe, like, a text-based agent—let's use, like, customer service as a canonical example—you have, like, some user question, you have some sort of triage agent that's, like, a small model that goes to other models here that will do, like, the harder tasks. ### Delegation **Anoop Kotha** [8:37] So you'll have, like, an O3 to do, like, the refund, or maybe, like, an L4 might need to do some, like, cancellation. And typically that's, like, how you build a normal agent today. For voice, one pattern that we're starting to see more and more of is some sort of, like, delegation through tools. So you would have, like, the real-time API be your, like, frontline agent. So it does a lot of the normal interaction patterns that you're used to. So it can respond to, like, the easy questions that you sort of want your model to answer. And you know you're confident in your model answering. But then there's also other questions that might be a bit harder. And then you want your model to, like, do a tool call to these other agents that are run by smarter models behind the hood. So that's a pattern that we'll show a quick video of, of, like, how this would work. I've authenticated you now. What item would you like to return? Yeah, I want to return the snowboard I bought. Got it. I'm reviewing the policy and will get back to you with next steps shortly. **Anoop Kotha** [9:34] So here's a scenario where we're actually using, like, a much smarter model, O4 mini, and delegating to that, because we don't want a smaller model like the original. It looks like the return you are making is within the 30-day return policy. We will process the return and send you a confirmation email with next steps shortly. **Anoop Kotha** [9:54] So that's, like, one pattern we've seen on the architecture side. Another point I touched on is, like, the way you basically customize your brand and what you're sort of building. So the prompts you're using is a huge part of that. ### Customization **Anoop Kotha** [10:05] And the way to think about this is, when you're prompting models today in the text-based world, you only can control, like, the instructions you're giving it and sort of what you want the model to do. But in voice-based applications, you can also control the expressiveness of the voice and how you want the model to speak or sound. So this is, like, an example prompt here of, hey, you can control, like, the demeanor, the tone, the level of enthusiasm, all of those things that don't really get captured in a text agent that you do have much more control over in a voice-based agent. Here's just a fun website you can use to play around with this: openai.fm, where you can play around with different voices and also click through sample prompts that we've created. This is just a great place to get started on, like, actually playing around with the expressiveness of voices and seeing that there is a ceiling and, like, a lot of fluctuation you can get in performance just by the prompts that you do use. Another side of this is, when you're prompting models in the text-based approach, you probably have, like, a lot of few-shot examples. Like, hey, I want you to do these few steps and then do these next few steps. And go through, like, a typical conversation flow. The same thing you can sort of mimic in voice. So if you see in this picture, there's, like, some idea of, hey, I want you to do some greeting, the description of it, and then the instructions for the users. And you can just go through that. So this is, like, a sample way of you to think through how you should be prompting models, especially for more complex tasks that you want the real-time API to handle. The next big bucket of things that are sort of important or maybe, like, a bit different for the voice agents you're building is tools. Tool use in general is, like, very important, because you want your model to connect to other things that are sort of important for you. ### Tools **Anoop Kotha** [11:53] And the first step is, honestly, just start simple. You don't need to connect 10 tools or, like, 20 tools to a given model. Keep the number of tool calls pretty limited for a given agent. And that maybe goes back to the first diagram that we showed of, you want to delegate to different models or agents to do different tasks. So here, a given agent, don't give it a ton of tools to start with and then slowly add on more tools as needed. Handoffs. So if you ever use the agents SDK, this is the concept of, if we have a given model, you want to pass it to another. And here, you can manage context between handoffs as well. So the thing to remember here is, you might have a given agent and you're going to another one, but you want to keep the context the same between them. So a good pattern to do is actually summarizing the conversation state and then passing it to the other agent so you don't lose any of the context and it's not lossy. The final one is delegation. So the concept we showed a bit earlier of, like, using O3 mini or O4 mini to do the tool call. That's, like, natively built into the agents SDK if you'd like to try it out. Evals. So I think this is a pretty big part of the, like, voice experience. And maybe, like, breaking it down into, like, four key buckets. I think the first and most obvious thing is always start with observability. You're not going to get very far if you don't know how your data looks. ### Evals **Anoop Kotha** [13:17] So just make sure you have traces of what's going on with your agents, the audio and everything, so you can actually look at it and take actions. And the next thing is, when you're actually starting to do evals, like, the most important thing is actually have SMEs to label these data and actually go and iterate on prompts from there. It sounds maybe a bit too easy or it's, like, not the most appropriate or maybe the best approach, but it is the most effective that we've seen when working with customers. It lets you go a lot faster. And then from there, the next step is transcription-based evals. So this is where you would have your traditional LLM as a judge evals or, like, testing your function calling performance on certain business criteria and having a rubric in place. And then there's this next bucket where a lot of people ask us, how do we actually test the audio that's generated? And for the most part, it's there's not a lot of things that audio evals get you that maybe text evals don't. But the things that maybe it does get you is, like, understanding the tone, understanding the pacing, or other things that are maybe a bit harder to capture just through text. And this is where you can use, like, a completions model or some other audio model. So, for example, GPT-4 Audio to, like, understand the tone, the pacing, and what other intonations that you care about for your business. And then the final one that we've started to see a bit more is actually synthetic conversations. So, for example, maybe you have two real-time clients going back and forth across to each other. So maybe one is your real-time API agent or whatever, and then the other is a set of customer personas that you have. And you can simulate, like, many, many of those conversations and then extract those evals across maybe, like, transcriptions or audio and then have a much better way to understand what you've sort of built. Cool. And then the final thing here is guardrails. Because, especially for the real-time API, it's pretty fast, you want to have guardrails that make sure that you have the safety that's needed to actually have confidence in the solution you're rolling out to users. ### Guardrails **Anoop Kotha** [15:24] So the main things we have here is, like, run them async, generally because the generated text from the real-time API is much faster than the spoken audio that you get back from the model. So you have room to play with the latency here. The next thing also is, you have control to set, like, whatever debounce period you have here. So in the code snippet, to theright-hand side of the screen, you'll see that, like,right now it runs every 100 characters to run the guardrails, but you can control that if you want to. ### Conclusion **Anoop Kotha** [15:52] Awesome. And then maybe just some learnings that we've seen from other folks in the field. Lemonade. So AI, like, insurance company. They're, like, building one of the things we, like, noticed from them that made them successful is, they, like, focus really early on evals, guardrails, and, like, feedback mechanisms for their team. And even if it wasn't scalable, they realized that and it actually made them move a lot faster in the end. And then finally, for Tinder, they realized that customization and brand realism was pretty important for them when they were launching their Riz Chat experience. So that's what they focused on and it created a much better experience for their users. And now I'll hand it off to Toki to close this out. **Toki Sherbakov** [16:31] Cool. Yeah, just to wrap this all up, so I kind of what I said in the beginning is we're really entering this multimodal era. So away from or in addition to text-based models we have now, obviously video and image models, but now audio. So very much in that multimodal era. And this real-time speech-to-speech technology is emerging. And I think we had a kind of low-key update yesterday. Our real-time API, we had a new snapshot that we released that's actually a lot better. So I really feel like now is the time to build. So you kind of have this first movers advantage to build now. I think the technology is getting to that point where it is good enough to build scalable production applications. So really excited to see what you all build. Please test it out. We'll be around after the talk if you guys have any questions, but thanks for the time. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.