Building voice agents with OpenAI — Dominik Kundel, OpenAI

Awesome. Well, hi everyone. My name is Dominik. I work on Developer Experience at OpenAI, and I'm excited to spend the next 2 hours to talk to you all about voice agents. Um, the QR code was already on the slides; if you just entered the room, you want to try to download the dependencies as soon as possible.

So head over to that QR code, or to that starter repository, and follow the instructions to install. That might take a while with the internetright now, so please do that as soon as possible. You have, like, 15 minutes of me rambling about stuff before we get started with, uh, with actually coding.

Um, so I said we're going to talk about voice agents. I want to first put everyone on the same page, because I know we all have different, uh, definitions of agents, and there's going to be a lot of definitions flying around at this conference, naturally.

Uh, so when we're going to talk about agents, we're talking about systems that are going to accomplish tasks independently on behalf of users. And most importantly, um, they're going to be essentially a combination of a model that is equipped with some set of instructions, that then has access to tools that can both be used to work on that goal.

And then all of that is encapsulated into a runtime to manage that life cycle. And that's an important definition, because, uh, today we launched the OpenAI Agents SDK for TypeScript. If you've heard of the one for Python, today we basically released the TypeScript equivalent.

Um, and so we're going to use that, and it maps those exact patterns. So if you're unfamiliar with the Agents SDK, it's basically an SDK that provides you with an abstraction based on the best practices that we learned at OpenAI to build agents.

Um, and it comes with a couple of different base foundational features, including things like handoffs, guardrails, streaming input and output, tools, MCP support, built-in tracing, so you can see actually what your agents did and how they interacted with each other.

And then additionally to those features that are coming from the Python SDK, the SDK we launched today in TypeScript also includes human-in-the-loop support with resumability, so that if you need to wait for human approval for a while, you can deal with that.

And most importantly, native voice agent support. What that means in practice is you can use that same those same primitives that we already have in the Agents SDK, and you can build voice agents with that that handle handoffs, have output guardrails to make sure that the agent is not saying things it's not supposed to.

Uh, tool calling, context management, meaning keeping track of the conversation history so you can use it in other applications. And built-in traces support, um, so that you can actually replay conversations, listen to the audio of the user, and properly debug what happened.

Plus native interruption support, if you've tried to build interruptions, you might know how hard this is. If you haven't, be glad you don't have to. Um, both WebRTC and WebSocket support, meaning it's actually can communicate both on the, uh, on the server for things like Twilio communicate, like phone call voice agents, or directly in the client in the browser.

That's what we're going to use today, using WebRTC. But first, why why would we be interested in voice agents in the first place? Um, one of the things that I'm most excited about is it makes technology much more accessible to people.

There's something magical about being able to talk to a voice a to, like, a voice agent, and just have it, like, see it do things. It's also, um, much more information dense. I can convey information much faster, but also it can contain a lot of information through the type of tone and voice that I'm using, the emotions.

So it's much more information dense than sort of just basic, basic, uh, basic text is. One of the cool things is also it can act as, like, an API to the real world. You can have a voice agent go and, like, call a business for you, and, like, have a conversation with them, where maybe there isn't a there isn't an API for that business.

And so when we talk about building voice agents, there's essentially two types of architectures that have emerged on when building these. The first one is based on your traditional text-based agent, and just sort of wrapping it into a chained approach, where we have a speech-to-text model that is taking the audio and then turning it into text so that we can run our basic text-based agent on it.

And then we take that and we, uh, run that agent, take the, uh, text output, and run it through a text-to-speech model to generate audio that we can play again. This has a lot of strengths. One of the most common reasons why we raised this is it's much easier to get started with if you already have a text-based agent.

You can take that, wrap some audio around it, and you have something that you can you can interact with. But, uh, the other aspect is that you have full access to any model. Text is the main modality that any LLM has, and so you can use really any of the cutting-edge models.

It also gives you much more control and visibility of what the model did, by being able to actually look into the exact text that went in and out of the model. But it also comes with some challenges. Turn detection is one of the one of the big ones, where you need to now take into consideration what did the user hear by the time that they interrupted someone, uh, interrupted the voice agent.

Then translate that part back into text. Make sure that, like, your transcript is appropriately adapted so that the model doesn't think it told the user something that it didn't. Um, chaining all of these models together adds latency on every possible level, and so that's another big challenge.

And then you're losing some of that audio context,right? You're transcribing the audio, and if you've ever tried to convey a complicated topic over a text, you know it's a bit harder than dealing the dealing with the same thing using using your own voice.

And so an alternative to that chained approach is a speech-to-speech approach, where we have a model that has been trained on audio, and then takes that audio, uh, to directly interact on the conversation and, uh, make tool calls.

Meaning there's no transcribing in the process. The model can just natively deal with that audio. And that translates into much lower latency, because we're now skipping those speech-to-text, text-to-speech processes. We can also now have much more contextual understanding of the audio, um, including things like tone and voice.

And all of that leads to a much more natural, fluid, uh, level of conversation. But there are some challenges with this. One of the most common ones is reusing your existing capabilities. Everything is built around text, so if you already have some of those existing capabilities, or a very specialized agent for a certain task, uh, it's harder to reuse those.

Also dealing with, like, complex states and complex decision-making. It's a bit harder with these models, since they've been really focused on improving on the audio conversational tone, less so on being very complic uh, complex decision-makers. But there is a solution that we can get around with this.

Um, again, taking inspiration from what we do with text-based agents, we can actually create a, um, delegation approach using tools, where we have a front-line agent that is talking continuously to the user, and then that one uses tool calls to interact with much smarter reasoning models, like O4 mini or O3.

Uh, actually, let me, at this point, give you a quick demo and see how the internet goes here. Um, so I have a real-time agent here that we built with the Agent SDK. It's going to be very similar to what you're going to build later on.

Um, but when I start talking to this,

hello there.

Hello there. How is the AI Engineer World's Fair going?

So we can now give it ta like, give it tasks to, like, call tools that I gave it. Like, hey, what's the weather today?

Let me check the weather for you. One moment, please. Transferring you to the weather expert now. Actually, I can directly help you with the weather information. Could you please specify the location you're interested in?

Oh, yeah. What's the weather in San Francisco? So you can see here it's actually dealing with the interruption.

Today is sunny. Enjoy the bright and pleasant day. Is there anything else I can assist you with?

No, that's it. Thanks.

You're welcome. If you need anything else, feel free to ask. Have a great day.

And so in a similar way, we can actually trigger the more complicated back-end agents as well. So I have a tool for this to handle refunds that will call out to O4 mini and evaluate the refund. So, hey there.

Um, I have one more thing. So I recently ordered this skateboard that I tried to use, and it seems like I'm really bad at skateboarding, so I want to return it. Um, it is slightly scratched, though.

I'm here to assist, but it sounds like you'd need customer service for that. I recommend contacting the company where you bought the skateboard. They can provide you with the return.

Looks like I didn't add the tool. Maybe I did? Oh, I asked for I didn't ask for a refund. Um, let's try this once more. Hey there. Um, I bought a skateboard recently that I tried, and apparently I'm really bad at using it, so I wanted to return it.

It is slightly scratched, so can you give me a refund?

Hello there. How is the a hello there. How is the AI Engineer?

The joys of internet. Um,

hey. Um, I recently ordered a skateboard from you, and it, uh, failed. Like, I ca I can't use it. I'm struggling to use it. It's slightly scratched. Can you give me a refund, please?

Hello there. How is the a I'm going to assess your request for a refund.

There we go.

Let's get started.

It is slightly struggling with this, like, weird echo that we're having here.

The skateboard arrived damaged, and you're eligible for a full refund. We'll process that for you.

Allright. Um, but you can see here that, uh, it was able to call that more advanced tool, and, um, actually process that request. And one of the nice things is that, like, while time to first token is often a really important thing, the longer a conversation goes, like, your model is always going to be faster than the audio that has to be read out.

And so this this is, like, a really helpful thing, where by the time that the model was able to say, like, hey, I'm going to check on this for you, it already had completed that LLM call to the O4 mini model to get the response there.

Allright. Um, let me oh, one more thing. Since we talked about traces, one of the nice things now is we can actually go back here into our traces UI. And this launched today. Um, you'll be able to actually look for any of your real-time API cases, look at all the audio that it dealt with, um, and all the tool calls.

So we can actually see here that the tool call was triggered, what the input was, the output. We can listen to some of the audio again.

I'm going to assist.

Um, to understand what happened. And then because both this and the back-end agent use the Agents SDK, we can go into the other agent as well, which was the O4 mini one, which we can see here. And we can see that it received the context of the full past conversation, the full transcript, um, as well as additional information about the request, and then generated the response here.

So this allows us to then get a full complete picture of, like, what happened both in the front end and the back end. Let's jump back into the slides and cover a couple of more things before we get coding.

Um, and that's about best practices. So I would group when you're, uh, group the best practices of, like, building a voice agent into three main things to keep in mind. Uh, the first one is to start with a small and clear goal.

This is super important because measuring the performance of a text-based agent you will hear a lot about evals at this conference, uh, is already hard enough. But with voice, uh, voice agents, it's going to be even harder. So you want to make sure that you're very focused on, like, what is the first problem you want to solve, and keep it focused on that.

And give it a no like, limited number of tools so that you're fully centered on this. The Agents SDK makes this really easy, because you can then later on add additional tools to additional agents and deal with, like, handoffs between them.

Uh, but this way you can kind of really stay focused and make sure that one of your use cases is great, and then hand off other ones to human agents, for example. The second one is what I elaborated on, which is building evals and guardrails very early on, uh, so that you can feel both confident in what you're building, but also confident in, um, that it's actually working, so that you can then continue to iterate on it and know when it's time for you to, like, grow the complexity of your voice agent.

Uh, as of today, you can use the traces API for uh, the traces dashboard for that. Uh, but alternatively, some of our customers have even built their own dashboards, like Lemonade, to really get an end-to-end idea of the customer experience, and then even replay some of these conversations with their agent as they're iterating on it.

The other thing that I'm personally super excited about with these models is, uh, both our speech-to-speech model and our text-to-speech model are generative models. Meaning you can prompt them the same way that you can prompt a LLM around tone and voice, and you can give it emotions, roles, personality.

Uh, we built this little microsite called openai.fm. It's a really fun website to play around with, where we have a lot of examples of different personalities and how that, uh, that style of prompt can then change what is being read out by our text-to-speech model.

And so that's a great way for you to not just limit, one second, uh, limit the experience of your model, um, or, like, the personality of your model by the voice that you picked, but also by the prompt and instructions that you're giving it.

There was a question there. Would you mind using the mic that isright behind you, just so that it's on the recording?

Hello, sir. So my question regards to the previous slides on Lemonade. So you're displaying how they have this dashboard where they can show all of this. Is this a dashboard that OpenAI provides, and Lemonade just integrates it, like, an iframe or something?

No. So in this case, they built their own, um, solution for it.

Okay. And does OpenAI then provide all the JSON or the data structure that we can just plug into the?

So the the way the real-time API under the hood works is that you get all the audio data, and you can do whatever you want with that, basically. You're getting all the necessary audio events, so you can use those data structures.

So we're not storing them by default. You can use the traces AP uh, the traces dashboard. We don't have an API for it yet, but you can use the traces dashboard to, um, get a a basic look of, uh, of that.

But it's not iframeable.

But you mentioned it's only audio data. This shows not just audio, but also the transcription and all of that as well,right?

So the the traces dashboard, if we go back to it, um, does show, um, all of the transcripts and stuff as well, as long as you have, um, transcription turned on, which I don't seem to have turned on for this particular one.

Um, but it should like, you can turn on transcription, and you should be able to see the transcripts as well.

Okay. Thank you.

You're welcome.

Allright. Um, let's go back to this. Um, the other part with it is, as I said, you can prompt both the personality. You can also be very descriptive with the conversation flows. One of our colleagues, um, found that giving it conversation states and sort of this JSON structure is a great way to help the model think through sort of what processes and what steps it should go through, the same way that you would give a human agent a script to operate on.

If you're struggling to write those scripts, though, uh, we also have a custom GPT that you can use to access that. And I'll share all of those links and a copy of the slide deck later on in the Slack channel, so if you're in that, uh, you should be able to access those.

But with that, um, that's primarily what I wanted to talk through from a from a slides perspective. So from here on, what I want to do is, um, build with you a voice agent. We'll see how that goes with the internet.

Um, also, if you have headphones, now is a great time to bring them out. It's going to be really weird when we're all going to talk to our own agent. Um, but we're going to try this and see how that goes.

Um, so if you came in later, please scan the QR code, go to that, uh, GitHub repository, and set that up, install the instructions. There is no code in it yet other than, like, a boilerplate Next.js app and a, um, empty, like, package JSON that installed just, like, the dependencies that we needed so that we are not all trying to run npm install at the same time.

But what I want to do is build a first agent. Um, so if you want, you can just straight up copy the code that is on here. But I'm going to, um, actually go and type it along with you all so that you get a feeling for what's happening and we have a good idea of, like, timing.

So if you want to take a picture now and just code ahead, do that. And otherwise, uh, I'm going to switch over to my code editor, and we're going to do this together.

Oh, and if you're running into trouble, the Slack is a great way to post questions that are technical questions. And Anoop, who's over there, um, is going to try to help you. Um, alternatively, raise your hand, but it's a bit easier if you're just Slacking the messages there, and we can kind of multi-thread the problem.

Um, allright. Let's go and build an agent. So if you cloned a project, you should see an index, uh, .ts file. Go and open that, and you should be able to import the agent class from the openai/agents package.

That's what we're going to use to create the first agent. Yeah.

Can you use live mode? Because it flies.

Oh, yeah. Good call. Um,

is that better?

Yeah.

Cool. That seems a bit it seems worse on my side than yours, but I think as long as you all can read that, I'll be fine. Um, allright. So what what I want you to do is go and import an agent, and we're going to define our first agent.

And as I mentioned, um, primarily the an agent has a co like, a few centerpieces, the first one being a set of instructions on what to do. So we can give it instructions. I'm going to say, you're a helpful assistant.

That's sort of the most boilerplate thing you can do. We do need to also give it a name, and that's, uh, so that we can actually keep track of them in our traces dashboard. I'm going to say my agent here.

This can be anything that helps you identify it. And then we need to actually execute this agent. So we can import a run function here. And then

we can await the run here. I'm going to

run this agent with just, Hello, how are you, and then log out the results. And with the results, we get a lot of different information because, uh, essentially, when we run an agent, it's going to do a lot of different tasks, from executing all necessary tool calls, if there are any, to validating output guardrails, et cetera.

But one of the most common things that you just want to log out is the final output. That's whatever the last agent in an execution said. Um, so in this case, it's going to be, uh, a set of text.

And then you should be able to run npm start, um, oh, npm run start

01. And that should execute it. And then you should see something like this, um, depending on what your model decides to generate. And by default, this is going to run GPT-4.1, um, as the model. But if you want to experiment with this, you can set the model property here, and we can set it to

O4 mini, for example, and then rerun the same thing. So this is the most basic agent that you can build. But one of the things that really makes something an agent is if it can execute tools. So we can import a tool here.

There we go. And we can define a get weather tool. Um,

one of the things here is you have to specify what arguments the model is going to receive. And one of the ways that you can do this is through a tool called Zod. If you've never heard of it, it's essentially a way to define schemas.

Um, and what we'll do is we'll both use that Zod schema to inform the model on what the parameters for this function call are, but we're also going to use it to validate then what are the actual arguments that the model tried to pass in, and do they fit to that schema.

So we get full type safety here, if you're a TypeScript developer and you care about that. So in this case, we have, um, a get weather tool, and then we can give that tool to the agent, and we can change this to, what is the weather in Tokyo?

Uh, is what Cursor wants to check. So if I run this again

let me move this slightly. We can see it's going to take a bit longer now, and that's because it ran some tools. Um, and now it's telling me the weather in Tokyo is sunny. And if you're wondering, well, did it actually run a tool, we can go into

our traces dashboard here

and look at the trace. We have a my agent here.

And in there, we can see it ran, tried to call the tool, executed the tool, and got the weather in Tokyo is sunny back, and then took the response to generate the final response. So the traces dashboard is a great way for you to see what actually happened behind the scenes.

How are we feeling? Can I get a quick temperature check? Are able are people able to follow along? I see Zeke is giving a thumbs up there. So, um, allright. So this is a text-based agent. I wanted to show you this just to get a bit familiar with the, um, overall Agents SDK so that we can jump into building voice agents.

Uh, the first one we're going to build or, like, the first thing we need to understand about a voice agent is the slight differences between a voice agent and a and a what we call a real-time agent. Essentially, a real-time agent is just a specialized version of an agent configuration.

There's just a few fields you can't pass in. But they can be used in what is a what is called a real-time session, because with voice agents, there's a lot more things to deal with than just executing tools in a loop.

Um, one of the most important things is you need to deal with both the audio that's coming in, uh, process that, and then, like, run the run the model with that, and then generate deal with the audio that's coming out.

But you also need to think about things like guardrails, handoffs, other life cycle things. And so the real-time sessions are really dealing with all of that. So let me show you how that works. Um, for this, what we're going to do is we're going to go in the same project.

There's a 02, and it has a page TX uh, TSX in there. Um, this is a Next.js app that, really, I just gut it to have, like, the bare minimum in there. But this is a great way for us to just, uh, build both the front end and the back end part of the, um, voice experience, because this voice agent that we're going to build is going to run in the browser.

In order to make sure we're not leaking your API credentials, one of the important things is you need to use an ephemeral key. That is a key that is short-lived and is going to be generated by your server and handed off to your client so that they can use that to interact with the real-time API over a protocol called WebRTC.

Um, for that, you should see a token.ts file in your repository that just calls out to the real-time API to generate a session and then return a client secret, which is that ephemeral key that we can then use to authenticate with the SDK.

You do not have to do this if you're building a real-time agent that is running on your server, for example, in the case of a Twilio app or something else, where you can just directly interact with the, uh, OpenAI API key.

But if you're running anything in the browser, then you actually need to generate this client key just so that you're not, you know, giving your API key to the world. So with that in here, we can actually go and build our first real-time agent.

So similar to previously, we're going to import a an agent class here, but in this case, it's going to be a real-time agent.

And we're going to import it from the real-time package, uh, which is just a subpath in the same package, so you don't need to install a different package here. But now we can define a real-time agent that works the same way.

We have a name. We give it instructions. Um, just sort of going with a default suggestion here. And now we actually need to connect that agent to a real-time session. So I have a connect button here for running this example.

Let me start it up here with npm run start 02. That command should be in your README as well. Uh, it's going to start a development server, and we can go over here, reload this, and you can actually see it just has, like, a little connect button that,right now, it doesn't do anything.

So let's connect that up. I don't need this anymore. Um, so let me just move this to the side.

Um, so in this onConnect function, it gets triggered whenever we press the button. We want to deal with that connected state. So we're what we're going to do here is we're first going to fetch that token. And this code what this basically does is it's going to import that server action, which is a Next.js concept that just makes sure that, like, this code is going to run on your back end.

If you're using a different framework, you you should be able to just go and fetch this key from your back end server. Um, and then once we have that token, we can go and create a new real-time session.

So what we're doing here is we're going to give it the first agent that should start the conversation up. Uh, I'm going to specify the latest model that we released today along with the Agents SDK. This model is a if you've used the real-time API before, it's an improvement, especially around tool calling.

It's much better on that front. Um, we have a couple of different customer stories on our Twitter if you want to check that out. Um, and then I'm going to give it not there. I don't know why Cursor insists on that.

Um, the last step that we need to do is we need to connect to that session. So this is where we're going to give it that API key so that the mo that we can connect to the real-time session under the hood.

Just so that it's easier for us to deal with all of this, I'm also going to close the session. But I've got one thing here that is an oddity of, uh, React. We do not want to generate that session every time.

So I'm going to on every re-render, so I'm going to create a what is called a ref here. Again, if you're new to React, um, this basically is just a variable that's going to persist through re-renders. So we need to slightly change this here, where we're going to assign that to session.current so we can maintain that.

And then that also allows us to say, if there is a session.current set, uh, we want to actually close that connection when we press the disconnect button. That just makes sure that we're disconnected from the audio again.

So I'm going to leave that on the screen for a second, and then we can test this out. But if you already typed this, go into your browser, refresh, press connect, and you should be able to talk to your agent.

Allright. Let's try mine. Let me move this to the other side so it's not blocking your cut.

Hello?

Hi there. How can I assist you today?

Allright. So you can see, um, there's just a few lines of code. We didn't have to deal with things like figuring out how to set up the microphone, how to set up the speakers. By default, if it's running in the browser, it will do deal with all of that automatically.

If you do want to pass in your own microphone source or other things like that, you can do that as well. Um, if this is running on a server, you have a, um, both a

send audio function that allows you to send an audio buffer in, or you can listen to the audio event, which is going to emit all of the audio buffers that are coming back from the model so that you can pass it to whatever your source is.

Um, so that's our first basic agent. Any any questions so far?

Can you send the date to Red Hole?

Hmm?

Can you send that code to the Red Hole? To their boss? Sorry.

He wants you to send the code to Red Hole.

Can you push it? Can you push it up?

Uh, I can push it, yeah.

I'll copy it to the client.

Good call. Thank you. Um, allright. So now that we have that, let's go and actually give it a tool. So this is really where the benefit of the, um, Agents SDK comes in. We can actually use that same tool definition that we did earlier.

Um, so I'm just going to follow the autocomplete here. We should be able to just give that tool now to our agent and save. Oh, I need to import Zod again to do that schema validation. This is especially important on the real-time side because the real-time model currently do not does not support strict mode.

So the JSON might not fully comply with your, um, um, schema unless you're running you're giving us a Zod schema, and we'll go and validate that this actually fits that schema. So that makes your code a bit easier.

So with that, we can go back. Hey, what's the weather in San Francisco?

The weather in San Francisco is sunny today.

We can disconnect it here. Um, also, this does now deal with interruptions. So, hey, can you slowly count to, like, 20?

Sure. I'll count to 20 for you. 1, 2, 3, 4, 5, 6.

OK. Stop.

Sure. I'll stop counting.

How far did you count?

I counted up to 6.

It's pretty pretty close. It's always hard with the timing to, like, perfectly get it. But, uh, normally, that's enough to, um, deal with the context. But it is super crucial to have that interruption timing so that, like, your model doesn't think it read out, like, the full customer policy, um, but the customer interrupted it halfway through, for example.

Um, allright.

Question. You don't have to, um, manage all the events to actually do that anymore?

No. Um, so the the real-time session will handle all of those events. That is a great proxy to if you are curious about these events, um, that you're alluding to. What we can do is listen to the

transport event and let's do this later. Uh, this will log out all of the events that are happening under the hood. So if we open the dev tools here and rerun this.

Hey.

Hey there. How can I help you today?

So you can see all of the events that normally you would have to deal with are being dealt with. You still have full access to them, so you can both read them, but you can also send events yourself.

So it's going to handle all of the things, but continue to par uh, pass them on to you if you want to do your own logic on top of that.

I'm going to push that code for you so you can pull it.

Cool. Um, allright. Since we already have this commented out code, the other part of this that typically is a request that you want to deal with is, I want to show, like, the transcript. I want to see what sort of is being transcribed.

And the important thing here is I'm using the word transcribe because even though the speech-to-speech model is dealing with the audio directly, um, and there is no transcription sp a step in between, by default, we're going to transcribe all of the conversation at the same time.

Um, you can turn that off if you want to. Um, if you're using the API directly, you have to actually turn it on. Um, in the Agents SDK, it's turned on by default because it's such a common request, and it enables us to do a couple of additional features that we'll cover later on.

But this is going to give us that whole history every time. So I'm just going to, um, log that history here.

Or rather, I'm going to there we go. Import that. I'm going to set that as a variable. And then because it's React, we can let's create a list here. We're going to go over all of this. I need to filter because it has both tool calls and messages.

And I only want to show the messages for this. Um, so you should be able to does it why is it like that? Um, let's see. Close this. Refresh. Hey.

Hello. How can I assist you today?

How's the weather today in San Francisco?

The weather in San Francisco today is sunny. Anything else you'd like to know?

So you're automatically getting that conversation. If you are interrupting the model, one of the things that happens is the transcript is going to disappear. And that's because, uh, the the model currently does not adjust that transcript, and instead, it's going to be removed.

And we're going to remove it from that object as well just so that you get the most accurate representation and you're not thinking that, like, the model read out a certain piece of text. And again, with everything that we're doing here, we can actually go back into traces, and we can see that same representation here with the weather call and everything.

So, again, helps with the debugging.

And go briefly back to the slides. So we covered we set up our first agent. Yeah.

How do you store the conversation history?

The question was, how do you store the conversation history? Um, so it's currently fully stored in memory. Um, so basically, there's going to be a bunch of events that are that I logged out that are being emitted by the real-time API.

All of those are going to be sent over to the client and then stored in memory in a conversation, like, in just like an array, essentially. So you can do whatever you want with that by listening to that history updated event.

So if you do want to store it somewhere, you can store it. Um, the other part is the traces part is automatically going to be stored on the OpenAI, uh, platform as long as you both enable that tracing.

You can disable it. Um, by default, in the Agents SDK, it's enabled. And then, um, the other aspect of that is if you are a ZDR customer, so a zero data retention customer of OpenAI, you don't have access to that traces feature.

Um.

How about the conversation context, like the session? How how how how many previous sessions of voice conversation do you have to?

Um, so the question was how much of the, like, voice context, how much of the con previous conversation is being used. Um, that's going to depend and sort of, like, dealt with directly by the real-time API. So, like, the real-time API, when you do start that session, um, that holds the source of truth for that whole conversation session.

So what you're receiving on the on the client side is just a copy of whatever is happening at that point. It's not the source of truth of what we're going to adapt to pass into the model.

So the difference of off or open is that you lose all the conversation that happened all the voice, uh, that happened before. Like, if I have the voice like, if I have a conversation and then AI AI, I will have the all the conversation from the previous old session.

Is it inference?

Um, the question is, how does it work with, like, the inference cost and, like, whether you're passing it like, passing in that whole conversation. Um, and Noop is nodding. Um, he's the bigger expert there. But yes, um, we're actually like, you can log the the like, we're keeping track of the usage.

There's an event that you can, like, log out to see your token cost. Um, so you have an idea of, like, what is being actually passed in. So, like, with every if we're going back here to this example, you can see these response done events.

Um, I don't know. Where is the um, shouldn't it be on the response done? Uh, it is being sent over. I just do not knowright now. It's not showing. Oh, there. So you can see here, um, it outputs the detailed information of your tok token usage at any point in time.

So while you don't have, like, access to, like, what is exactly what was passed into the next response generation, um, you can keep track of the cost as it's happening.

You're welcome. Uh, yes. Do you there's a microphoneright over there. That might be easier than you yelling across the stage to me.

I I see that the format that you're using is PCM 16. Is there a way in which we can modify the output formats of the audio files so we can save in memory?

Um, yeah. There are different different audio modes that you can use, um, including, like, for example, ULAW for that is, like, helpful for phone calls, for example.

Sorry. Uh, another question on the usage. Um, does that, like, final assistant response roll up all the tokens from, like, all the intermittent tool calls as well? Does that make sense? Like, the agent needs to, like, kind of reason through and then format tool calls.

So I'm assuming it's not just the output tokens for only the assistant response,right?

It like, every tool call is a response in general as well. So, like, it it works the same way that, like, the responses API works, for example.

OK. So, like, eachright? Because we're using this and we have, like, tool calls and tool call outputs,right?

Yeah.

And I couldn't find the, like, usage attribute on the tool call output. Is it somewhere in those, like, raw events that are outputted?

Do you do you know, Noop?

OK. No worries. I I know it's, like, kind of early on.

We can we can follow up on it.

Allright. Thank you.

You're welcome. Uh, yeah. Do you want to head over to that microphone that isright behind you? It just makes it a bit easier.

Is that what that was?

Oh, yeah. There's in the meantime.

Just a quick question. Can I go back to the slides explaining the different modes of the, uh, the audio agents? Like, the text in, text out. That's the first one.

Oh, yeah.

Text to speech. That's the second one. Uh, I didn't get the third one. And, uh.

Oh, you mean this?

Yes.

Yes. It's uh, when you just showed us the GPT-4 real-time, that one.

Yeah.

That model, is that, uh, this PPT this slide is about?

Yeah, exactly. So, like, where it's, like, when we did the refund, um, it kind of followed this pattern where it performs a tool call. Like, the, like, real-time API agent can perform tool calls. It performed a tool call to trigger a separate agent that was the refund agent that, in my case, used O4 mini to execute that task and then hand that back.

OK. Got it. Thanks.

You're welcome. Yes.

I'm I'm currently using, like, regular, uh, OpenAI Agents. So what will be the challenge that I will face when I will when I will want to change my regular my regular agents to, uh, real-time agents?

So there's a couple of different challenges. Like, one is, like, anything that you're doing around latency. Like, like, anything you're, uh, doing around, um, voice, latency is always king. So, like, you want to figure out what are the best ways to I actually have a slide around this.

Um, like, when you when it comes to things like tool calling, you want to find ways to do things like buying yourself some time. So you will typically see some prompting around, like, announce what you're about to do next, um, before you're doing it.

And that's to do that little trick around, uh, while the while the previous audio is still being read out, the agent can already perform the tool call and wait for the tool call to come back. Because similar to a text-based agent, the model can't do like, can't receive additional data as, like, like, do another thing.

Um, outside of, like, we can interrupt the response, but it can't finish that response, if that makes sense. And so you want to do these sort of, like, buying time. The other thing is, like, if you're building a real-time agent, the longer your prompt gets, at one point, it increases the likelihood that it gets confused.

So you want to make sure you're properly scoping those use cases and, like, through what what we call handoffs where you have different agents that are more specific scoped to specific steps in your experience.

Thank you.

You're welcome.

Yes.

Just to go through more about memory. Earlier, you said that some of that permissioning will be stored in memory. Is that short-term, long-term, session memory? Like, like.

Yeah. So you're, uh.

Like, a memory tool on top of that agent?

Yeah. So the question is about memory. Um, basically,right now, the the the let me correct this. Um, when we go back to this demo, what you're seeing here is essentially just, like, a copy of the events that we're receiving back.

So this is, like, helpful as a visualization of the history. Uh, that being said, the actual, like, memory in the sense of, like, an LLM agent memory is the session context that is happening on the real-time API side.

There are events that you can use to update that. Um, we actually have an update history event that you can pass in what you want the history to be. But what that does is essentially, like, fire off events to the real-time API to say, like, delete this item from the history or add this new item.

Um, and you can give it a previous item ID. So, like, you can, for example, like, slot messages into a specific spot if you wanted to. Um, does that make sense? But there's, like, no, no, like, advanced, like, long-term memory solution like you like you were alluding to.

Cool. Yes.

Hi. Do you have tips for handling input from low-fluency users? Like, say someone who's just learning a language and they have, like, multilingual input and maybe broken grammar and their pronunciation's not so good.

Uh, I don't think I have any, like, best practicesright now that I could share.

Can it handle it just off the shelf or?

Um, it can handle, like, switching languages and and things like that.

OK. But it might not be able to handle low-fluency.

Um, I don't know if we have any use cases.

Yeah. We have we have some customers that are, like, language learning companies. So there there is some that are using it that way. But I don't think I have any, like, best practices that I can share.

OK. Thank you.

You're welcome.

Sorry. Back in the code, is there a callback for the interrupt and does it include the last transcription?

Um, there is a call for the interrupt, but, uh, there is no, um,

there is no actual, like, event that.

There's no param or E that comes with it or anything like that?

No. There's currently no, um, transcript. So what you can do is, if you're getting this, you can call.

Get history or something?

Uh, there's just history.

OK.

Um, so, like, this always is up to date.

Cool.

Um, so you can you have access in that moment.

OK.

The the thing that we do have is, um, for tool calls specifically, um, you're getting some additional context. And that context has a, um, history parameter that you can, like, push into.

OK.

Um, it's more documented in the in the documentation.

In the API.

Yeah.

OK. Great. Thank you.

You're welcome.

Awesome. Um, let's move a bit on and show a couple of other things. So we talked about tools. As I said, like, one of the benefits is you can reuse the same syntax that you're doing with text-based ones.

Um, it's also a good way for you to then communicate with your back-end systems using HTTP. Um, follows sort of a, um, general practice around, like, keeping both the tool calls as low latency as possible. Like, send out a tool like, for example, if you know a task is going to take longer, start the task, give it a task ID, and have the agent have a tool to check on the status, for example.

Like, that helps getting back to it because, again, while the tool call is going on, the model is sort of stuck. So you want to you want to make sure to, like, get get back to that as soon as possible.

Um, one of the other things that you can do is human appro uh, hu uh, human approval. I can show you that quickly. There's essentially a use uh, it needs approval that, um, you can either specify as a function that will be evaluated before the tool ever gets triggered.

This is a great way if you have, like, a more complex logic on, I need approval for this. You can also give it just straight up, I always need approval, at which point there is a another event here, um, tool approval requested.

And then that gets a, um, event here so we can do things like, um,

go to old prompt. Um, and then we can go and approve that tool call again.

I don't know why the autocomplete is not working.

Um,

yeah.

Approve. There we go. And, uh,

why is it

oh, this is wrong.

This is where I go into the docs because I do not remember why this is autocompleting the wrong way. But everything I'm showing you is in the docs. Um, so we can just

oh, took the wrong thing,right? The first

there we go. Approval request.

And then we can do.

Approval request.

Approval request. Thank you. It's like the classic thing when you're on stage and you can't really

there we go. So in this case, I'm just going to always approve. But if we now go in hey, um, can you tell me the weather in Seattle?

So we can, in that case, approve it. It's always going to approveright now because I'm not actually checking the status. But, um, that means you can build, like, a human-in-the-loop approval experience. This is really convenient, especially if you're running it in the browser and you just want to have, like, a confirmation of, like, the tool is hallucinating things before the customer actually submits it.

And does it do it directly? Can it actually say, you okay if I do this or not?

Um.

Or does it do that and then do voice text to voice and all that stuff?

So the the basically, this is happening uh, so the question is, does it automatically do this? Like, the what we're doing and the reason why this is separate is the model is asking for this tool to be executed, but we're intercepting this, um, basically before we're ever generating or executing the response.

This is intentional so that, like, you don't have to deal with like, we want you to think through why should this tool need approval as opposed to doing that somewhere halfway through your tool execution and you have to, like, deal with the consequence of rolling back every decision that you've made, for example.

And so by default, if this is just needs true, it cannot get past that until the execution was approved, at which point it stores it in the context that is stored locally and then bypasses that security. So this is not happening on the on the model level.

Um, I'm going to remove that again. Uh, so the other thing we talked about already, but I want to show it in practice, is handoff. So a handoff is essentially just a specialized tool, uh, call that resets the configuration of the agent in the session so that, um, we can update the system instructions, we can update the tools, and make sure that we can nicely scope the tasks of what we're trying to, um, solve.

So what you cannot do I know people are probably going to ask about this is you can't change the voice of the agent mid-session. You could define different voices on different agents, but the moment that you're like, the first agent that starts talking, that's the voice that we're going to stick with throughout the entire conversation.

So that's a caveat to just keep in mind. Um, but they're still very helpful to, let's say, have a weather agent here.

An agent squared.

Sorry?

You can have an agent squared.

Um, and what we're going to do in this one is call the weather agent. Um,

um,

let's give we'll do this. And then what we can do is we can actually give it a handoff description. So if you don't want to have this in your system prompt, but you just want to help the model understand when to use this, you can say, like, uh, this agent is an expert in weather.

And then this one is going to have the weather tool. We're going to remove it from this one, and we're going to give it a handoff instead to that other weather agent. So now, if I'm going to restart this, hey, can you tell me the weather in, uh, New York?

The weather in New York is sunny, so you might want to grab your sunglasses if you're heading outside. Enjoy the day.

Allright. That's that's the model's best attempt at a New York accent. Uh, we'll take it. Um, but you can see there that, like, it automatically handed off from that first agent to that second one and let it handle it.

You can, through prompting, do things like, do you want it to announce that it's about to hand off? Do you not want to do that? Sometimes it's a bit awkward if you're forcing it to always do it. So, like, I would not necessarily try it, but maybe that's the type of experience that you'll want to have.

Um, so that's handoffs. Let me do you a favor and push that code.

Push, baby, push. So, uh, the agent can't change voice when passing to another one, but it can change accents?

Yeah. So, uh, that's a good question. The vo uh, the the agent can can't change the voice, but it can change the accent. Um, that's because the, um like, again, this goes back to, like, the model is a generative model.

So you can prompt it to have different, like, pronunciations, tonality, like, voice in that sense, but it cannot change the voice model that is actually being used to generate that output.

So may maybe as a summit of that, is it, like, the whole real-time request body that can't be changed or just the voice? So can I, like, create a tool that could adjust the speed if someone was saying it was talking too fast for the the noise reduction?

You should be able to change I I have not tried the speed because uh, speed parameter changing it mid-session because it literally came out today. Um, uh, I don't know enough of you tried this. No. Um, but, like, you can like, essentially, a handoff does change the session configuration.

Like, if we look back at, like, um, the

like, one of the transcripts here, like, now that we have a handoff, um, let's go to this trace. So you can see here that, like, it called the transfer to weather agent. Um, but then, like, the this these instructions were talk to uh, talk with a New York accent.

Um, so in this case, it did change the instructions midway through the session. And the same way, like, when that handoff happens, we take away the tools. We give it new tools. So you can change those tools. You could have a tool to change the tool, but my recommendation would be, like, use a different agent for that.

Um, but then, like, the speed control, like, you could you should be able to send off an event, but I have not tried that.

Yeah. Or maybe, like, the like, the macro like, basically, like, if you had something and someone was, like, in a noisy environment.

Mm-hmm.

Like, hey, you seem to be getting interrupted. Could you adjust or could we catch in the background and start adjusting that parameter so just the voice is protected at this part?

Yeah. So the the question is, like, for example, if someone is, like, in a noisy environment, like, could you have the agent detect that and then use, um, like, adjust some of the session configuration to deal with that and just the voice is protected?

I don't know, honestly, which parameters are protected or not. The good thing is the like, the API will throw an error if if a thing didn't work. Um, so it's a good way it's a good thing to experiment with.

Um.

You can do that in Python.

Hmm?

With the previous ones, you could do that in Python.

Oh, yeah. Um, well, with the Python, uh, agents SDK, we're doing the chained approach. We don't have a real-time build-in yet, so.

So just calling the old API in Python, you could change it.

Oh, allright. Yeah, yeah. Um, then it should work, yeah. Uh, if you can do it in Python, like, it should just work. Cool. Um,

so the other thing we talked about, um, is this delegation part. So that's what I, like, had mentioned earlier. That was in the diagram. So this is essentially where you want to be able to have certain complex tasks dealt with by a more intelligent model.

Um, and the way we can do that is essentially just creating another, um, agent except on the back end. And because it's because the TypeScript SDK works both in the front end and back end, we can do that through, um,

uh I think I have a let's see if we have a file here or not. Uh, we can do that using the same SDK. So I'm going to create on the in the server folder here a new, uh, file.

I'm going to call it just agent. And in here, we can build our regular text-based agent. So this is essentially the same code that we've done before. Um, and we can say this is a I don't know. Um,

call it the riddler. Um, you are excellent at creating riddles based

on a target demographic

and topic. Um, and we'll just give it a model of O4 mini.

Also, reminder for those if you are trying to follow along, um, and you run into troubles, post in the Slack, and the noob can help you with that. Um, so we have that new agent here. We're not going to give it any tools or anything.

Uh, and then we can export a function here

that we just call run agent. And this is just going to take some input and then return that output. And we can go back into our front end code, create a new tool here, create riddle.

Um, and this one, we're just going to have take, like, two parameters, the demographic and the topic, and then call out the run agent function that is going to run on the server we can pass in. Actually, I realize I didn't specify this.

Do demographic and topic and then create an input here of

this. The other thing you want to do when you're using server actions in, uh, Next is put that use server at the top. That makes sure that this file executes on the server. And then we can pass in that demographic.

And again, if you're using a different framework, this is just the equivalent of a fancy fetch request. So, like, if you want to do an HTTP request to your service, if you want to maintain a WebSocket connection to your own back end, you can do all of those things to, like, talk back to other systems.

So with that, we can give that to our main agent.

Riddle?

Riddle. Oh.

Tools.

And then what you want to do in these cases is, like, um,

you can tell it, like, announce when you are about to do a task. Down, say, you are calling a tool.

Things like that can be helpful to, like, by itself sometime.

Hey there. Um, can you tell me a riddle, uh, for, like, a five-year-old Star Wars fan?

Hey, are you there?

I'm still here. I'm working on creating that Star Wars riddle for you. It should be ready in just a moment. Here's a riddle for your little Star Wars fan. I'm not.

So you can see that, like, because it announced that, like, what it's about to do, the tool call came back before it even finished what it previously said. And so, like, that's, again, one of the benefits of, like, if you can get your agent to, um, balance out that and, like, by itself sometime, this is a good way to, uh, you know, deal with the more complex tasks.

And, like, it also means that you can, like, for example, take all of the, like, more reasoning-heavy workloads and take it out out of the voice agent model.

Uh, um, for delegation.

Yeah.

Is it possible to delegate to more than one agent, like, simultaneously, or is it just one in the current SDK?

Um, you can it's it's tool called. So, like, I think you like, you would have two options,right? Like, you could do par like, parallel tool calling, or you could, um, like, have one tool that then triggers running multiple agents,right?

So, like, my recommendation would be that part, potentially, uh, so that you're not relying on the model making theright decision of calling multiple tools at the same time. Like, you want to make the decision-making for the voice agent always as easy as possible.

Allright. Thanks.

Yeah.

Previously, when you did can you go back to the previous page? The.

Yeah.

Oh, no. Sorry. Your example.

Which example?

The one you were running where it had the output.

Oh, here.

So on line three there, where it said, "I'm still here."

Yeah.

Is that coming from your SDK? Do I have to use the SDK to do that, or is it a real-time API that you?

No, it's a real-time API. It responded because I asked, like, hey, are you there?

Yeah.

Um, so it it it tri it realized that, like, it didn't start anything and, like, interrupted and was like, hey.

It realized there's a tool call going on.

Yeah.

Cool.

Um, yeah. So the thing is that, like, I didn't render out the tool calls in here,right? So, like, what basically happened between this and this was, like, it started off a tool call. Um, and then that tool call because I interrupted it then, it stopped that tool call.

Because I interrupted it, it stopped the the generation. It also reset that, like, transcript here. It's a good indicator that, like, the interruption happened. And so when I said this, it remembered it was trying to call a tool, did that tool call, um, and then gave you back the response.

So that's all the just the the regular real-time API.

Any any other questions around this? Oop. Yeah.

What is the cost per minute?

What's the cost per minute? We charge per token. There's some translations. I don't know. Anoop, do you have the?

I haven't found one for you.

It's more expensive than TTS and ASR, but it depends on what model versus what you're using.

Right. So it it's more expensive than TTS and, uh, uh, and, like, speech-to-text chained up with a model uh, like, in most cases, but it depends on the use case, um, and sort of, like, your model choices and stuff.

So a bit harder to say, like, what the per minute pricing is because, again, it's by tokens. And it also depends on, like, if you have, like, transcription turned on and, like, how many function calls you have and things like that because it's a mix between audio and text tokens.

Um, so one of the interesting things and this is not a thing in the in the regular API. This is a agents agents SDK specific thing is, um, guardrails. So, like, the agents SDK, both in Python and in TypeScript, has this concept of guardrails that can either protect your input or your output to make sure that, like, the agent is not being meddled with or does things that are against policy.

Um, we took that same pattern and moved it over to the real-time side, um, where essentially we're running these guardrails that you can define in parallel on on top of the transcription at all times. You can kind of specify you can see it at the bottom here, like, how often you want to run them or if you only want to run them when the full transcript is available.

But this is a great way for you to, like, make sure that the model doesn't violate certain policies. You want to make sure that these run as efficiently as possible because they're still running in the client. Um, but this is a good way to still fix cer like, stick to certain policies.

And if it violates those, it will interrupt it. Now, there is sort of the bit of the cave a bit of a caveat where because we're running this on a transcript, um, it results in a bit of a timing aspect where if it vi if it would violate your guardrail in the first couple of words, chances are it will say those first couple of words and then get interrupted.

If it is happening at a later point in time, um, then the transcript will be complete or, like, the text output is not really a transcript. The text output will be done before the audio is done speaking every like, is is done saying everything.

And so in that case, it will just correct itself. So in that ca like, to give you an example of, like, this is a guardrail that just checks, like, is there the word dom in the output. In this case, like, if I would ask it, like, hey, please call me dom, chances are it will call me dom and then self-correct.

If I tell it to tell me a story and only introduce dom in the second act, then it will, um, catch that it was trying to do that in a at a much earlier point because that transcript is going to be done before the audio is being complete uh, before the audio is being read out to the user.

So the user will never hear dom, but instead, the model is going to be like, okay, I'm sorry. I, I couldn't help you with that. Like, let's do something else instead. And you can give it, um, policy hints essentially on, like, why it violated this policy or what it should do instead.

So you can give it these this, like, output info where you can inform the model, like, why this happened.

Is the transcript still being done with Whisper?

No. Um, you can choose what, uh so the question was, is the transcript still done with Whisper? Uh, you can you can switch the transcript models. So we have we released in March, uh, two models. One is GPT-4 mini transcribe and

we have two,right? Yeah. And GPT-4 uh, uh, GPT-4 transcribe. I was trying to remember. Like, we have only one text-to-speech model, but we have two transcribe models for transcribe models.

Awesome. Um, this was the main part of what I wanted to walk with you all through. So, um, one, I'm going to, um, post all of the links and the slides in the Slack channel, which let me go back to

where was that slide? There we go. Um, so in that Slack, I'm going to post all of the resources. So if you want to check them out afterwards, I'll also like, I already put a bunch of the resources that I talked about into the bottom of that starter repository, so you should have access there as well.

And, uh, I'm happy to hang around, answer any questions. Yes.

What is the wrong guiding rule for voice setting in the coder? Um, my understanding is for text coding itself, when I type something in, the wrong guiding rule is it goes to the same coder in which you do your coding before, so those same key value variables are actually a bit of a catch.

Um, is it a bit tricky because you need, like, low latency as well? Because if you have a client on the other end, who was going to deploy the most profitable thing? So.

Yeah. Um.

I wondered if that's a it's posed in the API or is it, yeah, is it necessary?

Yeah. The question was around how prompt caching works, um, and sort of whether we like, for prompt caching, that, like, we guide the request to the same, um, same system again to run and whether, like, there's any control with that with real-time because latency obviously matters.

Um, I don't think there is any controls about that. Um, no. Um, I'm getting a no from there. So, like, I don't think there's any any controls around thatright now. Yes.

Hey. So we all know that having, uh, natural conversations is involves more than just spoken words. It involves detecting emotion and adjusting tone. Um, it also involves cadence and even humming to let the other person know that you are listening.

Mm-hmm.

I wonder if the current speech-to-speech model is capable of having that kind of natural conversation.

Um, part of this is, like, a a prompting challenge. So, like, it it definitely can have pretty natural-sounding conversations. And, like, I think this is sort of the the part where I highly recommend to check out the openai.fm page.

Um, because this is, like, it's it's pretty interesting to see sort of, uh, if we go to um, let's see if we find the um, actually, I didn't show one neat feature

that I normally call out on the playground. If you're just getting started with real-time and you don't even want to write any line of code, um, this is a great way to just have conversations and try things out.

But one of the things it has is this, um, it has a couple of s-system prompts. One of my favorite ones to show sort of this is the bored teenager. Um, so if we start this. Hey there. Um, so I'm at AI Engineer World's Fair, and everyone is super stoked about voice agents.

Can you show me some excitement of this whole thing launching today?

Let's see.

I guess it's cool, whatever. There's always new stuff launching. People get excited, but, you know, it's just voice agents. Not really my thing to get all hyped up about it.

So you can see in this case, like, it it put its own pauses in there and stuff. Like, this wasn't a pause because the model was waiting,right? Like, um, it can deal with a lot of that sort of adjusting tone and voice.

And it can do similar things, like reacting to, like, someone talking in and stuff.

Okay. Thanks.

You're welcome. Yes.

So again, on the new, uh, API that's been releases, what else has changed? So maybe has this improved? Is there anything else that's improved in terms of turn detection or voice?

Um, we have not released any new VAT models. Um, we primarily released, like, a new base model for uh, or, like, a new model for the, um, GPT-4 real-time model that is just better at function calling and has been overall well-received from our alpha testers.

Yes.

Can you inject, uh, can you inject audio as a as a background audio, like ambience audio, like typing audio and so on?

Um, you ca like, basic.

You can do it like you're in a real office.

Right. Yeah. Um, you can just intercept the audio that is coming back from the model and then, like, overlay your own audio.

Oh, yeah.

Hi. Um, I was wondering what your support is for, like, multiple speakers. Uh, if there's more than one person in a conversation, can it detect who's talking and and do pauses that way?

There's no no current, like, speaker detection in the model. Um, so it might struggle with that. Yeah.

Um, I just wanted to ask, um, about, like, custom voices. Is it limited to the preset voices you have, or can I upload my voice as a sample, for instance?

It's currently limited to the voices that we have. We keep adding new voices, though.

Is there going to be support to add custom voices anytime in the future?

Um, at any time in the future?

Uh, not not, like, not, like, years.

Uh, what I mean, what I can say is, like, we're trying to make sure we're finding, like, the safest approach. Like, we have an article on online that talks about sort of, like, the responsible take we're trying to take on this on making sure that, like, the vo the voice like, if we're providing custom voices, that it comes with theright guardrails in place and stuff to avoid abuse.

Allright. Thank you.

You're welcome. Yes.

Is there still a 30-minute timeout on the session? And if so, what is the recommendation for the longer session?

I don't think that has changed, um, to my knowledge. My personal recommendation would be that, like, one of the things that you can do and this goes back to, like, for example, in the demo that I was showing, um, if you're keeping track of the transcript and stuff, you can re like, when you're starting a new session, you can populate that context by creating new items using the API.

So, like, one of the things that you could do is starting a new session, if you know what the previous context was because you kept track of it, you can then, um, basically inject that as additional context.

Is there an event we look into?

Um, what type of event are you looking for?

Timeout event?

Oh, if there's a timeout event. Um, I do not knowright now. But all of our events are documented in the API reference. Yeah.

So when you say, uh, real-time API can call tools and function calls, uh, do we include, like, system file reading and writing in those packages?

Can you sorry. Can you repeat that question once more?

Oh, I was wondering if the real-time API can each function call, uh, functions such as system file writing or reading.

Oh, uh, whether the function calls can do things like system file reading and stuff. I would say it depends on where you're running that a uh, where you're running that, um, real-time session. So if you're running it on the server, you can do anything you can do on the server.

If it's if it's running in the browser, then you're limited to whatever things are available in the browser.

I see. So not so locally. Type of challenge?

Um, no. You should be able to, like you could create a, like, WebSocket-based, like, um, voice agent that runs on on your device. I mean, like, it's going to use the real-time API for the model, but then, like, because the actual tool calls will all get executed on your system, you should have access to whatever aspect of your system your program has access to.

Cool. Yes.

So even before we get to voice agents, we all need bigger, better, and more diverse evaluation sets, especially for anything around function calling and parameterization. Uh, do you have any best practices or suggestions for how we now take evaluation into the voice world?

Should we keep things in text and then turn them, you know, use text-to-speech to have voice versions of it? Just if you have any suggestions for how we evaluate the the full range of inputs that we would expect users to bring to this.

I mean, one of my suggestions would be if you can go to the leadership track, go to Anoop's talk, um, which is tomorrow, I think. Yeah. Um, he's going to talk a lot more about, like, additional best practices of what we've learned, um, in, uh, building voice agents.

I would say, like, if you can hold on to the audio, like, it it's helpful. Um, obviously, transcriptions, definitely. But, like, it it's sort of, like, the audio is still the thing that is, like, the most powerful thing, especially for speech-to-speech models, where you have the model act on the speech, not on the text,right?

And, like, this is one of the one of the few things where, like, the the chained approach obviously makes some of this much more approachable because if you have if if your if your agent is running on text anyways and you can just store the text and rerun it, that makes that part of evals a bit easier.

That makes sense. And then also for those of us who might be thinking about launching a new voice agent, how would you suggest evaluating it before we get to that stage that we'd have, uh, customer interactions to work with?

Um, I would start with, like, having, like, a thing that goes back to, like, um, like, some of this is, like uh, where is it? There. Um, like, human review is an excellent solution for this,right? So, like, have it like, have a system like, like, one of the big things with, like, things like Lemonade and stuff is, like, they're able to go through all of these calls and, like, get an idea.

But they also have their own, like, predetermined set of examples that they might want to test as they're developing the agent. So, like, that's a great first way. I would scope clearly the problem you're trying to solve as well.

Like, if you're if you're trying to sort of boil the ocean, it makes a lot of this significantly harder as opposed to, like, well-scoping what the agent should be able to do and what it shouldn't be able to do.

Makes sense. Thank you.

You're welcome.

Get getting on the same path of our friend over there. Uh, actually, when I'm testing my agents, my text agents, conversational, uh, I use Promptful.

Mm-hmm.

It's a platform for doing all the prompted testing. Could I put another agent to talk with this this agent to do all the evaluation?

I think you can try it. Um, like, I don't think it shouldn't work. Um, um, like.

So I put another voice agent talking to that agent to try to execute all the prompts, and then I could get, like, the transcription.

Yeah.

Observability.

Yeah. I mean, I mean, I know we have we have use cases where customers also use our models to, like, prompt humans,right? So, like, um, where it's, like, for for, like, training use cases or other things, for example.

Awesome.

Um, so it should work out. But I don't know if anyone uses that kind of approach in Lemonade does.

Oh, cool. So the second the second picture is is exactly that.

Awesome. Thank you.

You're welcome.

Any any other questions?

Sorry.

Yeah. Go ahead.

Slightly related, uh, do you have something around wake word detection on the real-time API roadmap or.

Sorry.

Patterns for wake words?

Oh, reports? Um.

No, no. Wake words. So that's, like, activating the.

Oh, wake words.

Yeah. The KCD.

Um, no, we don't have any wake words built in or anything.

No patterns either? Like, any patterns to avoid costs?

Um, no.

On-device, I guess?

No.

No. Okay.

You could basically, like, what you can do is you can build your like, you can turn off our, uh, voice activity detection and then build your own, um, and then basically use that. So, like, you could use a model that has, like, a like, a VAD, uh, voice activity detection model that has wake words in it and then, like, do it that way and basically commit all of that audio to our API and then send, like, a commit like, a commit event.

Cool.

Thanks.

Well, awesome. Thank you so much for taking the time and spending the afternoon with me.