Voice Agents: the good, the bad, and the ugly

Hey everyone. My name is Eddie Siegel, I'm the CTO at Fractional AI, and today I'm here to talk to you about AI voice agents and all the challenges we've run into building with them, and ways we've been able to work around those challenges.

So to start, voice models are just generally tough to wrangle. I mean, the ability with LLMs is tough to begin with,right? They hallucinate, they're generally difficult to evaluate. Coming up with metrics to know how well your system works is really challenging, especially if you're dealing with something that is conversational in nature, where there's no objective metric to know how well you did or how poorly you did in a particular interaction.

And latency is challenging for many different kinds of applications that leverage LLMs. They're especially— it's an especially important challenge when you're dealing with conversational UIs that are meant to be sort of fluid the way a human conversation would be fluid.

And when we're dealing with audio and these voice agents, everything's on hard mode. You deal with all these existing AI problems, plus you're trying to deal with things like transcription, which is not as straightforward as you might think.

You're dealing with a streaming environment instead of this sort of batched back-and-forth kind of interaction that you get with text. And this all makes it really, really hard to do development. So let's go into a case study with an actual application that we built recently, so that we can see in practice why the sort of theory was not exactly what we encountered in practice.

So for this case study, we're talking about automating this sort of consulting-style of interview. And what I mean by that is, like, going into big companies and interviewing employees there to learn information. Not interview as in a job interview, but interview as in, "Let me go into some Fortune 500 company and understand how people are doing their jobs."

This is a thing that consultants do all the time. When they're trying to research how something works within a company, they have to go in and do this sort of in-depth qualitative research, where they're going and meeting with employees at a company and asking them questions.

It's a pretty expensive process. You know, if I want to send a bunch of consultants into Coca-Cola to ask employees about things they're doing, the consultants have to spend a whole bunch of time. It's very inefficient. You have to get people's schedules to line up and do this sort of, you know, how do you cram a calendar full of these interviews back to back?

Lots of overhead. But it's necessary. This is kind of a difficult thing to automate,right? You might think, "Why not just send them, you know, a form to fill out or something?" And there's certain cases where you probably can, but in a lot of cases, you need the human touch,right?

You need the interviewer to be navigating, you know, "When do I sort of improvise? How do I build trust? What follow-up questions do I want to ask?" And also, people being interviewed answer differently when they're asked to type out questions to specific— or answers to specific questions, whereas, like, if they're given the freedom to just ramble, they often say more stuff.

And so the goal here is, let's build an AI interview agent that can replace this function here, where our AI system should conduct interviews just like a human would conduct these interviews. It should feel less like filling out a form and more like having a conversation.

But our system will be able to interview hundreds of people at once and not deal with this sort of scheduling issue and the high cost of sending consultants into a company. And one nice outcome of this system should be that we get automatic transcription of what happened during the conversation so that later on we can extract whatever data we need to extract and aggregate it across all these different interviews.

So let me just dive into a quick demo and show you what we ended up building so you can see what it looks like in practice. So here I just set up a test interview that's going to ask me just a few basic questions, like, you know, who I am and what I'm here to do.

But on the back end, this can be configured with all sorts of different sort of use cases that we'd have in mind.

Hello. Let's begin. What is your name and role in your organization?

Hi. I'm Eddie Siegel. I'm the CTO at Fractional AI.

Pleasure to meet you, Eddie. As we continue, I'm curious to know what you hope to accomplish with this interview. What are your main goals?

Mainly, I just want to show a quick demo of what your capabilities are so that I can talk about what it took to build you.

Got it. You'd like to showcase a demo of what I can do and discuss what went into creating me. That's a clear goal. Shall we move on to another question, or is there anything else you'd like to add?

No, I think that's all I got from my end. Thanks.

Thank you for taking the time to talk to me, Eddie. It was great hearing about your role and your intentions to showcase a demo of my capabilities. I'm looking forward to being a part of your demonstration.

So now you see what the final product looks like. We can talk a little bit about how we started building something like this. And when we first started, we figured there's this great OpenAI real-time API that lets you build, you know, voice agents like this.

And we just did the naive starting thing. We integrated with the OpenAI API. We built a big prompt that explains, you know, you're trying to conduct this kind of interview, and it tells them all these things about the interview, and then puts all the interview questions we might want to ask into the prompt and tries to explain how to navigate asking those questions.

The problem is, like, we quickly realized that we wanted some additional functionality here. And one of the early things we realized we wanted was we wanted to show a little roadmap on the side of the screen that shows the list of questions you're going to be asked, which one's being askedright now, and ideally would let you click on this and jump around from question to question.

And this really just was not well-suited to the monolithic prompt approach that we had at the beginning, because there's really no way to know what question the LLM is currently asking, and it's difficult to sort of coax it into moving around and asking a different question if they do click around.

So something we introduced here was, let's just ask one question at a time, put that one question into the prompt that we're sending to the LLM, and then introduce some tool use here where we give the LLM access to a tool where it can say, "I want to move on to the next question now."

And when it says it wants to move on to the next question, then we'll feed it the next tool, sort of determine— sorry, the next question deterministically. And then we also added some additional prompts that we inject into the stream with the LLM in the case when the user just clicked around in this roadmap, so that the LLM can be made aware the user is, you know, we're moving on to this question because the user clicked next or because the user skipped around.

And so this is what it ends up looking like on our back end. We have different prompts for what happens when a user is revisiting a question, what's happening when a user is skipping to something new. And this enables some really cool behavior.

When we skip around, the agent can now do things like saying, "Sure, let's move on to, you know, XYZ. We can come back to that later." Or, "Let's go revisit that question we were talking about, about ABC."

Very quickly, once we started playing with this and actually using this in practice over and over again, we found that the LLM had a real tendency to go down these rabbit holes where, you know, now that we gave it this tool to use to say, "I want to move on to the next question," it was hard to get it to call that tool the appropriate amount of time.

LLMs just love chit-chatting and asking follow-up questions and digging down rabbit holes and being very encouraging about things that they hear. And so this resulted in sort of a reluctance to move on to the next question. And if we forced it too hard, then we would eliminate its ability to improvise at all, which is obviously not what we want.

So we introduced at this point an additional background agent that's listening to the conversation and running a separate side thread to decide, with a totally separate call to a non-voice, text-based LLM, based on the transcriptright now, do we think we're kind of like off track or on track?

And we called this the drift detector agent. And so this is like a totally separate prompt, and it's just looking through the history, and it's trying to decide, "How are we doing? Are we on topic? Are we off topic?

Has the question we're trying to ask been answered, and is it time to move on?" And when we get back a result that says pretty strongly, "It's time to move on," we can prevent this sort of rabbit hole from continuing any further by, like, forcing the tool use and, you know, flipping the flag on the API call to OpenAI to say, "Okay, you have to use this tool now, and you have to move on to the next question."

Broadly, it's pretty hard to get these human-like interviews tuned into theright place. They're always either, like, following up too little or following up too much. You want them to be able to rephrase questions, but sometimes they'll rephrase them in ways that are not conducive to the overall interview.

And because of our approach of, you know, giving it one question at a time, it doesn't always know what the whole interview is going to be like. And also, this linear flow, or the only thing it can do is decide to go on to the next question, is pretty restrictive.

It's either digging in really hard into the question it's on, or it's saying, "Okay, give me what's next." And this sort of limited the natural flow of the conversation. And so we introduced some things to try and correct for this even further.

The first is we added goals and priorities as a first-class concept in the sort of plan for the interview. So rather than just telling the LLM, "Here's the question you're askingright now," we're telling it the why behind the question so that it can be informed when it's trying to rephrase or informed when it's, like, trying to decide how to ask follow-up questions.

And then we introduced yet another side agent, which we call our next question agent. And this guy is just in charge of figuring out, "What should we ask next?" And so the first point is, this is what the goals look like.

This is a screenshot from our admin interface. But, you know, we have this question that it's going to ask. You know, in this case, the question is, "What are your main responsibilities and daily tasks?" And without the goals, you know, it may repeat this question verbatim.

It'll probably rephrase it ever so slightly, but it's going to ask something pretty similar. And then when it comes to digging into follow-up questions, who knows? Like, if it's going to be satisfied after your answer, like, "Okay, I understand this person's responsibilities and tasks or not," sometimes it's going to be satisfied, sometimes it won't.

So instead, we add in these goals, and we say there's a high-priority goal of, "We want to get a clear picture of this person's regular activities," and a medium-priority goal of, "We'd like to be able to start to suss out where AI might be useful."

This is an imaginary use case where the reason you're doing these interviews is you're trying to find ways to help with AI. And so this really can, like, help guide what the follow-up questions will look like. And then we have the tuning of this side, you know, next question bot that's just running on the transcript in the background as the conversation's going, and it's sort of taught how to be a good interviewer and is this interviewer on track.

And it has the ability to say, "No, like, it's time to go down this path now," and guide the rest of the conversation.

Speaking of transcripts, you know, these background bots are running off of the transcript of the conversation, and you also see the transcript live. When you're being interviewed, you don't just see what the model is asking you. You see what you said to the model.

And luckily, OpenAI makes this really easy. When you make API calls, you get back, you know, what did the user actually say? You get a transcript kind of, you know, baked into the API response. But a challenge in the implementation here is that the OpenAI real-time API, it has its core model that's listening and understanding you, but it uses a side model called Whisper that's doing the transcription of what the user said.

That core model is this fully integrated model that's able to think about what it's hearing. It's operating in the sound domain. If you clap, or if you cough, or if you bang on the table and you ask it what you just did, it'll be able to tell you.

Whisper does not work that way. Whisper is converting everything into text. And so, you know, if you clap, you might get some surprising results. And so, like, this is the result of me just not speaking. I just sent some silence for a little while.

And despite the long conversation history and all the evidence that I was speaking English, it decided, "I guess he's not speaking English anymore." Or another one where there's, like, a little bit of background noise and I didn't say anything, and this is the transcript that came out.

So, like, clearly I was not saying, "Creeping dippity dippity dippity dippity," but, you know, it said that I was. And this is not a great user experience, and there's no way to tune this. There's no, you know, if we were working with Whisper at a low level, there might be things we can do here.

The OpenAI real-time API does not give you knobs to really control this. And so to handle this at the UX level, we added an entire separate agent that is taking in the whole context of the conversation as it's happened so far.

Here's the entirety of the transcript, and its entire task is, "Should we hide this piece of the transcript from the user? Do we think we'd have a transcription error?" And we still capture the transcript in that case, but now we can at least hide, you know, these embarrassing examples so that it just doesn't show up as anything to the user.

And the nice thing about that is, if these blocks are hidden, the core model still knows what's going on. So I don't include in these screenshots what the response was, but typically the response was like, you know, "I didn't really get that.

What?" You know, and sort of re-ask the question, "What do you hope to accomplish with this interview? I didn't hear what you said before." And so hiding the transcript actually helps a lot in these cases for the user experience.

So as you can tell, this is getting kind of complex. Like, we're just adding lots and lots of agents to sort of band-aid over different things. We're doing a test interview and seeing how it went and realizing that we've got these issues and then trying to, you know, piecemeal fix them in this kind of vibes-driven way, which is not the worst thing for the very beginning of the development cycle, but this thing is starting to get complex.

And now we have a lot of different prompts to update. We have all these side agents. When we have an issue, it's not clear which one to update or if it even worked. And sometimes you're going to introduce regressions,right?

You perform poorly in one interview setting, and so you update your, you know, your drift detector to sort of handle that case. And maybe you just made yourself worse at the last thing you've tried to fix with that drift detector.

This is a very challenging situation. It's common to all LLM-based development, but it's extra hard in this domain. We want to know how well it's working, and the answer is evals. We need some systematic way to sort of measure the performance of this system.

And so what we ended up coming up with in this case was a set of metrics that measure all the different attributes we wanted to see. And each one of these metrics you're looking at is, you know, something, an automated test suite that we're able to run over a conversation, and then we ask an LLM as judge to measure how clear was this conversation, how complete was this conversation, how professional was the agent, all of these different things we see here.

And behind each one of these is a prompt that we've tuned to try and do this evaluation. This is really, really useful for getting more objective. It's still not completely objective. You know, we don't have a perfect ground truth to measure against here.

There's always improvement that we can make in the prompts that are measuring these things, but this is really useful for getting further outside of this sort of purely vibes-driven style of iteration and more sort of metrics-driven style of iteration.

As I mentioned, there's no good ground truth here. When you're designing evals for, you know, any system that is going to do sort of, you know, the sorts of things you'd want to use AI to do, your ideal situation is you've got some objective metrics, you've got some historical data to run your metrics and your current system against, and you can know objectively, like, how well are you doing.

That clearly doesn't exist here. There's no way to say with perfect understanding, "How well does my system work?" But still, we want to make sure that, like, we don't ship a system that we think is okay and then find out from users in practice.

You know, we have a bunch of these interview agents go out into Coca-Cola, and we find out from Coca-Cola we're annoying all their users for some reason, or we've got this horrible edge case. So as we thought about this problem, what we ended up introducing was synthetic conversations where we would come up with, we would use LLMs to fake users and to do fake interviews as the interviewee, and we'd run a whole bunch of these and at least be able to measure how well we think we would do against the types of users we want to be able to interview.

This also just helps with automation so that it's not us just clicking around a bunch and taking these interviews. It gets very tiring to talk to your computer to test this thing out. And so the way these synthetic conversations work is we can just create a persona.

That persona is used as a prompt to the LLM where we tell it who it is. You know, in this case, I said, "You're a snarky teenager that's in charge of a Fortune 500 company." In practice, we come up with a whole list of the types of people we expect to be interviewing, and they're different personalities.

They're different job functions, and we have a whole roster of them. And then for each one, we can run it where the agent interviews it, and then it uses that persona to try and answer the questions. And then afterward, we can run that very same eval suite over them, and now we can get average metrics over a broad population of the types of people we'd be interviewing.

So to wrap it up, you know, that was a case study with us building with voice, but I think it highlights in general that this approach of, you know, maybe we just call OpenAI, it's got these nice voice capabilities, now we'll have a nice robust voice app is kind of wishful thinking.

You know, the basic kind of call the API, do some prompt engineering, and get it into a good place is very helpful. It gets you a very, it gets you very far in the development process, but it's not enough to build your robust app.

As you're trying to build that more robust app, some really helpful additions are adding some out-of-band checks where you've got separate agents operating in the text domain instead of the audio domain, trying to make decisions about whether the thing you're trying to do is working properly, trying to get the overall thing back on track.

And tool use can be very powerful for constraining behavior and for getting sort of, it's like sort of a hack to instrument the LLM and understand what's going on. Instead of letting it just sort of do whatever it wants, it's got to call your tools in order to do what it wants, and so now your tools know what's happening.

And then lastly, as with all LLM-based projects, evals are critical to measure success. They're also critical to guide development. It's hard to know what to do without some metric. And even in this domain where you've got no objective source of truth, there are still ways to harness evals to get a really robust development process.