Shipping an Enterprise Voice AI Agent in 100 Days - Peter Bar, Intercom Fin

Today I'm going to be talking about Fin Voice, and Fin Voice is a voice agent for phone support. Um, and we designed it to be a frontline teammate for inbound calls, so it picks up the phone, answers customers' questions, and then escalates to a human agent when needed.

And we built this experience in about 100 days, so today in this talk I'll share what it took to get there. And I'll also talk about why I believe voice is the next big frontier in AI for customer service.

So first, a little bit of context. My company, Intercom, so we're a customer service platform and also an AI agent company. And you might be familiar with us because of our messenger product, seen it, you might have seen it in a mobile app or on the website.

But yeah, it's been our foundation for years. But we evolved over time, we became a complete customer service platform a few years ago, added robust tooling for other channels like email, WhatsApp, and phone. And then two years ago,right after the launch of GPT-4, we launched Fin, which is an AI agent on text, on chat.

And Fin's growth has been incredible. We have over 5,000 customers, and also in terms of performance, it's reaching average resolution rate of 56%, and for some customers achieving 70, 80%. And this is defined as a percent of interactions handled by Fin that are resolved without human intervention.

And Fin is also a full system for continuous optimization, so it's not just the agent, but also tooling for analyzing conversations, training the behavior of the agent, and also testing and deploying changes.

But yeah, up until now we didn't have the voice channel, and that's what we're changing with Fin Voice. It's the same system, but now it can answer phone calls. And a few thoughts on why voice, why are we investing in this channel.

So for a lot of users, voice is simply the preferred way to get help. So when an issue isurgent or sensitive, they don't necessarily want to type, they just want to talk. And if you look at some of the top-level data, over 80% of support teams still use phone support.

And if you think about all the conversations globally, our customer service interactions, over one-third of them are happening over the phone as well. So it's not a legacy channel that's going away, it's still widely used, and it's also quite costly.

If you think about the average cost of handling a phone call in the US, with human support it's about between 7 and 12 dollars. And with Voice AI agents, it can be at least five times cheaper.

And a few more benefits of Voice AI in customer service. So first, availability: 24/7 support, you can call your bank on the weekend. No wait time, so instantly available, there's no need to be to stay in, be on hold in a queue.

No IVR menus, no need to press 1 to go to support, press 2 to go to payments, because all, everything is happening via natural speech. And also multilingual, so AI agents can support 30, 40 plus languages, obviously better for the, for the users.

And on the business side, major cost savings, and also scalability as the business grows, or as everyone needs to handle peak times, AI agents are much better for that.

So how we built Fin Voice. Over the next few minutes, I want to cover seven main areas that had the biggest impact on how Fin Voice came about. I'll try to be more practical, practical, focusing on some of the product decisions we made and the challenges we faced.

And yeah, the first one is the use case, so the starting point for Fin Voice, then also the scope of our MVP, the tech stack behind it, the, how we approached the conversation design, how we integrated it with the support teams, and also how we thought about evaluation and pricing.

So starting with the use case. If you look at a lot of the Voice AI startups in this space, they typically start with a narrow problem space, something like scheduling a dentist appointment or booking a table at a restaurant.

And we looked at some of those options, but eventually decided to go for a more flexible knowledge-based agent. So an agent that can answer helps, help article questions like what are your pricing plans, or what's your returns policy.

And why did we decide to go this way. So first, we had strong evidence from chat. Fin over chat has been handling those kinds of conversations for years, and our customers have constantly told us that they're seeing the same type of issues over the, over the phone as they see on chat.

And we also validated this for extra analysis of call transcripts, and this confirmed that a very large percentage of all the queries could be solved with the, with the knowledge-based, with the help articles content, rather than sitting with the API integrations.

And we were also thinking about the initial wedge use case, so like the, what's the lowest possible risk way for companies to integrate Voice agents. And we looked at the in-office hours and outside-of-office hours use cases. So we pitched out-of-office hours as an initial wedge, because essentially it allows the team to not affect the main workflows, and try out this technology, build up more confidence over time, and later deploy it for their, on their, in their main office hours.

But in out-of-office hours, it just replaces their voicemail experience. And there's also a few other use cases we looked at. Authentication, so verifying user identity on another channel. Info gathering, so the agent getting stuff like order ID, account ID, and also smart routing to theright team.

So these use cases are still very high leverage because they can save a lot of time for the support team agents, but they're not necessarily solving the issue end to end. So we're still focusing on those, but they weren't necessarily the primary use case for this initial version of the product.

So now moving on to what we shipped first.

When we, when we started, started this, like the biggest challenge there was to ship something meaningful as soon as possible. We had access to a lot of customers, because we already had thousands of customers using our native phone support product.

So it's mostly about how can we test as quickly as possible. So we focused on the three main experiences: testing, deploying, and monitoring the agent behavior. So first test, this is what we call the Fin Voice playground. And here was like a lightweight test environment for the customer service managers to go in and simulate a few sessions, ask the questions based on the knowledge base, get those answers, and get an idea how this product actually works.

And it was also managed, we shipped this probably within the first four weeks of the project. So it was like the fastest possible way to get some feedback from customer service managers in terms of how it's actually performing, so we can do optimization not just based on our internal views, but also based on customer feedback.

Then the deploy experience, this was to allow customer service managers to actually deploy it on their phone lines, and it included some basic configuration in terms of agent behavior, and also how it should interact with the customer service team's workflows.

And lastly, observability or monitoring. We really wanted to provide some visibility into, into what's actually happening on those calls with an AI agent. So we had those experiences that did show the transcripts recording, and also the transcript summaries and call outcomes to customer service agents.

Cool. And now moving to the tech stack. I'm not going to go for the technical detail of everything. I'm sure that there's going to be a few more talks on this that you might attend as well. But I'll mention some core components in Fin Voice.

So there's the main chained loop for, for the Voice agent, STT, LLM, TTS. So speech to text, converting speech into text. LLM for actually generating the response, and text to speech for converting text back into audio. But there's also another approach with the voice to voice models, where everything is processed directly in audio whilst keeping the text layer entirely.

So, and the voice, voice to voice approach has the benefit of potentially faster or natural more standard speech, but also gives you less control over the output. So in our approach, we did start with real-time API by itself from the get-go, and it allowed us to test very, very quickly.

But eventually we did evolve our stack, but we're still using real-time API as part of the, of the core architecture. There are two other components I want to mention, RAG and telephony. So RAG obviously is super critical for a lot of agent experiences, but obviously it is important for the agent answering questions based on the knowledge base.

And then telephony, so actually being able to put the agent on the phone lines. And we had a bit of a head start because our agent on chat already had the RAG set up, and already we already had a native phone support product, so we got some of those things for free.

Now once we have the technical foundation in place, there's a question about how do we actually design the conversations for voice. So

Intercom has a background in chat, but we knew from the get-go that the approach for voice will have to be a bit different. And that voice is not necessarily just chat with sound. So there are like three key differences I want to mention.

There's obviously many, many more, but there's a few that I thought it's worth mentioning. So latency.

On chat, it's actually, I think, okay to wait for a few seconds for a response, at least from the user's perspective, there's a lot of tolerance. But obviously it doesn't really work on voice. If the agent goes silent for, for a second or two, maybe longer, the user might assume that something has gone wrong.

So in terms of our approach, for the simple queries, we got it to about one second, so we didn't need to do anything extra. But for more complex queries, when they're running a bit longer, three, four seconds, then we added injected filler words, something like, let me look into this for you, let me look it up, to maintain the conversation flow while we generate the answer in the background.

Then thinking the second one is answer length. So again, on chat, it's actually probably desirable in those agents, customer service agents, to provide a bit of a longer response, to provide as much context as possible to the user, and it's easy to skim through the answer to find theright information.

But again, it wouldn't work for on voice. You don't want to wait there for a minute or two listening to the agent. So for more complex responses, or for responses with multiple steps, we're breaking out the answers into multiple chunks and deliver it chunk by chunk, and after each, we ask the user to confirm whether they would like to listen to the next step.

And this works really well for something like troubleshooting when you have a few steps to follow. And lastly, the user mindset. So what's something interesting in, during our early testing of real phone calls is that some customers would interact with Fin Voice like with an old school IVR.

So we're just using single words like support, password reset, yes, no. But then throughout the conversation, I've listened to a lot of those calls, it's like they change their behavior during the call and actually start using full sentences whilst they hear the agent using full sentences.

And one of my colleagues like summed it up nicely that crazy how the human speaks more like a bot and the bot speaks more like a human. And I think this will change over time. To some extent, it changed over chats, and it's about obviously Voice agents getting better and people more being more common as a technology, people are going to get used to it.

But for now, for now it's on us to make those conversations sound as natural as possible, so we help with this transition.

Now thinking about how Fin integrates into support workflows. So this was super important, and definitely surprising for me when, when we got to this point is that majority of the feedback wasn't about the voice, about the model, or about the latency.

It was actually about how does it work with the support team workflows. And don't get me wrong, I do think that all those core model experiences are super important, but this actually became a bigger blocker for a lot of those teams.

So we put all focus that those integration points are as smooth as possible. So we did a bunch of other things, but I just mentioned two here. One is the escalation paths. So getting the calls,

getting configurations for how the calls get escalated to the human support team, and also the context handoff. So after every AI agent call, we generate a transcript summary that gives a bit more context to the human agent that gets the call, so what happened on the call.

And yeah, these are not like super flashy features, but they were absolutely essential to get this from this demo stage to the deployment stage for larger customers.

And then how do we know it's working. So there's a few topics I wanted to touch on. One is the manual and automated evals. So we had a test, a set of test conversations that would be running through on every major code, code change.

Initially it was mostly manual, just in a spreadsheet, but over time we added some automation. Number two is internal tooling. This was super critical

for troubleshooting. So essentially we built some internal, just streamlined web apps to review the logs, the transcripts, the recordings. So anytime one of our customers raised an issue, we can review in detail what happened in the conversation, actually troubleshoot with the logs.

Number three, resolution rate. So this is our North Star metric, which actually tells us whether we're delivering value for our customers. So we define it as

essentially either the user confirming on the call that the issue was resolved, or the user disconnects after hearing at least one answer and then doesn't call back within 24 hours. And yeah, this is the main, the main metric, metric, metric we track.

Obviously there's more in customer service, but this is kind of the main success metrics that we have. And lastly, LLM as a judge. This is more experimental, but we're using another LLM to analyze the call transcripts to help us identify issues or opportunities for improvement.

And lastly, how to price it. So just want to touch on the cost and some of the pricing models. The typical cost ranges between 3 and 20 cents per minute, and the cost here will depend on the, the complexity of your queries, but also on the providers that you choose.

And in terms of the pricing models, so the main two dominant on the markets are usage-based pricing and outcome-based pricing. Probably usage is still the most dominantright now. So usage-based pricing, very simple. It's like per minute or per call, very predictable, but it doesn't necessarily capture the quality of the agent.

So the incentives are not very well aligned between the providers and the customers. This changes with outcome-based pricing because you only charge if you actually resolve something for the customer. So it has a lot of benefits, but also it also reduces risk because for a very long call and the call that you unresolve, the provider actually needs to take the cost.

So, so yeah, so there is risk there. But over time, I do expect the market will converge toward outcome-based pricing because the incentives are way better aligned.

Cool. And a few final thoughts. So to recap, we built a voice AI agent and shipped it in about 100 days. We got several enterprise customers to use it on their, on their main phone lines. And when I think about like some of the main takeaways from this experience, it's actually like getting to theright performance and those like latency and the, and the resolutions outcomes obviously super important.

But it also is not just a model problem, it's also a product problem. So it's about picking theright use case, designing for the realities of those phone conversations, building the tools, both your internal and external, integrating with the supporting workflows and actually building trust with them, because they are ultimately going to be decision makers whether they want to release it.

And yeah, and it's about making it feel effortless, even if there's a lot of complexity behind the scenes. And that's everything for me. Thank you very much. And yeah, if you're building in this space, we'd love to chat with you.