The Agent Development Life Cycle — Zack Reneau-Wedeen, Sierra

Hey everyone, my name is Zack Reneau-Wedeen. I'm going to be telling a few stories, and hopefully we'll leave you all entertained and with an idea of how we build agents and improve them at Sierra. So, in a nutshell, Sierra is the conversational AI platform for businesses.

And just poll of the room, out of curiosity, how many people have heard of Sierra? So, most of the room, but not all. If you've heard of us, you probably associate us with chat experiences and perhaps with customer service.

And that's a lot of what we do, but I would say that we're kind of broadening out in both cases. Probably by the end of this year, most of our interactions will be over the phone, so that's already a big area for us.

And we'll also have a lot more touch points. We have a lot of customers, which I'll show today, who are using us for sales, for subscription management, for product recommendations, kind of all pieces of the customer experience. I noticed yesterday, were a lot of people here yesterday?

Some people. So it was funny to watch people were reflecting on, you know, how much has happened in AI, and they had these timelines and they went way back in time. And so Colin from AugmentCode went all the way back to 2023.

Waseem from Rider was talking about purpose-built models and went all the way back to 2020. And Grace from Luxe Capital went even further. She went back to 2019, although if you zoom in, you can see actually the first thing here is still from 2020.

So everyone was reflecting on ancient history in AI, and it was all this decade. So I'm going to zoom back even further, 2016, in the AI caves. And I know what you're thinking, you know, AI goes back to the '70s and all that, but it definitely felt like the caves in 2016.

I know because if you zoom in on the bottomright, you can see I'm actually down there. I was working at Google with a bunch of amazing computer vision engineers. And what that meant in 2016 is we were really trying to help computers understand the difference between chihuahuas and blueberry muffins.

And, you know, it's not actually that simple. It's not just chihuahuas and blueberry muffins. You know, it's dogs and bagels, dogs and mops, and of course dogs and fried chicken. And so in other words, what we were doing is we were building the first version of Google Lens.

And at this time, I lived in New York City. I was in the East Village, and I had about a 30-minute walk to work. And on my walk, I would see a bunch of stuff. New York's one of the greatest walking cities in the world.

And I would say, what's going on there? What are they even doing? Or, oh, I wonder if that bookstore is nice. Or, I wonder if this restaurant is tasty. Or, oh my goodness, look at that dog. And so there were also a bunch of flowers on the walk.

At this time, Google Lens was in its infancy, and one of the very few things that computer vision models were actually good at that had some consumer application was identifying plants. You might still know this today. It's kind of in the, you know, is that bug poisonous category.

And so I'd ask questions on the walk, like, you know, can it tell the color of the plant in addition to the species? Or, what's that? What type of fern or palm is that? And there's a bunch of flower shops on this walk, so I'd even walk in.

And these are all actually photos from 2016 from my walks to work. And I would go in and test them all out. And as you can imagine, you know, sometimes it was accurate, and sometimes, you know, it wasn't necessarily wrong, but it wasn't really on the nose either.

And so it felt like a slot machine. And I think everyone here who's dealing with AI can probably understand that feeling of,ugh, it worked five times in a row, why didn't it work the sixth time? Whether it's the nondeterminism of the inputs or the nondeterminism of the outputs, that's just part of what it means to be building with AI.

So let's fast forward a bit to present day. Google Lens, you can not only search what you see, you can also shop what you see. You can do this on Google Images, on YouTube, you can do it with your camera.

You can translate non-Latin character sets into English so you can read the washing machine in Tokyo and actually figure out what settings in your Airbnb you should use. You can do your math homework. I'm a little bit too old to have benefited from this, but apparently it's a brave new world out there for the kids.

And of course, this is from the Google Lens homepage, you can still identify flowers. So this is all very mind-blowing, but in my opinion, it comes down to consistent step-by-step iteration over a decade. And when we think about what drives this, we're all engineers in the room, we understand that you need a process to iteratively improve, to get better without also getting worse.

And this, over time, has kind of been considered software development lifecycle. How do you continuously improve? How do you implement, test, maintain, analyze, design, and go through this as many times as you can? Let's rewind a bit more.

2012. The AI caves, you know, the drawings are a little bit less sophisticated. I'm not there yet. I've been oblated. And I pulled some headlines from around this time. You can see this is around when Google Brain was watching cat videos and identifying them on YouTube, and it was a big breakthrough.

I don't know if anyone remembers how big this model was. It was about a billion parameters. This was a huge breakthrough. If you think today, the frontier models are about a trillion parameters. So it was one one-thousandth. It was as if this whole room had like a quarter of a person in it.

And so it was still very impressive at the time. There was also a theory, you know, everyone thought computers would be limited in terms of what they can achieve. I think this is a less popular theory today. What I'm trying to say is it was a long time ago.

This is also around the time that Marc Andreessen published his famous essay that said, "Software is eating the world." And that took a lot of people by storm. If you looked at Stanford University on campus, you would have seen some early-stage startups forming on the lawn.

Does anyone know which startups I'm talking about? You can call it out.

No.

Okay. You might be thinking Snapchat. Not that one. I did actually hear DoorDash in the back. Very good guess. Not that one either. Of course, I'm taught you look like stylish people, so I think you'll know what I'm talking about.

I'm talking about Chubbys.

Chubbys had a contrarian idea that was alsoright, which was not only is software eating the world, but teeny shorts for men are also going to take over. And as I mentioned, they were correct, which you can see here, and you can also see here.

Fast forward to 2024. Kit Garten, SVP of Commercial at Chubbys, we were fortunate enough to host her in Sierra's office. And Chubbys has had an amazing brand since they were founded, and they've always been on the forefront of customer experience.

They've always been thinking about how to level up and how to make the experience more fun and better for their customers. And so it clicked immediately for Kit that the same way you needed a website in 1995, the same way your business needed a social profile and a mobile app this millennium, in 2025, you need an AI agent to represent your business and to help your customers.

So Kit and Chubbys partnered with Sierra. We came up with an AI agent, which is affectionately called Duncan Smothers. First and foremost, he's incredibly capable, but almost as importantly, he's always down to clown. Duncan Smothers is on the Chubbys website and can help you with a variety of cases.

I got permission from Kit to show some of these conversations to you today, so you can see what some of the Sierra interactions look like under the hood and some of the things that these agents are capable of.

So on the left here, you have a customer asking a question about sizing and fit. Duncan is able to empathetically help them while asking questions like, what's your waist size, and offer product recommendations. At the end, he gets a thumbs up from the customer.

Another example, another thumbs up. This is inventory tracking. Duncan can tell what's in stock and help customers choose new items. And then finally, package tracking and refunds. So more customer love. In this case, Duncan is able to inform the customer, actually, there's a couple different tracking numbers for your order, and in the second case, issue a refund.

And so when we talk about autonomous agents, agents actually taking action, not just answering questions, this is what we're talking about. And the results for Chubbys have been they're able to help more customers, more quickly, and with higher satisfaction.

The way that we get to this is because we believe at Sierra that every agent is a product. That means that you can't just drag and drop a bunch of boxes. You need a fully featured developer platform. You need a fully featured customer experience operations platform in order to work on this the same way you would work on your mobile app, the same way that you would work on your website if you want the best results.

And so when Chubbys is partnering with Sierra, it's not just using the product. It's actually partnering with our team. And so we have dedicated agent engineering and agent product management functions that you can think of sort of as forward deployed with our customers, working closely with Kit and her team on a daily basis.

By the way, remember that face that you just saw on the last slide? Was anyone here at the AI Engineering World's Fair back in June? Nice. Got some whoops from the audience. So I know Ben was there. He's up there on stage introducing everyone, and the energy was electric.

You can see the crowd is packed. When I got there, the first thing I did was I sat down at the Deepgram workshop. This was about three months into me building voice agents at Sierra, and I was very interested in what Deepgram had to say.

What did they think of the latest multimodal models? How are they handling latency? How are they handling tone and phrasing? All of these problems that were new at the time. And I sat down next to a man named Sean.

And Sean and I were nerding out about how to increase the speed of our developer loop by using the say command on Mac and then using a program called Loopback in order to pipe that into the browser so that we didn't have to wear headphones and talk and look awkward in the office.

Sean gave me his contact info. He was interested in Sierra. And a few months later, there we are working together in the office. So when I told our company and our founders, hey, I'm going to the AI Summit, you know, I hope it's as productive as the last one.

I'm excited to learn. They said, go find more Seans. So I'm hopeful that people in the audience will say hi after this. Whether or not you're interested in working at Sierra, I'm interested in meeting you. And so I hope to meet you later today.

Anyway, back to Duncan Smothers. The point of the software development lifecycle, the point of our agent engineering team, is that even if Duncan is not perfect today, he should be getting better every single day. And so what we did is we sought out to build something like the software development cycle, borrowing as many concepts as we could and inventing new ones where we needed to.

The issue is that large language models are like building on top of a foundation of Jello. And so you can't just take everything out of the box and have it just work. While traditional software is deterministic, fast, cheap, rigid, and governed by if statements that always follow logic, large language models can be nondeterministic.

They can be slow. They can be expensive to run. They're very flexible, though. They are creative. They can reason through problems. And so we wanted to create a methodology that takes advantage of all the strengths of large language models and then also is able to invoke traditional software where it's helpful.

And that brings me to slide 78, the agent development lifecycle. So at Sierra, this is the process by which we build and improve AI agents. You might be thinking about it like, oh, that looks kind of like the software development lifecycle.

And I think the devil is in the details, so I'm going to dive in a little bit. It's not that these are revolutionary or innovative concepts. It's that each one of them involves iterative refinement with customers in production to make it as productive and as bulletproof as possible.

So if we dig into quality assurance, for example, if you work at one of your customer, one of our customer companies, you have access to Sierra's experience manager. What that means is that you can dive in and look at every conversation, and you can look at high-level reports of how is the agent performing in real time.

You can provide feedback. So for example, if Duncan Smothers has incorrect inventory, maybe it made one API call to one warehouse, but it didn't make all the API calls that it needed to, or one of them timed out, whatever it may be.

You can report this issue. It then will lead to an issue being filed, which leads to a test being created, and then once that test is passing, we can make a new release. And over the course of time, a Sierra agent will go from having a handful of tests at launch to hundreds and then thousands of tests as it improves.

Another example here is it's not always that the agent is making a mistake. Sometimes there's an opportunity to go above and beyond. Chubbys actually has each of its agents have a budget in order to delight customers. And so in this case, Duncan Smothers could actually, you know, DoorDash the shorts from a retail location if they're not available online.

So this is the agent development lifecycle at work. But the thing is, a year ago, we were doing this all manually. This was kind of early on in the history of Sierra, and we were learning what works at each of these stages.

And with the improvements to AI, we're actually able to add AI to each part of this lifecycle and speed up the improvements in the present day. But it's bigger than just Duncan. The agent development lifecycle is more effective the larger the customer is.

And while Duncan handles hundreds of thousands of requests, we have customers that are doing tens of millions. So the more valuable the velocity and change management are when you're that big.

And the change also comes from everywhere. It's not just that, oh, there's an issue with the agent and we need to improve it. There's tons of stuff going on outside. There's all those graphs at the beginning of this presentation showing how fast our space is moving.

You have models being upgraded. You have new paradigms like reasoning models. You have multimodality and more and more. When we think about how these impact the agent development lifecycle, reasoning models are a force multiplier toward each step. We're actually able to be more effective applying AI to development, to testing, to QA, and every step in between.

Now, another one that's near and dear to my heart, I mentioned the Deepgram workshop eight months ago, which was an accelerant in my understanding of the voice landscape is building for voice. And I started working on this about a year ago, and in October, we were able to launch voice generally available at Sierra.

One of our large customers that has benefited from the agent development lifecycle that has, you know, tens of millions of customers in the United States is SiriusXM. And with Sierra's voice capabilities, they're able to pick up the phoneright away every time to answer their customers.

The way that we think about voice, I think, is similar to the way that we think about web development today. If you remember 10, 15 years ago, a lot of websites were, you know, m.website.com. You had two separate websites for mobile phones and for desktops.

And then we graduated to responsive design. And this is how we think about our AI agents at Sierra too. Under the hood, it's the same platform. It's the same agent code, but it's able to be responsive to whatever channel someone reaches out in and whatever modality you're operating in.

Of course, you can still customize the same way you might have a different layout. You can still have different phrasing. You can still parallelize requests to achieve lower latency. But it basically just works out of the box. I'll close with a few thoughts.

This is something I've been thinking about a lot lately. One of the most fascinating and fun parts about building with AI is that large language models remind us of ourselves. In short, they're unpredictable, they're slow, and they're not that great at math.

But also, it allows us to be great designers by having empathy in a way that we probably couldn't ever before with computers. And so you can actually put yourself in the shoes of the robot. You can put yourself in the, I don't know, primordial soup of the Jello, and you can think about what it would mean to actually build a good experience.

And as someone who's building voice agents, and a bunch of you, I bet, in the audience are, I know there's kind of this thought on, are these multimodal agents the real deal? You know, should I just kind of wire everything together and hope it works?

And the question I've been asking myself a lot lately and what our results have kind of shown us is, you know, how would you do if someone just passed you transcribed text of your conversation partner with a few hundred milliseconds of delay and then you had to respond on the spot?

And so what we're building at Sierra is much more robust and very exciting to me. And I hope to talk to you all about it. I think on my badge, it says voice-to-voice models is the thing that I'm excited about.

And so here is kind of a sense of the robustness and the richness of what you can create when you let large language models have the same inputs and same experiences that humans have. And so thank you for your time today.

I look forward to a lot of engaging discussions, and it's great to talk to you all.