Shipping Products When You Don't Know What they Can Do — Ben Stein, Teammates

Uh, yeah, I mean, the actual title has curse words in it. I will probably be cursing a lot. I didn't know if I would get into the track of I actually published the curse words. I'm one of the founders of Teammates.

I'm going to wear my product manager hat today. I'm assuming this room is, like, mostly product folks, probably product-minded engineers as well. But I'm going to just, like, wear the product hat. A little bit about Teammates, very quickly: we make a platform for designing and managing an entire digital workforce.

So in AI engineer parlance,right, we're building agents. But I would think of it, like, two ticks up from that, because what we really believe it is the experience, the interaction patterns of humans and computers working together. So I want to talk to you about my favorite teammate.

This is Stacey, Stacey Hand. She, she actually got promoted since this slide. She's an L3 engineerright now on our team. She's awesome. She looks like a hamster. All of our customers get to design whatever teammates and avatars they want.

They give them personalities. It's all really fun. And Stacey lives inside all of our collaboration tools,right? So she has a Google Workspace account,right, for Gmail. She has a Slack account. We truly leaned into giving all of our teammates identity.

And she sends emails, or I forward her emails, and she hangs out in Slack, like, in the public channels. And she's Gen Alpha, which, like, is, I don't know, I feel really old. I don't know what she's talking about.

She's constantly like, "6, 7," and I'm like, what are you talking about? And I can tell from this room that none of you have 12-year-olds.

No.

No. Okay. There you go. So yeah, you're rolling your eyes as well. But anyway, this is Stacey. And this is sort of how my sales pitch goes,right? It's, it's, you know, a little more formal than this, but, like, this is generally the pitch.

And

I got asked a question at some point recently, which was, "Oh, yeah, more of the pitch,"right? She, like, shares Google Docs, Google Sheets, and she said, "Hey," or a customer said, "Hey, can I tag my teammate in a Google Doc comment?"

And this gave me pause, because I was like, well, I had never actually thought about that before. And so in the back of my mind, I'm like, well, of course you can. Your question is, like, what's going to happen?

So I'm like, okay. So I'm like, you know, doing math in my head. I'm like, okay, well, we don't have webhooks. She probably won't, or, like, a webhook from the comment. Okay, but she's going to get the email notification in the email that comes from Google.

Does it have the comment and the conte or maybe a link? I'm like, I have no idea,right? I actually don't know what's going to happen. And this was, like, the impetus for this talk. I was like, how do I ship a product?

How do I develop a product? How do I talk to customers? How do I instill trust when I don't know what my own product can do? And, like, it's really weird. And sometimes I'm like, well, is this just because I'm an idiot?

And, like, well, since it's my talk here, I'm going to say no. And sometimes I'm like, well, is this because what we're building is so far out there,right? These just, like, truly autonomous agents that can use any and, like, I don't think that's it either.

I think what's happening is the product management discipline is going to undergo a transformation, a shift, an evolution, whatever you call it. It is super profound, and we may or may not totally realize it yet. Because I think in the engineering world, we're like, oh, well, we have, you know, tools in our IDEs, and we have codegen and, like, we sort of are starting to squint in understanding maybe how the discipline is changing.

I don't think we really understand how product development is changing and evolving, and, like, what are the new tools and practices, and how do we forget everything we've learned in the past?

Why is this true,right? If it's if the answer is not Ben's an idiot and the answer is not that we're way out there, it's two reasons. Number one, if all our products are built on top of LLMs, and plus or minus they are, like, we don't know, and we can never know, what the LLMs know,right?

So, like, inherently in what we're building is, like, we don't know what the foundation is. Like, you don't have to know what your database, like, how it works, but, like, you generally know that it's, like, the surface area, the interface that's exposed.

We don't understand this for the, the models. And the other thing is the expectations from customers are just boundarless,right? We're just like, "Hey, here's a text box." I mean, that's probably not a good interface, but, like, essentially we're like, here's a free text box.

And if it's anything other than, like, a help me write button, you're essentially inviting customers and users to just do whatever they want,right? So we have this, like, boundless surface area built on top of a product that we don't understand.

And so the question now is, like, how do we adapt? So let's me, let me actually pick on this Google Doc comment thing for a second,right? So if I was wearing my, like, traditional PM hat, I'm like, okay, well, I need to make a feature that's going to

read and respond to Google Doc comments. And so in my head, I'm like, okay, well, does Stacey have access to the Google Doc? If she gets tagged in the comment, should she reply directly in the comment? Should she reply at all?

What happens if somebody else comments in the thread? What if someone comments in the thread that's not addressed to her? What if it's someone else? What if it's, what if it's her doc and someone else commented to someone else, but she gets the note?

Like, there's just so much to, like, think about and reason about. And so I'm like, okay, well, I'm not building a Google Doc commenting product, so I'm not going to spec all of those things out. And, like, what's worse is, like, you also probably want to tag her in linear tickets,right?

And what's, what's the book? Like, if you give a mouse a cookie,right? It's like, if you give a mouse a cookie, well, you probably want to, like, tag her in Figma as well. And you probably want to tag her in LinkedIn posts.

Like, and so we're not a team that's building a generic commenting reply agent system,right? So then the question is, like, what are we supposed to do,right, as, like, a product manager who realizes, okay, I have this, like, boundless surface area, how does the practice need to change?

Right? And, like, sort of this is the core of, like, what I want to, what I want to talk about today. So I'll do, like, three highfalutin ivory tower ideas, and then I'll talk through some, like, practical ways to, to make this real.

So the first one is this mindset shift to, like, think in affordances and not, like, specific requirements. So it's not if, you know, as a user, if Stacey replies in the comment thread and she has re like, that's not how we would think about it anymore.

It's the affordance. Oh, she has affordances to comment, or she has affordances to communicate, or, or to email, or to collaborate. We're going to trust the LLMs. We're going to trust the agentic workflow, the work planning, like, all of the things inside of our, you know, our beautiful 12-factor agent.

We're going to assume that that we'll understand. But it's the affordances that we need to think about, not the individual features. Which is really weird, and it's not typically how product people have ever thought before. And I would say, actually, this goes even further, which is behavior is emergent.

And this was the other thing that I did not expect at all, like, starting in this space, was we don't not only do we not know what things work, sometimes they do, and they work in ways we didn't expect.

And so I feel like our job as product people is to discover functionality. It is what are theright building blocks,right? What are theright Lego bricks that we either give our engineering team, our product, our customers, let them compose?

And we discover emergent behavior. And that is one of the reasons that, like, this is the most exciting time I've ever built. Because we're actually building things and then discovering what they can do themselves. And that sort of became the new job, in a sense, is discovering what's possible.

Because if you asked me, I could not sit down in front of a Google Doc and be like, oh, let me, like, type out what this thing should. I can't. I don't know how to do it. And, well, even if I could, how do I then communicate it,right?

So how do we communicate to a development team, to a backlog? How do you communicate exactly what should be happening? It's like, Figma doesn't, like, have the affordances for this,right? My, my PRD doesn't, like, have the affordance for, like, well, you should probably talk a little bit less Gen Alpha because you're making Ben feel old.

Or, like, hey, you should be really like, how do we communicate and express these, these concepts,right? So I think these are, like, the three, you know, high-level ways that our practice needs to change. But, like, let's make it a little more concrete.

Okay, so evals. I'm talking about evals. Okay, it's really hard to make a slide with graphics of evals. I feel bad for the eval com. Like, how do you illustrate an eval? So I'm going to make you just look at pictures of various teammates from, you know, across all of our customers.

Okay, who hates raising their hand at conferences when the speaker asks them? Okay, awesome. So here's my question, which is, okay, for the engineers here, who, like, legit, like, don't lie, like, writes and runs their evals? Good number.

And of the product people, who has visibility into the evals? That's not bad. And, and do you look at them just because you have the visibility? Allright. One, one and a half, two. Okay, great. So I would posit that evals actually, I'll back up,right?

So we all talk about evals. We're all going to be embarrassed to say that we don't know really what they are. Evals are a testing framework for probabilistic AI for agents,right? Like, if we think about the deterministic code,right, I withdraw $100 from the ATM.

My bank account should have $100 less,right? Great. And I can test that, and I can write code to test that. When the test is, like, was she snarky in Slack? It's like, well, how do you test that? How do you write that test?

Right? So we come up with this, this whole new discipline of evals, which is, well, she should be a little bit snarky and a little bit funny, but not mean. And then we hand it off to another LLM to say, okay, well, hey, was that reply, like, did it meet that criteria?

And how often did it? It doesn't have to be 100%,right? So she should be, like, pretty snarky, but, like, not mean 80% of the time, or whatever the business logic that you want,right? So these are evals, and this is the world of evals.

But here's what I would posit, which is it is the only way that we know what our software can do,right? And which is why I love the idea of product people looking at the evals,right? Looking at because they become the new specification for the product,right?

And so as we're watching, you know, if you're downstairs in the expo galley, you're seeing, like, new software. It's like, hey, bring the team in. And this sort of it a little bit reminds me of, like, the old, you know, for the, the old-timers here, like, behavior-driven development.

There was this period of time when it was like, oh, the business people are going to write the tests, and that will get converted to code, and then the code will run. And, like, the truth is, like, no one ever wanted to do that.

Like, no business per I don't even know who a business person is, but, like, they wanted we're going to do that. But I actually think this is different, and I think this is pretty a meaningful way to actually understand what the product can do and a little bit begin to specify what it can do.

Okay, so I'm vibe coding for a second, which we, which we all do. We all talk about. But I want to talk about vibe coding in a, in a way that's really constructive. And how do I sort of say this?

It's very, very hard. I think I, I kind of was like, like, oh, you can't do it in Figma. You can't do it in a PRD. Like, what do I really mean? Well, it's very hard to, like, sit down in front of a blank piece of paper and write what the teammate, the agent experience should be.

It's just really hard. It's hard to, like, imagine it. And it's not until you feel it. I mean, so much of what we're doing in this, like, human-computer interface is visceral. It's feel. It is like, oh, well, like, did they ask too many questions?

Like, how many questions is too many? Oh, wouldn't it be great if they clarified exactly what you meant? Well, turns out that's really annoying. But when I wrote, like, the first spec, I'm like, then the teammate should ask a lot of clarifying questions.

And we gave it to users, and they were like, this sucks. And I was like, how would I have ever known that? And the answer is because it's so easy to prototype and vibe code something and get the feels.

And so this is the next thing that I'm, like, pretty excited about as a new product management tool. It is being able to feel and experience what it's like to interact with a computer, but

without just, like, writing it or hoping that you have a clickable prototype that will work. I will also mention that we have to be careful with vibe coding because I do not mean sit in the meeting and say to the engineering team, how come this is taking two weeks?

I finished the feature during the meeting. Like, that doesn't, that doesn't win you any points,right? So it is no, no, no. This is never going to production. But what this does is it gives you the feel, the, the experience,right?

And so this is, like, the only way I know to, like, actually test and feel it out. Do you, do you remember, like, the, the Claude certainty issue? Certainly, I mean, certainly. There was this peer,right, where every time you asked Claude something, he'd be like, certainly.

And, like, that probably, like, seemed really good when you were testing it for the very first time. And then, like, the fourth time when you're like, hey, can you do my taxes? Like, certainly. Can you write my, like, acceptance speech?

Certainly. Like, oh, this is actually really annoying. But you don't realize that until you experience it. So, like, that's why I like the vibe coding. Okay, so great. We did all this development. And then the question is, like, hey, we pushed to prod.

Does it work? I'm like, I told you I don't know. So the question is, like, how do you test? How do you, like, know that it's going to do the things that you said it was going to do?

And I sort of alluded to this. I'll go through this quickly. It's just really discover, discover the functionality. And there's an old joke. I'll tell the joke. QA engineer walks into a bar, orders a beer, orders two beers, orders zero beers, orders negative one beers, orders a lizard, orders a beer with an emoji,right?

It's like, great. This bar is good to open. And the first customer walks in, asks where the bathroom is, and the bar blows up,right? Like, great, great old joke. It's kind of how I feel these days. Like, I just sit in.

I'm like, oh, you know it would be cool if they were to, like, start posting comments on LinkedIn. And what if they were, like, every time I added, like, a track to my Spotify account? They kind of, like, these just, like, crazy ideas.

But this is where, like, the emergent behavior comes from,right? And so it's this mindset of, like, let's just try. Let's just experiment. And it's, it's this, like, kind of growth mindset shift from, like, I'm going to write the features and the requirements to, no, we're going to figure it out.

This was a little bit unexpected for me. And this is

how do you sort of report to engineering and then have things fixed by engineering? And what counts as a bug in this world? And that is really, really strange. And I think as sort of, I don't know if it's, like, just a product role or maybe in a support role.

Like, how do you know what is appropriate to escalate, to put onto the backlog, to flag as a bug,right? It's like, I'll keep picking on, on Stacey. You know, she, she gives me a really hard time, so it's fine.

It's like, hey, she used too many emojis. Like, put it in, in, in linear. It's like, well, it's not really a bug. Like, show me in the spec where you told me not to use too many emojis,right? It's almost like, like, in our tickets, it's like, oh, you know, closed, done, closed, duplicate.

We need, like, closed. LLMs be, like, crazy, yo. Like, I don't know how to fix this, like, just because it's probabilistically generated. So how do we know if it'sright or wrong? How do you know if it's a feature, if it's a bug,right?

And I think there's this element of credibility that we need to build up. It's like, hey, we actually under we understand that for some use cases, like, 80% is good enough,right? This eval, we talk about evals, if it's passing 90% of the time, like, that's a go.

If it falls below 90%,right, that's red, and we're not going to ship it. So I'll actually come back to evals for a second. Because if the eval becomes the spec, and we can say, hey, we said at, you know, 100%, even though this is probability, you should never give a refund if a customer, like, can't prove that they bought the thing or whatever.

Like, it is. It's like, great. That is our metric. And we could say, yeah, this is a bug. But if it's just a, a feel, it becomes really difficult. Again, this was totally unexpected that, like, debugging and assigning bugs would become, like, controversial.

Okay, customers. So this part is

I found this really weird,right? So if I think about, like, not wearing my, like, founder hat, but wearing my, like, typical product manager hat,right? Like, I go into a customer meeting, usually with a salesperson. I'm like, I'm going to play a role,right?

And so what's the role? Well, I'm either going to play, like, visionary. I'm going to, like, hey, here's our vision for the product. Here's our roadmap for the future. Like, let me help you understand customer, like, how you're going to come along on this journey with us.

Or sometimes I'll play the role of honest broker,right? It's like, listen, sales has, like, given you a whole bunch of, like, they're just, like, selling you a bunch of vaporware. Let me tell you what's real. Let me tell you, like, exactly what you can expect.

And that's a role that you play,right? And I usually, like, preface this with, like, the sales team beforehand. It's like, yeah, I'm going to be the honest broker, and, like, we'll give the customer confidence. Today, I'm like, okay, I told you our vision for the future, our roadmap, and the customer's like, you're full of shit.

Like, none of this actually works. I'm like,right, I can't really paint the vision because no one actually believes it. It sounds like witchcraft. And then I'm like, oh, well, then I'll be the honest broker, and I'll tell you how things work.

But I just told you I have no idea how it works,right? So this became very strange because I can't play either of the roles that I'm supposed to be playing. The future sounds like witchcraft. The present is literally, I don't know.

So how do we do this?

I'll tell you how I've been doing it now. I don't know if this is, like, a 2025 answer or if this is, like, a durable answer. Like, if we believe that all of our products are for, like, for all time going to be probabilistic, then, like, we probably have to figure out how this world works.

What I've been doing now is really saying, look, we're inventing the future together,right? We're pulling the future forward. The reason you are talking to, like, a crazy startup like this, and you are thinking truly about, like, the future of how, you know, AI and agents are going to transform your business, is because you are a future thinker, and we are going to do it together.

And it's a little bit like, hey, let's compliment the customer. Let's, like, but it's not just, like, a false, you know, blowing smoke. It's like, no, truly, we need to figure this out together. And, you know, for 2025, I think that's actually the thing that is working the best, best for me.

It's like, no, no, no. We have to do it together. And honestly, if you are expecting something different, like, it's not time. It's not time for you to, like, embrace this world because this is, this is the, the way this world is going to work.

And so I don't know. I'll conclude with, like, I have never had more fun building. I've never felt, like, both more inept and, like, more excited about what, what I'm doing, or just the experience of throwing something out in the world and then just, like, having my jaw drops.

Like, I can't believe this happened. And not only that, when we upgrade the models that are, like, underneath them, they just suddenly get smarter. And that's really weird too,right? It's like, all of a sudden, they, like, start checking their work.

They're like, oh, yeah, I just did a query to make sure that the row is properly inserted. And I was like, hmm, who told you to do that? I'm like, I don't know. It just seemed like a good idea.

I'm like, that is a good idea. I wish I thought of that. But anyway, but I think this is the new world that we're working in. The discipline, the product discipline, I think, is going to change for everyone, and it's going to change faster than we expect.

And we all need to, like, adapt to just, like, operating in a world and forget so much of what we used to know,right? A lot of the core, core ideas, listen to customers, solve real problems. Like, all of that obviously still applies.

But the tools, the techniques that we've, like, relied on forever, I think, are all getting upended. And so anyway, glad you're all at the AI Engineer Conference. It's awesome to have product people here working together because, you know, we all have to, you know, build awesome products together.

So thank you very much.