Design Patterns for AI Trust: Juries, Libraries, and Agent Tiers — Alex Bauer, Upside.tech

Well, hopefully this is the one talk that's different from all the others today. So let's start with audience participation. Who here has kids? Ooh, okay, there's a couple. Who here kind of has a parent-child relationship with Claude?

Okay, there we go. I have a kid. This is Otto. He learned to drool this week, which surprisingly is something that these things don't come out of the box with. He also discovered his fingers, but he doesn't know about his hands.

So they're fascinating. They're in front of his face, but he doesn't know how he got there, and he can't bring them back. But anyway, I finished these slides this morning at 2 a.m., and he decided that playtime started at 5.

So we're going to see how the rest of this talk goes. But clearly, I need practice telling bedtime stories. So we're going to do one of those first. Once upon a time, a go-to-market team set out on a simple quest.

They wanted to know one thing: how their business was doing. Between them and the answer lay one small, tidy pile of data.

And a dragon. Because, of course, the best quests always have dragons. And every quest also has guardians. The first was large, worried, and had a great many questions about their security posture. They defeated it in the way that important bosses are usually defeated: not with a sword, but with paperwork.

The next monster was worse. It had many heads, and no two of them could agree on what a single word meant.

To beat it, our heroes had to accept a hard truth: the systems they trusted had never actually promised to tell them the whole story. So they stopped believing any single one of them and started building something underneath all of them, a place where the data finally agreed with itself.

And then came the final boss. It didn't attack. It did something even worse. It asked one question, the hardest one of all.

Now, our heroes had learned a few things by then. They'd seen what happens when you send a brand-new adventurer in with no guidance at all. And they realized the secret: you don't need a genius adventurer. You need a well-prepared one.

First, you write down how your world actually works. Then you give every adventurer a knowledgeable guide. They can ask for directions. And then the adventurers start teaching each other so that no one ever has to start from anything and nothing again.

So they gathered up everything they'd learned: the permissions, the definitions, the knowledgeable guide, the habit of asking for a second opinion before they believed anything. And they forged all of it into a single tool. And when they had finally reached that tidy data pile, they found out that the dragon had never been the villain.

And so the kingdom entered a golden age of growth and creativity. The end.

Just kidding. That's only where the bedtime story ends. The actual talk starts here. Because every part of that story is a real problem. And I want to show you how three of them actually get solved. So our agenda for today is I'm going to talk a bit about the age of agentic GTM, and then we're going to walk through three actual examples that we either have worked through as a company recently or that we've built into the product that we sell to our customers.

So briefly, to establish credibility here, I'm going to give you one minute about our company, Upside. We are the data layer for the age of agentic go-to-market, which means that we start by helping you get go-to-market data ready for AI.

And that means knowing what actually happened, with a data foundation and a dashboard you can use, and then helping you answer things that used to be impossible at scale with things like deep research or a multi-touch attribution model.

We do a lot to help you make AI correct about your business so that it's not hallucinating and giving you trust issues. And then we make sure that this data is in all the places that you want to use it.

So a lot of things you used to spend money to buy are actually things that you, the person closest to the problem, can now build. So with that established, this is the age of agentic go-to-market. We have this interesting situationright now where go-to-market teams have not historically been the place where you'd find a high density of builders, at least compared to product and engineering.

Obviously, there are exceptions to this, but in general, most marketers and salespeople were never writing a ton of code. And that's very much not to say they weren't aware of their problems and didn't have ideas of how they would solve them.

But the historical toolbox here was basically spreadsheets and PowerPoint slides. And AI has changed that. Claude does for building basically what the bicycle did for mobility. It makes it accessible. And so spreadsheets and slides are no longer the default business tool that everyone has available, which means that a lot of people who historically had ideas and didn't consider themselves builders now can be.

Before AI, I would have described myself as technical enough to be dangerous. I can read code. I can write code. You probably wouldn't want my stuff in production generally if you could help it. But I'd be interested, who in this room today considers yourselves an actual engineer?

Okay, awesome. Thank you for being brave. Who would tell themselves that they are also technical enough to be dangerous?

Allright. And who doesn't read code at all?

Okay, thank you for being brave in the back. Those categories should have been mutually exclusive, collectively exhaustive. So if anybody didn't raise your hand, I recognize that there's no free coffee here, and I am the first to talk after lunch.

But the thing about AI is it makes all of us technical enough to be dangerous. What AI has done for those second two groups is basically provided you with an infinite supply of valedictorian interns with computer science degrees who can help you implement all of the solutions that you probably know better than any engineer that you'd work with on a different team.

And that incredibly speeds up iteration. And it also kind of makes me feel like I'm Harry Potter walking around with a magic wand for the first time. But essentially, when you set interns loose on problems led by non-technical people to build things, you're going to get some interesting results.

And a couple of years ago, I remember that everyone was talking about the AI hallucination problem. Somehow, that big word seems to not come up as often anymore. But I think we have its older sibling now. It's a trust problem.

And if you ask Claude to do something like report on revenue, it doesn't say, "I'm not sure." It says, "Here you go." And it gives you a wrong answer that looks exactly like beingright. So a lot of the talks this week, I think, have been around the bleeding edge of technical optimizations.

And a lot of the influencer fluff you see on LinkedIn and Twitter tells you about the magic improvements if you do your prompt with exactly theright incantation. I think both of those things might well be true, but I don't generally have time to do either of them.

And in practice, I find that the most practical thing for working with AI is something that we all know pretty intuitively anyway, because we have a lot of experience working with other people. And in that framing, establishing trust actually isn't new either.

We already know a lot about how to do this for people. So the main thesis of the talk today is actually, when in doubt, manage your agents like other humans. If you were helping a team of humans figure out how to do something, great.

Do that with your AI agents too. And if you take nothing else from this talk, this is my one practical tip. Use commander's intent when you prompt. This is something that comes out of the armed forces doctrine. But basically, tell your agents why you want them to do something, and they will do it a lot better.

And by the way, this works for humans as well. People don't generally like being micromanaged. So give them commander's intent, and you'll get great outcomes. But beware, because the agents have been trained on material from other humans, and so they like to micromanage themselves.

And that's not usually what you want. Don't tell Claude to improve itself. You'll get micromanagement. You have to pull it back and say, "Remember, we're talking about the why." Anyway, onto the examples. So the first one here is I needed to redo our entire website recently.

And at my previous company, which was 600 people and I led the product marketing team, this would have meant several product marketers, a designer, a couple of consulting PMs, a web team, and it would have taken two months.

And so in general, with AI, sometimes you just kind of had to give it a go to see if it's improved enough from the last time you tried. And so I did that, you know, YOLO mode. Here are the sources.

Go read everything. Please give me my website. And it was obviously a fail, even with using a nice plan mode in Claude first. I'm sure this will work eventually, but it doesn't today. So you have to do some scaffolding.

And by scaffolding, what I mean is you have to tell it what to know about your business. So I maintain this list of anchor assets for our entire company. I think documentation is something that humans generally should have used anyway, but that was often something that was also generally delegated and forgotten.

Anyway, I maintain these anchor assets. And one of the really important ones here is the product capabilities reference, because on a website, you want to talk about your product. And this is something that I actually did use AI to compile.

So this wasn't me sitting here and typing all of these out. But if you pull up one of these cards, you'll see that it tells you what does the product capability do, why it matters for a bunch of the personas, which are also defined in these anchor assets.

And then it also shows the track record of how Claude found all of this. So these are citations from across every system that it's connected to. I can follow them back if I need more details, but that helps me know that it didn't hallucinate the important parts.

And if you look back at what we had on this website, that actually ladders pretty much directly up to what was on the homepage. And then if I were to go into these product pages, it also ladders up there.

So the step here was define the structure first and then turn Claude loose. Don't try and YOLO it from the beginning. If this kind of thinking is new to you on product capability references, by the way, this I think is probably the best book on how to do it.

There's also a nice summary here that someone wrote up online. Particularly steps five to six, I think, are the important ones for this. But the whole book is worth reading, honestly.

This ticker is staring me in the corner of the space. So I'm going to give you five more seconds to scan that, and then we're moving on.

Two other important things: a general DSLOP skill. There are a ton of these floating around. And also consult persona bench. I have a set of agents that inhabits each of the important personas, and I can have them review something on demand and give their opinions.

But the second one is our radiant librarian. So this is something that actually exists inside our product. And I'm going to show you how it works with a diagram. So this is the librarian. If you, the user, have a question, "How much pipeline did we create in Q1?"

then your agent is happily going to go off and say, "Oh, okay. Well, quarter means January to March. I'll just look at created date." And that is probably not correct. So what we actually have it do is consult the librarian first.

And the librarian has access to documentation and the library of knowledge items about your company and the schema of prior failed queries. And so it basically gives your agent a just-in-time memory of all the important things. So we know that the fiscal year here is actually February through April.

And new pipeline means things that only meet stage two or later. And then you get a nice trustworthy answer with citations back rather than something that you discovered for the first time. I was going to do a live demo of this, but I think we're going to skip that for the purposes of time and move on to example number three, which is what we call our jury and judge workflow.

So we started this company to solve multi-touch attribution, which is kind of like the holy grail of anything in go-to-market. And it took us two years to get anywhere close. But along the way, we built this AI-native data layer and then discovered that that was actually what everybody needed for everything else.

And when we came back to multi-touch attribution enabled by Opus, we actually get really interesting results that are actually pretty impressive. So we're going to look at how this workflow works, because there's this classification of challenges in go-to-market where there is no empirically correct answer.

And we have a model for how we deal with that in the real world. It usually involves a trial by a jury of your peers. So here, we're doing a very similar thing where you might say, "I'd like attribution on the Acme deal."

And your agent is not going to go off and immediately answer this on their own. They are going to spin up a team of independent analysts who all look at the data independently and come up with an evidence-sided opinion for what they think the attribution credit of that deal should be.

And none of these are necessarily correct, but they are independent research that then goes back to the consensus judge who says, "Oh, I'm not treating these as fact. I'm treating them as input. And I'm going to weigh the reasoning quality of each of these analysts.

And then I'm going to help you come up with the final version here. And if there's not enough consensus, then I'll escalate and expand the jury." But my job is not to do research on my own. It's to say, "Here's the final result based on a whole team of independent analysts doing a really good job of research."

And this is something that we see in the real world. We would do it with humans too. Turns out multiple researchers with somebody who helps at the end is better than a single person kind of perseverating on that forever.

So thus ends the three examples. But I have a bonus. So bonus isright here, agent tiers. In my experience, you can't fix stupid. So this basically means friends don't let friends use really bad harnesses or low-intelligent models for important work.

An example of this is Slackbot released its MCP client functionality a couple of weeks ago. So you can now hook Slackbot up, and it will talk to MCPs. And I thought, "Oh, that's awesome." Now everyone who has Slack can suddenly consult the Upside librarian.

And I was so excited to try it. And it turns out that this is horrifically stupid. So basically, the general case here is any AI product where it's been crowbarred into a pre-I subscription model is probably not something that you should be using for anything important, because the margin on those plans just doesn't leave enough space for an intelligent reasoning model to work.

So find something that is at least tier two here. And tier two means it's got to be a powerful model. You need to have attributes like sub-agents, plan mode, full MCP support. It should be able to use file editing.

And there are a lot of these out here. So do not let your team just use the ChatGPT web interface and expect that it will be a great result.

And that ends the talk. So I hope you stick around for the rest of the track. I know Jeff from X is coming up later, and he's got a really banger of a talk about how they basically created Upside for all the rest of the world's data.

And I shall be outside for a few minutes if you have any questions. Thank you.