BDD, ADR, PRD, WTF: Capturing Decisions for Humans and AI Alike — Michal Cichra, Safe Intelligence

Hi, I'm Michal. Uh, welcome to "Capturing Decisions for Humans and AI Alike." Yesterday, with a team from Safe Intelligence, we have released Spec 27, a new product to test agents. Before that, I was in Microsoft, Red Hat, and spent 10 years working on a single product.

The consistency problems we face with AI, and the story of capturing decisions, show up in every product I have seen. And these notes are distilled from that experience. And you can find me at the booth. Um, so BDD, PRD, ADR—like, that's a lot of acronyms.

Uh, why does any of that matter? So let's unpack it from the end. You probably know this story. Uh, I hope it's not an urban legend. Uh, but scientists put five monkeys in a cage with bananas on a ladder, then gave them a cold shower every time a monkey tried to get a banana.

Other monkeys beat up the poor fella. Then they replaced the monkeys one by one, and none of the originals remained. And yet, they have beaten up every monkey that tried to climb the ladder, not knowing why.

So humans and LLMs, they suffer from the same trait: limited context. People forget. LLMs context compacts. Humans leave. LLMs have no memory. After a while of operating a product, the team starts asking, "Why do we have this flow?

Why is this goal of this feature? Why is this code shaped like that? Why—where does this belong?" And you might not have the founding engineer available to answer. And these problems show in every org, uh, maybe with AI much sooner than they used to.

So ADR is Architecture Decision Record. It records why you do something and how you enforce it, or how you want to do that. And you can cover examples by reference docs and code snippets. For example, we split code in layers to prevent N+1 queries.

We enforce that split by linting imports in modules. And we also enforce reading from database. It returns, uh, plain shapes instead of ORM objects. So we cannot, um, cannot make these, uh, these queries. And to prevent duplication. And also linting it by module imports.

And another, like, 50 ADRs that define the architecture of the product. There is not a single format that you need to use. It's just a concept. Um, it's a text, so there is no specific, uh, way how to enforce it.

You still need a tool to enforce it. But the tool will tell you that this is the rule. Why are you doing this? And how are you supposed to fix it? Then the agent will go and try to find this document, why this reason exists, and more information about how to fix it.

Also, you can define, like, which files it actually concerns to. Like, is it some Python files or some folders? And how you actually enforce it. PRD is a Product Requirements Document. Uh, that's something lighter. When you're building a feature, you describe why that thing exists and what problems it solves.

And how a user goes through the app to actually interact with it. What's the journey through the application? It can be very light. It doesn't need to be really long and exhaustive, like a massive document. Uh, you can just capture why, the problem, and the goal, and the journey that connects them.

And it's not just for the agents, but also for you six weeks from now, when you forget why you did that.

Now, BDD. Um, it's a Behavior-Driven Development. You have probably seen spec-driven development lately. Uh, but if you practiced it, uh, you might have suffered the same thing as me. How do you validate that the product actually adheres to the spec?

It's a markdown document. You describe how it's supposed to work, but how do you know it actually works like that? One thing harder than reading an AI code is reading AI tests. Um, so what if you had an intermediate layer that actually describes how the product behaves in a human language?

And BDD is not new and shiny, but it's, it can be executable and readable. So enter Cucumber. It's almost forgotten, suddenly useful again. It's definitely easier to review than your average tests. You can connect scenarios directly to your PRDs and critical user journeys.

It can be readable, executable, and it closes the loop that a spec-driven development leaves open. These rules, uh, these specs are later parsed by steps, and they are executed as code. But what you can do is that you can actually write and read these.

And you can review these, and you can understand these. The language is on you. It doesn't need to be, um, enforced. Like, you there are multiple ways how to write these, uh, these features. And

that's it. But they describe how you're supposed to go through the application, why this thing exists, and how it runs. And similarly, they can refer back to all the documents that you have about why things exist.

So, and as a bonus, making consistent UIs with agents is just another level of hard. Like, Design System and Pattern Library are the way to build consistent UIs. Like, there was the way before AI, and it is the way now.

So you document your language. You say, for example, a primary button is this and that. It is blue. It has this shape. It has this color. And it's this size. And you say your rules. You say, "We will have only one primary button visible on a site at any on a on a page at any point in time."

And then you can enforce these rules. Similarly, you define components and patterns. So, for example, if you have multiple colors of these buttons and multiple states, you define components, and you define previews, and you demonstrate how they work.

And you create snippets of previews so you can actually see them. And the agents can see them. And then you can go and review and, like, do these actually adhere to the principles that I have? Do they adhere to the visuals?

And then you reuse them. As with code, you build these from the ground up. From small pieces into bigger ones, you compose them, and you reuse them. Otherwise, it's, uh, chaos, like with a code.

So, cool. These are cool ideas, but how do you actually enforce this? So my team and agents stick with it. How do I keep it consistent? Well, with the loop. So you probably have heard about closing the loop, reinforcement loop, the harness, how to remind the agent that there are rules and how to follow them.

So our loop is simple. It is git hooks, scales, CI, and, uh, linters and a bunch of other checks. Agent's goal is to deliver a pull request. And they to do that, they need to use git. So we use git hooks to run predefined tasks.

And these tasks are later executed on a CI. They are the same tasks that they are executed as, as hooks. If, for example, agents would get lazy and not want to execute them or skip them, then they get caught.

And we include linting, formatting, type checking, code duplication, architecture checks, document linting, everything that's, that's possible. So there was a time where code reviews were about style and tabs and spaces, and there is no space for that anymore.

All these things are not for discussion. They are rules, and they are enforced, and they are automated because there is no space for discussion about these anymore. It's more about a high-level concept. What you cannot find, you cannot enforce.

So, for example, we enforce architecture of the product and of the code. We separate modules, um, and their imports. So what you can use from where. For example, our end-to-end BDD test suite cannot access the database. So we forbid from accessing any module that could access the database and basically force the module to the models to iterate without the database and really use only the browser features of the application.

Similarly, in the product itself, we enforce we cannot talk to the database from rendering templates. So we know that there are no N+1 queries, ever. You just define ways to prevent these problems from happening, ever. You cannot keep finding them.

You need to prevent them entirely.

Then the com the agent tries to commit it and push it, and they get feedback on the commit and get rejected. And they get linked back to the document, and they go read it and fix it and iterate.

So there are some drawbacks. Um, it is, uh, sorry, it's not drawbacks. Um, so this loop is generic. Uh, this loop where they do some work, they, they push it, and they get feedback, and they iterate. But the loop can be multiple things,right?

Like, sometimes you're working on a product feature, sometimes you're working on a UI, sometimes you're working on more back-end issues, back-end-ish things. So that loop is the same, but what changes is the focus of the loop. So we have different scales.

Uh, there is ADR that's whenever there is an ADR mentioned, the agent will look up ADRs, how to operate with them, uh, how to find code that affect that's affected by these ADRs. For PRD, the same. For, for UI loop, we actually skip a bunch of checks and rather force it to iterate in a browser quickly.

And test skill that actually identifies tests to run based on code coverage and file changes. So we run just the focused part of the suite and not the entire suite. And some goal execution to actually keep decisions that the model made so we can review them later.

But all of these provide focus in the loop, but the loop stays the same. There are drawbacks. Uh, it is very context-heavy. Like, you can run out of half of the context, uh, in, uh, starting the research. Um, but I have no fear of context compacts.

Like, this actually, for, like, the last half year, actually works, I think. So in my sessions, there are 20, 50 context compacts, and it's, it's okay. Because the important things survive, and the agent will always look them up again.

So, and that's the goal anyway,right? Like, you want to have multiple-hour sessions with a clear goal that the agent can operate autonomously with the rules that you define. So that's the goal anyway. Like,

so

there are decisions that you can record. There are parts of the product that you can describe why these exist. There is, uh, Cucumber or BDD that can have executable specifications that you can actually read and review and understand.

Design systems can help you to build consistent UI from components and, um, again, enforce it that, for example, there are no inline styles anywhere else. And you employ harness to loop it all together. So may the spec be with you.

That's it.