Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro

For those of you who haven't heard of us, Kiro is an agentic ID. We launched Generally Available this most recent Monday, I think the 17th, but we launched public preview on, uh, in July. I think July 14. So we've been out there for a few months, getting customer feedback, all that good stuff.

We're going to talk a little bit about using Spec-Driven Development to sharpen your AI toolbox. I did a show of hands; about a quarter of the people here are familiar with Spec-Driven Dev. My name is Al Harris, principal engineer at Amazon.

I've been working on Kiro for the last 2 years, and we're a very small team. We were basically 3 or 4 people sitting in a closet doing what we thought we could do to improve, um, the software development lifecycle for customers.

So we were charged with building a development tool that answered, um, that improved the experience for Spec-Driven Development. We were theoretically funded out of the org that supported things like QDev, but we were purposefully a very different product suite from the QE system to just take a different take on these things.

So we wanted to work on scaling, you know, helping you scale AI Dev to more complex problems, improve the amount of control you have over AI agents, and improve the code quality and maintain it, uh, reliability, I should say, of what you've got out the other end of the pipe.

Now we're back to new content. So our solution was Spec-Driven. We took a look at some existing stuff out there and said, "Hey, vibe coding is great, but vibe coding relies a lot on me as the operator getting thingsright.

That is me giving guardrails to the system. That is me putting the agent through a kind of a strict workflow." We wanted Spec-Driven Dev to sort of represent the holistic SDLC because we've got, you know, 25, 30 years of industry experience building software, building it well, and building it with different practices,right?

We've gone through Waterfall, Agile, XP. We have all these different ways that we represent what a system should do, and we want to effectively respect what came before. So this animation looked a lot better. It was initially just the left diamond, but the idea was, hey, you know, you basically are iterating on an idea.

I think like half of software development is discovery requirements, and that discovery doesn't just happen by sitting there and thinking about what should the system do, what can the system do. We realized, though, kind of working on this, that the best way to make these systems work is to actually synthesize the output and be able to feed that back really quickly.

Things like your input requirements, to actually do the design and feed back, you know, realize, "Oh, actually, if we do this, there's a side effect here we didn't consider. We need to feed that back to the input requirements."

And so this compression of the SDLC evolved to bring structure into the software development flow. We wanted to take the artifacts that you generate as part of a design. That's the requirements that maybe a product manager or developer writes.

That's going to be the acceptance criteria. What does success look like at the end of this? And then we want to take the design artifacts that you might review with your dev team, you might review with, you know, stakeholders, and say, "This is what we're going to go build and implement the thing."

And we want to make sure that you can do this all in some tight inner loop. And that was initially what Spec-Driven Dev was. What Spec-Driven Development and Kiro is today, or at least was before we went GA, was, you give us a prompt and we will take that and turn it into a set of clear requirements with acceptance criteria.

We represent these acceptance criteria in the EERS format. EERS stands for the Easy Approach to Requirements Syntax. And this lets you really easily, it's effectively a structured natural language representation of what you want the system to do. Now, for the first four and a half months this product existed, the EERS format looked like kind of an interesting decision we made, but just that sort of interesting.

And with our launch, our general availability launch on Monday, we have finally started to roll out some of the side effects of that, which is property-based testing. So now your EERS requirements can be translated directly into properties of the system, which are effectively invariants that you want to deliver.

For those of you who have or, like, have not, I guess, done property-based testing in the past using something like, I think it's a hypothesis in Python or Fast Check and Node, Closures Spec Library is another example. These are approaches to testing your software system where you're effectively trying to produce a single test case that falsifies the invariant that you want to prove.

And if you can find any contrapositive, then you can say, "This requirement is not met." If you cannot, you have some high degree of confidence where the word high there is doing a little bit of heavy lifting because it depends on how well you write your tests.

But you can say with a high degree of confidence that the system does exactly what you're saying it does. Yeah. So a property, we'll get a little bit more into property-based testing and PBTs a little later, but this is the first step of many we're taking to actually take these structured natural language requirements and then tie this with a through line all the way to the finished code and say, "If your code, if the properties of the code meet the initial requirements, we have a high degree of confidence that you have reliably shipped the software you expected to ship."

So with Spec-Driven Dev, we take your prompt, we turn it into requirements, we pull a design out of that, we define properties of the system, and then we build a task list and we go and you can run your task list.

Effectively, the spec then becomes the natural language representation of your system. It has constraints, it has concerns around functional requirements, non-functional requirements, and it's this set of artifacts that you're delivering. So I don't think I have the slide in this deck, but ultimately the way I look at spec is that it is, one, a set of artifacts that represent sort of the state of your system at a point in time T.

It is, two, a structured workflow that we push you through to reliably deliver high-quality software, and that is the requirements, design, and execution phases. And then, three, it is a set of tools and systems on top of that that help us deliver reproducible results, where one example of that is property-based testing.

Another example of that, which is a little less obvious but we can talk about later, is going to be, I don't even know what to call it, requirements verification. So we scan your requirements for overambiguity. We scan your requirements for

invalid constraints, e.g., you have conflicting requirements, and we help you resolve those ambiguities using sort of classic automated reasoning techniques. And I could talk a little bit more about sort of the features of Kiro. I think that's maybe less interesting for this talk because we want to talk about Spec-Driven Dev.

We have all the stuff you would expect, though. We have steering, which is sort of memory and sort of cursor rules. We have MCP integration. We have, you know, image, yada yada. So we have ways to, and we have software hooks.

So let's talk a little bit about sharpening your toolchain. And I'm going to take a break really quick here. Just pause for a moment for folks in the room who had maybe tried downloading Kiro or something else and just say, "Are there any questionsright now before we dive into how to actually use spec to achieve a goal?"

No questions. It could be a good sign. It could mean I'm not talking about anything that's particularly interesting. So I actually want to, like, talk in some concrete detail here. This is a talk I gave a few months ago on how to use MCPs in Kiro.

And so one of the challenges that people who had tested out Kiro had, that might be a little easier to see, was that

they felt that the flow we were pushing them through was a little bit too structured. Like, you don't have access to external data. You don't have access to all these other things you want. And so one thing that we said on our sort of journey here towards sharpening your, oh, you know what?

It's out of order. Here's my nice AI-generated image. So you can use MCP. Everybody here, I assume, is familiar with MCP at this point, but Kiro integrates MCP the same way all the other tools do. But what I think people don't do enough is use their MCPs when they're building their specs.

And so you can use your MCP servers in any phase of the Spec-Driven Development workflow. That's going to be requirements generation, design, and implementation. And you can use, we'll go through an example of each. So first of all, to set up a spec in Kiro is fairly straightforward.

We have the Kiro panel here, which is a little ghosty. And then you can go down to your MCP servers and click the plus button. You can also just, my favorite way to do it is to ask Kiro to add an MCP and then give it some information on where it is.

And it can go figure it out usually from there, or you just give it the JSON blob and it'll figure it out. Once you have your MCP added, you'll see it in the control panel down here, and you can enable it, disable it, allowlist tools, disable tools, et cetera.

So you can manage context that way. Worth noting, changing MCP and changing tools in general is a cash-breaking operation. So if you're very deep into a long session, maybe don't tweak your MCP config because it will slow you down dramatically.

But let's talk about MCP and spec generation. So something I, the Kiro team uses Asana for reasons I don't know, but it's our task tracker of choice. But so one thing I want to do is maybe go and say, "I don't want to write the requirements for a spec from scratch.

My product team has already done some thinking. We've iterated in Asana to kind of break a project down." This is not always how things work, but sometimes how things work. So in this case, I have a task in Asana.

I don't know. I did the wrong thing. That's what I get for zooming. So I have this task in Asana that says, "Add the view model and controller to this API." In this case, this was a particular demo app that I can share in a few minutes.

And we even had, like, it's kind of peeking under here, but we had some details about what we wanted to have happen. Now I can go into Kiro and just say, "Start executing task XYZ URL from Asana." And Kiro is going to recognize this is an Asana URL.

I had the Asana MCP installed. It goes and pulls down all the metadata there. Da, da, da. So it's going to break out and from there

start determining what to work on.

Oh, it's funny. These titles are backwards. Basically, create a spec for my open Asana tasks. Again, go pull from Asana all the tasks, and then for each one, generate requirements based on those tasks. So I think I had like six tasks assigned to me.

One is do user management, do some sort of

property management, da, da, da. It pulled them in, generated the requirements, and then in this case, title is wrong. Apologies. Start executing task. This is, I want to go and do the code synthesis for this. And I will take a quick break here to talk about how you can do this in practice.

So for those of you who are, you know, following along in room, feel free to fire up your Kiro, open a project, and then pick an MCP server. I'll share a few repos here really quick that you can play around with.

So

I have an MCP server implemented. I have...

this lofty views, which I think implements the Asana. And then these should all be public. Let me just double-check. Yeah. Okay. So for example, if you wanted to extend my, I have a Nobel Prize MCP, which curls, perhaps unsurprisingly, there is a Nobel Prize API.

So you can use UVX to install it, or you can get cloned this Al Harris at Nobel MCP. This is just one example. Another one here is if you want to play around with the sample that's in the video.

I have Al Harris at slash lofty views. I'll leave these both sort of up on the screen for a few moments for folks who do want to copy the URLs.

But while that is happening, oh no. Let's put you on the same window.

So what I'll demo quick is the usage of an MCP to make, like, spec generation much easier or more reliable. So here I have, let's see.

I've got a lot of MCPs. Which ones do I actually want to use?

Let's use the GitHub MCP.

Oh no.

Ignore me.

That's bad. Okay. Well, I have the Fetch MCP. So in this case, I could, for example, come in here and say, "Hey, I've generated a bunch of tasks here. It's lofty views app. This is basically a very simple CRUD web app.

But I want Kiro to

use the Fetch MCP to pull examples from similar products that exist on the internet." You could also use, you know, Brave search or Tavoli search MCP servers, but in this case, I'll just use Fetch because I've got it enabled.

So let's say, oh, actually, we can run the web server and use Fetch. That's a good example.

This is one example of you can at any point in the workflow generating a spec go through and, you know, use your MCP servers to get things working. Oh no. This is what I get for not using a project in a while.

No, we'll cancel that. We can actually do something a little more interesting, which is a separate project I've been working on. So I've been working on an agent core agent, and that might be, I know the project works, which is the reason I'll fire it up here.

What should I call it?

Well, maybe we'll do live demos at the end. So that's sort of like the most basic thing you can do with Kiro is just use MCP servers, but any tool uses MCP servers. I actually don't think that's particularly interesting.

So let's say in sort of this process of trying to sharpen our Spec-Driven Dev toolkit, we've finished up with the 200 grit. We've added some capabilities with MCP. It's useful, but it's not going to be a game changer for us.

I want to come in here and actually get it up to the 400 grit. Let's start to get a really good polish on this thing. I want to customize the artifacts produced because you've got this task list. You've got this requirements list, and I don't agree with what you put in there, Al.

You could say that. A lot of people do. And that's a great starting point. So here's something I heard earlier in the week at, you know, earlier in the conference is that people like to do things like use wireframes in their mocks.

Use wireframe mocks because in your specs are natural language. Using specs as a control surface to explain what you want the system to do. Therefore, I want to be able to actually put UI mocks in here. So the trivial case is that I just come in here and say, "Kiro's asked me here, does the design look good?

Are you happy?" And I said, "This looks great, but could you include wireframe diagrams and ASCII for the screens we're going to build here?" I'm adding, this is again from that lofty views thing. I'm adding a user management UI, but I want to actually see what we're sort of proposing building, not just the architecture of the thing.

So Kiro's going to sit here and churn for a few seconds, but you can add whatever you want to any of these artifacts because they're natural language. So they're structured, which means we want

some sort of reproducibility in what they look like, but ultimately what they look like doesn't matter because we've got the any machine here, the agent sitting that can help translate it to what it needs to be. So Kiro's churning away here.

It's thinking, thinking, and then it's going to spit out these text wrapped ASCII diagrams. I'll fix the wrapping here in a second in the video. But ultimately, like,

you know, it does whatever you want. So if you want additional data in your requirements, you can do that. If you want additional data in the design, like this, you can easily add that. Here we've got sort of these wireframes and ASCII that help me sort of rationalize what we're actually about to ship.

And then I can, again, continue to chat and say, "Actually, in the design, I don't want, you know, maybe I don't want this add user button to be up at the top the entire time," in which case I could chat with it to make that change easily.

And now we're on the same page up front instead of later during implementation time. So we've again sort of left shifted some of the concerns. So that's one example. You know, I want to add UI mocks to the design of a system.

Another example, though, could be this, oh, this is a quick snapshot of the end state there where now my design does have these UI mocks. But another example that I actually like a little bit more is this including test cases in the definition and tasks.

So today the tasks that Kiro will give you will be kind of the bullet points of the requirements and the acceptance criteria you need to hit. But I want to know that at the end state of this task being executed, we have a really crisp understanding that it is correct.

It's not just like done. Because anybody who's used an agent can probably testify that the LLMs are very good at saying, "I'm done. I'm happy. I'm sure you're happy. I'm just going to be complete." Oh, yeah, the tests don't pass, but they're annoying.

I tried three times to get them to work. I'm just going to move on. No, I don't want that. I want to actually know that things are working. So in this case, I've asked Kiro to include explicit unit test cases that are going to be covered.

So my task here, for example, in creating this agent core memory checkpointer is going to have all of the test cases that need to pass before it's complete. And then I can use things like agent hooks to ensure those are correct.

We'll run this sample a little later in the talk. This is the thing I'm ready to live demo. Yeah. So this is another example where you can, again, you're working on your tool bench. You're sort of, you have all these capabilities and primitives that you control, and you can tweak the process to work for you, not just the process that I think is the best one.

And then sort of last but not least, the 800 grit. At this point, we're getting a final polish on the tool. We might be stropping next, but we want to, you know, you can iterate on your artifacts, but you can also iterate on the actual process that runs.

So one thing you might have, and I do this a lot, is I'll be chatting with Kiro and I say, "Hey, I want to, in this case, I want to add memory to my agent in agent core. Let's dump conversations to an S3 file at the end of every execution."

Kiro's going to say, "That's great. I know how to do that. I'm going to research exactly how to do that thing. I will achieve this goal for you." But ultimately, what I've done is actually introduce a bias up front, which is I'm steering the whole agent using S3 as the storage solution just because maybe I'm familiar with it, but it's probably not the best way to go about it.

So then after it had synthesized the design and all the tasks and all this stuff, I came back and said, "Well, like, we don't need to stick to this rigid Spec-Driven Dev workflow that has been defined by Kiro.

I can ask for alternatives. Like, is this the idiomatic way to achieve session persistence? I don't know. Maybe there's a better way. Maybe if we're talking AWS services, it's not S3, it's Dynamo or yada yada." Kiro's going to come in here and say, you know, "Good question.

Da, da, da. Let me research." It's going to go through, call a bunch of MCP tools that I've given it access to. This kind of ties back to that. You should be using MCP. And then it comes back with this recommendation that I didn't know was a feature, which is agent core memory.

It says it's more idiomatic and future proof. That maybe is TBD and should be checked a little closer. But, or you could use S3, which is the thing you recommend. Now, actually, I bet there's far more than two options here.

So you could probably keep asking the agent, "Are there other options?" yada yada, and it would go and continue to investigate. But you should not lock yourself into the rigid flow that is sort of the starting point here.

Yeah. So that's actually, I think, it for my deck. What I will talk about

is let's just run through that sample I just had up there, which is that. So basically, let me delete it and I'll just do a live demo of sort of specs in Kiro and how we can fine-tune things a little bit.

So this project is a Node.js app. It

is a CDK. Again, I'm not trying to sell more AWS. This is just the technologies I'm familiar with, so I can move a lot more quickly. So I wanted to know a little bit about Agent Core, which is a new AWS offering, and as somebody building an agent, I should probably be familiar with it.

So, and I'm not familiar enough with it. So we've got some of the people here who know a lot about it. So I'm going to put my hand up a little bit in, you know, you caught me. So I set up a CDK stack, which is just, you know, IaC technology to deploy software.

I'm familiar with it and I love it. So I have a stack here that lets me deploy whatever an Agent Core runtime is. I don't know. I ask Kiro to do it. We vibe-coded this part. So we vibe-coded the general structure.

We got an agent. We got IaC set up. I then vibe-code added commit lint. I added Husky, a few things like this that I like for my own TypeScript projects. Prettier and ESLint, I think. So we have a basic product here, or like a basic project here that I know I can deploy to my personal AWS account.

Now, I'm going to come in here and, oh, and then importantly, this is super important because I don't know how the hell Agent Core works. And I could go read the docs, but the docs are long and they're complicated.

And I'm really just trying to build out a POC to like learn about it myself. So I added two MCP servers. Oh no, maybe I didn't. Let me check. Oh, okay. Yes. Sorry. Buried down here at the bottom.

So this is my Kiro MCP config. I added one important MCP server here, which is the AWS documentation one. There's other ways to get documentation. You can use things like TESL level seven, but in this case, this is vended by AWS, so I have some confidence that it might be correct.

So I used this to help the agent have knowledge about sort of what technologies exist. And I think I used Fetch quite a bit as well. So these are the two sets of, these are the two sets of MCP servers I provided the system.

Let's create and move on. So, and I'll just rerun this from scratch. So what I had done yesterday evening, or maybe the evening before, was I sat down and I have this system sort of basically working, and now I want to start doing Spec-Driven Development.

So I want to add this session ID concept, and then I want to rewrite conversation to an S3 file, blah, blah, blah. This is the whole sort of bias thing I showed you earlier. We're going to fire that off through Kiro.

It's going to start running, chugging away, and then it's going to, you know, see if the spec exists. Okay. The folder does exist. It's probably going to realize there's no files there and start working away. But from here, I'll sort of live demo.

It's going to read through requirements. It's going to read through existing docs. It's going to read through existing files, gather the context it needs, churn away.

But in a moment, once it generates sort of the initial requirements and design, I am going to challenge it to use its own, you know, MCP servers. I want you to go and do some research on the best way to do this and provide me some proposals.

And this is why I was hoping to get the clip on mic working, because I've got to set this down for a moment.

Okay. So, you know, I don't know if this is the best way to do this. Go read docs, go use Fetch, da, da, da. It's going to keep kind of churning away here and then come back to me after it's probably got a few ideas and proposed it.

But this is an example of me just using additional capabilities. Use Fetch, use the docs MCP, use whatever you can to get the best information, and don't take at face value the things that I said. These are usually things we have to prompt pretty hard to get the agent to do, but if you're doing it in real time, it works fairly well.

Again, the agent, all of these agents are going to be very easy to please. So, you know, just because I said something in the studio docs, it may or may not actually be the most important thing from the agent's perspective down the road.

So, you know, okay. So it's done a little bit of research. It understands the LangGraph, which is the agent framework we're using, already has this knowledge of persistence. Da, da, da. And actually, in this case, it didn't find, it did not use the MCP for Agent Core docs.

It didn't find that Agent Core has this knowledge of persistence. So maybe you like, let's assume I don't, I still don't know that exist because I didn't dry run this a few days ago. We might have to find that later in the design phase.

So the first thing it's going to do is kind of iterate over all my requirements here. You know, change the requirements based on what it now knows about LangGraph and how it can natively integrate with the checkpointing. But it's still really crisply bound to this like S3 decision that I made implicitly in the ask.

So that is just something to be aware of. Anything you put in the prompt is effectively rounding for the agent, for better or for worse. I see it's still iterating. So yeah, comes through, says, "Does this look good?"

We change, da, da, da. I'm going to say, "Looks great. Let's go to the design phase." So now Kiro is going to take my requirements and take me into the design phase of this project. I can make this.

So things are a little bit bigger. But

here's an example of what I meant by these ears requirements. So the user story here is, as a dev, I want to implement a custom S3-based checkpointer so the agent can use LangGraph's native persistence mechanism with S3. Great.

That sounds reasonable to me as a person, you know, sort of co-authoring these requirements. This here, this sort of when-then-shall syntax, this is the ears format. And the structured natural language is really important for us to pass this through non-LLM-based models and give you more deterministic results when we parse out your requirements.

Because ultimately, our goal is to actually use the LLM for as little, not as little as possible, but less and less over time. We want to use classic automated reasoning techniques to give you high-quality results, not just, you know, whatever the latest model is going to tell you.

So Kiro's gone through, spits out a design doc. Let's actually just look at this in Markdown. This sure, you got a server, da, da, da, checkpoint or goes to S3. That makes sense. Pseudocode. Again, in a real scenario, maybe I read this a little bit more closely.

And what's actually, this is the new thing we shipped in on the 17th, is that now Kiro is going to go through and do this formalizing requirements for correctness properties. And soright now, what the system is doing is it's taking a look at those requirements you generated, the requirements we agreed upon with the system earlier.

These look good. I agree with them, yada yada. It's taking a look at the design, and it's extracting correctness properties about the system that we want to run property-based testing for down the road. This is something that may or may not matter for you in the prototyping phase, but should matter for you significantly when you're going to production.

Because if these properties are correct and these properties are all met, the system aligns one-to-one with the input requirements you provided. Yeah. So while this is chugging away, any questions yet? Any folks kind of curious about this?

Yeah.

Oh, we're hearing them there.

What would you say is the main difference between the planning mode and the user has?

I haven't used the planning mode in a couple of weeks. So it's, I'm, things move so fast, it's a little wild. But I think ultimately what we would say is that Kiro's Spec-Driven Dev is not just LLM-driven, but it is actually driven by like a structured system.

And so planning mode, I'm not sure if there's actually like a workflow behind it that takes you through things. But yeah, this is our take on it for sure. I'm not familiar enough to give like a more concrete example, unfortunately.

Similar. I mean, it doesn't give you like this document. I think that this document is cool. It's creating you the document is cool. But what Cursor does is to basically create you a plan.

Just an execution plan. Okay. Oh, I see. So I think that the fundamental difference there, does that plan get committed anywhere, or is it just ephemeral?

It's committed.

Okay. So what I want over time is not just how we make the changes we care about, but it is actually the documentation and specification about what the system does. So the long-term goal I have is that as Kiro, we are able to do sort of a bidirectional sync.

That is, as you continue to work with Kiro, you're not just accruing these sort of task lists. And so I'm just going to say, "Go for it," to go to the tasks. But we're not just accruing task lists, but actually if I come back and let's say change the requirements down the road, we will mutate a previous spec.

So I'm looking at really just a diff of requirements, which as you go through the greenfield process, you're going to produce a lot of green in your PRs, which is maybe not the best because I'm just reviewing three new huge Markdown files.

But on the next time or the subsequent times that I go and open that doc up, I want to be seeing, "Oh, you've actually, you know, you've relaxed this previous requirement. You've added a requirement. And that actually has this implication on the design doc."

That is the process the Kiro team internally uses to talk about changes to the Kiro system. So we review, our design docs have in general been replaced by spec reviews. So we will, you know, somebody will take a spec from Markdown, they'll blast it into our wiki basically using an MCP tool we use internally.

And then we'll review that thing and comment on it in sort of a design session as opposed to, you know, this Markdown file or a wiki from scratch. So it becomes sort of, if, well, it's actually not like an ADR because it's not point in time.

It is like this living documentation about the system. But yeah, yeah. Thanks for the question. There's one over here.

This may be more a Spec-Driven Development question, but are there like, like is there like a template for a set of files that you fill out? Likeright now you're in the design.md. Are there like, is this, is the design.md the spec and it's a single doc or are there?

Oh, great question. So the, yeah, the question was, are there, and correct me if I'm wrong here, but the question is, are there a set of templates that are used for the system? And is the questionnaire driving at, can you change the templates or is just, are there?

Okay. So the, yeah, question is, are there a set of templates? There are implicitly in our system prompts for how we take care of your specs. So you'll see here at the top nav bar here,right now we're really rigid about this requirement design task list phase, but we know that doesn't work for everybody.

For example, if you're starting, we get this feedback from a lot of internal Amazonians actually that I want to start with a, I have an idea for a technical design and I don't necessarily know what the requirements are yet, but I know I want to make, maybe design is even the wrong word.

I want to start with a technical note. Like I want to refactor, this comes up a lot for refactoring actually. So I want to refactor this to no longer have a dependency on, here's a good example. Here we use a ton of mutexes around the system to make sure that we're locking appropriately when the agent is taking certain actions because we don't want different agents to step on each other's toes.

But maybe I want to challenge the requirements of the system so I can remove one of these mutexes or semaphores, I should say. So I might start with something like a technical note and then from there sort of extract the requirements that I want to share with the team and say, you know, I had to kind of play with it for a little while to understand what I wanted to build, but I still want to generate all these rich artifacts.

So today it's this structured workflow. We're playing a lot around with making that a little bit more flexible. But the structure is important because the structure lets us build reproducible tooling that is not just an LLM. So I think that that's an important distinction we make is that our agent is not just an LLM with a workflow on top of it.

The backend may or may not be an LLM or it may or may not be other neurosymbolic reasoning tools under the hood. And so we try to keep that distinction a little bit clear that you're not just talking to like Sonnet or Gemini or whatever, you're talking to sort of an amalgam of systems based on what type of task you're executing at any point in time.

Although when you're chatting, you are talking to just an LLM. But yeah, so we have a template for the requirements. We have a template for this design doc because there's sections that we think are important to cover. And again, like if you disagree and you're like, "I don't care about the testing strategy section," just ask the agent to do it.

And similarly, the task list is structured because we have sort of UI elements that are built on top of it as well. It's like task management and do we have, we'll get there when we do some property-based testing, but there's some additional UI we'll add for things like optional, you can have optional tasks and stuff like that.

And so we need the structure there for our task list LSP to work, for example. Yeah, thank you for the question. Anything else before we truck on?

Cool. I may need somebody to remind me what we were doing. Oh, that'sright. So we went through and we synthesized the spec for adding memory and some amount of persistence to my agent. By the way, I didn't introduce you to this project.

This project is called Gramps. It is an agent that I'm deploying to Agent Core to learn about it. I mentioned that, but what I didn't tell you is that it is a dad joke generator, a very expensive one since we're powering it via LLMs.

But effectively, your dad joke generator, joke should be clean. It should be based on puns, you know, obviously bonus points if they're slightly corny, but endearing, yada yada. So we're deploying this to the backend. So the reason I want memory is because every time I ask the dad joke generator for a joke, it gives me the same damn joke.

And that's just super boring. And my kids are not going to be excited about that. So I want memory so that as I come back for the same session, I get different jokes over and over again. That's the context on the project.

So we've come through here and we actually said, we generated this thing, we did the task list. I said, "Hey, is this the idiomatic way to do it?" But what I know is that we didn't actually, we're not using Agent Core's memory feature, which is probably a big oops.

And so, you know, quick show of hands. Do we want to make the mistake and go all the way to synthesis and deployment, or should we fix it now? We want to fix it now because we know better.

No, I want to make the mistake. Let's keep on trucking. I have three yeses and a room full of nothing. So we're going to make the mistake and then come back and fix it later. So let's say run

all tasks

in order.

The reason I mentioned in order, which seems very specific, is because this is a preview build of Kiro. And so somebody just added to the system prompt, I should only do one task at a time. And I found that if I say run all tasks, it thinks I somehow mean do them all in parallel.

So that'll be fixed before these changes get out to production. So Kiro's going to keep kind of going through here and chewing away on the system in the backend. It has steering docs that explain how to do its job.

It has, which I guess I should show you guys. Steering again is like memory. So I have some steering on how to do commits, you know, how I like to have commits, but also steering on things like, how do you actually deploy this thing?

How do you deal with Agent Core? And then how do you run the commands that are necessary for you to deploy this to my local dev account? And those are mostly just an example again of sharpening your tools.

Like I went through this kind of painful process of figuring out, you know, you have to use this parameter on the CDK, the CDK command, you have to use this flag, otherwise it doesn't work correctly. And so once I go through that pain of learning, I just say, "Kiro, write what you learned into a steering doc," and it will usually do a very good job of summarizing.

And so it generated automatically this Agent Core Lang Graph workflow MD file. Yeah. So I mean, it's just going to kind of go away here and truck on and do its job. And we can watch it in the background.

But in the interim, I think at this point we're at a pretty flexible spot. So for folks who want, feel free to use Kiro, try out Spec-Driven Dev on your own. I'm going to keep just kind of running this in the background and taking questions and comments, but that's kind of it for the scheduled part of today.

Yep.

How does Kiro work for like existing large code bases or this?

Yeah. Yeah. Question was, how does Kiro work for large and existing code bases? Basically the brownfield use case. And the answer is, it depends on what you're trying to do. For Spec-Driven Dev, you can ask Kiro to do research into what already exists.

So when you start a new spec, it will usually start by reading through the working tree. But the agent is generally starting from a scratch perspective,right? It needs to understand the system. In practice, what that means is that you're going to end up with a bunch of things like if your system already had good separation of concerns, the components in your system are highly cohesive and they're sort of highly coherent and highly cohesive, it's going to have a great job,right?

It's going to be able to say, this is the module that does this thing. I don't need to keep 18 things in my context to do my job. And it's going to do well. If you, let's just take an example that's off the top of my head.

If you were trying to launch an IDE very quickly leading up to an AWS launch and you, you know, took a lot of tech debt along the way that you need to unwind. And, you know, nobody here would do that, I'm sure.

But in case you did that like me, then your agent might actually have a much harder time traversing the code base in the same way that a dev would,right? So from just kind of that perspective, the more reliable things like your test suite are and the more understandable things like module separation and sort of decomposition of concerns are, the better the agent will do.

And versus true, of course. Now, for things like understanding the code base, this is a bad example because this is a very small code base, but we do have things like, you know, code search and workspace.

I don't know what to call these, context providers. So you can come in here and just say, I want to do code. Oh, what is it?

I might have turned this off actually. Oh, I did turn it off because the code base isn't big enough. We'll do things like indexing in the background so that the agent, like you can do semantic search over what you've got if you're just chatting.

But in general, Kiro should go in and do sort of background search to figure out how to do its job. Like as the code base scales up, it's going to be less, do probably less well overall, but that's one thing we're working on as a team.

Did that answer your question or did I kind of glance off the side a bit?

Yeah, there. Got it.

Okay, cool. Anybody else?

I went to

people size.

How long are you willing to wait for indexing to complete? So one example I have is that the code OSS, if it's not supremely obvious by looking at it, Kiro is a code OSS fork, just like, you know, cursor windsurf.

One of the challenges we've had is that the code OSS code base is very large, fairly large. There's other big ones out there, but that's kind of my large code base because I'm not forced to get to work in it fairly frequently.

And so there's definitely some perceived slowdown when you're dealing with something large like that, especially when you talk about code base indexing. It's a very active area of work for us though. So we're trying to do things like either remove indexing from the critical path so that you're not waiting there on some kind of slowed down render thread because indexing is running.

But in practice, there should not be. I mean, again, the agent may practically do less well, but we are going to be talking in a couple of weeks at Reinvent about how some of the temple features in Kiro were built via spec in a code base we did not understand particularly well because we're just not VS Code devs.

And Kiro did a fine job of it. But again, that's a testament to the fact that code base is reasonably well structured. And like if you've taken the time to understand how it works, it's very understandable. If you have not, it might be a little bit opaque to stare at.

Yep.

In terms of indexing, is it like just putting

as much information from the code base into context or is just, is there a way to like create some kind of like vector database of all the code base and then like query it?

Yes. So the question was, what do you mean by indexing? Because indexing can mean a bunch of different things. And what I mean is that the agent is actually not provided the. We want to keep the agent context as small as possible.

We use the index for most like secondary effects. Things like if you're doing a code search or if I do something like search for pound, what's the file in here? Server, like we use it more for these types of UI than giving it to the agent because the agent does, this is sort of anecdotal and based on our benchmarks, does better when given less context, but given the tools to understand where to go find things.

Something we've heard a lot about is sort of incremental disclosure here at this conference. And that's, again, we don't want to load too much at the beginning of the context and conversation with the agent. We want the agent to self-discover theright context for the task.

Yeah. Thank you. Yep.

You guys managing session length? Like is there any kind of compression or pruning?

Yeah. So the question was, how do we manage session length? We have no incremental pruning today or incremental summary. You basically just accrete context until you hit your limit, which I thinkright now I'm on auto, which has like a 200K token limit, similar to the Sonnets.

So we don't have a very sophisticated algorithm here yet. We've looked at a few things, but our number one concern actually is prompt caching hit rate. And so in a normal use case, I can achieve something like 90, 95% cache token usage here on per turn, which means that my interactions are very fast.

And that's, or they're much faster than the alternative, which is I'm sending 160K tokens to Bedrock Cold. So that's one of the reasons we've actually not done much experimentation with incremental summary. Our summarization feature exists when you hit the cap.

It's not great. It's something we're trying to ship an improved version very, very shortly, e.g., in the next couple of weeks, which should be faster. Today it's like a one-off operation that can take up to 30 or 45 seconds, which is a horrendous experience.

We're hoping to fix that here and make it sort of a real-time experience.

There's a follow-on to that.

Yep.

Managing stapleness between sessions, is that how you're relying on this area of Spec-Driven too long?

So sort of.

That is not the only reason. I mean, the Spec-Driven gives us less performance and more to do with the agent.

Because if we can give you theright result, the way I, and I think that we talk about it internally as this team is if I spend 10 seconds giving a prompt to the agent and then it goes off and it gets it wrong, it's like, that's kind of no skin off my back,right?

I burned however many tokens and, you know, a couple cents of credit usage with whoever my LM provider is, but I spent 10 seconds generating a prompt. If I spend five to 10 minutes with the system producing a detailed design doc, or let's just say even a detailed set of requirements, I want it to do a fairly good job.

If I spend an hour generating a design doc, reviewing it with my team, and then synthesizing from that, I want it to get itright. So the goal necessarily is not just latency, but actually accuracy when we talk about that.

Now, it's a both-and. You need to do both, but Spec comes more from the goal to have highly reproducible output.

I'm going to go over here first and then you.

Yeah. How do each of these task agents pass context to each other? And then are you only supposed to run this parent task because it just finished all like 3.1, 3.2, 3.3, but then it still thought that 3.1 wasn't done and ran that in 3.2.

Oh, did it?

Yeah. Well, no, mine.

Oh, okay. Yeah, yeah. So if you,

the question is if you're in the UI and you're like running tasks and I can just kind of pull up my task list here. So if I just hit start, start, start, each of these is going to be a new session, which means the context is completely unique.

Personally, I like to just, if I can, if I've got the context base to afford it, I just say, do all the tasks because I find that more understandable and I think I actually get better performance. But by default, each task will be a new session that has no shared context with the previous ones.

So the session is effectively just seeded with your specification. And then like here, you're working on a spec that does all this stuff, block of text, and you are doing this task, da da da. Don't do any other tasks, just do this.

So that sounds like a bug.

Is there ever been a sub-agent for certain things?

We don't have sub-agents yet in Kiro. Some we're working on. Yeah. Yeah, because I mean, ideally,right, if we click on task three and I've got three, one, three, two, three, three, and they're separated, there's no good reason I couldn't have different systems working on them.

Yeah.

Right here.

We do have in the Kiro CLI custom agents that you can also run off.

Yeah. Kiro CLI is a concept of custom agents, which can be run sort of as a task. And it's something we're playing withright now in Kiro Desktop. Min, I think you had another one.

Yeah. I'm sorry if I missed this, but in the spec folder, as you do more and more of these tasks over time, is it just all in one design requirements tasks? Your whole project is defined there or is it grouped by?

Oh, that's a good question. Yeah. So I will have many, I will have, the question was, as you do more, you generate, let's say, more specs over time, are you sort of just creating one massive spec? And no, let me open a different project.

So this is, for example, the Kiro extension, which is like a 1B extension inside the Kiro IDE. This is where the agent itself lives. And so we have pruned some specs, but there are specs in here that we can talk through or I can just kind of demo.

So these are, the way I think about it is the spec sort of represents a feature or a problem area in the project. And so, for example, I can blast this a little larger. So, for example, we have like some of these are just tests.

We've done things like, oh, could we have a prompt registry? Could we have a prompt registry file loader? They may or may not make it all the way to production. I want telemetry on the chat UI. So these are just like somebody will go off and spend, maybe represents a few days of work for an SDE.

Agent SMD support is a good one where we just, you know, I sort of said, research what Agent SMD is and build it in the way you build steering in, like support in the same way. The spec is fairly unlikely for us to come back and revisit in the future.

So I may actually just delete it, which is what we've done with some of the older ones. But a good example of one that we might come back to is our message history sanitizer. So one thing we've had issues with, or we had issues with early in the development of Kiro is that we would send these sort of invalid sequences of messages because let's say the Anthropic API required tools to be in the same order they were invoked in the responses, but the system wasn't doing that.

So we built this whole sanitizer system that has a bunch of requirements around, let's see, very specifically, yeah, when conversation is validated, the system shall verify that each user input is either non-empty content or tool responses. So we had things where like empty strings would get passed in, but there was a tool response.

This is a good example where you've come in over time and actually just added, maybe not to the requirements, but to the acceptance criteria of the requirements as new validation rules are uncovered. Yeah.

So how do you handle like that? So, for example, you have like telemetry up there.

Yep.

You got a feature that needs telemetry. Is it going to go back and update that spec too or you're just?

It should. Yeah. So if you, usually you'll see, and let me just ask

a new chat here.

No, that's a terrible idea.

So here I've asked, I've made an inspect mode. I've made some requests to add UI telemetry to the thing. I'll help you add it. Let me first check if there's any relevant runbooks. Then explore the code base and send the implementation.

It might go do a little bit of research here. And then flip of a coin. Again, it's an LM, so it may or may not discover the existing spec, but ideally it will, after doing its research, say there exists a spec already for things like UI telemetry.

I'm going to go and amend that one. And if it doesn't, in this case, like I would come in and just ask it to as sort of the operator of the system. But over time, again, we want that to be easier for you as a user to not have to think about so much.

We can watch it while it chugs along.

Is there anything pre-configured in Kiro that makes it better to work with AWS?

No, not really.

Was that a question?

Oh, question was, is there anything in Kiro that is pre-configured to make it work better with AWS? No. We are sort of purposefully. We're in, we are brought to you by AWS, which, so, you know, Andy Jassy and Jeffy B pay my check, but we're not like an AWS product that's deeply, deeply integrated with the rest of the AWS ecosystem.

Now, that said, I still answer emails when somebody says, why is this other thing we built with AWS not working with Kiro? Yay. But similarly, like if you're building on GCP or Azure or whatever, or you're running some on-prem system, the product should work just as well for you.

That's our goal.

A good answer potentially is the AWS documentation MCP service.

Yes.

So there are MCP services that you can add into any of these things that will make them better at work with AWS now.

Yeah, that's a good point. So like in this case, I actually had to add the AWS MCP documentation here. We could, of course, have natively bundled this, but I don't want to ship this to customers who don't need it.

Yeah. Because again, AWS is not the only docs that we might care about. By the way, coming back to your question, so it did find the existing spec for telemetry. It read it, it read different sections of it, and now it's actually making amendments to it so we can follow the diff as it shows up here.

So it's added new requirements to the pre-existing spec. So this is effectively another case where we're mutating the system as opposed to just adding this sort of never-ending spiel of specs.

I guess what I'm wondering is like, how does it know or decide where to put the spec? You know, if you break down your project into these different categories, I would imagine like crossover.

Yeah. I mean, that's sort of like software development in a nutshell though,right? Like how do you actually define the seams between different parts of your system, different concerns, the product?

Right. If you want to like build something, like I have a task and it's going to cross, it'll require changing like three or four things.

Yep.

You're going to change three or four specs and then run tasks across three or four specs.

Oh, yeah, yeah. No, it should not do that. It would probably, so again, I don't have a good example offhand that we can do for that, but my perspective would be that if you're working on something that is cross-functional, by the way, the question was, if I'm working on something that, let's say I have a spec for security requirements and I have a spec for API design, like the API shapes, and I have a spec for logging, and I am changing something in the API public interface that is a security-facing concern because we're redacting logging PII.

I think that's maybe a semi-tangible use case that we can all imagine coming down from our governance teams. I want to, I would imagine that you either pick one of those to load the requirements into, or you create sort of a cross-functional spec, but that would come down to, I think, you as an operator making that decision.

In much the same way that if I, how you actually implement it might be, you would not necessarily implement my PII, API redaction module as a standalone thing. It's going to be a cross-cutting theme across your code base, I'd imagine.

This is also a good example that's like multi-group webspace came out when it went GA on Monday, and now you can like drive different, so like in your example you just went through with like APIs and auth and like even the front-end thing, you can bring in those projects if you have them separately and then still work.

Yeah. Thanks, Brooke.

Just to mention more, the spec generates the code after that? Like what code you can specify? How does that work?

Yeah. So we have now synthesized effectively the spec. So we sat down, we defined the requirements, design, and task list. I've had Kiro now go through and run all the tasks in this spec. So it ran them one at a time.

It basically worked on small bite-sized pieces of work, chunk by chunk. And then now this is done. So what we've actually produced is not just like the completed spec, but it went here into my agent and it did a few things in the CDK repo because it's doing persistence to S3.

I'm sure it added a bucket. Yep. Some new bucket with encryption and yada yada. It then went into the agent, added the S3 checkpoint saver. It looks like it, you know, created a checkpointer. It adds this to the graph and it kind of passes this all the way through the system.

And the S3 checkpointer here, I'm sure, has some knowledge of how to write the checkpoints to and from S3. So like we have gone not just for defining the system, but we've now produced it end-to-end or we've delivered it end-to-end, including property tests, I believe.

Yeah.

Oh, I have an answer to an earlier question related to like some specific AWS-related features like that makes it easier to work with. The Kiro CLI comes with a used AWS tool, which helps with the CLI.

Yep. Yep. So what Rob's pointing out is the Kiro CLI, which we just rebranded this week, has a used AWS tool, which is basically a wrapper over the AWS SDK to make some of those things easy. But again, BYO, use GCP tool as an MCP server if you were so inclined, if that's your tool of choice.

And I believe, don't quote me on this, because the CLI is kind of new to me, I should say. But I believe you can turn off tools in the CLI as well. Let me know if that's notright, Rob.

Yeah, you can turn off.

Yeah. So that's like you're actually not strict in the desktop product today, you can't control the tools, the native tools built in, but in CLI you can.

So I intuitively get the benefits of having a spec. Have you done any work to empirically see like how a project or

a problem would work with or without a spec?

Yeah. We do have benchmarks covering the data offhand. I think part of that's in our blogs. So if you go to the kiro.dev/blog or it's on the site, we talk really crisply about some of the lift that things like property-based testing give to task accuracy.

Science team's always working on that stuff.

I've heard a blog about specs and Kiro discussed distinguished engineer

for databases. Yep. His spec blog post really steps it up. And I think it has the DOT specifically that you were asking for, but I think it will be useful.

Yep. Yeah.

How does it work? I understand the feature side of it, but how does it work in a non-functional side like the agency dealing with a little bit more harder problems?

Well, yeah. I mean, that is ultimately the goal here,right? Is we're saying you're making a slightly larger investment upfront, but we believe that the structure we're bringing is going to help you increase the accuracy of your result. So while we've got a team of people who are basically working on making spec better, my job when I fly back to Seattle is to make Kiro as a whole much faster.

One, execution time, like the end of like lagginess in the UI, but two, how do we get tokens through the system faster? How do we get responses to you faster so that like you're not sinking as much cost into Kiro to use a spec?

Yeah, yeah. I'm not talking about the Kiro tool itself.

Okay.

Code generated from the spec.

Oh, oh, yeah. Okay. Yeah. You mean like the non-functional requirements of the generated code? So that's going to come down to, I think, what you're specifically trying to do. So you could add one of the slides I had here was talking a little bit about how to tweak the process and tweak the artifacts for your use cases.

Again, you could very easily add something like, I want non-functional requirements for speed and runtime and things like lock contention to be considered in the design phase. Yeah, that's something you could certainly add.

So you could generate a code in Rust or Java or.

Yeah, totally. Yeah.

And it will vary in the functional depending on what language it's generated?

I mean, it would have to. Like, yeah, there's no other way, I think, to approach it. Again, I'm just, I'm familiar with Node, so I'm doing everything here in Node, but you can use this with any language. I think technically we say we support Java, Python, JavaScript, and, Jesus, JavaScript, TypeScript, Java, and Rust.

But in practice, there's no reason that this doesn't work with any language. I mean, it's just an LLM. There's nothing language-specific or framework-specific in the system. And for those of you, so there was a conference earlier this week hosted by Tessel, which are doing sort of specs for knowledge base.

As long as you've got theright grounding docs in there, and this is sort of

their argument, is that it should not matter what you're building. Like that's all just informed by the context you're building for your system.

This is also a really good point for steering. So steering, you can get the agent to develop code in the way you want. Like being a developer is all about making trade-offs. And the problem with Kiro out of the box is it's like so polite because it's trying to be everything to everyone.

And especially like with latency and cost and other things like that, just tell it in steering what you want it to prioritize, and then that will influence any code that gets generated.

Yep.

Even like how it designs based on that as well. So if there's something that's very specific to your use case or the industry or whatever, it just shows in a steering file and then people work in.

Yeah, that's exactlyright. So for example, I will have Kiro generate

commits for me. And one of the things I personally care about is that I can track commits I generate versus commits that Kiro generates, being the ones that come from the system. And so my steering doc, while short, includes things like very specifically my requirement for Kiro is, just use the UI,

da da da, attributed to the co-author of Kiro agent, which is trivial, but also I want it to happen every time. So in this case, it just generated a commit co-authored by Kiro agent, da da da. So that's an example of like you could add whatever you want in there, not just something related to Git commits, but you could do code style, you could do, you know, code style, code coverage.

Whenever you add a spec or you're adding a new module, make sure that you annotate it with coverage minimums that are 90% because that's the thing I care about. You can kind of put anything you want up in there.

The good news is it looks like what we built works. Kiro's very happy with itself at least, and it looks like all tests passed. But yeah, so we can deploy this to the backend and see how things work.

We're technically just about at time. So, you know, if anybody has any other questions, I'm going to stick around here for a while, but thank you all for joining, listening, and learning a little bit more about Spec-Driven Dev.