Agents Don't Do Standups: Building the Post-Engineer Engineering Org — Mike Spitz, PFF

Alright, so I'm in the slot before the snacks, but I'm here to speak about the post-engineering org and a case study we've been doing at PFF. I started in January and we finished it in March. PFF is a sports data company; we help NFL and NCAA teams figure out what they should be doing, and we also have a consumer arm which does fantasy football sport betting.

And for those of you who know American football, there's a draft happening end of the month, and so it allows people to basically play as one of these teams. We're a fully distributed engineering team, so we've got engineers in India, we've got engineers in Spain, and all the states in America.

But first, let's get into some kind of stats. So we have 100 million page views annually. We have 9 million drafts happen on an annual basis, so it's a fairly popular tool. And the issue that we had was we were 200 employees, around 20 engineers, and we were falling behind competitors.

We were focusing on sport betting stuff, but a lot of our consumers were actually interested in this. And then with Claude Opus coming out, I started experimenting around November on a personal level, and then it got spun out to two engineers.

One of them was our strongest front-end engineer, and the other one was one of our strongest full-stack engineer. So really, the question I was asking was, instead of figuring out how we can help engineers go and output more, how do we help make the agents quicker?

Right? So I think if we all think historically software engineering, you've got the Agile manifesto, Software Craftsmanship, you've got a lot of good perks and benefits for engineers, from foosball tables to sleeping pods to pretty much every perk that most other industries really don't have, and it's because we're the bottleneck.

And if companies are able to go and optimize for that, it gives them a lot of benefit. But now things are changing. So I just want to go straight into how the case study ended up. So we had 25 times more deploys, so the two engineers were deploying five times every day, and the other team was a team of around 10 engineers.

They were doing it pretty much one deploy every five days. There is a big obvious caveat here,right? Small engineering teams are always going to be quicker than the big ones. So there is an element of that multiplier that is just from it being a smaller engineering team.

However, that smaller Tiger team still had to coordinate all of those daily deploys, still had to go and coordinate all of those deploys with the bigger team, so there was still that kind of issue that was happening. And this one, this is always hard.

How do you validate like the output is actually helpful, the number of PRs isn't helpful, the amount of code isn't helpful? So we basically blended the number of tickets with the code complexity, and we found that their output was at 10x.

Got another slide which might look a little bit hectic, but just bear with me. So these were the features that we went and constructed. With those two engineers, it took them under two months. If you're going to do this before, we were estimating it was going to take four months.

The big thing you can probably have a look in the top half is one of the engineers gets unblocked in under a month and can then start building other stuff. Whereas in the old way, they're both blocked for three months.

And so you get this thing where you now have a compounding increase where it's not just faster for that stuff, but you're now able to do a lot more stuff than you were able to do before. And the one thing that I really want to go and basically go and highlight is it doesn't matter if the output's more, it doesn't matter if the number of deployments are higher.

What really matters is basically if the customers are happy. And so we did statistically significant surveys, and the average quality score was 8.6 out of 10. What was interesting was before AI, we would probably average 7, 7.5. So we weren't really delivering what the customers had been interested in.

So Scrum did not survive. We were basically having a look not just on an engineering front and all of the delivery gains from that, but also from a process standpoint. And so no need for a project manager. We don't need to play multiple games of the telephone, and everything we were doing was optimizing to be as quick as possible.

Engineers aren't the bottleneck, so we don't need to have all the old ceremonies that we had before. So what did we have? It looks pretty basic. We basically had huddles. So those huddles were basically every other day. They'd be like half an hour, maybe an hour.

You'd have the engineers, someone from product, someone from the design team, and you would speak about the things you've been building the last couple of days. You'd get instant feedback. We were trying to deploy to production as fast as possible in an MVP state and get as much feedback as possible also.

And that development flow is we have a spec. I think most of us are probably fairly familiar around this, but we get the agent to go and interview us, we get feedback on that spec, and then we make a lightweight design document.

That's done by the agent, so we have like a skill. And the nice thing about that is it analyzes how we've done all the LDDs before, and so anything that we're building is in the same kind of ethos as everything else that's been built.

So it's not a Claude code specific kind of thing. And then those LDDs, they get distributed and we get feedback from all of the engineers. And then at that stage, we automatically create all the tickets and then the PRs after that.

So sprint planning, we don't have sprint planning anymore because we don't need to have an hour going and basically estimating tickets because those estimations don't really make any difference. Something we're not doing at the moment, but we will probably have to do once the subsidization of tokens ends, is we'll probably have to go and estimate the token expenditure to go see if we want to actually spend this amount of money on it.

The daily standups we don't have to do because all these tickets get auto-updated. That is specific to the status of the PR. So if the PR is open, it goes auto in progress. If it goes into review, it updates it.

If it gets merged, it gets closed. These things were obviously manageable to do before, but it's all been made a little bit easier. Sprint refinement we don't need to do because that happens in the spec and the LDD flow.

And then when we automatically create all of the tickets, we structure it in a manner so the tickets are made so none of them are blocking each other. And if there are ones that are blocking each other, that information is flagged up and highlighted.

And retrospective, this one might be a little controversial, but we rely on the customer satisfaction, the customer survey. That's the main thing really. And then you've got the normal development metrics like the deployment frequency. We ask all of our engineers to really flag when there is an issue, to flag it immediately instead of hanging on until the end of the sprint to go and go and flag that issue, where sometimes I think all of us have probably felt that our feedback hasn't really been heard because it's been a sprint before.

Oops, sorry. So how do we start? You've got to pick the engineers which have the best system knowledge. And I think every engineering team probably has one or multiple engineers where if anyone ever gets kind of stuck or hung up on anything, they're like, you should speak to them, they'll be able to sort it out.

The other one is you should go slowly. So there is an appetite to give everyone all the coding assistance and open it all up, but I think with this, you want a slow-faced approach. And then you should experiment in non-critical systems.

So that's what I did in the two months before the case study. I was making small proof-of-concept features, going and pushing it to production. Didn't really get much traction, so if there was a bug or a mistake, it didn't really matter.

And then we moved it to the thing which gets 100 million page views after that. The big thing here though is not everyone can drive a sports car, and that's allright. That's not a big issue, but I think everyone needs to be a little bit honest around the engineering org and the engineering team.

And this new era is going to be hard for a few engineers. And I think the type of engineer they'll really strive is one who is curious. It's the engineer who, if they haven't figured it out, they're going to spend a bit of time just figuring out how something's been built.

They'll be able to smash this easily. The old style of engineer which needs something really prescriptive as a spec, I think they're going to struggle.

So in the engineering side of things, you need verifiable, deterministic tasks. I think a bunch of people have spoken about this today. It ranges from everything from all the different types of tests, but there's also things which are specific for your product or your org.

So for us, it's feature flag because we do a trunk-based development, or it could be generating the interactive elements, generating the analytics for the interactive elements.

Next thing is the agentic code reviews. So with us, we didn't really like relying on agents to do code reviews for system design and what engineers would usually do code reviews for. We use agents to do the code reviews that engineers hate getting any feedback from.

So that's like the variable names, this doesn't fit the style, or those kind of opinionated manners is much easier to offload. And then you can remove that whole kind of emotional aspect out of it, and then you just allow the engineers to focus on the big picture.

We still need people involved, but it's really heavy on the spec, really heavy on the LDD, the lightweight design document where we figure out how we're going to build this. And then the thing I haven't touched on at the moment is the product.

We're in an era where everyone's able to create anything in an hour, but a lot of these tools have the brand feel and the product feel of something that's been created by a Claude code. And so if you really want to get the best out of it, you really need to make sure that the engineering team is spending time making sure it still feels like basically every other product from the company.

So how would I recommend tackling this? You should go see your engineering development lifecycle like a kind of a factory. So you should think you're in a factory. How do you break that up into each small composable element?

So in a car factory, there's one thing which is building a door, another one fitting a steering wheel. The exact same thing happens for engineering,right? So you have the branch name, you've got creating feature flags with trunk-based development, you've got the if you build APIs with a specific software design pattern, you should abstract that into a composable skill.

The one thing I would flag is I'm not a big fan of

consuming other people's skills if they have strong software kind of opinions that are in contrast to the engineering org because you're just going to kind of end up in issues. So you just need to make sure that any skills that are getting composed are still kind of matching.

And yeah, this is just at the moment pretty much everything is fully autonomous from the spec, LDD, ticket, from the PR. We have a QA process also. So what happens is whenever we merge a PR, it automatically deploys onto staging.

When the deployment on staging has happened, we will spin up a QA agent which has a look at all of the tickets that have happened, has a look at the acceptance criteria, and it will go and then QA against that.

And then if everything's passed, I mean, great. And then if everything hasn't, it will flag what those items are. And the bit we haven't gone and done at the moment, but

I'm naming in the next couple of months, is to then have an agent have a look at those tickets, find out where the acceptance criteria hasn't happened, and then automatically create the PRs. And so then you get into this flow where agents can basically self-heal.

And the cool thing about that is it lets us do multiple things in parallel because we're now in a position where we basically trust agents as well. Where do we still use people? I mean, I've experienced this, but they do like to use

they do like to use shortcuts. So you've got to make sure the security side of things is behaving. Product fuel, I spoke about that a little bit earlier. And then the scale and engineering complexity for tasks, the LDDs really meant to help on this.

So I think we've all experienced where it'll make a thousand lines of code or over-engineer something. If you're really prescriptive on that engineering document at the start, you can really prevent that from happening.

So what things should you go and aim for? You should start with boring, repetitive tasks, ideally things engineers hate because you're going to get the most amount of buy-in from that. Remove as much redundant process as possible. And I know it can be strange when you've been doing something for like two decades plus to just chuck it all out, but I would ask you, what is the purpose of this meeting?

What's the purpose of this process? Is it just because everyone else has been going and doing it before, or is it because it actually helps out? Make sure your team's personal kind of engineering culture and the patterns are encoded in skills.

So if there are software design patterns that you really do, so for us, whenever we build an API, we focus on the service repository pattern. Make sure your guardrails are fully functional before you get into an autonomous flow, and you should start out with the best engineers.

Things you should not do is to try and onboard everyone at the exact same time. I think a big reason why a lot of these things haven't really panned out is because we've given everyone a Claude code codex and given the demo hackathon of we've done everything, they should now be able to sort it all out.

It's just not that easy. Every engineering org is completely different, and every engineering org's style is really different. And you just need to go and have a slow-faced approach to make sure that that transition is happening smoothly. And that's why I think those small companies are at a huge advantage over the big enterprise companies because it's really easy for me to scale this out when I've got 20 engineers.

It's really hard for me to scale this out if I was in charge of 100 or if I was in charge of 1,000 engineers or if I was in charge of 10,000 engineers. But you don't want to be too shy.

You don't want to be too conservative because the fact is that there's a lot of other companies that are going full speed. And

for us, even from our perspective, I felt a few months behind, and I was kind of feeling it. And I think if you think about the compounding impact I was talking about at the start, that's just going to carry on happening.

So a few months behind at the moment might be six months behind, in a few months might be 12 months behind a little bit afterwards. And then again, just

take it slow. It needs a phased kind of approach and rely on your engineers to let you know whether you can scale things faster or if it needs a bit more time. I know I spoke to you about a lot of features that we built, but I didn't show you any of the features.

If anyone does want to have a play around, feel free to scan the QR code and then you can have a look at all this stuff. Cool. Thank you, everyone.