Building AI Agents with Real ROI in the Enterprise SDLC: Bruno (Booking.com) & Beyang (Sourcegraph)

So my name is Beyang. I'm the CTO and co-founder of a company called Sourcegraph. We build dev tools for big, messy codebases.

Yeah. And I'm Bruno, Bruno Passos, and I lead the product side of Developer Experience at Booking.com. And yeah, over the past year I've been overseeing the GenAI innovation side of Booking as well.

Cool. And today we're here to talk about how we're partnering to build software development agents that automate a bunch of toil inside Booking that are actually having real ROI and impact. So how many people have heard this before?

You're working inside a large company, the CEO comes in and says, like, hey, we need to adopt AI. And then folks are like, OK, what does that mean? How do we measure it? Maybe like FOMO purchase, Copilot, or something like that.

And then six months later, someone else, maybe the CFO, is asking you, hey, so what's the ROI of that AI tool we just adopted? Or what's the measurable impact of the agents that we're building? This is a question that I think a lot of people aren't quite sure how to answerright now.

But Bruno and Booking have been sort of on the leading edge of answering this question, and very proactive at acquiring and building the best tools, and also following through to demonstrate how they're actually impacting their org.

It's very kind of you to say we are leading this. I think we areright at the beginning, and I couldn't feel further from actually the forefront of it. But let me start by talking a little bit about Booking.

I am sure that most of you would have heard about this company. Our goal is to make it easier for everyone to experience the world. And my team's goal is to make sure that our developers have their path cleared so that they can do their best work.

Now, are we close to that? In some parts of the company, yes. Other parts, we couldn't be further away from it. To set a little bit of context, we are one of the largest online travel agencies in the planet, and we serve about 1.5 million room nights with more than 3,000 developers.

Can you raise your hands who work in a company that has more than 1,000 developers? Quick show of hands.

Good number of people.

OK. More on the dev side or on the technical side, we serve over 250 merge requests at a given year, with 2.5 million CI jobs running at a given year as well. And we are extremely data-driven. Our company has gotten to where it got to over experimentation and being obsessed about data.

And the reason I'm going into this is because as we experiment, and in the form of primarily A/B tests, we start adding those experiments and feature flags into the codebase. And as we push forward to bring new features to our users, most likely those experiment flags or dead code will stay in the codebase.

And now, fast forward decades, our codebase became extremely bloated. Fun fact, my kids were looking at me editing this slide, and they said, what are feature flags? And I said, well, they stay in the codebase, and they start polluting the codebase, and they were like, code farts?

And I'm like, now you're going into code smells. It's a different topic, but let's move forward. And so as the codebase starts to bloat up and become bigger and bigger, cycle times also become larger and longer. And the time that developers spend to debug and to work on that codebase just becomes over 90% toil,right?

Who here is familiar with this?

That's even more hands than 1,000 developers. And so we survey our developers at least a quarter on how they're feeling, how they're feeling about working on that particular codebase. And it just becomes harder and harder for them to do anything.

And so we had to do something about it.

So I've seen the best developer minds of my generation destroyed by decade-long dead feature flag migrations. It's crazy. A lot of you didn't actually say that, or? But I mean, seriously, though, there are probably geniuses out there. I was talking to someone from PWC the other night and described the system that they're building to update all the kind of legacy code in their system.

And it was amazing. The guy was really smart, really brilliant, really interesting tech. But wouldn't it be great if those sorts of minds were unlocked to actually work on new features and thinking about user problems rather than all this kind of legacy craft?

And so in a nutshell, that's why Sourcegraph exists as a company. So our mission is to make building software at scale tractable. And so you might be familiar with a couple of the products and tools we've built along the years.

Code search, it's kind of like a Google for your code, allows any human developer to find in and build a working understanding of what's going on. We have a tool for large-scale refactoring and code migrations. You might have heard of our AI coding assistant, Cody.

It's a context-aware code generator that's tuned to work well in large, messy codebases. And the topic of this talk is really about the agents that we're building to automate toil out of the software development lifecycle. So a bunch of different products that we've built over the years, the unifying theme really is to accelerate things in the developer inner loop, augment human creativity there, and then to automate as much of the BS out of the outer loop as possible.

Allright. So as Beyang talked about Sourcegraph's search, just over two years ago we started using their product. And it was a big success within our community because they were able to search that bloated codebase much, much easier and find small pieces of context lying here and there.

I totally encourage you to have a look at this particular product. It's awesome. And so about a year ago, January last year, we started experimenting with Cody. And why? Because Cody also has Sourcegraph's search as context, and so it became extremely useful for us to use a tool that had that context to be able to experiment with the GenAI topic.

And now we are hoping to reach the path of building agents with Cody and Sourcegraph search built in. Allright. So if I summarize very quickly, and hopefully this illustrates how fast things are moving forward, in January, we started with Cody.

We gave everyone the ability to start using the tool in the company. So all our 3,000 developers had the opportunity to use it. Some started using it, some used it, didn't see any value with it, and then stopped using it.

And that started intriguing us. And so back then,right in the beginning of the year, we had the choice of one LLM to use across the entire company and some token

limits, limiting what we could do with it. And so the first thing that we started pairing with Sourcegraph, and we appreciate the partnership on that, was to remove all the guardrails that we had in order to be able to really give it a go.

And so Sourcegraph was very quickly to be able to give us multiple LLMs per developers. We could choose that. And why that was important is because we found that LLMs had expertise,right? And so if we were going to excavate our codebase, our bloated codebase, a particular LLM would do better than someone that was working on a completely new piece of service and developing features there.

And so fast forward to July, we started training developers, and that became incredibly important because the people that started using and didn't see the value, when they started getting trained, they started using it and falling in love and becoming what we call now daily users.

And I'll explain why that's important. And then we started looking into more metrics. Back in January, the main metric was hours saved. And I mentioned that we are a data-driven company, and hours saved wasn't the most statistically relevant metric that we could use.

It was based on research only over a couple of developers, a few developers, and that wasn't cut. Raise your hand here if you heard folks out there in the beginning of the hype talking about thousands or 80, 100,000 hours they saved with GenAI.

Has anybody ever heard that? And then you go back to your company and say, why are we not doing this? I call that semi-BS. And so we had to start going into other metrics, something that were more statistically relevant.

And so we started brainstorming with that. Come October, we defined new KPIs, which I'll go deeper into it, and metrics to measure GenAI. And fast forward to November, end of last year, we then started finding traces that developers were 30% plus faster if they were using Cody on a daily basis.

And that's 12 plus day in a month to take away weekends and the times that they were not coding. And most importantly, we were able to partner with Sourcegraph to be able to create an API layer in front of Cody so we could be creative in using with some of the tooling that we use, like Slack, Jira, and being able to extract some of that away from the IDE.

Allright. So as we finish around October, we started looking into those KPIs. And what was important for me is that we defined something that we could measure within a year. Why? Because things are moving so fast. And it was really helpful to ground us to what can we measure within the next year.

And so we defined four KPIs: the lead time for change, quality, codebase insights that would then go into how we could modernize some of our bloated codebase. And so some of the metrics, when I say short, mid, and long term, these were metrics that we could see results in the short term, in the mid term, and in the long term.

And that long term is precisely a year. And so we started seeing results with time to review MRs. Developers that were using Cody on a daily basis would ship 30% more MRs than the ones that didn't. And one very interesting piece is that their MRs were lighter.

They had less code in it, which I still don't know what to make out of it, but we are working on it. And then on the quality side of things, we are hoping to go into the vulnerability. Can we show some of the vulnerabilities we've had in the past?

Give the context, the codebase's context, and try to see where we can predict where the new vulnerabilities will appear or if they're still lingering in our codebase. And then we started out. The obvious one is test coverage. Can we increase test coverage?

Can we create test coverage on the legacy so the new stuff, when we replatform, passes that particular set of tests? And then we went into coding sites, which is more related to, like, can we track what parts of our codebase are not being used, some feature flags that are still lingering but shouldn't be there, and the code that is not performant enough.

And all of this is to feed into our ultimate goal, which is, can we bring the time to replatform our codebase from, yeah, years to months,right?

OK. So while all this is going on, one of the things we noticed is that the same engineers that were using the coding assistant to generate code were also playing around with the underlying APIs. And so what we realized is that asking people to customize prompts leads to them wanting to build and compose those calls into longer chain automations that we now call agents.

There are a lot of pitfalls that we encountered

in the early stages of this, like helping people understand what the expectations were with respect to what the LLM can do and what it can't do. But the long story short is, at some point, we basically said, F this.

It's not really working. Let's just put our brains together, fly out to Amsterdam. We'll do a week-long joint hackathon and build some agents together. And so the first thing to come out of that hackathon was this thing that generates GraphQL.

So Booking has a huge GraphQL API. Go play the video.

It's seriously more than a million tokens long. So it does not fit into the context window of any of the existing LLMs. Even if you could shove it inside context, it's not going to do a good job of integrating that context into something that's coherent.

There are a ton of hallucinations. And so what we did is we built this system that basically searches this very, very long GraphQL schema, finds the relevant nodes wherever they are in this schema tree, and then agentically figures out which ones are relevant, and then walks up that tree to pull in the relevant parent nodes.

And so on theright-hand side, you can kind of see it's like inner dialogue. This is like its thought process for reasoning about which nodes of the schema to pull in. And then after it's done that reasoning, it generates a response.

And so if you do this naively, the UI looks very similar, but you just end up getting garbage, which is what we're seeing before we ran this hackathon. After we sat down and actually worked through the specific prompts and stuff to make this work well, we saw far better results.

Allright. So a pretty interesting one that we started working through in terms of agents were the automated code migration. Could we go into that legacy piece? Functions with over 10,000 lines to give you context and speed up that replatforming effort.

And so code search, structured meta prompts, and then the concept of dividing that particular codebase to conquer the small bits were really, really interesting. One of the things that I totally recommend if you're starting to embark on a journey like this is pairing with some experts to bring that expertise into your offices was incredibly valuable to us.

And we started seeing back to when I mentioned that the developers were using Cody and stopping and feeding back doesn't add any value, was pure lack of knowledge. Folks didn't know how to work LLMs out. They didn't know how to pass theright prompt in theright context.

And this was a pretty important piece for us to be able to work on this particular agent. And so when we go into this, we had developers working for months at this point to try to figure out the size of the problem that we had to then be able to divide and conquer.

And then we came within two days within a hackathon, we were able to really define and understand where the call sites were coming from. And then being able to define how big the problem is was important for us to be able to have a start point and then collect the low-hanging fruits that were available for us.

So all of this is still in experimentation mode, but we've seen a lot of values and a lot of sort of firing that smoke going from months of understanding the codebase to days.

Cool. And so the last agent that really came out of this joint effort was targeted at code review. So this is something that we found is pretty universal across many different enterprises. Everyone who does not do code review here?

One hand. OK. I'll talk to you later, sir. So everyone does code review. And what we found, originally we didn't think this was a very interesting space because there's like two dozen startups now that are popping up that do AI code review.

But when we talked to Booking, we talked to other enterprises, what we found is that code review is kind of very specific to your organization. There's a long tail of rules and guidelines and other things that you want to bake into your review process.

And a lot of the tools that are off the shelf there aren't super customizable. And so what we built is this interface where we're going through and productizing the process of building a review agent that's tailored to your team and your organization.

So the basic idea is that you define a set of rules that you want to hold in the code. And then those are defined in kind of a simple flat file format. And then the agent will go and consume those rules, apply the relevant ones to the specific files that are modified in any given PR, and then very selectively post up comments that are tuned to those rules.

So it's very not noisy. We're trying to optimize for precision over recall here in the feedback that we're giving the developer.

Allright. So knowing what we know, in the beginning of this year, we've been working on this for a year together with Sourcegraph. Then a few ideas started popping into our minds of how we could go forward here. And one of the things that I'd love to leave you with is the concept of declaring what are the rules of your service,right?

So think of your CI pipelines today. When they give you errors, could we anticipate and shift this left to the IDE so those errors appear there and they appear in the form of, here is an error and here is a fix?

So hopefully the service gets to a point where it's self-healing,right? And we started seeing that we could do that, that there are areas that we can start using, giving all the context, all the prompts that the developers started creating via the prompt library that we created, and asking those questions, automate those questions to the server to see what comes out in terms of knowledge out of that codebase.

And so we think this is ultimately what we are trying to achieve within, I would say, as short as the end of this year in terms of agents. But

lots here to go.

Sorry. Can I just say one more thing about that last slide? I think that we have the potential here to solve one of the problems that has plagued software development since its inception. So who here has read the Mythical Man month before?

Yeah, basically everyone. So it's this problem of any software that becomes successful eventually becomes a victim of its own success. Because if you have revenue, if you have users, that's going to generate feature requests, bug reports. Any business that's prioritizing that is going to take on tech debt in order to compete, quite frankly.

And over time, as you add contributors to the codebase, you lose this cohesion of vision. You lose the set of standards that you want to maintain and hold. With declarative coding, now you can have the senior engineers, the architects, the people in charge of the organization define constraints and rules that must hold through the codebase and enforce those rules both at review time as well as inside the editor for the code that's written by human or AI.

Yeah. For bigger organizations, all your compliance rules, all the things that the developers need to work on, but it's not necessarily feeding new features to your end users, I think those could be perfect examples of, yeah, being declared into your service.

But anyway, the main important thing so far in this past year that we've been pairing to be able to figure this out has been education. The more we educated the developer and hand-held the entire business units to be able to show them the value, but then have them experiment within two days of workshops and hackathons, have them experiment with the tool, they were coming out the other side incredibly passionate about what it can do, but also becoming that daily users that we are trying to transform them so hopefully to defend that 30% plus increase on speed.

So educate your folks. If you take one thing from this, it's education.

And if you want to dive deeply into any of this, we've got a booth downstairs. Feel free to stop by. We'll talk shop. Or also tomorrow, I'm giving an expo talk that covers some of the more nitty-gritty details of how some of those agents were implemented.

So thank you. Thank you all.