What We Learned Deploying AI within Bloomberg’s Engineering Organization – Lei Zhang, Bloomberg

I don't have a joke about a dog, I don't have a joke about the hot dog either, so I will just jump to the topicright away. Um, so my name is Lei, and I lead the Department of Technology Infrastructure in Bloomberg.

So we're basically a group of technologists focused on global infrastructure: think data centers connectivities, developer productivities, think SDS tooling, and also reliability solutions, think telemetry and incident responses,right. So, um, depends on the audience. Sometimes, you know, you're familiar with what Bloomberg is, sometimes you don't, so I thought it might be a good idea to talk a little bit about our company.

Um, so there's no better way to talk about our company by sharing some numbers. I want to highlight a few numbers. We have more than 9,000 engineers, and most of them are software engineers. We handle a lot of market techs, which is in the billions and 600 billions, I believe.

And, um, we also have tons of folks focused on AI research and engineering, so we have more than, really today it's 500-plus employees focused on AI products for, sort of, our customers. So takeaway here is we are, I guess, you know, building a lot of software and use a lot of data to empower our flagship product, which is called the Bloomberg Terminal, and to really support our users to make the most important decisions for them to do their job the best.

Um, in the technical lens, a lot of time kind of into explain that we actually have one of the largest private networks in the whole world. We also have one of the largest JavaScript code bases in the world.

Um, we because the domain where I am, so Bloomberg Terminal is really you can think of a software that supports thousands of different applications. We call them functions,right. Um, email is a function. News is a group of functions.

Um, let's say fixed income price to yield calculation to spread calculation is another function. Um, trading workflows is another group of functions. So there's many, many, many different types of functions. As you can imagine, we kind of have to utilize different technologies to really support those functionalities.

Uh, we also been increasingly more than used, but also contribute to open source communities. Um, for this audience, I guess I want to call out, you know, we kind of helped creation of the CaseServe, Envoy AI Gateways, and among many, many other things that we deploy in-house and support the communities.

Again, in summary, there's a lot of software, there's a lot of data. We kind of have to figure out how to make the best of AI tooling to support us to do our engineering work. Allright. So get to what is AI for Coding.

Um, we started about two years ago, maybe a little bit more than that. Um, and as, I guess, the rest of the world, we look at the toolings provided. And, you know, I apologize if your logos are not here.

Um, but as you can imagine, it's kind of like overwhelming,right. There's so many things, and every day there's news about this is great, this is great. Um, so at the time, we actually didn't know what all the AI solutions can help us to boost our productivities as well as stability.

But one thing we knew at the time is, um, unless we deploy and try, we wouldn't know what's the best way to benefit from all the awesome work and, you know, a lot of folks are contributing to. So at the time, we quickly formed a team of people, start kind of like release, um, a set of capabilities so that people start iterating on, um, utilizing the toolings.

And then, of course, you know, we are a data company, so kind of want to get a sense of how we measure the impact and, um, what we can do from the capability we provide,right. So we look at the typical developer productivity measurements.

We ran a few surveys. It was very obvious that people felt like there's much quicker proof of concept. People roll out tests. There's a lot of one-time use scripts being generated. And then the measurements dropped actually pretty quickly when you go beyond all the greenfield type of thing,right.

And then we start thinking like, okay, so what are the things that we should really be doing using all those wonderful things so that we can really make a dent in the space. And then at the time, we also kind of like also be thoughtful of unleash a very powerful tooling,right.

The benefits is it's very fast. The challenge is also it's very fast,right. For any of you who actually dealt with hundreds of millions of lines of code, you probably understand the system complexity is a at least exponential or at least polynomial, I guess, function of your live code or software assets,right.

So at some point, you kind of want to be very careful what you do with your software assets. And what we thought, maybe we should look at some of the basics. One idea we had is, um, allright, so AI for Coding.

There's a narrow definition of what Coding is, but there's also a broader definition of what software engineering,right. And then maybe we can also look into some of the work our developers don't really prefer to do. For instance, um, some maintenance work, some of the migration work, some of the, I don't know, maintenance work and stuff like that.

So I want to give some examples of the things that we've been trying and we think there's pretty good return on investment. So the question we ask ourselves is, how do we evolve our code base,right. The first one is, allright, wouldn't it be cool the day you get a ticket saying, hey, you know what, this piece of software needs to be patched, and at the same time, you have a pull request with the fix, with the patch, and also with the thinking why the patch happened that way,right.

So it's kind of like we're trying to broadly deploy something called Uplift Agents. Um, broadly scan through our code base and figure out what the patch would be applicable and be able to apply those patches. Step back a little bit.

We did have a regex-based refactoring tool. Um, it works to some extent, but it's limited,right. Now with, um, LLMs and other toolings, we are able to see very much better results from the Uplift Agents. So there are a few challenges in case you also plan to deploy such capabilities.

The first one is, I guess, any AI or ML, it would be really nice if there's some deterministic verification capability. Oftentimes, it's not so easy, especially if you don't have test cases, if you don't have good linter, if you don't have good verification.

The patch can sometimes be

difficult to be applied. And one thing we also realized when we deploy AI tooling is the average open pull requests increased and time to merge also increased. Because you're spitting out new code and then still we have to review the code and merge the code,right.

So time to merge becomes a challenge sometimes. And the last one is, um, I think it applies to any engineering thing is the shift becomes what do we want to achieve rather than how we want to achieve,right. So the second example that I want to share is, um, the other area that people kind of like sometimes really impact our productivity in a negative way or impact our stability in a negative way is how we handle incidents.

So we're trying to develop and then deploy, um, incident response agents. Um, now

the importance of this is if you really think about GI tools, it's really, really fast and it's also unbiased,right. In my instance, it can go through your code base really quickly. It can go through your telemetry system very quickly.

It can go through your feature flags very quickly. It can go through your, um, I don't know, code trace very quickly. And in an unbiased lens, when we do troubleshooting, sometimes we have this biased view, it must be this.

It turns out to be not the case. So there's many, many interesting benefits by deploying agents from this perspective.

And then the second question is becoming interesting is imagine you have an organization of 10,000 people, um, let's say 9,000 people, as I described. A lot of people trying to fix those problems,right. And you can have 10 teams who want to build a pull request review bot.

You have 20 teams who want to build an incident response agents,right. They become very quickly chaotic and sometimes can have duplications. So before I talk about the Paved Path, I'm going to give an example of the incident response agents.

So basically, this is what, you know, an incident response agent will look like. The key part is we're going to need to build a lot of MCP servers to connect to the metrics and logs dashboards you have, connect to the topology you have, whether it's network topology or it's the, um, your service dependency topology, your alarms, your triggers,right, your SLOs.

And then we kind of don't want people just start building MCP servers without a Paved Path. So we created a Paved Path in partnership with our AI organization, and I will talk a little bit what that means. Before that, um, I do want to explain a little bit some of the platform principles.

Some companies allow teams to have a lot of freedom at the same time responsibility in the sense a business unit can build whatever infrastructure, whatever platform. Um, some organizations have a very, very strong tight abstraction of the service infrastructure and typically kind of have to use their platforms,right.

So Bloomberg is kind of in the middle. If you look at the golden ones, we kind of believe in provide a golden path, um, with enablement teams. So my team is really an enabling team. And one of the guiding principles for us is we want to make easy things extremely easy to do.

Uh, sorry, theright things extremely easy to do, and we want to make sure the wrong things ridiculously hard to do. So that's the guiding principle here. Now move on. So what is the Paved Path here? So the Paved Path is, uh, we have a gateway so that teams can easily figure out which model works the best.

They can do quick experiments. They can, um, we can have visibility of what kind of model is being used, and we can also guide through teams which model is a better fit for the problem they want to solve.

Uh, we have a tool discovery, basically MCP directory via hub so that, let's say, team A wants to do something, they will go to the hub. Okay, someone is building an MCP server already. Maybe I should partner with them to build it together,right.

Uh, tool creation and deployment is via a path. It's basically a, um, you know, a standard platform service where you can do SDLC and we provide runtime environment for you as well, taking care of all auth and side of things as well.

So it really reduces friction for teams to deploy their MCP servers. And then this is kind of interesting is we want to make demo very easy so that, or I should really say proof of concept very easy so that people can try have idea generation.

Um, because we believe in creativity comes from some freedom of trying different new things. But we also want to make sure the production requires some quality control. Um, because at the end of the day, stability and system reliability is at the core of our business.

This is sort of the Paved Path that we deployed, um, and enabled the rest of engineering, really the 9,000 software engineers to do their job. Okay. And, um, with all this, and then we start maybe, okay, yes, we got Paved Path.

We have some good ideas of how to evolve our code base. Help out our people,right. Um, now this is where I find that

any new things, any adoption of new things provide opportunity to leverage the strengths you have and also identify some of the weaknesses that you may have. So, um, in Bloomberg, we have a well-established training program. Uh, it's more than 20 years.

So there's onboarding training. It depends on entry level or depends on senior level. Um, so we have this whole training program to prepare folks before they join the team. And what we did is we just incorporate AI coding in the onboarding training program and also show them how to best utilize them with our principles and our technologies,right.

There's a huge benefit here because, um, if any of you run into the challenge of adoption somehow running into a chasm,right, the rest of the org is not adopt as quick as possible. Whenever we have folks join a company, they learn how to do things in a new way.

When they go back to their team, they were like, hey, why don't we do that,right. They're going to challenge some of the senior folks as well to say, hey, there's a new way to do this type of thing.

So why don't we do that. So we actually find this program extremely effective, uh, to be a change agent for anything you want to push out. And then a bunch of results. There's a lot more familiarity and comfort with the tooling.

Um, and also the important part is there's a lot more nuanced insights of where it's add value,right. The second one is, um, oftentimes we run organization to push, uh, new initiatives. So within Bloomberg, we have something called, um, a champ program and a guild program.

That's basically across organizational tech communities where people have similar interests and similar passion. They get together and get stuff done. So, um, we had this for more than 10 years now. Uh, we sort of bootstrapped engineering AI productivity community two years back, leveraged the community we have already, and then have some few results.

Uh, because we have this pretty much everyone passionate about this and we'll be in that community. So organically, it deduplicates efforts and there's shared learning, uh, shared learning happening. And it also helps to boost in-resource contributions and the vision engineer idea,right.

Oftentimes, team A wants to do something and team B, let's say a platform team, have different prioritization. And the way we solve this is via in-resource or via visit engineer, which is move someone over the team, work for six months, a year, get it done, and then we can move on.

Um, the last one is interesting. So our data shows individual contributors have a much better, stronger adoption than our leadership team. Now, if you think about this, a lot of software TLs and managers

in the age of AI, they kind of don't really have, um, enough experience to truly guide their teams to build software,right. So oftentimes the stuff that they learned before might not be exactly applicable, but still very valuable, but there's some missing piece there to make sure they can continue to guide the team to do theright thing.

So we're rolling out leadership workshops to make sure our leaders are equipped with whatever knowledge they need to have to drive the technology, um, innovation.

So, um, I'm going to close my part and to share with you what, uh, the part I feel most excited about. The part I feel most excited about is that with a lot of, um, creativity and innovation in the GNI space, it actually changed the cost function of software engineering, meaning the trade-off decision of whether we do something versus we don't do something actually changed because some of the work becomes a lot cheaper to do and some work becomes a lot more expensive to do.

I tend to think it is a great opportunity for engineers and engineering leaders to get back to some of the, uh, basic principles and sort of ask a soul-searching question. What is a high quality software engineering and how can we use a tool for that purpose?

So that's it. Thank you very much.