Production software keeps breaking and it will only get worse — Anish Agarwal, Traversal.ai

Um, hi everyone. Thank you for, uh, coming to our talk. So he was kind enough to already introduce us. So I'm the CEO, Matthew is the first person who joined us. If you have any difficult questions, please direct it towards Matt.

Um, now, if you think about the three major categories of, of software engineering, at least as we see it, there's three things that show up to me. There's system design, where you think about, you know, how do you actually architect a system.

A lot of the, the talks we saw in this, uh, track have been about that, in some sense, at a high level. Second is actually developing software,right, putting in the business logic of your particular company and, and all of the DevOps that comes with it.

And then when you, when your software actually hits production, invariably it's going to break, and then troubleshooting of these production incidents,right. At the heart of it, those are the three things interspersed with each other as you think about developing production-grade software.

Now, what, what's been happening with the magic of, um, AI software engineering tools like Cursor or Windsurf or GitHub Copilot, and there's many more others, the, the part around development is getting narrowed,right. That's the part we're relying on all these dis- different systems to do it for us, and increasingly so.

So over the next year, I'd say more and more of what we see, uh, is going to make the development part of it really seamless,right. The question is, what happens to the other two parts of this entire so-software engineering workflow?

The first one being system design, and the other one being troubleshooting,right. Now, what is the promise that is, um, as we think about what software engineering could look like, is that we'll get to focus on just the most high-impact and creative work that happens in engineering, which is system design,right.

So we're hoping that AI will write our code for us and also troubleshoot all the production incidents, so we just get to focus on the fun stuff, which is the creative work of how do we put all these different pieces together.

That's the hope. Um, I actually think that to make that happen, there's something missing,right, which is the, the problem of troubleshooting. How do you automate that part of it? And I think if we just continue in the way we're going, it's actually going to look the opposite, is that most of our time, I think, is going to be spent doing on-call for the maj vast majority of us.

And why is that? I think troubleshooting is going to get more and more complex as we go along. Uh, first is that as we write as these software engineering, AI software engineering systems write more and more of our code, humans are going to have less context about what happened.

They don't understand the inner workings of the code. They don't have all the context in their minds,right. Second, we're going to push these systems to the limits. We're going to write more and more complex systems, just like the things we saw in the previous talks,right.

So the system's going to get more complex, and we're going to have less understanding of it. And as a result, troubleshooting is going to get really, really painful and complex. And I think that's where we're going to spend most of our time in this world, which is just QA and on-call,right.

And that would be kind of a sad existence for ourselves if that's what happens,right. So this, I think, is the grim reality if we don't do something about it. So now, if you think about the workflow of troubleshooting, what does it look like?

In my head, you'll have all of these different wonderful companies like Grafana, Datadog, Splunk, Elastic, uh, Sentry, whatever it might be. And what they do, essentially, is they help process data and help you visualize it,right. And so if anyone who's been on a Datadog dashboard or whatever it might be, you have these beautiful dashboards, thousands of them, to give you some cut as to what is the health of your system.

Now, something will break invariably in production. Then what happens next? The next step, as I see it, is what I call dashboard dumpster diving,right. You'll go through all these different thousands of dashboards to try to find the one that explains what happened, and you'll try to have many different people doing it in parallel.

At some point, you might come up with some sort of promising lead, being like, "Okay, maybe that was it. That's the dashboard that kind of explains what happened, or that's the log that kind of explains what happened." Then you want to connect it to some sort of change you made in your system,right.

As we think about root cause analysis, it's typically you're trying to connect it to some particular change in the system, whether it's a pull request or a change of your configuration files, whatever it might be,right. And that's where you start, say, that's the second stage of your troubleshooting, which is you start aggressively to code-based till something, you know, some sort of, uh, inspiration hits you.

And then most of the time, it doesn't hit you, and then you kind of bring more teams into the mix because you know it's maybe not your issue, and suddenly you have 30, 40, 50, 100 people in a Slack incident channel trying to figure out what happened.

And this loop keeps going on and on and on,right. So the status quo of what incident production incident debugging looks like clearly is, is not optimal. And I think it's only going to get worse based for all the reasons I talked about earlier, okay.

And obviously, this problem has been around since software has been written, and it's not like we're the first people to think about thinking of this issue,right. The problem is that the existing approaches we've taken to troubleshooting or using machine learning AI towards it have not worked, and I don't believe will work, for really fundamental reasons,right.

So the first one is what we call AI ops, broadly speaking, which is we're using traditional machine learning and statistical anomaly detection type techniques to help figure out what happened,right. The problem is, if any of you have actually tried these techniques in production systems, it leads to too many false positives.

Your system is too complex, it's too dynamic. If you just try to come up with some sort of numerical representation of your data, it's going to be not representative enough. And what you'll see is that you'll have thousands of alerts happening.

Maybe one of them is useful, but you just don't know which one,right. So typically, it leads to more signal sorry, more noise than signal, is what AI ops has led to, sadly. Okay. Now, we have this new world of LLMs, okay.

So one option is, okay, let me I'm sure many people here have taken a log and put it in ChatGPT and said, "Explain to me what happened. What, what's going on in this log?" Right. That's something I think all of us have done.

Now, that's okay if you know which log you want to look at. But if I'm actually dealing with production-grade systems, you're going to have petabytes of data. You might have a trillion logs. Which one do I look at?

You can't take all of these different logs, the trillion logs, and put it in a context,right. Even if you have infinite context, it doesn't matter. The size of these systems are so large that forget about context window. It doesn't even fit in a memory.

It doesn't even fit in a cluster. That's why you have such difficult retention policies for data,right, for logs and metrics. And second, the problem with these LLMs are, is that they don't have a really good understanding of the numerical representation of the data.

So they have a good semantic understanding. They don't have a good numerical re-representation, and also the context isn't big enough. Okay. Now, the third thing we might think about is, let's build an agent,right. Like, let's build a React-style agent.

The problem with that is that if you think about these React-style agents, what they're going to do is they're going to assume you have some access to some sort of runbook, some sort of meta-workflow that they can rely on to help you make the decision of what next tool to call.

The problem is, any runbook you actually put into place is deprecated the second you create it,right. I don't know how many of you have used runbooks, but typically what we've found, in my experience and the team's experience, is that they're typically deprecated by the time they're built,right.

And so that workflow that the agent is going to go through is not going to be, um, is, is, is just not going to be optimal. And also, if you try to make it do a much more broad search of your system, if you give it these simple tools, it's going to be take too long,right.

If you just put it into, into like a loop, to-tool-calling loop, it might take days to run if, if it doesn't time out,right. And typically, if you're trying to solve an incident, you need to solve it in two to five minutes for it to be really useful,right.

So every minute counts when things are down. So as a result, none of these three things, if you do them just by themselves, is enough to really trouble-troubleshoot these fundamental issues in all of them that I think need to be solved by thinking about them in a more collective way.

And that's really what we're trying to do at Traversal,right. Well, what you need to do with Traversal is what we're trying to do at Traversal is really go to out-of-sample autonomous troubleshooting, which is that you have a new incident.

We've never seen it before. Can you troubleshoot it from first principles debugging? And to do so, I think we need to combine a few different ideas. The first one being statistical, the second one being semantic, and then the third one being a novel agentic control flow.

And with statistics, what I mean is, is causal machine learning. So that's where a lot of our research came. So the idea of causal machine learning is this idea of being correlation isn't causation. So how do you get these AI systems to pick up cause and effect relationships from data programmatically?

That's what we spend a lot of time thinking about. And obviously, that problem shows up a lot in production incidents because typically, when something breaks, a lot of things break around them,right. So there's a lot of correlated failures, which are not the root cause.

And so thinking about it programmatically is what the study of causal machine learning is. The second being semantics,right, which is actually trying to push the limits of what these reasoning models can give you, uh, to help you understand the semantic, rich semantic context that exists in, in log fields, in the metadata source of the metric, and so on and so forth,right, or in code itself,right.

And so by combining causal machine learning, which is the best of what statistics gives you, and reasoning models, which is the best of what semantics gives you, you now have at least the tools, the basic tools to put into place so that you can actually deal with this issue.

Now, you have to figure out how do you actually make this work in an agentic system. And what we found is this idea of swarms of agent, where you have these thousands of parallel agentic tool calls happening, giving you this kind of exhaustive search to all of your telemetry in some sort of efficient way.

That's what brings it together and helps you actually deal with this issue,right. So just to repeat, it's statistics, which is causal machine learning, semantics, which is these reasoning models, and this novel agentic control flow, which you call the swarms of agent.

All of this put together in theright, elegant way is what actually helps you autonomously troubleshoot,right. And a lot of the work we, we rely on is based on our years of research, years of experience as researchers. We've written a number of papers on these various different fields and also relying on a lot of the research that has happened, uh, in general in the field over the last couple of years that we're relying on to actually make this into a reality.

Um, and so if you think about the, the troubleshooting workflow we talked about earlier and all the pains of it, each of the different things can help remove or alleviate some of those pains. So the idea of finding the promising lead from your sea of information, that's where this agent swarm and causal machine learning put together really help.

And then from the promising lead, connecting it to a specific change in your system, whether it's a pull request or change log, that's where a lot of the work that's happening in code agents and also vector search, we're relying on it, uh, as this get better, we get better,right.

And as a result, because you're building this context in real time and agents are doing it, they're pulling in theright team with theright context at theright time. And so don't people want to get pulled into an incident and being like, "I don't really know why I'm being pulled in here," which I'm sure many of you have been experiencing in your time as engineers.

Um, and so now I'm going to hand it over to Matt to actually talk through a real-life, uh, case study, uh, that we have done with one of our customers.

Hi, everybody. Uh, I'm Matt, as Anish said. Thank you. Uh, and no, I, I don't o-own any other outfits. Um, so I'm going to tell you a little bit about how I got involved with Traversal, and I'm going to show you how we are helping out some of our customers.

So, uh, in my career, I spent most of my time in high-frequency trading. Uh, what I wanted to spend my time doing there was, uh, you know, focusing and writing beautiful code. What I ended up doing much more often than not was debugging production incidents.

So I got sick of this, as you might understand, and, uh, very happily quit. Some time later, I, I found myself in a class on causal AI. Uh, the guy teaching the class seemed pretty smart. It, uh, looking back, then it may have just been the fact that he was much better dressed than your average computer science professor, and he had a, you know, a fancy accent.

Um, but he, he mentioned in class one day, sort of an offhand comment, it seemed, that, "Oh, you might be able to use causal AI to automate incident remediation workflows." And I'm sitting in the back, back of the class, and my head explodes.

Like, this sounds like an amazing idea. And so I casually come up to him at the end of class and say, "You know, maybe we might want to do a research project on this." And, uh, little did I know that at the time, he had actually started a company to solve this problem.

So he in-invited me to join him on that journey, and I'm very grateful for that. And so fast forward a year, though, and we have, uh, we built something that we're quite proud of, and I'll, I'll show you how it works.

Uh, but to begin, uh, I'm going to be talking about a client of ours, DigitalOcean. DigitalOcean is a cloud provider. They serve hundreds of thousands of customers every day. And I'm going to tell you about what, uh, the life of an on-call engineer was like prior to Traversal.

So imagine you're in the middle of a productive workday. You're focused in, you're writing code. Um, and then you get hit with a message, uh, describing something that's broken horribly. It's, it's causing issues for customers. It might say something like, "Powered some potential compromise of some host assigned to, you know, a bad application."

And my apologies in advance here. This is not a demo per se. This is us telling you in the real world how, uh, we are, we are solving issues for our customers. So apologies for the redactions that you're seeing here.

So you get this context, and then you get thrown into an incident Slack channel. And in this Slack channel, you and 40, 50, 60 other engineers begin frantically searching to find what's the cause of the customer issue,right. But you're not looking through some documents.

You're not looking through a database table. You're looking through hundreds of millions of metrics, which are viewable on thousands of dashboards. And beyond this, you also have tens of billions of logs that you might want to find. And these are coming from thousands of services.

And what's the thing you're looking for here? You're looking for something that's comparatively microscopic. So this might be a single log that describes the thing that went wrong, the root cause of the incident. And, you know, if you're lucky, after a few hours of everybody frantically searching, the incident gets resolved, and, you know, everybody can go back to work.

That is until the next production incident comes, and the cycle repeats itself. So this is a very familiar situation to me. Unfortunately, it may be a very similar situation to all of you. Uh, but things have changed for DigitalOcean.

So

what Traversal's been able to do for DigitalOcean is make their mission-critical infrastructure far more resilient. So you'll see here that, uh, mean time to resolution, uh, MTTR, has reduced pretty dramatically for DigitalOcean. We've measured that's about 40% reduction in the amount of time that it takes to find and resolve production incidents.

And all of these minutes mean a lot of pain for, uh, engineers that's, that's now been removed, and also thousands of dollars for each minute. So now, I'll be able to show you how this works. So in the world post-Traversal, rather than this frantic search going on all the time, when an incident kicks off, the Traversal AI, the ambient Traversal AI, begins its investigation.

And the thing that it begins its investigation with is the same little bit of context that engineers get when the incident starts. And given the small amount of context, Traversal AI orchestrates a swarm of expert AI SREs to sift through petabytes of observability data all in parallel.

And so what was done previously manually is done exhaustively and automatically. And then after about five minutes, Traversal comes back to the usersright where they are in the incident Slack channel and tells them what happened. So here you can see that, uh, Traversal identified an issue.

There was a deployment that introduced changes, a cascade of issues throughout the entire system. And when engineers see this, they can roll it back. They can move on and, and get back to their good work. And you'll note here that, uh, Traversal, the engineers at DigitalOcean noted that this was, this was the thing that solved the issue.

This was the correct finding. And this is happening all the time. Engineers that want to dive in further can look in the Traversal UI, where you have a well of information that Traversal unearths, uh, describing what happened with the incident.

It cites relevant observability data, just like any good engineer would. It gives you confidence levels for the potential root cause candidates. It explains its reasoning. And if you want to dive in even further, Traversal allows you to interact with an, uh, AI-generated impact map.

And you can even ask follow-up questions to Traversal to describe to ask questions like, "How does this incident impact the part of the stack that I care the most about?" And this entire experience is exactly the kind of thing that I was dying for in my previous roles, and I'm so excited to share that it's now a reality.

So thank you. Uh, so, but this is not just happening for DigitalOcean. We're working across a very heterogeneous group of, uh, enterprise environments. Uh, so you'll see here, the thing that I want you to focus on here is that we're talking to basically every obser-obser-observability tool out there, and we're talking to them on a massive scale.

We're talking trillions of logs. So yes, it's an AI agents problem, and I, I like thinking about the AI agents problem, but in reality, this is just as much an AI infrastructure problem.

Furthermore, the pro-problem that we're solving here is one where we have a massive set of data, and we're looking for this small piece of information that's telling you everything that you need to know. This is sort of a needle in the haystack problem.

And we're focusedright now on observability, but there are plenty of other domains where there are since similar principles that can be applied. In my experience, that when you're doing network observability, you're working on cybersecurity, you find similar patterns here that I hope that you all in your respective domains can take from this, that these strategies of exhaustive search and swarms of agents can be applied in this, in these domains as well.

Uh, not only is this an amazing problem to work on, frankly, we've assembled an amazing group of people to do it. Uh, we've got AI researchers from the top AI labs in the world that, that are pushing the limits of what AI, what AI can do.

We have people from top dev tools companies who know what it takes to build a tool that developers love to use. We have fantastic AI product engineers who know, who know how to make delightful AI products. And of course, we've got high-frequency quant finance traders who can deal with pain and know what the pain of production downtime actually means.

But I think what's, uh, most special about the Traversal team is not necessarily the skills that the, that we bring to the table, but, you know, it's, it's really unusual. It's really rare to be able to find a group of people that aren't really all out there for themselves, that everybody's working for the betterment of each other.

And we have a lot of fun coming into work, and everybody loves showing up to the office. And this is what makes Traversal, in my opinion, an amazing place to work. And so if the team or the problem sounds interesting to you, please, uh, scan the QR code, look at one of these websites, and, uh, help us try to create this picture of me that we haveright here for all, all the engineers in the world.

Thank you very much.