Your Support Team Should Ship Code – Lisa Orr, Zapier

I'm so excited to tell you about how at Zapier we are empowering our support team to ship code. Before I tell you about that, has anybody here visited the Grand Canyon? That's a good amount. Anybody rafted through the Grand Canyon?

I see one person. I just got off an 18-day trip rafting through the Grand Canyon, over 200 miles. It was incredible. No internet, no cell service. The moment I got off, I found out I was giving this talk.

I didn't think about work at all on the river, but once I got off, I started thinking about the parallels between the Grand Canyon and Zapier. And we have one thing in common, and that is erosion.

Now, natural erosion happens over millions of years with wind, water, and time. It creates the beautiful canyon that we experience, and it's never stopping, always continuing. At Zapier, we have over 8,000 integrations built on third-party APIs, and they are constantly changing, which I'm now thinking of as app erosion.

We've been around for 14 years. Some of our apps are that old. API changes and deprecations impact us and create reliability issues. Again, it never stops. So I like to think of our apps as like layers in the Grand Canyon, and they need constant attention.

So if we were to create our own Zapier Canyon and our apps would be at the walls, here's our support team flowing down the middle, watching out for app erosion. And we have a backlog crisis. Tickets were coming in faster than we could handle them.

It creates integration reliability issues, poor customer experience, and even churn. So to solve for app erosion, we kicked off two parallel experiments. The first was moving support from just triaging to also fixing these bugs. That's experiment number one.

Experiment number two, we were asking, can AI help solve app erosion faster? So let's jump into experiment one. This got kicked off two years ago, but had to start with the why. We needed to get that buy-in to empower our support team to ship code.

So app erosion is one of the major sources of bugs coming through from support to engineering. So there's a big need. Support is eager for this experience. A lot of them want to go into engineering eventually. And unofficially, many support members were already helping to maintain our apps.

This moves us into how we started this out. Put on some guardrails. We started with just four target apps to focus our fixes on. Engineering was set to review any merge requests coming from support, and we kept the focus on app fixes.

So jumping into experiment two, this is what I've been leading for the last couple of years. How can we use code gen to help solve for app erosion? And so fortuitously, the name of this project is Scout, which ties in so well to the Grand Canyon experience that I've just been through.

As any good product manager, we start with discovery. We did some dogfooding, so I shipped some app fixes. We shadowed engineers and support team members as they were going through the app fix process. We designed out what are the pain points experienced along the way, what are the phases of the work, and how much time is spent.

One big discovery we had is how much time is spent gathering the context, going to the third-party API docs, even crawling the internet looking for information about a bug that's emerging. Maybe somebody else has already discovered and solved for it outside of Zapier.

Internal context, logs, all of this is a lot of context to go and search for as a human, and a lot to grok and work through. This is something we knew we needed to solve for.

Where we started with all this great opportunities and pain points is we started building APIs that we believed would solve for these individual pain points. And some of these APIs are using LLMs for our diagnosis tool, gathering all that context on behalf of the support person, engineer, and curating that context, building a diagnosis that's using an LLM.

And then some aren't. Like we have a unit test generator, but the test case finder is simply using a search query to look for theright test cases to pull in for your unit test. We built a bunch of APIs.

We had a bunch of great ideas. So there was a lot for us to test with, but we ran into some challenges in this first phase. We had APIs, but they were not embedded into our engineers' process. So our tool, I just said, they don't like to go to so many web pages to find all their context.

They would love all this information to come to them. And yet our web interface, where we've created a playground we call Auto Code internally, where you can come and play around with our APIs. And our ask to the teams was, come try out our APIs and give us feedback.

Now, this is just one more window to go to, so we didn't get a lot of engagement. Also, because we had shipped so many APIs, our team was spread pretty thin. Cursor launched at the same time, which has gotten great adoption at Zapier.

We're all huge fans of Cursor. But from our side, it made some of our tools no longer necessary. But there was one major win in this phase, which is one of our APIs became a support darling. It's Diagnosis, that number one pain point of needing to go out and find all of your context, curate it for yourself so you can start solving the problem.

We were doing that on the support team's behalf with the Diagnosis API.

And support loved it enough that they decided to embed it into their process. They asked us to build a Zapier integration on our Auto Code APIs so they could embed it into their Zap. That creates the Jira ticket from the support issue.

And now Diagnosis is included.

So embedding tools is the key to usage, as we find out. So how can we embed more of our tools? Well, then MCP spins up, and that solves our problem. We can now embed these API tools into our engineers' workflow.

Specifically, our engineers are pulling in these MCP tools as they're using Cursor. Our builders using Scout MCP tools are leaving the IDE less, spending more time in one window. Still coming into challenges, one of our key tools, Diagnosis, is so valuable to pull all that context and to provide a recommendation, but it takes a long time to run.

Now, we might run down that runtime. However, as you're working synchronously on a ticket in your IDE, this was frustrating. We also weren't keeping up with the customization needs. Not only did MCP launch, and we started leveraging it, Zapier MCP launched too.

And some of our tools, if we weren't keeping up with the customization needs, our engineers internally looked to Zapier MCP, which is great. We're all on the same team, solving the same problem, but some of our tools had a dead end.

Also, adoption was scattered. We had a whole suite of tools, and we thought there was value in each of them as it solves for different problems across the different stages. Not every engineer was using our tools, and if they were using our tools, they're only using a few of them.

So we have tool usage. We're happy about that. But we were under the hypothesis that true value is going to come from tying these tools together. So what if we owned orchestration of these tools? Rather than saying, here's a suite of tools, you use them as you wish, what if we combined them and created an agent to orchestrate this?

So this we are calling Scout Agent. We take that Diagnosis, run that against a ticket, use that information to actually spin up a code gen tool, which will then produce a merge request using all theright context. So who would benefit the most from orchestration?

There are several integration teams at Zapier who are solving for these app fixes of various levels of complexity, and there's the support team. So when we're saying, who should be our first customer of Scout Agent, we were thinking, hmm, it should probably be the team fielding small bugs that are emergent and coming hot off the queue, which is the support team.

And now our two experiments merge. And we have Scout Agent we are building for the support team. And this is the flow of how it works. Support is submitting an issue to Scout Agent. We first categorize the issue.

We next assess its fixability. Not every issue that comes from support can be fixed. If we think it's fixable, we'll move on to generating a merge request. At that point, the support team, this is the first time they're picking up the ticket, it already has a merge request attached to it.

They'll review and test. If it's not satisfying what they believe is the actual solution or what the solution should be to best address the customer's need, they will make a request for an adjustment. That can happenright in GitLab, which is where we do our work.

And Scout will do another pass. And hopefully, at that point, we've gotten itright, and support can submit that MR for review from engineering. How we are running Scout, it's all kicked off by a Zap. This is a picture of one of our Zaps.

There are many Zaps that run this whole process, and it embedsright into our support team's Zaps. We do a ton of dogfooding at Zapier. We first run Diagnosis and post that result to the Jira ticket, saying what the categorization is, if we believe it's fixable.

And then if we do believe it's fixable, we then are kicking off a GitLab CI/CD pipeline. And we run three phases in that pipeline: plan, execute, and validate, generate this merge request. The tools used in this pipeline are Scout MCP.

So all those APIs we invested in a year ago now are really coming together, and we're orchestrating it within the GitLab pipeline. And we're also leveraging Cursor SDK. Once the merge request has been completed, we attach it to Jira, and support picks it up.

The latest addition to this is doing a rapid iteration

once a ticket has been posted with the merge request and the support team is looking at it. And they say, you know, it needs some tweaks to save them more time so they don't have to go pull that down to their IDE, do the fixes, and push it back up.

They can simply chat with the Scout Agent in GitLab. That'll kick off another pipeline, which does that phase with that new feedback and posts the new merge request. On our side, we want to make sure Scout Agent is working.

So we ask three questions. Is the categorizationright? Was it actually fixable? And was the code fix accurate? So far, we have two evals, 70% to 75% accuracy for categorization and fixability. As we get more feedback and process more tickets, those become our test cases, and we can move forward improving Scout Agent over time.

So what has been Scout Agent's impact on app erosion? 40% of support team's app fixes are being generated by Scout. So we're doing more of the work on behalf of the support team. This is resulting in, for some of our support team, it's doubling their velocity from one to two tickets per week, which already is amazing.

That's going from a support team that wasn't shipping any fixes, well, unofficially they were sometimes, to now shipping one to two per week per person, to now shipping three to four with the help of Scout. Another process improvement, Scout puts potentially fixable ticketsright there in the triage flow, takes away a lot of the friction of looking for something to grab from the backlog.

It's not just the support who's benefiting. It's also engineering. Engineering managers said it's a great example of when it works. This tool allows us to stay focused on the more complex stuff. If you take away anything from this talk, I hope it is that there is a really powerful magic between support and empowering them with code gen and allowing them to ship fixes.

Because they have three superpowers. The first, they are the closest to customer pain, which means they're closest to the context that really matters for figuring out what's the problem and how to solve it. They're also troubleshooting in real time.

These tickets aren't stale. The context is fresh. The logs aren't missing. You put this ticket into an engineering backlog months later, you might not get access to those logs anymore. And then three, they're best at validation.

Again, you put the same ticket into an engineering backlog. The solution an engineer might come up with may change the behavior, and that might be good for some customers, but it might not necessarily be best for that one customer who wrote in about the problem.

And one other major benefit of this is our support team members who have been part of this experiment are now engineers.

I want to say thank you to the amazing team who's helped build this process or build all the tools and the Scout Agent. Andy is actually here in the audience. So shout out to Andy. If you want to talk about any of the technical bits, he's here.

And I want to impress upon you two things. We're hiring, but mostly, if you haven't rafted through the Grand Canyon, please consider it. It's life-changing, and you should go with Orrs. Thank you very much.