The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp

My name is Beyang. I'm the CTO and co-founder of a company called Sourcegraph. We build developer tools, and today I want to share with you some of the observations and insights that we've had on this sort of emerging skill set of how to wield coding agents.

That sound good to everyone? Allright, cool. OK, so let's check in on the agent discourse. I don't know if you all saw this, but a couple of days ago there were some spicy tweets about the efficacy of AI coding agents, or inefficacy depending on your perspective.

So Jonathan Blow, who's a really talented developer, he basically single-handedly coded up the indie game Braid, if you're familiar with that. So he's kind of like god-tier status in terms of coding ability. Retweeted Alex Albert, who's also someone I respect and admire a lot, works at Anthropic, basically claiming that all the hype around coding agents and code AI in general was just hype,right?

There's no substance there. And then there were some responses, and there was kind of a spectrum of responses too. We had some other big names in the developer world, like Jessie Frazelle. She was one of the early contributors and maintainers of Docker.

She's also really legit. She said basically something to the effect of, like, I think you'reright, but you're in the top 0.01% of programmers, Jonathan, for the rest of us down here in this room that aren't on Mount Olympus.

It actually helps a lot, but not super helpful if you're really, really good. But then we also had folks like Eric S. Raymond, who's like one of the fathers of open source, who had a very spicy reply. He's basically like, look, I consider myself to be pretty decent at programming, and these things help a lot.

And then the kind of my favorite one of this was actually the Top Hacker news post that was written by

Thomas Tachek, who is a really legit security engineer. Some of you may have seen this trending. He's basically taking the opposite view of, like, there's some really smart people there who are very AI skeptical, but they're nuts. Like, these things are really useful.

So I'm guessing if you're at this conference, you probably lean toward coding agents are substantively useful, and there's something there. I don't know, just a guess. But I think even within this room, there's probably a spectrum of best practices and opinions about where agents are good, whether they're restricted to small edits or front-end applications or weekend vibe coding, whether they actually work on your production codebase.

And I think this is just indicative of the dynamic technical landscape that we're inright now. And a couple months back, I read this blog post from this guy, Jeff Huntley. So Jeff was a senior engineer at Canva at the time, and his role at Canva is really interesting.

He basically went around interviewing all the developers inside of Canva using AI tools like Cursor and other things and seeing how they're using it. And he basically came to the conclusion that most people were holding it wrong, which is really interesting.

And he came up with a blog post about all the different anti-patterns that he was seeing. But my summation of that blog post is, like, the number one mistake that people are using with coding agentsright now is they're trying to use coding agents the same way they were using AI coding tools six months ago, and therefore they're wrong, which is kind of crazy because normally if you're using a tool, the best practices don't change in six months.

Typically, the things that you learn that are good will still be present and topical and relevant six months down the line. But I think we're in a really interesting moment in timeright now. And why the sudden change? I think it's because of this step function transition that we've experienced in model capabilities in the past six months.

So we've all been around since the dawn of generative AI, the ancient year of 2022,right? November 2022 was when ChatGPT launched,right? And every year now, this is now the year three, three after ChatGPT,right? We're now living in the AI future.

But I think there's already been kind of like three distinct waves or eras, largely driven by the evolution of frontier model capabilities. And the model capabilities really dictate the ideal architecture that becomes dominant at the application layer. So in the GPT-3 era, all the models were text completion models, which meant all the applications that people were building were these like copilots or autocomplete tools.

So the dominant UX paradigm was like, you type some stuff, it types some stuff, you type some more, and that's how you would interact. And then ChatGPT came along with GPT-3.5, which was instruct-tuned to interact like a chatbot.

And suddenly people realized, like, oh, it's not just completing the next thing I'm talking about. I can actually ask it questions like I can a human now. And then some other people came along. We were part of this crowd.

We realized, like, hey, you know what's even better than just asking it questions? You can actually copy-paste stuff into the chat and say, like, here's some code from my codebase. Use that as an example and pattern match against that.

And that helps it generate a little bit better code or less fake code or less hallucinated code than it did before. And that basically meant that everyone at the application layer was building a ragbot in 2023, so like a chatbot plus a rag retrieval engine.

But now I think we've entered a new era. And I'm not sure if everyone realizes it, or maybe this is I don't know. Who agrees with this statement? Who thinks it's a real paradigm shift? OK, and then who here is like, oh, that's a bunch of bullshit?

Anyone? Feel free to I like OK, OK, so maybe I can just skip this slide.

So we're now living in the era of agents, and the new model capabilities really dictate a new application architecture. And so one of the things that we asked ourselves at Sourcegraph is, a lot of the existing tools in the market, they were designed for the era of GPT-4 and Claude 3.

So a lot of the application stuff, features, and UX and UI was really built around the capabilities, or in some cases, the limitations of the chat-based LLMs. And so if we're going to design a coding agent from the ground up to unleash the capabilities of tool-using LLMs, agentic LLMs, what would that look like?

OK, so here are my spicy takes. These are controversial design decisions that I think are better to make in the age of agents. And many of these go against the best practices that kind of emerged in the chatbot era.

OK, so number one is the agent should just make edits to your files. It shouldn't ask you at every turn, like, hey, I want to make this change. Should I apply it? If it's asking you and it's wrong, it's already done the wrong thing and has wasted your time.

Humans need to get more out of the inner loop and more kind of like on top of the loop, like still steering it and guiding it, but less micromanaging and managing every change. Second thing is, do we still need a thick client

to manipulate the LLMs? Do we still need a fork VS Code? That's like the salty way of saying this,right? The VS Code fork became the culmination of the AI coding application, I think, for the chatbot era. But there's this question of if the contract of an agent is you ask it to do stuff and then it does stuff, do you really still need all that UI built around context management and applying the proposed change in the codebase?

Or can you just ask it to do stuff and expect it to do theright thing? Third, I think we're going to move beyond the choose-your-own-model phase. So I think in the chatbot era, it was very easy to swap models in and out.

And you'd like, oh, a new model came along. Let me swap it out and see how well it attends to the context that my retrieval engine fetches. In the agentic world, there's a much deeper coupling because the LLM that you're using essentially becomes the brains of these agentic chains.

And so it's much harder to rip and replace. And I think a lot of people in this room who have tried mixing and matching different models in the context of agents have found that swapping out a different model and expecting similar results is very different.

A lot of the LLMs out there aren't even good at the basics of tool use yet. So it's very difficult to just replace the brains. Four is I think we're going to move past the era of fixed pricing.

Agents eat up a lot of tokens, and so they look expensive relative to chatbots. But the comparison that more and more people are making is how much human time is it saving? So they're still cheap relative to human time saved.

And the fixed pricing model actually introduces a perverse incentive now, where it's like selling gym memberships,right? If I sold you a membership to my chatbot and you're now paying me $20 a month, my incentive now is to push the inference costs as low as possible.

And the easiest way to do that is to use dumber models. But dumber models just waste more of your time.

Sorry, this is a long list. Hopefully it's not too tedious, but I think these are important points. The second and last point I'll make is I think the Unix philosophy is going to be more powerful here than vertical integration.

So in developer tools, the ability to use simple tools in ways that compose well with other interesting tools is really powerful. And so I think, especially with agents, where there's less of a need to create a lot of UI around it, you're going to start to see more command-driven tools, command-line tools, and things like that.

And then last but not least is we had an existing RAG chat coding assistant. Maybe some of you have used it. It was called Cody. It still exists. We're still supporting it. It's still in heavy use across many Fortune 500 companies.

But we decided to build a new application from the ground up for the agentic world because we didn't want to be constrained by all the assumptions and constraints that we built into the application layer for the previous generation of LLMs.

And one analogy I like to draw here is

the early days of the internet,right? In the early days of the internet, the way people jumped into the web was using an interface on the left. This is before most people knew what the internet was about, what it was capable of.

And that was theright interface for the first generation of the internet because what can you do with the internet? Well, there's a bunch of different things. You can look at trending celebrities. You can buy automobiles. You can look at movie reviews, all these things you might not have thought of.

And so it's useful to have in front of you. But at some point, it gets a little tedious, like clicking through all the different hyperlinks and navigating your way through. And then the real power of the web was sort of unleashed by just like the one simple text box where you just type what you're looking for and you get to it.

And I think with agentic UIs, that's what we should be striving for, both in developer tools and in a lot of different application paradigms. OK, so what does that look like in practice? So when we went to design this thing, our coding agent is called Amp.

And Amp has two clients. And this is what they look like. So both are like very, very bare bones. A lot of people look at this and like, what is this? It's just a text box. What can I do with it?

And that was by design for all the reasons I just mentioned. One client is just a simple VS Code extension that allows us to take advantage of some nice things that you get in VS Code, like being able to view diffs.

That's really important in the agentic coding world. I often joke that that's now I use that view more than the editor view now. And the second was a CLI. So just stripping things down to bare bones. It has access to all the same tools as the VS Code extension does, but it's just something that you can invoke in your command line.

You can also script it, compose it with other tools. OK, so what does this actually look like in practice? I want to do something a little bit risky here, which is in the past, I've done a lot of like, hey, here's me building a simple app, like those sorts of demos.

But I actually want to show off where we think this is most useful, which is like, hey, I'm working on an application that has real users. Let me actually make a contribution to that codebase with all the existing constraints.

And so I actually want to I'm just going to code a little bit. I don't even know how far we're going to get. But this is Amp. This is VS Code running Amp in the sidebar, and it's open to the Amp codebase.

And what I want to do is implement a simple change to this application. So the change that I'm going to make is Amp has a server component. And the server exists as a way to provide the LLM inference point.

It also provides team functionality. We have a way to share what different teams are doing or what different users are doing with AI so you can kind of learn from other users. There's a leaderboard. It's fun. But there's also these things called connectors, which allow Amp to talk to external services.

So our issue tracker is linear. And so I've integrated linear into Amp here. But I'm kind of annoyed because it's using this generic network icon. And I would really like to customize this icon such that when you plug in the linear MCP endpoint, it uses a more appropriate icon, like a checkbox or something issue-y.

So I've already filed this as a linear issue. And I'm just going to ask, can you find the linear issue about customizing the linear connector icon? Then implement it.

So what this will do is it has access to a set of tools. I can go over here to the tool panel and see what tools it has access to. Some are local. Some are built-in. It's got the standard tools, like read and edit file, run bash command.

You can also plug in things like Playwright and Postgres via MCP. And then linear is also plugged in through this. So we're basically talking to the linear API through the MCP server. And what this will do is it will use the linear issues API, and it will search at issues.

It found 50 issues. And the one that I was referring to is at the top here. So add a special icon for the linear connector. And now it's going to go and implement the thing for me. And one thing to note here is it's just making these tool calls on its own.

I'm not asking it to use specific tools. We've also tried to make the information that you see minimal. So you don't need to see all the API tool calls that it's making underneath the hood or crowd out the transcript with a bunch of things.

Most of the time, we just want to keep it simple because the contract we want to provide to users is like the feedback loops here are more robust, and you don't have to micromanage this as much. Another thing I want to point out here is the search tool that this is using is actually a sub-agent.

So it's actually spinning off a sub-agent loop that uses a form of agentic search. It has access to a bunch of different search tools: keyword search, regular grep, looking up file names. If you want to inspect what it's doing, you can click the expand thing and see what different paths it's taking, what files it's reading, what things it uncovered.

But again, by default, we think this is like an implementation detail, and hopefully it should just surface theright thing. So it's working. It's gathering context. Another thing I want to call out in this interface is as we've gotten more feedback, we've kind of designed this thing to be more multi-threaded.

So there's a quick keyboard shortcut that allows you to quickly tab through the different threads that you're running. And it's a common paradigm in our user community to be running more than one of these things at a time.

And it takes a little bit of use to get used to the context switching. Developers hate context switching,right? We like to be

in flow, in focus. Typically, what we see here is the secondary thread will either be something that's a lot shallower so that you can quickly page back to the main thread. Or what I like to do is while the agent is working, I actually like to understand the code at a deeper level myself so I can better understand what it's going to do.

So I could ask something like, can you show me how connectors and connections work in Amp? I can ask it to draw a picture of that. So we'll kick that thread off in parallel. We'll check back in on what this guy is doing.

So it's found it's read a bunch of files. It's read some front-end files. Our front-end is written in Svelte. And as you can see, it's being fairly thoughtful about reading the appropriate files before it actually goes and does the work.

And we find that this is really important to make the feedback cycles more robust. Otherwise, the anti-pattern is you just get into the weeds of steering it manually.

It's also got this to-do list thing at the bottom that helps it structure and plan out the longer-term tasks so that it doesn't go immediately dive into the code. This is a classic mistake that human developers make too, where you dive into the code too early, and then you get lost in the weeds, and then it takes a while to dig yourself out.

OK, so it's making some changes. One other thing that I like to point out here is I mentioned that I use the diff view in VS Code now, probably more than the editor view. VS Code actually has a really nice diff view.

I have a hot key so I can open it up quickly. And most of my time in VS Code now is spent just reviewing the changes it makes. And I actually like this a lot better than GitHub PRs or Git diff on the command line, just because it's in the editor.

You can see the whole file, and jump to definition even works.

So yeah, we'll just wait a little bit for it to do its thing. I actually think it's probably made looks like it's getting there. It's probably just running tests. Let's see if we go back here if it's updated the icon at all.

[Ladies and gentlemen, there are keynote presentations.]

OK, so it hasn't gotten there yet. But I think it's on theright track.

[Does it write its own tests?]

The question was, does it write its own tests? Yes, it typically writes its own tests. And if it doesn't, you can prompt it to do so. So it's doing a lot of things. It's reading a lot of files.

It's making these edits incrementally and then checking the diagnostics. And then now let's see if it works. OK, cool. So you see here the icon has been updated. And this is without me really steering it in any fashion.

Notice here on this page that this icon didn't update, though. And so this is actually not surprising to me because this change, as many changes in production codebases are, is often more nuanced than it seems at the surface.

So in this case, the reason it's not getting it here is because

this is the admin page. And the piece of data we need to know we need to read in order to tell that this is a linear MCP rather than a generic MCP is actually part of the config. We have to look at the endpoint of the MCP URL.

And in order to do that, you have to read the config. But the config might also contain secrets. It doesn't contain secrets in this case, but it might contain secrets in other cases. So we actually prohibit those secrets from being sent to non-admin pages.

So it's not surprising to me that the first pass, it didn't get thatright. But let's see if it can get I'll just nudge it a bit. So I notice that the icon changed on admin connections, but not on settings.

Can you investigate why?

And in the interest of time, we'll check back on this later. How about that? We'll let it run. We'll see if it can find its way to theright solution there.

Is it OK if I go a little bit over since we started a little bit? OK, cool. Is it OK with you all if I go a little bit over? OK, are you still having fun?

[OK.]

OK, cool. So that was like a brief demo of just the interaction patterns in the UI. We try to keep it really minimal. We've released this to a small group so far. The sign-up is now publicly open. It's been open for about two weeks.

But we haven't done a lot of marketing around it. And that's kind of been intentional because we're really trying to design this for where we think the puck is going. And so we've done a lot to curate this community of people who are trying to experiment with LLMs and figure out how the interaction paradigms are going to change over the next 6 to 12 months.

And so our user community is really people who are spending nights and weekends a lot of time with this thing to see what they can get it to do. And so actually, one of the most insightful things and actually the main topic of this talk is lessons that we've learned from just looking at what our power users are doing and seeing what interesting behavior patterns they're kind of implementing.

And so the average spend for agents is growing. It's a lot more than the average spend was for chatbots or autocomplete. But one other interesting thing that we've noticed among the user base is that there's a huge variance in terms of how much people use this thing, to the point where there's an upper echelon of users that are spending thousands of dollars per month just in inference costs.

And at first, we're like, this has got to be abuse,right? Someone out there has poked, found some way to exploit the inference endpoint, is using it to power some Chinese AI girlfriend or whatever. But actually, no. When we spoke to the people using it, we actually found that they were doing real things.

And we're like, hmm, that's interesting. What the hell are you doing? And from those insights and the conversations, we basically have encapsulated a series of best practices or emergent power user patterns for how the very dominant users, the most active users, are using this thing.

And this has informed our product design process as well. So one of the first changes that we made was we noticed that a lot of the power users were writing very long prompts. It was not like the simple kind of Google style, like three keywords and just read my mind and expect something good to happen.

They actually wanted to write a lot of detail because they realized that LLMs are actually quite programmable. If you give them a lot of context, they will follow those instructions and get further than if you just give them a one-line sentence.

And so we made the default behavior of the Enter key in the Amp input just newline. So you have to hit Command Enter to submit. And this throws a lot of the new users off because they're like, wait a minute.

Why isn't it just Enter? If I'm in Cursor or whatever, it's just Enter. That's easy. That's intuitive. But actually, what we want to push users to do is to write those longer prompts because that actually yields better results.

And I think that's one of the things that prevents people who are still in the kind of chat LLM mode from unlocking some of the cool stuff that agents can do.

Another thing that people do very intentionally is direct the agent to look at relevant context and feedback mechanisms. So context was very important in the chatbot era. It's still important in the agentic era. Now, agents do have a good amount of built-in knowledge for how to use tools to acquire context.

You saw that before when it was using the search tool to find different things. And it was executing the test and linters to see if the code was valid. But there's still some cases, especially in production codebases, where it's like, oh, we do things in a very specific way that are kind of out of distribution.

And so some

less agentically inclined users at that point will just give up. They're like, ah, agents aren't capable of working with back-end code yet. But what we've noticed is the power user is like, actually, let me try to just tell it how to run the build in this particular subdirectory, run the tests.

And that helps it complete the feedback loop so that it can get the validation to get further.

Feedback loops are going to be a big theme of this talk. So another dominant paradigm here is constructing these front-end feedback loops. So a really common formula is you have the Playwright MCP server, and then there's a thing called Storybook, which is basically a way to encapsulate or componentize a lot of your front-end components.

It makes it very easy to test individual components without loading your entire app. And you probably should have been doing this anyways as a human developer because you get a fast feedback loop. You make a change, see it reflected instantly.

You get the auto-reload and then go back to your editor. But with agents, you kind of notice it more because you're no longer in the weeds doing the thing. You're like, oh, you're almost like the developer experience engineer for your agent.

It's like, how can I make it loop back faster? And so what the agent will do is make the code change, use Playwright to open up the page in the browser, snapshot it, and then loop back on itself.

And it does that via Storybook because it's much faster than reloading the entire app.

[You put Playwright as a tool for you.]

Yes. So it's one of the default recommended tools. So it'sright here. And actually, it looks like that run completed. I wonder if

it looks like it did approximately theright thing. Sorry, just to jump out of the slides with her a little bit. So now you can see the icon is customized on the settings page, not just the admin page. And if you look at how it did that, I think it did theright thing.

So if we look at the diff, it actually looked at the surrounding code and was like, oh, there is an existing mechanism for plumbing non-secret parts of the config through to the UI. Let me kind of use that as a reference point.

And it actually plumbed exactly that field through to the front end. So now if I add additional fields to the MCP config that do contain secrets, this is whitelisted. So it'll still only send the endpoint URL over to the client, basically what it needs to make that icon customization.

So yeah, it's not a super impressively visual change. But a lot of such changes in messy production codebases are like that. And it's cool to see the agent be able to tease out that nuance.

OK, I know we're a little bit over time. People mind if I keep going? OK, cool.

[Take it away.]

There's some additional tips and tricks. Most of this talk is just sharing what we've learned from our power users. So another thing that we've noticed is there's this kind of prevailing narrative that agents are going to make programmers lazy.

It's going to make us so we don't really understand what's going on in the code. So we're going to ship more slop. But we've actually found the inverse happen with the power users. They're actually using agents to better understand the code.

And so this is a really good onboarding tool. We just hired this guy, Tyler Bruno. He's a very precocious young developer. He's actually still in college, but he's working full time in addition to taking classes. So really bright, but also a bit green.

He's been using Amp to just quickly ramp up on how the different pieces connect together. And it can draw diagrams and point you to specific pieces of the code. And it's really good at accelerating that. And then a corollary to this is we all do a form of onboarding to new code whenever we do a code review.

By definition, code review is new code. And oftentimes, it's new code that contains bugs or is hard to understand or is a bit of a slog. And so rather than just ignore the code that the AI generates and just commit it blindly, we find that our user base is actually using this tool to do more thorough code reviews.

So I've adopted this practice myself, where if I have to review a very large diff, the first thing I do is ask the agent to consume the diff and generate a high-level summary so I can have a high-level awareness.

And then I ask it, hey, if you were a smart senior dev, what's the entry point into this PR? Because often, half the battle is just finding theright entry point. And psychologically, I often put off code reviews because I'm like, oh, it's going to be a pain.

And it's going to take forever just to figure out where I should start reviewing it. So I'll just do it tomorrow. But this thing, it helps lower that activation energy and make code reviews more thorough and actually, dare I say, a little bit fun and enjoyable now.

Sub-agents are also a thing. So we implemented the search tool as a sub-agent in the very beginning. But we're seeing more and more use cases emerge for sub-agents. And the general best practice with sub-agents is that they often are useful for longer, more complex tasks because the sub-agent allows you to essentially preserve the context window.

So the quality of the LLM will degrade over time. Sonnet 4 has a context window of 200K. But we see degradation typically around 120 or 130K. And by the time you hit 170 tokens, you start to see more kind of off-the-rails and crazy behavior.

But sub-agents allow you to encapsulate the context used up by a specific subtask, like implementing a small feature such that it doesn't pollute the main agent. OK, so that was a quick tour of a lot of best practices.

Just to recap the anti-practices, the common anti-patterns are just micromanaging the agent, using it like you would a chatbot where you have to kind of steer it at every interaction or review every edit it's making. Another common anti-pattern is just under-prompting, so not giving it enough detail.

LLMs, their knowledge comes from two places. It either comes from their training data or from the context that you give it. And so it's fine if you do a five-word prompt if you're coding up a 3D flappy bird game from scratch because that's well represented in the training set.

They're really good at that. They're trained to do that. But if you're trying to make a subtle, nuanced change to your large existing codebase, you should be giving it all the details that you would give a colleague on the team to point them in theright direction.

And then last but not least, agents are not a vehicle to TL;DR the code. If anything, they're the opposite. You should be using them to do much more thorough code reviews more quickly. The human is still you're ultimately responsible for the code that you ship.

And you shouldn't view this as a human replacement. It's really a tool that you can wield to make yourself 10, 100x more effective. Last tidbit. So one of the things that we've noticed among the very, very, very top 1% of the 1% is

this inclination to run multiple of these things in parallel. So Jeff Huntley, who wrote that blog post that I showed earlier, he started putting out these Twitter streams that are about four hours long each. And it's basically just he's working on a compiler on the side.

And what he does is he constructs prompts for three or four different agents to work on different parts of the compiler. And he's gotten to the point where he's prompting it such that he feels confident enough in the feedback loops where he just hits Enter, lets them run, and then he goes to sleep.

And then this thing just runs on Twitter for a while. And I think he's doing this to kind of spread the word. It's like, hey, you can use this for serious engineering. Compilers are not some vibe-coded weekend project.

They're real tech. They're difficult to build. And it is possible to use agents for code like this. But it has to be a very intentional skill that you practice. And so I think it's cool. I think there's a lot of people thinking in terms of agent fleets and where the world is going.

But I do think that the way that we'll actually get there is by building these composable building blocks that allow people like Jeff to go and combine them and come up with interesting UIs. I think this is just running in TMux or some window manager.

OK, so the takeaways I just want to leave you with is, one, contrary to what some might say and look, there's a lot of smart senior developers out there who think AI is overhyped and maybe parts of it are.

But I think coding agents are very real. And it is, I think, a high-ceiling skill. It's like, I think we will probably invest in learning how to use these things in the same way that we invest in learning how best to use our editor or our programming language of choice.

And I think the only way you can learn this stuff is by doing it and then sharing it out with others. And one of the reasons we built the kind of thread-sharing mechanism into Amp is to help encourage knowledge dissemination so that if you discover an interesting way of using it, you can share that out with your team.

But yeah, that's it. If you want to kind of see a recap of the best practices in this talk, we've actually put out an Amp Owner's Manual that guides new users how to best use it. I'll also be around afterwards.

We have a booth in the main expo hall. I'm supposed to say, too, if you stop by the booth, we'll give you $10 in free credits. So if anything you saw here was of interest to you and you want to try this out, stop by and say hi.

[I noticed you still type a "can you" and then you correct your typos, which I guess you said you shouldn't do.

Yeah.

Is it habit?

It's part habit and it's part paranoia that in a live demo setting, there will be some typo token that will trigger off-the-rails behavior. But I think that was more of a concern that I learned in 2023 when it actually mattered.

Because these days, LLMs are more and more typo-robust, I would say.