How Coding Agents change Software Development Forever - Hailong Zhang

Hi, everyone. In this presentation I'm going to talk about coding agents and how to work with coding agents. This is my agenda. First, I'm going to talk about the future of software development workflow, and then I'm going to show you an example of a coding agent, and this agent will boost the unit test.

And finally, I'm going to dive deeper into how we solved real-life problems with agents, how we build the agents.

Generative AI has shifted a lot of things, and people believe AI coding is going to be the future. And we believe a lot of the routine works will be handled by AI, including entry-level coding tasks, debugging, testing, and operating.

But still, I think the creative works, including product design, architecture design, and difficult issues, still need to be handled by humans. So it must be a human and AI agents collaborate together to solve problems in the same workflow in the future.

The collaboration has two types. First is synchronous, and another one is asynchronous. I think people may be pretty familiar with the synchronous one, GitHub Copilot or Cursor. This is AI live inside your IDE, and this AI works simultaneously with human.

When you're typing, it's working,right? So we have already had this kind of product for a long time, since 2020, and it's getting mature since 2023 and rapidly grows in 2024, and it's widely adopted today. But the asynchronous one is pretty new, and it just started in 2024.

The asynchronous one is more like a bot inside your workflow, for example, GitHub Bot. And this bot can be triggered either manually or automatically. And this bot will complete the task without humans' attention, fully autonomous, and will submit a deliverable once it's done.

So it's totally different experience, and these two types of agents are all important. And I think people need all these kinds of agents to solve the real-life problems.

Imagine this is your workflow in the future, and the workflow will have a lot of AI agents, small agents living inside your repos, workflows, in different stages, to solve different problems, including unit tests, including fixing bugs, writing documents, submitting code reviews, and releasing from these kinds of tasks.

And so human can focus on the more creative part.

Next, I'm going to show you a detailed coding agent, which will boost unit test. I think unit test is more important today than before, because in the AI era, a lot of code is generated by AI. As we just talked, if you're using Cursor, you must be familiar with tab.

For each of the tab, multiple lines of code in multiple parts of the file will be generated. And it's very hard for you to focus on every generation, and you may overlook something. The coding speed is really, really fast, but it's also easy to generate bugs.

So one of the ways to solve the issue, or prevent the issue, is to write a unit test. People talk about unit test. People think unit test is important. But when writing unit test, people hate it. Yeah, developers, every developer hates to write unit test.

So we built a Gru, an AI agent, to help developers to write a unit test and managing unit test.

This is a screenshot that pull request of unit test generated by Gru. And the process is triggered either manually or automatically by a pull request. When humans submit a pull request, Gru will detect the change of the code and determine if this change needs more unit test or change existing unit test.

And Gru will do the coding work, writing the test, and to run the test to make sure everything works, and then prepare and submit a pull request, including all the information showing in this screenshot, the summary of the test, and the coverage improvement of the test.

Now, human needs to review this pull request and determine if this unit test is good or not to merge into the repo.

Let's look at the performance of Gru in production. This is also a screenshot from GitHub. It's a list of pull requests. And as you can see, over 50% of the pull requests are merged and accepted by humans. So 50% is not a very large figure, and there are a lot of rooms for improvement.

But I think in real life, 50% is already meaningful in production and helps solve a lot of problems.

And also, Gru handles around 80% of the unit test itself in its own repo. We are using Gru every day, and most of our tests are generated by Gru. From the commits you can see, Gru is already in the counts of commits.

Gru is already the first contributor in our team. And we believe more and more agents will become contributors in people's repo this year, 2025.

Next, I'm going to dive a little deeper into how we build Gru to solve real-life problems.

To build an agent, I think the first and the most important thing is to define the problem itself. A clear, concrete, and doable problem is crucial. For example, unit test is a problem, but software engineering is not a problem.

And once we have the problem, we need to build the data sets for the evaluation purpose and also the evaluation harness. And then we work on LLMs, building context. And last, we need agent OS or framework to orchestrate everything together.

As we all know, agents work on top of LLMs, and Gru works on all frontier models, either from OpenAI, Anthropic, DeepSeek, Google, etc. And we evaluate the models on the different scenarios, try to find the best model for each of the scenarios.

That means even within the same job, Gru may use different LLMs for different stages. And also, we fine-tune models to improve the performance. For example, we fine-tune GPT-4o with human-labeled unit test code to improve the generation of the test code.

Building context in agents is really important. And we think it's necessary to build the context for specific tasks. For example, here in unit test, we build the context for each of the languages and the frameworks. And we also gather all the information from the environment.

For example, the GitHub issues, code reviews, commits, pull requests, README, code itself. And all this information needs to be gathered together and to be filtered and to be selected to fit into the context.

We talk about unit test a lot, but our vision is not only unit test. We want to build agents for different software engineering tasks. For example, the refactor task, the E2E test task. It is almost impossible for us to build every agent from scratch.

So we build the framework, or we can call it agent operating system, agent OS. Different tasks may share similar runtime, similar tools, similar context. So we abstract the common infrastructure. And this operating system will enable us to build agents in this domain in a really fast pace.

The agent era is coming. Let's embrace agents in workflow. Thanks for watching.