The Code AI Maturity Model and What It Means For You: Ado Kukic

Hello, and welcome to my talk on how we're thinking about the levels of Code AI. My name is Ado Kukic, and I'm the Director of DevRel at Sourcegraph. At Sourcegraph, we're building Cody, the only AI coding assistant that knows your entire codebase.

To help educate our customers and users, as well as shape our thinking of Code AI, we've been using a concept that we call "levels of Code AI" internally. These levels have really resonated with our community, so we wanted to publicize them and start a conversation with the broader developer community, and where better to do it than at the AI Engineer Summit.

When we talk about Code AI, we refer to software that builds software. Today, 92% of developers are using Code AI tools, whereas this number was just 1% a year ago. Our founder and CEO, Quinn Slack, has shared a bold prediction that in 5 years, 99% of code will be written by AI.

While we await that future, let's talk about how we see the levels of Code AI today. We see 6 distinct levels across 3 different categories: human-initiated, where humans are the primary coders; AI-initiated, where AI starts to take a proactive role in software development; and AI-led code, where AI has full autonomy over a codebase.

We'll contrast these levels of code with the SAE levels of autonomy for vehicles. Let's dive in. At level 0, the developer writes all code manually without any AI assistance. The developer is responsible for writing, testing, and debugging a codebase.

AI does not generate or modify any part of the codebase, but IDE features like symbol name, completion, can provide a bit of assistance. This level reflects the traditional software development process before introducing any AI assistance into the development workflow.

A vehicle operating at level 0 is fully reliant on the human driver for acceleration, steering, braking, and everything in between. At level 1, the developer begins to use AI that can generate single lines or whole blocks of code based on developer intent.

For example, a developer might write the signature of a function, and the AI will infer the context and generate the implementation details for said function. At level 1, the AI assistant has been trained on millions of lines of open-source code and can leverage this to provide superior completions based on the developer's guidance.

SAE level 1 vehicles still require the full attention of the human driver, but offer features such as cruise control or lane centering that make driving an easier, safer, and more comfortable experience.

At level 2, the AI coding assistant has superior understanding and context of the codebase it is interacting with. Where at level 1 the context is broad and general, a level 2 AI coding assistant has specific context about the codebase that it is working in.

This allows the AI assistant to make better suggestions for code completions. For example, if you are working in a Node.js codebase and we're using the Axiom library to handle HTTP requests, a level 2 AI assistant would provide auto-complete suggestions based on the Axiom library as opposed to a different Node HTTP library like Fetch or Superagent.

At SAE level 2, we get partial automation. The human driver is still in control and can override anything the car does at any time, but features such as traffic-aware cruise control or automatic lane changes can make driving a much smoother experience.

At level 3, the developer provides high-level requirements, and the AI assistant delivers a codebased solution. The AI coding assistant goes beyond generating singular snippets of code to building out full components and even integrations with other pieces of software.

Rather than writing the code themselves, a developer could instruct a level 3 Code AI assistant to add a user authentication to an application that they are building, and the coding assistant would generate all of the code required. The coding assistant could then explain to the developer the code it wrote, how it works, and how it integrates with the rest of the application.

SAE level 3 is also the first level where the vehicle itself takes on the primary role of driving, with the human driver being a fallback in case the vehicle cannot drive itself safely. The vehicle can perform most of the driving tasks but may encounter situations where it cannot adequately perform these tasks, so it's forced to give control back to the human driver.

At level 4, the Code AI assistant can proactively handle coding tasks without developer oversight. Let's imagine a few scenarios where a level 4 Code AI assistant would play a role. A level 4 capable Code AI assistant could continuously monitor your code changes and autonomously submit PRs to ensure your documentation stays up to date.

Even better, the coding assistant could monitor bug reports from customers and submit PRs to fix the issues. The human developer could then simply review the pull requests and merge them. Level 4 SAE vehicles can perform virtually all driving tasks under specific conditions.

For example, Waymo operates a fleet of fully automated self-driving taxis in cities where they have high-quality mapping data and can provide a safe driving experience for passengers without human drivers. A customer simply hails a Waymo taxi using a mobile app, provides a destination, and the vehicle is responsible for taking the passenger to their final destination without any additional human input.

At level 5, the AI assistant requires minimal human guidance on code generation and is capable of handling the entire software development lifecycle. The developer provides high-level requirements and specifications. The AI then designs the architecture, writes production-quality code, handles deployment, and continuously improves the codebase.

The developer's role is to validate that the end product meets the stated requirements, but the developer does not necessarily look at the generated code. The Code AI assistant has complete autonomy to take code from concept to production. A self-driving car capable of level 5 driving automation can perform all driving tasks under all conditions, humans optional.

The car is responsible for making all the decisions. At this level, a steering wheel or any ability for a human to override the car is unnecessary. So there you have it, the 6 levels of Code AI, or at least how we're thinking about them at Sourcegraph.

Do you agree? Disagree? We'd love to hear your thoughts. Find us at booth G5 and let's chat. And if you'd like to try Cody for yourself, get it for your IDE of choice at cody.dev. Thank you, and I'll see you on the show floor.