Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft

Okay. Hi everyone. Um, my name is Marlene, and I am a Senior Developer Advocate at both Microsoft and GitHub. So I work at— I work in a group called Core AI, which looks at how developers are using AI across our products.

Um, so this is kind of new. To start off today, I wanted to show you some stats about GitHub from GitHub Octoverse, uh, last year's GitHub Octoverse report in 2025, which shows data about how developers are using GitHub.

What we saw from our report was that more code was added to GitHub last year than ever before. So about a billion commits were pushed to the platform in 2025, which is GitHub's most active year ever. Okay? What we know now, in 2026, is that this growth is accelerating.

So a couple of— we haven't actually released any official stats yet, but a couple of days ago our COO, Kyle Dagle, tweeted that we're seeing about 275 million commits to the platform every week. And if we extrapolate that over time, we're going to see about 14 billion commits by the end of the year.

So that's 14 times the amount of growth we saw last year, or commits we saw last year, which, by the way, last year again was our biggest year ever with a billion commits. One thing that we know is that there's a growing share of these commits that are co-authored by AI agents.

We haven't released the data yet, but we can actually track and see, you know, some Claude, for example, co-signs commits and so does Copilot, but Codex doesn't, but we can also kind of track based off of, uh, some wording in the code.

I had a question when I saw all of this growth in terms of how much code we were seeing, and that question is: does AI actually make developers more productive? We're seeing all of this code; does it actually correlate with productivity?

One of the best resources I've seen that tries to answer this question is actually from AI Engineer, from a talk last year. And this graph shares findings from that talk that's from a Stanford University study of 120,000 developers.

And in this study, um, it found that while yes, AI can make developers more productive, it's actually how the developers are using AI that matters the most. So this graph from the study actually shows us that clean codebases amplify AI gains and AI productivity, while unchecked AI in a codebase is going to amplify entropy.

To illustrate this point, the speaker from this talk gave a case study example of a company that used AI in an unchecked way in their database. And what you can see is that the number of PRs that the team was pushing out increased, but at the same time, the code quality that the team was seeing decreased, and actually they spent a lot more time reworking that code, refactoring that code.

And so overall, though there was effective output increase of like 1%, AI didn't really improve the productivity from this team.

So what we learned from this study is that a lot of value that we are wanting to see as developers from AI relies or hinges on us having a clean codebase. So for developers that are using AI tools, we want to focus on things like good test coverage, type coverage, and things like good documentation, modularity, and so on.

So I'd actually also argue that we need to start standardizing some practices across our teams and across our industry. And this is something that's a bit of a controversial topic because some people at this conference believe in just closing their eyes and shipping, and that's also okay.

But I think, in my ideal world and from the study we've seen, I would recommend standardized practices for keeping a codebase clean. So how can developers maintain create and maintain clean code? This question is actually not a new question.

In our industry, over time, we have seen several methods that have tried to make, uh, maintaining a clean codebase a central part of their philosophy. One of those approaches that I've actually seen a lot of developers that are doing, uh, agentic coding with coding agents talking about is test-driven development, or TDD.

Uh, Simon Willison, who's very popular, just recently published a blog post about how he's using this specific flavor of TDD called Red-Green-TDD. And here, what happens is a developer the first thing that happens is the developer gets an incoming feature request.

As soon as they get the request, they immediately start by writing a failing test because the feature doesn't exist. After that, the developer focuses on getting the test to pass. And in this green phase, when they're trying to get the test to pass, historically, you should not be focusing on the quality of the code.

All you're focusing on is speed and getting the test to go green. So in the past, developers maybe would copy code from, uh, Stack Overflow and so on and get the test passed. But then after that, the final phase of this is the refactor phase.

And in this phase, you're just focusing on code quality. So you're taking that code that you made pass and refactoring it so that it follows all the best practices.

So not everyone is a fan of TDD. And like many things in this industry, TDD was pronounced dead in 2014. Um, and one of the most common complaints that I've seen on the internet about TDD is that it focuses too much on code coverage with unit tests and that it doesn't actually test the system.

So DHH, who created Rails, published this blog post in 2014 and was kind of talking about this, that is, an overfocus on unit tests. And we know that when we over-index on code coverage, there are several issues that come up.

One of the issues is that there's a tendency to test implementation details. So, um, take an example like we see on the screen where we are having an order calculation with a discount. If the test is tied directly to a method like calculate, just simply renaming the name of the test, even if the functionality is still fine, is going to break those unit tests.

So that's not going to be great. But even if we test specifically, uh, the behavior of the system, like the final end result of the price we're looking for, um, or we test on something like a stable contract, like our API or a module that doesn't change but we export it, it should survive any refactors of our internal code.

I would say that if you're interested in learning more about this and behavior-driven, uh, TDD, I would recommend the talk by Ian Cooper called TDD Where It All Went Wrong. It's a very good talk. Another thing that we see is that in the age of AI, that, uh, many developers are using AI to generate tests.

What they've noticed is that AI sometimes generates self-affirming tests. So while the code coverage test might pass and your unit test suite is all green, the behavior of the system is not being validated, and that's where the problem lies.

So for the rest of this talk, I'm going to be focusing on showing you how you can avoid these problems and start to test for functionality using Playwright. Playwright is an open-source testing framework that's built by Microsoft, and it automates end-to-end testing in the browser by simulating user interactions.

And the link that you see on the screen there is, uh, is the documentation. So Playwright supports a number of different, uh, languagesright now: Python, TypeScript, C#. And the example script that you can see on the screen is what a test would typically look like.

So you have that line that says, "Page go to," telling the, um, the script that it starts at the toy's page, is where we want to start. And then we're going to look for the placeholder search. And then we're going to fill that search bar with that, uh, let with the word Furby.

And that will actually do run the search for us automatically in the browser, for example. You can also use headed or headless mode. So you don't necessarily have to look at the browser while your tests are running. You can actually just have them running in the background as well.

Um, why? Okay. So going back to that idea of TDD, when we're using Playwright with AI, it actually should speed up the full process of TDD for us. So a lot of developers in the past have really complained about how TDD is slow and that it's not effective for teams that want to move fast.

But if we have AI, then what happens is that red part and the green part are fast. So we're focusing on getting our agents to generate these behavioral tests, the Playwright tests. Then we're focusing on getting the agent to quickly generate, as fast as it can, code that's going to make the test pass.

And then I would recommend that developers are going to spend the most amount of time, so it grows bigger, on that refactoring stage. So they're spending time looking at the code the agent has generated and making that code better.

There's a number of ways you can connect your coding agents today with Playwright. One of those ways is through the Playwright MCP server. You can use the CLI tool if you'd like that instead. Or you can use something, uh, called Playwright Agents.

And when you're using Playwright Agents, you'll run the command that you can see on the screen. And once you run that command, it's going to install for you, um, three agent.md files. So the first one is going to be a planner, second is a generator, and the third is a healer.

So the planner will plan which tests to run. The, uh, generator is going to actually generate the tests, and then the healer will fix those tests for you. Okay.

So I do want to show you a demo. And I am going to hope the demo gods are smiling today. Uh, so we will we'll give this a try. Oh, oh no. Okay. Here we go. So I want to give us a scenario.

The scenario is that, oh, you can't see my you are only looking at my PowerPointright now, and I don't know how to stop that. Uh, let me close the PowerPoint, maybe, and see if that will help.

Um, I know I want to just show the screen. Sorry.

Hopefully they'll give me more time. Okay. Perfect. That's working as expected. Okay. Perfect. So the scenario that we're going to imagine today is imagine I'm a developer. I'm working at a toy company called Tailspin Toys. And a few days ago, I got an email from the search product management team, and they asked me to add some new search and filter features to the site.

They asked me to add in a search bar with tick search for simple searches and Azure AI search for more complex ones. And they've also asked me to add in a sidebar so customers can filter by category and price.

So I'd like for Copilot to help me with this task and also for us to use this, uh, TDD-first style of development. So this is GitHub Copilot CLI. And the first thing that we can do is we're going to try to get the agent to get the information that we saw in that email and bring in the features here into our terminal.

And, uh, for this, we're going to use something called Work IQ, which is Microsoft. It's a skill that Microsoft has developed that lets developers connect to the M365 suite. So Outlook, PowerPoint, whatever it is you would like, and to bring that information here into the terminal.

So if you're using the M365 suite for work, I can definitely recommend it. And what I will also mention with TDD, in the past when we've done things like unit tests, um, typically people what would trigger writing a unit test is adding a new method to a class.

But actually, in this new world, what we want to focus on is the behavior. So we want to focus on a feature. So if a feature request comes, that is what the trigger is for the test to be written.

So now we have our list of what needs to be actually developed. And I'm passing in a second, uh, I'm passing in the second prompt. And I'm asking Copilot to help me develop these features using Red-Green-TDD to start by writing the Playwright tests that fail for each feature.

And I'm telling it not to commit the changes just for the sake of this example. And I do want to point out that the first thing the agent is going to do is it's going to start to examine my codebase.

So it's going to understand what's in my codebase. I have the Playwright MCP server already installed in my CLI, uh, into Copilot CLI. And it knows what it needs to do to create the tests to be able to test for these functions.

So the agent is going to understand what the codebase is going to look, uh, look like and then going to write the tests for it. Uh, this process is actually going to take a while. So in the meantime, I'm going to switch over to a new tab.

And I'm going to run the command to get Playwright the Playwright tests. So earlier today, I asked, uh, I got the agent to generate those failing tests. And then I got it to do the green phase where the agent just creates the code to get the tests to pass.

And then now I'm asking my agent to go ahead and run the Playwright tests to actually test for that search bar and filter feature for us. So like I mentioned before, I have the Playwright MCP server already installed.

You can see it installed here. And our agent is just going to look for the test file. And if everything works correctly, it's going to start writing some tests running some tests. So we see it's opened the correct page.

It's typing in different inputs, which is testing for the search bar is working. We saw Furby was correctly found. Simon was correctly found. And now it's clicking buttons. So also testing the category filter is working correctly. Again, my hands are not on the keyboard.

This is all Playwright and Copilot. So super cool. And now it's correctly finding all of the toys in the specific price range. So when I run these functionality tests, all I can see actively that, okay, the agent has written this code.

The code is working as I expected. The app is working as I expected. So there's so many different ways that you can test your app by functionality and all of our tests pass. Uh, now once our tests have passed, that's when I would say we step into the next phase of actually going ahead and, um, and, and running out our writing our refactor.

So refactoring the code the agent has created to generate these tests that pass. So a final thing I will do is I will give you some best practices, um, with Playwright. The first thing I would say is that when Playwright runs those functionality tests, it's going to take screenshots of all the tests that it's run.

I've gotten into the practice of adding those screenshots to a PR. So if I've made some changes, I'll add them to a PR. The second thing is that you don't have to run it where it launches the browser like you saw in the example.

You can run it in headless mode so it runs in the background. And then a final thing is I would say commit your code, uh, before you actually get it to fix the tests or, you know, commit before it starts to make changes to your code.

Because if you don't commit, it might not remember what it what happened in the past. So that's something to do. And then I would also say to generate one feature one test per feature as well. As a final note, these are some resources you could take a look at.

Ah, I forgot to add the link to the GitHub GitHub repo. But all of these slides are going to be available at that link there. You can check out the documentation, and you can connect with me on social media as well.

So yeah, thanks, everyone. I think that's all the time I have today.

I think we have two minutes for questions. Does anyone have any questions about this? Yes, I see a question there.

Um, like, I use Playwright to get a storybook because you need to, like, set up a sort of state, uh, like, how would you, like, now. This example, the eCommerce website is quite simple.

Yeah.

How would you what are tips for more complex tasks and admin panels where you have a lot of, uh.

We have a lot of, uh.

State management as well.

I mean, I think if you have a lot of state management, I would focus on maybe I would recommend using Playwright agents where it downloads the specific agent.md file because that's going to have some specialized instructions that are better at handling state and things like that.

So I found that agents, um, Playwright agents specifically has a lot of good instructions already built into it that should help with that. Another thing that you could do is if you didn't want to use Playwright for everything, you could also just directly test your APIs.

If there is an API available, that's something you could do. Um, so yeah, that's what I would recommend. Allright. And are there any other questions? Maybe one more question. Uh,

I'm not sure. Yeah.

Can Playwright also check, like, different, uh, like sizes, like from, like, desktop size to, like.

Yeah.

Okay.

Yes. Yes, it can. It can check your mobile versus on desktop. It should just work. Yeah. One more. Yeah.

Is it only for browser-based, uh, checkings or if I'm developing a Mac app or an iPhone app?

It's browser-based for the moment. Yeah. For the moment, it's only browser-based. Yeah. Okay. I think that is all all the time I have for today. Thanks, everyone. Sorry about the no link to the GitHub.