# How to Improve your Vibe Coding — Ian Butler AI Engineer · 2025-08-03 Ian Butler (CEO of BizMyth) presents benchmark data showing that popular coding agents like Devin and Cursor have less than 10% true positive rates for finding bugs, with Cursor producing a 97% false positive rate across 100+ repos and 1,200+ issues. He argues that these agents overrun codebases with false positives, causing alert fatigue and bugs reaching production. To improve vibe coding, he recommends adding bug-focused rules (e.g., OWASP Top 10, explicit bug classes like auth bypasses), managing context by feeding diffs and preventing file summarization, and using thinking models which were significantly better at finding deeper bugs. However, even thinking models show high variability in which bugs they find across runs, indicating agents never holistically examine a file. The talk concludes with a plug for BizMyth.sh, which automates PRs and vulnerability scanning. ## Questions this episode answers ### What is the true positive rate of AI coding agents like Devin and Cursor at finding bugs? Ian Butler reports that in their benchmark, Devin and Cursor have less than a 10% true positive rate for finding bugs. Overall, three out of six agents had a 10% or less true positive rate from over 900 reports. Cursor specifically had a 97% false positive rate across 100+ repositories and 1,200+ issues, meaning almost all reported bugs were not real. This leads to alert fatigue and bugs reaching production. [0:32](https://aie.addtry.com/b7e73c87-6805-4f24-aa72-23de47482af0?t=32000) ### How can I configure my AI coding agent's rules to better find security bugs? Ian Butler recommends adding scoped instructions to your agent's rules file. Specifically, feed it the OWASP Top 10 to bias it toward security issues. Name explicit bug classes like auth bypasses, SQL injection, and protocol pollution rather than vague requests. Also, require fix validations—tell the agent to write and pass tests to confirm fixes before code is accepted. Structured rules prime agents for higher-quality output and reduce false positives. [3:18](https://aie.addtry.com/b7e73c87-6805-4f24-aa72-23de47482af0?t=198000) ### Do thinking models perform better at finding code bugs than regular models? Yes, according to Ian Butler, thinking models significantly outperformed non-thinking models in their benchmark at finding bugs. Their thought traces show them expanding considerations and diving deeper into code. However, even thinking models have high variability between runs—the bugs they find change from run to run—indicating they don't holistically examine files like a human would. Despite this, Butler recommends using thinking models in tools like Claude Code or Cursor for better bug detection. [2:49](https://aie.addtry.com/b7e73c87-6805-4f24-aa72-23de47482af0?t=169000) ## Key moments - **[0:00] Intro** - **[0:32] The Problem** - [0:32] Devin and Cursor each achieve less than 10% true positive rate in bug finding, reveals Ian Butler's benchmark. - [1:31] Cursor posted a 97% false positive rate across over 100 repos and 1,200+ issues, says Ian Butler. - **[1:48] Three Tips** - **[3:04] Writing Rules** - [3:18] OWASP Top 10 fed into agent rules primes models for better security bug detection, advises Ian Butler. - **[4:26] Context Management** - **[5:29] Thinking Models** - [5:29] Thinking models significantly outperform non-thinking models in bug detection across codebases, Ian Butler states. - **[6:45] Wrap-up** - [7:00] "May your vibes be immaculate." ## Speakers - **Ian Butler** (guest) ## Topics Vibe Coding ## Mentioned BizMyth (company), Claude Code (product), Cursor (product), Devin (product) ## Transcript ### Intro **Ian Butler** [0:15] My name's Ian. I'm the CEO of BizMyth. We're an end-to-end agentic coding solution, kind of like Codex. We've been working on evals for how good agents are at finding and fixing bugs for the last several months, and we dropped a benchmark yesterday discussing our results. ### The Problem **Ian Butler** [0:32] So, one thing to point out about agents currently is that they have a pretty low overall find rate for bugs. They actually generate a significant amount of false positives. You can see something like Devin and Cursor have a less than 10% true positive rate for finding bugs. This is an issue when you're vibe coding, because these agents can quickly overrun your codebase with unintended bugs that they're not able to actually find and then later then fix. Overall too, it's worth noting that in terms of needle in a haystack, when we plant bugs in a codebase, these agents struggle to navigate more broadly across those larger codebases and actually find the specific bugs. So, here's the hard truth,right? Three out of six agents on our benchmark had a 10% or less true positive rate out of 900 plus reports. One agent actually gave us 70 issues for a single task, and all of them were false. And, like, no developer is going to go through all those,right? You're not going to sit there and try to, like, figure out what bugs actually exist. So, bad vibes,right? Implications: most popular agents are terrible at finding bugs. Cursor had a 97% false positive rate over 100 plus repos and 1,200 plus issues. The real-world impact for this is that when developers are actually building with this software, there's alert fatigue, and it reduces the effectiveness of trusting these agents, which means bugs are going to go to prod. So, how do you clean up some of the vibes? ### Three Tips **Ian Butler** [2:01] Like, we did this large benchmark. We've been doing this for months. We have, like, practical tips for you when you're working kind of in your IDE with these agents side by side. So, the first thing to note is bug-focused rules. Every one of these agents has a rules type of file. You want to basically provide scoped instructions that provide additional detail on security issues, logical bugs, things like that. The second issue here is context management. So, the biggest issue we saw with agents when navigating codebases was after a little bit of time, they'd get confused. They would lose logical links to stuff they've already read, and their ability to reason and come up with connections across a codebase stumbled significantly. Obviously, when it comes to finding bugs, this is a problem because most significant and real bugs are complex multi-step processes that are nested deeply in codebases. And then finally, thinking models rock. Thinking models were significantly better at finding bugs in a codebase. So, whenever you're using something like Claude Code, Cursor, whatever, try to reach for thinking models. They are just significantly better at this problem. Okay, so I mentioned rules earlier, and I think there's some, like, practical tips you can take away for improving your vibe coding. OWASP is, like, the world's most popular kind of, like, you know, security authority for bugs, I would say, give or take. ### Writing Rules **Ian Butler** [3:18] When you're creating your rules files, try to feed some specific security information, like the OWASP Top 10 to the model. What you're doing here is biasing the model, so when it's actually looking at your code, it's considering these things in the first place. Right now, we find when you don't actually supply models with security or bug-related information, their performance is significantly lower than otherwise. Second, you're going to want to prioritize naming, like, explicit classes of bugs in those rules. Like, don't be like, "Hey, Cursor, just try to find me some bugs in this repository." Be like, "Hey, Cursor, I want you to examine my repository for auth bypasses or protocol pollution, SQL injection, auth bypasses,"right? You want to be explicit about this. That kind of primes the models to be looking for these issues. And then finally, with rules, you want to require kind of, like, fix validations. You always want to tell the model, "Hey, you have to write and get tests to pass before this is coming into the codebase. You have to ensure they've actually fixed the bugs." We've seen more broadly across the 100 repositories we benchmarked in the thousands of issues we've seen from many agents that structured rules eliminate the vague check-for-bugs requests and the produce alert fatigue. Instead, they prime agents for much higher quality output. ### Context Management **Ian Butler** [4:26] So, okay, context is key too,right? So, I mentioned agents struggle significantly with, like, cross-repo, you know, navigation and understanding. In fact, a lot of the agents, when they reach their context limits, kind of, like, summarize or compact files down. When that compaction happens, the ability to detect and understand bugs reduces significantly. So, it's actually on you as users in the IDE to kind of manage your context more thoroughly for these agents. You want to make sure you're feeding, you know, either diffs of the code that was changed to the agent. They're able to actually understand cause and effect better from that. You want to make sure key files aren't being summarized or being taken out of the context window. And you want to actually ask one thing we found really effective in the benchmarking was asking agents to come up with a step-by-step component inventory of your code. So, have it index, like, these are the classes, these are the variables, this is how the use is happening across the codebase. When it does that inventory, it becomes much more able to find bugs. So, okay, thinking models rock. We saw across our benchmarking, basically just implicitly, that thinking models were far more able to find bugs. If you go through their thought traces, you're actually able to see them kind of expand across a few different, like, considerations in the codebase. ### Thinking Models **Ian Butler** [5:45] And then when they find those considerations, they will actually dive deeper into the chain of thought for finding those bugs. That means in practice, they do find deeper bugs than just non-thinking models were able to across the benchmark. However, I still want to note here, even with thinking models, there's, like, a pretty significant limitation in their ability to actually, like, holistically look at a file. We found, again, over hundreds of repos and thousands of issues that when agents were run, the top-line number of bugs found would remain the same, but they would actually the bugs themselves would change run to run. So, agents are never holistically really looking at a file like you or I would be looking at a file. There's high variability across runs. We think that's a very big limitation of current agents, by the way. We think for consumers, you shouldn't have to run your agents 100 times to get, like, the whole holistic kind of, like, bug breakdown, but that's kind of a still-in-progress problem. So, they're more thorough, and they just perform better across the benchmark than other models were able to. I'm going to quickly plug us. So, we're BizMyth.sh. We create PRs automatically. We're linked into GitHub, GitLab, Jira, and Linear. We scan for vulnerabilities, we provide reviews, and we also have on-prem deployments, which I know is a big sticking point for people. ### Wrap-up **Ian Butler** [7:00] May your vibes be immaculate. If you scan this QR code, it'll take you to our site. There, we have a link to the full benchmark with breakdown of methodology results. You can dive into the actual data itself, and you can see the SM100 benchmark here along with our full dataset and exploration so you can understand just how well current agents are actually at finding and fixing bugs. Yep, I'm Ian Butler. Thank you so much. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.