Benchmarking semantic code retrieval on Claude Code — Kuba Rogut, Turbopuffer

Hi everyone, uh, welcome to benchmarking semantic search—or semantic code retrieval—on Claude Code. My name is Kuba, I'm here from Turbopuffer. For those unfamiliar with Turbopuffer, we are a serverless, full-text and vector search database built from first principles on top of object storage.

We serve some of the fastest-growing AI companies in the world, and if you'd like to know more about the talk or any questions about Turbopuffer or anything, just find—feel free to find me after. Uh, so let's get started.

So, for those of you who are unaware, Claude Code by default—or I mean, yeah, by default—doesn't use semantic code search. There's actually a tweet from Boris—those unfamiliar with Boris, he's essentially like the founding father of Claude Code, and he talks about how Claude—the early versions of Claude Code—did actually use semantic search with a local vector DB, but they just kind of found agentic search, essentially, you know, grepping through your file system, kind of worked better and, you know, seems to work simpler for Claude Code.

However, one of Turbopuffer's customers—actually, fun fact, one of our very first customers, Cursor, does use semantic code search and does in-index codebases into Turbopuffer. And you may think, like, this seems like a lot of work, and it kind of is.

But the reason they do this is because they see real performance gains because of this. They have this amazing blog post about their semantic search. They also have one about indexing codebases, which I highly recommend as well. But you can see on theright, the kind of performance gains they see with their composer model is like 24% increase in an—or relative improvement in answer accuracy, and I think it's like 12.5% or 13% across all models.

And, you know, I—this benchmark came up before Composer too, but you can probably imagine there's some sort of similar performance gains with that new model as well. And in the bottomright, you can see—this is from the same blog post—they talk about how in they performed an online A/B test where they found that adding semantic—either allowing or disallowing semantic code search—led to like a 2.6% increase in code retention in large codebases, and a 2.2% decrease in dissatisfied user requests.

And you may also think, like, these numbers seem kind of small, like what, 2.6%, 2.2%. Well, keep in mind that, you know, out of 100 queries, not all queries will ever really need semantic search or, like, benefit from it.

So these—they even state in the blog post that these numbers look small because not every—you know, you can imagine a very simple tool call or very simple query would not really use semantic search. So what we think about here at Turbopuffer and why we kind of really think vector search and why Cursor probably sees this, like, real performance gain is we think about how embeddings are cached compute.

And you could also, you know, what do you mean cached compute? This sounds like a bunch of baloney you're throwing at me. You know, if we were to walk through a Claude Code trace and like a Cursor type trace of kind of the same thing, you can imagine that on the left side is kind of like a, you know, grepping through the file system, or what they call agentic search, how it just has like grep through—this is for finding metadata filtering and understanding it in your codebase.

That's essentially grep through, read the files, grep through more if it doesn't find theright things. And again, this is repeated on every session across, you know, every agent running in, you know, for the same codebase. So that even if you're asking the same question or trying to do the same thing multiple times, you always have to do this compute again.

And you can see, like, in this case, 6,000 tokens isn't a lot in one time, but across every session and every agent, this really starts to add up. And if we—you know, on theright, this is more of a Cursor style or Cursor level trace where you have this upfront cost where you have to, like, chunk, embed, and index the codebase, but then you essentially have this cache of the semantic meaning.

So that when a agent wants to understand, you know, where or how does metadata filtering work, it can simply query, you know, how is metadata filtered, and get a lot—get the chunks it needs and kind of the understanding it needs a lot faster and with token savings.

And again, this doesn't—it's not a lot of savings in one time, but we all, you know, we're not really running one agent anymore. Like, I'm running like three at one time, so there's definitely some long-term savings. So what we did is we built a simple CLI tool for Claude Code.

We call it Turbo Grep. It's essentially, you can imagine, it's just a simple way of using a tree splitter library to essentially parse through your codebase, chunk it, embed it using the Voyage Code model, and then upload it to Turbopuffer.

This is just a simple file system walkthrough. There is an open-source library for the v1 version, and then soon the v2 version will be open-sourced.

Here's a little video of, for example, a tool call trace for Claude Code where you'll see it will call the TPuff tool. Now it's called TPuff search password reset token generator. This is in the Django repo. This is one of the repos I was testing it on.

And you can see that it gets the content and then is able to kind of give the full explanation to the user. And, you know, it's very easy to say, like, oh, this works better, but the important thing is obviously to benchmark it.

Let's see for real, what—how much better does it work? Cursor has their own internal ContextBench, and there's this paper, this public paper called ContextBench, where essentially the benchmark is not really testing whether or not the coding agent solved or didn't solve a problem.

It tests when it, in the process of solving the problem, did it find certain files, did it find certain lines, and did it find certain symbols. Because they, you know, they kind of have this thesis that, like, it, you know, it's also important how you get there, not just, like, the end goal.

The process really matters for understanding, like, are agents actually looking for theright files. So essentially it's a human-labeled dataset of, like, in order to complete this task, the agent should have looked at this file, these few lines, and these few symbols in order to, like, actually complete the task well.

And I tested it with three conditions. Essentially raw Claude Code out of the box. Then I tested it with Claude Code with a max of 50 line reads at a time. And then the same thing with the windowed reads with the TPuff search tool.

You may be asking, like, why this like 50 line read thing or limit. It's because it became really noisy, really fast if it's just reading, like, long files. It's really hard to, like, understand and get a difference because it all—if it just reads a whole, like, 1,000 line file, it doesn't really make a lot of difference in the numbers.

So we'll start with the first results of it. This is precision. Precision is essentially the measure of how many files did it read, like, in its total process. How many of them were actually golden files. So if there was—if Claude read 10 files and 8 of them were needed, it hits 80% precision.

You can see the baseline, it hit like 65% precision, 33% on line precision, and 43% on symbol. And it kind of goes up as we add windowed grep. And then windowed grep plus semantic search got it to like 87% file precision, for example.

And this is also due to the fact, like, Claude Code by default, like, is really exploratory. It loves to read as much as it can and, like, tries to read everything, for example, and it kind of shows up in the 65% precision.

And, like, it's hard to, like, translate these numbers, but a more, like, English or, like, human version of it is that, you know, Claude Code, one in every three file reads is actually just a completely wasted file. And with windowed grep, it was one in five reads was an irrelevant file.

And then with semantic search, it was only one in eight files was a, you know, quote-unquote wasted or irrelevant file. And it kind of, like, scales up as you kind of add these tools. And, you know, obviously this is already, like, pretty good.

Then we have recall. Recall is essentially how many of the needed files did it find. So, for example, if there was 10 files in the task that it should have found, if Claude found five of them—five files in its total trajectory—and only three of them were actually these golden files, it would hit 30% recall.

By default, again, Claude Code actually does win the file recall. And again, part of this is because it just loves to explore every single file it can. Then we have line recall where it drops a lot more. This is kind of a because it loved to read a lot of files, but also read a lot of files that didn't have a lot of golden context lines.

So you can kind of see that even though it did explore a lot, it also kind of explored the wrong things a lot. And then with grep and grep plus semantic search, or windowed grep and windowed grep plus semantic search, kind of got the same recall.

You can kind of see on theright, like, kind of the what happened with the behavior between these three conditions. And you may be thinking, like, well, you know, semantic search didn't really, like, add improvements here. Like, what went wrong?

Like, actually some decreases. And we can, like, dig in a little further into the recall numbers for certain tasks. This is across 50 tasks by default. But if we break it down into where semantic search won, you know, quote-unquote won, and where just, like, windowed grep won, we see some stark differences between, like, certain tasks performed a lot better with semantic search allowed and then certain tasks performed a lot better with no semantic search.

And this kind of proves, you know, certain tasks require different types of tools. For example, when semantic search won, it was really good at finding a lot of behavior-adjacent files that didn't have the same kind of keywords. For example, it was like I think one example off the top of my head was trying to, like, it was like multi, like, first handling lots of different ORMs that had to, like, handle across different libraries.

And it didn't, by default, the keyword search didn't, by default, find all of them, but behavior semantic search was able to kind of, like, understand these are all related files. And then grep won, when it was really good at just the task was a lot about, like, tracing through imports.

And if it, like, was able to find the keywords in, like, the first or second tool call, it was able to, like, just keyword search through that and find the relevant files. So again, it's like two different types of how to find files, but they kind of lead to, like, really different results.

So in summary, like, what does this mean? Well, we saw semantic search, like, did boost precision quite a bit. And we, like, kind of understand, like, grep and semantic search kind of find different code different ways. And an interesting thing to note as well is these numbers weren't as great as Cursor's because part of, like, part of Claude Code is it's built for just greping.

Like, that's what Anthropic kind of focuses on. Like, it's not built to under like, to really understand, like, when to call semantic search or how to call it. We kind of, like, add as an extra tool and it's like, hey, like, here's this cool tool.

You probably should use it sometimes, but it's very hard for it to have a true understanding of when to use it, why to use it, versus, like, for example, Cursor's composer model. They understand this is a built-in tool that it knows when and how to use it, and that's why they saw this, like, 23.5% performance gain.

So, like, in summary, you know, we think long-term winners will kind of provide these lightweight tools to find theright context in various different ways. I think it's something important to think about. You can't just, like, grep through everything, unfortunately, in a file system.

We think there's a lot of different ways to access lots of different types of information. And the people that provide these, like, easy tools to provide to shrink down these billion context windows into theright million will win in the long term.

That's the general talk. Thank you. If you have any questions, feel free to come up.

I have my own idea of what semantic search is, but could you define what your definition is of semantic search?

Sure. So this was just doing vector search. It was just performing vector search, embedding it using Voyage's code model, and then just embedding the query sentence or query tokens and just sending them back to Turbopuffer. Yeah.

Sorry, which embedding model?

Voyage Code 3.

Okay. And which vector DB?

Turbopuffer.

Ah, okay.

Yeah.

Sorry.

No worries. Yeah, yeah. Yeah, for those who are unfamiliar with Turbopuffer, we are the vector database that powers companies like Cursor, Anthropic, Notion. So when you use something like Cursor, you have by proxy used Turbopuffer. So you may know us just not by name, I guess.

I was just going to ask, you may not have a benchmark for this, but how does it perform on shit code?

Oh, that's tough. I mean, it's hard to say.

Is greping better or is semantic better if the code's like a mess and you need to go and sort it out?

So I think it works best when there's a lot of, like, comments on code because it kind of provides that semantic meaning.

Alongside like documentation that's kind of inline?

Yeah, if it's like inline documentation, that was like a big boost. I believe one of the repos, I remember, like, looking through some trajectories and, like, asking Claude, like, why did it perform so well here versus not. And it was when it kind of explains me, like, when I was looking through as well, like, those with, like, really good comments, for example, just, like, comments about the function, it's able to, like, really understand a lot more because you just kind of give this context to the model and the embedding model.

So then it can, like, actually search better. Because that's part of it. Like, the embedding is not the hard part. It's, like, figuring out what meaning really is of that chunk. Yeah.

You may have you mentioned some of it now, but of course, semantic search is just similarity search. So if your query doesn't really match the format of what you're querying against, then you get some kind of innate distance.

Do you do any kind of pre-processing of what your of the target data before you like, do you do any parent-child where the parent is query-ish and the child is the real code, or how do you?

In this case, it was just simple, just the code.

Yeah, just the raw code.

Yeah, just the raw code, just the thing. But I can't speak for, like, how these more complicated and sophisticated customers use this, but I can imagine it's definitely something of providing not just code-level meaning, but, you know, as you said, it's a parent-child relationship of, like.

The authentication flow, like, that could be a good query to do a similarity search against, but the code itself is more, like, raw.

Yeah. Yeah. Like, Cursor has their own embedding model, which I think kind of helps with this of, like, how do you translate code into more of like a human-level query. And I mean, they're kind of an experts on that.

I can't speak for how they do it, but I just know they do do that.

I think they actually do what you said, like, they create fake comments on top of your code and they embed the code with the comments.

Yeah.

So that's how they can have, like, the higher recall when they search for the code.

Oh, okay.

So they add they kind of inject comments.

Yeah. Yeah. Yeah, it's something that I think definitely could work. Yeah.

Interesting.

In the back?

Yeah. I was going to ask, like, how do you see, I guess, the vector database kind of working with the harness and how, like, when do you encode, like, long-term or mid-term reduce the size of all the files that are created?

I mean, I think it depends. Like, obviously the easiest way, like, people love greping because it's zero cost. Like, if you're able to, like, download everything to your local file system and just, like, grep through it, like, yeah, it works.

I think vector DBs are built essentially like multiplayer and, like, this, like, super maybe in a sense, like, hard-to-understand or complicated relationships between lots of data. For example, like a knowledge base, like a Notion, like, you can imagine kind of hard to, like, really grep through that really easily on your local machine.

Like, it's in a sense best to, like, have that vectorize for the agents. And even stuff like, we have customers doing stuff like with multimodal data. Like, you can't really grep through a video file. You can't grep through an audio file.

You can't grep through an image file. Like, maybe you can glob on the file name, but, like, get a true understanding of these kind of multimodal data as well is something that we find a lot of customers are doing.

Go ahead. It just kind of depends on the workload and I yeah, if you're hitting, like, at some sort of even, like, miniature scale, like, a vector DB kind of, like, helps offload a lot of this work into, like, you know, cache compute of cache semantic meaning.

Any other questions?

Perfect. Thank you all.