How to look at your data — Jeff Huber (Chroma) + Jason Liu (567)

Allright, welcome everybody. I'm Jeff Huber, the cofounder and CEO of Chroma, and I'm joined by Jason. We're going to do a two-parter here. We're really going to pack in the content. It's the last session of the day, and so we thought we'd give you a lot.

Everything in this presentation today is open source and code available, so we're also not selling you any tools. And so there'll be QR codes and stuff throughout to grab the code. So let's talk about how to look at your data.

All of you are AI practitioners. You're all building stuff. And this probably these questions probably resonate quite deeply with you. What chunking strategy should I use? Is my embedding model the best embedding model for my data? And more.

And our contention is that you can really only manage what you measure. Again, I think Peter Drucker is the original who coined that, so I can't take too much credit. But it certainly is still true today. So we have a very simple hypothesis here, which is you should look at your data.

The goal is to say, look at your data, I think at least 15 times this presentation, so that's two. And great measurement ultimately is what makes systematic improvement easy. And it really can be easy. It doesn't have to be super complicated.

So I'm going to talk about part one, how to look at your inputs. And then Jason's going to talk about part two, how to look at your outputs. So let's get into it. Allright, looking at your inputs, how do you know whether or not your retrieval system is good?

And how do you know how to make it better? There are a few options. There is guess and cross your fingers. That's certainly one option. Another option is to use an LLM as a judge. You're using some of these frameworks where you're checking factuality and other metrics like this, and they cost $600 and take three hours to run.

If that is your preference, you certainly can do that. You can use public benchmarks. So you can look at things like MTEB to figure out, oh, which embedding model is the best on English? That's another option. But our contention is you should use fast evals.

And I will tell you exactly what fast evals are. Allright, so what is a fast eval? A fast eval is simply a set of query and document pairs. So the first step is, if this query is put in, this document should come out.

A set of those is called a golden data set. And then the way that you measure your system is you put all the queries in, and then you see, do those documents come out? And obviously, you can retrieve 5, or retrieve 10, or retrieve 20.

It kind of depends on your application. It's very fast and very inexpensive to run. And this is very important because it enables you to run a lot of experiments quickly and cheaply. I'm sure all of you know that experimentation time and your energy to do experimentation goes down significantly when you have to click go and then come back six hours later.

All of these metrics should run extremely quickly for pennies. So maybe you don't have yet you have your documents, you have your chunks, you have your stuff in your retrieval system, but you don't have queries yet. That's OK.

We found that you can actually use an LLM to write questions and write good questions. I think just doing naive, like, hey, LLM, write me a question for this document, not a great strategy. However, we found that you can actually teach LLMs how to write queries.

These slides are getting a little bit cropped. I'm not sure why, but we'll make the most of it. To give you an example, so this is actually an example from one of the MTEB kind of the golden data sets around embedding models, one of the benchmark data sets.

This also points to the fact that many of these benchmark data sets are overly clean. What is a pergola used for in a garden? And then the beginning of that sentence is, a pergola in a garden... Real world data is never this clean.

So what we did in this report, the link is in a few slides, we did a huge deep dive into how can we actually align queries that are representative of real world queries. It's too easy to trick yourself into thinking that your system's working really well with synthetic queries that are overly specific to your data.

And so what these graphs show is that we're actually able to semantically align the specificity of queries synthetically generated to real queries that users might ask of your system. So what this enables is, if a new cool, sexy embedding model comes out, and it's doing really well in the MTEB score, and everybody on Twitter is talking about it, instead of just going into your code and changing it and guessing and checking and hoping that it's going to work, you now can empirically say whether it's good, better, or not for your data.

And the kind of example here is quite contrived and simple. But you can actually look at the actual success rate. OK, great. These are the queries that I care about. Do I get back more documents than I did before?

If so, maybe you should consider changing. Now, of course, you need to re-embed your data. That service could be more expensive. It could be slower. The API for that service could be flaky. There's a lot of considerations, obviously, when making very good engineering decisions.

But clearly, the North Star of success rate of how many documents that I get for my queries, super fast and super useful, and makes your improvement of your system much more systematic and deterministic. Allright, so we actually worked with Wits & Biases, looking at their chatbot to kind of ground a lot of this work.

So what you see here is, for the Wits & Biases chatbot, you can see four different embedding models. And you can see the recall at 10 across those four different embedding models. And then I'll point out that blue is ground truth.

So these are actual queries that were logged in Weave and then sent over. And then there's generated. These are the ones that are synthetically generated. And what we want to see is a few things. We want to see that those are pretty close.

And we want to see that they are always the same kind of in order of accuracy. We don't want to see any big flips between ground truth and generated. And we're really happy to see that we found that answer.

Now, there are a few fun findings here, which is and of course, they're going to get cropped out. But that's OK. Number one, the original embedding model used for this application was actually text embedding 3 small. This actually performed the worst out of all the embedding models that we evaluated just for in this case.

And so probably wasn't the best choice. The second one was that actually, if you look at MTEB, Gina embeddings v3 does very well in English. It's like way better than anything else. But for this application, it didn't actually perform that well.

It was actually the Voyage 3 large model which performed the best. And that was empirically determined by actually running this fast eval and looking at your data. That's number three. Allright, so if you'd like access to the full report, you can scan this QR code.

It's at research.chroma.com. There's also an adjoining video, which is kind of screenshotted here, which goes into much more detail. There are full notebooks of all the code. It's all open source. You can run it on your own data.

And hopefully, this is helpful for you all thinking about how, again, you can systematically and deterministically improve your retrieval systems. And with that, I'll hand it over to Jason.

Thank you.

So if you're working with some kind of system, there's always going to be the inputs that we look at. And this is what we talked about, maybe thinking about things like retrieval. How does the embeddings work? But ultimately, we also have to look at the outputs.

And the outputs of many systems might be the outputs of a conversation that has happened, an agent execution that has happened. And the idea is that if you can look at these outputs, maybe we can do some kind of analysis that figures out what kind of products should we build, what kind of portfolio of tools should we develop for our agents, and so forth.

And so the idea is, if you have a bunch of queries that users are putting in, or even a couple of hundred of conversations, it's pretty good to just look at everything manually. Think very carefully about each interaction.

And then only use these models when they make sense. And then oftentimes, if I say this, they can say, what if we just put everything in O3? And then here, generally, only use the language models if you think you're not smarter than the language model.

Then when you have a lot of users and an actual good product, you might get thousands of queries or tens of thousands of conversations. And now you run into an issue where there's too much volume to manually review.

There's too much detail in the conversations. And you're not really going to be the expert that can actually figure out what is useful and what is good. And ultimately, with these long conversations with tool calls and chains and reasoning steps, these outputs are now really hard to scan and really hard to understand.

But there's still a lot of value in these conversations. If you've used a chatbot, whether it's in Cursor or any kind of like Cloud Code system, oftentimes you do say things like, try again. This is not really what I meant.

Be less lazy next time. It turns out a lot of the feedback you give is in those conversations. We could build things like feedback widgets or thumbs up or thumbs down. But a lot of the information exists in those conversations.

And the frustration and the retry patterns that exist can be extracted from those conversations. And the idea is that the data really already exists in this conversation. If we think of a simple example in a different industry, we can imagine the analogy of marketing.

Maybe we run our evals and the number is 0.5. I don't really know what that means. Factuality is 0.6. I don't know if that's good or bad. Is 0.5 the average? Who knows? But imagine we run a marketing campaign and our ad metric or our KPI is 0.5.

There's not much we can do. But if we realize that 80% of our users are under 35 and 20% are over, and we realize that the younger audience performs well and the older audience performs poorly, what we've done is we've just drawn a line in the sand on who our users are.

And now we can make a decision. Do we want to double down on marketing to a younger audience? Or do we want to figure out why we aren't successfully marketing to the older population? Do I find more podcasts to market to?

Should I run a Super Bowl ad? Now, just by drawing a line in the sand and deciding which segment to target, we can now make decisions on what to improve. Whereas just making them ads better is a sort of very generic sentiment that people can have.

And so one of the best ways of doing that is effectively just extracting some kind of data out of these conversations in some structured way and just doing very traditional data analysis. And so here we have a kind of object that says, I want to extract a summary of what has happened, maybe some tool that it's used, maybe the errors that we've noticed in conversations that happen, maybe some metric for satisfaction, maybe some metric for frustration.

The idea is that we can build this portfolio of metadata that we can extract. And then what we can do is we can embed this, find clusters, identify segments, and then start testing our hypotheses.

And so what we might want to do is sort of build this extraction, put it into an LLM, get this data back out, and just start doing very traditional data analysis. No different than any kind of product engineer or any kind of data scientist.

And this tends to work quite well. If you look at some of the things that Anthropic Clio did, they basically found that code use was 40x more represented by cloud users than by GDP value creation. They go, OK, maybe code is like a good avenue.

And obviously, that's not really the case. But the idea is that by understanding how your users develop a product, you can now figure out where to invest your time. And so this is why we built a library called Cura that allows us to summarize conversations, cluster them, build hierarchies of these clusters, and ultimately allow us to compare our evals across different KPIs.

Again, so now, if we have factuality is 0.6, that's really hard. But if it turns out that factuality is really low for queries that require time filters, or factuality is really high when queries revolve on contract search, now we know something's happening in one area, something's happening in another.

And then we can make a decision on what to do and how to invest our time. And the pipeline is pretty simple. We have models to do summarization, models to do clustering, and models that do this aggregation step.

And so what you might want to do is just load in some conversations. And here, we've made a fake data set, fake conversations from Gemini. And the idea is that first, we can extract some kind of summary model where there's topics that we discuss, frustrations, errors, et cetera.

We can then cluster them, define cohesive groups. And here, we can find maybe some of the conversations are around data visualization, SEO content requests, or authentication errors. And now we get some idea of how people are using the software.

And then as we group them together, we realize, OK, really, there's some themes around technical support. Does the agent have tools that can do this well? Do we have tools to debug these database issues? Do we have tools to debug authentication?

Do we have tools to do data visualization? That's something that's going to be very useful. And at the end of this pipeline, we're sort of presented with these printouts of clusters. We know what the tools are, how the chatbot is being used at a higher level, SEO content, data analysis, and at a lower level, maybe as blog posts and marketing.

And just by looking at this, we might have some hypothesis as to what kind of tools we should build, how we should develop even our marketing, or how we can think about changing our prompts. We can do a ton of these kinds of things.

And this is because the ultimate goal is to understand what to do next. You do the segmentation to figure out what kind of new hypotheses that you can have. And then you can make these targeted investments within these certain segments.

If it turns out that 80% of the conversations that I'm having with the chatbot is around SEO optimization, maybe I should have some integrations that do that. Maybe I should re-evaluate the prompts or have other workflows to make that use case more powerful for them.

And again, the goal really is to just make a portfolio of tools, of metadata filters, of data sources that allows the agent to do its job. And oftentimes, the solution isn't really making the AI better. It's really just providing theright infrastructure.

A lot of times, if you find that a lot of queries use time filters and you just didn't add a time filter, that can probably improve your evals by quite a bit. We have situations where we wanted to figure out if contracts were signed.

And if we just extracted one more step in the OCR process, now we can do these large-scale filters and figure out what data exists.

And generally, the practice of improving your applications is pretty straightforward. We all know to define evals, but not everyone that I work with has really been thinking about something like finding clusters and comparing KPIs across clusters. But once you do, then you can start making decisions on what to build, what to fix, and what to ignore.

Maybe you have a two-sided quadrant. Maybe we have low usage and high usage. And you have high-performing evals and low-performing evals. If a large portion of your population are using tools that you are bad at, that is clearly the thing you have to fix.

But if a large proportion of people are using tools that you're good at, that's totally fine. If a small proportion of people use something that do something that you're good at, maybe there's some product changes you didn't make.

Maybe it's about educating the user. Maybe it's adding some prefiller or automated questions to show them that we can do these kind of capabilities. And if there are things that nobody does, but when we do them, they're bad, maybe that's a one-line change in the prompt that says, sorry, I can't help you.

Go talk to your manager. These are now decisions that we can make just by looking at what proportion of our conversations are of a certain category and whether or not we can do well in that category. And as you understand this, then you can go out.

You can build these classifiers to identify these specific intents. Maybe you build routers. Maybe you build more tools. And then you start doing things like monitoring and having the ability to do these group buys. So now you have different categories of query types over time.

And you can just see what the performance looks like, where 0.5 doesn't really mean anything. But whether or not a metric changes over time across a certain category can determine a lot about how your product is being used.

By doing this, we figured out that some customers, when we onboard them, they use our applications very differently than our historical customers. And we can now then make other investments in how to improve these systems. And ultimately, the goal is to create a data-driven way of defining the product roadmap.

Oftentimes, it is research that leads to better products now, rather than products justifying some research that we don't know is possible. And again, the real marker of progress is your ability to have a high-quality hypothesis and your ability to test a lot of these hypotheses.

And if you segment, you can make clearer hypotheses. If you use faster evals, you can run more experiments. And by having this continuous feedback through monitoring, this is how you actually build a product. Regardless of being an AI product, this is just how you build a product.

And so if we look at the takeaways, really, when you think about measuring the inputs, we really want to think about not using public benchmarks, building evals on your data, and focusing first on retrieval, because that is the only thing an LLM improvement won't fix.

If the retrieval is bad, the LLM will still get better over time. But you need to earn theright to sort of tinker with the LLM by having good retrieval. And then lastly, if you don't have any customers or any users, you can start thinking about synthetic data as a way of augmenting that.

And once you have users, look at your data as well. Look at the outputs. Extract structure from these conversations. Understand how many conversations are happening, how often are tools being misused, what are the errors, and how are people frustrated.

And by doing that, you can do this population-level data analysis, find these similar clusters, and have some kind of impact-weighted understanding of what the tools are. It's one thing to say, maybe we should build more tools for data visualization.

It's another thing to say, hey, boss, 40% of our conversations are around data visualization. And the code engine or the code execution can't really do that well. Maybe we should build two more tools for plotting and then see if that's worth it.

And you can justify that because we know there's a 40% of the population that is using data visualization. And we do that maybe only 10% of the time. This is impact-weighted. And ultimately, as you compare these KPIs across these clusters, you can just make better decisions across your entire product development process.

So again, start small. Look for structure. Understand that structure and start comparing your KPIs. And once you can do that, you can make decisions on what to fix, what to build, and what to ignore. If you want to find more resources, feel free to check out these QR codes.

The first one is to Chroma Cloud to understand a little bit more about their research. And the second one is actually a set of notebooks that we've built out that go through this process. So we load the wasted and biases conversations.

We do this cluster analysis. And we show you how we can use that to make better product decisions. So there's three Jupyter notebooks in that repo. Check them out on your own time. And thank you for listening.

We do have time for we do have time for one quick question and, of course, as well outside as well.

Perfect. Thank you.

If anybody wants to grab the mic there and over there.

So Jason.

Yep.

You're famous for spicy takes.

What's the spicy take today?

It's not KPI, by the way. That's not the spicy take.

I think more agent businesses should try to price and price their services on the work done than the tokens used. Yeah. Price on success. Price on value. Very unrelated to this talk.