Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel

My name is Kobe Crawford. I'm a developer advocate at Snorkel. We are the frontier AI data lab, and what that means is that we produce datasets for foundation models to help climb on. Our research team is highly integrated with the work that we do in terms of our production work, and we put a lot of emphasis on how we integrate research in that.

This company's origins actually begin from Stanford University AI Research Lab, and the work that they were doing there actually was part of one of the CEO's PhD thesis. That became a library that was used open source for a while, and then we've grown into focusing on delivering things with datasets for our customers.

One of the things that's been a consistent through line for Snorkel since they got started

as a company in 2019 is that the core thesis has been that the quality of data is critical, and that the data that you're looking at you want to make sure is top quality in all those cases. We look at how that applies to the datasets that we provide, as well as as things move into the agentic space, how that applies to agentic tasks.

What we wanted to show in this context is how data quality impacts things in the context of task quality, and that task quality and data quality are largely the same thing.

We're going to talk about the particular research objective here was looking at, again, how task quality affects the training outcomes that you get when you're trying to improve models. Then from there, we're going to talk about the techniques and the path that we chose to verify that these behaviors were actually happening for us.

Does the task quality actually matter? You can tell from our thesis about it that we actually expect that, of course it does. The way that we're going to break down talking about that is to make sure we understand where we're approaching this.

We're talking specifically in the context of agentic TerminalBench style tasks. We're working with a flow that is going to be a containerized environment and then a task definition within that. We want to show that when you're looking at how

you look at the tasks themselves and how they're built, that what we do in the agentic context is still also governed by the same data quality premise. That applies in terms of talking about task quality as well as data quality.

If your architecture changes, if the harness that you're using changes, these kinds of things, all of those things are also obviously impactful things, but underlying all of that is still that data quality is at the center of it.

Here, what we're doing actually puts some specific kinds of rigor and delivering empirical evidence to validate that this is true. We want to not just sort of say we accept this as a thing that we like to say is true, we actually want to verify that that's the case.

In the definition of task quality, we're talking about basically four core things. If you've worked with these kinds of environments, the Harbor framework, OpenEnv, when you've built tasks for agentic purposes, what we're talking about in the context of evaluation, benchmarking, RL, is that we are creating an environment in which that is going to run.

We have it containerized for reproducibility and isolation, and that also allows it to parallelizing for rollouts, these kinds of practical elements of how that works. Inside of that, you have looking at what's in the logic of the task, you want to talk about that the task is achievable, that it is non-trivial, that it is functionally correct, that the logic actually plays as expected, and then the environment itself is reliable.

That environment reliability is a key as well. Those four criteria for us as we work on this, the Snorkel team has built in our research harnesses, we've built a setup where we verify all four of those criteria. Our tests to verify those criteria are the tests that we use.

If a task passes all of those tests, then we consider it an accepted task, is accepted, and then becomes something that we can use for our training and research purposes. Then if it's not accepted, then it would be put in the rejected bucket.

We use those two buckets as a basis for talking about how we're going to compare what is a high quality task, the accepted ones, to what is a low quality task.

Let's look at those comparisons for just a moment and make sure that when we actually take a look at it and level set, does the acceptance criteria that we use tend to correlate with actual performance behaviors that we think we want to see, and how they differ.

The way that we did that was we used Sonnet 4.5, this is obviously some months ago, and Codex, which had GPT-5.2, 5.1, and sometimes 4.0 included in terms of the tests that were run. We used those two for running these tests, and we compared how those tasks were completed.

In the completions, we found that our accepted tasks averaged twice as many tool calls, demonstrating more difficulty, more steps needed, and more engagement with external tools. A lower pass rate, so higher difficulty intrinsically, and then also more output tokens needed, so there was more reasoning that was done by the models to actually do that.

The failure modes, however, let's go back to the pass rate for a second. It's possible in the context of a lower pass rate that you could have failure modes that actually don't show meaningful signal. We actually wanted to dig into the failure modes a little bit as well.

The next step here is to just say, okay, what does it mean to be talking about those failures? To do that, we broke down the failures into categories to identify the kinds of failures that represent something meaningful, like this model is not performing the task completely because the model's not achieving a logical conclusion that it needs to, versus a failure that's more say of a kind of a degenerate case where you have a problem that is an environmental problem, something that literally makes it so that no model would be able to solve that problem and not complete that task in this particular flow.

We have a breakdown of those, and then given those, again, we wanted to compare the accepted versus rejected tasks and see where the failures are occurring in those tasks and see what we get from that.

Here's a summary breakdown of each of these things across the percentage of failures that we saw in each of these different categories. You can see, I like to highlight in particular the logic error and the incomplete task

bars and just observe that you can see that in each case, you can see a reversal of which one had a higher percentage appear in those there. Then this breakdown, even with the comparative analysis based on percentages of failures, you can see that you can see the difference in terms of where you see the overrepresentation or underrepresentation of these kinds of failures across the types of failures that we categorized, and again, between where the rejected tasks versus where the accepted tasks had errors.

The general tendency that we are taking away from this is that the accepted tasks are producing cleaner failures. These are failures due to the task itself being more difficult, truly more difficult, that the steps that it needs to accomplish are more difficult, and that means that this is a test that would be actually very useful for the model to be able to heel climb on, provide some data samples that could help it actually be able to be improved in terms of those performance patterns, versus something where it's just like a failure that's not super meaningful in terms of like it's just a tactical thing that's happening inside of the context that's not working.

With that in mind, we then take it as we've accepted that we've actually put together enough analysis that gives us a pretty strong sense that the accepted tasks are also higher quality tasks in the main. The same thing again, that given that we have a differentiation between the higher quality tasks and the lower quality tasks, now we want to actually see, can we see an impact on model performance when we use it.

Following forward from that, we just actually run a training run, an RL training run with the same model, the same compute budget, the same number of tasks in each case, and then look at the difference there. That's where we've level set that we have a set of tasks that we consider high quality, a level set of tasks that we consider lower quality, and that that usage is going to help us say something about that.

We trained it twice, and we wanted to see what we got. The performance uplift is

actually very meaningful. We're talking about about a 1% improvement with using the low quality tasks. After the RL training was done, the low quality tasks only improved the base model by about 1% improvement, but the improvement was about a 6% improvement with the higher quality tasks.

That uplift of a 5x uplift difference based on just quality is really striking from our point of view. We think that that cements the intuition that it really is important for the data quality to be high. The way that Snorkel generates datasets and the way that we put together RL environments, we're using human expertise and having experts in the loop for generating the data.

We have a strong feeling that the expert in the loop is an important element of delivering data quality. Between those, that ultimately gives you

we can talk more about how we use our platform to help make that work by our experts be something that we scale and we can deliver quality at scale. The key is that we want to put the emphasis on making sure that quality is the first thing that people think about, about what you need your data to what you need from your data to make sure that you're getting good results.

That's the end of what we got out of that. Hopefully, if you have additional questions, we have just a couple of additional minutes. Here's a couple of quick links to the research page as a summarizing how our research team works and what we do and what we put our emphasis.

Then the leaderboard is pointing to a couple of benchmarks that Snorkel built and curates ourselves that are similar to things that you've seen. We have one called agentic coding, which is focused specifically on these TerminalBench style tasks. We're doing the same kinds of evaluation, but we apply a certain kind of rigor to how we're going about it that we wanted to make sure that people could see the difference.

Thanks very much, and please come in with questions.

Can I take any follow-up questions? I'll start here.

I'm just kind of wondering if you would look at another ABC, I guess, of all the tasks rejected, accepted, to see if there's an effect of the rejected tasks actually pulling it back down, or actually the model can get over it as long as all the tasks are in there.

Good question. The interesting thing about

I don't have a specific and quick direct answer from that analysis that we did in this particular case, but one of the things that we saw in surveying and working with, for example, working with the TerminalBench team and looking at the tasks that were in TerminalBench 1 versus what we did for TerminalBench 2, and in some other contexts around some of the SweepBench, some of the variants of the SweepBench, we've been doing some analysis internally to compare the various public benchmarks that are out there and looking at what we see across those things.

When we look at that, you certainly see that the failure rates and where the models have been improving over time, whether they're getting to saturation faster or the benchmarks themselves are getting to saturation faster. Sometimes we ended up seeing some noise about that because you end up with a certain number of tasks that never get completed.

Then we start to see that these tasks will never be completed literally just because they actually can't be. Because of that, we ended up finding that that was actually sort of a source of noise in the process of actually evaluating whether the model improvement was actually happening.

In that way, I can't say it's like sort of like whether or not the models are actually improving ended up being something that ended up being more masked by the quality of task issues as opposed to being something where we could tell did the models actually improve or not despite their presence, being some more source of noise.

I saw a couple of other questionsright here.

Do you have any sense of how the input is making this task? There's a correlation sometimes, and input can be a little bit less prescriptive, could make it harder, or it could be very prescriptive, could make it easier.

I don't know, but I'm just wondering if you encountered something like that.

Yes, absolutely. In fact, the way that we've been looking at it, a lot of times what makes a task sort of one of the rejected tasks is it being under-specified in terms of when the task is defined in a way that the desired testable outcome is not clearly specified in the task definition up front, but then on the back end, the tests themselves expect certain things to pass that were never actually requested.

Those kinds of mismatches are sort of some of the places where you can see where the task becomes, or at least appears to be harder because the tests don't match what the requested task set up. Also, sometimes there are implicit dependencies in the testing that the task doesn't specify be there, and then without knowing that the dependency is required in the first place and hasn't been fed into the context of the model, then the model doesn't even have theright context to be able to approach those dependencies.

Just a follow-up or a quick thing.

Classifying them as a fail could be an issue because not every task needs to be complete. There's an iteration, there's a journey. Typically, when we solve problems, it's never a one-shot kind of thing in most of the problems in the world.

Yeah, differentiating that under-specified could maybe improve. I don't know.

Certainly, certainly. I 100% agree that those kinds of things about what we ultimately want the models to do tends to go like that. In the context of building benchmarking tasks, we work to make it so that we have something that's verifiable on the back end, and that in principle, if we do itright, that the skills that are being learned are still going to be more applicable in the context of unverifiable results or things where there's going to be an iteration that needs to occur following the step that you're working on currently.

Yeah, yeah.

Thank you.

Sure, sure. Yes.

In terms of future challenges, next steps, are you working on tasks that are not as straightforward, verifiable, and then maybe more towards the very long horizon?

We certainly are looking at all of those, and different projects of ours are in those different spaces, especially once you get outside of the places where verification isn't easy, and coding and math make it straightforward, and then things that are more fuzzy are different.

We have an open benchmark grants program that we're working with where we're partnering with folks who are developing benchmarks and evaluations in some more of the less verifiable areas. There's a lot of very interesting stuff that we're doing that has to do with

one of them, there's an interesting organization that's working on, I'm trying to remember the name of it, but it was about sort of like really looking at things that involve sort of emotional level things and stuff like that.

There's lots of very human-centric thinking. Even the notion of what is correct or not is something where we want to have actually multiple possible outcomes and then score them differently, but have them all sort of fit on the spectrum somewhere.

There's a lot that we're trying to do that vary things in a lot of different dimensions. Yeah, it's a very interesting space. One last question in the back.

Yeah, I was just wondering an extension of this conversation. How does your system handle this annuator, I guess, we may be disagreement, I guess, in the multi-step task that you're saying. I guessright now it's maybe more verifiable, like you say, because the answer is a bit discrete

for coding and math. In the world, when you say there's many possibilities, how do you manage that?

I'll speak to inter-annotator agreement sort of centrally, and then the complexity you're talking about 100% is something that the longer the horizon, the multiple steps involved, and the different dimensions are an issue. Last question, yeah? Or are we at dinner time?

Do I still have one minute left?

Yeah.

Okay, brilliant. The way that our platform works, we actually do a number of things to bring together human annotators as well as using LLM judges. That's partly to help us replicate and scale what our human annotators are delivering, but also these kinds of agreement, we feel like the way that we are doing things with rubrics these days and providing sort of like a longer list of data points and criteria that need to be met, that as we sort of build out a set of rubrics that then can be used both by LLM judges and people, that we're actually sort of like looking at high-level qualitative things as well as individual sort of like more quantitative comparisons.

Through a longer list of things, it builds on a rubric, and then using the human annotators and the experts to help us give us the information, the ground truth information that we can inform LLM judges to look at, we're actually looking to make sure, for example, that we actually test and get

inter-annotator agreement very high between both individual humans as well as between the LLM judges and humans, and then use all of those comparisons to do quality assessment. It's part of our assessment process, and we use that for each of these kinds of tests.

In the context here where it's explicitly verifiable and tests will pass or tests will fail, it's still obviously an easier domain than others, but we still keep using that sort of guiding principle across all these domains.

Allright, well thank you very much. I really appreciate your time. It's great to have you here.