# When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain AI Engineer · 2026-07-20 Dmitry Petrov of DataChain argues that AI agents fail on unstructured physical data because their intuitions assume cheap recompute, while large-scale video, sensor, and robot data requires a different harness. He cites Anthropic's finding that agents achieve only 21% accuracy on data projects without specific data harnesses, and OpenAI's need for six layers of context even on structured data. Petrov demonstrates DataChain, an open-source Python framework that uses Pydantic schemas to turn messy binary files into queryable databases, an execution engine for distributed processing, incremental checkpoints to avoid recomputing on failure, and a knowledge base of datasets and source code so agents can answer follow-up questions in seconds instead of reprocessing terabytes. In a live demo with Claude Code analyzing 90 dashcam videos, the harness took 24 minutes to extract 100,000 object records, then instantly answered 'how many clips have people?' without rerunning inference. Petrov emphasizes that the key is organizing metadata into star schemas and sharing data lineage across teammates so no one pays the compute cost twice. ## Questions this episode answers ### What accuracy do AI agents achieve on data-related projects? According to Dmitry Petrov, citing Anthropic, agents initially achieve only 21% accuracy on data projects. This poor performance comes without specific data harnesses and context. He explains that working with physical, unstructured data—like video or sensor feeds—demands a different approach than structured business data, which frontier labs often focus on. The low accuracy highlights why specialized infrastructure is essential. [0:30](https://aie.addtry.com/67b3476c-e779-4ea5-9956-39ed82b9f5bb?t=30000) ### Why does working with unstructured data like video create a metadata explosion? Dmitry Petrov compares unstructured data to a neutron star: a few files, like 2,000 videos, can contain millions of internal objects—clips, frames, detections—each with attributes and connections. This nested structure causes an explosion of metadata. Managing it with flat JSON files or a separate database leads to complexity, while a unified Pythonic schema approach provides a more manageable alternative. [3:09](https://aie.addtry.com/67b3476c-e779-4ea5-9956-39ed82b9f5bb?t=189000) ### How can AI agents avoid recomputing everything when answering new questions over large video datasets? Petrov demonstrates that after an initial intensive processing pass (e.g., 24 minutes on 90 dashcam videos), the extracted metadata is stored in a database. Subsequent queries, like counting videos with people, run against this materialized layer in seconds without reprocessing raw data. This approach replaces expensive recompute with fast recall, saving time and cost. Incremental checkpoints further prevent loss if processing fails. [8:39](https://aie.addtry.com/67b3476c-e779-4ea5-9956-39ed82b9f5bb?t=519000) ## Key moments - **[0:00] Intro** - [0:30] Anthropic finds agents achieve only 21% accuracy on data projects without specific data harnesses. - [1:46] Agents handling physical data need a harness providing abilities to see, run, touch, verify, and remember data. - **[2:33] Seeing Data** - [2:33] "It looks like a neutron star. It's small on the surface, but the mass of this object is tremendous." - [4:51] DataChain uses pydantic schemas in Python codebase to eliminate SQL islands when handling unstructured data. - **[5:56] Demo Setup** - **[8:39] Exploring Results** - [8:39] DataChain harness processes 90 dashcam videos in 24 minutes using YOLO, storing metadata in a database. - [10:47] Processing 90 videos with DataChain yields 100,000 detection records, illustrating the data explosion in unstructured data. - **[13:39] Execution Engine** - [13:57] Data harnesses require an execution engine that can recover from failures without recomputing from scratch, says Dmitry Petrov. - **[18:01] Checkpoints & Tests** - [18:01] Incremental updates and data checkpoints prevent losing expensive LLM compute on failures, a must-have for data harnesses. - **[20:07] Metadata Layers** - [20:07] To answer questions fast, stop running Python scripts on raw data; instead organize metadata layers using star schemas. - [22:15] Teams often pay double to quadruple the compute cost by recomputing the same data slices, warns Dmitry Petrov. - **[22:38] Knowledge Base** - [23:06] OpenAI's data agent blog post identifies source code as the most crucial context; DataChain encodes it in knowledge base MD files. - **[25:58] The Stack** - [26:28] At large scale, coding agents' intuition fails because the physics of data changes; a data harness is required, not just stronger LLMs. ## Speakers - **Dmitry Petrov** (guest) ## Topics AI Infrastructure, Agent Engineering ## Mentioned Anthropic (company), OpenAI (company), Claude Code (product), Codex (product), Copilot (product), Data Version Control (product), DataChain (product), Git (product), Pedantic (product), Ray (product), S3 (product), Spark (product), YOLO (product) ## Transcript ### Intro **Dmitry Petrov** [0:03] How could an agent work with physical data? Video recordings, sensor data, robots' telemetry, and sometimes all of those combined in a single multimodal project. If you've tried this, you probably have seen how badly it fails, and today we'll discuss the reasons and all the physics laws behind the problems and how to fix it. This year, two frontier labs published very interesting results—and surprising results—that agents in general are not good at data. Anthropic published that accuracy for data projects on their agents is only 21% until you add specific data harnesses to them and provide context. OpenAI published a whole layers of context—6 layers of context—in order to make the data agent work, and all of those are on structured data which lives in a very houses with tables, execution engine, and all this luxury. In my life, I don't have this luxury, unfortunately, because I live in a very extreme side of the data universe: messy, unstructured data. I've worked with data for about 10 years. I built a Data Version Control project, Git for data, and now work on DataChain. In order to make agent work for unstructured data, for physical data, we need to build not only the brain—which we already have,right, it's LLM—but we need to make harness data harness for agents to understand this physical world. It should see the data properly, it should be able to run, kind of giving him a leg. It should be able to touch data, verify the result, run tests, and also remember the crucial important datasets, important result, so this information could be reused in the future project. And the first we start from the how you see data. What does it take to understand all this complicated binary file that you have in your object storages? ### Seeing Data **Dmitry Petrov** [2:48] In reality, how it usually looks, there are just a several files, sometimes several thousand of files,right. It doesn't seems like a big deal,right, but what is usually happens, those files are very complex inside. That's what makes unstructured multimodal data complicated. And from tutor, because video recordings might have clips inside, clips might have frames, frames, objects, objects, confident, type, class, label. There's connection between those pieces together. It makes kind of like explosion. It looks like a neutron star. It's small size on the surface,right, it's just a size of one city, but the mass of this object is tremendous. It's more than the mass of our sun. And 2,000 objects, 2,000 files of videos could easily generate you millions of object inside the videos. And how people usually do deal with these problems? They usually go through like two major steps. First step, let's put this meta information to JSON files and put it on S3 next to the images,right, and they end up with millions of JSONs. Crazy latency, not efficiency, not consistency. And the next idea, why don't we use database? Brilliant. And the most advanced team do exactly this. Let's put a centralized database when all the metadata is in there. Great. But this way you end up with a two system, with a two programming languages, and all the mess around two different stacks. And this stacks useless for most of the researchers because they don't want to deal with this complexity. We found that the easiest way for researchers and developers to deal with the schema is pedantic. So you use the same language for the data, for the schemas, as well as code. There are no SQL island in your code base. So in that way, you kind of transition from the messy world of unstructured data to the structure. Don't you think you needed just like transpilers to the SQL? From Python, and I'll show you how it works. But that's the way when you can create schemas and work with the schemas efficiently. I want to emphasize that the problem we are solving here, it's very specific for unstructured data, physical data. It doesn't exist in the structured data world,right, for example, in those OpenAI and Anthropic blog posts, because they work with the structured business logic,right. We work with the physical data binaries and such. Let's see how it works. I will be showing you our open-source project, DataChain, with data harness to coding agents. So first you need to install the tool,right, through using pip install. So we have already done this. Then skill needs to be installed. So you choose the coding tool that you use. ### Demo Setup **Dmitry Petrov** [6:21] We will be doing Claude Code, but we support three more different coding agents. And then you run your favorite coding agents. Of course, we are skipping permissions just to make it faster,right. And then you define the prompt and just solve your problem. So in this case, we will be analyzing motions in video cameras. That's open dataset from dashcam recordings. Video, it's a very usual use case for physical AI projects. Majority of the project do include video recordings as one of the modalities, sometimes the major modalities. And that's one of the, I would say, more interesting and more challenging problems to solve in this physical AI world. Let's use this modality as an example. This data harness asks some questions from users to understand the scope better. So in this case, we need to choose the model. So it decided to use a YOLO model, and we need to choose the size. So let's use the smallest one. We need to choose the velocity. So there's a few ways how to track, like speed, but let's use the simplest one. So the second one adds more or will require more time for the compute. And granularity, so pair detection frame, pair track. Let's do just pair frame in this case. And the dataset itself, so we ask for January data,right, 91 clip, but it also ask for if we need to extend scope and analyze more images. So let's stay here. ### Exploring Results **Dmitry Petrov** [8:39] So it took us 24 minutes to analyze those 90 videos. And now we have all the information. We can ask questions about it. For example, how many videos have people in it? This information lives in the database,right, as we discussed, and it can very quickly query this database and return the result,right. So okay, 92, oh, 82 out of 91 clips have some people detected,right. And you don't need to kind of go through all the JSON files and download and analyze and parse,right. Information is here in the local database. It can easily, quickly answer the question. We can even see the source code. Okay, just an easy, nice Python code to get data, get data, dataset,right, analyze, apply some simple filters when label is equal to person, count the number,right, and then see the total number. And it goes like it runs against database, so super fast. So how big is how many records do we have here? So that's about the size of our neutron star,right. It's not supposed to be big because we analyze only like 90 videos. And 90 videos generated 100,000 of records,right. So that's how explosion is happening. If you analyze thousands of videos,right, it goes to like a million scale. And if you go deeper to the objects, it can easily be multiplied by 10, 20, even hundreds. Let's take a look at the source code. This code was generated by the agents, power agents using the harness. The magic part here is the data model. So this data class that was generated based on our requirements, and it's just a pedantic, usual pedantic data model with a file,right, the video file, frame ID, timestamp, class ID. That's how you know if it's a person or car or some object. Confidence score, binding box, there are a few more items here. And the object is nested. If you look at the if you take a look at the binding box,right, there's several columns, but files is more interesting one. File has path and checksums, like a version of the file, ETag, size of the file. So everything that your cloud storage provides for you. And this information is just become a row in a database instead of a JSON file in S3. That's why you can easily answer all the questions, all the analytical questions,right, like before we ask how many people are there. And it's just a matter of like a simple Python code that runs against the database to return like results super quickly. That's how you can make sense of all the messy data you have on your storage,right, in a file, but you analyze this using regular analytical tools and tricks. ### Execution Engine **Dmitry Petrov** [13:39] With the schemas, you can see what is inside the object. You can see the scale of the problem. But how to do the actual heavy lifting when we deal with the terabytes of this messy data? It turned out that data harness have to have execution engine in order to deal with physical data. It has to work efficiently. Sometimes it spent a lot of resources and tokens if you use LLMs. And if it breaks somewhere in the middle, so you should be able to recover and catch up with all the result that it's already processed. So you don't want to waste your resources. In the SQL world, structured data world, execution engine is your data very fast,right. That's obvious and easy. In unstructured data world, you have to reinvent one. And in some teams, they use different types of orchestration tools or distributed compute on Ray or Spark. I like model of Dusk, which connects compute with the data, structured data. But unstructured data needs its kind of like own approach when you connect the Python and schemas with the execution engine. To simplify it, you can connect those Python functions, pedantic schemas, input parameters, output parameters for the function, as well as the data very house with all the meta information and files on the storage, all together in a holistic experience. This way, it will be more simple for developers to code, and it might be simple for the engine to distribute jobs to different machines or different threads based on the files they process, the schemas they produce, so far and so on. It's kind of like a Dusk approach if you wish, but for unstructured data world. To make the execution efficient, you need to run this data processing, data crunching function very efficiently,right, in a parallel way or distributed way. And we use data models and the data storage models a lot. So it kind of connects the file, which is a file in your S3 bucket, to the result. That's a set of detection object, which goes to the data very house,right. So this function connects the storage and the data very house in a regular Python way. So that's a very typical Python code. There are no like extra assumptions about this code,right. We don't even use like annotation for the function. It's just a function with a specified types. So the types are important, and all of those are pedantic. And result, you just generate the objects, and the engine connects all the storages and very house together,right. That's how you define the parallel parallelization layer. You can say, I just need to run it in 40 machines,right. That's how easy it's supposed to be for researchers to run like distributed compute,right. You just specify how many resources you need, so you got it. And the function itself,right, so we just use it, run it through generator, or it can be like mapper with one-to-one function, one function, one file returns like one object to the database, or generator, one function returns like multiple objects in the database. And it is saved as a like dataset,right, table under the hood in your database. ### Checkpoints & Tests **Dmitry Petrov** [18:01] Processing of those heavy files usually takes a lot of time. And sometimes it's very expensive because you use LLM in order to understand what is inside of your binary files. And the last thing you want is to lose this compute. It's really sad when you process hundreds of thousands of files and it fails in the middle because of bugs or IPI call issue, and you want to execute everything from scratch,right. You are losing the whole half of this compute and sometimes doing this over and over again. Incremental update and data checkpoints, it's a must-have in this data world. If you fail, you fix the bug, run it again, and catch up all the result that you have already. If you got more files in the bucket, you run the script again, and it gets only the new files and update data based on the new compute only without recomputing the other stuff. Running tests is a very fundamental part of coding agents. They do this all the time for quality control, for reasoning, to better understand what is around in their universe. It's kind of like a hands of your agents, hands of your harness. In data, running tests is super slow process. And how to do this? Because quality and accuracy of the questions, it's even more important in the data question, data projects, component to software projects. As Anthropic put it in theright way, in software projects, there's a lot of ways how to you can solve a particular problem. In data, there is usually only one way and only one current correct answer to solve the problem. How to answer questions very fast on your binary data mass. And first of all, you should stop running all this like complicated Python scripts on top of raw data. That's the most expensive, the slowest way of doing this. ### Metadata Layers **Dmitry Petrov** [20:25] Instead, you need to organize layer of meta information, all the like datasets or tables around this raw data that could answer this questions very quickly. And this is something that data industry knows for like a dozens of years. It's a dimensional, multi-dimensional data modeling, star schemas, one big table approach, and all this like fancy theories around this. And we need to use this more in order to make our unstructured messy data processing to be more efficient. And we actually incorporated these techniques inside the agents. And when you ask a question, instead of answering the questionright away, agent ask itself, do I have a proper datasets, proper metadata to answer this question quickly in a single like SQL-ish query? And if not, it tries to build this layer. It tries to make sure this layer is general enough to answer not your particular question, but a set of questions related to the one that you ask. And that's where you are building these layers and layers of information that could be reused by you and your teammates. Now you spent a lot of time, a lot of resources to compute those useful slices of the data, and you can use those to answer some interesting questions efficiently without running expensive compute. And guess what? People are doing the same job over and over and over again. You are paying double, triple, quadruple price to solve the same problem. So if you discover this Nitro star or black hole, it's better to share this information with people so they won't be wasting their resources and their time to doing this stuff. ### Knowledge Base **Dmitry Petrov** [22:38] And coding agent, as well as data agent, doing those tricks with memory a lot. Your coding agent like Copilot, Codex, PI knows a lot about your source code, about with all the indexes and such. On the data harness, you need to build and provide this context to your agents. And that's not only the fact that, hey, there is a dataset, you need to provide a lot more information with this. Why this dataset was built kind of like a context from the session, description of the dataset, which usually enriched by LLMs. Source code, probably the most important part here is the source code. That's one of the conclusion in OpenAI data agent blog post. And this information needs to be exposed in a way like knowledge base, some way that can easily be used by agents, by people, so you are not wasting this time over and over again. And the knowledge base is organized in a very traditional way, I would say. Just a set of MD files,right. So you see, this is how it looks like in my directory, but when I get more datasets. So the dataset that we created, that one, this is the MD file. So description of the dataset,right. The session context, so why this dataset was created based on the discussion, dependency to the storage, to the directory that we pointed to in the beginning, preview of the data, very useful information to kind of have a sense of the data, this schema, some stats on the data, and the source code. As we discussed, the most crucial part to understand what is data is about. And all the spaces together,right, the source data,right, in the bucket, the source code in the knowledge base, and the result in the data very house create like a data lineage,right. So everything is connected. Agents knows everything. If you share the knowledge graph, then all your teammates already know about this, about this dataset, about the resources you spent in order to process. So next time if someone asks question about this directory, the recompute won't happen. Agents will be using your result, the result which based on resources that you already that you have already spent,right. That's the magic of the data harnesses when agent knows everything about your data. Let's put all the pieces together in a single stack. ### The Stack **Dmitry Petrov** [25:58] In the bottom of the stack, there is some huge mass of your unstructured physical data in object storages. No one makes sense of this data, and you need to run expensive compute, LLM calls to extract some meta information through the compute engine and organize this meta information in a datasets, in a dataset slices through the some dataset DB. Knowledge base is a way how you share this information. This is the world when your favorite coding agents, such as CodePilot, Codex, Cloud Code, do not operate efficiently. Their intuition pushes them in a wrong direction because laws of physics changes. And in order to make it, you don't use stronger models. Everyone use frontiers. Instead, you are building data harness, data harness that understand the laws of this physical data. And that's the way how to make your favorite coding agent efficient with these problems. We implement some of those principles in DataChain project, which is open source. So please check it out and put some mass on your agent. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.