YOLO

When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain

Jul 20, 2026 · 27:33

Dmitry Petrov of DataChain argues that AI agents fail on unstructured physical data because their intuitions assume cheap recompute, while large-scale video, sensor, and robot data requires a different harness. He cites Anthropic's finding that agents achieve only 21% accuracy on data projects without specific data harnesses, and OpenAI's need for six layers of context even on structured data. Petrov demonstrates DataChain, an open-source Python framework that uses Pydantic schemas to turn messy binary files into queryable databases, an execution engine for distributed processing, incremental checkpoints to avoid recomputing on failure, and a knowledge base of datasets and source code so agents can answer follow-up questions in seconds instead of reprocessing terabytes. In a live demo with Claude Code analyzing 90 dashcam videos, the harness took 24 minutes to extract 100,000 object records, then instantly answered 'how many clips have people?' without rerunning inference. Petrov emphasizes that the key is organizing metadata into star schemas and sharing data lineage across teammates so no one pays the compute cost twice.

Moondream: how does a tiny vision model slap so hard? — Vikhyat Korrapati

Nov 14, 2024 · 19:26

Vikhyat Korrapati built Moondream, a tiny open-source vision language model under 2 billion parameters that matches LLaVA 1.5, a model four times its size, on VQA v2 and GQA benchmarks. He attributes this to focusing on image understanding over world knowledge and investing heavily in synthetic training data—a pipeline that generated 35 million images and used two orders of magnitude more compute on data than training. Key lessons: community engagement was critical, open-source builds trust, and safety guardrails should be application-layer, not baked in. He argues tiny models will dominate production due to cost and privacy advantages, and that prompting will replace custom model training for most vision tasks. Moondream raised a seed round from FullySysAscent and the GitHub Fund.