Training Data Curation

How We Built Zeta2: Training an Edit Prediction Model in Production — Ben Kunkle, Zed

May 30, 2026 · 10:50

Ben Kunkle, edit predictions lead at Zed, explains how they built Zeta2, a small specialized model for edit prediction in production. The pipeline pulls opt-in production edit traces, distills them through a frontier teacher, and routes bad predictions through a repair step before formatting for the student. To validate settled data, Zed originally ran 10 frontier model predictions per example and measured Levenshtein distance to the final state, but for 100,000 training examples that is a million frontier model requests — prohibitively expensive. The fix: Zeta2's student model now approaches teacher quality, so they run it 50 times instead at negligible cost. Ideal training examples sit in the middle of the Levenshtein distance distribution: too close to the settled state is obvious, too far is noise. A metric called reversal ratio — how often the model undoes exactly what the user just typed — was the key diagnostic for catching bad model behavior before shipping.

Z.ai GLM 4.6: What We Learned From 100 Million Open Source Downloads — Yuxuan Zhang, Z.ai

Nov 22, 2025 · 19:39

Yuxuan Zhang from zAI details the technical roadmap behind the GLM 4.6 open-source model series, which has surpassed 100 million downloads and tied for number one on the LMSYS Chatbot Arena alongside GPT-4o and Claude 3.5 Sonnet. The training pipeline uses 15 trillion tokens of pre-training data, followed by 7 trillion tokens of code and reasoning data, repo-level contexts at 32,000 tokens, and 100 billion tokens of long-context agent trajectories up to 128,000 tokens. Zhang introduces SLIME, a hybrid synchronous/asynchronous RL framework that decouples agent-environment interaction from GPU training to avoid bottlenecks. He explains why single-stage RL at 64,000 tokens outperforms multi-stage approaches for preserving long-context abilities, and shows that token-weighted loss converges faster than sequence-average loss for code RL. The multimodal GLM 4.5V model handles native resolution images and video with temporal index tokens, enabling GUI agent capabilities. Deployment is supported via vLLM and SGLang, with an API at z.ai and an open-source coding assistant.

Robotics: why now? - Quan Vuong and Jost Tobias Springberg, Physical Intelligence

Jul 26, 2025 · 18:07

Physical Intelligence's Quan Vuong and Jost Tobias Springberg describe their mission to build a model that can control any robot to do any task, arguing that software intelligence is the main bottleneck in robotics. They explain Vision Language Action models (VLAs) as adaptations of vision language models that output robot actions instead of text. To train these models, they built a data engine from scratch, collecting 10,000 hours of successful episodes via teleoperation in six months. Their latest model, PAIO-5, achieves open-world generalization by training on data from multiple homes, matching or surpassing performance on held-out scenes. They demonstrate this with a policy that performs long-horizon tasks like cleaning an unseen bedroom for up to 10 minutes autonomously. They also highlight a remote coffee-making demonstration on a robot they never touched, showing model portability across hardware.

A Taxonomy for Next-gen Reasoning — Nathan Lambert, Allen Institute (AI2) & Interconnects.ai

Jul 19, 2025 · 19:21

Nathan Lambert, senior research scientist at AI2 and founder of Interconnects.ai, argues that next-generation reasoning models require a taxonomy of four traits: skills, calibration, strategy, and abstraction. While current models excel at math and code (skills), they overthink easy problems, wasting tokens and latency—calibration is needed to match output length to task difficulty. The real frontier is planning: models must learn strategy (choosing the right direction) and abstraction (breaking problems into tractable sub-tasks) to enable long-horizon agents like Deep Research and Claude Code. Lambert traces how OpenAI's Q*→Strawberry→o1 took 12–18 months of human data to teach backtracking and verification; planning should be easier because humans can write 5-step plans. He predicts post-training compute could reach parity with pre-training, citing DeepSeek's shift from 0.18% post-training compute in V3 to an estimated 10–20% for R1. The path forward: collect diverse verifiable questions, filter by difficulty, run stable RL—then scale.

OpenThoughts: Data Recipes for Reasoning Models — Ryan Marten, Bespoke Labs

Jul 19, 2025 · 19:59

Ryan Marten, co-lead of the OpenThoughts collaboration and founding engineer at Bespoke Labs, reveals the missing data recipe for open-source reasoning models, presenting OpenThoughts 3, a state-of-the-art 7B reasoning dataset that outperforms DeepSeek R1 Qwen 7B and Nematron Nano on benchmarks like AIME, Live Code Bench, and GPQA Diamond. Through over 1,000 experiments and 5,000 datasets, key findings include that sampling multiple reasoning traces per question scales performance by 16x, Qwen 32B surpasses DeepSeek R1 as a teacher model, synthetic question generation is highly effective, and filtering by difficulty or response length works better than embeddings. Surprisingly, verification of answers in SFT distillation did not improve results, and focusing on fewer high-quality sources outperformed maximizing diversity. For domain-specific reasoning, Marten advises starting with the OpenThoughts recipe, using synthetic data generation (via the open-source Curator library), and rigorous evaluation (via EvalComet). A legal reasoning example shows that distillation can surpass the teacher model. All resources are open-source.

Training Albatross An Expert Finance LLM: Leo Pekelis

Feb 13, 2025 · 16:20

Leo Pekelis, chief scientist at Gradient, explains how they transformed an open-source model into Albatross, a finance LLM that tops leaderboards on both general and domain-specific tasks. The key was an automated data pipeline using membership inference to curate finance data from a massive corpus, followed by continual pre-training and alignment via supervised fine-tuning and preference optimization. He also details a one-million-token context extension on a Llama 3-based model that achieves 100% needle-in-the-haystack scores, enabling in-context learning with thousands of examples to reduce hallucinations. The models, v-alpha-tross and the extended-context Llama 3, are open-sourced on Hugging Face.

The GenAI Maturity Curve or You Probably Don't Need Fine Tuning: Kyle Corbitt

Feb 9, 2025 · 18:03

Kyle Corbitt, CEO of OpenPipe, argues that most teams don't yet need fine-tuning and should start with prompted models like GPT-4. He presents a GenAI maturity curve where the trigger to fine-tune is when you hit constraints on cost, latency, or quality consistency—for example, if GPT-4 is 80-90% correct but inconsistent on the last 10-20%. Fine-tuning shifts the paradigm frontier outward, enabling models like fine-tuned LLaMA 38B to outperform GPT-4 at 1/25th the cost. The process has four steps: capture production logs to know your input distribution, prepare high-quality data (using GPT-4 outputs or iterative labeling), train with one-click tools, and evaluate with inner-loop (LLM-as-judge) and outer-loop (business metrics) evals. OpenPipe and other providers make deployment trivial via OpenAI-compatible APIs. The talk delivers a concrete decision framework and a walkthrough so any engineer can fine-tune in under an hour.

LLM Quality Optimization Bootcamp: Thierry Moreau and Pedro Torruella

Feb 8, 2025 · 53:05

Thierry Moreau of OctoAI demonstrates how to fine-tune Llama 3 8B on a PII redaction task using OpenPipe and OctoAI, achieving 47% better accuracy and a 200x cost reduction (from $30 to $0.15 per million tokens) compared to GPT-4 Turbo. He explains that fine-tuning should follow prompt engineering and RAG, and works best for specialized tasks like function calling. The talk walks through building a fine-tuning dataset from the PI Masking 200k dataset, using OpenPipe to train a LoRA for $40, deploying it on OctoAI, and evaluating it to show the fine-tuned model scores 0.97 accuracy versus GPT-4’s 0.68. Moreau emphasizes that this continuous deployment cycle requires monitoring data drift and retraining, but tools like OpenPipe and OctoAI make it accessible even for teams without deep ML expertise.

The Hierarchy of Needs for Training Dataset Development: Chang She and Noah Shpak

Oct 15, 2024 · 16:32

Chang She (CEO of LanceDB) and Noah Shpak (AI data platform lead at Character AI) argue that data infrastructure is the critical bottleneck for LLM training, and LanceDB's columnar format solves the 'new cap theorem' for AI: needing fast scans, random access, and handling large multimodal blobs simultaneously. Noah explains how Character AI structures pre-training around wide domain coverage and post-training around granular analytics like token counts and difficulty scores, using synthetic data, quality scoring, and dataset selection to improve models. Chang details how Lance format provides zero-copy schema evolution, time travel, and indexing extensions for vector, scalar, and full-text search, enabling a single table to serve SQL analytics, PyTorch training, and production vector search. The episode emphasizes that speed and iterative dataset management are key to accelerating AI research, with LanceDB facilitating cheap random access and low-infra billion-scale vector search.

Domain adaptation and fine-tuning for domain-specific LLMs: Abi Aryan

Nov 14, 2023 · 25:09

Abi Aryan's talk covers domain adaptation and fine-tuning for large language models, contrasting prompting, RAGs, and three fine-tuning methods: adaptive, behavioral, and parameter-efficient. Adaptive fine-tuning adds small adapter modules (0.15% of parameters) for new domains like biochemical engineering; behavioral fine-tuning optimizes label space for a single task; and parameter-efficient methods like LoRA and QLoRA reduce model size via low-rank adaptation and four-bit precision, ideal for low-resource devices. Aryan emphasizes data quality—deduplication reduces memorization—and practical tips: batch size of 32 or 64, starting with 100 epochs, using Adam optimizer, gradient checkpointing for memory savings, and in-context learning with dynamic examples to handle drift. Evaluation should combine metric-based (Bleu, Rouge), tool-based (Weights & Biases), model-based, and human-in-the-loop approaches, though full pipeline considerations (data collection, base model choice, storage) are critical for robust applications.