The Hierarchy of Needs for Training Dataset Development: Chang She and Noah Shpak
Oct 15, 2024 · 16:32
Chang She (CEO of LanceDB) and Noah Shpak (AI data platform lead at Character AI) argue that data infrastructure is the critical bottleneck for LLM training, and LanceDB's columnar format solves the 'new cap theorem' for AI: needing fast scans, random access, and handling large multimodal blobs simultaneously. Noah explains how Character AI structures pre-training around wide domain coverage and post-training around granular analytics like token counts and difficulty scores, using synthetic data, quality scoring, and dataset selection to improve models. Chang details how Lance format provides zero-copy schema evolution, time travel, and indexing extensions for vector, scalar, and full-text search, enabling a single table to serve SQL analytics, PyTorch training, and production vector search. The episode emphasizes that speed and iterative dataset management are key to accelerating AI research, with LanceDB facilitating cheap random access and low-infra billion-scale vector search.
