Scaling Enterprise-Grade RAG: Lessons from Legal Frontier - Calvin Qi (Harvey), Chang She (Lance)
Jul 29, 2025 · 16:40
Calvin Qi of Harvey and Chang She of LanceDB detail how Harvey builds enterprise-grade RAG for legal use cases, tackling massive scale (tens of millions of documents per corpus), spikey workloads, and non-negotiable privacy. They explain why domain-specific reranking is essential for ambiguous, expert queries like 'What is the applicable regime to covered bonds issued before 9 July 2022 under Directive EU 2019/2062?' and stress that evaluation-driven development—from expert reviews to automated metrics—is key to accuracy. Chang shows how LanceDB's AI-native multimodal lakehouse, built on the open-source Lance format, enables low-latency, high-throughput retrieval across billions of vectors (indexing 3–4 billion in under 3 hours) while supporting offline analytics, GPU indexing, and compute-storage separation. The talk emphasizes that creative data understanding, iteration speed via good evals, and flexible infrastructure are critical for scaling RAG in domain-specific, privacy-sensitive environments.
