RAG at scale: production ready GenAI apps with Azure AI Search
Feb 13, 2025 · 21:53
Pablo, a Distinguished Engineer at Microsoft, explains how Azure AI Search enables production-grade RAG applications at scale by addressing key scaling challenges: data volume, query load, workflow complexity, and diverse data types. He demonstrates the platform's hybrid retrieval combining vector search (HNSW, exhaustive), keyword search (BM25), and filters, with a two-stage system using cross-encoder re-ranking for improved quality, achieving 100ms re-ranking latency. Azure AI Search supports multi-billion vector indexes through 10–12x storage limit increases and quantization (int8, single-bit) preserving 90–95% precision, with oversampling for full-precision re-ranking. Integrated vectorization automates ingestion from Azure sources (blob, Cosmos DB, OneLake) with change tracking and format handling (PDFs, Office documents, images). OpenAI uses Azure AI Search to back ChatGPT file uploads and the Assistants API, scaling user limits 500x after Microsoft's infrastructure upgrades.
