Vector Search Benchmark[eting] - Philipp Krenn, Elastic
Jun 27, 2025 · 14:10
Philipp Krenn from Elastic dissects 'benchmarketing' and explains why most vector search benchmarks are unreliable due to selective scenarios, outdated competitor versions, and omitted quality metrics like precision-recall. He highlights that read-only benchmarks don't reflect real workloads, filtering can slow HNSW-based search, and implicit biases favor the benchmarker's own system. Krenn advises building automated, reproducible benchmarks (like Elastic's nightly Rally tool) to avoid the 'boiling frog' problem of gradual performance degradation. He concludes that only running your own tailored benchmarks yields trustworthy results, urging listeners to learn from even flawed benchmarks rather than dismissing them entirely.
