Deanna Emery

Evaluating AI Search: A Practical Framework for Augmented AI Systems — Quotient AI + Tavily

Jul 29, 2025 · 20:33

Julia Neagu (CEO of Quotient AI), Deanna Emery (founding AI researcher at Quotient AI), and Maitar Asher (Head of Engineering at Tavily) argue that traditional monitoring approaches like static benchmarks (e.g., SimpleQA) fail to capture the complexity of production AI search because the web is dynamic and truth is subjective. They present a framework combining dynamic datasets generated via a LangGraph-based agent that creates evidence-based Q&A pairs from multiple search providers, and holistic evaluation using three reference-free metrics: answer completeness, document relevance, and hallucination detection. In a two-part experiment comparing six anonymized AI search providers on a static (SimpleQA) and a dynamic benchmark of ~1,000 rows, ranking shifted significantly—provider F went from worst on SimpleQA to best on the dynamic benchmark. Reference-free answer completeness correlated 0.94 with correctness, and document relevance showed an inverse correlation with unknown answers. The speakers stress that evaluation should diagnose issues (e.g., incomplete answers with relevant documents suggest need for more retrieval) and guide improvements, ultimately aiming for…

Navigating RAG Optimization with an Evaluation Driven Compass: Atita Arora and Deanna Emery

Nov 12, 2024 · 18:14

Atita Arora (Qdrant) and Deanna Emery (Quotient AI) demonstrate how evaluation-driven optimization improves Retrieval Augmented Generation (RAG) systems, using Qdrant's vector database and Quotient's evaluation platform. They detail a 10-experiment pipeline on Qdrant documentation, starting with naive RAG and incrementally testing chunk sizes, embedding models, LLMs (Mistral, GPT-3.5), re-rankers (MixedBread, Cohere, ColBERT), and hybrid search. Key metrics—faithfulness (hallucination reduction), context relevance, and chunk relevance—guide decisions: increasing chunk size hurt faithfulness (0.76 to 0.72), while smaller chunks with larger retrieval windows improved it. Switching to GPT-3.5 lifted all metrics, but poor context relevance revealed retrieval issues. Adding Cohere re-ranking boosted faithfulness to 0.82, and hybrid search with re-ranking achieved 0.85, proving domain-specific terminology demands tailored search strategies. The episode stresses avoiding over-engineering without metrics, keeping evaluation datasets current, and using domain understanding for systematic RAG improvement.