Apache Lucene

How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust

May 28, 2026 · 20:43

Phil Hetzel of Braintrust argues that agent observability differs fundamentally from traditional observability, which only answers 'is the system up?' Agents are non-deterministic, with individual traces exceeding a gigabyte and spans reaching 20 megabytes, packed with unstructured text that traditional tools like Datadog or Grafana cannot handle. Braintrust built a custom database from scratch—featuring a write-ahead log for instant visibility, analytical indexes for filtering, and a forked Rust-based Tantivy index for full-text search—to ingest and query this data in real time. Unlike traditional observability, agent observability involves non-technical experts (e.g., clinicians, lawyers) who grade traces and write justifications that become training signals for automated scoring functions. Braintrust also runs lightweight LLMs on traces to perform clustering and topic modeling, surfacing user intent, sentiment, and failure modes to accelerate the iteration loop between production issues and fixes. The episode also covers Braintrust's shift from ClickHouse to a custom database due to ClickHouse's limitations in text indexing.

Information Retrieval from the Ground Up - Philipp Krenn, Elastic

Jul 27, 2025 · 1:48:07

Philipp Krenn of Elastic demonstrates that vector search is only a feature of information retrieval, not its foundation, in this hands-on workshop on retrieval for RAG. He contrasts classic keyword search—using tokenization, stemming, stopword removal, and BM25 scoring with the inverted index—with sparse (Elser/Splade) and dense (OpenAI text-embedding-small) vector embeddings. Krenn explains why hybrid search, combining lexical and semantic methods via reciprocal rank fusion (RRF), outperforms any single approach, especially for brand or exact-match queries where keyword search remains superior. He also covers scoring normalization, chunking strategies, and the new `text_similarity_reranker` retriever, showing how Elasticsearch's retrievers API enables two-stage retrieval with a cross-encoder for re-ranking. The session emphasizes that retrieval quality depends on evaluating against a golden dataset or LLM-as-judge, and that 'it depends' is the correct answer for choosing between PgVector and dedicated search engines.