Elastic

User Signal Dies at the Retrieval Boundary - Sonam Pankaj, StarlightSearch

Jun 28, 2026 · 15:37

Sonam Pankaj, CEO of StarlightSearch, argues that production agents fail because retrieval is static and eval signals never cross into runtime context. She introduces utility-ranked memory, where memories are re-ranked by a utility score combining semantic similarity with outcome history—passing runs raise a memory's score, failing runs lower it. In a product SQL agent demo, a search for 'gaming mouse' initially failed, but after marking that output as a failure and noting a wireless mouse was relevant, the agent's trajectory updated in real time to find the correct product. Benchmarks on Tao Bench show reflect memory improves performance from 66% to 76% without skills and 80% with skills; on agentic tasks, reflect achieves 61.3% versus 35.7% baseline and 58.2% with other memory systems. Pankaj explains that after enough reviews, memories can be baked into skills, allowing continuous improvement without manual prompt rewriting or model fine-tuning.

Agentic Search for Context Engineering — Leonie Monigatti, Elastic

May 8, 2026 · 1:03:13

Leonie Monigatti from Elastic argues that context engineering is 80% agentic search—the search tool that decides what to pull from files, databases, memory, and the web. She identifies three failure modes: the agent not calling any tool, calling the wrong tool, or generating incorrect parameters, and shows how detailed tool descriptions and agent skills (progressive disclosure) reduce these. In demos, semantic search fails for keyword 'JEPA' due to embedding similarity; a general-purpose ESQL tool with an agent skill correctly queries the database. Shell/bash tool enables file system search but requires iterative grep, while the custom Gina Grap CLI provides semantic grep for fuzzy queries. Practical recommendations: start with general-purpose tools, log agent behavior, and add specialized tools for frequent queries to balance low floor (easy successes) and high ceiling (complex queries). Hybrid agents combining shell and database tools achieve higher accuracy by verifying results.

Information Retrieval from the Ground Up - Philipp Krenn, Elastic

Jul 27, 2025 · 1:48:07

Philipp Krenn of Elastic demonstrates that vector search is only a feature of information retrieval, not its foundation, in this hands-on workshop on retrieval for RAG. He contrasts classic keyword search—using tokenization, stemming, stopword removal, and BM25 scoring with the inverted index—with sparse (Elser/Splade) and dense (OpenAI text-embedding-small) vector embeddings. Krenn explains why hybrid search, combining lexical and semantic methods via reciprocal rank fusion (RRF), outperforms any single approach, especially for brand or exact-match queries where keyword search remains superior. He also covers scoring normalization, chunking strategies, and the new `text_similarity_reranker` retriever, showing how Elasticsearch's retrievers API enables two-stage retrieval with a cross-encoder for re-ranking. The session emphasizes that retrieval quality depends on evaluating against a golden dataset or LLM-as-judge, and that 'it depends' is the correct answer for choosing between PgVector and dedicated search engines.

Production software keeps breaking and it will only get worse — Anish Agarwal, Traversal.ai

Jul 10, 2025 · 18:13

Anish Agarwal and Matthew Schoenbauer of Traversal.ai argue that as AI writes more code, production troubleshooting will become vastly harder, requiring a new approach combining causal machine learning, reasoning models, and agentic swarms to autonomously resolve incidents in minutes. They explain that traditional AI ops generates too many false positives, LLMs can't handle petabyte-scale data, and simple agents depend on deprecated runbooks. Their Traversal AI orchestrates thousands of parallel agentic tool calls to sift through trillions of logs and metrics, identifying root causes and citing observability data. A case study with DigitalOcean shows a 40% reduction in mean time to resolution (MTTR), with the system delivering findings in about five minutes. The episode details how this approach turns frantic incident Slack channels into autonomous, cited root-cause analysis, freeing engineers to focus on system design.

Vector Search Benchmark[eting] - Philipp Krenn, Elastic

Jun 27, 2025 · 14:10

Philipp Krenn from Elastic dissects 'benchmarketing' and explains why most vector search benchmarks are unreliable due to selective scenarios, outdated competitor versions, and omitted quality metrics like precision-recall. He highlights that read-only benchmarks don't reflect real workloads, filtering can slow HNSW-based search, and implicit biases favor the benchmarker's own system. Krenn advises building automated, reproducible benchmarks (like Elastic's nightly Rally tool) to avoid the 'boiling frog' problem of gradual performance degradation. He concludes that only running your own tailored benchmarks yields trustworthy results, urging listeners to learn from even flawed benchmarks rather than dismissing them entirely.