Arise Phoenix

Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents - Aparna Dhinakaran

Jun 10, 2025 · 14:25

Aparna Dhinakaran, co-founder of Arise, details how to build a self-improving evaluation stack for AI agents using her company's open-source tools Arise Phoenix and RiseX. She argues that production agents require three evaluation layers: tool-call correctness (right function and arguments), trajectory accuracy (correct order of steps), and multi-turn session consistency (context retention). Dhinakaran demonstrates with real traces from Arise's own Copilot, showing how a bottleneck in search Q&A correctness (only 50% accuracy) was traced to incorrect argument passing in a tool call. She stresses a dual iteration loop: improving agent prompts and simultaneously refining eval prompts (LLM-as-judge) to avoid static evaluation criteria. The talk concludes that continuous eval improvement is essential for agents to move from working once to working reliably in production.

The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

Jun 3, 2025 · 18:52

Jonathan Fernandes, independent AI engineer, details the RAG stack his team settled on after 37 failed attempts, covering orchestration (LlamaIndex), embeddings (BAAI BGE small), vector database (Qdrant), LLMs (GPT-4, Qwen, Llama), reranking (Cohere), monitoring (Arize Phoenix), and evaluation (RAGAS). He demonstrates a live prototype in Google Colab using a London railway knowledge base, showing how a naive RAG returns irrelevant results (e.g., suggesting black cabs for "where can I get help at the station"). By swapping components—replacing in-memory storage with Qdrant, using an open-source embedding model, upgrading to GPT-4, and adding Cohere reranking—the answer improves to "go to booth number five next to the Eurostar ticket gates." For production, he deploys via Docker Compose with NVIDIA embedding/reranking models and Ollama for serving. The episode also stresses the importance of tracing latency per component and using RAGAS for systematic evaluation across many queries.

Optimizing LLMs in Insurance with DSPy: Jeronim Morina

Feb 16, 2025 · 19:29

Jeronim Morina from AXA Germany argues that AI engineers must stop relying on manual prompt tuning and instead adopt DSPy's automated optimization combined with first-principles thinking. He explains how his team built a customer-facing insurance chatbot by first crafting clear problem definitions, annotating evaluation data, and avoiding data leakage. After initial struggles with fragile prompts and overly complex error-handling code, they modularized their system into DSPy modules and created custom metrics for German text. Morina emphasizes that tools like DSPy require a steep learning curve and a solid baseline of hand-written prompts and basic evaluations before optimization. The episode calls for engineers to focus on real-world impact, measure everything, and break down problems into discrete, optimizable steps.