Aparna Dhinakaran

The Unreasonable Effectiveness of Prompt Learning – Aparna Dhinakaran, Arize

Dec 23, 2025 · 10:56

Aparna Dhinakaran, co-founder of Arize, argues that prompt learning—applying RL techniques to prompts rather than model weights—can continuously improve coding agents by auto-tuning system prompts from runtime feedback. She details a process using Claude Code and CLIMB on the SWE-bench dataset: agents generate patches, unit tests yield results, then an LLM-as-a-judge eval produces natural-language explanations of failures. These explanations feed a meta prompt that iterates on the system prompt rules. On 150 SWE-bench examples, this approach improved Claude Code's issue-resolution rate by 5% and CLIMB's by 15%. She contrasts their method with DSPy's GEPA optimizer, noting theirs required fewer loops due to carefully engineered eval prompts.

Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents - Aparna Dhinakaran

Jun 10, 2025 · 14:25

Aparna Dhinakaran, co-founder of Arise, details how to build a self-improving evaluation stack for AI agents using her company's open-source tools Arise Phoenix and RiseX. She argues that production agents require three evaluation layers: tool-call correctness (right function and arguments), trajectory accuracy (correct order of steps), and multi-turn session consistency (context retention). Dhinakaran demonstrates with real traces from Arise's own Copilot, showing how a bottleneck in search Q&A correctness (only 50% accuracy) was traced to incorrect argument passing in a tool call. She stresses a dual iteration loop: improving agent prompts and simultaneously refining eval prompts (LLM-as-judge) to avoid static evaluation criteria. The talk concludes that continuous eval improvement is essential for agents to move from working once to working reliably in production.

Ensure AI Agents Work: Evaluation Frameworks for Scaling Success — Aparna Dhinkaran, CEO Arize

Apr 23, 2025 · 15:28

Aparna Dhinakaran, CEO of Arize AI, explains that evaluating AI agents requires testing three core components—routers, skills, and memory—each at different trace levels. Routers must be checked for correct skill selection and parameter passing; skills need LLM-as-judge or code-based evals for chunk relevance and answer correctness; and convergence measures whether the agent takes a consistent number of steps to complete a task. For voice agents, additional evaluations on audio chunks—sentiment, speech-to-text accuracy, and tone consistency—are necessary. Dhinakaran demonstrates Arize's own Copilot, where evals run at every trace step (router choice, argument passing, task completion) to isolate failures. She argues that observability-driven evaluation frameworks, with multiple eval layers, transform experimental agents into production-ready enterprise tools.

Lessons from the Trenches: Building LLM Evals That Work IRL: Aparna Dhinkaran

Feb 6, 2025 · 18:49

Aparna Dhinakaran, co-founder of Arize AI, distinguishes between model evals (e.g., Hugging Face leaderboard) and task evals for real-world LLM systems, arguing that production applications need component-level evaluations like router and parameter evals. She demonstrates a chat-to-purchase app where a router function call misidentifies user intent, showing how Phoenix open source tool traces errors and provides explanations to iterate. Dhinakaran advises using categorical over numeric LLM-as-judge scores because numeric outputs tend to be binary (0 or 10) and lack granularity. Presenting needle-in-haystack research, she notes GPT-4 struggles retrieving facts placed early in large context windows, and in retrieval-with-generation tasks, Anthropic’s Claude 2.1 outperforms GPT-4 due to verbose reasoning, a gap closed by prompting GPT-4 to explain itself first.