Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse
Jul 18, 2026 · 17:39
Annabell Schäfer, Growth Engineer at Langfuse, argues that successful auto-improvement loops require domain expertise and high-signal target functions, not just generic evaluators. She details an experiment classifying arXiv papers with a minimal loop using GPT-5 for nano and Claude Opus 4.8 as an optimizer, achieving a 15% accuracy jump from 68% to 83% in four iterations. The first iteration alone gained 10% by adding classification rules and examples based on error analysis of a 200-item dataset. Schäfer advises replacing vague metrics like correctness with yes/no quality criteria (e.g., 'answer uses knowledge base'), and working with domain experts to identify failure modes and define what 'good' means. She emphasizes validation to prevent overfitting and treating the system as generalizing from representative examples, not just burning tokens on endless loops.
