Prompt Engineering is Dead — Nir Gazit, Traceloop
Jun 27, 2025 · 14:19
Nir Gazit, CEO of Traceloop, argues that manual prompt engineering is ineffective and demonstrates an automated, test-driven alternative. He improves a RAG-based documentation chatbot by 5x without hand-tuning prompts, using an LLM-as-a-judge evaluator with 20 example questions (each with 3 expected facts) and a CrewAI agent that iteratively optimizes the prompt based on evaluation scores. The initial score of 0.4 jumps to 0.9 after just two iterations. Gazit warns of overfitting to the small dataset and suggests splitting into train/test sets. The approach replaces manual tweaking with a 'gradient ascent' on evaluators, effectively vibe-engineering prompts.
