Voice Agents: the good, the bad, and the ugly
Feb 22, 2025 · 18:48
Eddie Siegel, CTO at Fractional AI, details the real-world challenges of building an AI voice agent for consulting-style interviews, moving from proof of concept to production. The agent, designed to conduct hundreds of research interviews simultaneously, struggled with hallucinations, latency, and Whisper transcription errors like 'creeping dippity dippity dippity dippity'. To constrain the LLM's tendency to chitchat and rabbit-hole, Siegel's team introduced a 'drift detector' agent and a 'next question' agent, both running as separate text-based side threads. They also added tool use for question navigation, explicit goals and priorities for follow-ups, and a 'transcript cleaner' to hide garbled outputs from the user interface. To move beyond vibes-driven iteration, they built an automated eval suite with metrics like clarity and completeness, using synthetic conversations with personas such as a 'snarky teenager in charge of a Fortune 500 company' to simulate diverse interviewees. The talk offers a no-nonsense playbook for getting voice agents into production, focusing on out-of-band checks, tool use, and evals even without objective ground truth.
