Building Effective Voice Agents — Toki Sherbakov + Anoop Kotha, OpenAI
Jul 20, 2025 · 17:17
Toki Sherbakov and Anoop Kotha from OpenAI argue that speech-to-speech models have reached a 'good enough' tipping point for production voice agents, highlighting the shift from chained architectures (transcription + LLM + TTS) to the real-time API's speech-to-speech approach. They detail trade-offs across latency, cost, accuracy, UX, and integrations depending on use case—consumer apps prioritize low latency and expressiveness, while customer service demands accuracy and tool integration. Key design patterns include delegating complex tasks to smarter models like O4 mini, prompting for voice expressiveness and tone, and starting with few tools. For evals, they recommend starting with observability, using SMEs for labeling, then transcription-based LLM-as-judge evals, audio evals with GPT-4 Audio to assess tone and pacing, and synthetic conversations between two real-time clients. Guardrails should run async with a configurable debounce period (e.g., every 100 characters). Examples from Lemonade (early focus on evals and guardrails) and Tinder (customization for brand realism) illustrate successful approaches.
