Executive Summary
If you're putting a voice agent in front of customers this year, this guide is about the part that's hardest to budget for: proving the agent actually works. The one-line version: a voice agent that passes every text-style metric can still fail badly on a live call, because voice failures are temporal and acoustic, not textual. Automated simulation is the only way to test at useful scale, and it still isn't enough on its own. Human-in-the-loop review is what closes the remaining gap.
We'll cover the metrics worth tracking on two axes (audio performance and conversational quality, each with a reference table), how architectural choices between chained and speech-to-speech systems change what you can observe, how automated platforms run simulated callers against your agent, and a structured qualitative method (open and axial coding) for turning real user failures into permanent test cases. The reader I have in mind is the engineer or product owner responsible for a voice agent's quality; you don't need a speech background to follow along. The running example is a deliberately silly tech-horoscope agent, which keeps the evaluation mechanics visible without any domain complexity in the way.