"Our AI handled 10,000 conversations" sounds good and tells you very little. Handled how? Correctly? Did those customers buy, come back angry, or ask again on WhatsApp? Measuring an AI agent well means looking at outcomes and quality, not just volume.
The core metrics
AI resolution rate
The share of conversations the agent closes without a person. It's the headline number, with two caveats:
- Count a conversation as resolved only if the customer didn't come back about the same thing within a few days.
- Look at it by intent. 95% on shipping questions and 20% on returns tells you where to work.
Handoff rate and reasons
How often the agent passes to a person, and why: customer asked, value threshold, sensitive topic, no source found. The "no source found" group is your improvement list.
Accuracy, by review
No dashboard measures accuracy for you. Each week, review a random sample of AI-resolved conversations and mark each one correct, partly correct or wrong. Twenty to thirty a week is enough to spot problems early.
First reply time
How long customers wait for the first response, AI or human. With an agent this should be seconds; watch the time to first human reply after a handoff too.
Customer satisfaction
A one-question survey after the conversation, split by AI-resolved and human-resolved. If the two are far apart, read the low scores.
Commercial metrics
For pre-sale conversations, track:
- Conversations that included a product recommendation
- Add-to-cart and orders after a conversation
- Leads captured and booked calls
Be careful with attribution. A conversation before an order doesn't mean the conversation caused it. Compare similar shoppers with and without a conversation, or run a holdout on some pages, before claiming lift.
Read sessions, not just charts
Averages hide the interesting part. When a number moves, open the conversations behind it. Session traces that show the question, the sources retrieved and the steps taken let you see why the agent did what it did. That's what LiveCart's Analyze view is for.
A simple weekly routine
- Check resolution and handoff rates by intent.
- Read ten handed-off conversations and note missing sources.
- Review twenty AI-resolved conversations for accuracy.
- Fix sources or workflows, then publish a new version.
- Compare next week.
Thirty minutes a week, consistently, will improve an agent more than any one-time setup.
