Why 'automation rate' is not a goal, and what is?

Almost every AI customer support vendor throws the number in your face: 73% of your tickets get automated. 87%. 92%. The message is clear, higher is better. But spend a little longer thinking about it and you see that automation rate as a KPI leaks like a sieve.

Our claim is simple: automation rate is a measurement, not a goal. Turn it into a goal and you optimise for the wrong thing, losing exactly what customer support exists for, trust.

Why the KPI leaks

Automation rate is defined as 'the percentage of conversations resolved without human intervention'. That definition hides three problems.

1. What counts as 'resolved'? A conversation where the bot says 'I can't help you with that, please contact support' technically has zero human involvement, but it resolves nothing.
2. What counts as a 'conversation'? Someone who asks at 11pm where their order is and gets the right answer counts as one. Someone with a complex warranty claim that crosses four departments over three sessions counts as three. The hard cases, the ones that actually matter, weigh in one-for-one with the easy ones.
3. What don't we count? People who give up after a single bot message and never come back. For the bot that's a success (no handoff), for your business it's a lost customer.

The three KPIs that actually say something

At Conveya we look at three metrics in combination. None of them says enough on its own, together they tell the real story.

1. Resolution rate (per channel, per category)

Not 'did the conversation end without a human', but 'did the customer get what they were looking for'. Operationally you measure that by sending a single-question survey 24 hours after the conversation: 'Did you get an answer to your question?', yes/no/partly. Combine that with the number of cases reopened within 7 days.

2. CSAT of automated conversations (versus human conversations)

After a conversation, ask for satisfaction on a 1-5 scale. Compare the AI conversations with your human conversations in the same period. A healthy benchmark: AI CSAT should sit at most 0.3 points below human CSAT. Below that you start losing goodwill.

3. Cost-per-resolution

All costs (LLM tokens, infra, human time on escalations) divided by the number of resolved cases. This is the KPI your CFO understands. And it's the only one that shows whether automation saves money or just shifts the work around.

How to set up the right KPI mix

1. Split your cases by category (order status, refund, product question, other). Each category has a different optimal automation rate. 'Order status' can and should be 95%. 'Refund' belongs somewhere between 30-50% because that's where you want human judgement.
2. Set a minimum CSAT gate. As soon as the CSAT of a category drops below a threshold (we use 4.2 out of 5), the bot goes back to a supporting role instead of handling cases end to end.
3. Measure weekly, not daily. Daily fluctuations say nothing and lead to panic tuning.
4. Stop paying VP bonuses on automation rate. Pay them on resolution rate × CSAT × cost reduction. Then you get a team that optimises for the right thing.

What this means for your vendor choice

If a vendor waves automation rate around in the sales pitch without the resolution + CSAT + cost components alongside it, you know enough. Ask for all four numbers as standard in your RFP, and refuse demos that show only the automation percentage. The vendors who can deliver will show it. The rest won't.