An AI resolution rate usually counts conversations where the customer did not ask for more help after the AI’s answer, within a time window the vendor chooses. It does not check that the answer was correct. A customer who gives up, finds the answer elsewhere or simply stops replying can count as resolved. Treat the number as a vendor-defined activity count, and judge quality on replies you can check yourself.
What does a resolution rate actually count?
Each vendor defines “resolution” itself, and the definitions differ. Two examples from vendors’ own documentation:
- Intercom (Fin). A resolution is counted when, after Fin’s last answer, “the customer either confirms the answer was satisfactory (confirmed resolution), or exits the conversation without requesting further assistance (assumed resolution)”. Its help page adds: “A customer leaving after receiving Fin’s answer without requesting further help is counted as an Assumed Resolution and will be charged.” (Intercom Help: Fin outcomes, checked 30 September 2026.)
- Zendesk. Its documentation for the automated-resolution platform in place before 18 May 2026 counts a messaging resolution “after 2 hours of inactivity” if an AI evaluation confirmed the response was relevant and the last interaction was one of a listed set, which includes the customer giving positive feedback or no feedback at all; for email, the conversation ends 72 hours after the first email. Resolutions are “verified by a large language model” (Zendesk help: About the automated resolutions platform (prior to resolution tiers), checked 30 September 2026). Zendesk has since introduced resolution tiers, so check the current terms for your own account.
Both definitions share one idea: a conversation is resolved when the customer does not come back for more help. That is a reasonable billing rule. It is a weak quality measure.
Why can a wrong answer still count as resolved?
Because silence is ambiguous. A customer stops replying for many reasons:
| What really happened | Counted as |
|---|---|
| The answer was right and complete | Resolved |
| The answer was wrong, and the customer gave up | Resolved |
| The customer found the answer on your website instead | Resolved |
| The customer phoned or emailed a person instead | Resolved, unless that contact was linked to the conversation |
| The customer acted on a wrong answer and complains a week later in a new thread | Resolved, often |
| The customer asked a follow-up question | Not resolved |
Only the first and last rows are what the number is meant to capture. A tool that gives a confident wrong answer and ends the conversation can score better than one that honestly says “I don’t know, let me pass you to someone”.
Why are vendor averages hard to compare?
A published resolution rate is an average across many customers, counted under that vendor’s own rules. Before comparing two numbers, you would need to know:
- The definition. Does silence count, or only confirmed resolutions?
- The time window. Two hours of inactivity and three days of inactivity give different numbers for the same conversations.
- The mix of questions. A shop full of “where is my parcel?” questions resolves more than a software company with technical questions.
- Who is counted. All conversations, or only those the AI attempted? Are conversations handed to a person removed from the base?
- Who chose the sample. A headline number on a website may be a best case rather than a typical customer.
Without those answers, two resolution rates are two different measurements that happen to share a name.
What does this mean if you answer customers by email?
Much of this debate is about chat widgets that answer on their own. If you use AI to draft email replies that a person sends, the resolution rate barely applies: a person decides what goes out, and a wrong draft that gets corrected never reaches the customer.
What you need to know instead is how much work the drafts save and how often they are wrong. Both can be counted in your own inbox, without trusting anyone’s definition.
Which numbers should a small business track instead?
Numbers you can count yourself, from your own mailbox, week by week:
| Measure | How to count it | What it tells you |
|---|---|---|
| Sent unchanged | Drafts sent without any edit, out of all drafts sent | How much writing the tool saves |
| Fact corrections | Drafts where you changed a number, date, promise or policy | How often the tool is wrong where it matters |
| Follow-up rate | Threads where the customer wrote back with a question about your answer | Whether your replies are complete |
| Reopened threads | Customers who come back about the same problem within a week or two | Whether the problem was actually solved |
| Coverage | Customer emails that got a draft, out of all customer emails | How much of your inbox the tool handles |
The sent unchanged share is the most useful single number, because it combines quality and time saved and nobody can inflate it but you. The “sent unchanged” rate explains how to count it and how to read it.
What should you ask a vendor who quotes a resolution rate?
Five questions that turn a headline into something you can judge:
- How exactly do you define a resolution, and does a customer who stops replying count?
- What is the time window before a conversation counts as resolved?
- Is the number an average across all customers, or a selected example?
- What share of resolved conversations reopen, or come back through another channel?
- Can I see the individual conversations behind my own number?
A vendor that answers all five clearly is giving you something you can use. If the answer to the last one is no, the number is marketing.
How does ReplyPen measure quality?
ReplyPen does not report a resolution rate. It writes Drafts in your own Gmail or Outlook thread that you send, and its Reports show how many Drafts went out unchanged and how many you edited, plus coverage and a time-saved estimate, week by week. Each Draft carries a Check note listing the sources it relied on, so you judge the email in front of you instead of an average.
For the routine that goes with these numbers, read How to check an AI-written answer before you send it.
