In this article
- Why do AI-written replies need a different kind of check?
- What should you read first?
- Which facts must you always verify?
- How do you check a fact quickly?
- What are the warning signs of a guessed answer?
- What does a check look like on a real draft?
- What should you do when a fact is wrong?
- When should you not send a draft at all?
- How long should checking take?
- How do you check replies as a team?
- How do you know your checking is working?
- A checking routine you can copy
- How does ReplyPen help with checking?
- Related reading
Check an AI-written reply in a fixed order: does it answer the question the customer asked, are the facts that commit you right, and does it promise anything you would not? Verify numbers, dates, policies and customer details against the source the answer came from, not against memory. If the reply cannot show where a fact came from, treat that fact as a guess until you have checked it.
Why do AI-written replies need a different kind of check?
A reply written by a colleague usually fails in visible ways: a typo, a missing step, an awkward sentence. An AI-written reply fails in the opposite way. The grammar and tone are usually fine, and the mistake sits in a confident sentence that reads exactly like the correct ones.
That changes what checking means. Proofreading for style finds little. What you need to check is content: whether each fact matches your business, and whether the reply says anything you have not agreed to. Why AI answers sound right and still get the facts wrong explains where these mistakes come from.
What should you read first?
Read the customer’s email before the draft. It takes a few seconds, and it is the only way to notice a draft that answers the wrong question well.
Then answer three questions, in this order:
- Did it answer what was asked? Including the second question in the email, which drafts often skip.
- Is anything in it a commitment? Money, a date, an exception, a promise that someone will do something.
- Does it rest on a source? Can you see which help article, page or past reply the answer came from?
If the answer to the first question is no, stop there and rewrite or regenerate. Checking facts in a reply that answers the wrong question is wasted time.
Which facts must you always verify?
Some parts of a reply cost you when they are wrong, and some do not. Spend your checking time on the first group.
| Check | Examples | Check against |
|---|---|---|
| Numbers | Prices, amounts, discounts, quantities | Your pricing page or price list |
| Dates and durations | Refund windows, delivery times, deadlines, “within 2 business days” | Your policy pages or help articles |
| Promises | “We’ll refund you today”, “we’ll send a replacement” | What you or your team can actually do |
| Customer details | Order number, plan, what they bought, their name | The customer’s record in your shop or billing system |
| Instructions | Steps, menu names, links | The product itself, or the current help article |
| Exceptions | “unless”, “except”, “only if” | The full policy, since drafts tend to drop the conditions |
A quick test for anything in between: would the customer be right to hold you to this sentence later? If yes, verify it before sending.
Everything else (the greeting, the phrasing, whether a sentence could be shorter) is a matter of taste. Fix it if you want to, but it is not what makes a reply safe to send.
How do you check a fact quickly?
Check it against the source the answer came from, not against your memory. Memory is where out-of-date facts live: last year’s shipping time, a price from before the change.
The fastest route is a reply that shows its sources. If a draft about returns lists the returns article it used, you open that article, find the number, and compare. That takes seconds. If the draft shows nothing, you have to decide yourself where the answer should have come from, find that page, and then compare, which is most of the work of writing the reply yourself.
When a draft does list its sources, check two more things:
- Is it the right source? A draft about an annual plan that cites the monthly plan’s article is wrong even if it quotes the article correctly.
- Is the source current? An accurate quote from an outdated page is still an outdated answer.
What an AI reply should show about its sources describes what a useful source list contains and what to do when it is empty.
What are the warning signs of a guessed answer?
Some patterns suggest the draft filled a gap instead of answering from your sources. None of them proves a mistake, but each one is worth a second look.
- A specific number your help pages never mention. “Within 5 to 7 business days” when you have never written that down.
- Generic policy language. “We offer a 30-day money-back guarantee” could be any company’s policy.
- Extra reassurance. A draft that is longer than your usual reply, with more apologies and promises than the question needs.
- Confident answers to unusual questions. The rarer the question, the less likely your sources cover it.
- No source for the main fact. The rest of the reply cites articles, but the one sentence that matters does not.
What does a check look like on a real draft?
A customer writes: “Hi, I ordered the desk lamp on Monday, order 4821. Can I still change the colour to black?”
The draft:
Hi Sarah, yes you can! Orders can be changed until they are shipped, which usually happens within 2 business days. I’ve changed your lamp to black, and the new total is €89. Best, Tom
Its sources: the help article “Changing an order after checkout” and your reply to a similar question in March.
Going through the routine:
- Did it answer the question? Yes: can the colour change.
- Commitments. “I’ve changed your lamp to black” says an action happened. Nobody changed anything. That sentence has to go or become a promise you will keep.
- Numbers and dates. The help article says orders can be changed until they are packed, usually the morning after ordering, not “until shipped, within 2 business days”. The draft misread its own source. The order was placed on Monday, so it may already be packed.
- The price. “€89” appears in neither source. It is a guess, and it has to be checked against the price list or removed.
The corrected reply:
Hi Sarah, thanks for checking. Orders can be changed until they are packed, usually the morning after ordering, so I’m checking with the warehouse whether yours has gone out. I’ll reply by noon tomorrow with either the change confirmed or your options. Best, Tom
Three of the four checks found something, and the draft still read well. That is the normal pattern: the writing is rarely the problem.
What should you do when a fact is wrong?
Fix the reply, then fix the cause, so you do not correct the same thing next week.
- Correct the draft and send it, or send nothing until you know the right answer.
- Find out why it was wrong. Usually one of three causes: the source was missing, the source was outdated, or the draft misread a correct source.
- Fix the source if it was missing or outdated: add the answer to your help center or facts page, or update the page.
- Tell the tool if it misread a correct source, using whatever feedback or correction it offers, and check whether the correction sticks on the next similar email.
The third step is the one that pays off. A missing help article causes the same wrong draft every time the question comes in; one article fixes all of them.
When should you not send a draft at all?
Some emails should get a reply written or at least rewritten by a person, however good the draft looks:
- The customer is upset, or the email is a complaint
- It involves a refund, a dispute, a legal claim, or personal data requests
- The draft relies on no source for its main answer
- You cannot verify a fact in the time you have
- The email is from someone important to your business, and the reply sets a precedent
For these, use the draft as a starting point at most. When you cannot answer fully, a short holding reply with a specific time (“I’m checking this with our warehouse and will reply by 3 pm tomorrow”) is better than a complete-looking guess.
How long should checking take?
For a routine question with a draft that shows its sources: well under a minute. Read the question, read the draft, open the one source that backs the key fact, compare, send.
If checking regularly takes longer than writing the reply yourself, one of three things is going on:
- The drafts are missing sources, so every fact needs its own search. Add the missing articles or pages.
- The drafts are often wrong, so you read every line with suspicion. Look at which kinds of email go wrong and keep those away from the tool for now.
- You check everything with the same care. Save the full check for the six kinds of fact in the table above, and skim the rest.
How do you check replies as a team?
On a shared inbox, agree on the routine so everyone checks the same things.
- Write the checklist down. The table of facts above, adapted to your business, on one page.
- Name the risky kinds of email that always need a second person, such as refunds above a certain amount, or anything legal.
- Keep one facts page that everyone checks against, instead of each person’s memory.
- Give that page an owner, so a price or policy change reaches it the same day it happens.
- Share what went wrong. A short weekly note of drafts that needed a factual correction, and why, shows which sources to fix.
How do you know your checking is working?
Keep a simple tally for a few weeks. For each draft, note whether you sent it unchanged, changed the wording, or corrected a fact. The share sent unchanged tells you how much work the tool saves; the fact corrections tell you where the risk is. The “sent unchanged” rate explains how to count it and how to read the result.
Be wary of the opposite number: a vendor’s resolution rate. It counts conversations where the customer did not come back, which is not the same as conversations where the answer was right. Why AI resolution-rate claims mislead small businesses goes through what those numbers actually measure.
A checking routine you can copy
- Read the customer’s email.
- Read the draft. Does it answer the question, all of it?
- Mark every number, date, promise, customer detail, instruction and exception.
- Open the source behind each marked item and compare.
- If a marked item has no source, verify it yourself or take it out.
- If the email is on your “never auto-trust” list, have it reviewed or rewrite it.
- Send, and note whether you changed a fact.
- Once a week, fix the sources behind the facts you had to change.
How does ReplyPen help with checking?
ReplyPen was built around this routine. For each customer email it leaves a Draft in your own Gmail or Outlook thread, and with every Draft a Check note: a short list of the help articles, website pages and past emails the Draft relied on. You see in seconds where an answer came from, and when the sources do not cover the question, you can see that before you send. Nothing is sent without you. Reports count how many Drafts you sent unchanged and how many you edited, week by week.
