How to test an AI receptionist before you buy

Preparation

• Give every provider the same knowledge base: hours, three services with prices, one contact person, your emergency rule.
• Use the same call forwarding setup and call from a mobile phone.
• Write down ten real requests from your business, including two difficult ones.

The ten checks

1. Greeting and disclosure: Does the agent say it is an AI assistant, and does the recording notice play if recording is on?
2. Latency: Time from the end of your sentence to the start of the answer. Measure ten times and note the average. See latency and barge-in.
3. Interruptions: Does it stop talking when you interrupt, and does it use what you said?
4. Names and numbers: Last name, email address, phone number and, depending on your business, an address or a policy number. Count the errors.
5. Accents and noise: Call once with a strong accent or fast speech, once from a car or the street.
6. Knowledge gaps: Ask something your knowledge base does not cover. Does it invent an answer? See hallucination.
7. Transfer: Ask for a person. What happens when nobody answers? See warm transfer.
8. Emergency: Describe an emergency typical for your industry. Does it tell you to dial 911 or alert the right person right away? See E911 and emergencies.
9. Booking: Does it book into your real calendar or software, and what happens with a double booking?
10. Follow-up: Does the summary arrive, is it correct, and where is the transcript stored?

What to request in writing

• The terms, the privacy policy and, if you handle health information, the business associate agreement
• The list of subprocessors with locations
• A statement on training on call data
• The latest SOC 2 report, if the provider states one
• Billing details: increment, overage price, simultaneous calls, cancellation

Score each provider on the ten checks and compare the results with the prices in the calculator.