A support lead approves an agent after asking five clean questions: What are your hours? Where is the setup guide? How do I reset my password? The answers look right, so the widget goes live. Two hours later, a customer asks whether a late refund still qualifies under a payment exception. The agent gives a confident answer and leaves out the condition that protects the business.
Chatbot knowledge base testing needs harder inputs than a demo. Use real customer wording, incomplete context, policy edge cases, and questions the knowledge base should refuse. Build a reusable test set, score every answer against the same checks, and rerun it after knowledge base changes.
A Demo Conversation Does Not Prove Readiness
A demo proves that the agent can answer questions someone chose in advance. It does not prove that the agent can handle the way customers actually write.
Production questions are short, misspelled, vague, and sometimes stitched together from several issues. A customer may write, “can i get refund if used it 2 weeks and bought annual?” Another may ask, “is this included,” without naming the plan or feature. A third may ask for an account detail that the general knowledge base cannot safely provide.
Your test set should include all three situations. It should check whether the agent answers when the source is clear, asks for the missing detail when context is incomplete, and refuses or hands off when the source does not support an answer.
The useful output is not a polished transcript. It is a repeatable release check. Every case has an expected result. Every answer receives the same score. When a policy, product page, or agent instruction changes, you run the set again instead of trusting the last demo.
Build a Test Set From Real Customer Questions
Start with language your customers already use. Search conversation logs, support inboxes, live chat transcripts, and current FAQ entries. Pull questions that were easy to answer and questions that made a support agent stop and investigate. The second group usually exposes the expensive failures.
Group the cases by intent so you can see where failures cluster:
- Product questions, such as feature availability, limits, and compatibility.
- Billing and refund questions, including renewal, cancellation, credits, and exceptions.
- Setup questions, including installation, configuration, and troubleshooting.
- Account-specific requests that need identity or private account data.
- Unsupported requests where the knowledge base has no approved answer.
Give each test case a fixed record. This prevents reviewers from judging an answer by memory or personal preference.
| Field | What to record |
|---|---|
| Customer message | The wording a customer would actually send |
| Expected answer | The approved facts and conditions the answer must contain |
| Source document | The page, file, policy, or Q&A pair that supports the answer |
| Acceptable fallback | The response or handoff path when the source has no answer |
| Risk category | Billing, refund, privacy, compliance, sales, or ordinary support |
Do not write every case as a clean FAQ question. Add alternate phrasing, spelling mistakes, short questions, multi-part questions, and questions that combine two policies. “Can I cancel and still get a refund?” tests more than one rule. “refund after renewal??” tests whether the agent can ask for the missing context without inventing a policy.
Sensitive wording deserves special treatment. If a refund term, legal requirement, or compliance statement cannot be paraphrased safely, create an exact Q&A pair for it. AssistLoop supports files, website content, pasted text, and exact Q&A pairs as training sources. See the guide to training sources for AssistLoop for the available options.
Keep the original customer wording in the test set. Do not replace it with a cleaner version after the case fails. The rough wording is part of the test.
Score Every Answer on Four Checks
Use four pass or fail checks for every response. A case passes only when all four checks pass.
| Check | Pass condition |
|---|---|
| Factual accuracy | The response matches the approved product detail or policy |
| Source grounding | The response is supported by the intended knowledge base source |
| Answer completeness | The response covers every part of the question and includes required conditions or limits |
| Safe handling of unknowns | The agent does not guess and follows the approved next step when the source has no answer |
Factual accuracy asks whether the answer is correct. If the policy says refunds are available within thirty days of purchase, an answer that says sixty days fails even if it sounds clear.
Source grounding asks where the answer came from. A plausible sentence is not enough. The response should reflect the approved source, not a detail the language model filled in from general knowledge. This is where outdated pages and conflicting documents matter.
Answer completeness catches partial answers. Consider this customer message:
“Can I cancel my annual plan after renewal and get a refund?”
A passing answer might say:
“You can cancel the annual plan from your billing settings. Refund eligibility depends on the post-renewal refund window in the billing policy. If you are outside that window, contact support for a review.”
That answer names the action and the condition. The exact wording should match your approved policy.
This answer sounds helpful but fails:
“Yes, you can cancel your annual plan and request a refund after renewal.”
It leaves out the refund condition. A customer could read it as a promise. Mark it as a completeness failure, and possibly an accuracy failure, depending on the policy.
Safe handling of unknowns is a separate check. The right answer to an unsupported question is not a longer answer. It may be a clarifying question, a link to a known process, or human handoff. Documentation from Cekura and Giskard also treats answer evaluation as a defined test activity rather than an informal preview.
Test the Questions That Cause Expensive Mistakes
Prioritize failures by what they can cost you. A wrong greeting is annoying. A wrong refund answer can create a chargeback. A careless answer about private data can create a privacy incident. A false claim about a product capability can lose a sale.
Put these cases at the top of the test set:
- Disputed charges, refund exceptions, cancellation, and renewal questions.
- Requests involving personal data, account ownership, identity, or payment details.
- Compliance and legal wording that requires approved language.
- Product claims that affect a purchase decision.
- Requests for exceptions that only a person can approve.
Test conflicting documents on purpose. Give the case a current policy and an outdated FAQ, then check which source the answer follows. Test what happens after you remove or replace the old source. A knowledge base can contain correct information and still produce the wrong answer when the source set is unclear.
Account-specific questions need their own group. A general support article may explain how to change an email address. It must not reveal which email belongs to a particular customer. Test both the general process and a request for private account information. The expected result may be a verification step or a handoff, not an answer from the public knowledge base.
Requests such as disputed charges, policy exceptions, and suspected account takeover usually need a person. A longer AI response does not create authority the agent does not have. Use human handoff in AssistLoop when a team member needs to review the case and the judgment call belongs to them.
Turn Failures Into Knowledge Base Changes
A failed test is useful only when someone can reproduce it. Record the failed customer message, the answer given, the expected answer, the source used, and the reason for failure.
Then classify the problem before changing anything:
- Content problem: The source is missing, vague, outdated, or contradictory.
- Instruction problem: The source is correct, but the agent is not told how to apply it.
- Retrieval problem: The needed source exists, but the agent uses a different passage or document.
- Routing problem: The question should go to a person instead of receiving an AI answer.
Fix the source material first when the policy is missing or contradictory. Updating an instruction cannot resolve two documents that disagree. Give the policy one approved version, remove the stale guidance where possible, and run the failed case again.
Add an exact Q&A pair when a sensitive answer needs controlled wording. This works well for refund terms, legal language, eligibility conditions, and responses that must include a particular limit. Do not use a Q&A pair to hide a policy that the source documents still contradict.
After a fix, retest the failed case and nearby cases that could be affected by the same change. A new refund condition may fix one question and break a cancellation question. A new setup instruction may help one installation path while confusing another.
After launch, review conversation logs for new wording. Add real customer messages to the test set when they expose a gap, an unclear source, or an unsafe fallback.
Set a Production Gate Before You Publish
Write a release checklist before the agent reaches the site. It should require every high-risk case to pass, every tested answer to reflect the correct source, and every unsupported request to follow the approved fallback.
Billing, refund, privacy, and account-related cases need a second review. One person can run the test. Another person should confirm that the expected answer and fallback match the policy the business intends to enforce.
Run the full test set after changes to pricing, policies, product documentation, or agent instructions. Do not run only the cases that motivated the change. The nearby cases are where unintended changes appear.
Track failures by category instead of relying on one overall pass rate. A high average can hide one serious billing failure. Your review should show how many cases failed for accuracy, grounding, completeness, unknown handling, and routing. The release decision should consider the risk of each failure, not just the percentage that passed.
Production approval should depend on risk-weighted failures. A polished demo and a single accuracy score cannot tell you whether an agent is safe to publish.
Use the Same Test Discipline With AssistLoop
Train an AssistLoop agent on files, website content, pasted text, or exact Q&A pairs. Then test the questions your support team actually receives before publishing the widget. The point is not to make every response sound impressive. The point is to confirm that the agent uses approved information, includes the conditions that matter, and stops when a person needs to decide.
Review conversation logs after launch. Add real customer wording to the test set so the suite follows live traffic instead of staying frozen around the questions you wrote on day one. When a case needs a judgment call, use human handoff.
If your immediate job is to train a website agent, test its knowledge, and send difficult conversations to a person, create an AI agent and run this workflow before publishing the widget.
FAQ
What should a chatbot knowledge base test case contain?
A chatbot knowledge base test case should include the customer’s message, the expected answer, the source document, and the approved fallback when no answer exists. Add a risk category so high-impact failures receive more scrutiny than ordinary product questions.
How do you test a chatbot with real customer questions?
Test real customer wording, including incomplete questions, spelling mistakes, short messages, multi-part requests, and questions that combine policies. Clean FAQ wording alone will miss the conditions that cause failures in live support.
What should you score in chatbot knowledge base testing?
Use factual accuracy, source grounding, answer completeness, and safe handling of unknowns as separate checks. A response can be factually correct but still fail if it omits a required condition or answers a question the knowledge base does not support.
When should you retest a chatbot knowledge base?
Rerun the full test set after changes to pricing, policies, product documentation, or agent instructions. Retest the failed case and nearby cases because the same source change can affect more than one intent.
Written by