A refund policy changes on Monday. On Tuesday, your agent still quotes the old condition because the retired document was never removed. Chatbot knowledge base maintenance works best when a change starts the review, not when answer quality has already fallen. This playbook uses four triggers to keep source material current, remove conflicts, and test the answer customers actually receive.
Treat maintenance as an operating process, not a cleanup task
A knowledge base should be reviewed when something changes. Waiting for a visible drop in answer quality means customers have already found the gap for you.
Use four triggers throughout the process:
A product or policy change.
A failed ingestion.
Duplicate or conflicting guidance.
A customer conversation that exposes missing information.
Each trigger needs one owner. The owner records four details in a maintenance log: the source that caused the review, the change made, the person who reviewed it, and the validation result. That record stops the same question from being investigated from scratch next month.
The log can be a shared document, a database, or a field in your support workflow. The format matters less than the audit trail. A reviewer should be able to see why an answer changed and what test proved the new answer was safe.
For risk management context, the NIST AI Risk Management Framework treats documentation, monitoring, and ongoing review as part of responsible AI operation. The same principle applies to a customer-facing agent: training is a launch step, not the maintenance process.
Start every review with a source freshness check
Begin with the material that the agent is allowed to use. List every source that can change:
Website pages, including pricing, shipping, returns, and product pages.
Policy documents with effective dates.
Product documentation and release notes.
Exact Q&A pairs written for answers that must stay precise.
For each source, record its effective date, last revision date, and owner. Add a status that anyone on the team can understand:
| Status | Meaning | Next action |
|---|---|---|
| Current | The source is approved and reflects the present rule | Keep it available and monitor changes |
| Needs review | The source may have changed or lacks a clear owner | Confirm its wording and approval |
| Blocked | The source cannot be checked or ingested | Fix access or readability before editing answers |
| Retired | The source no longer applies | Remove it or replace it with the approved version |
Do not add a new answer beside an old one when the old answer is no longer valid. Two documents can both look reasonable to a reviewer and still produce conflicting answers for a customer. Remove or replace the outdated material first.
A source freshness check also catches a common mistake: editing the answer manually before confirming that the source is available. If the source is missing, you may fix one response while leaving the underlying ingestion problem in place.
Use four triggers to decide what to audit
The trigger tells you where to start, but it does not tell you that the review is complete. For each issue, trace the customer-facing answer back to its source, check related entries, and run a test after the correction. That prevents a narrow edit from leaving a second outdated answer in place.
A product or policy change
Compare the old wording with the new wording. Mark every limit, date, exception, and condition that changed. Then review dependent Q&A pairs and test questions that use the changed rule.
For a refund policy, do not test only “What is your refund policy?” Test the customer phrasing that exposes the actual decision: “Can I get my money back after the trial?” The agent needs the current condition, not a general summary that sounds correct.
Retain the approved wording in one source. If an exact phrase matters for legal or billing reasons, use an exact Q&A pair and connect it to the approved policy document.
A failed ingestion
Check the source before changing any answers manually. Confirm that:
The source still exists at the expected location.
The content can be read in its current format.
The updated version is present.
The agent has received the current source after the ingestion or training event.
A failed ingestion can look like a bad answer. If the latest product page never entered the knowledge base, rewriting five Q&A pairs only hides the original fault. Fix the source path or file, then test the answer again.
Duplicate or conflicting guidance
Group entries that cover the same customer question. Compare their wording, limits, dates, and exceptions. Choose one approved source of truth, then delete or rewrite the rest.
Conflicts often hide in small details. One entry says cancellations are allowed within 30 days. Another says 14 days. A third mentions an exception for annual plans. The agent cannot safely resolve that conflict from formatting alone. Your team has to decide which rule applies and remove the discarded versions.
A customer conversation that exposes missing information
Capture the exact question, the answer the customer needed, and the source that should support it. Do not add a guessed answer simply to make the conversation pass a test.
If no approved source covers the question, mark the item as blocked. Ask the appropriate owner to provide the rule, then add the answer only after it has been reviewed. A confident answer without a source creates a support risk that is harder to spot than an honest handoff.
Remove overlap before you add more content
More content does not fix a messy knowledge base. It can give retrieval more material that says almost the same thing.
Search for entries that answer the same question with different wording, limits, dates, or exceptions. Search both the source titles and the text of the answers. A duplicate may not use the same phrase as the customer question.
Keep exact Q&A pairs for wording that must stay precise. Refund conditions, account requirements, and legal policy language are poor candidates for loose paraphrasing. An approved Q&A pair gives the agent a clear answer to a known question.
Split broad documents when one source contains unrelated rules that could be retrieved together. A single document covering shipping, returns, billing, and account access is harder to inspect than separate material with clear topics. The goal is not more files. The goal is fewer ambiguous paths to an answer.
Test two versions of every important question:
The direct question used in the knowledge base.
A customer phrasing that uses different words or leaves out the formal policy term.
For example, test both “How long do I have to return an item?” and “Can I send this back next month?” If the answer works only for the formal wording, the entry is not ready for customer traffic.
Turn customer conversations into a maintenance queue
Conversation logs show what customers tried to ask, not what your team expected them to ask. Review them on a recurring schedule and tag each issue as one of four types: missing guidance, stale guidance, conflicting guidance, or retrieval failure.
Start with questions that block a purchase, create billing risk, or lead to a human handoff. Those conversations have a clear business or customer cost. They also give the reviewer enough context to fix the real problem instead of polishing a generic FAQ.
Write the missing answer in the same language and level of detail customers use. If customers ask “Can I change the delivery address after ordering?”, an internal answer such as “See fulfillment policy” is not enough. The entry should state the rule, the condition, and the next step, provided an approved source supports each part.
Connect the new answer to that source. This makes later reviews easier. When the policy changes, the owner can find the dependent answer instead of relying on memory.
Use human handoff as the safety path while an answer is under review. The team receives the full thread and conversation history, along with the context needed to understand what failed. That gives the reviewer the original customer wording, the agent’s response, and the point where the conversation moved to a person.
Validate every change before you call it done
Create a small test set for every material update. Include the changed policy, the old wording, common customer phrasing, and at least one edge case.
Record the result in the maintenance log:
| Test field | What to record |
|---|---|
| Test question | The exact wording sent to the agent |
| Expected result | The approved answer or required handoff |
| Actual result | The response the agent gave |
| Follow-up action | The correction, owner, or reason no change was needed |
The test needs to prove three things. The agent gives the current answer. It does not repeat the retired answer. It hands the conversation to a person when the source does not support a safe response.
Run the test after a source update or retraining event. A source marked current is not proof that the agent used it correctly. The answer still has to survive direct wording, customer wording, and the edge case you chose.
This is also where you catch a false success. A page may have been crawled, but the answer can still rely on an older Q&A pair. A new document may be readable, but a duplicate may still contain the previous limit. Validation checks the customer-facing result rather than the status of the source alone.
Google’s guidance on creating helpful content asks publishers to show clear purpose, first-hand knowledge, and evidence that content is maintained. The same standard is useful here: record what changed, why it changed, and how you checked the result.
Apply the playbook to your own AI agent
AssistLoop lets you train an agent with uploaded PDF, DOCX, or TXT files, crawled website content, pasted text, and exact Q&A pairs. The training sources documentation shows how those sources support an agent’s knowledge base. Use the source freshness check before you add material, and remove retired content instead of stacking another answer on top of it.
Review conversation logs and deflection data to decide what needs training next. Tie each maintenance decision to a real customer question. A guess about what customers might ask is weaker evidence than a logged conversation with the exact wording and outcome.
For questions that need a person, use human handoff while the knowledge base is being corrected. The agent can step aside without losing the thread your reviewer needs.
AssistLoop also supports training through website crawling and exact Q&A pairs, so the same review process applies across different source types. Start with the four triggers, assign an owner, and record the validation result for every change. When you are ready to test the process, create an AI agent. You can review the rest of the platform’s AI support agent features as your maintenance workflow grows.
FAQ
How often should a chatbot knowledge base be reviewed?
Review it whenever a product or policy changes, an ingestion fails, duplicate guidance appears, or a customer conversation exposes a missing answer. A recurring conversation-log review can catch issues that do not produce an obvious source change.
What should I check first when maintaining a chatbot knowledge base?
Start by identifying the approved source, its owner, effective date, and last revision date. Mark it current, needs review, blocked, or retired, then remove outdated material instead of placing a new answer beside it.
How do I prevent conflicting chatbot answers?
Keep one approved source of truth for each customer question. Use exact Q&A pairs when wording must stay precise, and remove or rewrite entries that contain older limits, dates, or exceptions.
How can I tell if a knowledge base update worked?
Record the exact customer question, expected answer, actual answer, and follow-up action. Test the updated wording, the retired wording, a natural customer phrasing, and an edge case before closing the review.
Written by