A customer asks, “What does your plan cost?” Your agent finds one answer on a crawled pricing page and another in an older PDF. Chatbot knowledge base prioritization is the process of deciding which source should control the answer when several sources cover the same topic. The goal is simple: every high-risk question needs a source of truth, a test question, and a human handoff rule.
Rank sources before you tune the agent
Start with content governance, not prompt changes or model changes. An LLM can retrieve and summarize information, but it cannot reliably decide which of two conflicting policies your business intends to enforce. Your team has to make that decision first.
This matters most when the answer affects money, access, eligibility, or a buying decision. A vague instruction such as “use the most accurate source” does not tell the agent how to judge accuracy. A fixed source policy does.
For each important topic, document three things:
- The approved source.
- A question that tests whether the agent finds it.
- The point at which the conversation goes to a human.
For example, a refund policy might use an approved policy document as its source, a test question such as “Can I get a refund after 30 days?”, and a handoff rule for exceptions that depend on account history.
Do this before adding more documents. A larger knowledge base does not fix unclear ownership. It gives the agent more material to retrieve when the underlying policy is still undecided.
Score each source on four signals
Review every source against four signals: authority, freshness, scope, and escalation risk. The scores do not need to be complicated. Their purpose is to force a decision when two sources overlap.
Authority asks who approved the content. Official policies, current product documentation, and approved support answers should sit above informal notes, copied pages, and draft content. A private support note may explain how your team handles a case, but it should not outrank the published policy unless someone has explicitly approved it for customer answers.
Freshness asks when the content was last reviewed, not only when it was created. A page can look official and still describe an old price, product limit, or cancellation rule. Record the last review date and lower the priority of content that has not been checked since the underlying product or policy changed.
Scope asks how closely the source matches the question. A narrow document for a specific plan should outrank a broad overview when the customer asks about that plan. The same applies to product areas, customer types, regions, and exceptions.
Escalation risk asks what happens if the answer is wrong. Billing, refunds, account access, legal wording, and eligibility deserve stricter review than a low-impact question such as where to find a settings page.
Use a table like this during the review:
| Source | Topic | Owner | Last reviewed | Authority | Scope | Action if it conflicts |
|---|---|---|---|---|---|---|
| Refund policy PDF | Refund eligibility | Finance | 2026-09-10 | High | Refund terms | Use approved wording and hand off exceptions |
| Help center article | Refund overview | Support | 2026-07-12 | Medium | General refunds | Update after policy review |
| Internal support note | Manual exception process | Support lead | 2026-08-01 | Medium | Selected accounts | Keep internal and require human review |
The action column is where the table becomes useful. “Review later” is not a source policy. Name the owner, correction, or handoff that should follow the conflict.
Resolve conflicts with a fixed decision order
Use this order when sources disagree: topic scope, authority, then freshness.
First, identify the exact question. A current source about one plan may outrank an older general page because it matches the customer’s scope. Next, check who approved the source. Then confirm when it was last reviewed. A current, official refund policy should outrank an older general help article even if both contain the same phrase.
For wording that should not be paraphrased, use exact Q&A entries. Refund conditions, cancellation rules, eligibility exceptions, and legal language are good examples. An exact answer gives the agent a controlled response instead of asking an LLM to rewrite sensitive language from several documents.
Do not merge conflicting answers into a vague compromise. Mark the lower-priority source for correction. If the conflict could affect the customer’s decision, hand off the conversation rather than presenting a blended answer with false confidence.
Consider the pricing example. A crawled pricing page shows one plan price. An uploaded policy PDF shows another. The approved, current source wins only after its owner and review date are confirmed. If nobody can confirm which source applies, the agent should say that a team member needs to check and hand off the conversation. Guessing creates a customer-facing policy that nobody approved.
This method is about governance, not a claim that every AI platform exposes a manual source-ranking control. The team still needs a decision order even when the underlying retrieval system chooses among sources automatically. For additional background on source selection and knowledge-base retrieval, see this knowledge source prioritization explanation and this knowledge base implementation guide.
Test retrieval with questions that create collisions
Do not test only questions that have one obvious document match. Build questions where two or more training sources could produce an answer.
Include pricing, product limits, policy exceptions, and customer wording that does not match the title of the source. A customer will ask “Can I cancel after the trial?” rather than “Trial cancellation policy.” The test should reflect that difference.
Record each test in a simple log:
| Test question | Expected source | Agent answer | Approved wording matched? | Handoff required? | Next action |
|---|---|---|---|---|---|
| Can I cancel after the trial? | Current cancellation policy | Returned answer | Yes or no | Yes or no | Correct source or add exact Q&A |
| What does the Pro plan include? | Current product page | Returned answer | Yes or no | Yes or no | Update page or review retrieval |
| Can you make an exception for my account? | Human review | Returned answer | Not applicable | Yes | Add handoff rule |
Test direct questions, follow-up questions, and paraphrases. An agent may find the right document for “What is your refund policy?” and then choose the wrong source after the customer adds, “What if I bought it two months ago?” Context can change the relevant scope.
Review conversation logs after launch. When you find a wrong-source answer, classify the fix before changing the prompt:
- Add an exact Q&A pair when the approved wording needs to stay precise.
- Correct or remove the lower-priority source when the content is outdated.
- Add a handoff rule when the answer depends on account facts or an exception.
This turns each confirmed failure into a content decision. It also keeps the team from responding to every bad answer by adding another broad document.
Apply the framework to AssistLoop training sources
AssistLoop lets you train an AI support agent on several types of content. Organize those inputs by the job they need to do rather than uploading every document your team owns.
- PDFs: Use for approved policies and documents that need controlled review.
- Crawled website pages: Use for current public product information and published guidance.
- Pasted text: Use for controlled internal guidance that has a clear owner.
- Exact Q&A pairs: Use for answers that must stay precise, such as refund conditions or eligibility exceptions.
See AssistLoop’s training sources for the available ways to add content. Before adding a source, write down the customer question it is meant to answer. If nobody can name that question, the source may add noise without improving support.
Keep the first training set small. Include the sources that answer buying questions, repeated support requests, and high-risk policy questions. Then run collision tests before putting the agent on your site. The AssistLoop pricing page shows the plan details and limits to review before deployment.
Embed the trained agent as a website widget only after the tests pass. The widget should expose the approved answer, not the fact that your team has several documents that disagree.
Create a human handoff rule for conflicts, case-specific judgment, and answers where a mistake could create financial, legal, or account risk. With human handoff in AssistLoop, your team receives the conversation history in a shared inbox, so the customer does not need to start over.
AssistLoop is the wrong fit if you need the agent to make unsupported policy decisions from conflicting documents. The safer setup is to resolve the source conflict first or send that conversation to a person.
Keep the knowledge base accurate after launch
Assign an owner to each high-risk source. Review it when the underlying policy or product changes, not only after the agent gives a bad answer. The owner should be able to confirm whether the content is still approved and whether its scope has changed.
Use conversation logs to find repeated questions that lack a clear source. Add the smallest authoritative piece of content that closes the gap. A focused Q&A pair is often better than another long document that repeats existing material in different words.
Retest every question affected by a source update. A new pricing page or policy PDF can change which answer the agent retrieves, including answers that worked correctly before the update. Keep the old test question in the suite so you can see whether the change fixed the collision or moved it elsewhere.
Start with questions that affect buying decisions, create repeated support work, or carry a high cost when answered incorrectly. When that small source set is ready, create an AI agent and test it with real customer wording before expanding the knowledge base.
FAQ
How should you prioritize chatbot knowledge base content?
Rank each source by authority, freshness, scope, and the risk of a wrong answer. Official policies and current product information should usually outrank broad, older, or informal content.
What content should be added to a chatbot knowledge base first?
Start with questions that affect customer decisions, create repeated support work, or carry high escalation risk. Add current policies, product documentation, and approved support answers before lower-impact material.
How do you test whether a chatbot is using the right knowledge source?
Create questions that could match more than one source, then compare the returned answer with the approved source and wording. Include direct questions, follow-ups, and customer-language paraphrases.
When should a chatbot hand a conversation to a human?
Use human handoff when sources conflict, the answer depends on case-specific judgment, or a wrong answer could create financial, legal, account, or policy risk. The handoff rule should be defined alongside the source policy.
Written by