Use a chatbot support coverage model to classify what your agent can answer, what should reach a person, and what has no safe answer. This framework gives support leaders a review method based on real conversation logs instead of a single deflection percentage.
A hypothetical example: at 9:07 a.m., your support dashboard shows a healthy deflection number. By lunch, three customers have repeated the same billing question, one has asked for a person twice, and another followed an outdated cancellation instruction. A chatbot support coverage model helps you see the difference between a conversation the agent handled and a customer problem it solved.
Coverage is the share of real customer intents your agent can handle safely. It is not the number of pages in your knowledge base or the number of conversations the agent receives. The practical method is simple: classify each intent, assign one operational state, then review those assignments against conversation logs on a regular cycle.
What a chatbot support coverage model measures
A useful coverage map has three states:
| State | Meaning | Appropriate outcome |
|---|---|---|
| Approved answer | The agent has approved source content and can answer without private account data. | The customer receives an answer they can act on immediately. |
| Human handoff required | The request needs judgment, sensitive context, dispute handling, or a person’s involvement. | The agent transfers the conversation with enough context for the team to continue. |
| No safe coverage | The source is missing, conflicting, outdated, or too vague to support a reliable answer. | The agent states the limit or routes the request without inventing instructions. |
Classify intents before measuring deflection. A high deflection rate can hide an incorrect answer if the customer leaves instead of escalating. It can also hide a failed handoff if the customer gives up before a person joins the conversation.
The unit you measure is the intent, not the wording. “Where is my order?” and “Can you tell me when package 4831 arrives?” use different words, but they express the same intent. “Can I get a refund?” and “Why was my refund rejected?” belong to different intent groups because the second may require account history and judgment.
Your coverage register should answer one question for every recurring intent: what should happen when this request arrives? The answer may be an approved response, a human handoff, or a clear unsupported state. Treating handoff and no safe coverage as valid operational states is more useful than forcing every request into automation.
How to classify each support intent
Start with the customer’s goal. Avoid labels such as “billing” or “account” when they combine requests with different risks. Use labels that tell your team what the customer wants done.
Examples include:
- Refund eligibility
- Order status
- Account-specific billing
- Product setup
- Cancellation requests
Then apply three tests to each intent.
1. Can the agent cite approved content?
An intent is answerable only when the agent has a source that states the answer clearly. A refund policy written for internal staff may omit conditions customers need to know. A product setup page may describe the happy path but not what to do when the first step fails.
Exact Q&A pairs are useful when wording matters. Use them for refund conditions, cancellation terms, or other answers that should not be paraphrased loosely. Broader product explanations can come from documentation or approved website content.
2. Can the customer act without private data?
General billing policy can be answerable. “Do you charge annually?” does not require access to a customer record. “Why did you charge me twice?” does. The second request needs account-specific data, investigation, or both.
Account-specific answers should use verified identity and a connected system, or go to a person. Do not classify an intent as covered because the agent can produce a plausible sentence about it.
3. What happens if the answer is wrong?
Route an intent to human handoff when the cost of a wrong answer is high. Billing disputes, cancellation requests with retention implications, sensitive account issues, and conversations involving clear frustration deserve a person even when a policy page exists.
This means one broad category can contain all three states. General billing policy may receive an approved answer. A disputed charge may require human handoff. A question about a new billing rule that is absent from the knowledge base has no safe coverage.
Here is a clearly labeled sample walkthrough. The dialogue is invented to show the method, not drawn from AssistLoop customer data.
Customer: I was charged twice for my subscription. Can you refund one of the charges?
Agent: I can explain the refund policy, but I can’t inspect your billing history or approve a refund here. I’ll send this conversation to the support team so they can review the two charges.
Assigned state: Human handoff required.
Source and review: The refund policy is the approved source for general eligibility. The request still needs account-specific billing data. Review the handoff rule on the next coverage cycle.
Outcome to record: Whether a person received the conversation, whether the customer had to repeat the issue, and whether the duplicate charge was resolved.
If the customer instead asks, “Do annual subscriptions qualify for refunds?” and the approved refund policy answers that question clearly, the state may be Approved answer. If no current policy covers annual subscriptions, assign No safe coverage. Do not let the broad label “refunds” hide those different outcomes.
Use this rule in the register:
An intent is AI-answerable only when the source is approved, private account data is not required, and the customer can act on the answer immediately.
If any part of that rule fails, choose handoff or no safe coverage. Do not reward the agent for answering a question it should have declined.
How to build the coverage map from conversation logs
Review a representative sample of recent conversations. Export the logs if your support system allows it, or review them directly. Group messages by customer intent rather than by the exact phrase used.
For each sampled intent, record the following:
| Field | What to record |
|---|---|
| Intent | The customer’s actual goal. |
| Coverage state | Approved answer, human handoff required, or no safe coverage. |
| Source | The file, page, pasted text, Q&A pair, or connected system used. |
| Answer quality | Correct, incorrect, incomplete, or unsupported. |
| Customer response | Resolved, repeated, corrected, abandoned, or escalated. |
| Next action | Training change, handoff rule, documentation fix, or API work. |
A published research framework on knowledge-base coverage and correctness makes the same distinction between whether an answer is supported and whether it is correct. Use its coverage and correctness framework as a research reference, accessed August 26, 2026, then apply the fields to your own logs.
Separate knowledge-base gaps from product and account-data gaps. If the answer exists but is buried in an unclear page, improve the training source. If the answer depends on a live order, subscription, appointment, or account record, documentation alone cannot solve the problem.
Recurring unsupported intents deserve priority. The same unsupported request appearing repeatedly gives you a concrete choice: add an exact Q&A pair, write better documentation, define a handoff rule, or build an Agent Action that can complete the request.
What good coverage looks like in practice
Good coverage needs more than one percentage. Use separate calculations for coverage state and service quality. To keep the unit consistent, assign each sampled conversation one primary intent and one coverage state before calculating the measures.
For a sample of conversations, calculate:
- Answer coverage: approved-answer conversations divided by all sampled conversations, multiplied by 100.
- Safe coverage: approved-answer conversations plus human-handoff conversations, divided by all sampled conversations, multiplied by 100. This treats a planned handoff as a valid outcome.
- Unsupported rate: no-safe-coverage conversations divided by all sampled conversations, multiplied by 100.
- Answer correctness: correct approved answers divided by sampled approved-answer conversations, multiplied by 100.
- Repeat-question rate: conversations where the customer repeats the same intent divided by all sampled conversations, multiplied by 100.
- Handoff completion: required handoffs where a person takes over divided by all required handoffs, multiplied by 100.
- Customer outcome rate: conversations marked resolved divided by all sampled conversations, multiplied by 100.
Here is a clearly labeled sample calculation using 100 conversations. It is a measurement example, not an industry benchmark.
| Measure | Sample count | Calculation | Result |
|---|---|---|---|
| Approved-answer coverage | 50 approved-answer conversations | 50 / 100 | 50% |
| Safe coverage | 50 approved-answer + 30 handoff conversations | 80 / 100 | 80% |
| Unsupported rate | 20 unsupported conversations | 20 / 100 | 20% |
| Answer correctness | 44 correct approved answers | 44 / 50 | 88% |
| Repeat-question rate | 18 repeated conversations | 18 / 100 | 18% |
| Handoff completion | 27 completed handoffs | 27 / 30 | 90% |
| Customer outcome rate | 76 resolved conversations | 76 / 100 | 76% |
The sample shows why one automation number is weak. An 80% safe-coverage rate may look good, but the 18% repeat-question rate tells you that some answers are not working. The 20% unsupported rate points to missing knowledge, missing system access, or a handoff rule that has not been defined.
Create a baseline before changing the knowledge base. Keep the same intent groups in the next review so you can compare like with like. A content update may improve refund eligibility while leaving cancellation requests unchanged. A new order-status action may reduce handoffs but expose a separate identity problem.
Look for false coverage. These are conversations where the agent answered, but the customer still had to repeat the question, correct the response, or ask for a person. False coverage is more useful than a raw deflection count because it shows where the system appears successful in a dashboard while failing in the conversation.
This is a baseline method, not a universal scorecard. The NIST AI Risk Management Framework describes AI risk measurement as context-dependent and tied to the system’s risks and intended use, rather than one threshold for every operation (accessed August 26, 2026). Set a baseline from your own conversation logs, record the review date, and compare each change against that baseline.
Where a coverage model breaks
A knowledge base can contain the right topic and still fail to cover the question. The policy may be conditional. The page may be outdated. The language may be written for internal readers rather than customers. Treat source presence as a starting check, not proof of coverage.
An agent may also give a confident answer outside the approved material. That is unsupported confidence, and it belongs in the no-safe-coverage state. A plausible answer is not evidence that the intent is covered.
Coverage can fall after a pricing, product, refund, or onboarding change. Recheck affected intents after the change instead of waiting for a quarterly review. A new price can make an old billing answer wrong. A changed setup flow can turn a previously safe instruction into a dead end.
Human handoff is part of coverage. The model is working when it sends a sensitive or high-risk request to the right person with the conversation history available. Check whether the handoff starts, whether the team receives the context, and whether someone completes the next step.
Be clear about limits. Account-specific answers need verified identity or a connected system. Unsupported requests should receive a clear limit, not invented instructions.
How to apply the model with AssistLoop
Start by training an AI agent on approved material. AssistLoop accepts website content, uploaded PDF, DOCX, and TXT files, pasted text, and exact Q&A pairs through training AssistLoop on your data. Map each approved-answer intent to the source that supports it. Use exact Q&A pairs where a policy needs controlled wording.
For requests that need a person, use human handoff. Paid plans let a visitor escalate mid-chat, and the team can pick up the thread with its conversation history. That gives you a measurable handoff state instead of treating every non-automated conversation as a failure.
Use Agent Actions when coverage depends on a completed action or live system data. An Action can call a REST endpoint for tasks such as order-status checks, meeting bookings, lead capture, or another API-backed operation. The distinction matters: adding an FAQ about order status does not make live order status available.
Review conversation logs after training and workflow changes. Check whether the intended state improved and whether new false coverage appeared. If a new Q&A pair fixes refund eligibility but causes customers with disputed charges to receive the same answer, split the intent and add the handoff rule.
The Free plan provides 150 message credits and does not include human handoff. See AssistLoop pricing for current plan details.
Last verified: August 2026.
A review cycle support leaders can run
Run the review as a coverage register, not as a general impression of whether the agent “seems good.” Start with recurring intents from recent conversation logs and assign one of the three states to each.
For every sampled intent, check the source, answer quality, customer outcome, and escalation path. Then prioritize work in this order:
- Fix incorrect answers.
- Define handoff rules for requests that need judgment or private context.
- Fill repeated knowledge gaps with documentation or exact Q&A pairs.
- Add an Agent Action when the request needs live data or a completed transaction.
After each content or workflow change, sample the same intent groups again. Record what changed, which state is expected now, and whether the customer outcome moved with it.
A compact register should use these columns:
| Intent | Current state | Approved source | Owner | Last review | Next action |
|---|---|---|---|---|---|
| Customer intent name | Approved answer, human handoff required, or no safe coverage | Source supporting the state | Responsible team member | Date of the latest review | Next documented change or test |
The register is the operating tool. It tells you what the agent may answer, where it must stop, and which unsupported requests deserve product or documentation work. Create your AI agent and test the first intent group against real conversation logs before expanding the map.
FAQ
Does chatbot coverage mean every customer question must be automated?
No. Coverage describes the safe outcome for an intent, which may be an approved answer, a planned human handoff, or a clear unsupported state. This lets support teams measure safety and service quality without treating automation as the only acceptable result.
How often should a support team review chatbot coverage?
Review recurring conversation patterns after meaningful content, pricing, product, or workflow changes. The exact cadence should match your conversation volume and the risk of incorrect answers, rather than follow a universal calendar.
How can you measure coverage without using an industry benchmark?
Create an intent-level baseline from your own conversation logs. Pair the coverage state with correctness, repeat questions, handoff completion, and the customer outcome, then compare the same intent groups after each change.
When should you add an API integration to a chatbot?
Add an API-backed Agent Action when the request requires live system data or a completed transaction. Order-status checks and meeting bookings are examples where documentation alone cannot complete the customer’s request.
Written by