A customer asks where to find a setup guide. That belongs in automation. The same customer asks why a refund has not arrived, and the right next step may be verification or human review. What should a chatbot answer depends on the risk, the source, and what must happen after the reply.
For a support team, the practical rule is simple: classify every question as answer, clarify, verify, or hand off before adding it to an agent’s knowledge base. A smaller set of dependable answers beats a broad agent that guesses.
What should a chatbot answer?
A chatbot should answer questions that meet four conditions:
- The question is common enough to justify automation.
- The answer is stable and unlikely to change without a source update.
- Reliable support material exists, such as an approved policy, product guide, or exact Q&A pair.
- The outcome is low risk if the answer is given without private account access or human judgment.
That includes setup instructions, feature availability, shipping windows, published refund terms, supported file types, and basic troubleshooting. It can also include pre-sales questions when the answer comes from current product documentation.
It should not give a direct answer just because the question looks familiar. “Why was I charged?” may be a general billing question, or it may be a dispute about one customer’s account. “Can I change it?” is incomplete until the agent knows what “it” means.
The framework below separates those cases:
| Decision path | Use it when | Example |
|---|---|---|
| Answer | The source is clear, current, and safe to share | “Do you support SSO?” |
| Clarify | The request is missing information | “Can I change it?” |
| Verify | The answer depends on an authenticated account or system record | “Has my refund been sent?” |
| Hand off | The case involves risk, emotion, judgment, or a dispute | “I was charged twice. Fix this now.” |
This is a support operating rule, not a writing preference. A correct knowledge-base paragraph can still be a poor response if the customer needs a transaction completed, an account checked, or a sensitive case reviewed.
Microsoft’s guidance on writing for bots makes a similar point about context and clear language. Your support team still has to decide where the agent should stop.
Sort support questions into four decision paths
The four paths are a sequence for making a routing decision, not four labels to add after the conversation is over. Start with the customer’s intent, check whether the request is complete, then ask whether the answer needs private data or human authority. That order prevents a familiar topic, such as billing or refunds, from receiving an unsafe generic answer.
1. Answer directly when the source is stable
Use a direct answer for a documented fact or procedure. The agent can explain how to reset a password, where to find an invoice, which plan includes a feature, or how long standard shipping takes.
The source needs an owner and a review path. A policy copied from an old help article is not reliable simply because it is written clearly. If the answer depends on a current plan, price, eligibility rule, or product flow, check the source before treating the category as ready for direct answers.
2. Ask for clarification when the request is incomplete
Clarification is useful when one short question can identify the customer’s intent. It should narrow the request, not make the customer do the support team’s classification work.
Customer: “Can I change it?”
Agent: “Do you mean your plan, shipping address, or subscription date?”
The agent should ask for the smallest missing detail, then continue. It should not send a long list of possible interpretations or repeat the same question after the customer has already answered it. Once the customer says “shipping address,” the agent can decide whether a documented instruction is enough or whether an order record must be checked.
3. Verify when the answer depends on the account
An agent should not guess about a customer’s invoice, subscription status, order, usage, or refund. It needs a trusted system and a safe way to confirm the user’s identity.
AssistLoop supports verified user identity through a signed token and shared secret. That gives an application a way to identify a logged-in user before exposing account-specific information. Verification is only one part of the workflow. The required account data or action must also be available through the configured integration.
4. Hand off when the case needs authority or judgment
Send the conversation to a person when the customer is disputing a charge, requesting an exception, reporting a security concern, making a complaint, or showing strong frustration. A correct general policy may not resolve the case.
Human handoff gives the team the conversation history instead of making the customer start over. That matters after a failed attempt, when repeating the same article can make an already poor interaction worse.
Consider the refund example:
- “What is your refund policy?” can receive the published policy.
- “Am I eligible for a refund?” may need order and purchase details.
- “Why has my refund not arrived?” needs a transaction check.
- “Your team promised an exception and now I was charged again” needs a person with authority to review the case.
The topic is the same. The decision path changes with the customer’s situation.
How the answer decision works in a real conversation
A support conversation should move toward a useful outcome, not toward the longest possible answer. Here is a short example:
Customer: “Can I cancel my subscription?”
Agent: “You can cancel from Settings > Billing. Do you want instructions, or are you asking whether you qualify for a refund?”
Customer: “I canceled yesterday. I still see a charge.”
Agent: “I can check the account if your identity is verified. Otherwise, I’ll send this to the support team for review.”
The first response gives a documented path and asks one clarifying question. The second response recognizes that the request has become account-specific. It does not repeat the cancellation article.
For wording that must stay exact, use an exact Q&A pair rather than relying on a general summary. Refund terms, security instructions, eligibility rules, and legal wording are examples. The point is not to make every response stiff. It is to prevent a source with strict wording from being paraphrased into a different promise.
Answer quality depends on the material behind the agent:
- Website crawl: useful for public product pages, documentation, and current service details.
- Uploaded files: useful for approved PDFs, DOCX files, and TXT files that are not public.
- Pasted text: useful for a short policy or internal instruction.
- Exact Q&A pairs: useful when the question and approved wording are both known.
- Conversation logs: useful for finding questions the source does not cover and answers that customers misunderstood.
AssistLoop lets teams train an agent on their data using these sources. Training is not a one-time publishing task. A pricing page, refund policy, or setup flow can change, and the agent needs the current version.
Here is a common failure:
- The refund policy changes from 30 days to 14 days.
- The old help document remains in the training material.
- A customer asks for a refund on day 20.
- The agent confidently cites the old 30-day policy.
The answer sounds good and is still wrong. A support lead reviewing conversation logs can spot the outdated wording, check the source date, remove or replace the old document, and test the same question again. Logs are where answer quality becomes an operating process instead of a launch-day opinion.
Rasa’s conversation design guidance also treats context as part of the conversation rather than an afterthought. For support teams, that means the agent’s next decision should depend on what the customer has already said.
What good looks like: the numbers
A chatbot benchmark is useful only when you define what counts as success. A high resolution rate can hide a poor experience if the agent closes conversations without answering the question or sends customers to the wrong policy.
Comm100’s 2024 Customer Support Benchmark Report uses chatbot resolution rate as a support measure. The report recorded a 65.7% chatbot resolution rate for the period it studied. Treat that figure as a comparison point, not a target for every support team. Resolution must mean that the customer’s issue was handled, not simply that the conversation ended.
Source: Comm100 Customer Support Benchmark Report. Last verified: August 2026.
For your own review, track the four paths separately:
| Metric | What it measures | Framework connection |
|---|---|---|
| Direct-answer accuracy | Whether the agent’s answer matches the current approved source | Answer |
| Clarification success rate | How often one clarification question identifies the customer’s intent and leads to a useful next step | Clarify |
| Verification completion rate | How often an authenticated customer reaches the required account data or action | Verify |
| Appropriate handoff rate | How often risky, emotional, disputed, or failed cases reach a person with the full context | Hand off |
Start with a dated baseline from your conversation logs. Review a fixed sample, record the customer’s initial intent, and mark whether the final outcome was useful. Then compare the same measures after changing a source or routing rule.
The important number is not the largest share of conversations answered by AI. It is the share that reached the correct path without making the customer repeat the problem. A direct-answer rate that rises while handoff quality falls is not progress. It means the boundary is too wide.
What belongs in the first automation pass
Start with questions your team answers repeatedly from approved material. Do not begin by trying to automate every category in the inbox.
Group the first candidates by intent:
- Product and setup: How do I install the widget? Where is the export button? Which file types can I upload?
- Policies and eligibility: What is the refund window? Which plan includes human handoff? Are annual subscriptions available?
- Troubleshooting: Why is the widget not visible? Why did the upload fail? How do I reconnect an integration?
- Availability: Is this feature available in my plan? Do you support this language, region, or integration?
- Pre-sales qualification: Does the product support our workflow? Can it call our API? How does the team receive a handoff?
For each category, record the source, the decision path, and the fallback. A category is ready for direct answers only when the source is clear and the customer can act on the response without private data or special judgment.
Keep these out of the first direct-answer pass:
- Private account details without verified identity.
- Discretionary refunds, credits, or exceptions.
- Legal, security, or safety judgments.
- Complaints that need an accountable person.
- Requests that require a system action the agent cannot safely perform.
A practical review method takes the last set of conversations your team can access and labels each request as answer, clarify, verify, or hand off. Mark the source used and whether the conversation reached a useful outcome. Automate only categories with a clear source and a clear fallback.
This approach may produce a shorter knowledge base. That is a good result. Dependable answers build trust. A wide agent that invents an answer forces your team to repair trust one conversation at a time.
Where chatbot answers break
A chatbot answer usually breaks at the boundary between a documented fact and a situation that needs context. The wording may be fluent, but fluency does not prove that the source is current, that the customer was understood, or that the agent has the authority to act.
The source is out of date
A confident response can be wrong when a plan, policy, price, or product flow changed. Put an owner on material that changes often. Review the conversations that mention a recent change, then test the old question and the new wording.
The failure is easy to miss because the answer can match the old document exactly. For example, a customer may receive an outdated refund window and have no way to know that the policy changed. Treat every policy change as a training change: identify the affected source, remove stale material, test the old customer wording, and check a live conversation after the update.
The request is ambiguous
“Why was I charged?” could mean a new invoice, a duplicate charge, a renewal, or a usage fee. “Can I change it?” could refer to almost anything. Ask one precise question before choosing a source.
A vague clarification creates another failure. “Can you provide more details?” makes the customer guess what information the agent needs. Ask about the missing object or event instead: “Do you mean a renewal charge, a duplicate invoice, or a usage fee?” If the customer still cannot identify the issue, stop extending the question loop and move the conversation to a person.
The subject carries risk
Billing, authentication, security, complaints, and exceptions need more than fluent text. Use verified identity and a configured account workflow when the agent has the required access. Use human handoff when the case needs judgment or authority.
A published policy can answer “What is your refund window?” It cannot decide whether a disputed charge deserves an exception. A general security article can explain how to reset credentials. It should not ask a customer to disclose a password or make a judgment about a suspected account takeover. Separate information delivery from decisions that affect an individual account.
The customer keeps failing to get an answer
Repeated rephrasing is a stop signal. If the customer has tried twice, the source does not fit, or the agent cannot identify the intent, transfer the full thread to a person. Asking the customer to restate the problem again shifts the system’s failure onto the customer.
The handoff should include the conversation history and the reason for transfer. A support teammate can then see which question was asked, what the agent answered, and where the exchange stopped working. Without that context, the customer pays the cost of the failed automation by starting over.
The source does not cover the real question
Conversation logs show the gap between what your team documented and what customers ask. Review unanswered questions, group similar wording, then decide whether to add a source, write an exact Q&A pair, add a clarification prompt, or create a handoff rule.
Review these failures by decision path rather than by a single overall automation rate:
| Failure found in logs | Likely fix | Correct path |
|---|---|---|
| Correct answer exists but the agent misses the customer’s wording | Add an exact Q&A pair or clearer source text | Answer |
| The request has two plausible meanings | Add a targeted clarification prompt | Clarify |
| The answer changes by account or transaction | Add verified access and the required integration | Verify |
| The customer disputes a charge or requests an exception | Add a handoff rule with the full thread | Hand off |
| The same question fails after repeated attempts | Stop the loop and route to a person | Hand off |
The support lead’s job is not to make the agent answer more questions at any cost. It is to improve the boundary between a useful answer and an unsafe guess.
How to put the framework into AssistLoop
The setup follows the same four-path decision:
- Train an AI agent on approved support content.
- Embed the widget on your site.
- Review conversations and label the outcomes.
- Configure human handoff for cases the agent should not resolve alone.
Start with material your team already trusts. Use website content for public documentation, uploaded files for approved internal material, pasted text for short instructions, and exact Q&A pairs for wording that must stay precise. Then check the conversation logs for questions that still need clarification or a new source.
For work that can be completed safely, Agent Actions can book meetings, capture leads, or call a REST API for workflows such as order status when the required integration is configured. That changes the answer from “Here is a link” to a completed step, but only when the action has the required data and permissions.
The widget also affects trust. Use widget customization to set the agent’s name, greeting, avatar, brand color, light or dark theme, and suggested replies. A clear identity and useful prompt give customers a better idea of what the agent can handle.
The Free plan does not include human handoff, so a team testing this framework should separate answer quality from the production handoff experience. You can test whether the source produces a useful answer, but a full answer-to-human workflow requires a plan that includes handoff.
Source: AssistLoop pricing. Last verified: August 2026.
The operating loop is short: review the logs, fix the source, test the response, and move the question to handoff when the risk is higher than the time saved. When you’re ready to test the boundary with your own support content, create your AI agent.
FAQ
Should a chatbot answer every FAQ?
No. A chatbot should answer FAQs directly only when the source is reliable, the wording is current, and a wrong answer would have a low-risk outcome. Questions involving private data, disputes, or exceptions need another decision path.
How do you know when a chatbot should hand off to a person?
Use handoff when the case involves account risk, strong emotion, a complaint, a discretionary decision, or repeated failed attempts. A handoff is also appropriate when the agent lacks the data or authority needed to resolve the request.
Can a chatbot handle account-specific questions?
Yes, but only when the user’s identity is safely verified and the agent has access to the required account data or action. Without both conditions, it should ask for a safe next step or transfer the conversation.
What should a support team review when chatbot answers are wrong?
Inspect the conversation log first, then check the source material and its date. Test whether the request was ambiguous, and update the training content, clarification prompt, verification flow, or handoff rule that caused the failure.
Written by