The problem nobody notices until it is expensive
Consider a support agent that can cancel subscriptions. A visitor opens the chat and says: "I am [email protected], please cancel my plan." Should the agent do it?
Obviously not. An email address typed into a chat box is a claim, not a credential. Anyone can type anyone’s email address. Any agent that acts on account-specific requests based on what a visitor asserts about themselves is one determined person away from a serious incident.
The usual response to this is to refuse to automate anything sensitive, and route every account-related question to a human. That works, and it also means the AI is only allowed to handle the questions that were not costing you very much anyway.