A fluent answer can fail in several ways
An assistant can invent a fact, omit a qualifying condition, overstate certainty, or combine true statements into an unsupported conclusion. Review meaning at the claim level.
Three practical examples
Invented rule: “Every validation test requires a screenshot.” The answer presents an unsupported universal instruction.
Omitted condition: A procedure permits an exception only after a specified approval. The summary describes the exception and leaves out the approval condition.
Unsupported completeness: The assistant says “No gaps exist” after reviewing only selected sections. The wording exceeds the evidence.
Design a useful response boundary
Encourage source-linked claims and clear statements of missing information. Provide a way to ask for clarification or decline to answer within the application’s task. Test whether those behaviors actually occur when needed. Avoid displaying confidence language that users could mistake for a validated probability.
Example corrected output: “I did not identify a mismatch in the sections provided. I did not review the omitted appendices or establish overall protocol completeness. The reviewer should verify the cited sections against the approved requirement set.”
What changes for this type of AI?
Related work: testing, supplier assessment, records, change control, monitoring.
Exercise
The assistant never invents source titles but frequently misses exceptions. Is hallucination testing enough?
Show suggested answer
Expand content evaluation to omissions and meaning changes. Different errors require different challenge cases and metrics.
Expected records
A claim-level findings record identifying unsupported statements, omissions, meaning changes and required escalation. Adapt record names and detail to the applicable procedures; these are practitioner examples, not a universal mandatory document list.