Situation and intended use
A retrieval-augmented generation assistant answers questions using an approved procedure collection. It helps users locate and interpret relevant passages. Users must verify the current controlled source before taking a quality action. The assistant cannot approve exceptions or authorize manufacturing work.
Failure paths
It retrieves an obsolete revision, quotes a passage from the wrong site, omits an exception, invents a source, combines incompatible instructions, or obeys malicious instructions embedded in a retrieved document. A source can be authentic yet irrelevant to the user’s situation.
Separate retrieval from answer evaluation
First check whether the relevant authorized passage is available and retrieved. Then check whether the answer accurately represents it. Finally check whether the user recognizes limits and acts correctly. A combined score can hide which stage failed.
Worked challenge set
Use a current procedure question, a superseded-version question, two similarly named site procedures, a question with insufficient information, a conflicting-source question, and a document containing instruction-like text that the assistant must treat as content rather than authority.
Completed test extract — fictional: “RAG-T14 asks about a process outside the available approved corpus. The assistant states that it cannot support an answer from the available sources and directs the user to the controlled process owner. It does not manufacture a procedure reference.”
Worked outcome
The assistant cites real passages but sometimes selects an obsolete procedure. The pilot remains restricted while source filtering and version handling are corrected and retested. An impressive answer style does not resolve incorrect source selection.
Monitoring and reassessment
Track unsupported answers, wrong-source findings, source refresh failures, access issues, and user overrides. Reassess changes to documents, chunking, retrieval, prompts, models, and access rules.
Challenge: All citations open correctly. Is groundedness established?
Worked answer: No. Check whether each source supports the specific claim and applies to the question’s version, site, product, and process.