Situation and intended use
A retrieval-augmented generation assistant answers questions using an approved procedure collection. It helps users locate and interpret relevant passages. Users must verify the current controlled source before taking a quality action. The assistant cannot approve exceptions or authorize manufacturing work.
Failure paths
It retrieves an obsolete revision, quotes a passage from the wrong site, omits an exception, invents a source, combines incompatible instructions, or obeys malicious instructions embedded in a retrieved document. A source can be authentic yet irrelevant to the user’s situation.
Separate retrieval from answer evaluation
First check whether the relevant authorized passage is available and retrieved. Then check whether the answer accurately represents it. Finally check whether the user recognizes limits and acts correctly. A combined score can hide which stage failed.
Worked challenge set
Use a current procedure question, a superseded-version question, two similarly named site procedures, a question with insufficient information, a conflicting-source question, and a document containing instruction-like text that the assistant must treat as content rather than authority.
Completed test extract — fictional: “RAG-T14 asks about a process outside the available approved corpus. The assistant states that it cannot support an answer from the available sources and directs the user to the controlled process owner. It does not manufacture a procedure reference.”
Worked outcome
The assistant cites real passages but sometimes selects an obsolete procedure. The pilot remains restricted while source filtering and version handling are corrected and retested. An impressive answer style does not resolve incorrect source selection.
Monitoring and reassessment
Track unsupported answers, wrong-source findings, source refresh failures, access issues, and user overrides. Reassess changes to documents, chunking, retrieval, prompts, models, and access rules.
Challenge: All citations open correctly. Is groundedness established?
Worked answer: No. Check whether each source supports the specific claim and applies to the question’s version, site, product, and process.
Limitations and unresolved questions
Evidence applies to the assessed sources, formats and questions. Unsupported documents and changed retrieval behavior require reassessment; fluent answers are not proof of completeness.
Relevant regulations and guidance
These sources apply within their stated scope. The suggested worksheet fields, test designs and fictional decisions are practitioner recommendations, unless a source requirement is expressly identified. Completing a template does not establish compliance.
Computer Software Assurance for Production and Quality Management System Software ↗
Regulatory guidance / GMP guideline · United States · February 2026
Scope: Medical-device manufacturers assuring software used in production or their quality management system.
Relevance: For medical-device production or QMS software, supports selecting assurance activities from intended use, risk and available evidence. It is nonbinding guidance and does not determine other sectors’ obligations.
Status: Final / published · Checked 2026-09-21
Publication identity, scope and status checked against the linked official source. This is not an independent audit of every retained guide statement.
Read source details and applicability →EudraLex Volume 4, Annex 11 — Computerised Systems ↗
Regulatory guidance / GMP guideline · European Union · January 2011 revision
Scope: Computerised systems supporting activities in the applicable EU pharmaceutical GMP context.
Relevance: For computerised systems used in EU GMP activities, informs lifecycle, supplier and record controls. Confirm applicability to the actual activity; the example does not establish it.
Status: Final / published · Checked 2026-09-21
The Commission index confirms the January 2011 operative revision. The linked PDF could not be retrieved in this check; retained detailed section mappings need review against an authorized copy.
Read source details and applicability →Was this useful? Leave feedback
Was this resource useful?
Tell us what helped and what could be clearer. Usefulness feedback does not establish technical validity or regulatory compliance.
Loading contribution options…
Questions and moderated discussion
Questions and discussion
What additional information would you need before making this decision?
Loading contribution options…
Suggest a correction
Sent privately to the editorial moderation queue. Your account email and correction are not published as a comment. Do not include confidential or personal information.
Resource: RAG procedure reviewer: a citation can still be wrong
Loading sign-in status…