Break the problem into stages
Check source eligibility, extraction and indexing, retrieval relevance, answer support, and user interpretation. A failure at an early stage can make later answer evaluation misleading.
Worked example
Worked example — fictional
A user asks how Site B handles an overdue calibration. The system retrieves Site A’s procedure because its title is similar. The answer accurately summarizes Site A, but it is wrong for the user’s context.
The improvement is not merely asking the model to be more accurate. Examine metadata, filters, ambiguity handling, and whether the system should ask which site applies.
Evidence to collect
For each case, identify the expected relevant source or the reason no answer should be given. Record retrieved passages and their versions. Evaluate whether the answer’s claims are supported and whether omitted exceptions change the meaning. Challenge access boundaries so unauthorized material does not enter retrieval or output.
Completed finding: “The answer is supported by the retrieved passage but fails contextual relevance because the source applies to a different site. This is recorded as an incorrect response, not a successful grounded answer.”
What changes for this type of AI?
Related work: testing, supplier assessment, records, change control, monitoring.
Exercise
Can a high citation rate establish answer quality?
Show suggested answer
No. Citations can be irrelevant, obsolete, or insufficient for the claim. Evaluate the support and applicability of each material claim, not only whether a link is present.
Expected records
A retrieval/source evaluation record separating eligible sources, retrieved passages, claim support, access and final outcome. Adapt record names and detail to the applicable procedures; these are practitioner examples, not a universal mandatory document list.
Relevant regulations and guidance
Regulations set requirements within their scope; guidance offers recommendations or explains applicable expectations; standards depend on adoption or incorporation. Voluntary frameworks do not establish compliance. The lesson's controls, test designs and example records are practitioner suggestions unless explicitly identified otherwise.
Voluntary framework · National Institute of Standards and Technology · NIST AI 600-1 · July 2024
Scope: United States · Cross-sector AI governance and risk management, including generative AI.
Relevance to this lesson: Provides context for information integrity and generative-AI risks; the staged retrieval tests are practitioner examples.
Publication status: Final / published. Source checked 21 September 2026.
Publication identity, scope and status checked against the linked official source. This is not an independent audit of every retained guide statement.
Source details and applicabilityRegulation · U.S. Food and Drug Administration · Current eCFR text
Scope: United States · Electronic records and signatures within FDA Part 11 scope, tied to the underlying required records.
Relevance to this lesson: If the retrieved or approved information forms records in scope, assess the relevant record controls.
Publication status: Final / published. Source checked 21 September 2026.
Publication identity, scope and status checked against the linked official source. This is not an independent audit of every retained guide statement.
Source details and applicability