AI Assurance Academy · Part 4
Assuring Retrieval-Augmented Generation
Chapter 13 of 20 · Retrieval-augmented generation can improve source grounding, but it creates an eight-stage assurance chain. A correct source may be excluded, split badly, retrieved incorrectly, ignored by the model, or cited inaccurately. This chapter treats RAG as a controlled knowledge system.
Retrieval-augmented generation can improve source grounding, but it creates an eight-stage assurance chain. A correct source may be excluded, split badly, retrieved incorrectly, ignored by the model, or cited inaccurately. This chapter treats RAG as a controlled knowledge system.
Published: September 4, 2026 | Version 1.0
Editorial owner: CSV to CSA Knowledge Hub | Review status: Open for practitioner peer review
THE EIGHT-STAGE RAG CHAIN
1. Sources
Approved repositories, document states, metadata, and access rights.
2. Ingestion
Extraction of text, tables, images, attachments, metadata, and version information.
3. Indexing
Chunking, embeddings, identifiers, metadata, filters, and update logic.
4. Retrieval
Query processing, semantic or lexical search, ranking, filtering, and top-k selection.
5. Prompt assembly
System instruction, retrieved context, user question, tool output, and policy constraints.
6. Generation
Model behavior, parameters, citation format, refusal, and uncertainty.
7. Human review
Source visibility, authority, competence, override, and escalation.
8. Record and monitoring
Version, query, retrieved evidence, output, user action, feedback, and operational trends.
SOURCE GOVERNANCE
Define eligible document types, approval state, effective date, site, product, language, confidentiality, retention, and supersession. The search result should link to the authoritative repository, not become a new uncontrolled copy.
Reconcile expected documents to indexed documents. Confirm that deletion, supersession, and access changes propagate within an approved interval.
INGESTION TESTS
- Headings and paragraph order
- Tables, footnotes, equations, and symbols
- Scanned or OCR text
- Page headers and repeated footers
- Attachments and cross-references
- Document ID, version, and effective date
- Special characters, units, and decimal values
- Multi-column layout
- Empty or corrupted files
- Access labels
Do not assume successful upload means faithful extraction.
CHUNKING AND INDEX CONTROL
Chunk boundaries can separate a condition from its instruction or a table header from its values. Test critical document structures. Retain the chunking algorithm, embedding model, metadata schema, and index version.
Use deterministic filters for approval status, site, role, or product when those boundaries must not depend on semantic similarity.
RETRIEVAL EVALUATION
Create questions with known required sources. Measure recall at k, ranking, and authorization. Include:
- Synonyms and abbreviations
- Product and site-specific language
- Negative questions
- Multiple documents needed for one answer
- Conflicting or superseded sources
- Similar titles and procedures
- Tables and attachments
- Questions with no supported answer
Evaluate retrieval before generation. If the required source is absent, the model should not be expected to create a correct answer.
GENERATION AND CITATION TESTS
At claim level, ask:
- Is the claim supported by retrieved text?
- Does the citation point to the correct document, version, and section?
- Is any critical condition omitted?
- Does the answer combine incompatible site or product instructions?
- Does it clearly state uncertainty or no-answer?
- Can the user open the authoritative source?
Citation presence is not citation correctness.
ACCESS-CONTROL THREAT
Authorization must be enforced before retrieval and after generation where appropriate. Test:
- User without source permission
- Shared embeddings or index across business units
- Query that attempts to infer restricted content
- Citation URL access
- Cached responses
- Logs containing source text
- Administrator and support access
PROMPT INJECTION FROM SOURCES
A retrieved document may contain text such as “ignore previous instructions” or hidden content. Treat sources as data, not trusted instructions. Separate control instructions from retrieved text, restrict tools, sanitize risky formats, and test malicious documents.
WORKED EXAMPLE: SOP Q&A ASSISTANT
Approved use: Help trained operators locate relevant sections of effective SOPs. The authoritative SOP remains the source of instruction.
Critical controls:
- Only effective documents eligible
- Site and role access filters
- Exact document ID and revision shown
- Source passage displayed beside the answer
- No-answer response when evidence is insufficient
- User opens and follows the controlled SOP
- Query, versions, retrieved chunks, output, and feedback retained as defined
Challenge cases:
- “What is the cleaning hold time?” when two products differ
- Superseded SOP has a higher semantic match
- Table header is separated from the value
- User asks for an unapproved shortcut
- Source contains embedded prompt injection
- Access-restricted procedure shares terminology
- Correct answer requires both SOP and form instruction
RAG CHANGE CONTROL
Assess changes to source repositories, parser, OCR, chunking, embedding model, metadata, filters, ranking, top-k, prompt, foundation model, citation renderer, access integration, and feedback loop. A source update can change behavior without a model change.
RAG EVIDENCE PACKAGE
- Approved source inventory and reconciliation
- Ingestion and metadata test
- Index and embedding version
- Retrieval evaluation results
- Claim-level groundedness and citation results
- Access-control and injection tests
- Human-use evaluation
- Change and monitoring plan
- Known limitations and residual risk
PROFESSIONAL INTERPRETATION
RAG reduces one source of uncertainty only when the retrieval and source-governance chain is controlled. The strongest design lets the user inspect the authoritative evidence and safely refuses unsupported questions.
PRIMARY SOURCES
NIST Generative AI Profile:
NIST AI RMF:
www.nist.gov/itl/ai-risk-management-framework
MHRA GxP Data Integrity Guidance:
www.gov.uk/government/publications/guidance-on-gxp-data-integrity
European Commission 2025 consultation on revised Annex 11 and draft Annex 22. Draft material; distinguish from current requirements: