AI Assurance Academy · Part 2
Risk Analysis for AI Failure and Uncertainty
Chapter 6 of 20 · AI risk analysis must move beyond a generic statement that “the model may be wrong.” The practitioner must identify how a specific error arises, how it propagates through the process, whether it can be detected, and which controls actually interrupt the chain.
AI risk analysis must move beyond a generic statement that “the model may be wrong.” The practitioner must identify how a specific error arises, how it propagates through the process, whether it can be detected, and which controls actually interrupt the chain.
Published: September 4, 2026 | Version 1.0
Editorial owner: CSV to CSA Knowledge Hub | Review status: Open for practitioner peer review
LEARNING OBJECTIVES
- Build a feature-level AI failure chain
- Separate model uncertainty from process consequence
- Evaluate detectability and control independence
- Link risk directly to assurance intensity and monitoring
THE AI FAILURE CHAIN
Use this structure:
Hazardous situation or process need → initiating condition → AI failure mode → interface or human response → downstream process effect → quality or safety consequence → available detection → residual risk.
Example:
Rare visual defect → insufficient representation in training data → classifier produces a confident false accept → reject gate remains open → defective unit continues → product nonconformity may reach the next process → downstream sampling has limited chance of detection.
This chain supports a better test than “verify model accuracy.”
TWELVE PRACTICAL AI FAILURE MODES
1. False negative
The system misses the condition of interest. Often critical in anomaly, defect, or safety signal detection.
2. False positive
The system identifies a condition that is not present. Consequences can include unnecessary rejection, investigation burden, or alarm fatigue.
3. Miscalibration
Confidence does not reflect the observed likelihood of correctness. Users may overtrust a high score.
4. Out-of-distribution use
Input differs materially from evaluated data: new product, instrument, language, site, image condition, population, or workflow.
5. Data or concept drift
The relationship between inputs and expected outputs changes over time.
6. Hallucination or unsupported generation
The system produces fluent content not supported by approved sources or available evidence.
7. Retrieval failure
The RAG system finds an irrelevant, obsolete, incomplete, unauthorized, or inaccessible source.
8. Automation bias
The reviewer accepts the output despite contradictory evidence.
9. Interface distortion
Data is truncated, reordered, encoded incorrectly, rounded, or mapped to the wrong field.
10. Silent component change
Model, prompt, embedding, index, safety layer, or vendor routing changes without assessment.
11. Adversarial manipulation
An attacker or untrusted source influences training data, retrieval, prompts, or tool execution.
12. Excessive authority
An agent performs an action beyond approved scope or cannot reliably stop, reverse, or explain it.
RISK DIMENSIONS THAT CHANGE ASSURANCE RIGOR
Consequence
What happens if the output is wrong, missing, late, biased, or unauthorized?
Reliance
Is it informational, advisory, decision-supporting, decision-automating, or action-taking?
Uncertainty
How variable is the output across inputs, time, sampling, model versions, or repeated runs?
Detectability
Can a qualified human or independent control identify the error before impact? Measure this where important; do not assume it.
Control independence
Does the control depend on the same model, same data, or same failure mechanism? A second LLM reviewing the first may not be independent.
Exposure
How frequently and broadly is the system used? A low-probability failure can matter at scale.
Reversibility
Can the decision or action be stopped or corrected before harm? A draft is more reversible than a released batch decision.
EVIDENCE QUALITY
How representative, independent, current, and reproducible is the evaluation?
RISK-TO-EVIDENCE MAPPING
Lower consequence, high detectability, reversible use
Potential evidence: supplier assessment, configuration check, scenario test, sampled output review, basic monitoring.
Moderate consequence or uncertainty
Potential evidence: independent challenge set, stratified metrics, boundary testing, human-factors study, targeted scripted and exploratory testing, stronger monitoring.
High consequence, low detectability, or action-taking use
Potential evidence: formal design review, independent evaluation, predefined statistical acceptance, robust negative and boundary tests, deterministic interlocks, role segregation, approval gates, continuous surveillance, tested rollback, and stronger change control.
This is not a fixed matrix. Explain why each activity reduces uncertainty or interrupts a failure chain.
WORKED EXAMPLE: DEVIATION-SEVERITY TRIAGE
An NLP model ranks deviations as routine, elevated, or urgent for investigator queueing. It does not assign final severity.
Failure chains:
False low ranking → urgent record waits → containment or escalation is delayed → product impact may increase.
False high ranking → queue is flooded → investigators disregard alerts → a future true urgent case may be overlooked.
Subgroup disparity → terminology from one site is systematically under-ranked → site-specific urgent cases wait longer.
Potential controls:
- Deterministic rules for known critical words, products, and events
- Urgent items remain visible even without AI prioritization
- Required daily review of unranked or low-confidence cases
- Queue display shows evidence and uncertainty
- User can override with reason
- Stratified performance by site, product, language, and event type
- Time-to-review monitoring and retrospective miss analysis
Test design:
- Enrich the challenge set with rare serious cases
- Include negation, abbreviations, misspellings, copied text, and contradictory statements
- Evaluate urgent-class recall and false negatives separately
- Measure reviewer detection when the model is wrong
- Compare operational time-to-review with and without AI
CONTROL EFFECTIVENESS QUESTIONS
- Does the control address the exact failure?
- Is it preventive or detective?
- Is it independent?
- Is it timely?
- Is it available during outage or peak volume?
- Is its performance measured?
- Can users bypass it?
- Is execution recorded?
RISK REGISTER ENTRY
Use case and feature:
Failure mode:
Initiating conditions:
Affected inputs or subgroups:
Process effect:
Potential consequence:
Existing controls and evidence:
Detectability:
Selected assurance activities:
Monitoring and alert:
Residual risk and owner:
Review trigger:
PRACTITIONER REVIEW QUESTIONS
1. What is the most dangerous plausible false negative?
2. What error appears safe only because human review is assumed?
3. Are rare cases sufficiently represented?
4. Does a control share the model’s failure mode?
5. What change could invalidate the risk conclusion?
PROFESSIONAL INTERPRETATION
AI risk is not equivalent to model error rate. Risk emerges from the interaction of error, use, authority, detectability, and process controls. A modest model can be safe in a constrained workflow; a high-performing model can be unsafe when its output silently drives an irreversible decision.
PRIMARY SOURCES
FDA CSA final guidance, risk-based framework:
www.fda.gov/media/188844/download
FDA draft AI credibility guidance for drug and biological product regulatory decision-making. Draft; not for implementation:
NIST AI RMF 1.0:
www.nist.gov/itl/ai-risk-management-framework
NIST Generative AI Profile: