Connect specific AI failures to consequences, controls and a functional risk assessment.
Author
Sandip Thorat
Last updated
21 September 2026
Reading time
2 min
Related video
1:50
On this page
Related recorded lesson
Risk Analysis for AI Failure and Uncertainty
1:50
This recording uses an earlier chapter sequence and may use a different example. The written lesson below contains the current exercise and worksheets.
Read the video transcript
Risk is a failure chain, not a score
Statements such as the model may be wrong are too broad to guide assurance. Build a specific chain: an initiating condition causes an AI failure; the interface or user responds; the error enters the regulated process; a quality, safety, data-integrity, or record consequence becomes possible. Then identify what prevents, detects, or limits the chain. A numerical rating is useful only after this reasoning is visible.
Model uncertainty is only one source
Consider false acceptance, false rejection, hallucination, omission, bias, poor calibration, out-of-distribution input, stale knowledge, retrieval failure, prompt manipulation, silent truncation, unavailable service, and excessive agent authority. Also examine human behavior: reviewers may accept plausible output, skip sources, or use the service outside its approval. Many serious failures arise from interactions around the model rather than the model algorithm itself.
Challenge whether controls can really break the chain
Credit a control only when it addresses the specific failure. Human review is weak if the reviewer lacks the source, time, competence, or authority to challenge the output. Monitoring is weak if it detects harm after the regulated decision is irreversible. Supplier testing is weak if it covers a different model or use. Document control independence, trigger, evidence, owner, response time, and known limitations before reducing risk.
State residual uncertainty honestly
After controls and evidence, state what remains uncertain: rare events, subgroup performance, supplier opacity, future data shift, reviewer behavior, or an incomplete recovery path. Reduce exposure through read-only use, limited populations, confidence routing, staged rollout, or stronger review. The residual-risk decision should name the conditions of acceptance, accountable approver, monitoring signals, containment action, and triggers for reassessment.
Name the error and its consequence
“Hallucination risk” is too general to design an adequate test. Identify the unsupported statement, the use of that statement, and what could happen next.
Worked failure analysis — fictional
Failure
Example
Possible workflow effect
Invented requirement
Claims every test needs a screenshot
Unnecessary work and misleading review findings
Missed gap
Fails to detect absent alarm testing
Reviewer may overlook an important omission
Wrong source
Uses an obsolete procedure
Review applies outdated expectations
Lost qualification
Removes “unless independently verified” from a source summary
Meaning changes
Unauthorized disclosure
Returns another team’s restricted content
Confidential information exposed
Wrong action
Creates a task under the wrong deviation
Investigation becomes misleading or fragmented
Controls need failure-specific support
Citations can help source verification but do not ensure relevance. Human review can help detect errors but needs demonstrated effectiveness. Permission controls can block actions but do not make text correct. Use a combination appropriate to the failure.
Completed risk statement: “If the assistant omits a protocol gap and the reviewer treats the generated list as complete, the final review may miss required coverage. The workflow will preserve full reviewer responsibility and evaluate missed issues in the combined process before any expanded reliance.”
Exercise
Which is more informative: “AI is high risk” or “the assistant can omit an escalation condition from a deviation summary used for triage”?
Show suggested answer
The second statement identifies something that can be challenged, controlled, and connected to a decision. Use the approved rating method after explaining that failure.
Expected records
A functional risk assessment connecting each specific failure to consequence, control, evidence and residual-risk decision. Adapt record names and detail to the applicable procedures; these are practitioner examples, not a universal mandatory document list.
Worksheets
Each worksheet includes instructions, a completed fictional example and a blank version. Adapt it to your applicable procedures and requirements.
Regulations set requirements within their scope; guidance offers recommendations or explains applicable expectations; standards depend on adoption or incorporation. Voluntary frameworks do not establish compliance. The lesson's controls, test designs and example records are practitioner suggestions unless explicitly identified otherwise.
Guidance · International Council for Harmonisation (ICH); regional implementation by FDA and other authorities · R1; January 2025 typographical correction
Scope: International · Quality risk management across the pharmaceutical product lifecycle.
Relevance to this lesson: Provides pharmaceutical quality-risk context; it is not a universal device or AI risk-classification rule.
Publication status: Final / published. Source checked 21 September 2026.
Publication identity, scope and status checked against the linked official source. This is not an independent audit of every retained guide statement.
Educational chapter · Editorial version 2.0. The update date records substantive site editing; a completed independent technical review is not claimed.
AI tools assisted drafting and structural editing. Examples and results labelled fictional or synthetic are teaching material. Adapt the methods to the actual intended use, applicable requirements and approved procedures.
Estimated reading time uses the written lesson at 220 words per minute, with links and Markdown formatting removed. Video time comes from the media file and is separate.
Sent privately to the editorial moderation queue. Your account email and correction are not published as a comment. Do not include confidential or personal information.
Tell us what helped and what could be clearer. Usefulness feedback does not establish technical validity or regulatory compliance.
Loading contribution options…
Questions and discussion
Ask a question or share a perspective. Contributions are reviewed before publication.
Loading contribution options…
CSV to CSA Newsletter
Practical updates for Quality and Compliance professionals
Read practical examples covering software validation, risk-based testing, supplier assessment, maintaining the validated state, and AI assurance. Selected regulatory updates explain the source, its scope, and what practitioners should consider.