AI Assurance Academy · Part 1
From Deterministic Validation to AI Assurance
Chapter 1 of 20 · AI changes the assurance problem because correct behavior can no longer be reduced to “the same input always produces the same output.” A professional CSA strategy therefore validates the complete sociotechnical system: intended use, data, model, application logic, human decisions, operating controls, and evidence over time.
AI changes the assurance problem because correct behavior can no longer be reduced to “the same input always produces the same output.” A professional CSA strategy therefore validates the complete sociotechnical system: intended use, data, model, application logic, human decisions, operating controls, and evidence over time.
Published: September 4, 2026 | Version 1.0
Editorial owner: CSV to CSA Knowledge Hub
Review status: Open for practitioner peer review
Scope: AI-enabled computerized systems used in life-sciences production, quality, laboratory, clinical, safety, or regulatory processes. Educational content; not legal or regulatory advice.
LEARNING OBJECTIVES
- Distinguish conventional software verification from AI system assurance
- Define the validated state for a probabilistic system
- Build an evidence argument around the complete intended-use workflow
- Recognize when a deterministic control should surround an AI component
THE PRACTITIONER POSITION
AI does not make established CSV principles obsolete. Intended use, requirements, supplier management, data integrity, testing, change control, security, and lifecycle review remain essential. What changes is the nature of uncertainty.
Traditional application logic is usually specified as explicit rules. AI behavior is influenced by data distribution, learned parameters, prompts, retrieval sources, model settings, vendor updates, and user context. A test pass at release is therefore necessary but may not be sufficient evidence for continued fitness.
The assurance unit should be the business decision or controlled process—not the model in isolation. A highly accurate classifier can still fail in use if the wrong population is submitted, the interface truncates an input, users misinterpret confidence, or exceptions are not escalated.
THE SEVEN-LAYER ASSURANCE MODEL
1. Intended-use layer
Define the user, decision, input, output, regulated process, record, operating boundary, and prohibited use. “Use AI for investigations” is not testable. “Rank open deviation records for investigator review without changing disposition or approval status” is specific.
2. Process-risk layer
Describe foreseeable failure chains. Consider false acceptance, false rejection, omission, hallucination, bias, stale knowledge, overreliance, unavailable service, and unauthorized action. Connect the failure to product quality, patient safety, subject protection, data integrity, or a required record.
3. Data layer
Control source, provenance, selection, labeling, transformation, representativeness, separation, retention, privacy, and leakage. Data is both an input and part of the design evidence.
4. Model layer
Identify model family, version, training status, configuration, hyperparameters where applicable, known limitations, output variability, and supplier controls. A commercial model name alone is not a configuration record.
5. Application layer
Verify prompts, retrieval, business rules, interfaces, access, workflow states, records, security, and deterministic guardrails. Many production failures occur around the model rather than inside it.
6. Human-control layer
Define who reviews, what evidence they see, when they can override, what requires escalation, how disagreement is recorded, and how competence is maintained. A nominal “human in the loop” is weak if the reviewer cannot detect error or is rewarded for accepting recommendations quickly.
7. Operations layer
Monitor performance, data shift, user behavior, incidents, overrides, supplier changes, latency, availability, and residual risk. Establish triggers for investigation, restriction, rollback, retraining, or retirement.
WHAT “VALIDATED STATE” MEANS FOR AI
For an AI-enabled system, validated state is a controlled state of knowledge. The organization can identify what is deployed, explain the approved use, retrieve the supporting evidence, detect material change, understand current limitations, and demonstrate that performance remains acceptable.
The state is lost when, for example:
- A hosted model changes without assessment
- A prompt or knowledge base changes outside change control
- Data distribution shifts beyond evaluated conditions
- Monitoring is unable to detect the failure that matters
- Users apply outputs to an unapproved decision
- An agent receives a new tool permission without risk review
A PRACTICAL CSA EVIDENCE ARGUMENT
Build the conclusion as a chain:
Approved intended use → credible failure analysis → proportionate controls → representative evaluation → controlled deployment → effective human oversight → monitoring and change response.
Each arrow needs evidence. The final validation summary should explain why the combined evidence is sufficient. It should not merely list documents.
WORKED EXAMPLE: AI-ASSISTED COMPLAINT CODING
Intended use: Suggest one of 12 approved complaint categories to a trained complaint specialist. The user must review source text, may change the suggestion, and remains responsible for final coding. The AI cannot submit, close, or prioritize a complaint.
Foreseeable failures:
- Serious event is assigned a routine category
- Ambiguous wording produces an overconfident suggestion
- Product or language subgroup performs worse than the aggregate
- Interface removes negation or truncates the narrative
- User accepts the recommendation without reading the complaint
Controls:
- Deterministic serious-event keyword and rule checks operate before AI
- Low-confidence and out-of-distribution cases route to manual coding
- The interface shows the original narrative and supporting rationale
- Category change and override are recorded
- Final submission always requires trained human confirmation
- Monitoring is stratified by product, language, seriousness, and site
Assurance activities:
- Independent challenge set with difficult, rare, and ambiguous cases
- Class-specific recall and false-negative analysis, not accuracy alone
- Interface tests for encoding, truncation, negation, and missing fields
- Human-factors study measuring reviewer detection and automation bias
- Security and privacy review of complaint text transmission
- Post-release review of overrides, misses, and subgroup performance
The AI model is only one component of the accepted system.
DETERMINISTIC GUARDRAILS
Use deterministic controls when a boundary must not be probabilistic. Examples include blocking an unapproved model version, enforcing role permissions, preventing direct batch disposition, requiring a signature, applying a fixed range check, validating a schema, or stopping an agent before a critical action.
Do not claim that a guardrail eliminates risk unless its own failure modes and independence are understood.
MINIMUM GOVERNANCE RECORD
- System and model inventory identifier
- Intended and prohibited uses
- Responsible process owner, model owner, data owner, Quality, and IT owner
- Deployed model, prompt, retrieval, application, and configuration versions
- Risk analysis and control rationale
- Evaluation datasets, methods, thresholds, and results
- Supplier evidence and unresolved gaps
- Human-oversight design and training
- Monitoring metrics, limits, and escalation
- Change categories and revalidation triggers
- Residual-risk decision and approval
PRACTITIONER REVIEW QUESTIONS
1. Can the team state the approved decision boundary in one paragraph?
2. Which failure would create the most serious downstream error?
3. Can the reviewer detect that failure before harm?
4. Which component changes behavior most often?
5. What evidence proves current performance, not only launch performance?
6. What deterministic control prevents unauthorized or irreversible action?
PROFESSIONAL INTERPRETATION
AI assurance is not “model validation plus CSV.” It is an integrated control argument for a changing, data-dependent system. The mature CSA team spends less time forcing AI into a legacy template and more time making uncertainty visible, testable, monitored, and governable.
PRIMARY SOURCES
FDA, Computer Software Assurance for Production and Quality Management System Software, Final Guidance, February 2026:
FDA and EMA, Guiding Principles of Good AI Practice in Drug Development:
NIST, AI Risk Management Framework:
www.nist.gov/itl/ai-risk-management-framework
ISPE, GAMP Guide: Artificial Intelligence, published July 2025. Industry guidance; not a regulation:
ispe.org/publications/guidance-documents/gamp-guide-artificial-intelligence