AI Assurance Academy · Part 5
End-to-End Case Study and Inspection Readiness
Chapter 20 of 20 · This capstone applies the complete series to an AI-assisted deviation-triage service in a cloud eQMS. It demonstrates how to make a release decision, assemble evidence, respond to a monitoring signal, and explain the system during an inspection.
This capstone applies the complete series to an AI-assisted deviation-triage service in a cloud eQMS. It demonstrates how to make a release decision, assemble evidence, respond to a monitoring signal, and explain the system during an inspection.
Published: September 4, 2026 | Case study version 1.0
Editorial owner: CSV to CSA Knowledge Hub | Review status: Open for practitioner peer review
Scenario type: Fictionalized professional learning case. Educational content; not legal or regulatory advice.
THE PROPOSED SYSTEM
A global manufacturer processes 18,000 deviations each year across six sites. A private SaaS AI service will rank new deviations as routine, elevated, or urgent. It will also retrieve relevant approved procedures and draft a nonauthoritative chronology.
The system will not determine product impact, root cause, CAPA, reportability, batch disposition, or final priority. A trained site Quality investigator makes the decision.
SYSTEM BOUNDARY
- Cloud eQMS deviation record
- API extraction and schema validation
- Deterministic urgent-event rules
- Classification model
- Approved-procedure repository
- Parser, embeddings, index, and access filter
- RAG prompt and hosted LLM
- Investigator review interface
- eQMS priority and chronology fields
- Identity, audit trail, monitoring, and incident systems
- Suppliers for eQMS, model, and cloud
INTENDED USE
“For newly opened English-language deviation records at the six approved manufacturing sites, the system ranks review priority and drafts a chronology from the deviation text and cited effective procedures. An authorized investigator reviews source data, AI rationale, confidence, and citations; selects final priority; edits or rejects the chronology; and remains responsible for the investigation. The system cannot approve, close, assign product impact, determine root cause, create CAPA, or disposition material.”
TOP FAILURE CHAINS
1. Urgent event under-ranked
New terminology or missing context causes a false routine ranking → review is delayed → containment is delayed → quality impact may increase.
2. Unsupported chronology
LLM invents a sequence or confuses a prior event → investigator anchors on false narrative → investigation scope narrows.
3. Wrong procedure
Superseded or wrong-site SOP is retrieved → draft reflects an invalid requirement → user follows incorrect process.
4. Interface mapping error
Negation or unit is truncated → model receives misleading text → ranking and chronology are wrong.
5. Automation bias
Reviewer accepts a high-confidence recommendation without examining the source.
6. Silent supplier change
Hosted model behavior changes → prior test results no longer represent production.
RISK CONTROLS
- Deterministic urgent terms and event rules operate before the model
- Every new deviation remains visible independent of ranking
- Low-confidence and unsupported-language cases route to manual review
- Original text and evidence displayed beside recommendation
- No default acceptance
- Investigator records final priority and override reason
- RAG index includes only effective site-authorized procedures
- Critical claims require source citation
- Model, prompt, index, and application versions recorded
- Daily review of unranked, failed, and low-confidence cases
- Supplier change notification and version pinning where available
- Manual fallback and immediate suspension capability
DATA AND SUPPLIER STRATEGY
Development data is split by deviation case and time. The independent test period is later than training. Rare urgent events are enriched in a separate challenge set. Results are stratified by site, product family, terminology, and event category.
Supplier evidence is leveraged for cloud security, standard model service controls, availability, and hosted release management. Customer evidence covers intended-use performance, eQMS configuration, interfaces, deterministic rules, RAG sources, human review, and operational monitoring.
ACCEPTANCE STRATEGY
Classification:
- Predefined urgent-class recall and lower confidence limit
- Manageable false-alert workload
- Minimum site-level evidence
- Zero unexplained misses in defined critical challenge cases
- Calibration and abstention behavior
RAG and chronology:
- Required procedure retrieved within top-k
- Correct document ID, revision, and section citation
- No unauthorized or superseded source exposure
- No invented event, batch number, disposition, or root cause in critical challenge cases
- Appropriate no-answer behavior
Integrated workflow:
- Correct field mapping and complete records
- Role and access enforcement
- Override and audit trail
- Interface timeout, retry, and duplicate prevention
- Outage fallback and queue reconciliation
Human oversight:
- Representative investigators detect seeded AI errors at the predefined rate
- Review time remains feasible
- Source consultation and escalation are demonstrated
RELEASE DECISION
The model meets aggregate limits but misses two urgent cases from one site using new abbreviations. The deterministic urgent-rule library does not include them.
A mature decision does not simply average the misses into a passing score.
Disposition:
- Add and test deterministic rules for the critical abbreviations
- Expand the site-specific challenge set
- Require manual same-day review of all new deviations at that site during a controlled pilot
- Restrict the initial release to advisory ranking
- Monitor site-specific overrides and time-to-review daily
- Establish an action limit and authority to suspend
- Open a follow-up model improvement change; do not contaminate the locked test set
Residual risk is accepted by the process owner and Quality for a time-limited pilot because the independent manual review prevents reliance on the missed ranking.
MONITORING PLAN
Daily:
- Failed or unranked records
- Interface reconciliation
- Deterministic urgent-rule hits
- Low-confidence cases
- Manual review completion
Weekly during pilot:
- Override and disagreement by site
- Urgent-case misses
- Alert volume and workload
- Citation and chronology sample
- Use outside approved scope
Monthly after stabilization:
- Stratified performance with adjudicated outcomes
- Data and terminology drift
- Supplier/model change
- Incidents, complaints, and deviations
- Benefit and reviewer workload
INCIDENT AFTER THREE MONTHS
Monitoring detects rising overrides at Site 4. Investigation finds a new equipment family and local shorthand absent from development data. Confidence remains high.
Immediate action:
- Suspend AI ranking for affected equipment at Site 4
- Continue deterministic rules and manual review
- Preserve affected records and versions
- Review prior records from the change date
- Notify model and process owners
CAPA:
- Update intake governance so new equipment triggers AI impact assessment
- Add equipment metadata to the operating-envelope check
- Build an adjudicated challenge set
- Reevaluate after controlled model or rule change
- Add a site/equipment drift metric
- Test the effectiveness of both detection and prevention controls
INSPECTION-READY NARRATIVE
Be prepared to explain the system in ten minutes:
1. What the AI does and explicitly does not do
2. Which regulated process and records it supports
3. The component and supplier boundary
4. The most important failure chains
5. Why the selected controls interrupt those failures
6. How data and evaluation represent intended use
7. How human oversight was tested
8. What is recorded for each decision
9. How change and drift are detected
10. One real issue and how the organization responded
EVIDENCE INDEX
- Approved intended use and architecture
- Scope and risk assessment
- Data provenance and dataset versions
- Supplier assessment and agreements
- Requirements and acceptance rationale
- Model, RAG, integration, security, and human-factors results
- Validation summary and residual-risk approval
- Training and procedures
- Monitoring and alert records
- Change, incident, CAPA, and periodic review
- Retirement and fallback plan
INSPECTION QUESTIONS TO REHEARSE
- Show the currently deployed model and prompt versions.
- Why is human review an effective control?
- How do you know Site 4 was affected?
- Which supplier evidence did you rely on?
- How do you prevent superseded SOP retrieval?
- What happens when the service is unavailable?
- Show one override and how it informed monitoring.
- How can you reconstruct an AI-assisted decision?
- Which changes require revalidation?
- Who can suspend the system?
LESSONS LEARNED
- Feature-level intended use prevented uncontrolled expansion
- Deterministic rules protected rare urgent events
- Aggregate accuracy would have hidden site failure
- Human oversight required performance evidence
- Supplier documentation reduced duplication but did not validate the tenant
- Monitoring found a change that launch testing could not predict
- A constrained pilot made residual risk manageable
PROFESSIONAL INTERPRETATION
An inspection-ready AI system is not the one with the largest validation package. It is the one whose owners can show a coherent, current, and evidence-backed chain from intended use to risk, controls, evaluation, release, monitoring, and response.
PRIMARY SOURCES
FDA CSA final guidance:
FDA and EMA, Guiding Principles of Good AI Practice in Drug Development:
FDA, Part 11—Scope and Application:
NIST AI RMF:
www.nist.gov/itl/ai-risk-management-framework
EMA AI reflection paper:
www.ema.europa.eu/en/use-artificial-intelligence-ai-medicinal-product-lifecycle-scientific-guideline
CORRECTIONS AND CONTRIBUTIONS
Use this fictionalized case for team training. Share improvements only in generalized form and remove employer-confidential, patient, product, security-sensitive, and personal information.