Situation
A retrieval-augmented AI assistant proposes related deviations, common themes, and missing investigation questions for a qualified investigator.
Intended Use
Support investigation preparation using approved records. The AI cannot classify severity, determine root cause, approve a conclusion, initiate CAPA, or close a record.
Decision Summary
| Decision element | Case record |
|---|---|
| GxP assessment | Missed critical similarity, false linkage, obsolete source, privacy exposure, or automation bias can affect investigation quality and timeliness. |
| Potential impact | The investigation can overlook recurring failure or pursue an unsupported conclusion. |
| Known issue | The fictional evaluation misses two related deviations stored only in scanned attachments after a parser change. |
| Validation conclusion | Scanned attachments are excluded and the update is not released until critical-miss criteria and reviewer controls pass. |
GxP Assessment
Missed critical similarity, false linkage, obsolete source, privacy exposure, or automation bias can affect investigation quality and timeliness.
Affected GxP Functions
- Record retrieval and similarity
- Source version and access
- Proposed theme and question generation
- Reviewer disposition
- Monitoring and change assessment
Failure Scenarios
- Critical prior deviation is missed
- Unrelated record is presented as similar
- Restricted record is retrieved
- Unsupported root-cause wording is accepted
- Model update changes performance without detection
Potential Impact
The investigation can overlook recurring failure or pursue an unsupported conclusion.
Existing Controls
- Read-only approved corpus
- Source citation
- Role-filtered retrieval
- Qualified reviewer disposition
- No autonomous Quality action
Existing Evidence
- Independently labeled reference set
- Security assessment
- Human-workflow study
- Audit and version records
Evidence Gaps
- Rare critical events
- Poor-quality attachments
- New model version
- Reviewer workload effect
Testing Approach
- Evaluate correct links, false links, missed critical links, and source correctness by record type and criticality
- Challenge prompt injection and restricted access
- Test reviewer detection and override under representative workload
- Verify monitoring, stop, rollback, and revalidation criteria
Deviations or Known Issues
The fictional evaluation misses two related deviations stored only in scanned attachments after a parser change.
Validation Conclusion
Scanned attachments are excluded and the update is not released until critical-miss criteria and reviewer controls pass.
Reassessment Criteria
- Model, prompt, parser, retrieval, source, access, or workflow change
- Critical miss or privacy incident
- Acceptance-criteria or supported-population change