Situation
An AI-assisted reviewer proposes comments on validation protocols. Qualified reviewers decide whether each comment becomes a Quality finding; the assistant cannot approve a protocol.
Intended Use
The assistant identifies potential missing approvals, inconsistent identifiers, weak acceptance criteria, traceability gaps, and unsupported conclusions in permitted document formats. Every potential issue must show the source passage and requires reviewer disposition.
Failure Scenarios
- A critical omission is not identified.
- A correct statement is reported as an issue.
- An obsolete procedure is retrieved.
- A plausible comment cites the wrong passage.
- A reviewer accepts fluent output without checking the source.
Existing Controls and Evidence
Source documents are read-only, approved references are versioned, outputs are labeled as potential issues, reviewers can accept, reject, clarify, or mark not applicable, and final approval remains outside the tool.
Testing Approach
Use an independently labeled reference set stratified by document type, format, issue type, and criticality. Measure correct findings, false positives, false negatives, precision, recall, critical misses, source correctness, and final reviewer decisions. Challenge scanned tables, missing pages, conflicting sources, prompt injection, and repeated execution. Test reviewers under representative workload.
Deviations and Validation Conclusion
In this fictional evaluation, the combined process misses one critical obsolete-limit reference in a scanned table. That format remains manual-review only. The team corrects source-version display and parsing, then repeats the affected challenge set with independent reviewers before expanding use.
Reassessment Criteria
Reassess model, prompt, parser, retrieval, source corpus, document type, interface, reviewer workflow, critical-miss threshold, or supplier-service changes.