CSVtoCSAPractical Software Assurance
SearchSubscribe
Menu

Fictional worked example · CSVtoCSA editorial

AI-Assisted Protocol Review

Reference findings, false positives, false negatives, critical misses, source verification, and reviewer performance in one controlled evaluation.

Published 12 September 2026 · Context updated 21 September 2026 · Version 2.0

All organizations, identifiers, test results and decisions in this example are fictional teaching material.

Information available and assumptions

The AI proposes protocol findings for qualified reviewers; it cannot approve a protocol. Independent expected findings include critical issues and scanned-table formats.

Scope and decision record

Standard case study record

Scope and decision fields

Read these fields before the detailed worked case. All organizations, identifiers, test outcomes, and decisions are fictional teaching material.

GxP Assessment
The AI assistant proposes validation-protocol review comments but cannot approve the protocol or create a Quality finding. Parsing, source retrieval, critical-miss performance, and reviewer disposition are GxP-relevant.
Change or Issue
The evaluation must determine whether source-linked potential issues improve review without hiding critical omissions or overloading reviewers with false positives.
Functions Affected
  • Document parsing
  • Requirement and procedure retrieval
  • Potential-issue generation
  • Source display
  • Qualified reviewer disposition
Potential Impact
A missed critical acceptance criterion or an unsupported review comment could weaken protocol approval and later validation evidence.
Deviations / Known Issues
The fictional study misses one obsolete acceptance-limit reference in a scanned table. That format remains manual-review only until parsing and critical-miss retest meet the predefined criteria.
In this example

Situation

An AI-assisted reviewer proposes comments on validation protocols. Qualified reviewers decide whether each comment becomes a Quality finding; the assistant cannot approve a protocol.

Intended Use

The assistant identifies potential missing approvals, inconsistent identifiers, weak acceptance criteria, traceability gaps, and unsupported conclusions in permitted document formats. Every potential issue must show the source passage and requires reviewer disposition.

Failure Scenarios

  • A critical omission is not identified.
  • A correct statement is reported as an issue.
  • An obsolete procedure is retrieved.
  • A plausible comment cites the wrong passage.
  • A reviewer accepts fluent output without checking the source.

Existing Controls and Evidence

Source documents are read-only, approved references are versioned, outputs are labeled as potential issues, reviewers can accept, reject, clarify, or mark not applicable, and final approval remains outside the tool.

Testing Approach

Use an independently labeled reference set stratified by document type, format, issue type, and criticality. Measure correct findings, false positives, false negatives, precision, recall, critical misses, source correctness, and final reviewer decisions. Challenge scanned tables, missing pages, conflicting sources, prompt injection, and repeated execution. Test reviewers under representative workload.

Deviations and Validation Conclusion

In this fictional evaluation, the combined process misses one critical obsolete-limit reference in a scanned table. That format remains manual-review only. The team corrects source-version display and parsing, then repeats the affected challenge set with independent reviewers before expanding use.

Reassessment Criteria

Reassess model, prompt, parser, retrieval, source corpus, document type, interface, reviewer workflow, critical-miss threshold, or supplier-service changes.

Limitations and unresolved questions

The observed scan failure prevents expanded reliance. Supported-format restrictions, source/parsing corrections and reviewer evidence must be assessed before extending use.

Relevant regulations and guidance

These sources apply within their stated scope. The suggested worksheet fields, test designs and fictional decisions are practitioner recommendations, unless a source requirement is expressly identified. Completing a template does not establish compliance.

Computer Software Assurance for Production and Quality Management System Software ↗

Regulatory guidance / GMP guideline · United States · February 2026

Scope: Medical-device manufacturers assuring software used in production or their quality management system.

Relevance: For medical-device production or QMS software, supports selecting assurance activities from intended use, risk and available evidence. It is nonbinding guidance and does not determine other sectors’ obligations.

Status: Final / published · Checked 2026-09-21

Publication identity, scope and status checked against the linked official source. This is not an independent audit of every retained guide statement.

Read source details and applicability →

EudraLex Volume 4, Annex 11 — Computerised Systems ↗

Regulatory guidance / GMP guideline · European Union · January 2011 revision

Scope: Computerised systems supporting activities in the applicable EU pharmaceutical GMP context.

Relevance: For computerised systems used in EU GMP activities, informs lifecycle, supplier and record controls. Confirm applicability to the actual activity; the example does not establish it.

Status: Final / published · Checked 2026-09-21

The Commission index confirms the January 2011 operative revision. The linked PDF could not be retrieved in this check; retained detailed section mappings need review against an authorized copy.

Read source details and applicability →

Artificial Intelligence Risk Management Framework (AI RMF 1.0) and NIST AI 600-1 ↗

Industry guidance / framework · United States · AI RMF 1.0 (2023); GenAI Profile (2024)

Scope: Cross-sector AI governance and risk management, including generative AI.

Relevance: A voluntary framework for describing AI risks, evaluation and ongoing management. Its suggested practices are not regulatory requirements.

Status: Final / published · Checked 2026-09-21

Publication identity, scope and status checked against the linked official source. This is not an independent audit of every retained guide statement.

Read source details and applicability →
Was this useful? Leave feedback

Was this resource useful?

Tell us what helped and what could be clearer. Usefulness feedback does not establish technical validity or regulatory compliance.

Loading contribution options…

Questions and moderated discussion

Questions and discussion

What additional information would you need before making this decision?

Loading contribution options…

Suggest a correction

Sent privately to the editorial moderation queue. Your account email and correction are not published as a comment. Do not include confidential or personal information.

Resource: AI-Assisted Protocol Review

Loading sign-in status…

Correction details