A person in the workflow is not automatically an effective control
The reviewer needs suitable competence, time, source access, authority, and a usable way to reject or escalate output. The interface should make the generated status clear and preserve the final human decision.
Worked example
Worked evaluation — fictional
Reviewers assess representative protocols under a planned comparison of assisted and unassisted workflows. Include false AI findings, missed issues, ambiguous source material, and difficult cases. Evaluate final reviewer outcomes as well as time.
Balance or randomize assignment where appropriate, avoid exposing the same cases in a way that creates a memory advantage, and define how disagreements are adjudicated. Keep the analysis aligned with the study design rather than claiming causal improvement from an informal demonstration.
Useful observations
Did reviewers open the source? Did they accept unsupported claims? Did the assistant cause them to miss issues they would otherwise find? Were false findings resolved? Did time savings persist after corrections and documentation?
Completed control description: “Reviewers verify each accepted finding against the approved source and complete the required protocol review independently of the assistant’s issue list. Training includes rejection of plausible false findings and escalation of unsupported cases.”
Reviewer prerequisites
For the fictional PRA-01 protocol-review evaluation, each participant needs relevant process knowledge, access to approved source documents, an understanding of the assistant's limitations, sufficient review time, authority to reject and escalate, and training in the actual review procedure. Record evidence of these prerequisites. Training attendance alone does not establish effective review.
Challenge the claimed human control
These are test situations to include in the evaluation design. The table is a practitioner suggestion, not a prescribed regulatory test set.
A proportionate evaluation approach
- Define the review task, proposed reliance, failure consequences and acceptance criteria before execution.
- Prepare representative cases with expected results established independently of the AI and the study participants.
- Include incorrect outputs, omissions, conflicting sources and ambiguous cases as well as correct outputs.
- Evaluate qualified users who represent the proposed user population, with realistic source access and time constraints.
- Compare assisted and unassisted review where this can answer the decision question without compromising required review.
- Balance assignment and order where appropriate. Do not let a participant see the same case in both conditions or reveal expected answers before their remaining tasks.
- Score final review outcomes, including critical misses and accepted false findings; also observe source checks and escalation.
- Record coverage limits, deviations and corrective actions. Satisfaction and time saved are secondary observations.
- Retest affected controls on suitable fresh cases before relying on the claimed improvement.
Fictional results: component and complete workflow
Six qualified reviewers each complete four sessions: two assisted and two unassisted. Across 24 sessions, each of the 12 documents is reviewed once in each condition by different reviewers. Each condition therefore contains the same 40 distinct reference issues, including 10 critical issues. No participant sees a document twice. Case assignment and order are balanced, and scoring is independently adjudicated.
The frozen AI outputs are the Chapter 11 results: 32 correct and six false findings, with eight reference issues missed. In the assisted condition, reviewers find four of those eight omissions, including one of the two critical omissions. They reject four of the six false AI findings but accept two. For this simplified teaching example, reviewers introduce no further false findings.
The assisted workflow still misses one critical issue. It fails the fictional acceptance criterion for expanded reliance. Zero critical misses in the small unassisted condition does not prove zero underlying risk, and this limited repeated-document design does not establish a causal advantage for either approach. Reviewers and documents contribute dependent observations; report this when planning statistical analysis or a larger evaluation.
Act on ineffective review
Do not approve the proposed reduction in independent review. Remove misleading completeness wording, require source checks, make rejected and unresolved findings visible, and investigate whether the task, interface, procedure or time allowance contributed to the miss. Restrict intended use, add independent checks or withhold the use if those controls cannot be supported.
Record the critical miss as DEV-02 and the two accepted false findings as DEV-03. Update the affected requirements and test the changed workflow with representative qualified users and protected fresh cases. A new training attendance list, satisfaction survey or shorter average review time cannot close these deviations by itself.
Exercise
Reviewer satisfaction rises, but critical issues remain missed in final reviews. Is the oversight control demonstrated?
Show suggested answer
Satisfaction is useful usability feedback, but the remaining missed issues require assessment. Do not claim the control is effective for the intended reliance without adequate outcome evidence.
Expected records
A human-review evaluation plan, prerequisite evidence, case assignments, independently scored final outcomes and corrective-action/retest records. Adapt record names and detail to the applicable procedures; these are practitioner examples, not a universal mandatory document list.