An AI reviewer can save time and still weaken a review

An assistant produces a list of protocol findings in seconds. The reviewer finishes faster. That looks promising, but time alone does not establish whether the review improved.

The assistant might help the reviewer find a subtle unit mismatch. It might also distract the reviewer with false findings or create an impression that everything outside the generated list has already been checked.

Examine the final decision

Fictional example. An evaluation set contains 40 reference issues. The assistant identifies 32 and misses 8, including 2 critical issues. Its proposed findings include 6 false findings. A human may correct those errors, but that must be evaluated rather than assumed.

If reviewers accept false findings and miss the same critical issues, the combined workflow does not become adequate merely because a person clicked approve. If reviewers identify additional issues and reject unsupported suggestions, that is useful evidence about the combined process, subject to the study’s design and limits.

Count the full effort

Include time spent preparing inputs, reviewing sources, dismissing false findings, correcting output, documenting decisions, and resolving tool failures. Record whether different experience levels affect outcomes. A demonstration by the developer is not the same as use by representative reviewers.

Protect the comparison

Use a planned evaluation with suitable cases and a fair comparison. Consider assignment order, learning effects, repeated templates, and independent adjudication. Preserve unfavorable findings. Describe uncertainty and avoid claiming that an informal pilot proves a general improvement.

Improve the workflow

Make sources accessible. Avoid language that promises completeness. Keep the assistant’s authority narrow. Train reviewers to challenge plausible output and to perform the required review. Evaluate those controls under representative conditions.

A useful AI reviewer is one whose role and limitations are understood and whose contribution is supported by evidence. Fast generation is an input to that evaluation, not its conclusion.