AI-Assisted Document Review

This method evaluates an assistant that suggests document-review findings. It does not treat generated findings as approved quality conclusions.

Step 1: Define a narrow review task

For example: identify candidate mismatches between a draft protocol and an approved requirement set. Exclude final approval, interpretation beyond the approved scope, and unsupported claims of completeness.

Step 2: Establish reference findings independently

Qualified reviewers examine the test documents using an agreed rubric. Record disagreements and their resolution. Keep the final evaluation set separate from prompt development. Include clean documents as well as documents containing known issues.

Step 3: Score useful and harmful behavior

Count correct findings, false findings, and missed reference issues. Review severity and issue type separately. A large number of easy findings must not hide missed consequential issues.

Synthetic example

The evaluation set contains 40 reference issues: 10 classified as critical for this teaching exercise and 30 other issues. The assistant produces 38 distinct findings. Review confirms 32 correct findings and 6 false findings. It misses 8 reference issues, including 2 critical issues.

Precision is 32/38 = 84.2%. Recall is 32/40 = 80.0%. Critical-issue recall is 8/10 = 80.0%. These are illustrative results, not measured results for any real tool. The small critical subset leaves substantial uncertainty.

Step 4: Evaluate the combined workflow

Give reviewers representative cases with and without assistance under a controlled study plan. Check what errors remain in their final decisions, how they use sources, and how much time review actually takes. Do not assume human oversight catches every AI miss.

Reusable worksheet

Task; permitted use; reference-set method; separation from development; issue taxonomy; finding-matching rules; metrics and denominators; severity results; reviewer outcomes; limitations; decision; monitoring and reassessment criteria.