# AI Evaluation Scorecard

Version 2.0 · Updated 2026-09-21 · Testing

Record expected results, AI findings, missed issues, and performance measures for a defined task.

Adapt this template to your intended use, applicable requirements, and approved procedures.

## When to use it

Use before documenting the corresponding assessment or decision.

## Information needed before starting

- Defined task, assessment unit and supported document formats
- Independent reference issues with severity and predefined matching rules
- Recorded AI findings, missed issues and complete workflow observations

## Field instructions

- **Task:** Describe the specific task, users, outputs and decisions supported. Identify the assessed environment and limits of reliance.
- **Assessment unit:** Define the unit being scored, such as an individual issue, document or final review decision. Use consistent units in all denominators.
- **Reference issues:** Record independently established expected issues, their severity and source. Explain adjudication and coverage limits.
- **Correct findings:** Count AI findings that match an independently confirmed reference issue using the predefined matching rules.
- **False findings:** Count generated findings that are unsupported or incorrect under the matching rules. Preserve their disposition.
- **Missed reference issues:** Count expected issues the AI failed to identify; retain the issue identity and severity rather than reporting only an aggregate.
- **Precision:** Calculate correct findings / (correct + false findings). Report counts and state not estimable when the denominator is zero.
- **Recall:** Calculate correctly identified reference issues / all reference issues. Report counts and state not estimable when there are no reference issues.
- **Critical recall:** Calculate detected critical reference issues / all critical reference issues. List each critical miss and its consequence; overall recall must not hide it.
- **Key limit:** State dataset size, representativeness, exclusions and other limitations that prevent generalising the observed results.
- **Workflow concern:** Describe how AI output could mislead or overload the reviewer, and what evidence is needed about the complete human-review workflow.
- **Decision:** Record the supported use or restriction, the evidence and unresolved issues behind it, and the accountable decision owner.

Define the task and unit of analysis before entering numbers. Record the reference-set method and protected evaluation boundary. Report subgroup results and limitations.

Are denominators clear? Are duplicate findings handled consistently? Are the cases independent enough for the analysis? Does the conclusion reflect missed consequential issues?

## Blank template

System / record ID: ____________________

Prepared by / date: ____________________

Required review / approval under local procedure: ____________________

| Field | Your entry |
| --- | --- |
| Task | ____________________ |
| Assessment unit | ____________________ |
| Reference issues | ____________________ |
| Correct findings | ____________________ |
| False findings | ____________________ |
| Missed reference issues | ____________________ |
| Precision | ____________________ |
| Recall | ____________________ |
| Critical recall | ____________________ |
| Key limit | ____________________ |
| Workflow concern | ____________________ |
| Decision | ____________________ |

## Related guidance

- [Define AI intended use](https://csvtocsa.com/ai-assurance-academy/ai-in-csv-csa-chapter-3-intended-use-and-context-of-use)

- [Evaluate AI performance](https://csvtocsa.com/ai-assurance-academy/ai-in-csv-csa-chapter-11-performance-evaluation-and-metric-selection)

- [Review the AI protocol example](https://csvtocsa.com/case-studies/ai-assisted-protocol-review)

Canonical page: https://csvtocsa.com/worksheets/ai-evaluation-scorecard