AI Assurance Academy · Part 3
Performance Evaluation and Metric Selection
Chapter 11 of 20 · AI assurance can fail through mathematically correct but operationally irrelevant metrics. This chapter connects the harmful process error to metrics, thresholds, uncertainty, prevalence, subgroup performance, and release decisions.
AI assurance can fail through mathematically correct but operationally irrelevant metrics. This chapter connects the harmful process error to metrics, thresholds, uncertainty, prevalence, subgroup performance, and release decisions.
Published: September 4, 2026 | Version 1.0
Editorial owner: CSV to CSA Knowledge Hub | Review status: Open for practitioner peer review
START WITH THE DECISION ERROR
Ask which error matters:
- Missing a true defect or safety signal
- Incorrectly rejecting acceptable material
- Producing an inaccurate numeric prediction
- Ranking a critical record too low
- Retrieving the wrong controlled source
- Generating an unsupported instruction
- Completing the wrong agent action
Select metrics only after defining this failure.
CLASSIFICATION METRICS
Sensitivity or recall
Among true positive cases, the proportion correctly identified. Important when misses are harmful.
Specificity
Among true negative cases, the proportion correctly rejected.
Precision or positive predictive value
Among positive model outputs, the proportion actually positive. Important for workload and trust.
Negative predictive value
Among negative outputs, the proportion actually negative.
False-negative and false-positive rates
Express harmful errors directly and can be easier to connect to process impact.
F1 score
Balances precision and recall but can hide which error is more important. Do not use it as the sole metric when consequences are asymmetric.
Area under the ROC curve
Measures ranking across thresholds. It does not prove that the selected operating threshold is acceptable.
Calibration
Compares predicted confidence with observed frequency. A 90% confidence score should mean something demonstrable in the relevant population.
REGRESSION METRICS
Mean absolute error is interpretable but can hide rare large misses. Root mean squared error weights larger errors. Percent error can become unstable near zero. Always inspect the error distribution and clinically or technically meaningful boundaries.
For process predictions, evaluate error by operating range, equipment, product, site, and time—not only overall average.
RANKING AND RETRIEVAL METRICS
Recall at k
Whether the required item appears in the first k results.
Precision at k
How many of the first k results are relevant.
Mean reciprocal rank
How high the first relevant result appears.
Normalized discounted cumulative gain
Accounts for graded relevance and position.
Retrieval completeness
Whether every required approved source was eligible and indexed.
For RAG, measure retrieval separately from answer generation. A good answer on a poor retrieval set may be an unsupported guess.
GENERATION METRICS
Automated similarity metrics rarely establish factual validity. Use a controlled rubric:
- Claim supported by cited source
- Required content present
- No critical contradiction
- Citation points to the correct revision and section
- Uncertainty or refusal is appropriate
- No confidential or unauthorized content
- Instruction follows approved process
- Output is useful to the trained user
Define critical errors that fail a case regardless of average score.
PREVALENCE AND WORKLOAD
A model with high sensitivity and specificity can produce mostly false alarms when the event is rare. Calculate expected alerts and true cases under real prevalence.
Example:
At 0.2% prevalence across 10,000 records, there are 20 true events. A 95% sensitivity detects about 19. A 98% specificity creates about 200 false positives. Reviewers see roughly 219 alerts, of which only 19 are true. If the workflow cannot absorb that load, practical safety may decline through alert fatigue.
Evaluate the complete operating point, not a metric in isolation.
CONFIDENCE INTERVALS
Point estimates hide uncertainty. Report intervals for the metrics that support release, especially for rare classes and subgroups. If the lower confidence bound is below the minimum acceptable performance, the evidence may be insufficient even when the point estimate passes.
Avoid claiming precision that the sample size cannot support.
SUBGROUP PERFORMANCE
Choose groups connected to use and potential failure:
- Product and variant
- Site and instrument
- Language
- Image or signal quality
- Process stage
- User group
- Demographic or clinical factor when relevant
- Data source and time period
Predefine minimum subgroup evidence or control the limitation through restricted use, additional review, or monitoring.
THRESHOLD SELECTION
Select thresholds using training or tuning data and freeze them before independent testing. Document the tradeoff, expected workload, fallback, and change control.
Do not move the threshold after seeing the locked test result unless the evaluation is reclassified as development and a new independent assessment is performed.
WORKED EXAMPLE: ANOMALY SCORE FOR BIOREACTOR REVIEW
The model ranks process runs for engineer review; it does not control the process or disposition batches.
Evaluation plan:
- Historical confirmed excursions plus normal runs
- Time-based split to simulate prospective use
- Run-level independence
- Recall of significant excursions
- Alerts per 100 runs
- Lead time before conventional detection
- Performance by product, scale, sensor version, and site
- Review of false negatives by process SME
- Prospective silent-mode comparison before operational release
Acceptance should include both event detection and manageable review volume.
METRIC REVIEW TABLE
Decision:
Harmful error:
Primary metric:
Secondary metric:
Operating threshold:
Prevalence:
Expected workload:
Confidence interval:
Subgroups:
Critical-error rule:
Human performance:
Monitoring equivalent:
Action if missed:
PRACTITIONER REVIEW QUESTIONS
1. Does the metric represent the failure that matters?
2. Is the operating threshold fixed before independent evaluation?
3. What does performance mean under actual prevalence?
4. Which subgroup has the weakest evidence?
5. Can monitoring calculate a comparable production metric?
PROFESSIONAL INTERPRETATION
Metric selection is a quality decision. The strongest evaluation combines statistical performance, case-level error analysis, workflow impact, human performance, and explicit uncertainty.
PRIMARY SOURCES
FDA draft guidance on AI credibility for drug and biological product regulatory decision-making. Draft; not for implementation:
FDA and EMA, Guiding Principles of Good AI Practice in Drug Development:
EMA AI reflection paper:
www.ema.europa.eu/en/use-artificial-intelligence-ai-medicinal-product-lifecycle-scientific-guideline