AI Assurance Academy · Part 2
Dataset Design, Independence, and Leakage Control
Chapter 8 of 20 · An AI evaluation can look rigorous while materially overstating performance if the data split leaks related records, the challenge set resembles development data, or prevalence differs from intended use. This chapter shows how to design defensible datasets.
An AI evaluation can look rigorous while materially overstating performance if the data split leaks related records, the challenge set resembles development data, or prevalence differs from intended use. This chapter shows how to design defensible datasets.
Published: September 4, 2026 | Version 1.0
Editorial owner: CSV to CSA Knowledge Hub | Review status: Open for practitioner peer review
THE PURPOSE OF EACH DATASET
Training set
Used to fit model parameters.
Validation or tuning set
Used to select architecture, features, thresholds, prompts, hyperparameters, or stopping criteria. Once repeatedly examined, it is part of development.
Independent test set
Held outside development and used to estimate performance for the approved context. Access and reuse should be controlled.
Challenge set
Designed to stress rare, boundary, ambiguous, adversarial, or high-consequence conditions. It is not expected to represent routine prevalence.
Operational monitoring set
Production inputs and reviewed outcomes used to detect change and estimate current performance.
Investigation set
Records assembled to examine a specific incident, subgroup, or suspected failure. Do not quietly fold it into acceptance evidence without controlling bias.
SPLIT AT THE RIGHT UNIT
Random row-level splitting is often wrong.
Group by the entity that creates dependence:
- Patient or subject
- Production lot or batch
- Physical unit
- Instrument or camera
- Site
- Document family
- Complaint thread
- Time period
- Supplier or product family
If multiple images of one vial appear in both training and test sets, the model may recognize the unit rather than generalize to future vials.
LEAKAGE TYPES
Record leakage
The same or near-duplicate record appears across splits.
Entity leakage
Different records from the same entity appear across splits and share information.
Temporal leakage
Information available only after the prediction time is used during development.
Target leakage
A feature directly or indirectly contains the outcome.
Preprocessing leakage
Scaling, imputation, feature selection, or embedding is fit using the full dataset before splitting.
Prompt leakage
Evaluation answers, test examples, or reference outputs enter a system prompt or few-shot examples.
Retrieval leakage
The answer key or evaluation document is present in the indexed corpus when the goal is to test independent reasoning.
Human leakage
Evaluators know the expected class and unintentionally influence labeling or adjudication.
Vendor contamination
A public benchmark may have been included in foundation-model training. Benchmark success may not demonstrate generalization.
DATASET REPRESENTATIVENESS
Map the approved operating envelope to evidence:
- Products and variants
- Sites, equipment, and instruments
- User groups and languages
- Normal and abnormal conditions
- Seasonal or temporal variation
- Data completeness and quality
- Rare critical conditions
- Prevalence in use
- Demographic or clinically relevant subgroups where applicable
Document where the test set does not represent the intended use and control those gaps.
PREVALENCE AND METRICS
Sensitivity and specificity may remain stable while positive predictive value changes with prevalence. A model evaluated on a balanced 50/50 dataset may overwhelm users with false positives when the real event occurs in 0.1% of records.
Report both:
- Intrinsic discrimination metrics appropriate to the problem
- Operational metrics under expected prevalence and workflow
For queueing systems, include workload, number needed to review, time-to-action, and missed-event impact.
SAMPLE-SIZE REASONING
Do not set sample size by precedent. Link it to the performance claim and uncertainty.
Ask:
- Which metric supports the critical decision?
- What minimum acceptable value is required?
- What confidence interval is needed?
- How many positive, negative, and subgroup cases are required?
- Are observations independent?
- Are rare critical cases addressed through an enriched challenge set?
Where statistical estimation is not feasible, be explicit about limitations and use stronger controls or restricted use.
WORKED EXAMPLE: COMPLAINT SERIOUSNESS CLASSIFIER
Data contains 100,000 historical complaints but only 250 serious cases. A random split yields strong overall accuracy.
Problems:
- Accuracy is dominated by nonserious records
- Duplicate follow-up narratives may cross splits
- Coding practices changed two years ago
- One product family supplies most serious cases
- Rare foreign-language cases are absent
Better design:
- Split by complaint case and time
- Reserve the most recent period for independent evaluation
- Report serious-class sensitivity with confidence interval
- Build a rare-event challenge set across products and languages
- Analyze false negatives clinically and procedurally
- Evaluate reviewer detection and escalation
- Monitor future prevalence and site-specific performance
GENAI EVALUATION SETS
For generative systems, define prompt families and scoring rubrics. Include unanswerable queries, conflicting sources, obsolete documents, missing context, adversarial instructions, tables, scanned text, negation, and requests outside scope.
Use blinded review where feasible. Control reference answers and adjudication. Measure factual support, completeness, harmful omission, citation correctness, and actionability—not style alone.
DATASET CONTROL RECORD
Dataset ID and version:
Purpose:
Source and time window:
Unit of independence:
Split method:
Leakage checks:
Population and prevalence:
Subgroups:
Label process:
Sample-size rationale:
Access controls:
Known limitations:
Change and reuse restrictions:
PROFESSIONAL INTERPRETATION
An independent test set is a controlled asset, not a folder named “test.” Its credibility depends on how it was separated, protected, represented, labeled, and interpreted. Challenge data complements representative data; it does not replace it.
PRIMARY SOURCES
FDA draft guidance on AI credibility for drug and biological product regulatory decision-making. Draft; not for implementation:
FDA and EMA, Guiding Principles of Good AI Practice in Drug Development:
EMA AI reflection paper:
www.ema.europa.eu/en/use-artificial-intelligence-ai-medicinal-product-lifecycle-scientific-guideline