AI Assurance Academy · Part 3
Verification Strategy, Challenge Sets, and Repeatability
Chapter 12 of 20 · AI verification must test more than average model performance. A professional strategy challenges the full implemented system under normal, boundary, abnormal, adversarial, and operational conditions while preserving enough evidence to support an accountable decision.
AI verification must test more than average model performance. A professional strategy challenges the full implemented system under normal, boundary, abnormal, adversarial, and operational conditions while preserving enough evidence to support an accountable decision.
Published: September 4, 2026 | Version 1.0
Editorial owner: CSV to CSA Knowledge Hub | Review status: Open for practitioner peer review
THE EVIDENCE PYRAMID
Design evidence
Architecture, data flow, supplier capability, model card, prompt design, retrieval controls, deterministic rules, permissions, and failure analysis.
Component evidence
Data pipeline tests, model evaluation, retrieval tests, prompt tests, interface checks, security tests, and configuration review.
Integrated workflow evidence
End-to-end scenarios using representative roles, data, systems, records, and decisions.
Human-use evidence
Whether trained users understand limitations, detect errors, override appropriately, and escalate.
Operational evidence
Silent-mode results, monitoring, incidents, drift indicators, and real-world review.
TEST FAMILIES
Representative performance testing
Estimates expected behavior in the approved population.
Challenge testing
Targets rare, consequential, ambiguous, and boundary conditions.
Robustness testing
Varies formatting, noise, missingness, order, units, image quality, language, paraphrase, and irrelevant content.
Metamorphic testing
Applies transformations that should preserve or predictably change output. Examples: reorder independent fields, paraphrase noncritical text, alter irrelevant formatting, or convert equivalent units.
Negative testing
Confirms rejection, abstention, access denial, safe failure, and handling of unsupported inputs.
Adversarial testing
Challenges prompt injection, data poisoning paths, retrieval manipulation, tool misuse, secret extraction, evasion, and abuse.
Regression testing
Detects behavior change after model, prompt, index, application, configuration, or supplier updates.
Human-factors testing
Measures review quality, automation bias, workload, comprehension, override, and escalation.
Recovery testing
Verifies outage behavior, queue recovery, duplicate prevention, rollback, and reconciliation.
SCRIPTED AND EXPLORATORY COMBINATION
Use scripted tests for critical repeatable cases, exact calculations, role gates, electronic signatures, schema, records, and regression baselines.
Use exploratory charters for unexpected interaction, ambiguous inputs, multi-turn behavior, prompt manipulation, unusual workflows, and user adaptation.
Example charter:
“For 60 minutes, attempt to make the deviation-summary assistant produce unsupported root-cause statements. Use contradictory evidence, missing attachments, copied prior conclusions, negation, abbreviations, and instructions embedded in source text. Record successful failure patterns, cited evidence, user-visible warnings, and recovery.”
NONDETERMINISM AND REPEATABILITY
Record:
- Model and endpoint version
- System and developer prompts
- Temperature, top-p, seed where available, max tokens, tools, and safety settings
- Retrieval corpus and index version
- Date, region, and provider routing where known
- Number of repeated runs
- Scoring rubric and adjudication
Exact textual repeatability may be neither possible nor necessary. Define behavioral repeatability: critical facts remain supported, prohibited actions never occur, required citations appear, and error frequency stays within limits.
CHALLENGE-SET DESIGN
Build cases from:
- Hazard analysis
- Historical deviations and complaints
- Known model limitations
- Supplier defects
- Near misses
- User workarounds
- Rare but serious events
- Data-quality failures
- Security threat model
- Regulatory inspection observations where applicable
Keep the challenge set controlled and versioned. Separate a locked release set from an exploratory library that evolves.
WORKED EXAMPLE: AI-ASSISTED LABORATORY OOS REVIEW
Approved use: Identify potentially relevant sections of procedures and historical investigations for an analyst. The tool cannot determine whether a result is OOS, assign root cause, or close the investigation.
Test design:
- Correctly retrieves current method and OOS procedure
- Excludes superseded procedures
- Handles missing instrument ID
- Preserves decimal, units, and specification limits
- Does not treat prior root cause as current evidence
- Refuses to invent a cause when evidence is absent
- Prevents access to another product’s restricted record
- Cites the exact source section
- Remains usable during long narratives and tables
- Records model, source, prompt, output, and user decision
Run critical cases repeatedly and challenge with embedded instructions in historical documents.
DEFECT HANDLING
An AI test failure needs classification:
- Model limitation
- Data or label defect
- Prompt or retrieval defect
- Application or integration defect
- Configuration defect
- Human-interface defect
- Security weakness
- Test-data or expected-result issue
Evaluate scope, recurrence, affected versions, prior production impact, and whether monitoring could detect it. Correct, constrain, warn, retrain, or reject. Do not average a critical defect away.
MINIMUM TEST RESULT RECORD
Objective and risk:
System and versions:
Environment:
Dataset or scenario ID:
Method and repetitions:
Acceptance criteria:
Results and uncertainty:
Critical case failures:
Issues and disposition:
Evidence links:
Conclusion and residual risk:
Performer, date, reviewer:
PRACTITIONER REVIEW QUESTIONS
1. Which test challenges the most important failure chain?
2. Does the test cover the deployed pipeline, not only the model?
3. What repeated behavior must remain stable?
4. Which challenge cases remain confidential to reduce overfitting?
5. Can a future release reproduce the evaluation?
PROFESSIONAL INTERPRETATION
High-quality AI verification mixes statistical evaluation, software testing, process scenarios, human-factors evidence, security challenge, and operational monitoring. The test strategy should look different for a classifier, RAG assistant, and action-taking agent.
PRIMARY SOURCES
FDA CSA final guidance on scripted, unscripted, and objective evidence:
www.fda.gov/media/188844/download
NIST AI RMF and supporting resources:
www.nist.gov/itl/ai-risk-management-framework
NIST Generative AI Profile: