Requirements and Acceptance Criteria for Probabilistic Systems

“The AI shall be accurate” is not a verifiable requirement. AI requirements must define the context, population, error tolerance, uncertainty response, human control, records, and operating limits that together make the use acceptable.

Published: September 4, 2026 | Version 1.0

Editorial owner: CSV to CSA Knowledge Hub | Review status: Open for practitioner peer review

REQUIREMENT LAYERS

Process requirement

Defines the business or regulated outcome.

Example: Every potentially serious complaint shall receive qualified review within the established escalation time.

AI functional requirement

Defines the model-supported operation.

Example: The system shall assign a priority score and display the source statements that most influenced the recommendation.

Performance requirement

Defines measurable behavior in the approved context.

Example: On the locked independent test set, serious-event recall shall meet the predefined minimum, with the lower confidence bound exceeding the acceptance limit.

Safety and control requirement

Defines fallback, uncertainty, review, blocking, or escalation.

Example: Inputs below the minimum completeness rule or outside the supported language set shall not receive an automated recommendation and shall route to manual review.

Data requirement

Defines input source, format, completeness, quality, lineage, and representativeness.

Human-oversight requirement

Defines information, competence, authority, override, reason, and escalation.

Record requirement

Defines what must be retained to reconstruct the event.

Monitoring requirement

Defines current-performance indicators, limits, frequency, alert owner, and action.

CHANGE requirement

Defines which changes require assessment, testing, approval, or suspension.

CHARACTERISTICS OF A STRONG AI REQUIREMENT

Specific

Names the feature, actor, condition, and output.

Measurable

Has an observable acceptance method.

Contextual

Applies to a defined population, environment, and use.

Risk-linked

Controls or evaluates an identified failure.

Version-aware

Can be connected to the model, prompt, data, and application state.

Operational

Includes behavior when confidence is low, data is missing, service is unavailable, or an input is outside scope.

PERFORMANCE ACCEPTANCE FRAMEWORK

Define:

Metric

Select a metric aligned with the harmful error: recall, specificity, precision, false-negative rate, mean absolute error, calibration, retrieval recall, citation correctness, groundedness, or task completion.

Population

State the data, prevalence, sites, products, languages, equipment, and time period.

Threshold

Set a justified minimum and, where relevant, confidence interval.

Subgroups

Establish limits for meaningful groups rather than accepting an aggregate that hides failure.

Independence

Specify the locked evaluation data and separation from development.

Repeatability

For nondeterministic generation, define number of runs, parameter settings, and acceptable variability.

Failure handling

Define what happens if the limit is missed: investigation, restriction, retraining, rollback, or rejection.

GENAI ACCEPTANCE CRITERIA

Avoid one subjective score. Use a rubric that separates:

• Factual support from approved sources
• Citation correctness and completeness
• Harmful omission
• Unsupported assertion
• Instruction adherence
• Scope refusal
• Confidentiality and privacy behavior
• Robustness to adversarial or conflicting text
• User ability to detect and correct error
• Consistency across repeated runs

Define critical errors that cause automatic failure even when the average score passes.

EXAMPLE: RAG SOP ASSISTANT

Weak requirement:

“The chatbot shall answer SOP questions correctly.”

Better requirements:

• The system shall retrieve only documents effective for the authenticated user’s site and role.
• The response shall cite the controlled document ID, revision, and exact section used.
• When no approved source supports the answer, the system shall state that it cannot answer and shall not invent a procedure.
• The user shall be able to open the authoritative source from the response.
• Superseded documents shall be removed from eligible retrieval within the approved indexing interval.
• On the locked evaluation set, retrieval shall include the required source within the top-k results at the predefined rate.
• Critical procedural statements shall meet the defined citation-support threshold with zero unsupported critical instructions in the release test.
• Access-control tests shall confirm that unauthorized content is not retrieved or exposed.

The exact thresholds require process-specific justification.

NONFUNCTIONAL REQUIREMENTS

• Availability and latency within workflow tolerance
• Capacity and peak-load behavior
• Cybersecurity and abuse resistance
• Privacy and data residency
• Explainability or evidence display
• Accessibility and usability
• Auditability and record retention
• Version identification
• Backup, recovery, and fallback
• Portability and exit

REQUIREMENT-TO-EVIDENCE TRACE

For each important requirement, record:

Requirement → failure controlled → evidence method → dataset or scenario → acceptance result → issue → conclusion → monitoring metric.

Not every requirement needs a separate scripted test. Evidence can include design review, supplier evidence, code or configuration review, statistical evaluation, scenario testing, human-factors study, automated checks, monitoring, or procedural control.

ANTI-PATTERNS

• “AI shall be 95% accurate” without defining metric or population
• Acceptance based only on the supplier benchmark
• No criterion for abstention or out-of-scope input
• Average score hides a critical error type
• Human review is stated but not designed or tested
• Production monitoring is unrelated to release acceptance
• Thresholds selected after seeing test results
• Performance criteria are so strict that teams relabel failures instead of investigating them

ACCEPTANCE-CRITERIA WORKSHEET

Intended use:

Harmful error:

Primary metric:

Population and prevalence:

Subgroups:

Dataset independence:

Minimum threshold and rationale:

Confidence or uncertainty treatment:

Critical-error rule:

Repeatability design:

Human-control criterion:

Operational fallback:

Monitoring link:

Failure disposition:

PROFESSIONAL INTERPRETATION

The goal is not to make a probabilistic model behave like deterministic code. It is to define what level and pattern of performance is acceptable for the intended use, then surround the model with controls that make failure detectable and manageable.

PRIMARY SOURCES

FDA draft guidance on AI credibility for drug and biological product regulatory decision-making. Draft; not for implementation:

www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-artificial-intelligence-support-regulatory-decision-making-drug-and-biological

FDA and EMA, Guiding Principles of Good AI Practice in Drug Development:

www.fda.gov/about-fda/artificial-intelligence-drug-development/guiding-principles-good-ai-practice-drug-development

NIST AI RMF:

www.nist.gov/itl/ai-risk-management-framework