Model documentation is one part of the assessment
Review the supplied information about capabilities, limitations, version identification, changes, availability, security, data handling, and service terms. Then determine which questions remain about your application and use.
A general benchmark does not show whether your parser preserves a table, your retrieval respects permissions, or your reviewer detects a misleading answer.
Worked example
Worked example — fictional
A hosted model service can change the implementation behind an alias. The team’s evaluation record says only “latest model.” Later, behavior changes and the owner cannot identify what was used during evaluation.
A stronger approach records available model identifiers and deployment information, assesses supplier change controls, and defines what to do when a version cannot be pinned. That may include targeted evaluations, monitoring, restrictions, or deciding the service is unsuitable for the intended reliance.
Questions for the supplier
Which changes are announced? What version information is available? How are customer inputs and outputs handled? What retention and access controls apply? What happens during an outage? Can the organization export the records it needs? How are significant defects or incidents communicated?
What changes for this type of AI?
Related work: testing, records, change control, monitoring.
Exercise
The supplier reports excellent performance on a public benchmark. Can you skip the local protocol-review evaluation?
Show suggested answer
No. Use the benchmark as background evidence about the model, then address the local task, sources, configuration, users, and failure consequences.
Expected records
A supplier assessment covering available evidence, data/service terms, gaps, responsibilities and change notification. Adapt record names and detail to the applicable procedures; these are practitioner examples, not a universal mandatory document list.
Relevant regulations and guidance
Regulations set requirements within their scope; guidance offers recommendations or explains applicable expectations; standards depend on adoption or incorporation. Voluntary frameworks do not establish compliance. The lesson's controls, test designs and example records are practitioner suggestions unless explicitly identified otherwise.
Guidance · U.S. Food and Drug Administration · February 2026
Scope: United States · Medical-device manufacturers assuring software used in production or their quality management system.
Relevance to this lesson: Supplier evidence must be relevant to the in-scope configured production/QMS use.
Publication status: Final / published. Source checked 21 September 2026.
Publication identity, scope and status checked against the linked official source. This is not an independent audit of every retained guide statement.
Source details and applicabilityVoluntary framework · National Institute of Standards and Technology · NIST AI 100-1 · January 2023
Scope: United States · Cross-sector AI governance and risk management, including generative AI.
Relevance to this lesson: Supports attention to external dependencies and risk responsibilities.
Publication status: Final / published. Source checked 21 September 2026.
Publication identity, scope and status checked against the linked official source. This is not an independent audit of every retained guide statement.
Source details and applicability