More than the model can change
Assess changes to prompts, retrieval, documents, parsing, embedding models, permissions, tools, UI, review instructions, and downstream use. Identify which claims the change could affect.
Three examples
A prompt adds a requested output column. It may alter other response behavior, so examine the affected output and relevant regression cases.
A source corpus adds a new site’s procedures. Retrieval may now confuse similar documents. Assess context and permission filtering, not only the new files’ availability.
A model upgrade improves ordinary summaries but changes how the assistant handles missing information. Evaluate the unsupported-case behavior important to the control envelope.
Completed change rationale — fictional
Change CH-17 introduces a new table parser. Evidence will compare extraction on supported table types and rerun affected issue-detection cases. The intended use and permissions remain the same. Release depends on resolving material extraction differences and confirming the review workflow remains adequate.
Do not promise identical output
Where outputs vary, define acceptable substantive behavior and relevant stability rather than expecting every sentence to match. Preserve enough configuration information to understand what was evaluated and what is deployed.
What changes for this type of AI?
Related work: testing, supplier assessment, records, monitoring.
Exercise
The model version is fixed, but approved procedures are replaced nightly. Is the application unchanged?
Show suggested answer
No. Its information basis changes. Establish source-update controls, failed-refresh detection, and assessment appropriate to the effect on the intended use.
Expected records
A change assessment linking the changed configuration to affected requirements, risks, regression testing and release disposition. Adapt record names and detail to the applicable procedures; these are practitioner examples, not a universal mandatory document list.
Relevant regulations and guidance
Regulations set requirements within their scope; guidance offers recommendations or explains applicable expectations; standards depend on adoption or incorporation. Voluntary frameworks do not establish compliance. The lesson's controls, test designs and example records are practitioner suggestions unless explicitly identified otherwise.
Guidance · U.S. Food and Drug Administration · February 2026
Scope: United States · Medical-device manufacturers assuring software used in production or their quality management system.
Relevance to this lesson: In scope, assess changes that affect the supported intended use and risk.
Publication status: Final / published. Source checked 21 September 2026.
Publication identity, scope and status checked against the linked official source. This is not an independent audit of every retained guide statement.
Source details and applicabilityGMP guideline · European Commission · January 2011 revision
Scope: European Union · Computerised systems supporting activities in the applicable EU pharmaceutical GMP context.
Relevance to this lesson: Consider applicable GMP computerised-system lifecycle expectations without inferring that unchanged model code means unchanged use.
Publication status: Final / published. Source checked 21 September 2026.
The Commission index confirms the January 2011 operative revision. The linked PDF could not be retrieved in this check; retained detailed section mappings need review against an authorized copy.
Source details and applicability