AI Assurance Academy · Part 1
Classifying AI Systems and Drawing the Boundary
Chapter 2 of 20 · An AI assurance strategy fails early when the team treats “the model” as the system. This chapter provides a practical taxonomy and boundary method for predictive models, computer vision, generative AI, retrieval-augmented generation, embedded vendor features, and agents.
An AI assurance strategy fails early when the team treats “the model” as the system. This chapter provides a practical taxonomy and boundary method for predictive models, computer vision, generative AI, retrieval-augmented generation, embedded vendor features, and agents.
Published: September 4, 2026 | Version 1.0
Editorial owner: CSV to CSA Knowledge Hub | Review status: Open for practitioner peer review
Scope: Classification for lifecycle assurance; regulatory product classification requires separate qualified assessment.
LEARNING OBJECTIVES
- Classify AI by behavior, change mechanism, and role in the process
- Identify the components that belong inside the assurance boundary
- Distinguish AI used in a GxP process from AI that is itself a medical-device function
- Prevent uncontrolled “shadow AI” from entering regulated workflows
WHY CLASSIFICATION MATTERS
Classification is not a label for a validation plan. It is a way to expose different uncertainty and control needs. A static vision classifier installed on a packaging line, a hosted LLM drafting deviation summaries, and an agent that changes an eQMS record are all AI-enabled systems, but they require very different evidence.
CLASSIFY BY OUTPUT BEHAVIOR
Predictive or classification AI
Produces a score, class, ranking, forecast, or anomaly signal. Examples include visual defect classification, environmental-monitoring anomaly detection, and complaint triage.
Generative AI
Produces new text, code, image, audio, or structured content. It introduces risks such as fabricated content, unstable phrasing, unsafe completion, and unsupported reasoning.
Retrieval-augmented generation
Combines search or retrieval from an approved corpus with generation. Assurance must evaluate source governance, ingestion, indexing, retrieval relevance, prompt assembly, answer groundedness, and citation correctness.
Agentic AI
Plans and performs multi-step actions using tools. The critical questions concern authorization, state, tool permissions, sequencing, irreversible action, recovery, and evidence of each decision.
Embedded AI feature
Appears inside a SaaS platform, instrument, MES, LIMS, eQMS, or office tool. The vendor may provide little model transparency. Focus on intended use, enabled state, data flow, supplier evidence, customer configuration, and controls.
CLASSIFY BY CHANGE BEHAVIOR
Static model
The deployed parameters remain fixed until a controlled replacement. Inputs vary, but the model does not learn automatically in production.
Periodically retrained model
Training is repeated on a schedule or trigger and results in a new controlled model version. Retraining is a change that requires predefined data, evaluation, approval, deployment, and rollback controls.
Continuously learning model
Parameters adapt from production data during use. This creates major challenges for version identity, evaluation before use, traceability, and control. Do not assume it is acceptable for a critical GxP decision simply because a supplier offers it.
Hosted changing model
The provider may change weights, safety layers, routing, context limits, or behavior while the product name remains constant. The customer needs contractual notification, version controls where available, regression monitoring, and a response to silent change.
CLASSIFY BY PROCESS ROLE
Assistive
Summarizes, suggests, ranks, or drafts. A competent human makes the decision.
Decision-supporting
Its output materially shapes prioritization, investigation, or approval. Oversight must be proven effective, not merely present.
Decision-automating
The output directly determines a controlled action. This demands stronger evidence, deterministic constraints, and often independent confirmation.
Action-taking
The system changes records, configurations, workflows, equipment, or external systems. Agentic risk includes incorrect tool choice, excessive permissions, repeated action, and inability to stop.
THE SYSTEM BOUNDARY CANVAS
Map these elements:
- Upstream source systems and data owners
- Extraction, transformation, and feature pipelines
- Training, validation, and test datasets
- Model artifact or hosted endpoint
- Prompt templates and system instructions
- Retrieval corpus, embeddings, index, and ranking
- Application code and deterministic rules
- Interfaces, queues, and error handling
- Identity, permissions, and secrets
- Human review and approval activities
- Downstream records, decisions, and actions
- Logs, monitoring, incident, and change systems
- Supplier and subprocessor boundaries
For every boundary crossing, record the data, protocol, owner, failure handling, and evidence.
THREE REGULATORY QUESTIONS
1. Is the AI used as part of production or the quality management system?
FDA’s CSA guidance includes AI/ML and cloud technologies used in production or QMS software. Determine intended use at feature level.
2. Does the AI create or manage regulated electronic records or signatures?
Assess predicate-rule requirements and Part 11 scope, plus jurisdiction-specific requirements.
3. Is the AI itself a device software function or used to support a regulatory submission?
Separate product-software and submission-model frameworks may apply. Do not apply a non-product CSA rationale as a substitute for medical-device verification or clinical evidence.
WORKED EXAMPLE: THREE FEATURES IN ONE CLOUD PLATFORM
A cloud eQMS offers:
- A semantic search feature over approved SOPs
- A generative draft of a deviation summary
- An agent that opens a CAPA and assigns an owner
Search boundary: controlled documents, ingestion, index, query, ranked results, access filtering, link to authoritative source.
Drafting boundary: search boundary plus prompt, model endpoint, generated text, source display, user editing, acceptance, and record creation.
Agent boundary: drafting boundary plus planning, tool selection, CAPA-create permission, master data, assignment rules, confirmation, rollback, and audit record.
The same vendor platform contains three assurance units with escalating authority and risk.
SHADOW-AI CONTROL
Establish an intake route that is faster than bypassing governance. Detect browser extensions, embedded office assistants, public LLM use, API keys, and new SaaS features. Provide an approved experimentation environment using synthetic or de-identified data. Require production enablement through inventory, risk, privacy, security, and Quality gates.
BOUNDARY REVIEW QUESTIONS
- What exact component produces the output?
- Can the provider change it without notice?
- Which sources can influence the output?
- Which permissions can convert an error into an action?
- Where is the authoritative record?
- What remains if the AI is unavailable?
- Can every production version be identified and reconstructed?
PROFESSIONAL INTERPRETATION
The right boundary is wide enough to include the causes and controls of failure, but narrow enough to support feature-level decisions. A box labeled “AI platform” hides risk; a component-and-process map makes assurance possible.
PRIMARY SOURCES
FDA CSA final guidance, including AI/ML and cloud scope:
www.fda.gov/media/188844/download
FDA, Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations. Draft guidance; not for implementation:
www.fda.gov/medical-devices/digital-health-center-excellence/guidances-digital-health-content
EMA, Reflection paper on AI in the medicinal product lifecycle:
www.ema.europa.eu/en/use-artificial-intelligence-ai-medicinal-product-lifecycle-scientific-guideline
NIST AI RMF 1.0: