Complex assurance scenario
Agentic deviation triage
Constraining an AI agent that can read records and initiate workflow actions.
Context
An AI agent reads new deviations, proposes a risk category, requests missing information, and creates draft investigation tasks.
Intended Use
Prepare nonbinding triage recommendations and draft tasks; humans assign classification, due date, owner, and final workflow actions.
Boundary
Model, instructions, tools, permissions, records, identity, memory, retrieval, action logs, confirmation steps, and kill switch.
Failure Chain
Prompt injection in an attachment alters the plan → agent assigns an unauthorized task → investigation is misdirected → critical escalation is delayed.
Existing Controls
- Least-privilege service identity
- allow-listed read and draft actions
- human confirmation
- immutable action log
- rate limit and kill switch
Evidence
- Tool-permission tests
- adversarial content challenge
- state and interruption tests
- action replay review
- human-oversight study
Evidence Gaps
The agent has not been evaluated against multilingual narratives.
Test Design
Use scenario-based red-team testing across prompt injection, conflicting instructions, missing data, tool failure, loops, stale memory, and confirmation bypass.
Decision
Limit production to English-language records and draft-only actions with per-action confirmation.
Residual Risk
Users may normalize acceptance of reasonable-looking drafts.
Monitoring
Confirmation time, edits, rejected actions, loops, escalations, language, and policy violations.
Reassessment Triggers
- New tool permission
- model or instruction change
- new language
- unexpected autonomous action
Inspection Questions
- What can the agent actually execute?
- How is instruction precedence enforced?
- How quickly can it be disabled?