AI Threat Modeling

Review scope and evidence labels

This page defines AI Threat Modeling as a security methodology for identifying realistic attack paths, abuse cases and control gaps in AI-enabled systems.

Evidence labels used on this page:

Official guidance — based on public official guidance such as NIST material.
Community security guidance — based on public security community material such as OWASP or MITRE ATLAS.
Vendor documentation — based on public vendor documentation used to explain a method or vocabulary.
Author analysis — interpretation for AI security, architecture, governance and operational risk on this site.
Implementation evidence required — records that must exist before a mitigation can be treated as operating.

This entry is a methodology profile, not a claim of offensive certification. It is intended to support architecture reviews, AI governance, risk registers and secure engineering decisions.

Methodology snapshot

Field	Value
Category	AI security and threat modeling
Research type	Methodology
Core question	How can an AI system be misused, manipulated, bypassed or turned against its owner?
Main scope	LLM applications, RAG systems, AI agents, model-enabled workflows, AI-assisted development and vendor AI integrations
Useful inputs	System context, data flows, trust boundaries, identities, tools, APIs, model behavior, logs and business impact
Useful outputs	Abuse cases, threat list, mitigations, residual-risk decision, monitoring requirements and audit evidence
Reference families	NIST AI RMF, OWASP LLM Top 10, MITRE ATLAS, NIST adversarial ML, STRIDE
Review status	Public guidance reviewed; local threat models require system-specific evidence

What AI Threat Modeling means

AI Threat Modeling is the practice of asking how an AI-enabled system can fail under adversarial, accidental or misuse conditions before the system is trusted in production. Evidence: Official guidance; Community security guidance; Author analysis.

It extends traditional threat modeling with AI-specific concerns: prompt injection, data poisoning, model extraction, unsafe tool use, excessive context exposure, RAG manipulation, output misuse and unclear accountability. Evidence: OWASP LLM guidance; MITRE ATLAS; NIST adversarial ML; Author analysis.

For this site, AI Threat Modeling is the practical bridge between AI Security Architecture and daily engineering or governance work. The architecture shows the control model; the threat model tests where it can break. Evidence: Author analysis.

What it is not

AI Threat Modeling is not:

a one-time brainstorming meeting,
a replacement for security testing,
a guarantee that every attack path is known,
only prompt-injection testing,
only model evaluation,
only compliance documentation,
a substitute for secure design, monitoring or incident response.

A useful threat model must be updated when the system changes: model provider, prompt, RAG source, tool permission, API exposure, data classification, user group or business process. Evidence: Author analysis; Implementation evidence required.

AI-specific trust boundaries

AI systems often introduce trust boundaries that are easy to miss.

User or external input
        ↓
Application logic and policy checks
        ↓
Prompt / context / memory
        ↓
Model or AI service
        ↓
Tool, plugin, API or agent action
        ↓
Data sources, RAG index, logs and evidence store

Threat modeling should ask what can cross each boundary, who controls it, how it is validated, and what evidence remains if something goes wrong. Evidence: Author analysis.

Common AI abuse cases

Abuse case	What to ask	Typical mitigation evidence
Prompt injection	Can input override intended instructions or policy?	Prompt review, input handling, output checks, test cases
RAG poisoning	Can retrieved content inject instructions or false context?	Source governance, indexing controls, provenance, retrieval tests
Data leakage	Can prompts, responses, traces or logs reveal sensitive data?	Data-flow map, retention rules, access controls, log review
Excessive tool access	Can an agent call actions beyond its intended purpose?	Least-privilege permissions, approval gates, execution logs
Insecure output handling	Can generated output become code, command, HTML or decision input unsafely?	Output validation, sandboxing, human review, AppSec tests
Model or provider drift	Can behavior change without review?	Version records, change review, evaluation history
Supply-chain exposure	Can generated code, libraries or models introduce hidden risk?	Dependency review, SBOM, provenance, build evidence
Accountability failure	Can the organisation prove who approved, used and monitored the system?	Risk decision, owner, monitoring plan, audit evidence

Evidence: OWASP LLM guidance; MITRE ATLAS; NIST AI RMF; Author analysis.

Using STRIDE carefully

Traditional STRIDE categories can still help structure analysis:

Spoofing — fake identities, service accounts, model endpoints or agent identities.
Tampering — changed prompts, poisoned documents, modified context or altered model outputs.
Repudiation — missing logs or unclear records of who triggered an AI action.
Information disclosure — sensitive data revealed through prompts, responses, traces or embeddings.
Denial of service — costly loops, tool abuse, token exhaustion or workflow disruption.
Elevation of privilege — AI agent or tool access exceeding the intended permission boundary.

However, STRIDE should not be used mechanically. AI systems also need abuse-case thinking around model behavior, retrieval, human overtrust, vendor change and governance evidence. Evidence: Vendor documentation; Author analysis.

Practical workflow

A lightweight AI threat-modeling workflow can use these steps:

Define the AI use case and owner.
Draw the data flow and trust boundaries.
Identify users, service identities, tools, APIs and data stores.
List AI-specific abuse cases.
Map relevant OWASP LLM and MITRE ATLAS patterns.
Decide mitigations and residual risk.
Define logs, alerts and audit evidence.
Re-review after changes to prompts, models, RAG sources, tools or permissions.

Evidence: Official guidance; Community security guidance; Author analysis.

Relationship to existing Research entries

AI Threat Modeling connects to:

AI Security Architecture — threat modeling tests the architecture assumptions.
AI Governance — threat decisions need ownership and risk acceptance.
Secure AI-Assisted Development — generated or agentic software needs threat review before release.
Zero Trust — identity, least privilege and telemetry reduce blast radius.
Audit Evidence — threat models and mitigation decisions must be retained.
SBOM — supply-chain evidence supports analysis of generated software, libraries and components.

Governance relevance

AI Threat Modeling helps translate broad AI-risk language into concrete controls and accountable decisions. It can feed an AI risk register, security architecture review, model approval process, supplier review, incident-response plan and executive risk summary. Evidence: Author analysis.

For regulated or high-impact contexts, the important output is not only a list of threats. The organisation must show what was accepted, what was mitigated, what evidence exists and when the review will happen again. Evidence: Official guidance; Author analysis; Implementation evidence required.

Current limitations and follow-up research

This entry is a public methodology profile. Real AI threat models require system-specific details: data, users, architecture, deployment model, model provider, prompts, RAG sources, tool permissions, logging and business impact.

Follow-up Research nodes should cover:

OWASP Top 10 for LLM Applications,
MITRE ATLAS,
Prompt Injection,
RAG Security,
AI Red Teaming,
IAM for AI systems,
Model Supply Chain Security.