Threat Modeling for LLM Applications: Where to Start

LLM applications are often introduced as chat interfaces. A user asks a question, the model answers, and the first security review focuses on the prompt.

That is understandable, but it is too narrow.

A production LLM application is not only a model and a prompt. It is an application system with users, identities, data flows, retrieval pipelines, model-provider boundaries, tools, agents, logs and audit requirements.

If the model can read internal data, call an API, trigger a workflow, summarize regulated content or influence a business decision, the threat model must include the entire system.

The threat model for an LLM application should not start with the model. It should start with the system boundary: who can influence the model, what the model can access, what it can trigger, and what evidence remains when something goes wrong.

Start with the system boundary

The first question is not “Which model do we use?”

The first question is: what is inside the system, and what is outside it?

A basic LLM application boundary may look like this:

User
  ↓
Application
  ↓
Prompt builder
  ↓
RAG / knowledge base
  ↓
Model provider
  ↓
Tools / APIs / agents
  ↓
Output
  ↓
Logs / audit evidence

For each step, the architect should identify who controls it, which data crosses it, and whether the next component is trusted, semi-trusted or untrusted.

Important boundary questions include:

Who can submit input to the system?
Can external content enter the prompt through RAG, web browsing, email or file upload?
Does the application send data to an external model provider?
Which documents can the retrieval layer access?
Can the model trigger actions through tools?
Are tool calls executed under the user’s authority or under a service account?
What is logged, where is it stored, and who can read it?
Which parts of the system are covered by monitoring and incident response?

Without this boundary, the threat model becomes a list of generic LLM risks. With the boundary, those risks become specific to the system.

Identify the trust boundaries

Threat modeling becomes useful when every trust boundary is connected to a concrete abuse case.

LLM applications introduce trust boundaries that many traditional application reviews do not make explicit:

Boundary	Typical risk
User input → prompt	Direct prompt injection or jailbreak attempts
External content → RAG	Indirect prompt injection through documents, emails, tickets, PDFs or web pages
RAG → model context	Unauthorized data exposure or excessive context disclosure
Model → tool call	Tool misuse, unsafe parameters or unintended workflow execution
Agent → API	Excessive privilege or unclear delegation
Model output → UI/workflow	XSS, command injection, business logic abuse or unsafe automation
Application → model provider	Data leakage, unclear retention or contractual boundary risk
Prompt/output logs → storage	Sensitive prompt retention or weak incident evidence

This is why LLM threat modeling cannot be only a prompt review.

The prompt is one control surface. The system is the risk surface.

Model data flows, not only prompts

Prompts matter because they decide what the model sees. But a prompt is only one point in a larger data flow.

An LLM application may process:

user input,
system instructions,
prompt templates,
retrieved documents,
vector-search results,
personal data,
confidential business information,
API responses,
tool outputs,
model responses,
traces and logs.

The data-flow review should answer practical questions:

Does the prompt contain personal data or confidential information?
Is data sent to an external provider?
Is the provider allowed to retain, train on or inspect the data?
Are retrieved documents minimized before entering the prompt?
Are secrets, tokens or credentials excluded from model context?
Are prompts and outputs logged?
Are logs classified and access-controlled?
How long are prompts, outputs and tool traces retained?
Can the organization reconstruct which data influenced a specific answer?

This is where AI security connects to privacy, compliance, AI governance and audit evidence.

A policy may say that sensitive data must be protected. A data-flow threat model shows whether the system actually protects it.

Threat categories specific to LLM applications

Traditional web and API threats still matter: broken access control, injection, XSS, insecure configuration, dependency risk and logging failures do not disappear because a model is involved.

But LLM applications add their own patterns.

Direct prompt injection

Direct prompt injection happens when the user tries to override instructions, bypass restrictions or force the model into unsafe behavior.

The control is not only a stronger system prompt. The system also needs policy tests, input constraints, output validation and limits on what the model can access or trigger.

Indirect prompt injection

Indirect prompt injection is often the more important enterprise risk.

The malicious instruction is not typed by the user. It is hidden in a document, web page, PDF, ticket, email or knowledge-base article that the system later retrieves or an agent reads.

This matters for RAG and agentic systems because the user may be legitimate while the content source is hostile.

Data leakage through RAG

A RAG system can expose information that the original document system would not show to that user.

This happens when retrieval does not enforce per-user permissions, when document chunks lose access-control metadata, or when too much context is inserted into the prompt.

Negative permission tests are essential: the review should prove that a user cannot retrieve content they are not allowed to see.

Tool misuse

When the model can call tools, the threat model changes.

A model that can only answer questions has one risk profile. A model that can send emails, create tickets, change records, execute queries or call APIs has a much larger one.

Tool calls need allowlists, parameter validation, safe defaults, rate limits and human approval for sensitive actions.

Excessive agent permissions

An agent with a broad static service account is a powerful technical user controlled through natural language.

A safer pattern is to use short-lived scoped tokens, OAuth2 on-behalf-of flow, just-in-time authorization and least privilege. The agent should not automatically have more authority than the user or task requires.

Output misuse

Model output is data, not a trusted command.

If output becomes HTML, SQL, code, a ticket instruction, a workflow step or a configuration change, it must be encoded, validated and constrained before use.

Model and provider boundary risk

The model provider boundary is both technical and contractual.

The threat model should identify what data leaves the organization, which provider receives it, what retention policy applies, whether training is disabled, and which logs are visible to administrators or support teams.

Logging and evidence failure

After an AI-related incident, the organization needs to answer basic questions:

Who used the system?
What input was submitted?
Which documents were retrieved?
What tool calls were made?
Which identity or token was used?
What did the model return?
Which controls accepted or blocked the action?

If those questions cannot be answered, the system has an evidence failure.

Map threats to controls

A useful threat model does not stop at naming threats. It maps threats to controls and evidence.

Threat	Example controls	Evidence to keep
Direct prompt injection	Refusal tests, input constraints, policy test cases	Test results, blocked prompts, evaluation records
Indirect prompt injection	Untrusted-content labeling, context separation, retrieval filtering	Source metadata, retrieval traces, abuse-case tests
RAG data leakage	Per-user retrieval permissions, document classification, negative access tests	Access-control test results, retrieved chunk logs
Tool misuse	Tool allowlists, schema validation, parameter constraints, human approval	Tool-call logs, approval records, validation failures
Excessive agent permissions	Short-lived scoped tokens, OAuth2 on-behalf-of, least privilege	Token scopes, delegation records, access reviews
Output injection	Output encoding, validation, no direct execution	Sanitization tests, rejected outputs, workflow logs
Provider data leakage	Data minimization, redaction, DPA/vendor review, retention controls	Provider settings, data-flow records, contract evidence
Audit failure	Prompt/output/tool-call logging, trace IDs, incident runbooks	Correlated traces, log retention proof, incident reports

This table is not a complete control catalog. It is a starting point for architecture discussion.

The important principle is that every serious LLM threat should have:

an owner,
a technical control,
a test,
a log or evidence artifact,
a decision about residual risk.

A minimal LLM threat modeling worksheet

A first LLM threat model does not need to be a large formal exercise. A lightweight worksheet can already expose the most important risks.

System

What does the LLM application do?
Who owns it?
Is it public, internal, confidential or regulated?
Is it experimental or production infrastructure?

Actors

Human users
Administrators
External content authors
AI assistants
Autonomous agents
Model providers
API services
Retrieval services
Attackers

Data

User input
System prompts
Prompt templates
Retrieved context
Internal documents
Personal data
Secrets and credentials
Model output
Tool-call results
Logs and traces

Capabilities

Answer questions
Search a knowledge base
Read internal documents
Summarize regulated content
Call APIs
Send messages
Create records
Change configuration
Trigger workflows

Abuse cases

What if the user lies?
What if a document contains hostile instructions?
What if retrieved data belongs to another user?
What if the model calls the wrong tool?
What if the agent has more privilege than the user?
What if the output is treated as trusted code or HTML?
What if prompts contain secrets?
What if logs are needed after an incident but do not exist?

Evidence

Which controls were tested?
Which user, agent and tool call were involved?
Which documents were retrieved?
Which decision was logged?
Which risk was accepted?
What can be shown to audit or incident response?

This worksheet is intentionally practical. It helps teams move from abstract AI risk to concrete system design.

Use OWASP LLM Top 10 and MITRE ATLAS, but do not stop there

Frameworks are useful.

The OWASP Top 10 for LLM Applications helps structure application-level AI risks such as prompt injection, sensitive information disclosure, insecure output handling, excessive agency and supply-chain concerns.

MITRE ATLAS helps describe adversarial behavior against AI systems and gives security teams a language for tactics and techniques.

NIST AI RMF helps connect risk management, governance, measurement and operational controls.

But frameworks do not replace architecture-specific threat modeling.

A checklist can tell you that prompt injection matters. It cannot tell you whether your specific RAG pipeline enforces document permissions. A framework can name excessive agency. It cannot tell you whether your agent is using a scoped delegated token or a broad service account.

The framework gives vocabulary. The threat model must describe the real system.

Start before production

The best time to threat-model an LLM application is before it becomes invisible infrastructure.

Once a chatbot becomes an internal knowledge layer, once an agent starts calling APIs, or once a RAG system becomes the default way to search documents, the risk is no longer experimental.

The review should happen before:

connecting sensitive documents,
enabling tool calls,
adding agentic workflows,
exposing the system to external users,
sending regulated data to a provider,
relying on the system for operational decisions.

Threat modeling is not about predicting every possible model behavior. It is about understanding where untrusted input, sensitive data, model reasoning, tool execution and organizational responsibility meet.

If an LLM application can access data, call tools or influence decisions, it deserves a threat model before it becomes production infrastructure.