How to Run AI Threat Modeling for a New Feature in an Hour

The launch is Friday, the security queue is three weeks long, and someone just asked what happens if a customer pastes "ignore your instructions" into the chat box.

This guide shows how to run AI threat modeling in one hour. You get a timed agenda, a fill-in YAML template, and a file that lives next to the code. It won't replace a full security review for a high-risk system, but it finds the expensive problems early and gives the reviewer something to read instead of a blank page.

The skeleton is the four questions from the Threat Modeling Manifesto: what are we working on, what can go wrong, what will we do about it, and did we do a good enough job.

Prep for AI threat modeling (before the hour)

Invite three or four people: you (you run the clock), the engineer who built the feature, and someone from security. No security team? Invite the engineer who reviews pull requests most suspiciously.

Bring a one-page feature description, the list of tools the model can call, and the list of data it can read.

1. Draw the feature and mark trust boundaries (0 to 10 min)

Sketch the flow: user, app, model, tools, data. Draw a line wherever control changes hands, such as browser to API, or your service to a model provider. Then list the assets, meaning whatever would hurt to leak, corrupt or spend.

# docs/threat-model.yaml
feature: refund-assistant
reviewed_on: 2026-10-19
participants: [pm, backend-eng, security]
rerun_when: tools, data sources or model change

assets:
  - id: A1
    name: Customer order and payment history
  - id: A2
    name: Refund budget

trust_boundaries:
  - id: B1
    from: browser
    to: chat API
    note: everything the user types is untrusted
  - id: B2
    from: chat API
    to: model provider
    note: data leaves our network
  - id: B3
    from: agent
    to: refund service
    note: the agent acts with our credentials

Commit the file to the repo from the start. The id keys let later sections point back at earlier ones.

2. List every untrusted input (10 to 20 min)

Ask one question about everything that reaches the model's context: who wrote this text? Chat messages, uploaded files, fetched web pages, retrieved documents, emails, product reviews and tool outputs all count if an outsider can influence them.

untrusted_inputs:
  - id: U1
    source: user chat message
  - id: U2
    source: product reviews retrieved into context
  - id: U3
    source: uploaded receipt PDF

3. Review agent permissions per tool (20 to 30 min)

A successful injection can do exactly what the agent's tools allow, so permissions set the ceiling on damage. For each tool, record read or write, whose credentials it uses, its scope, and whether a human approves.

tools:
  - name: get_order
    access: read
    scope: current customer's orders only
    approval: none
  - name: issue_refund
    access: write
    scope: up to 50 USD per order
    approval: human above the cap
  - name: send_email
    access: write
    scope: address on file only
    approval: none

Circle every write tool that shares a context with an untrusted input. Those pairs feed the next step.

4. Write abuse cases (30 to 45 min)

Write one sentence per case: an attacker who controls this input makes the agent use this tool for this impact. Score impact and likelihood as low, medium or high, and stop around eight cases.

Seed the room with the OWASP Top 10 for LLM Applications, especially LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM06 Excessive Agency and LLM10 Unbounded Consumption.

abuse_cases:
  - id: X1
    story: Hidden text in a receipt PDF tells the agent to refund an order the customer does not own
    inputs: [U3]
    tools: [issue_refund]
    impact: high
    likelihood: medium
    mitigation: M1
  - id: X2
    story: A poisoned review tells the agent to email order history to an outside address
    inputs: [U2]
    tools: [get_order, send_email]
    impact: high
    likelihood: medium
    mitigation: M2
  - id: X3
    story: A user runs endless long chats to burn token spend
    inputs: [U1]
    tools: []
    impact: low
    likelihood: high

5. Assign mitigations with owners (45 to 55 min)

For every high-impact case, prefer a control in code or permissions over a line in the system prompt. A prompt instruction is a request, and an attacker's text can outvote it.

mitigations:
  - id: M1
    what: Refund service checks order ownership and the 50 USD cap, and the model cannot override either
    owner: backend-eng
    due: 2026-10-28
    status: todo
  - id: M2
    what: send_email recipient is fixed to the address on file and is not a model argument
    owner: backend-eng
    due: 2026-10-28
    status: todo

accepted_risks:
  - abuse_case: X3
    reason: Low impact, per-user rate limit comes next quarter
    accepted_by: pm

Every mitigation needs an owner and a date. Anything without both is a wish.

6. Decide and file it (55 to 60 min)

Agree the launch rule out loud: every high-impact abuse case has a finished mitigation or a risk accepted by a named person. Then commit the file and require security to review changes to it.

git add docs/threat-model.yaml
git commit -m "Add threat model for refund assistant"

# .github/CODEOWNERS
docs/threat-model.yaml @your-org/security-team

CODEOWNERS only blocks merges if your branch protection requires code owner review. To keep the file honest, add a check to the release job (pip install pyyaml):

# scripts/check_threat_model.py
import sys
import yaml

with open('docs/threat-model.yaml') as f:
    tm = yaml.safe_load(f)

mitigations = {m['id']: m for m in tm.get('mitigations', [])}
accepted = {a['abuse_case'] for a in tm.get('accepted_risks', [])}
problems = []

for case in tm.get('abuse_cases', []):
    if case['impact'] != 'high' or case['id'] in accepted:
        continue
    m = mitigations.get(case.get('mitigation'))
    if m is None:
        problems.append(case['id'] + ' has no mitigation')
    elif m['status'] != 'done':
        status = m['status']
        problems.append(case['id'] + ': ' + m['id'] + ' is ' + status)

for p in problems:
    print(p)
sys.exit(1 if problems else 0)

The script fails while any high-impact case lacks a finished mitigation or an accepted risk. Run it on release, not on every pull request, since mitigations are due before launch.

Gotchas

• Assume the injection works. Don't debate whether the model can be tricked. Ask what it can reach when it is.
• Time-box ruthlessly. Park side debates in a list and move on, or step 4 gets eaten.
• Re-run on change. A new tool, data source or model swap invalidates the file. The rerun_when key is there to remind you.
• Name the accepted risks. A risk with a name next to it is a decision; without one it's an oversight.

A threat model in the repo turns "is this safe to ship?" from an argument into a checklist, which is what you need when your AI feature has real tools and real customers on the other end.