Threat Modeling AI Features: The 12 Questions We Ask Before Shipping
Most teams ship LLM features with the threat model they'd use for a database query. They're not the same. Here are the 12 questions we work through before shipping AI features to production.
Key takeaways
- AI features have a different threat surface than typical software: prompt injection, data leakage, jailbreaks, cost-amplification attacks, output abuse, and training-data contamination.
- The 12 questions cover input safety, output safety, data handling, cost, observability, and human review.
- Skip these and your fastest path to incident is a creative user discovering they can do things you didn't intend.
The 12 questions
1. What's the worst output the model could produce here?
If it's "embarrassing tweet," low stakes. If it's "wrong medical advice," high stakes. Stake-level dictates everything else.
2. Can a user inject instructions?
Prompt injection is the most common LLM attack. Anything user-provided that goes into a prompt, including via RAG retrieval, must be sanitized or quarantined.
3. What's the data the model can see?
Map the data flow. What's in the system prompt, what's in retrieval, what's in tools. Any of it sensitive? Any of it the user shouldn't have access to?
4. What's the output destination?
Sent to the user? Stored in DB? Sent to a downstream system? Each amplifies risk differently.
5. Can output be used as input elsewhere?
If yes (e.g. the LLM writes to a DB that another LLM reads), you have a transitive injection risk.
6. What's the rate limit?
Without rate limits, you have a denial-of-wallet attack. Set per-user, per-IP, per-tenant.
7. What's the cost ceiling?
What happens at 100× expected volume? Caps, alerts, auto-shutoff.
8. Can the user trigger tool calls?
If your LLM can call tools (send email, write to DB, hit external APIs), every tool is a new attack surface.
9. What's logged?
Logs of prompts and outputs are often more sensitive than your normal logs. Treat them accordingly.
10. How do you detect abuse?
Anomaly detection on prompt patterns, output content classification, alerting on rate spikes.
11. What's the human-in-the-loop checkpoint?
For high-stakes outputs, a human must review before the output ships. Define the checkpoint.
12. What's the rollback?
If the model behaves badly, how fast can you swap to a safer version or disable the feature?
What we recommend
Run this list before any LLM feature ships. Update it quarterly as new attack patterns emerge.
Common pitfalls
The first: assuming "we use a frontier model, it's safe." Frontier models are less susceptible but not immune.
The second: forgetting that prompt injection can come from documents, not just user input.
The third: shipping without rate limits and discovering one user can run up a $4,000 monthly bill alone.
FAQs
Do we need a separate AI security review for every feature? Yes for high-stakes. A lightweight checklist for low-stakes.
Are there tools for this? Promptfoo, NeMo Guardrails, Lakera, Pillar Security all help.
What about red-teaming? Recommended pre-launch for any feature where misuse has real cost.