1. Why Threat-Model Before You Deploy
Threat modeling is a design-time activity, not an incident-response one. The whole point is to find the weaknesses while they are still cheap to fix — in an architecture diagram, not in production forensics. For an on-prem inference stack the case is especially strong, because on-prem shifts where the risk lives without removing it.
The reflexive argument for on-prem is "our data never leaves the building, so we're secure." That confuses one control (zero data egress) for a security posture. On-prem genuinely retires several cloud risks: no data crosses a third-party boundary, no external processor holds your prompts, no shared-tenancy inference endpoint can leak across customers. Those are real wins and they are why regulated and sovereignty-sensitive workloads come home.
But the surface you inherit in exchange is substantial and, crucially, it is now yours to defend:
- Model weights on disk. A multi-million-parameter artifact — often the most valuable IP in the system, sometimes a fine-tune trained on proprietary data — now lives on a filesystem you administer. Theft of that file is theft of the model.
- GPU nodes. Expensive, contended compute that is a prime target for both denial-of-service (starve legitimate users) and resource hijacking.
- The gateway. The single ingress that authenticates, authorizes, rate-limits, and routes every request. Its compromise is a compromise of everything behind it.
- The vector store and document corpus. The retrieval backend that decides which content reaches the model — and therefore an injection and poisoning target.
- Tool runners. The processes that turn model output into real actions (HTTP calls, database queries, code execution). Where "the model was fooled" becomes "damage occurred."
Threat modeling gives you a repeatable method to walk this surface systematically instead of relying on intuition. The classic framing — the four questions popularized by the threat-modeling community — is: What are we building? What can go wrong? What are we going to do about it? Did we do a good enough job? The sections below answer the first three for a concrete inference stack, using STRIDE as the "what can go wrong" taxonomy and MITRE ATLAS plus the OWASP LLM Top 10 to cover the AI-specific threats general-purpose frameworks miss.
Framework attribution, up front
STRIDE is a threat classification model developed at Microsoft (Loren Kohnfelder and Praerit Garg). MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is MITRE's ATT&CK-style knowledge base of real-world adversary tactics and techniques against ML systems. The OWASP Top 10 for LLM Applications is OWASP's community-maintained list of the most critical LLM risks (referenced here as LLM01–LLM10). This article uses all three together; none of them alone is sufficient.
2. Decompose the Stack into Trust Zones
You cannot threat-model a black box. The first real work is decomposition: draw the components, group them into trust zones, and mark the trust boundaries — the lines a request or a piece of data crosses when it moves from one level of trust to another. Every boundary crossing is where authentication, authorization, and validation must happen, and where most interesting threats live.
The zones below align with the layered sovereign-AI reference architecture. Read them outside-in, following a single request from an untrusted client down to the GPU and back:
The zones
- Zone 0 — Perimeter / clients. Browsers, internal apps, service accounts. Fully untrusted until authenticated. Everything here is attacker-influenceable.
- Zone 1 — Ingress & security. The API gateway, the identity provider / SSO, the rate limiter, and the input/output guardrails. This is the trust boundary between "the outside" and "the platform." Nothing should reach an inference node without passing through here.
- Zone 2 — Orchestration. The prompt orchestrator and the model router — the logic that assembles context, decides which model or adapter serves a request, and mediates tool calls. It holds real privilege and it consumes untrusted content, which makes it a confused-deputy risk.
- Zone 3 — Data & retrieval. The vector database, the document store, and the ingestion pipeline that populates them. This is the primary indirect-injection and data-poisoning surface.
- Zone 4 — Inference. The model-serving runtime (e.g. vLLM, TensorRT-LLM, or an equivalent server), the adapters/LoRA layers, and the KV cache. The model weights live here; so does a surprising amount of transient sensitive data in the cache.
- Zone 5 — Secrets. The KMS/HSM and secret store that hold API keys, model-decryption keys, and tool credentials. Highest trust; smallest surface; strictest access.
- Zone 6 — Observability. Logging, tracing, metrics, and the SIEM. It is both a defensive asset (detection) and, because it sees prompts and responses, a sensitive data store in its own right.
The data flows that matter
The high-value flows to annotate on your diagram — because each crosses a trust boundary — are:
- Client → gateway (crosses the untrusted → platform boundary; authN/authZ enforced here).
- Orchestrator → vector store (a query carrying, implicitly, the identity and permissions of the requesting user — or failing to).
- Retrieval → orchestrator → model context (untrusted document text entering the prompt: the indirect-injection channel).
- Model output → tool runner → external/internal systems (the point where text becomes action; egress boundary).
- Any component → secrets store (credential retrieval; must be least-privilege and audited).
- Every component → observability (telemetry egress; may carry sensitive prompt/response content).
The boundary people forget: the KV cache
The KV cache in Zone 4 holds the attention state for in-flight (and, with prefix caching, recently seen) requests — which means fragments of other users' prompts and retrieved data can persist in GPU memory. On a shared multi-tenant serving node without cache isolation, that is a cross-request information-disclosure boundary hiding inside what looks like a pure performance optimization. Treat cache-sharing as a security decision, not just a throughput knob.
3. Apply STRIDE Per Component
STRIDE enumerates six threat categories, each the inverse of a security property you want:
- S — Spoofing (violates authenticity): pretending to be someone or something you are not.
- T — Tampering (violates integrity): unauthorized modification of data or code.
- R — Repudiation (violates non-repudiation): denying an action with no evidence to prove otherwise.
- I — Information disclosure (violates confidentiality): exposing data to those not entitled to it.
- D — Denial of service (violates availability): degrading or denying legitimate use.
- E — Elevation of privilege (violates authorization): gaining capabilities you should not have.
The table below walks each zone/component through its most salient STRIDE threats and maps them to the corresponding OWASP LLM risk and a MITRE ATLAS technique conceptually (ATLAS technique names evolve across releases — treat these as the right tactic to look up, not verbatim IDs).
| Component (zone) | Key STRIDE threats | OWASP LLM risk | MITRE ATLAS (conceptual) |
|---|---|---|---|
| API gateway / IdP (Z1) | S forged tokens · E authz bypass · D flood | LLM07 Insecure Plugin/System Design | Initial Access via valid accounts / exposed API |
| Rate limiter / guardrails (Z1) | T filter evasion · D resource exhaustion | LLM01 Prompt Injection · LLM04 Model DoS | ML Attack Staging · evade ML model |
| Prompt orchestrator (Z2) | E confused deputy · T context injection | LLM01 Prompt Injection · LLM08 Excessive Agency | Execution via LLM prompt injection |
| Model router (Z2) | S mis-routing · I cross-tenant leak | LLM06 Sensitive Info Disclosure | ML Model Access / inference misuse |
| Vector DB / doc store (Z3) | T index poisoning · I unauthorized retrieval | LLM03 Data/Model Poisoning · LLM06 | Poison Training/RAG Data |
| Ingestion pipeline (Z3) | T poisoned source · S spoofed provenance | LLM03 Data Poisoning · LLM05 Supply Chain | ML Supply Chain Compromise |
| Model serving — weights (Z4) | I weight theft · T tampered artifact · D GPU exhaustion | LLM10 Model Theft · LLM05 Supply Chain · LLM04 DoS | Exfiltrate ML Model · Backdoor ML Model |
| Adapters / KV cache (Z4) | T malicious adapter · I cross-request cache leak | LLM05 Supply Chain · LLM06 | ML Supply Chain · inference-time data leak |
| Tool runners (Z2/Z4) | E over-privilege · I exfil via tool | LLM08 Excessive Agency · LLM02 Output Handling | Exfiltration via inference API / tools |
| Secrets — KMS/HSM (Z5) | I key disclosure · E credential reuse | LLM06 Sensitive Info Disclosure | Credential Access / secret theft |
| Observability / logs (Z6) | R log gaps · I prompts logged in clear · T log tampering | LLM06 Sensitive Info Disclosure | Defense Evasion / indicator removal |
The discipline of the exercise is what matters: force every component through all six letters. Even when a category yields "not applicable," writing that down is a decision on record rather than a blind spot. Repudiation, in particular, is the one teams skip — and it is exactly the gap that turns an incident into an un-investigable mystery.
4. Where STRIDE Ends and ATLAS Begins
STRIDE is necessary but not sufficient for an AI system. It was designed for conventional software, and it cleanly captures the "classic" threats — a spoofed token, a tampered database, a DoS flood. What it does not natively express are the threats that arise because the system contains a learned, probabilistic model that consumes untrusted data as if it were instructions. That is where MITRE ATLAS and the OWASP LLM Top 10 earn their place in the model.
You can shoehorn AI threats into STRIDE letters — prompt injection is arguably a form of tampering-plus-elevation — but doing so loses the specificity a defender needs. The AI-native threats worth calling out explicitly:
- Prompt injection (LLM01). Untrusted content in the context window overrides intended behavior. STRIDE has no category for "data becomes instructions because the processor cannot tell them apart." ATLAS captures it as an execution technique via prompt injection; it is the defining LLM threat and deserves its own coverage — see the companion prompt-injection defense-in-depth playbook.
- Model extraction / theft (LLM10). On-prem, this splits in two. The blunt version is filesystem exfiltration of the weights artifact — a Tampering/Information-disclosure problem STRIDE does see. The subtle version is functional extraction: an adversary with query access reconstructs a near-equivalent model by systematically probing it. STRIDE has no vocabulary for "steal the model by talking to it"; ATLAS does.
- Data / model poisoning (LLM03). Corrupting the corpus the model retrieves from — or, for fine-tunes, the training data — so behavior is subverted later. This is Tampering, but of a statistical, delayed-action kind that a code-integrity mindset under-weights. ATLAS's poisoning techniques frame it correctly as an ML-lifecycle attack.
- Model denial of service (LLM04). Beyond ordinary request floods, LLMs have AI-specific exhaustion vectors: crafted inputs that maximize output length, force worst-case attention cost, or thrash the KV cache. STRIDE's "D" covers the intent; the LLM Top 10 names the mechanism.
- Supply-chain compromise (LLM05). The provenance of everything you load: base weights from a hub, third-party LoRA adapters, tokenizers, the serving framework and its dependencies, and quantization tooling. A backdoored adapter or a tampered model file is a supply-chain attack that lands inside the highest-trust zone. ATLAS's ML supply-chain techniques are the right lens.
How to use the three together
Run STRIDE per component to get systematic coverage of conventional threats and force a decision on every boundary. Then overlay OWASP LLM Top 10 to make sure the AI-specific risks are named and owned. Then use MITRE ATLAS as the shared kill-chain vocabulary when you brief a security team or design detections — it maps your threats onto observed adversary behavior. STRIDE finds them, LLM Top 10 categorizes the AI ones, ATLAS operationalizes the response.
5. Secure-by-Design Controls, Per Zone
A threat model is only worth the mitigations it drives. Below, the controls are grouped by zone so they map directly onto the decomposition. The organizing principle throughout is secure-by-design: build the control into the architecture so security does not depend on every developer remembering to do the right thing at runtime.
5.1 Ingress & security (Zone 1)
- SSO and centralized authorization. Every request authenticates against the enterprise IdP; no local credentials on inference nodes. Authorize at the gateway so no downstream component has to re-derive "who is this and what may they do."
- Guardrails as a policy layer. Input and output guardrails run in-line at ingress/egress — content policy, PII detection, injection heuristics — as a probabilistic filter, never as the sole boundary.
- Rate limiting and quotas per principal. Cap request rate, token budget, and concurrency per user/service to blunt both DoS and functional-extraction probing.
5.2 Orchestration (Zone 2)
- Identity-aware retrieval. Propagate the caller's identity all the way to the retrieval query so a user can never retrieve chunks they are not entitled to. Authorization must live in the data layer, not just the UI.
- Treat retrieved content as untrusted data. Fence and spotlight it in the prompt; never let document text redefine policy. This is the confused-deputy mitigation.
- Least-privilege, human-gated tools. Scope every tool credential to the current user's permissions, and require out-of-band human confirmation for irreversible or high-impact actions.
5.3 Data & retrieval (Zone 3)
- Source allow-listing and provenance tiers. Only ingest from vetted origins; tag every chunk with a trust label at ingestion and carry it through to prompt assembly. Keep adversarial corpora in separate indexes from privileged ones.
- Ingestion sanitization. Strip hidden HTML/CSS, zero-width and bidirectional Unicode control characters, and instruction-like blocks before embedding — the common indirect-injection and poisoning carriers.
- Poisoning anomaly checks. Watch for bursts of near-duplicate documents or chunks with abnormal imperative-instruction density.
5.4 Inference (Zone 4)
- Signed, verified model artifacts. This is the single most under-invested control on-prem. Cryptographically sign every weights file, adapter, and tokenizer; verify the signature and a known-good hash at load time; refuse to serve on mismatch. It closes the supply-chain path (LLM05) and detects a tampered/backdoored artifact (LLM10) before it ever runs.
- Encrypt weights at rest, restrict read access. Model files live on encrypted storage with keys held in the KMS/HSM; only the serving service account can read them. Weight theft becomes theft of an undecryptable blob.
- KV-cache isolation. On multi-tenant serving, disable cross-request/prefix cache sharing between tenants — or partition it — so cache state cannot leak across principals.
- Resource limits on the serving runtime. Cap max output tokens, context length, and per-request compute to contain model-DoS vectors (LLM04).
5.5 Secrets (Zone 5)
- Centralized secrets with short-lived credentials. No secrets in env files, images, or prompts. Issue scoped, rotating credentials from the KMS/HSM; audit every retrieval.
- Isolate the blast radius. A compromised tool runner must not be able to read the model-decryption key or another tenant's credentials — segment secret access by service identity.
5.6 Cross-cutting: network & observability
- Network segmentation and egress control. Put each zone in its own network segment with default-deny between them and default-deny outbound from the whole stack. Allow-list only the specific internal hosts each hop needs. This is what makes zero data egress a structural guarantee rather than a policy promise — even a fully hijacked model cannot ship data to a host it cannot reach.
- Observability for detection. Log the full chain — query, retrieved chunks with provenance, assembled prompt, output, and every tool call with arguments — to a tamper-evident store (addresses Repudiation). Plant canary tokens in sensitive documents and the system prompt; alert if one ever appears in an outbound call. Protect the logs themselves, since they now hold prompt and response content.
6. Prioritised Checklist — Top Risks First
Not every control lands the same quarter. Sequence by expected loss: the highest-impact, most-likely, structurally-hardest-to-retrofit items first. A defensible order for an on-prem inference stack:
- Lock down model-weight confidentiality (LLM10). Encrypt weights at rest, restrict read access to the serving identity, and verify integrity on load. The crown-jewel artifact deserves the first control.
- Sign and verify all model/adapter artifacts (LLM05). Break the supply-chain path into your highest-trust zone before you ever pull a third-party adapter.
- Enforce authN/authZ at the gateway and identity-aware retrieval. Stop spoofing and cross-user disclosure at the two boundaries that gate everything else.
- Default-deny egress and network segmentation. The structural control that caps exfiltration blast radius regardless of how a request is compromised.
- Least-privilege, human-gated tools (LLM08). Ensure "the model was fooled" cannot become an irreversible action.
- Harden retrieval: source allow-listing, provenance tiers, ingestion sanitization (LLM01/LLM03). Close the indirect-injection and poisoning carriers.
- Contain model DoS (LLM04). Per-principal rate limits, token budgets, and serving-runtime resource caps.
- Isolate the KV cache on multi-tenant nodes (LLM06). Remove the quiet cross-request leak.
- Full-chain logging, canary tokens, and tamper-evident storage. You will not fix what you cannot see; make injections and exfiltration leave evidence.
- Re-model on every material change. A new tool, a new corpus, or a new adapter changes the surface — threat modeling is a loop, not a launch gate.
The honest framing
On-prem does not make you secure; it makes you responsible. You have traded a set of third-party risks for a set of first-party ones, and you now own every control that used to be someone else's job. Threat modeling is how you turn that ownership from a liability into an advantage — because on your own infrastructure, you can actually enforce every boundary you draw.
7. Bottom Line
A threat model is not paperwork; it is the design conversation that decides whether your inference stack is defensible. Decompose into trust zones, run STRIDE across every component, overlay the OWASP LLM Top 10 and MITRE ATLAS to catch what STRIDE cannot express, and let the resulting threats drive secure-by-design controls with a clear priority order. Do it before you deploy, and re-do it whenever the surface shifts. The reward for owning the whole stack is that every boundary you identify is a boundary you can actually enforce.