Red-teaming LLMs the way attackers actually do it.

What an AI red team actually tests

An LLM feature is rarely a single model call. It is a pipeline: untrusted user input, a system prompt, a retrieval layer pulling documents from a vector store, one or more tools or function calls the model can invoke, and downstream code that consumes whatever the model produces. Each hand-off is a trust boundary, and each trust boundary is where an attacker lives. A credible red team walks the entire OWASP LLM Top 10 across that pipeline rather than treating the model as a black box you only poke through the chat window.

The full 2025 catalogue anchors the engagement:

• LLM01 — Prompt Injection: instructions that override system intent, delivered directly or indirectly.
• LLM02 — Insecure Output Handling: model output trusted by downstream code, browsers or shells.
• LLM03 — Training Data Poisoning: tampered fine-tuning or RAG corpora that bias, backdoor or degrade behaviour.
• LLM04 — Model Denial of Service: resource-exhausting prompts and context abuse against self-hosted inference.
• LLM05 — Supply Chain Vulnerabilities: compromised weights, datasets, adapters and ML dependencies.
• LLM06 — Sensitive Information Disclosure: leakage of PII, secrets or proprietary data via responses, retrieval or logs.
• LLM07 — Insecure Plugin / Tool Design: over-trusting tool interfaces that accept unvalidated model-controlled input.
• LLM08 — Excessive Agency: agents with more permission, autonomy or functionality than the use case needs.
• LLM09 — Overreliance: unverified model output driving decisions without human or programmatic checks.
• LLM10 — Model Theft: extraction or exfiltration of self-hosted weights and their behaviour.

The four risks below carry the most impact on a real on-prem RAG/agent stack, so they get a full walkthrough: the attack, a realistic exploitation angle, how a blue team catches it, and the fix. Proof-of-concept material here is deliberately conceptual and redacted — the deliverable to a client contains reproducible detail; a public page does not.

LLM01 — Prompt Injection

The attack. Prompt injection is control-plane confusion: the model cannot reliably distinguish trusted instructions (the system prompt) from untrusted data (everything else in its context). Direct injection is the user typing adversarial instructions into the chat. Indirect injection is the dangerous one — the payload rides in on data the pipeline itself ingests: a PDF in the RAG corpus, a support ticket, a web page a tool fetches, a code comment, the alt-text of an image, or the JSON a downstream API returns to the agent. The user never sees it; the model reads it as instructions.

Exploitation angle (on-prem RAG/agent). Consider an internal assistant that answers HR questions over a document store any employee can contribute to. An attacker uploads a benign-looking policy document containing hidden instructions — placed in white-on-white text or a metadata field — that tell the model, when this document is retrieved, to append the contents of other retrieved chunks to a summary and phrase it as a helpful "related links" footer pointing at an attacker-controlled endpoint. When a victim later asks a question that pulls that document into context, the injected instructions execute with the victim's retrieval scope. The redacted shape of it: [ingested document] → retrieved into context → model follows embedded directive → tool call / output crafted by attacker. In an agentic setup the same primitive escalates: the injected text targets an available tool (e.g. "email", "http", "shell") and turns a read into an action.

How the blue team detects it. Log the full assembled prompt (system + retrieved chunks + tools) alongside the completion, and diff intent against action: retrievals that are immediately followed by an outbound tool call to a new destination are a high-signal anomaly. Screen ingested documents and tool responses for instruction-like patterns and invisible/steganographic text before they reach the context window. Track per-document "influence" — which corpus items disproportionately precede policy-violating outputs.

Mitigation. Treat every non-system token as untrusted data. Enforce a strong system/user separation, keep retrieved content clearly delimited and never grant it authority, and gate all consequential tool calls behind deterministic policy checks and human confirmation rather than model discretion. Sanitise and normalise documents at ingestion time (strip hidden text, validate metadata), and scope retrieval and tool permissions to the requesting user, not the corpus.

LLM02 — Insecure Output Handling

The attack. This is the mirror image of injection: not what goes into the model, but what the application does with what comes out. If downstream code renders, evaluates or forwards model output without treating it as attacker-controlled, the LLM becomes a confused deputy that smuggles classic web and injection vulnerabilities past your defences.

Exploitation angle (on-prem RAG/agent). A model response is rendered directly into a web dashboard as HTML — so a completion that contains markup executes in the analyst's browser, yielding stored XSS whose payload originated in a poisoned corpus document (chaining cleanly off LLM01). Same class, other sinks: model output interpolated into a database query becomes SQL injection; output passed to a shell or a "run this code" tool becomes command execution; a URL the model emits and a tool blindly fetches becomes SSRF against internal services. The redacted shape: model output → trusted sink (DOM / SQL / shell / HTTP) → no sanitisation → classic exploit. This is squarely OWASP WSTG territory — the LLM just moved the injection point upstream.

How the blue team detects it. Instrument the sinks, not just the model: alert on model-derived strings reaching HTML rendering, query construction, subprocess execution or outbound HTTP without passing an encoder/validator. Content-Security-Policy violation reports surface XSS attempts in the dashboard; egress monitoring surfaces SSRF to internal ranges.

Mitigation. Apply the same output encoding and validation you would to any untrusted input — context-aware HTML encoding, parameterised queries, allow-listed commands, and an egress proxy with an allow-list for any tool that fetches URLs. Never eval model output. Constrain tool arguments to typed, validated schemas so the model cannot hand a sink a raw string.

LLM06 — Sensitive Information Disclosure

The attack. The system reveals data the requesting user should never see — PII, secrets, proprietary content, or fragments of other tenants' documents — through model responses, over-broad retrieval, or verbose logs and error messages.

Exploitation angle (on-prem RAG/agent). The most common real-world failure is authorization collapse at the retrieval layer: the vector store is queried with the service account's permissions rather than the end user's, so a low-privilege employee asks a well-phrased question and the RAG pipeline happily retrieves and summarises board-level or HR documents they have no rights to. A second angle is secret leakage — API keys or connection strings embedded in a system prompt or a tool's context can be coaxed out, and stack traces surfaced to the user disclose internal paths and configuration. The redacted shape: low-priv user → query → retrieval runs as service account → cross-tenant / cross-clearance chunks in context → summarised back to attacker.

How the blue team detects it. Correlate the classification/ACL of retrieved chunks against the requesting principal and alert on any mismatch — a user pulling documents above their clearance is the signal. Run canary documents seeded across permission tiers and watch for them appearing in the wrong user's responses. Scan completions and logs for secret patterns and PII with DLP tooling before they leave the boundary.

Mitigation. Enforce authorization at the data layer, not the prompt: filter the vector search by the user's ACLs so unauthorized chunks are never retrieved in the first place. Keep secrets out of prompts and tool context entirely (use a broker with per-call scoping), strip PII at ingestion where it is not needed, and return generic error messages while logging detail internally.

LLM04 — Model Denial of Service

The attack. Self-hosted inference is finite and expensive. An attacker who cannot break confidentiality can still break availability by exhausting GPU memory, saturating the request queue, or driving latency and cost past acceptable limits — the on-prem equivalent of a resource-exhaustion DoS, and a real business risk when the model backs a customer-facing or operational workflow.

Exploitation angle (on-prem RAG/agent). Maximum-length inputs that fill the context window force the most expensive possible forward pass per request; prompts crafted to elicit very long or non-terminating generations pin a worker; and recursive or looping agent chains — a tool whose output re-enters the model and triggers another tool call — amplify a single request into many. Against a fixed GPU pool with no per-tenant quotas, a modest number of concurrent crafted requests can starve legitimate traffic. The redacted shape: crafted long / looping request × concurrency → GPU + queue saturation → latency spike → legitimate requests time out.

How the blue team detects it. Baseline tokens-per-request, generation length, GPU utilisation and queue depth, then alert on outliers and on single principals driving disproportionate load. Agent loops show up as an abnormal ratio of tool calls to user turns within one session — a strong runaway signal.

Mitigation. Enforce input-length caps, per-user and per-tenant rate limits and token quotas, and hard max_tokens and wall-clock timeouts on every generation. Bound agent recursion with an explicit step budget and loop detection. Isolate workloads so one tenant's spike cannot starve another, and put autoscaling or load-shedding in front of the inference pool.