When LLM Output Becomes an Exploit: OWASP LLM02 Insecure Output Handling

1. What LLM02 Actually Is

Insecure Output Handling (OWASP LLM02) is the vulnerability that arises when a downstream component accepts, trusts, and acts on the output of an LLM without validation, sanitisation, or encoding — as if that output were safe, well-formed, developer-authored data. It is not a flaw in the model. It is a flaw in the plumbing that sits after the model, in the same place where a traditional application would (correctly) distrust user input but forgets to distrust machine-generated text.

The mental model is straightforward and it is one that web-security engineers already own: an LLM is an untrusted input source for every system it feeds. Its response can contain HTML, JavaScript, SQL fragments, shell metacharacters, URLs, template directives, or Markdown — because it was trained on all of those, and because an attacker upstream may have coaxed it into emitting exactly the string they want. If you would never take a raw form field and drop it unescaped into your DOM, you must not do that with a model response either.

The confused deputy, again

An LLM sits between a low-trust source (the prompt, the retrieved documents, the tool outputs) and high-trust sinks (your browser, your database, your shell, your internal HTTP client). It holds the application's privileges but produces content that may have been dictated by an attacker. When a downstream sink honours that content literally, the model has acted as a confused deputy: it carried an attacker's intent across a trust boundary that the attacker could not cross on their own. Insecure output handling is the name for the missing check at that boundary.

How LLM02 differs from LLM01

LLM01 (Prompt Injection) and LLM02 are frequently confused because they often appear in the same kill chain, but they are distinct:

• LLM01 — Prompt Injection is about the cause: attacker-controlled text steers the model into producing an attacker-chosen output. It is an attack on the model's behaviour.
• LLM02 — Insecure Output Handling is about the consequence: a downstream system trusts and executes/renders that output. It is an attack on the code around the model.

The two are independent failure modes. You can have LLM02 without any prompt injection at all — a model can spontaneously emit an HTML tag or a stray SQL keyword that breaks a naive downstream parser, or a legitimate user can ask a question whose honest answer contains markup. And you can have LLM01 without LLM02 — a jailbreak that only produces disallowed text but never reaches an unsafe sink causes reputational harm, not code execution. The dangerous case is the chain: an indirect prompt injection (LLM01) instructs the model to emit a weaponised string, and insecure output handling (LLM02) lets that string fire in a browser, database, or shell. Fix LLM02 and you break the chain even when the injection succeeds — which is the whole point, because prompt injection remains an unsolved problem.

2. The Downstream Sinks That Bite

Insecure output handling only becomes an exploit at the point where output meets a sink that interprets it. Each sink corresponds to a well-known injection class; the LLM just supplies the payload. The sinks that matter in practice:

2.1 Rendering output as HTML → XSS

The most common and most damaging. If the application drops a model response into the page via innerHTML, a non-escaping template, or a Markdown renderer configured to allow raw HTML, then any <script>, event-handler attribute, or auto-loading tag in the output executes in the victim's session. In a RAG chat UI this is often stored XSS: the poisoned instruction lives in an indexed document, and the malicious output is generated and rendered for whichever user later triggers retrieval — no direct interaction with the attacker required. In a single-turn assistant it can be reflected XSS. Either way, the rendering surface is a security boundary, not a cosmetic detail.

2.2 Passing output to a shell or eval → command / code execution

Agentic systems that let the model "write and run" code, or that build a shell command from a model-suggested string, hand the attacker remote code execution if the output is executed directly. The same applies to any eval(), exec(), deserialization, or dynamic import that consumes model text. This is the highest-severity sink because it converts a text vulnerability into full runtime compromise of the agent host.

2.3 Building SQL from output → SQL injection

A common pattern is "natural language to SQL": the model emits a query fragment or a filter value that the application splices into a statement. If that fragment reaches the database as concatenated string SQL rather than a bound parameter, the model has just written your WHERE clause — and an upstream injection can make it write a destructive or exfiltrating one. Text-to-SQL is a legitimate feature; string-concatenated text-to-SQL is SQL injection with an LLM as the injector.

2.4 Using output as a URL for a server-side fetch → SSRF

If the agent takes a model-produced URL and fetches it server-side (to summarise a page, call a webhook, resolve a link), an attacker who controls the output controls the destination. That is textbook SSRF: the request can be pointed at internal services, cloud metadata endpoints, or other hosts the server can reach but the user cannot. The model becomes a request-forgery gadget sitting inside your trust perimeter.

2.5 Writing output into a template engine → SSTI

When model output is interpolated into a server-side template (email bodies, generated reports, dynamic pages) and the template engine evaluates it, template directives in the output are executed — server-side template injection, which frequently escalates to code execution depending on the engine. Model output must be treated as template data, never as template source.

2.6 Markdown auto-linking → silent data exfiltration

A subtle sink that needs no scripting. Many chat UIs render Markdown and auto-load images. If the output embeds sensitive context (retrieved records, secrets, prior conversation) into the query string or path of an image or link URL, the victim's client fetches it from the attacker's host on render — leaking the data in the request. No <script>, no tool call; the Markdown renderer itself is the exfiltration channel. This is why "we only render Markdown, not HTML" is not by itself a defence.

3. Attack Shapes (Defanged)

3.1 Indirect injection to a rendered XSS/beacon

An attacker plants instructions inside a document your RAG pipeline will ingest — a support ticket, a wiki edit, a scraped page. The embedded text, phrased as guidance to "the assistant," asks the model to include a specific HTML image tag or Markdown link in its answer, with a sensitive value folded into the URL. A benign user later asks an unrelated question; retrieval surfaces the poisoned chunk; the model dutifully emits the tag. Because the internal app renders that output as rich content, the browser either executes injected script (XSS) or auto-loads the image, beaconing the embedded data to the attacker's server. The LLM01 injection supplied the intent; the LLM02 rendering gap delivered it.

3.2 Model-generated SQL fragment reaching the database

In a text-to-SQL feature, an upstream injection steers the model to emit a filter or clause that the application concatenates into a live query. Instead of a harmless equality predicate, the fragment carries additional structure that changes what the statement does when the database parses it. The user sees a normal-looking answer; the damage happens in the query the application ran on their behalf. The root cause is not the model's creativity — it is that the fragment was trusted as SQL rather than bound as a value.

3.3 Tool-argument and URL abuse

Where the agent turns output into actions, a steered response can name an internal URL for a server-side fetch (SSRF) or place sensitive context into a tool argument bound for an outbound call. In both cases the exploit lives entirely in the gap between "the model said it" and "the system did it" — the missing validation of output before it becomes an operation.

4. Defences: Treat All Output as Untrusted

There is exactly one governing principle, and everything else follows from it: model output is untrusted input to every downstream system. The good news, relative to prompt injection, is that LLM02 is a solvable engineering problem. These are the same battle-tested controls the web-security world already relies on; the only new discipline is remembering to apply them to text that came from your own model.

4.1 Context-aware output encoding

Encoding is not one operation — it depends entirely on the destination context, and using the wrong one provides no protection. Encode at the moment of use, for the specific sink:

• HTML context — HTML-entity-encode before insertion into markup; prefer text nodes (textContent) over innerHTML; never render model text as raw HTML.
• JavaScript context — never place model output inside a script context or a string passed to eval; if data must reach JS, serialise it as JSON in a data attribute and read it inertly.
• URL context — URL-encode components and validate the scheme/host against an allow-list before any output is used as a link or a fetch target.
• SQL context — do not encode-and-hope; use parameterised queries / prepared statements so the value can never be parsed as query structure (see below).

4.2 Parameterised queries, never string SQL

For any database sink, bind model-derived values as parameters. A bound parameter is data by construction — the database cannot reinterpret it as SQL syntax — which neutralises the entire text-to-SQL injection class regardless of what the model emitted. If the model must influence structure (table, column, order), map its output to an allow-listed enumeration rather than interpolating it.

4.3 Never eval/exec on model output; sandbox tool execution

Do not pass model output to eval, exec, a shell, a deserializer, or a template compiler. Where code or command execution is a genuine product requirement, run it in a locked-down sandbox — an isolated, ephemeral runtime with no ambient credentials, a read-only or throwaway filesystem, resource limits, and default-deny network egress. The sandbox assumes the executed content is hostile, because it may be.

4.4 Allow-listing and schema validation

Prefer allow-lists to blocklists everywhere: permitted URL hosts, permitted tools, permitted actions, permitted output shapes. Where the task allows, constrain the model to a schema (typed JSON, an enumerated action set, a restricted grammar) and validate the output against that schema before use. A response that can only be one of N validated shapes has almost no room to carry an injection into a sink.

4.5 Content Security Policy and egress filtering (defence-in-depth)

• CSP — a strict Content Security Policy on the rendering surface limits what injected markup can do: disallow inline scripts, restrict img-src/connect-src to trusted origins. It will not stop every Markdown beacon, but it raises the cost and is a cheap, high-value backstop for the HTML sink.
• Egress filtering — default-deny outbound network access from the agent and rendering-adjacent runtimes, allow-listing only the hosts your system legitimately needs. Even a fully steered model cannot exfiltrate to a host the runtime cannot reach. This is the structural control that survives when encoding is forgotten.

# ILLUSTRATIVE — not production copy
raw = llm.generate(prompt)          # <-- UNTRUSTED. Same trust as anon user input.

# 1) HTML sink: encode for HTML, render as text, never as raw markup
def to_browser(raw):
    return html_entity_encode(raw)  # or set node.textContent = raw
    # NEVER: element.innerHTML = raw

# 2) SQL sink: bind as a parameter, never concatenate
def to_db(value):
    db.execute("SELECT * FROM docs WHERE owner = ?", [value])   # bound param
    # NEVER: db.execute("... WHERE owner = '" + value + "'")

# 3) URL / fetch sink: validate scheme + host against an allow-list
def to_fetch(url):
    u = parse(url)
    if u.scheme not in ("https",) or u.host not in ALLOWED_HOSTS:
        raise Reject("egress not permitted")
    return http.get(u)              # only reached for allow-listed hosts

# 4) code/command sink: do not eval. If unavoidable, sandbox it.
def to_exec(code):
    raise Reject("model output is never executed on the host")
    # if truly required: run in isolated sandbox, no creds, egress DENY

4.6 Relationship to LLM01 and LLM08

LLM02 is the middle link in a common chain. Upstream, LLM01 (Prompt Injection) is what supplies a malicious output in the first place — see the companion prompt-injection defense-in-depth guide for the input side. Downstream, LLM08 (Excessive Agency) is what determines how much damage a trusted-but-hostile output can do once it reaches a tool: an over-privileged agent with broad tools turns a rendering bug into an operational breach. Hardening output handling breaks the chain in the middle: even a successful injection is contained if its output can never fire in a sink and the agent it feeds is least-privileged.

5. Sink → Control Mapping

The fix for LLM02 is always chosen by the destination, not by the model. This table maps each downstream sink to the correct encoding or control.

Note: OWASP periodically re-numbers and re-scopes the LLM Top 10 across editions, and Insecure Output Handling has been reframed and merged with adjacent items in later revisions. Treat the LLM02 label as conceptual — verify the current item name and ID against the edition your organisation has standardised on.

6. The Honest Bottom Line

Insecure output handling is the most tractable item in the LLM security stack precisely because it is not a model problem — it is an application-security problem wearing an AI costume. The controls are decades old and well understood: context-aware encoding, parameterised queries, allow-listing, sandboxing, CSP, egress control. Nothing here requires a new research breakthrough. It requires the discipline to remember that the string coming out of your model is exactly as trustworthy as a string coming from an anonymous user — which is to say, not at all.

This is also why LLM02 is where you get the most leverage against prompt injection. You cannot yet stop a model from being steered into producing a hostile string, but you can absolutely stop that string from firing. Close every sink, keep the feeding agent least-privileged, and a successful injection degrades from a breach into a logged, contained non-event. That is the pragmatic sell: fix the plumbing you control, and the model's fallibility stops being your liability.