1. The LLM Stack Is a Log Source
A production LLM application is not a black box — it is one of the most instrumentable systems you will ever defend. Between the gateway, the retriever, the guardrail classifiers, the tool runtime, and the serving layer, a single inference produces a dozen structured events. Detection engineering is the discipline of turning that exhaust into signal: normalizing it into events, shipping it to a SIEM, and writing detections that fire on the behavioral shape of an attack. The premise is simply that an LLM gateway deserves the same detection coverage as a web proxy or an EDR agent — because functionally, that is what it is: a policy chokepoint that every request passes through.
The building blocks are already familiar to any detection engineer. Nginx has access logs; your LLM gateway has a request/response record. Sysmon has process-creation events; your agent runtime has tool-invocation events. The novelty is not the plumbing — it is knowing which fields carry attack signal and how the injection, exfiltration, and DoS techniques of the LLM world manifest in them.
OpenTelemetry is your transport
Emerging GenAI semantic conventions in OpenTelemetry give you a vendor-neutral schema for LLM spans — model, token counts, tool calls, and latency all become trace attributes. Export those spans to your log pipeline and the "LLM SIEM feed" is mostly a matter of parsing well-structured events rather than scraping unstructured stdout.
2. What to Log (and What to Redact)
You cannot detect on data you did not capture, and you cannot retain data you are not allowed to keep. Both constraints matter. Prompts and retrieved documents routinely contain personal data, credentials, and confidential business content, so the logging design is also a privacy design. The goal is maximum detection value at minimum retained sensitivity.
The fields worth capturing
- Request metadata — timestamp, authenticated principal (user/service ID), session/conversation ID, source IP, model and version, and a request ID that ties every downstream event together. This is the join key for the whole chain.
- Prompt fingerprints — rather than always storing raw prompt text, store a normalized hash and cheap features: length, token count, language, and boolean flags for known injection markers. Keep raw text only where policy allows, and treat it as sensitive.
- Retrieval sources — for every retrieved chunk: its document ID, provenance/trust tier, source URI, and retrieval score. This is the single most under-collected field and the one you most need for indirect-injection forensics.
- Tool invocations — tool name, arguments (redacted/hashed where sensitive), target host, and result status. Treat this exactly like a process-creation log.
- Guardrail & output-filter verdicts — every refusal, every classifier hit, every DLP match, with the rule that fired and its score. Refusals are not noise; they are attempts.
- Performance counters — prompt/completion token counts, end-to-end latency, and cost per request. These are your DoS and resource-abuse signals.
Privacy caveat: log the shape, not the secret
Hash or tokenize PII before it hits the SIEM; redact obvious secret patterns at ingestion; keep raw prompt/response text in a shorter-lived, access-controlled store separate from the searchable index. Detection can run on fingerprints and feature flags — you rarely need the raw sensitive text in the same place your analysts query all day. Define retention per field, and make sure your logging does not itself become the sensitive-information-disclosure incident you were trying to detect.
3. A Detection Catalogue
Below is a starter catalogue: for each attack pattern, the signal you watch, the conceptual MITRE ATT&CK tactic it lives under, and the OWASP LLM Top 10 risk it covers. ATT&CK's enterprise tactics were not written for LLMs — the mapping is a conceptual bridge to give your SOC shared vocabulary, and MITRE's ATLAS knowledge base is the more precise reference for AI-specific techniques. Use both honestly: ATLAS for the ML-native taxonomy, ATT&CK tactics for the language your existing detection team already speaks.
| Pattern | Signal to watch | ATT&CK tactic (conceptual) | OWASP LLM |
|---|---|---|---|
| Prompt-injection markers in inputs or retrieved docs | Imperative "ignore instructions / new role" phrasing, hidden/zero-width text, or instruction-like density in a user turn or a retrieved chunk | Initial Access / Execution | LLM01 Prompt Injection |
| Anomalous tool / agent actions | Tool call to a never-before-seen host, a privileged tool invoked in a low-privilege context, or an action sequence that deviates from the agent's baseline | Execution / Privilege Escalation | LLM08 Excessive Agency |
| Output-filter / DLP hits | Model output containing secret-shaped strings, system-prompt fragments, or another principal's data flagged by the egress DLP filter | Exfiltration | LLM06 / LLM02 Disclosure & Output Handling |
| Token / cost / latency spikes | Prompt or completion token counts far above baseline, recursive/looping generations, or a burst of expensive requests from one principal | Impact (resource exhaustion) | LLM04 Model Denial of Service |
| Retrieval poisoning | A sudden new high-frequency retrieval source, a burst of near-duplicate ingestions, or a chunk with abnormal imperative-instruction density climbing the rankings | Resource Development / Persistence | LLM03 Training/Data Poisoning |
| Jailbreak probing | Repeated refusals or guardrail hits from the same principal/session — the tell-tale iterate-until-bypass loop | Discovery / Credential-style probing | LLM01 Prompt Injection |
Reading the catalogue
Two of these are worth dwelling on because they are the ones defenders most often miss. Retrieval poisoning (LLM03) is a slow attack: the adversary is not hammering your gateway, they are quietly seeding your corpus so that a future benign query surfaces their payload. The detection lives at ingestion time and in retrieval-frequency baselines, not in the request path. And jailbreak probing is the LLM analogue of password spraying — no single refusal is interesting, but twenty refusals from one session in five minutes is an attacker tuning a payload against your guardrail. Detection engineering shines exactly here: correlating individually-benign events into a campaign.
4. Illustrative Sigma-Style Rules
Illustrative only — not production-ready
The three rules below are teaching sketches. They use a generic, made-up log schema and deliberately naïve conditions to show the shape of an LLM detection. Real rules need your actual field names, tuned thresholds, allow-lists to control false positives, and validation against your own baselines. Do not paste these into a SIEM and expect them to work — treat them as a starting grammar, not a signature feed.
4.1 Injection markers in the gateway log
Fires when a user turn or retrieved chunk contains phrasing that tries to redefine the model's instructions. Keyword matching alone is noisy — pair it with the trust tier of the span and treat a hit inside retrieved content as far higher signal than one in a user turn, because legitimate documents rarely address "the assistant."
# ILLUSTRATIVE — not production-ready
title: LLM Prompt-Injection Markers in Gateway Log
status: experimental
description: Instruction-override phrasing in user input or retrieved context
logsource:
product: llm_gateway
service: inference
detection:
markers:
text|contains|all:
- 'ignore'
- 'previous instructions'
role_markers:
text|contains:
- 'you are now'
- 'disregard the system'
- 'new instructions:'
high_trust_span:
span_type: 'retrieved' # injection inside DATA is high signal
condition: (markers or role_markers) and high_trust_span
fields: [request_id, principal, source_uri, retrieval_score]
level: high
tags:
- attack.initial_access
- owasp.llm01
4.2 Egress / exfiltration markers in model output
Fires when the model's output carries data toward an attacker — a URL to a non-allow-listed host, an auto-loading Markdown image, or a canary token that should never leave its source document. This is your last line before the response reaches a browser or a downstream tool.
# ILLUSTRATIVE — not production-ready
title: LLM Output Egress / Exfiltration Markers
status: experimental
description: Model output routing data to an external sink or leaking a canary
logsource:
product: llm_gateway
service: output_filter
detection:
external_link:
output_url_host|not_in_allowlist: true
markdown_image:
output|re: '!\[[^\]]*\]\(https?://' # auto-loading image
canary_leak:
output|contains: '{CANARY_TOKENS}' # seeded honeytokens
condition: canary_leak or (external_link and markdown_image)
fields: [request_id, principal, output_url_host, matched_canary]
level: critical
tags:
- attack.exfiltration
- owasp.llm06
- owasp.llm02
4.3 Token-flood denial of service
Fires on resource-exhaustion patterns: a single principal driving token counts or request volume far above their baseline, or individual requests whose generation runs away. Threshold-based rules like this are only as good as the baseline behind them, so compute the baseline per principal, not globally.
# ILLUSTRATIVE — not production-ready
title: LLM Token-Flood / Model DoS
status: experimental
description: Abnormal token consumption or request burst from one principal
logsource:
product: llm_gateway
service: metering
detection:
runaway_generation:
completion_tokens|gt: 8000 # tune to your model's norm
request_burst:
requests_per_min|gt: 60 # per-principal, over baseline
cost_spike:
request_cost_ratio_to_baseline|gt: 10
condition: runaway_generation or request_burst or cost_spike
fields: [request_id, principal, completion_tokens, requests_per_min]
level: medium
tags:
- attack.impact
- owasp.llm04
5. Canary Tokens as Tripwires
The highest-signal, lowest-false-positive detection in the whole catalogue is not a heuristic — it is a tripwire you plant yourself. Seed unique, unguessable canary tokens (honeytokens) into places an attacker would only reach by breaking a boundary: sensitive documents in the RAG corpus, the system prompt, high-value records. The token has no legitimate reason to ever appear in output, in a tool argument, or in an outbound request.
When one does surface — as in rule 4.2 above — the interpretation is unambiguous: something read a place it should not have and is now trying to move that data. There is almost no benign explanation, which is exactly what makes canaries so valuable in a discipline otherwise dominated by probabilistic, tunable signals. A canary in an egress attempt is an exfiltration caught in the act. Distribute distinct tokens across distinct sensitive sources and the token that fires also tells you which source was reached.
Where to plant them
One canary embedded in the system prompt catches system-prompt-extraction attempts. One per sensitive corpus document turns any indirect-injection-driven read into a detectable event. And a canary in a "decoy" high-value record catches the agent that was talked into over-reaching. Rotate them, keep the registry out of the model's reach, and alert on any appearance with critical severity.
6. From Detection to Incident Response
A detection that fires into a void is wasted engineering. Each rule above should terminate in a documented response path, because the join key you designed in Section 2 — the request ID that ties query, retrieved chunks, prompt, output, and tool calls together — is precisely what makes AI-incident DFIR tractable. When an alert lands, the responder should be able to pull the entire chain for that request and answer the forensic questions: what was retrieved, did a canary fire, which tool was called, what left the boundary.
That is why logging the retrieved chunks (not just the answer) is non-negotiable: without them, an indirect-injection investigation has no crime scene. Map each detection to a triage action — isolate the session, revoke the tool credential, quarantine the poisoned document, or force human review — so the SOC has a runbook rather than a shrug. The broader logging, canary, and anomaly-detection practice, and the AI-incident runbook itself, are treated as a discipline in the blue-team guide.
7. The Honest Bottom Line
Detection is the half of the LLM security story that vendors under-sell and defenders under-build, but it comes with a clear-eyed caveat: detections reduce dwell time and blast radius — they do not replace secure design. A Sigma rule that catches injection markers is worthless if the agent has ambient super-privileges and open egress, because by the time your rule fires the data is already gone. Detection is the layer that assumes the preventive layers around it will sometimes fail; it is not a substitute for building them.
So treat this catalogue as one tier in a stack. The preventive controls — least-privilege tools, default-deny egress, trust-tiered retrieval, output escaping — are covered in the defense-in-depth playbook, and the offensive counterpart that pressure-tests both prevention and detection is covered in the AI red-teaming approach. Detection engineering earns its place by turning a silent breach into an observed, timestamped, recoverable event — and that is a real, honest win, as long as you never mistake it for immunity.