1. Scope the Exercise Like a Kill Chain
A useful RAG red-team exercise starts with boundaries. Which ingestion sources are in scope? Which parsers and chunkers? Which vector indexes? Which users and tenants? Which tools can the assistant call after retrieval? The goal is to test the path from untrusted data to model behavior without touching production secrets or causing side effects.
- Entry point. Document upload, shared drive sync, email ingestion, ticket attachment, crawler, partner portal, or wiki edit.
- Transformation. Parser, OCR, HTML sanitizer, chunker, embedding model, metadata enrichment, and deduplication.
- Retrieval. Query rewriting, top-k selection, reranking, access-control filtering, and prompt assembly.
- Execution. Model response, tool selection, output rendering, egress controls, and user workflow.
Redacted by design
The exercise uses harmless marker text, canary IDs, and fake destinations. It validates control behavior without embedding working exfiltration payloads or harmful instructions.
2. Defanged End-to-End Path
2.1 Plant a benign canary document
The red team creates a document that looks like normal business content and includes a clearly marked, harmless instruction-like canary. It should never ask the model to leak real data or contact an attacker. The canary exists only to test whether low-trust document text can alter model behavior.
2.2 Track the document through ingestion
The team records parser output, chunk boundaries, metadata, trust label, source owner, embedding ID, and index promotion time. If the canary disappears, the sanitizer worked or the test needs adjustment. If it survives hidden in a chunk, the blue team should be able to find it later.
2.3 Trigger retrieval with authorized queries
Use ordinary user questions that could legitimately retrieve the document. The test should show whether the poisoned chunk wins retrieval, whether source trust is carried into prompt assembly, and whether the model treats the chunk as data rather than policy.
2.4 Observe model and tool behavior
A successful control stack either ignores the canary, fences it as untrusted, blocks tool use after low-trust retrieval, or raises an alert. A failure is not a dramatic exploit; it is any evidence that a document changed policy, tool arguments, output links, or confidence.
3. Blue-Team Evidence to Capture
| Evidence | Why it matters | Retention note |
|---|---|---|
| Raw document and normalized text | Shows whether sanitizer changed or preserved the canary | Hash raw bytes and parser output |
| Chunk metadata and trust label | Proves provenance survived ingestion | Keep source, owner, parser, and index version |
| Retrieval trace | Shows why the chunk reached the model | Store scores, filters, and top-k list |
| Assembled prompt | Shows trusted vs untrusted boundaries | Redact secrets but keep structure |
| Tool-call policy decision | Shows whether low-trust context could cause action | Log allow/deny reason and principal |
| Output rendering event | Shows whether links or Markdown were filtered | Log destination host class, not sensitive URL data |
Map the path to OWASP LLM01 for indirect prompt injection, LLM03 for poisoned retrieval data, LLM07 if a tool boundary is crossed, and MITRE ATLAS for staging malicious data before model execution.
4. Controls That Should Fire
- Ingestion quarantine. New sources and large content changes should be staged before production indexing.
- Trust-aware prompt assembly. Retrieved content must be fenced and labelled as untrusted data, not mixed into policy text.
- Tool gating after low-trust retrieval. If public or user-uploaded content influenced the answer, side-effect tools should be blocked or require confirmation.
- Canary alerting. Harmless markers in test documents should alert if they appear in tool arguments, external links, or policy-changing model output.
- Source-level rollback. The team must be able to remove a source, rebuild affected chunks, and invalidate cached answers quickly.
# ILLUSTRATIVE - harmless RAG red-team assertion
assert retrieved_chunk.trust in ["public", "user_uploaded"]
assert assembled_prompt.fences_untrusted_context is True
assert tool_policy.side_effects_allowed is False
assert output_filter.external_links_allowlisted is True
assert siem.event_type == "llm.indirect_injection_canary_seen"Run the Test as an Evidence Exercise
The field notes above are intentionally conceptual and defanged, but the controls are practical. Use AI red-teaming to validate the attack path safely, blue-team detection engineering to make the signals operational, and contact to scope a review of your own environment.