1. What LLM03 Actually Is
Data poisoning is the attack class where the adversary changes the data that teaches, conditions, or grounds the system, then waits for the model or retriever to behave badly later. For classical ML this often meant corrupting training samples. For LLM applications it also means poisoning a RAG index, a prompt template repository, a fine-tuning set, a preference dataset, or an evaluation suite. The common thread is persistence: the malicious influence is stored in the system and can fire long after the attacker has left.
A useful mental model is to treat every corpus as executable influence. The model may not execute code, but it does execute patterns: it imitates data, retrieves data, summarizes data, and uses retrieved content as context for tool decisions. Poisoned data therefore becomes a latent policy change. This is why LLM03 sits next to LLM01 Prompt Injection: injection is often the runtime symptom, while poisoning is the supply path that plants the instruction or backdoor.
On-prem does not remove poisoning
Keeping the vector store and weights inside your perimeter helps with confidentiality, but it does not prove the corpus is clean. Internal wiki pages, ticket exports, uploaded PDFs, feedback forms, and fine-tuning examples can all be attacker-influenced.
2. Attack Shapes (Defanged)
Conceptual only
The examples below describe the shape of the technique for defense and testing. They omit working prompts, trigger strings, and payload text.
2.1 Poisoned RAG document
An attacker places a plausible document into a location the ingestion pipeline trusts: a shared folder, a ticket attachment, a partner portal, a wiki page, or a web page scraped by the organization. The document contains normal domain content plus an instruction-like region crafted to survive chunking and rank for likely queries. When retrieved, it can steer the model toward unsafe disclosure, tool misuse, or a false business answer.
2.2 Fine-tuning backdoor
A small number of fine-tuning examples are crafted so that a rare trigger phrase or feature causes a specific behavior: policy bypass, a wrong classification, or a preferred answer. The trigger is not used in ordinary evaluation, so broad quality metrics stay healthy while the targeted slice is compromised.
2.3 Feedback-loop poisoning
If user ratings, thumbs-up examples, analyst corrections, or chat transcripts are recycled into future tuning without review, an attacker can gradually bias the assistant. This is low-and-slow poisoning: no single record looks decisive, but the aggregate feedback teaches the system the wrong preference.
2.4 Evaluation poisoning
A subtler variant targets the test set rather than the model. If the evaluation suite is weakened, duplicated, or made unrepresentative, unsafe releases appear to pass. This is especially dangerous when evals are used as automated deployment gates.
3. Blue-Team Detection
Poisoning detection is mostly provenance plus drift. You need to know what changed, who changed it, how the changed data moved through the pipeline, and which model behaviors changed afterwards. Without lineage, every suspicious answer becomes a manual archaeology exercise.
| Signal | Why it matters | Example response |
|---|---|---|
| Sudden corpus growth from one source | Poisoning often enters as a burst of near-duplicate files or chunks | Quarantine source, diff new chunks, require owner approval |
| Imperative language inside reference data | Documents that tell the assistant what to do are abnormal outside policy repositories | Score instruction density and review high-risk chunks |
| Retrieval rank shifts on sensitive queries | Poisoned chunks are often tuned to outrank legitimate documents | Run fixed query canaries before promoting an index |
| Eval slice regression with stable aggregate score | Backdoors hide in narrow trigger slices | Track targeted safety and security evals separately |
| Unexplained answer convergence | Many unrelated prompts produce the same policy or destination | Trace to shared retrieved chunks or fine-tune examples |
Carry MITRE ATLAS vocabulary into the incident record: the adversary prepared data, staged it in a training or retrieval path, waited for model execution, and sought impact through output or tool behavior. That mapping helps security teams treat poisoning as an attack campaign rather than a data-quality bug.
4. Mitigation and Build Controls
- Source allow-listing. Keep public, partner, customer-uploaded, and internal-authoritative corpora in separate indexes with different trust labels. Do not let untrusted web data ground privileged internal answers.
- Data lineage. Attach source, owner, ingest time, hash, parser version, and approval state to every chunk and every fine-tuning record. Log those fields into traces so an answer can be traced back to the exact data that influenced it.
- Staging indexes. New or changed corpora should land in quarantine first. Run poisoning checks, retrieval canaries, and security evals before promoting the index to production.
- Signed datasets and model artifacts. Treat curated corpora and training sets like release artifacts: immutable snapshots, checksums, signatures, review history, and rollback.
- Backdoor-focused evals. Include trigger-style and policy-conflict tests. The point is not to publish triggers; it is to prove the model does not change behavior on narrow, adversarial slices.
- Human review for feedback loops. User feedback can be valuable, but it should be sampled, moderated, deduplicated, and weighted by trust before it influences training.
# ILLUSTRATIVE - data promotion gate, not production code
for chunk in staged_index.new_chunks:
require chunk.source in approved_sources
require chunk.owner is not None
require chunk.hash == recompute_hash(chunk.raw_bytes)
require instruction_density(chunk.text) < policy.max_instruction_density
require not contains_hidden_text_or_bidi_controls(chunk.text)
run_retrieval_canaries(staged_index)
run_security_eval_slice(model_candidate, staged_index)
require security_owner_approval before promote_to_production5. Mapping to WSTG, ATLAS, and Related LLM Risks
| Control area | OWASP WSTG / ASVS lens | MITRE ATLAS lens | Related LLM risks |
|---|---|---|---|
| Ingestion authorization | Authentication, authorization, and access-control testing for upload and edit paths | Initial access and ML attack staging through data sources | LLM01, LLM06 |
| Corpus integrity | Input validation, file handling, and configuration management | Poison training data or stage malicious data | LLM05, LLM09 |
| Fine-tune release gates | Secure build and deployment verification | Backdoor ML model and model behavior manipulation | LLM05, LLM10 |
| Runtime tracing | Logging and monitoring verification | Detect execution and impact after poisoned context is used | LLM01, LLM07 |
The WSTG mapping is conceptual: RAG ingestion and training pipelines are not traditional web forms, but the same testing questions apply. Who can submit data? How is it validated? Can it cross tenant boundaries? Can a low-trust source change high-trust behavior?
Make Poisoning Testable
The practical test is whether these controls survive contact with a realistic attacker and a realistic SOC. Use the AI red-teaming methodology to pressure-test the failure modes, the blue-team service to turn the telemetry into detections, and the contact form to request a focused review of your own on-prem LLM stack.