1. What LLM09 Actually Is
Overreliance occurs when a workflow treats generated output as fact, judgement, or authority without an independent check. Hallucinations are one symptom, but the broader issue is misplaced trust: false citations, outdated policy interpretation, incomplete retrieval, weak math, unjustified confidence, or a correct answer applied to the wrong context. The security risk is that bad output can trigger operational, legal, financial, or safety decisions.
The pattern is familiar in security operations: a triage assistant marks an alert benign without evidence, a code assistant suggests an unsafe patch, a policy assistant invents an exception, or an incident runbook omits a containment step. None require a malicious user. A well-meaning user under pressure can over-trust fluent text.
Training users is not enough
User education helps, but production controls should assume fatigue, time pressure, and authority bias. Verification belongs in the product, not only in a slide deck.
2. Failure and Abuse Shapes (Defanged)
2.1 False authority
The model presents an answer with confident language and plausible structure. The downstream workflow accepts it because it looks complete, even though source evidence is missing or weak.
2.2 Retrieval miss
The answer is grounded in retrieved context, but the retriever returned the wrong slice or missed a newer controlling document. The model is faithful to the wrong evidence.
2.3 Citation laundering
A generated citation, policy name, or source reference is malformed or points to a document that does not support the claim. The citation gives the answer a credibility it has not earned.
2.4 Automation bias in security workflows
Analysts accept a model triage result because it reduces workload. An attacker benefits when malicious activity is summarized as routine or when a containment step is omitted.
3. Detection and Quality Signals
Overreliance is detected through disagreement, missing evidence, and workflow outcomes. The system should log grounding quality, source coverage, uncertainty, user overrides, reviewer changes, downstream incidents, and eval failures by task class.
| Signal | What it indicates | Response |
|---|---|---|
| Answer without source citations in a citation-required route | Ungrounded output reached decision point | Block or downgrade answer |
| Citation does not support claim | Grounding failure or citation laundering | Require reviewer or re-run retrieval |
| High-confidence language with low retrieval score | Model is overstating evidence | Normalize uncertainty and alert product owner |
| Reviewer frequently edits same answer type | Systematic task weakness | Add eval cases and improve prompt/retrieval |
| Model recommendation reversed after incident | Overreliance caused business impact | Post-incident eval and workflow control update |
For governance, NIST AI RMF provides the right management framing: identify contexts where erroneous output creates harm, measure reliability and uncertainty, manage with controls, and govern with accountability. MITRE ATLAS becomes relevant when an attacker intentionally induces or exploits the wrong recommendation.
4. Verification Patterns That Hold Up
- Grounded answers by default. Require citations to retrieved source spans for factual claims. Do not let the model cite documents it did not actually receive.
- Claim checking. Extract key claims and verify them against source text, structured data, or deterministic tools before presentation.
- Task-specific evals. Maintain eval suites for the decisions the system actually supports: legal clause extraction, SOC triage, policy Q&A, code change review, or customer response drafting.
- Confidence calibration. Use restrained language and explicit uncertainty states. Refusal or escalation is better than confident guessing.
- Human review where stakes are high. HITL should focus on specific evidence and decision fields, not a vague approve button.
- Separation from execution. Recommendations should not automatically trigger LLM08-style agent actions without policy and confirmation gates.
# ILLUSTRATIVE - groundedness gate
required_for_route: policy_advice
checks:
- each_material_claim_has_source_span
- source_span_accessible_to_user
- source_date_not_superseded
- answer_contains_uncertainty_when_sources_conflict
on_fail:
response_mode: ask_for_review
log_event: llm_grounding_gate_failed5. Mapping to WSTG, ATLAS, and Related LLM Risks
| Overreliance pattern | OWASP WSTG / ASVS lens | Framework lens | Related LLM risks |
|---|---|---|---|
| Ungrounded factual answer | Business-logic and data-validation testing | NIST AI RMF measure/manage reliability | LLM06 |
| Bad code or query suggestion | Secure coding and injection-prevention review | ATLAS manipulation of model output | LLM02, LLM07 |
| Automated action from unverified answer | Transaction integrity and authorization | NIST govern/manage; ATLAS impact | LLM08 |
| Poisoned evidence accepted as truth | Input validation and provenance testing | ATLAS staged malicious data | LLM03, LLM01 |
The right control is not generic skepticism. It is a workflow-specific definition of what counts as evidence, what the model is allowed to decide, and where the system must stop until a human or deterministic check verifies the answer.
Build Verification into the Workflow
These risks are not solved by a policy paragraph. They need adversarial testing, defensive telemetry, and architecture changes that can be inspected. Use the AI red-teaming path to validate controls, the blue-team guide to operationalize detections, and the contact route to request a focused review.