Stopping Data Leakage from On-Prem LLMs: OWASP LLM06 Sensitive Information Disclosure

1. What LLM06 Actually Is

Sensitive Information Disclosure (OWASP LLM06) is the risk that an LLM application reveals data it should have kept confidential: personally identifiable information (PII), authentication secrets and API keys, proprietary or regulated business data, internal system details, or — most insidiously — another user's data. The disclosure can be the whole answer or a single leaked field; it can be triggered deliberately by a probing attacker or fall out of an entirely benign question. What makes it distinct from a classic data breach is that the leak is mediated by a model that summarizes, paraphrases, and recombines whatever context it is handed, so the sensitive value need never appear verbatim to still escape.

The on-prem misconception

The comforting story is: "We run the model ourselves, so our data never leaves — LLM06 is solved." It is not. Running on-prem removes one specific egress channel, the third-party processor. It does nothing about the far more common failure mode inside enterprises: over-sharing across internal boundaries. If your retrieval layer can read the entire index and your model answers every user from that same undifferentiated pool of documents, then the finance team's assistant can happily summarize an HR investigation, and a contractor's chatbot can quote from a board deck. The data never left the building — it simply reached someone inside the building who was not authorized to have it. On-prem changes where the risk lives, not whether it exists.

2. How Data Actually Leaks

LLM06 is not one bug; it is a family of channels, each with its own root cause. You cannot defend what you have not enumerated, so map every path by which a sensitive value can reach a principal who should not see it.

2.1 Training / fine-tuning memorisation

If you fine-tune a model on sensitive corpora, the model can memorise and later regurgitate fragments of that data — verbatim strings, names, account numbers — even to users who had no access to the source. Memorisation is worst for rare, high-entropy sequences (exactly the shape of secrets and identifiers) and for records that appear many times in the training set. Once baked into weights, that data is not access-controlled at all: everyone who can query the model can, in principle, probe for it.

2.2 RAG retrieval crossing permission boundaries

This is the classic and most common on-prem leak: "the model answered from a document this user should not have been able to see." The retrieval layer performs a similarity search across the whole vector store, pulls the top-k chunks, and the model dutifully synthesizes an answer — with no check that the asking user is authorized to read the source documents. The vector store has become a permission-blind side channel around the access controls that the underlying systems (the wiki, the file share, the CRM) enforce carefully. Ingestion imported the content but discarded the ACLs.

2.3 Secrets in system prompts

Developers frequently stuff API keys, database credentials, internal endpoints, or privileged instructions into the system prompt because it is convenient. The system prompt is not a vault — it is context, and context leaks. A determined user can often coax the model into reprinting its instructions, and even without an attacker, verbose behaviour can surface fragments. Anything in the prompt should be treated as disclosable to the user.

2.4 Verbose error messages

Unhandled exceptions and debug output are a quiet disclosure channel. A stack trace, a raw database error, or a failed tool response echoed back to the user can reveal internal hostnames, file paths, query structure, connection strings, or the contents of an object that failed to serialize. The model may also helpfully relay the error verbatim.

2.5 Logs and traces capturing prompts + responses

Observability is essential for LLM systems — but naive logging captures full prompts and responses, which by construction contain user PII, retrieved confidential content, and sometimes the secrets from the system prompt. Those logs then flow into a SIEM, an APM tool, or a debugging bucket with far broader read access than the source data ever had. The telemetry pipeline becomes an unmonitored copy of your most sensitive data, sitting in a lower-trust store.

2.6 Embeddings inversion (conceptual)

Vector embeddings are sometimes assumed to be a safe, "anonymised" representation of text. Conceptually they are not: research on embedding inversion shows that a meaningful amount of the original text can be reconstructed from its embedding vector alone. So a vector database is not a privacy boundary — an actor who can read the vectors (a backup, a replica, an over-permissioned index) may be able to recover approximations of the sensitive source text, not just opaque numbers. Treat the embedding store with the same sensitivity as the documents it was derived from.

3. The Core On-Prem Problem: Retrieval Must Respect Authorization

The single most important principle for LLM06 on an on-prem RAG stack is this: retrieval must enforce the same authorization as the systems the data came from. If a user cannot open a document in SharePoint, in the file share, or in the CRM, then a similarity search must not be able to hand that document's contents to the model on their behalf. The vector store is not a neutral search index — it is a new access path to the same sensitive data, and it needs the same access control.

Access control at retrieval time, not just at ingestion

A common half-measure is to filter at ingestion: decide which documents are "sensitive" and simply not index them. That fails because authorization is rarely binary and rarely static. The same corpus is readable by different users to different extents; permissions change after ingestion; a document that was public becomes restricted. If you bake a single trust decision in at ingestion, you cannot represent "user A may see this chunk but user B may not," which is the actual shape of enterprise permissions.

The correct model is document-level and row-level access control evaluated at query time. Every chunk carries the identity of the principals allowed to read it — ideally the same ACL, group IDs, or row-security predicate as the source system. When a user queries, their identity and group memberships are propagated into the vector search as a metadata filter, so the similarity search only ever ranks chunks the user is already entitled to read. The permission check happens on every request, reflecting current authorization, not a snapshot from ingestion day.

Per-user filtering in the vector store

Concretely, this means the vector database must support metadata filtering that is applied as part of the nearest-neighbour search, not as a post-filter on results (post-filtering can silently return zero results, or worse, be bypassed). Each vector is tagged at ingestion with the source ACL — allowed user IDs, group IDs, tenant ID, sensitivity/classification label. At query time the application resolves the caller's identity and groups from a trusted session (never from anything the model or the user's free text can influence) and constructs a hard filter that the engine ANDs with the vector query. Multi-tenant deployments should go further and physically or logically partition tenants so a filter bug cannot cross the tenant boundary at all.

4. Defense-in-Depth: The Layered Controls

No single control stops LLM06. Identity-aware retrieval is the backbone, but it is surrounded by data-minimisation upstream and DLP-style filtering downstream, so that a gap in one layer does not become a disclosure.

4.1 Before ingestion: classify and minimise

• Data classification. Tag documents with a sensitivity label (public, internal, confidential, restricted/regulated) before they enter the pipeline, and carry that label onto every chunk. Classification is what lets every later layer make a decision.
• Minimisation. Do not ingest what the assistant does not need. Exclude corpora that carry high-sensitivity data with no retrieval value, and strip fields (raw identifiers, credentials, free-text notes containing PII) that add risk without adding answers.
• Never fine-tune on secrets or raw PII. To avoid memorisation (§2.1), keep secrets and high-entropy identifiers out of training data entirely; prefer RAG with access control over fine-tuning when the data is sensitive, so the data stays access-controlled outside the weights.

4.2 At retrieval: identity-aware access control

• Propagate the user's permissions into the query. Resolve identity and group membership from the session and translate them into a hard metadata filter ANDed with the vector search (see §3 and the pseudocode below).
• Tag every chunk with its source ACL and tenant. Ingestion must copy the authorization model of the source, not discard it.
• Fail closed. If identity or ACL data is missing or ambiguous, return nothing rather than everything.

# ILLUSTRATIVE — not production code
def retrieve(session, user_query, k=8):
    # 1. Identity + permissions come from the TRUSTED session,
    #    never from the prompt or any model output.
    principal   = session.user_id
    groups      = session.group_ids          # resolved server-side
    tenant      = session.tenant_id
    max_class   = session.clearance          # e.g. "confidential"

    # 2. Build a HARD authorization filter. This is ANDed with the
    #    vector search by the engine, not applied after the fact.
    authz_filter = {
        "tenant_id": tenant,                 # never cross tenants
        "acl_principals": { "$in": [principal, *groups] },
        "classification": { "$lte": max_class },
    }

    # 3. The similarity search only ever *sees* chunks the caller
    #    is already entitled to read.
    hits = vector_db.search(
        embedding = embed(user_query),
        top_k     = k,
        filter    = authz_filter,            # enforced in-engine
    )

    # 4. Fail closed: no identity / no ACL -> no data.
    if principal is None or not hits.filter_applied:
        return []
    return hits

4.3 On input and output: PII detection and redaction (DLP)

• Input DLP. Scan user prompts for PII and secrets before they are logged or forwarded, so you are not the one creating a new copy of sensitive data in a lower-trust place.
• Output DLP. Inspect the model's response before it reaches the user. Detect and redact PII, credential-shaped strings, and content that resembles the system prompt or another tenant's data. This is the last line of defence when retrieval filtering has a gap.
• Redaction over blocking where possible. Masking the sensitive span (leaving the useful answer) preserves utility better than refusing outright, though high-risk categories should hard-block.

4.4 Secrets: out of the prompt entirely

• Never put secrets in system prompts. API keys, DB credentials, and privileged tokens belong in a secrets manager, fetched at call time by trusted code and never placed into any text the model processes.
• Scope credentials to the user. Tool credentials should carry the caller's permissions, so a tool cannot read data the user could not (this also caps LLM06 blast radius, not just LLM01/LLM08).

4.5 Telemetry: scrub logs and traces

• Redact before you persist. Run prompts and responses through the same DLP redaction before writing them to logs, traces, or analytics. The observability store should never hold raw PII or secrets.
• Tighten access and retention. Restrict who can read LLM logs to match the sensitivity of the underlying data, and keep retention short. A debug bucket readable by the whole engineering org is a disclosure waiting to happen.

4.6 Errors, guardrails, and tenancy isolation

• Sanitise error messages. Never surface raw stack traces, DB errors, or internal paths to the user; return a generic message and log the detail (scrubbed) internally.
• Guardrails. Add a policy layer that blocks obvious disclosure attempts (requests to reprint the system prompt, dump records, or enumerate other users' data) — imperfect and probabilistic, so one layer among many.
• Tenancy isolation. In multi-tenant systems, isolate tenants at the storage layer (separate indexes/namespaces) so a single filter bug cannot leak across tenant boundaries. Defence in depth for the highest-severity failure mode.

5. Leakage Channel → Control

Each disclosure channel from §2 has a primary control (or two). Map them explicitly so nothing is left unowned.

Note: OWASP periodically re-numbers and re-scopes the LLM Top 10 across editions. Treat the "LLM06" label as conceptual — verify the current item name and ID against the edition your organization has standardized on.

6. The Honest Bottom Line

On-prem is a real and worthwhile control, but it solves the cloud-egress problem, not the disclosure problem. The moment you build a RAG system, you create a new, permission-blind path to all the data you indexed — and if you do not push authorization down into retrieval, you have quietly flattened every access boundary in your organization into one searchable surface. The good news is that the fix is not exotic: it is the same identity-and-access discipline that governs the rest of your stack, applied to the vector store and the model context.

Enforce the source system's authorization at query time. Classify and minimise before you ingest. Keep secrets out of prompts and out of weights. Redact PII on the way in and on the way out, including into your logs. Isolate tenants at the storage layer. None of these is sufficient alone; together they mean that a gap in one layer is caught by the next, and a disclosure becomes a contained, observable event rather than a silent breach. That is the honest sell for LLM06: not that data can never leak, but that leaking it takes more than one thing going wrong.