Securing generative AI features with the OWASP LLM Top 10

Say a bank ships an assistant that reads a customer’s statements and answers questions about them. It demos beautifully. Then a field that the model treats as content, a transaction description or an uploaded document, turns out to carry text the model reads as an instruction, and the assistant does something it was never meant to: calls a tool it should not have reached, or starts talking about data that belongs to someone else.

That is the whole problem with generative AI features in one sentence. The model will, sooner or later, follow instructions you did not write, and the only thing standing between that and a real incident is what you allowed the model to touch. The OWASP Top 10 for LLM Applications 2025 is a useful map of how these features fail, and for teams shipping a chatbot, copilot, RAG search or agent into a regulated product, it pays to read it as a design brief rather than a glossary.

Old failure modes under a new name

Most of the LLM Top 10 maps onto failures application security teams already know. Improper output handling is an injection bug. Excessive agency is a privilege problem. Sensitive information disclosure is a data-exposure problem. Unbounded consumption is denial of service with a billing dimension. The new ingredient is a component in the middle that is non-deterministic and that will, given the right input, act against your intent.

Prompt injection, LLM01, is the one to understand first, because it drives so much of the design. It arrives in two forms. Direct injection is an instruction typed by the user who is talking to the model. Indirect injection is an instruction hidden in content the model reads on the user’s behalf, a retrieved document, a web page, an email, an uploaded file, so the person who planted it never touches your app at all. The indirect case is the dangerous one for RAG and agents, because the model cannot reliably separate the data it was asked to summarize from an instruction smuggled inside that data.

So the defensive posture that works is architectural. You cannot make the model reliably refuse every malicious instruction, so you build the system such that the model’s cooperation with an attacker buys the attacker very little. Where the controls sit matters more than how clever any single one of them is.

1. 01User input and retrieved contentboth untrusted; the model may read attacker-influenced text from either
2. 02Model callthe system prompt holds behavior only, no secrets, no access rules
3. 03Output handlingtreat the response as untrusted; encode for the browser, parameterize for SQL, validate before any tool call
4. 04Tool and action layerleast-privilege tools; high-impact actions require explicit human confirmation
5. 05Retrievalscope the vector query to the user’s own permissions so the store cannot return documents they may not read

Treat the model’s output as untrusted input

The control that prevents the widest range of damage is also the least glamorous. Whatever the model returns, treat it the way you treat anything a user typed, because in effect it is. Model output that reaches a browser without encoding is a cross-site scripting bug. Output interpolated into a SQL string is an injection bug. Output handed to a shell, a template engine or an internal API without validation is remote code or server-side request forgery waiting to happen. This is LLM05, improper output handling, and it is where an otherwise contained chatbot turns into a foothold.

The rule is mechanical and easy to review. The model’s response goes through the same encoding and validation as any untrusted input at every sink: HTML-encode before rendering, parameterize every query, validate and allowlist before any action. If the model produces structured output that triggers a tool call, validate that structure against a strict schema and reject anything that does not match, rather than trusting the model to stay in format. The point of this layer is that it holds even when the model has been talked into producing something hostile, which is the case you are actually defending against.

Two sinks catch people out because they feel safe. Markdown rendered to HTML can carry active content and links that pull data out when the page loads, so render model output through a strict allowlist rather than trusting it to be plain text. And a response that drives a function call needs its arguments validated like any API input: an amount that must be positive, an account id that must belong to the caller, an action drawn from a fixed set. The schema check is what stops a talked-around model from passing a value your code would never have accepted from a form.

Give tools only the privileges the task needs

LLM06, excessive agency, is the risk that a model or agent has more permission or autonomy than the job requires, so that a bad decision, whether from injection or just a confused model, has reach. An assistant that only needs to read a customer’s own balance should hold a read-only, customer-scoped credential and nothing else. The instant you give an agent a tool that can move money, delete records or change a configuration, you have raised the ceiling on what a successful injection can do to match.

Two rules keep this in check. Scope every tool to the minimum: read-only where reading is enough, a single tenant or user where that is the context, no standing write access the task does not require. And for any high-impact action, put a human in the loop with a real confirmation step that states plainly what is about to happen. “Transfer 5,000 SAR to account ending 1234, confirm?” is a control. A silent tool call driven by model output is not. Design the autonomy down to what the feature genuinely needs, then stop.

Prompt injection has no complete fix, so the design goal is narrow: make sure a successful injection cannot reach anything that matters.

Keep secrets and access logic out of the prompt

A recurring mistake is to treat the system prompt as a secure configuration store, placing API keys, connection strings, internal URLs or access rules into it because it is “hidden” from the user. It is not hidden. LLM07, system prompt leakage, exists because these instructions can be coaxed out, and once an access rule lives in the prompt it can be both read and reasoned around.

Assume the system prompt will be revealed in full, and design so that revealing it costs you nothing. Keep it to behavior and tone. Enforce every access decision in your own code, outside the model, against the authenticated user’s real permissions, so that even an attacker holding your exact system prompt gains no privilege from it. This is also how you keep LLM02, sensitive information disclosure, in check: secrets and other users’ data never enter the context window in the first place, so there is nothing there to leak.

Logging is the mirror image of the same problem. You need enough of a record to investigate an incident, what the user asked, which tools ran, what the model returned, and you must not casually write full prompts and responses full of personal or financial data into logs a wide audience can read. Log the metadata and the decisions, redact the content, and hold whatever sensitive detail you genuinely must keep under the same access controls and retention limits as any other PDPL-relevant data.

There is a prior question specific to this market: where the data in the prompt goes. Sending a customer’s transactions or a citizen’s record to a model provider hosted outside the Kingdom is a cross-border transfer with PDPL implications, and for some government and banking workloads it pushes you toward a model deployed in-region or on-premises. Settle that before you design the feature, because retrofitting data residency onto a shipped product is slow and expensive.

Enforce the user’s permissions at retrieval

RAG is where a lot of Saudi and GCC products are heading, grounding an assistant in the organization’s own documents, policies, contracts, tickets. It is also where a quiet, serious authorization bug hides, and it is LLM08, vector and embedding weaknesses, in practice.

The failure is this: the vector store is treated as a single shared corpus, and retrieval pulls the most relevant chunks for a query regardless of who is asking. A junior employee asks a question and the retriever happily surfaces a chunk from a document only executives should see, and the model summarizes it back to them. The embedding did its job. The access control was simply never applied.

Fix it at retrieval time, with the same discipline you would use on any data query. Carry the user’s identity and permissions into the retrieval call and filter the vector search by them, so the store can only return chunks the user is cleared to read. Partition the index by tenant, so one bank’s documents and another’s can never be retrieved together, and tag chunks with their access level at ingestion so the filter has something to act on. The model should never be the thing deciding whether a user may see a document, because by the time the model sees the chunk, the decision has already been made for it.

Put a ceiling on consumption

LLM10, unbounded consumption, is broader than denial of service, because every model call costs money and a loop that would merely slow a traditional service can run up a serious bill against a metered API. A single abusive client, or a legitimate one caught in a retry loop, can turn a feature into a line item nobody budgeted for.

Put ceilings everywhere it matters: rate limits per user and per API key, a cap on tokens per request and per user per day, timeouts on model calls, and a hard quota that fails closed rather than billing without limit. Watch spend as a security signal, with alerts on sudden changes, the same way you would watch an unusual spike in traffic. For a consumer-facing feature in a market with sharp usage peaks, size these limits against your real peak and not a quiet afternoon, so a legitimate surge is not mistaken for abuse and a genuine abuse is not mistaken for a surge.

Three you inherit rather than write

Three of the ten are less about your request path and more about what you build on. Supply chain, LLM03, is the provenance of the model, the libraries and the plugins you pull in. A model or dataset from an unverified source can arrive with weaknesses already in it, so pin versions, check signatures, and treat a model artifact with the same suspicion you would apply to any third-party dependency. Data and model poisoning, LLM04, is the same worry turned toward training and fine-tuning data, and toward the documents you ingest into a RAG index from places you do not control. Misinformation, LLM09, is the plain fact that a fluent model states wrong things with full confidence, which in a banking or government context is a compliance exposure rather than a cosmetic flaw. Ground answers in retrieved sources the user can check, cite them, and have the model decline cleanly when a question falls outside what it was given.

How to test it, in two languages

Testing an LLM feature starts with an evaluation set: a collection of adversarial cases, organized by the category of behavior you are trying to prevent, that you run as a regression on every build. Keep the categories concrete, such as attempts to extract the system prompt, attempts to reach data belonging to another user, attempts to trigger a tool without authorization, attempts to produce output that would be unsafe at a downstream sink. The suite stores the categories and the expected safe behavior, not working attack strings, so it is a defensive asset and not an attacker’s handbook.

Run that suite in CI so a model or prompt change that regresses a guardrail fails the build the way a broken unit test would. Add a human red-team review before any launch, because an evaluation set only covers the failures you already thought of, and a skilled reviewer finds the ones you did not. Treat a model version upgrade as a change that re-runs the whole suite, since behavior shifts between versions and a guardrail that held last month can stop holding without anyone noticing.

One point specific to this region deserves its own line. Test every guardrail in Arabic and in English, independently. Safety behavior is usually tuned most heavily on English, and a control that refuses a category of request cleanly in English can behave differently in Arabic, including in the mix of Arabic and English and transliteration that real Saudi users type. A feature that is safe in your English tests and untested in Arabic is a feature you have not actually tested for your customers.

OWASP LLM risk	Design control	How to test it
LLM01 Prompt Injection	Assume injection can arrive from user input or retrieved content; limit what a compromised model can reach	Adversarial evaluation set by category in CI; human red-team before launch
LLM02 Sensitive Information Disclosure	Keep secrets and PII out of the context; redact before the model sees them; filter outputs	Probe for elicitation of secrets or other users’ data; assert none returned
LLM03 Supply Chain	Pin and verify model, library and plugin provenance; vet third-party models and datasets	Inventory model and dependency sources in the build; check versions and signatures
LLM04 Data and Model Poisoning	Control and validate training and fine-tuning data; isolate RAG ingestion from untrusted sources	Review data provenance; run known-answer cases after any retrain to catch drift
LLM05 Improper Output Handling	Treat output as untrusted: encode for HTML, parameterize SQL, validate before any shell or API call	Send hostile-shaped output through each downstream sink and assert it is neutralized
LLM06 Excessive Agency	Least-privilege tools; human confirmation for high-impact actions; no standing write access	Enumerate each tool’s permissions; confirm destructive actions cannot run without confirmation
LLM07 System Prompt Leakage	No secrets or access logic in the system prompt; enforce access in code	Attempt extraction; confirm there is nothing sensitive in the prompt to leak
LLM08 Vector and Embedding Weaknesses	Enforce the user’s permissions at retrieval; partition the index by tenant and access level	Query as a low-privilege user; assert no document above their clearance is returned
LLM09 Misinformation	Ground answers in retrieved sources, cite them, constrain the model from answering out of scope	Evaluate accuracy on a labeled set; confirm out-of-scope questions are declined
LLM10 Unbounded Consumption	Rate, token and cost limits per user and key; timeouts; quotas that fail closed; spend alerting	Abuse and load tests that confirm limits trigger before cost or latency runs away

The honest caveat to end on is about LLM01. Prompt injection, direct from the user or indirect through content the model reads such as retrieved documents, web pages or emails, does not have a complete fix today, and treating it as solvable leads teams to over-trust their input filters. Spend your effort on the assumption that an injection will eventually succeed, and make the blast radius small: untrusted output at every sink, least-privilege tools, human confirmation on anything that matters, permissions enforced in your code and not in the model. Those controls hold whether or not a given injection gets through, which is the only kind of control worth relying on here.

Sources

1. OWASP Top 10 for LLM Applications 2025, OWASP, 2025.