Prompt Injection Prevention: 7 Rules for AI Guardrails

What Prompt Injection Actually Is

Prompt injection is what happens when content the model reads is treated as instructions the model follows. It is not a jailbreak, not a content-moderation failure, and not something a better system prompt fixes. It has topped the OWASP Top 10 for LLM Applications since the list existed, and NIST classifies it as a form of evasion attack in its adversarial machine learning taxonomy.

The reason it resists the obvious fix is structural: instructions and data arrive in the same channel, as tokens, with no cryptographic or syntactic distinction between them. Delimiters do not create one. Neither do detection classifiers, which move the success rate down without moving it to zero - and a control with a nonzero bypass rate is not a control when the attacker can retry. That leaves one honest engineering position. Assume the model will, on some input, do exactly what an attacker wrote, then design so the worst available outcome is a wrong answer rather than a wire transfer.

1. Model Output Is Untrusted Input Everywhere

Model output is user-controlled data, and the vulnerability lives at the sink - the shell command, the SQL string, the innerHTML assignment, the deserializer - not in the prompt. This is the category OWASP files as improper output handling, and it is where LLM features turn into the vulnerability classes your AppSec programme already knows how to talk about: injection, SSRF, XSS, remote code execution.

sql = llm.complete(f"Write SQL for: {question}")
rows = db.execute(sql)   # output reaches the query sink verbatim

intent = QueryIntent.model_validate_json(llm.complete(prompt))
rows = QUERIES[intent.report_name](
    tenant_id=principal.tenant_id,   # never model-supplied
    start=intent.start_date, end=intent.end_date,
)

Never pass model output into an execution or query sink; map it through a declared schema to a statically referenced operation with bound parameters. For anything rendered in a UI, use the template engine's contextual escaping - never innerHTML, never a trusted-HTML construct, never a Markdown renderer configured to permit raw HTML. You give up the demo where the model answers any question about any table. What you get back is a feature you can put in front of an auditor.

2. Validate Against a Schema, Never Parse Prose

A model that returns prose returns a different shape on the day your traffic doubles, and string parsing turns that into a silent write of the wrong data. The failure is not dramatic. A field arrives as "none" instead of null, a confidence score comes back as a percentage instead of a fraction, a regex matches the first of two amounts. It lands in a database, and you find it in a reconciliation three weeks later.

const amount = reply.match(/\$([0-9.]+)/)[1];
await ledger.write({ amount });   // shape is whatever today's output was

const parsed = ExtractionSchema.safeParse(JSON.parse(reply));
if (!parsed.success) return fallbackToDeterministicPath(doc);
await ledger.write(parsed.data);

Parse model output into a declared schema, validate field types and value ranges, and define in code what happens when validation fails - retry with a repair prompt, fall back to a deterministic path, or fail the request. What you cannot do is leave the failure path undefined, because undefined here means the exception surfaces far from the cause. The same reasoning that governs error handling and exception discipline applies at this boundary, with the added wrinkle that the input is nondeterministic by construction.

3. Retrieved Content Is an Attacker's Message

Teams reliably treat user input as hostile and then treat retrieved context as trusted, on the unexamined logic that the system chose to retrieve it. That is indirect prompt injection's entire attack surface: a support ticket, a scraped page, a PDF a customer uploaded, a third-party API response, a code comment in a repository your agent reads.

const docs = await vectorStore.search(query, {
  filter: { tenantId: principal.tenantId },   // mandatory, not a post-filter
});
messages.push({ role: "user", content: asDelimitedData(docs) });

Insert retrieved documents, fetched pages, file contents, and third-party responses into the prompt as clearly delimited data in a user or tool message - never in the system instruction, and never by string interpolation of external values into it. Then scope retrieval itself: apply the requesting principal's tenant, ownership, and permission filters as mandatory constraints on every vector search.

Retrieval by relevance alone is a cross-tenant read waiting to be discovered - the same failure multi-tenant data isolation addresses in the query layer. Rule of thumb: if an attacker can get text into a system your agent reads, they can send your agent a message. Ask which of your ingestion paths an outsider can write to. The answer is usually more than one.

4. Tool Calls Run as the User, Always

The service account pattern is the default in every agent framework: the tool holds its own credentials, the framework calls it, and the user's identity stops at the chat boundary. From that point on, a successful injection does not need to escalate - it already has. Execute every model-requested tool call through the same authorization path, with the same principal scope, as a direct user request would take.

async function getInvoice(args: ToolArgs, ctx: RequestContext) {
  assertValid(ToolArgsSchema, args);
  return invoiceService.get(args.invoiceId, ctx.principal);
}   // same authz path as the REST route

Validate the arguments of a model-requested tool call against a declared schema - type, format, permitted ranges - before executing it. And never derive an authorization decision from model output: a model-selected identifier, role, or scope passed into a permission check is an attacker-selected one.

Authorization is evaluated server-side against the authenticated principal, before the operation runs, exactly as it would be for any other request. The tool is not a new trust boundary. It is the same one, reached through a different door.

5. Non-Reversible Actions Need Human Confirmation

"The model decided to" is not an authorization, and for actions that cannot be undone it must not be an execution path. Sending a message, moving money, deleting data, publishing content, changing a permission - these are the actions where a successful injection stops being an embarrassing screenshot and becomes an incident with a regulator attached.

// system prompt: "Always ask the user before sending email."
await sendEmail(args);   // an injection can claim the user said yes

const token = await confirmations.require(ctx.principal, pendingAction);
await sendEmail(args, token);   // rejects without a recorded approval

The confirmation has to be structural rather than a prompt instruction telling the model to ask first - an injected instruction can assert the user already confirmed, but it cannot forge a confirmation record. This is where autonomy dies, and it should: the design patterns for securing LLM agents literature makes the case directly that meaningful guarantees come from constraining what agents are permitted to do.

6. Prompt Payloads Are Built From Allowlists

The line looks harmless in review - JSON.stringify(user) into the message content - and it is harmless until an ORM relation, a new column, or an internal note field turns it into a data-transfer event with a third party. The same instinct produces the second half of this failure: logging the assembled prompt and the full response for debuggability, which writes everything the model saw into a store with a different access model and a different retention policy.

Build prompt and embedding payloads from an explicitly enumerated set of fields. Never pass a whole entity, request payload, database row, or conversation-history object. On the telemetry side, log the model identifier, token counts, latency, and outcome - never the assembled prompt, the retrieved context, or the full response. This is the AI-shaped instance of PII leak prevention in logs, errors, and telemetry, with a vendor boundary added.

While you are at the call site: set an explicit request timeout, a maximum output token count, and a per-request or per-principal call limit. Provider defaults for all three are generous, and unbounded consumption is the cost-shaped version of a denial of service.

7. Every New Tool Rewrites the Threat Model

The danger is not in any single tool but in the combination, and the combination is assembled by different teams, in different pull requests, over months. The lethal trifecta names it precisely: access to private data, exposure to untrusted content, and a way to communicate externally. Any two are survivable. All three in one agent is a data exfiltration path that requires no vulnerability in any individual component.

The team that added document retrieval was not thinking about the team that added the outbound webhook tool six weeks later. Neither change is wrong. Neither reviewer had the context to object. The trifecta assembles quietly, at the seams between two correct decisions, and it is visible only from a view of the whole codebase.

Resolve every outbound destination from a configured allowlist rather than a model-supplied URL, and never render a model-supplied URL as an automatically loaded resource - an image tag with attacker-chosen query parameters is a complete exfiltration channel. Diagnostic worth running: list every tool your agent can call and mark which ones read outside content, which read private data, and which can reach the network. If any agent has all three columns marked, you have the trifecta today.

How Pandorian Enforces This Guide

Every principle above survives a design review. Few survive six months of independent team decisions, because the violations aren't visible from inside any one pull request - they're visible only in the aggregate. Pandorian enforces this guide across three surfaces:

Caught at PR level

Model output assigned to innerHTML, a tool argument passed unvalidated, a new inference call with no token cap or timeout: flagged with the line, the rule, and a suggested fix, while the author still remembers why they wrote it.

Full scans

An on-demand scan across the whole codebase sets the baseline and sees compositions no diff contains. The retrieval tool and the outbound HTTP tool that jointly complete the lethal trifecta sit in different files, added months apart, by different teams.

Scheduled scans

The same scan on a cadence, catching drift introduced outside any PR: a new tool registered in a shared agent config, an allowlist widened during an integration push, a prompt template that grew an interpolated field.

A Taste of the Guideline Package

Four of the checks that ship in this package - each runs at the surface where it catches the problem earliest.

FAQ