Prompt Injection Testing Guide: Attack Patterns, Defenses, and Red-Team Checklists

Overview

Prompt injection is an attempt to influence a model through untrusted instructions. The instruction may appear in a user message, a document retrieved by a RAG system, a web page, an email, a code comment, or tool output. The risk is not limited to whether a model says something undesirable. In an AI agent, a successful injection may also change which tools are called, what data is returned, or which actions are proposed for approval.

Testing should therefore assess the complete application rather than only the system prompt. Include the model, orchestration code, retrieval layer, conversation memory, tools, permissions, output handling, and user interface. A strong test asks three questions:

• Can untrusted content alter the model's intended task or operating rules?
• Can the model reveal secrets, hidden instructions, private data, or internal reasoning that it should not expose?
• Can an injected instruction cause an unsafe external action without an appropriate control or human decision?

Run tests only against systems and data you are authorised to assess. Use synthetic secrets, test accounts, isolated environments, and reversible tools. Record the model version, prompt version, retrieved inputs, tool definitions, settings, and expected result so that a failure can be reproduced after a change.

Checklist by scenario

1. Direct prompt injection

Start with attacks placed directly in the user's message. Vary the wording rather than relying on one memorable phrase. Test requests that ask the model to:

• Ignore, replace, or reinterpret its governing instructions.
• Reveal system prompts, hidden policies, credentials, or confidential context.
• Act as a different role with fewer restrictions.
• Follow instructions hidden in quoted text, code blocks, markup, or an encoded string.
• Continue a supposedly authorised workflow without confirming identity or permission.

Check both the response and the application state. A refusal is not sufficient if the model still calls a tool, leaks a value in metadata, or places unsafe content in a downstream request.

2. Indirect prompt injection

Indirect injection is embedded in content that the application retrieves or processes. Add controlled test strings to documents, web pages, tickets, PDFs, spreadsheets, email bodies, and code repositories. Examples include an instruction addressed to the assistant, a request to send retrieved data elsewhere, or text claiming that a security review has already approved an action.

For every content source, verify that the system:

• Labels external content as data rather than trusted instructions.
• Limits which sources can influence decisions or tool parameters.
• Preserves source identifiers so suspicious content can be investigated.
• Prevents retrieved text from silently changing permissions or workflow state.
• Requires grounding and citation where the application makes factual claims.

For related retrieval design, see the RAG prompting best practices guide.

3. Tool-use and agent workflows

Test every tool independently and in combinations. Use harmless test endpoints or dry-run modes for email, file access, databases, payments, tickets, deployment systems, and other consequential integrations. Attempt to make the model:

• Call a tool outside the user's requested scope.
• Supply an attacker-controlled URL, file path, query, recipient, or parameter.
• Skip confirmation because an instruction claims to be urgent or pre-approved.
• Use one tool's output as authority for a more sensitive tool.
• Repeat an action after an error, timeout, or ambiguous response.

Define an expected action for each test: refuse, ask for confirmation, return a safe explanation, or use a restricted tool. The model should not be the sole enforcement point. Apply authorization, allowlists, validation, rate limits, transaction limits, and audit logging in application code.

4. Multi-turn and memory tests

Spread an attack across several turns. A user may establish harmless context, request a role change later, or persuade the assistant that an earlier approval covers a new action. Test conversation summarization and long-context behavior as well. Check whether untrusted text is copied into durable memory, whether one user's content can affect another session, and whether old permissions remain active after logout or role changes.

5. Output and data-exfiltration tests

Place synthetic canary values in system configuration, retrieved documents, and tool responses. Ask for them directly and indirectly: in a summary, a structured response, a diagnostic message, an error field, or a generated URL. Validate outputs against a schema and inspect logs, traces, citations, and tool arguments—not just the visible answer. Structured output can improve reliability, but it does not make untrusted instructions safe by itself. The JSON prompting and validation guide covers the application-side checks.

What to double-check

Before release, review the trust boundaries in a simple data-flow diagram. Mark where user input, retrieved content, model output, tool output, credentials, and external actions enter or leave the system. Then confirm the following controls:

• Instruction hierarchy: system and developer requirements are clearly separated from user and retrieved content. No prompt wording is treated as a substitute for access control.
• Least privilege: each tool has the smallest practical permission set, narrow parameters, and an explicit purpose.
• Human approval: irreversible, financial, privacy-sensitive, or externally visible actions require an approval step outside the model.
• Input handling: content is length-limited, normalised where appropriate, tagged by source, and safely parsed.
• Output handling: generated SQL, shell commands, HTML, URLs, and API parameters are validated or sandboxed before use.
• Observability: logs capture the relevant prompt version, model, retrieval sources, tool calls, decisions, and refusals without storing unnecessary sensitive data.
• Evaluation: attack cases are part of a regression set with pass criteria, severity levels, and an owner for remediation.

Keep the security tests separate from ordinary quality tests, but run both. A response can be accurate and still unsafe if it exposes data or takes an unauthorised action. The LLM evaluation guide provides a useful framework for combining quality and reliability checks.

Common mistakes

• Testing only the system prompt: attackers target retrieval, memory, tool outputs, and application logic too.
• Relying on keywords: blocking a few phrases will not address paraphrases, encoded content, or multi-turn manipulation.
• Giving the model excessive authority: a persuasive answer should never be the only authorization check.
• Using production secrets in tests: use canary values and isolated accounts so a test failure cannot become an incident.
• Checking only the final text: inspect tool calls, intermediate state, logs, citations, and side effects.
• Treating refusal as proof of safety: verify that the application also blocks unsafe actions and prevents leakage through alternate fields.
• Failing to version test cases: retain the exact attack input, expected behavior, model settings, and result for meaningful comparisons.

If your application uses multi-step orchestration, review each hand-off as well as the complete workflow. Prompt chaining can improve task control, but every intermediate output should be treated as potentially untrusted. See the prompt chaining guide for workflow-level error handling patterns.

When to revisit

Prompt injection testing is a recurring engineering activity, not a one-time launch gate. Re-run the relevant suite whenever you change a system or developer prompt, model, provider, temperature or tool configuration, retrieval source, memory design, parser, permission, or user-facing workflow. Repeat it before seasonal planning cycles or other periods when usage, data sources, and connected actions change.

Maintain a small, high-value regression set and a larger exploratory set. The regression set should cover direct injection, indirect injection, data disclosure, tool misuse, multi-turn manipulation, and output validation. Exploratory red teaming can then search for new variants without weakening the repeatable baseline.

After each run, record the failure, affected boundary, business impact, immediate containment, permanent fix, and regression test added. Recheck after major model or agent-platform releases because behaviour, tool formatting, context handling, and refusal patterns may change. A practical final sign-off is:

1. List every tool and its maximum permitted impact.
2. Run direct, indirect, multi-turn, and tool-use cases in an isolated environment.
3. Inspect responses, tool arguments, state changes, logs, and data exposure.
4. Confirm that application-side authorization and validation hold even when the model is manipulated.
5. Block release for unresolved high-impact paths, then add each discovered case to the regression suite.

Use this checklist again after the next workflow or tool change. Security improves when prompt engineering, AI development, and conventional application controls are reviewed together.