Red-Teaming AI Systems

Adversarial testing of your own system before someone else does it. What to try, how to organize it, and what to do with findings.

Ordinary evaluation measures whether a system works when used as intended. Red-teaming measures what happens when it is not.

Both are necessary, and the second is routinely skipped because it requires deliberately trying to break your own work.

What to probe

Six categories, covering the failure modes that actually occur.

Injection through content. Plant instruction-shaped text in anything your system reads — a document in the RAG corpus, a web page, a file comment, a message body. Check whether it gets followed. This is the highest-value test for any system with tools. See Prompt Injection.

Access control. Ask for data belonging to another user, another tenant, another permission level. Verify that filtering happens at retrieval and that permissions are checked against the authenticated user rather than the request. Test with a real low-privilege account, not an admin one.

Data leakage. Attempt to extract the system prompt, other users’ content, or details from the corpus that this user should not see. Assume the system prompt is extractable and confirm nothing sensitive is in it.

Boundary behaviour. Requests just outside intended scope. Does a documentation assistant give legal advice? Does a coding tool discuss unrelated topics? Both over-permissiveness and over-refusal are findings.

Resource abuse. Enormous inputs, requests designed to maximize output length, rapid repeated calls, prompts engineered to trigger long reasoning. Confirm bounds and rate limits hold.

Agent-specific failures. For anything with tools: impossible tasks, ambiguous tasks, tool failures, and whether irreversible actions can be reached without confirmation. See How Agents Fail.

How to run it

Write down what you are testing for. An unstructured session finds scattered problems and misses categories. A checklist by the six areas above is more thorough and repeatable.

Use accounts with real permission levels. Testing access control as an admin proves nothing.

Try repeatedly. Systems are nondeterministic. An attempt that fails once may succeed on the third try, and a single failure to reproduce is not evidence of safety.

Involve people who did not build it. Builders test what they anticipated. Fresh testers find the assumptions.

Automate what you can. Turn every finding into a test case. A model can generate variations on a successful technique to check whether your fix covers the class or only the instance — that distinction is the important one.

Turning findings into fixes

Categorize by what the fix actually is, because most findings are not prompt problems.

Architectural. Injection reaching a destructive tool means the tool should not be reachable from untrusted content. Fix the topology, not the prompt.

Enforcement. Access control failures belong in code — a permission check that runs regardless of what the model produced.

Validation. Malformed or unsafe output means an output check is missing.

Bounds. Resource abuse means limits.

Prompt. Some findings genuinely are prompt problems: unclear scope, missing permission to decline. The smallest category, and the one people reach for first.

A rough test for whether a fix is real: if a determined person can rephrase their way around it, it was a prompt fix and should have been architectural.

What red-teaming cannot do

It cannot prove safety. Finding no problems means your techniques found no problems.

Injection cannot be fully closed. For systems reading untrusted content, red-teaming establishes what an attacker gains, not whether they can get in. Design so that the answer to the first question is “not much.”

Techniques evolve. A test suite from six months ago misses recent approaches. Re-run periodically, and after model updates — provider changes shift behaviour without version changes.

Where to start

If you do one thing: put instruction-shaped text into your RAG corpus and see whether the system follows it. For any system with retrieval and tools, this single test finds the most consequential class of problem.

Then check access control with a genuinely low-privilege account. Those two cover most of what actually goes wrong.

What to remember

• Red-teaming tests unintended use; ordinary evals test intended use. Both are required.
• Six areas: content injection, access control, data leakage, boundary behaviour, resource abuse, agent failures.
• Use real low-privilege accounts, try repeatedly, and involve people who did not build it.
• Most findings need architectural or enforcement fixes, not prompt changes — if rephrasing defeats your fix, it was the wrong fix.
• It cannot prove safety, and techniques evolve — re-run after model updates.
• Start with injection into your own corpus, then access control.

Next: Model Supply Chain Risk