Conducting Adversarial Testing

Why a passing evaluation set is not evidence of safety

Your evaluation set contains questions real users asked. It passes, and that tells you the system handles the inputs you expected.

Nobody in that set was trying to break it. The people who will try are motivated, patient, and willing to send things no reasonable user would send. Adversarial testing is the practice of being that person first, against a system you own.

The difference is the input distribution. An evaluation set samples typical use. Adversarial testing searches for the inputs where behavior falls apart, which means it is exploratory work rather than a fixed suite, and finding nothing means you did not look hard enough rather than that the system is safe.

Everything here applies to systems you are responsible for and authorized to test.

Deriving your attack surface from the taxonomy

Skip the generic jailbreak list. Most published prompts target a model, and you are testing a system: your retrieval, your tools, your permissions, your data.

The OWASP Top 10 for LLM Applications gives you a principled starting point. Walk the list and ask which entries your system exposes.

A read-only chatbot over public documentation has LLM01 and LLM09 and almost no LLM06. An agent with write access to billing has LLM06 as its largest risk and LLM05 close behind. The list differs per system, which is the point: you are deriving a test plan rather than copying one.

Writing attempts against your own surface

For each entry you have, write attempts that would matter if they worked.

Instructions in retrieved content. Put a document in your corpus containing an instruction aimed at the model, then ask a question that retrieves it. This is the case people underestimate, because the text never passes through a field you validate. If any user can add documents, this is your largest surface.

Arguments to a tool. Steer the conversation toward a tool call and see what arguments arrive. Can you get a refund amount larger than the charge, an identifier belonging to another account, a path outside the directory?

Erosion over a long conversation. System prompts hold well in a three-message test and less well after forty. Take the conversation somewhere gradually rather than asking directly.

Output that reaches an interpreter. If model output becomes HTML, a shell command, or a query, try to get syntax through it. This is LLM05, and it is ordinary injection with a new source.

Consumption. Ask for something that loops, recurses, or generates until a limit stops it. Find out whether a limit stops it.

Record every attempt and its outcome, including the ones that failed. Next quarter, on a new model version, you want to rerun them rather than remember them.

Turning findings into permanent checks

A finding you fixed and did not record is a finding you will meet again after the next model upgrade.

Every successful attempt becomes a case in your regression suite. Most convert into computed checks that cost nothing to run: the forbidden phrase never appears in output, the refund amount never exceeds the charge, the tool is never called with an account the session does not own. That is the deterministic tier, and it is where adversarial findings should land.

Run them on every prompt change and every model change. Model upgrades are the event that reopens closed findings, because the behavior you relied on was never a guarantee.

Fixing in the layer that can enforce it

Most real findings are not prompt problems.

If the model can be talked into requesting a refund outside your policy, adding "never issue refunds outside the window" to the prompt reduces how often it happens. It does not stop it, because the prompt competes with everything else in the context, including text an attacker controls.

The fix is a check in the code that executes the refund. Then the model can request whatever it likes and the answer is no.

The taxonomy points the same way. LLM05 and LLM06 both describe failures of what the surrounding system permits rather than of what the prompt said. Prompt hardening is worth doing and it is a layer of defense, not the boundary.

Ask of each finding: which component could have refused this? Fix it there.

Further reading

• OWASP, Top 10 for LLM Applications: the 2025 list, with each entry described in full.
• Simon Willison, the prompt injection series: why LLM01 stays unsolved.
• Deterministic evals: where findings become permanent.