Jailbreaks and Guardrails
Why safety training is a tendency rather than a rule, and how to build the enforcement layer that does hold.
On this page
A model’s refusal behaviour comes from preference training. Human or AI feedback taught it to decline certain requests.
That means refusals are learned tendencies, not rules. No rule engine is consulted. There is no lookup table of forbidden topics. The model has a trained inclination that generalizes imperfectly, which is exactly why it can be circumvented and why it sometimes refuses things it should not.
Understanding this shapes what to build: model-level safety is a probabilistic layer, and anything you actually need enforced belongs in your code.
Why circumvention works at all
Safety training covers the distribution it was trained on. Requests far from that distribution generalize unpredictably.
Broad categories of technique, described at the level that helps you defend rather than execute: reframing a request as fiction or hypothetical, wrapping it in a persona, decomposing it into individually innocuous steps, obfuscating through encoding or another language, and exploiting the instruction-following bias that rewards answering over declining.
None of these exploit a bug. They exploit the gap between a trained tendency and a rule — a gap that closes with better training and does not disappear.
New techniques keep appearing because the surface is the entire space of ways to phrase a request. Treat provider safety as raising effort, not as a boundary.
The layer that holds
Guardrails you implement are enforcement. Guardrails the model provides are inclination. Build the first for anything that matters.
Input checks, before the model call. Length bounds, format validation, and rate limits per user. These are cheap and they close resource-exhaustion and abuse vectors that no amount of model behaviour addresses.
Output checks, after. Schema validation, pattern matching for sensitive data, and blocking on required-field failures. An output that cannot pass validation cannot reach the user.
A separate classifier. A small fast model examining input, output, or both — outside the main call, so it cannot be talked out of its job by the same prompt. This is the structural advantage: an independent check is not subject to the manipulation that reached the primary model.
Permission enforcement in your code. Whatever the model produces, actions execute against the authenticated user’s permissions. This is the defense that genuinely holds, and it is worth more than everything above it.
Human review on consequential actions. See Human-in-the-Loop Design.
Guardrails have costs
Every layer adds latency, cost, and false positives — and false positives are the ones that damage the product.
Over-blocking is a real failure mode, not a safe default. A medical application refusing to discuss symptoms, or a security tool refusing to explain a vulnerability class, is broken. Refusing legitimate use has a cost that is easy to overlook because it does not look like a security incident.
Measure both directions. Track the false-positive rate as carefully as the false-negative rate, and treat a high refusal rate on legitimate traffic as a defect to fix.
Scoping the problem down
The most effective safety measure is usually narrow scope.
A system that answers questions about your product documentation, using RAG with instructions to answer only from retrieved context, has a small attack surface. It has no tools, no external actions, and no reason to discuss anything else. Off-topic requests retrieve nothing relevant and produce nothing useful.
A general-purpose assistant with broad capability has a large surface by construction. If you do not need general capability, not building it is the strongest guardrail available — and it costs nothing.
Practical shape
For most applications: bound and rate-limit input, validate output against a schema, enforce user permissions in code, gate irreversible actions, keep scope narrow, log everything.
Add a classifier layer when you have measured a specific problem it addresses. Do not start there — the cheaper structural measures cover more ground.
And test adversarially before shipping. See Red-Teaming AI Systems.
What to remember
- Refusals are trained tendencies, not rules — which is why they generalize imperfectly in both directions.
- Circumvention exploits the gap between tendency and rule; provider safety raises effort rather than forming a boundary.
- Your guardrails are enforcement: input bounds, output validation, an independent classifier, and permission checks in code.
- Permission enforcement on the authenticated user is the defense that actually holds.
- Over-blocking is a real defect — measure false positives as carefully as false negatives.
- Narrow scope is the cheapest and strongest guardrail available.