Content Moderation
Filtering what goes in and what comes out. Where classifiers belong, and why over-blocking is as real a failure as under-blocking.
On this page
Two separate questions get bundled under moderation: should this request be processed, and should this response be delivered.
They need different checks at different points, and conflating them produces systems that block the wrong things.
Input moderation
Runs before the model call. Cheap, and it saves the cost of generating something you will discard.
What it catches well: clearly abusive requests, obvious attempts at prohibited content, and traffic patterns indicating automated abuse.
What it catches badly: anything requiring context. “How do I get rid of my neighbour’s cat” reads differently in a gardening forum and elsewhere, and a classifier sees only the string.
The important limitation is that input moderation cannot see what the model will produce. A benign-looking request can produce a problematic response, and vice versa — a request mentioning a sensitive topic frequently deserves a perfectly good answer.
Output moderation
Runs after generation, before delivery. More expensive — you already paid for the tokens — and better positioned, because it examines what would actually reach the user.
This is where the checks that matter belong:
Sensitive data. Pattern matching for identifiers, credentials, or PII that should not appear. Mechanical and reliable.
Schema conformance. Structured output that fails validation should never be delivered.
Content categories. A classifier over the generated text.
Grounding. For RAG, whether claims are supported by retrieved context. Catches hallucination rather than harm, and it is often the more valuable check.
Where the classifier should live
A separate model call, not an instruction inside the main prompt.
The reason is structural: an independent check cannot be talked out of its job by the same prompt that reached the primary model. If you ask one model to both answer and police itself, a request that manipulates the answer manipulates the policing too.
Provider moderation endpoints exist and are cheap. Small open classifiers work locally. A general model with a specific rubric works and costs more.
Over-blocking is a defect
Worth stating as strongly as under-blocking, because it is treated as the safe direction and is not.
A medical application that refuses to discuss symptoms is broken. A security tool that will not explain a vulnerability class is broken. A support system that declines a frustrated customer’s complaint because it detected hostility is broken.
False positives are invisible in incident reports and highly visible to users. Measure them: sample legitimate traffic, count what got blocked, and treat a high refusal rate on real requests as a bug with an owner.
The base rate argument matters here. If one request in ten thousand is genuinely abusive, a classifier with a 1% false-positive rate blocks a hundred legitimate requests for every real one caught. Precision at your actual base rate is the number that matters, not accuracy on a balanced test set.
Handling a block
Say something specific. “I can’t help with that” tells a user nothing. Naming the category, where you safely can, lets them rephrase a legitimate request.
Distinguish decline from failure. A user should be able to tell a policy decision from an outage.
Provide an appeal path for anything consequential — account actions especially.
Log blocks with the input. Without this you cannot measure false positives, and false positives are the thing you most need to measure.
Scope beats filtering
The strongest moderation is a narrow system.
An assistant that answers questions about your product documentation from retrieved context has very little room to produce harmful content. It has no tools, no general capability, and off-topic requests retrieve nothing useful.
A general-purpose assistant needs real moderation because it can do anything. If you do not need general capability, not building it removes the problem rather than filtering it — and costs nothing.
What to remember
- Input moderation is cheap and context-blind; output moderation is better positioned because it sees what would reach the user.
- Put the classifier in a separate call — self-policing inside the main prompt fails to the same manipulation.
- Sensitive-data patterns, schema validation, and grounding checks belong on the output side.
- Over-blocking is a real defect: measure false positives against your actual base rate, not on balanced test data.
- Explain blocks specifically, log them with inputs, and provide appeal for consequential actions.
- Narrow scope removes more risk than filtering does.