Content Moderation APIs

Why a classifier exists when models already refuse

Models decline harmful requests. So why run a separate check?

Because a refusal is a behavior and a classifier is a measurement. Refusals vary with phrasing, with context, and between model versions, and you cannot query them, log them consistently, or set a threshold on them. A classifier returns the same categories and the same scores for the same input, every time.

The other reason is position. A model refuses at generation time, which is after you have paid for the call and after user content is already in your system. A classifier runs wherever you put it, including before you call a model at all.

What the endpoint returns

OpenAI's moderation endpoint returns three things:

• flagged, a boolean summary.
• categories, per-category booleans.
• category_scores, confidence from 0 to 1 per category.

It accepts text and images up to 20 MB, though some categories such as harassment and hate apply to text only. It is free, which removes the usual argument for checking one side and not the other.

Read the per-category scores rather than flagged, which is somebody else's threshold applied to your product.

Input and output moderation solve different problems

Checking the input tells you what your users are sending. You can refuse before spending a model call, and you learn what is arriving.

Checking the output tells you what your system is about to say. This matters because your model is not the only author of its output. Retrieved documents, tool results, and user text all shape what comes back, so an output check catches what a prompt instruction did not.

Most systems want both, at different thresholds. An input check that is slightly too strict inconveniences a user who can rephrase. An output check that is too loose publishes something you cannot take back.

Thresholds are a product decision

The documentation is explicit about how to treat the numbers:

Treat moderation scores as signals for your application's policy, not as an > automatic blocking decision.

The reason is specific and easy to miss. A safety-conscious refusal that discusses harmful content can itself be flagged. So a rule that blocks anything flagged will suppress your own model declining to help, which is the correct behavior being punished by the guardrail.

Choosing a threshold means deciding what a false positive costs against what a false negative costs, and those differ by category and by product. A support tool for a bank and a creative writing assistant should not use the same numbers.

One maintenance note the documentation adds: custom policies that rely on category_scores "may need recalibration over time" as the underlying model changes. A threshold is not a constant you set once.

What to do on a positive result

Blocking is one option among several, and rarely the best one alone.

Block and explain. Appropriate for high-confidence, high-severity categories. Tell the user what happened, because silent failure reads as a broken product.

Route to review. For scores in the uncertain middle, hold the response and put it in front of a person. This is the option people skip, and it is the one that handles ambiguity honestly.

Flag and log. Let it through and record it. Useful when false positives cost more than the content does, and it gives you the data to set thresholds from evidence rather than from guessing.

Constrain rather than refuse. Sometimes the right response is a narrower answer instead of no answer.

Log every decision either way, with the scores. Without that record you cannot tune thresholds, and you cannot answer a user who asks why they were blocked.

What a moderation classifier is not for

The documentation prohibits sending known or suspected child sexual abuse material to the API, and states it is not a substitute for dedicated child-safety safeguards. If your product has that exposure, it needs a purpose-built path and a reporting obligation, not a classifier call.

More generally, a moderation endpoint is one control among several. It does not replace rate limits, authentication, permission checks in code, or adversarial testing of the system you built.

Further reading

• OpenAI, Moderation: the response shape, the guidance on thresholds, and the stated limits.
• Deterministic evals: a classifier is a computed check with a threshold.
• Conducting adversarial testing: finding what gets past it.