Constraining Inputs and Outputs

What each kind of limit prevents

Constraints get grouped together as "safety" and then applied without anyone saying which failure each one addresses. The result is a system with three limits, two of which do nothing about the risk the team was worried about.

Sorting them by what they prevent:

Length limits prevent runaway cost and resource exhaustion. A cap on input size stops someone pasting a book and billing you for it. A cap on output stops a single request generating for as long as it can. Neither prevents a harmful answer, which can be short.

Allowlists and enums prevent invalid values reaching the systems behind you. A priority field that can only be low, normal or urgent cannot arrive as anything else. This is the strongest constraint available, and it only applies where the acceptable set is knowable.

Output schemas prevent parse failures and make downstream code safe to write. They say nothing about whether the values inside are true.

Moderation classifiers prevent categories of harmful content, and are a different mechanism from all of the above. Content moderation APIs covers them.

The useful habit is to name the failure before adding the constraint. If you cannot say what it prevents, it is probably decoration.

Closed sets beat filtered strings

Where the set of acceptable values is knowable, enumerate it.

This is the ordinary security principle, applied to a new surface. A denylist has to anticipate every bad value. An allowlist has to enumerate the good ones, and anything outside fails by default. The asymmetry is that new bad values appear constantly and new good ones require a deliberate change.

What makes this applicable here is how often the set is knowable. A refund reason, a ticket category, a target table, a document type, a severity level. These come from your own domain and you already have the list.

// The model can produce any string, and you find out downstream.
{ "category": { "type": "string", "description": "Ticket category" } }

// The model can produce one of four values, or the call fails.
{ "category": { "type": "string", "enum": ["billing", "shipping", "returns", "technical"] } }

The second is enforced by the schema. The first is a request, and structured output covers what happens to requests.

What a schema does not tell you about the value inside it

Here is the limit worth being precise about, because schema constraints are often oversold.

Anthropic documents strict: true for tool definitions, which guarantees calls "always match your schema exactly". The same documentation notes that when a request lacks enough information to fill a required parameter, a model "might also infer a reasonable value", and shows a weather call with a unit nobody specified.

Both are true simultaneously. The schema is satisfied and the value is invented.

So a schema gives you a shape you can parse and tells you nothing about truth. customer_id: "CUS-00042" is a valid string of the right format and may refer to nobody. refund_amount: 40 is a valid number and may have no relationship to what the customer paid.

Which means validation does not stop at the schema. Check that the customer exists. Check the refund against the order. The schema removes a class of parse errors, and the class of errors where a well-formed value is wrong is untouched and is the one that reaches production.

Length limits as a cost control

Caps on input and output are worth setting, and worth setting for the right reason.

Every limit you will hit is counted in tokens, as stop estimating tokens covers, and output length drives both cost and latency, as inference covers. An output cap is the thing standing between you and a single request that generates until it hits the model's ceiling.

Two details make caps behave better. Set the output cap deliberately rather than leaving a default, because a default that is too low truncates mid-sentence and returns as a success. And check the field that reports why generation stopped, so a truncated answer is handled as a failure rather than parsed as a complete one.

What a length cap does not do is make the content safe. A one-line answer can promise a refund you do not offer.

Where a constraint has to live to be one

A rule stated in a prompt is a request. A rule enforced in code is a rule.

This is the same conclusion reached from three different directions on this site: what is an AI engineer reaches it about refund policy, MCP server about tool results, and system prompting about what a system prompt can enforce. The consistency is not a coincidence. It follows from the fact that a model reads your instruction and the user's input as the same kind of thing, which is what makes prompt injection possible at all.

Practically: put the constraint at the boundary where it can be enforced. Validate the input before it reaches the model. Validate the output after it returns and before anything acts on it. The prompt is where you ask for cooperation, and cooperation is worth having, and it is not enforcement.

Further reading

• OWASP, Top 10 for LLM applications: the taxonomy of what these constraints are defending against.
• Anthropic, Tool use: strict schema conformance, and the inferred-value behaviour alongside it.
• Structured output: the three levels of guarantee, and what a schema does not buy.
• Content moderation APIs: the classifier tier these constraints sit beside.