Structured Output

Why asking for JSON in the prompt is not enough

Write "respond with JSON" in a prompt and you will get JSON almost every time. Almost is the problem. A prompt states a preference, and the model weighs it against everything else in the context. At a thousand calls a day, a one percent failure rate is ten parse errors, and they arrive as a stack trace rather than a handled case.

The failures are rarely dramatic. A code fence wrapped around the object. A trailing comment explaining the answer. A field the model renamed because the value felt like a better fit for a different key. Each one is valid text and useless to your parser.

Three levels of guarantee, and what each one promises

Providers offer three distinct things, and the names blur together.

Asking in the prompt gets you a preference. Useful when a human reads the output and nothing downstream depends on the shape.

JSON mode guarantees syntactically valid JSON. It does not guarantee your JSON. The model can still omit a required field or invent an enum value, and what you get back parses cleanly into the wrong object. OpenAI draws the line itself: "only Structured Outputs ensure schema adherence."

Schema enforcement constrains the generation against a schema you supply, so the response has to match. OpenAI calls this Structured Outputs. Anthropic reaches the same place with strict: true on a tool definition, which makes calls "always match your schema exactly."

That middle level is where people get caught. They enable JSON mode, watch the parse errors disappear, and conclude the problem is solved. Then a field goes missing under load and the bug surfaces somewhere far from the model call.

Designing a schema the model can satisfy

Keep the structure flat. Deep nesting gives the model more places to make a structural decision, and each one is a chance to produce something your code did not expect.

Mark every field you depend on as required, and set additionalProperties to false so you get the object you asked for rather than a superset. Use enums wherever the value comes from a known set. "status": "refund_approved" chosen from three options is a much smaller target than a free string you have to normalize afterward.

Name fields the way you would name them for a colleague. The model reads your schema, and confidence_0_to_1 tells it more than conf.

Some JSON Schema features are unsupported, for performance and technical reasons. Check the current list in your provider's documentation rather than discovering it through a rejected request.

What a schema does not buy you

Schema enforcement is a guarantee about form. It says nothing about whether the values are true.

A response can satisfy every constraint and still be wrong. "refund_amount": 450 is a valid number in a required field, and it can still be a number the model invented from a policy it never read. Your validation does not end when the object parses. Range checks, referential checks against your own records, and the authorization check all still belong in your code.

This is the same boundary that function calling runs into from the other direction. Structured output shapes the answer; function calling requests an action. Both hand you well-formed data that you still have to verify.

Handling refusals as a separate case

When a model declines on safety grounds, a schema creates an awkward problem: there is nowhere in your object for "I won't do that."

OpenAI handles it by putting refusals in a separate refusal field rather than forcing them into your shape, which makes them detectable in code. Check that field before you parse. A client that treats a refusal as a malformed response will retry it, fail the same way, and bill you twice for the privilege.

Further reading

• OpenAI, Structured outputs: the JSON mode distinction, schema support, and refusal handling.
• Anthropic, Tool use: the same guarantee under strict.
• Streaming responses: why you cannot validate a schema you have not finished receiving.