Getting Reliable JSON Out of a Model

If your app parses a model's output rather than showing it to a human, you have a reliability problem that a prompt alone won't solve. There are two distinct failure modes, and most advice only addresses the first: output that isn't valid JSON, and output that parses perfectly but is wrong — a missing field, a string where you expected a number, an invented enum value. Structured-output modes have largely fixed the first. The second is yours. The design that works is validate in code, feed the validation error back once, and cap the attempts — because retries are the classic way an AI feature quietly becomes expensive.

Use the provider's structured mode, then don't trust it#

Most providers now offer a JSON or schema-constrained mode. Turn it on — it removes a whole class of problem: prose before the JSON, trailing commentary, markdown fences around the object.

What it doesn't guarantee is that the content makes sense. A schema can say priority is a string; it can't say the string should be one of three values you handle, or that end_date should be after start_date, or that the summary shouldn't be empty.

So the model's output is input to your program, and it gets treated like any other untrusted input.

Validate in code, not by reading#

Parse into a real type with a validating library — Pydantic, zod, whatever your stack has — rather than reaching into a dictionary and hoping.

class Task(BaseModel):
    title: str = Field(min_length=1, max_length=200)
    priority: Literal["low", "medium", "high"]
    due: date | None = None

Two things this buys beyond catching errors:

• Enums instead of free text. Literal[...] makes a whole category of wrong answer impossible to accept, and describing the allowed values in the schema also makes them less likely.
• A precise error message, which is the input to the retry below.

Validate the semantics too, where they matter — dates in order, totals that add up, IDs that exist in your database. A model will confidently reference a record that isn't there.

The retry: once, with the error#

When validation fails, the useful move is to send the model its own output and the specific complaint. This is the same principle as writing good tool errors: the error message becomes the next instruction, so it should say what was wrong and what would be right.

Your previous response failed validation: 'priority' must be one of
low, medium, high — got "urgent". Return corrected JSON only.

That usually works on the first retry, because the problem is specific and local.

Three rules to put around it:

• Cap the attempts — two is plenty. If a second, error-corrected attempt fails, a third rarely helps and you're paying for each one.
• Make the cap configurable as an environment variable, so you can change it without a deploy.
• Never retry in an unbounded loop. This is the pattern that turns a bad afternoon into a bill — and combined with an uncapped max_tokens, it's how people discover their spend limit was theoretical.

Retries are a container problem too#

Two platform-shaped details that bite specifically here.

Each retry is latency. A validate-fail-retry cycle can double or triple your response time, which matters if you're inside a request or a three-second interaction deadline. Acknowledge first, then work.

An in-flight retry sequence doesn't survive a restart. The container is replaced on every deploy and configuration change, with about a second's notice, so a multi-attempt sequence in progress simply stops. If the work matters, record the request before you start it so a restart is a retry rather than a loss.

Ask for simpler shapes#

The reliability of structured output is inversely related to how complicated the structure is. Before adding retry logic, try removing schema:

• Flat beats nested. Deeply nested objects fail more often and are harder to validate meaningfully.
• Fewer fields. Every optional field is a chance to omit or invent.
• Enums over free text, everywhere the set is known.
• One decision per call. Two simple extractions are often cheaper and more reliable than one call producing a complex document — and you can validate and retry them independently.

This is the same lesson as designing tools for a model: constrain the shape so wrong answers are hard to express.

Log the failures, carefully#

You want to know which validations fail, because the pattern tells you whether to fix the prompt or the schema.

Log the field that failed, the value it got, and a truncated copy of the raw output. Not the whole response — logs are capped and rotate, and a few full model responses will push out the things you needed.

If failures cluster on one field, that field's description is usually the problem, not the model.

Degrade instead of erroring#

The last question is what a user sees when both attempts fail.

For anything user-facing, a partial result usually beats an error. If four of five fields validated, use those and mark the fifth unknown. If nothing validated, say the request couldn't be completed and let them retry — rather than a 500 that implies your app is broken.

The exception is anything that writes or acts. Never act on partially-valid output for a destructive operation; fail closed there.

FAQ#

How do I stop a model returning malformed JSON?#

Use the provider's structured-output or JSON mode. That handles syntax; content still needs validating in your code.

Why validate if the schema is enforced?#

Schema enforcement covers types and shape, not meaning. Invalid enum values, contradictory dates and references to records that don't exist all pass.

How many times should I retry?#

Once or twice, with the validation error included. A third attempt rarely helps and every attempt is billed.

What should the retry message say?#

Exactly which field failed and what a valid value looks like. A precise complaint gets a corrected answer; "that was wrong" doesn't.

What happens if my app restarts mid-retry?#

The sequence is lost — the container is replaced with about a second's notice. Persist the request first if losing it matters.

Should I show partial results?#

For read-only output, usually yes. For anything that writes or takes an action, fail closed instead.

Wrapping up#

Treating model output as untrusted input is the whole idea, and everything else follows from it: a real type to parse into, a specific complaint when it doesn't fit, one corrective attempt, and a bounded cost. Simplify the shape before you add machinery — most structured-output failures are a schema asking for more than it needs. And keep the retry count small and configurable, because "just try again" is the single easiest way to turn a small feature into an expensive one.