Getting JSON Out of an LLM

Asking politely for JSON gets you JSON most of the time. Most of the time is not good enough for code, and there are ways to make it guaranteed.

The moment you put a model inside a program, you need machine-readable output. Prose is fine for humans and useless to a parser.

Asking for JSON works most of the time. The failure rate is what matters, because a 2% parse failure across ten thousand calls is two hundred incidents.

Why it fails at all

The model is predicting tokens, not filling a template. Valid JSON is a highly probable continuation after a JSON-shaped instruction, but nothing in the mechanism enforces it.

So the characteristic failures are exactly what token prediction would produce:

• Prose wrapped around the JSON — “Here’s the data you requested:” before the object
• Markdown code fences around it
• A trailing comma, or a missing closing brace
• A field name that drifted from user_id to userId
• An enum value slightly off — positive where you specified POSITIVE
• A single quote where JSON requires double
• Truncation, when the object ran past the token limit

Four approaches, weakest to strongest

1 · Ask, and show

Specify the schema in the prompt and include an example. Few-shot examples are unusually effective here because format is exactly what examples communicate best.

Return only a JSON object matching this shape, with no other text:

{"sentiment": "POSITIVE" | "NEGATIVE" | "NEUTRAL", "confidence": 0.0-1.0}

Example input: "This is fine I guess"
Example output: {"sentiment": "NEUTRAL", "confidence": 0.6}

Cheap, works with any model, and still fails sometimes.

2 · JSON mode

Many providers offer a flag that constrains output to syntactically valid JSON. This eliminates parse errors — but only syntax. You can still get valid JSON with the wrong fields, so the schema still needs stating in the prompt.

3 · Schema-constrained generation

The strongest available option. Supply an actual schema, and the provider constrains generation so only tokens permitted by that schema can be sampled.

This works by masking the probability distribution at each step: if the schema says the next thing must be "sentiment", tokens that would violate that get zeroed out before sampling. Malformed output becomes structurally impossible rather than merely unlikely.

Various names — structured outputs, guided decoding, grammar-constrained decoding — and it is the right default whenever a provider offers it.

4 · Tool definitions

If you are already using tool calling, tool arguments are schema-validated by construction. Defining a “tool” purely to receive structured data is a legitimate pattern, and the schema enforcement comes free.

Designing the schema

Schema shape affects reliability, not just convenience.

Flat beats nested. Deeply nested objects fail more often. Two flat calls usually outperform one deeply nested one.

Enums beat free text. "status": "APPROVED" | "REJECTED" is far more reliable than asking for a status string.

Short field names. Every field name is tokens, repeated on every call.

Give uncertainty somewhere to go. Without a null option or an UNKNOWN enum value, the model will invent a value rather than leave a required field empty — the same commitment pressure that drives invention elsewhere.

Reasoning before data, if you need reasoning. Put a reasoning field first in the object. Fields generated after the answer cannot influence it, so {"answer": ..., "reasoning": ...} produces rationalization while {"reasoning": ..., "answer": ...} produces actual step-by-step work.

Handling it in code

Even with constrained generation, defensive parsing is warranted:

• Validate against the schema after parsing. Syntactic validity is not semantic validity.
• Strip code fences before parsing. A cheap, high-yield defense.
• Retry once on parse failure, including the error in the retry prompt. Most failures are transient.
• Watch for truncation. An object cut off mid-generation means your output limit is too low, and no amount of retrying fixes that.
• Use temperature 0. Structured extraction wants determinism, not variety.

The cost of structure

Constraining output has a quality cost worth knowing. Forcing a model into a rigid shape can degrade the content inside that shape, particularly when the schema fights how the model would naturally organize the information.

If quality drops noticeably after adding constraints, the usual fix is a simpler schema or a two-step approach: generate freely, then extract into structure with a second cheap call.

What to remember

• JSON failures are token-prediction artifacts: fences, prose, drifted field names, truncation.
• Escalation: ask and show → JSON mode (syntax only) → schema-constrained generation (structurally guaranteed) → tool definitions.
• Prefer flat schemas, enums, short field names, and an explicit slot for uncertainty.
• Put a reasoning field before the answer field, or it does nothing.
• Validate after parsing anyway, and use temperature 0.

Next: Your Prompt Isn’t Working. Now What?