Stop Running JSON.parse Naked: Validating and Repairing AI Structured Output

This article is about one thing only: the engineering pipeline that turns AI structured output into reliably usable JSON. Boundaries first:

• vs the AI evals guide: that one is "how to evaluate model output quality"; this one is "how to make output stably become usable JSON" — production reliability, not evaluation methodology.
• vs the streaming output UX guide: that one covers typewriter effects and the presentation layer; this one covers the structural reliability of output content.
• vs the agent failure modes guide: that one is a panorama of failure modes; this one digs deep into exactly one failure — "malformed format" — and its engineering fix.

You've lived this scenario: you have an agent call a model to "extract structured info from user reviews," the prompt clearly says "return JSON only," week one is fine, then at 3 AM JSON.parse explodes — the model prepended "Sure, here are the extracted results:" before the JSON. Your service 500s, users complain, and all you want to ask is: I said JSON only, why didn't it listen?

The answer: a prompt is a suggestion, not a constraint. For structural reliability you need four defensive layers.

The four layers at a glance

Layer	What it catches	Cost
1. Native model structured output	JSON syntax errors, missing keys, hallucinated enums	Low (a parameter)
2. Schema validation (zod)	Semantic errors: right type, absurd value, missing required	Low (write a schema)
3. Repair retry	Occasional format/semantic glitches	Medium (one more call)
4. Degraded fallback	Consecutive failures	Decided by business

Remember the order: each layer catches only what it's supposed to, no overreach. Layer 1 ensures "looks like JSON," layer 2 ensures "is the right JSON," layer 3 handles "occasional flakiness," layer 4 handles "total meltdown." Most projects' bug is building only layer 1 (or worse, only "JSON only in the prompt") and then staring at the 3 AM 500.

Layer 1: native model structured output (and what it can't promise)

Mainstream models in 2026 all have native structured output, with different APIs (verified against official docs, 2026-10-17):

• OpenAI: in the Responses API, text.format = { type: "json_schema", strict: true, schema }, or client.responses.parse() taking a zod/pydantic schema directly. Note the docs' explicit line: JSON mode only guarantees "valid JSON"; only Structured Outputs guarantees "schema adherence." And OpenAI admits: on safety refusals or max_tokens truncation, the response can still violate the schema — structured output isn't a silver bullet, their own words.
• Anthropic: output_config: { format: { type: "json_schema", schema } } on messages.create() (the old output_format param is deprecated). Note newer models (Fable 5.1+) removed forced tool use tool_choice: {type: "tool"} — for JSON, use structured outputs or add strict: true to the tool.

Layer 1 guarantees the syntactic level: matched brackets, closed quotes, enum values in range. It cannot guarantee the semantic level: "price": "expensive" (a string where you expected a number), "date": "tomorrow" (a legal string where downstream wants ISO dates). Semantics need layer 2.

Layer 2: zod validation — strict vs lenient is a tradeoff

import { z } from "zod";

const ReviewExtraction = z.object({
  sentiment: z.enum(["positive", "negative", "neutral"]),
  // amounts: the model may return the string "99.9" or the number 99.9
  amount: z.coerce.number().min(0),
  // dates: accept a string first, then semantic-check with refine
  date: z.string().refine((s) => !isNaN(Date.parse(s)), {
    message: "date must be a parseable date",
  }),
  tags: z.array(z.string()).max(5).default([]),
});

// The key tradeoff: .strict() vs .passthrough()
// .strict(): extra fields are errors — when you fully own the schema
// .passthrough(): extra fields pass through — when the model occasionally
// improvises and you'd rather not explode
const StrictExtraction = ReviewExtraction.strict();

Two field lessons:

1. Use z.coerce against the model's "casual typing": models writing numbers as strings is routine; z.coerce.number() survives production an order of magnitude better than z.number(). But coerce isn't magic — "expensive" coerces to NaN and still errors, which is exactly what you want: absurd values should die at the validation layer, not flow downstream.
2. Error messages are layer 3's ammunition: don't just log zod's errors (e.g., Expected number, received string at "amount") — feed them verbatim into the retry prompt. A model that sees where it went wrong repairs far better than one that only hears "wrong format, retry."

Layer 3: repair retry — stitch the raw error into the prompt

async function extractWithRepair(input: string, maxRetries = 2) {
  let lastError = "";
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    const raw = await callModel(buildPrompt(input, lastError));
    const parsed = ReviewExtraction.safeParse(tryParseJson(raw));
    if (parsed.success) return parsed.data;
    // Key: feed the raw zod error back, not just "wrong, retry"
    lastError = parsed.error.issues
      .map((i) => `${i.path.join(".")}: ${i.message}`)
      .join("; ");
    // Cost fuse: retry budget
    if (attempt === maxRetries) break;
  }
  return fallback(input); // layer 4
}

function tryParseJson(raw: string) {
  // handle the classic "a sentence before the JSON" case:
  // extract from the first { to the last }
  const m = raw.match(/\{[\s\S]*\}/);
  return JSON.parse(m ? m[0] : raw); // may still throw; safeParse catches it
}

Three points on retry design:

• Cap retries at 2: a rule of thumb. Retrying once with the error attached recovers significantly; failing twice usually means the prompt or schema itself is broken — more tries just burn money. Make maxRetries a config, not a constant.
• Every retry must bring new information, not new luck: blindly retrying the same prompt burns cash. Effective retries carry new information — the raw zod error, the failed output, even "you got the amount field wrong last time."
• Put the retry budget on the dashboard: track "repair-retry trigger rate." If it jumps from 2% to 20% one day, upstream model behavior changed (new version, new defaults) — go look instead of raising the retry count.

Layer 4: degraded fallback — when consecutive failures hit

When all three layers fail (model flakiness, upstream API outage), you need a graceful way to lose. Three options by business tolerance:

Fallback	What it does	When
Human review	Push the raw input to a review queue; UI shows "processing"	Finance, healthcare — where wrong costs more than slow
Safe defaults	Return schema-safe defaults (e.g., sentiment: "neutral", tags: [])	Recommendations, tagging — where wrong is acceptable
Skip	Skip the sample in the pipeline, log it, backfill later	Batch processing — one failure doesn't sink the batch

The only selection criterion: which costs more, "wrong" or "missing." Human review costs people, defaults cost error propagation, skipping costs data gaps. Do that math and the fallback stops being a guess. And: fallback triggers must alert — fallback is an airbag, not a seat; every deployment gets a look.

Observability: failure metrics and (redacted) failure samples

Four layers built, one piece missing: how do you know it's working? Two metrics, one habit:

1. Structural failure rate: zod validation failures / total calls, trended daily. The health dashboard of the whole pipeline — a spike means upstream changed.
2. Repair-retry trigger rate: retries triggered / total calls. Healthy is usually <5%; a sustained climb means layer 1's (native structured output) config or model version needs review.
3. Retain failure samples (redacted): keep "whatever monstrosity the model actually returned" — redacted first, never store raw user input. These samples are gold for prompt iteration and a ready-made bad-case set for future AI evals.

Solo launch checklist

• ☐ Layer 1: native structured output enabled (OpenAI: structured outputs, not JSON mode; Anthropic: output_config.format)
• ☐ Layer 2: zod schema written, .strict() vs .passthrough() chosen per scenario, z.coerce on casually-typed fields
• ☐ Layer 3: retries feed the raw zod error back into the prompt, maxRetries ≤ 2 as a config item
• ☐ Layer 4: fallback chosen by "wrong vs missing, which costs more," with alerting on trigger
• ☐ Observability: structural failure rate and retry trigger rate on the dashboard; redacted failure samples retained
• ☐ Schema versioning (schema changes don't silently desync old data/old prompts)

Remember the chain: prompts are suggestions, parameters are constraints, validation is law, fallback is insurance. Stop letting JSON.parse run naked at 3 AM.