The JSON inspector: catching schema drift before your code does

Plausible JSON is not valid JSON

Ask a model for a list of products with prices and it will produce something that looks like the schema you described. The shape is right, the keys are named sensibly, the nesting is plausible. Then a consumer reads quantity and gets the string two, or reads an optional field and finds a nested object where a list belongs.

These failures are invisible in a chat transcript, because a transcript shows text and text has no types. They are also invisible in code review, because the call site assumes the contract, and the contract was a sentence in a prompt. The failure surfaces in production, in a downstream job, at the worst possible moment.

The inspector makes the difference visible. It parses the payload, walks the tree, and compares what arrived with the schema you pasted beside it: missing required fields, wrong types on declared fields, and keys the schema never allowed.

There is a second, quieter use: pasting an output you do not have a schema for, purely to see its shape. Depth, key count and structure tell you how much of your prompt actually landed, and whether the model padded the response with fields you never asked for.

What gets checked, and why that subset

Parsing comes first, because everything else depends on it. Malformed JSON is reported with the parser's own message, which usually names the position of the problem, and that is often the whole finding: a trailing comma, a smart quote where a straight one belongs, an unescaped newline inside a string.

Then types. Every declared property is checked against its type, including the distinction between integer and number. That distinction matters in a system that stores currency as an integer count of cents and silently accepts a float, and it is the one type error most often produced by models that reason in decimal.

Then required fields. A required array that is missing because the model had nothing to put in it is the most dangerous kind of absence, because the payload is otherwise complete and the consumer has no reason to check.

Then keys. With additionalProperties set to false, any field the schema never declared is reported, which catches the subtitle the model invented, the extra metadata block it decided was helpful, and the renamed key that changed spelling between runs.

• Parse errors, reported with a position.
• Missing required fields, per object.
• Wrong types, including integer versus number.
• Undeclared keys, when the schema forbids them.

Writing a schema that actually constrains

A schema that says an object has properties but names no required ones constrains almost nothing: any object passes, including an empty one. The minimum useful schema names its required fields, types every property it depends on, and forbids additional keys at the levels where an invented field would break something.

Nest it only as deep as your consumer reads. A schema three levels down is harder to maintain than the code it protects, and models are more likely to flatten a deep structure than to produce it correctly under pressure.

Keep the schema in the same repository as the code that consumes the payload. A schema that lives in a prompt or a wiki drifts the moment someone changes a field, and a drifted schema produces findings that make no sense to the person reading them.

And version it. When a validation fails three weeks after a deploy, the only question worth answering is whether the output changed or the contract did.

Same prompt, three runs

A single valid payload proves the schema fits one output. It says nothing about the distribution, and the distribution is what production sees. Run the same prompt three times and paste each result: if one run omits a required field and another returns a number as a string, the contract is being satisfied by luck.

That exercise takes about two minutes and it changes how you write the prompt. A constraint the model violates intermittently was never in the prompt, or was phrased as a suggestion. Adding it explicitly, with the consequence named, is more reliable than adding another validator downstream.

Keep the three payloads when you are done. They become a small regression suite: when the model or the prompt changes, run them again and see whether the failure you fixed has returned. Three files are enough to catch the common drift, and they cost nothing to keep.

The failures cluster by field, which is useful. If everything fails on the same optional array, the prompt described it as optional in a way the model read as unnecessary. If everything fails on types, the prompt never named the types and the model inferred them from examples.

Reading a schema you inherited

Schemas outlive the people who wrote them, and an inherited one has usually drifted from the code that consumes it. Read the consumer first, then the schema, and treat every difference as a question: which of these two is the contract?

Look for the parts that constrain nothing. A required array with one string type and no bounds, a properties block with no required list, an object with additionalProperties left open. Those are the fields where a model's invention will land, and tightening them is usually a one-line change with a large effect.

Check whether the schema's names match the language the team uses. A schema that calls something items while everyone says line_items will be misread by the next person who edits it, and misreading a contract is how the contract stops being true.

Then validate three real payloads against it and see what passes. If a payload that obviously breaks the intent of the schema validates cleanly, the schema is documentation rather than a constraint, and fixing it is the cheapest reliability work available.

Validating a stream, not a sample

One payload is a sample of one. A pipeline that processes a thousand records a day needs a contract that holds across the distribution, and the failures that matter are the ones that appear in one run in a hundred, because those are the ones that reach a customer without anybody noticing.

Take ten outputs from the same prompt and paste each one. Count the failures per field rather than per payload: if four of ten outputs miss the same required array, that is a one-line prompt fix worth more than ten validation rules downstream.

Do the same after any change to the model, the prompt or the schema. Version comparisons are where streams fail quietly: the new deployment returns the same fields with one different type, nothing throws, and a dashboard shows zero for a week.

And keep the contract in code rather than in a document. A schema that is imported by the producer and the consumer cannot drift without a build failure, which is the cheapest enforcement available.

Depth is a design decision

Every level of nesting is a level somebody has to handle, in code, in a template and in a review. Generated payloads tend to arrive deeper than the consumer needs because depth looks like thoroughness when you are generating.

Flatten where you can and name the remaining levels explicitly. A payload that is three levels deep and validated is still three levels deep every time somebody writes a template against it, and most of those templates will quietly assume the level is always present.

The exception is data that is genuinely nested: line items, variants, per-locale content. There, depth is the shape of the thing being described and flattening it produces a worse contract than the one you started with.

Schema notes worth writing down

Two sentences next to a schema prevent most of the confusion it will cause: what the payload is for, and which fields a consumer may rely on. Everything else can be read from the file, and these two cannot.

Record the units as well. A price field with no stated currency is a bug waiting for an international order, and an integer count of cents is invisible to everybody who did not write the schema.

Where structural validation stops

This checks shape, not meaning. Ranges, string formats, enumerations and cross-field rules are outside what is enforced here, and a schema-compliant payload can describe something entirely wrong: a price that is a valid number and a wrong price, a date that parses and is in the past when it should be future.

It also says nothing about whether the content is any good. A product description can satisfy a seventy-character limit and be nonsense; a summary can fit its token budget and misrepresent its source. Those are reading questions, and they belong to the claim tools and the console.

The practical role is to stop the cheap failures, so that human attention goes to the expensive ones. A pipeline that rejects malformed payloads automatically is a pipeline where nobody has to notice them, and where the interesting question is always about content.

For the content, take the payload into the schema output validator if you need range and pattern rules, or into the claim matrix if the payload is a set of facts. Structure first, then truth, in that order, because structure is the cheaper of the two to check.