Valid JSON is not a valid action: an eight-record experiment

The date had the right shape and did not exist

A record containing February 30 can be valid JSON. It can also satisfy a simple date-shaped pattern. The failure appears only when the application asks whether the date exists in the calendar. This is a useful example of why a structured AI response should not be treated as a completed action merely because a parser accepts it.

We created eight original synthetic records for a fictional entry task and ran three checks: JSON parsing, field-shape validation, and application-specific meaning checks. Seven parsed as JSON. Four passed the shape contract. Only one passed the final rules. The differences were intentional probes, not a sample of actual model outputs or a measure of a provider's reliability.

This directly executed lab makes three validation boundaries visible. Syntax asks whether the representation can be read. Shape asks whether the expected fields and types are present. Meaning asks whether the values satisfy the task's rules. Authorization and factual support remain additional questions even after all three pass.

Original TrendsWhat figure from the recorded local run. These counts describe a deliberately constructed challenge set, not a real-world model accuracy rate.

The fictional task contract

The task accepts exactly three fields: amount, currency, and date. Amount must be a finite number and later must be positive. Currency must have three uppercase letters and later must belong to the fictional application's USD/KRW allowlist. Date must first match a year-month-day pattern and later represent an actual calendar date.

These rules are not universal financial rules. A real application may allow refunds, additional currencies, or different representations of money. We chose a narrow positive-entry task so that each layer's responsibility is easy to see. Changing the domain rules changes the correct outcome for some fixtures.

The exact-key requirement also matters. One fixture adds an approved field. The fictional contract rejects it at the shape layer because the output is not allowed to grant itself approval. This illustrates a boundary between proposing data and deciding whether an action may occur. It does not claim that rejecting an extra field alone secures an application.

What we actually executed

The Python script defines the eight input strings directly. It parses them with the JSON library, checks the expected dictionary keys and value types, then applies the amount, currency, and calendar rules. Every input and result is saved to a downloadable JSON file.

There is no model call, private data, network request, or database write. The test evaluates the validators against known inputs. That lets us explain a failure precisely without attributing it to a model's reasoning. If you later collect permitted model outputs, you can evaluate them under a similar contract, but that would be a different dataset and study.

Download the runner and execute python3 run-output-validation.py results.json. Compare the result with the recorded output. The report contains the actual execution time, the scope statement, and separate flags for each validation layer.

Inspect the cases, not just the funnel

The fenced record contains a recognizable JSON object for a human reader, but the whole response is not the JSON value the contract expects. The numeric-string case is also understandable to a person, but it violates the selected type requirement. An application can choose coercion instead, provided it documents and tests that policy.

The three meaning failures are especially informative. The date matches the textual pattern, the negative value is a number, and the unsupported currency has the expected letter count. Each is structurally plausible. A shape validator that knows nothing about the task cannot resolve those rules simply by becoming more confident.

Why silent repair needs a policy

It is tempting to repair every rejected output automatically. Strip the code fence, convert the numeric string, replace the unsupported currency, and choose a nearby valid date. Some repairs are representational. Others change the user's intended meaning. Treating them all as harmless cleanup hides important decisions.

For example, removing a known wrapper may preserve the exact object, while converting February 30 to February 28 invents a date. Replacing an unsupported currency with the account default may change the value's meaning. Those changes should not be buried inside a generic “make valid” function without a clear rule.

A useful repair policy separates lossless normalization, explicit defaults, and unresolved values. Record what changed and why. If the required information is absent, return a reviewable draft or ask for clarification rather than fabricating a complete record. The operator's approval-boundary case study shows one architecture that provides a place for that review.

Validation should explain the next step

An error message should identify the failed boundary and an actionable correction. “Invalid response” leaves the user or developer to inspect everything again. “Date does not exist in the calendar” points to a specific value. “Currency is outside the supported set” is different from “Currency is missing.”

For an AI-assisted workflow, the application can pass a bounded correction request back to the generation step when appropriate. It should preserve the original input and reject instructions that expand the task. A retry to fix formatting should not become permission to invent missing business facts or approve an action.

Keep repeated failures visible. If the same field fails frequently, the problem may be an unclear input contract or insufficient context rather than a need for more retries. Measure the complete process, including the effort spent handling invalid outputs, before deciding that the assisted workflow is useful.

Copy this output-contract worksheet

Our executable test covers the first three rows only. The remaining rows are included to prevent a passing validator from being interpreted as blanket approval. A fully valid fictional record can still be unsupported by the user's message, belong to another account, or describe an action that was never authorized.

Use the worksheet when deciding where checks should live. Format checks belong near decoding. Domain rules belong with the application behavior. Permission and state checks belong at the action boundary. Putting all responsibility into one prompt makes the guarantees difficult to inspect and maintain.

A useful negative result is sometimes success

If the input cannot establish a required date, a workflow that returns an unresolved draft may be behaving correctly. An evaluation that rewards only fully populated objects can penalize that appropriate uncertainty. Define valid failure and clarification outcomes before comparing prompts.

Likewise, a parser rejecting a fenced response is not necessarily a system failure. It depends on the contract. If the caller expects raw JSON, rejection exposes a mismatch. If the product intentionally accepts a documented wrapper, then the parser should implement and test that rule. The important point is consistency between the declared interface and actual behavior.

This is why the lab reports each layer separately. A single pass percentage would hide whether failures came from syntax, shape, or meaning. The layer tells you what to change and whether the rejected input should ever have been accepted in the first place.

Expand the fixtures around your real risk

For a scheduling tool, add timezone transitions, unavailable attendees, and conflicting dates. For an inventory tool, add unknown item identifiers, unsupported units, and quantities beyond permitted bounds. For a publishing tool, add missing source links, invalid destinations, and a draft that changes after review.

Use synthetic examples that preserve the relevant behavior. Define expected outcomes independently of the implementation so the test can reveal a wrong rule rather than merely repeat it. Include valid boundary cases as well as invalid ones; otherwise, an overrestrictive validator can look successful by rejecting everything.

This lab's one valid record is enough to illustrate the layers, but not enough to establish broad acceptance coverage. A production-quality contract needs a richer set of valid examples and failure modes. Do not adopt the fictional currency or amount rules without replacing them with your application's actual requirements.

The relationship to structured generation

Asking an AI system for a structured response can make downstream handling easier because the desired representation is explicit. It does not remove the need to check values against application rules and evidence. The useful architecture treats generation as a producer of candidate data and validation as an inspectable boundary before action.

This article does not compare providers' structured-output features or make claims about their current guarantees. The observed result comes entirely from local Python checks on authored strings. If you evaluate a provider, record the actual model, schema, settings, inputs, and outputs, then report which failures occurred under that configuration.

Keep the distinction between guaranteed shape and justified meaning in your product language. “The response matches our schema” is a narrower claim than “the record is correct.” Readers and users benefit when those promises are stated separately.

What the eight records taught us

Seven syntactically readable responses narrowed to four shape-valid objects and one object meeting the fictional domain rules. The most useful failures were plausible values: a date that looked right, a number with the wrong sign for the task, and a currency-shaped string outside the allowed set.

Before connecting AI output to a persistent action, write the layers down. Decide what can be normalized, what requires clarification, and what must be rejected. Then retain the original response and the validation explanation so the next person can understand why the workflow proceeded or stopped.

Valid JSON is a good starting boundary. It is not the final decision about truth, permission, or completion. A dependable AI feature keeps those decisions visible all the way to the result.

Sources, materials, and limits

• Original executable validation lab and recorded eight-case output.
• StockNote approval case study: source-inspected separation between proposed fields and final action.
• Original contract worksheet and fictional entry rules. No external model benchmark, financial guidance, production data, or authorization test was performed.