A Valid JSON Response Can Still Be a Bad Contract

A response can be valid JSON and still be an invalid product response. Parsing proves only that the bytes have a recognizable shape. It does not prove that required fields exist, values are safe, relationships make sense, or the agent had permission to make the claim.

For QA engineers, structured output testing needs two layers: syntax and contract. Syntax asks whether a consumer can parse the response. Contract asks whether the parsed object satisfies the rules that make it usable. The second layer is where most of the risk lives.

A small synthetic example

Imagine an incident-triage assistant that returns this object:

{
  "severity": "high",
  "owner": "payments",
  "customer_impact": 12,
  "next_action": "restart-service",
  "confidence": 0.91
}

The JSON is well formed. A test that calls JSON.parse and checks for a 200 response will pass. But the object may still be wrong in several ways. The allowed severity values might be low, medium, and critical, with no high value. customer_impact might be required to be an integer greater than or equal to zero, but the system may have inferred it from an unverified sentence. restart-service might be a mutation that this assistant is not authorized to perform. The confidence value might be presented as a probability even though the evaluator never calibrated it.

None of these defects is a parsing failure.

Test the contract, not the formatting

Start by writing the output contract as executable rules. JSON Schema is one option, but the important property is that the rules are explicit and machine-checkable.

1. Required fields and types

Assert that every required field is present and has the expected type. Reject a numeric string when the consumer expects a number. Reject null unless null is part of the contract. Set additionalProperties deliberately: unknown fields can be harmless, or they can hide a model-version change that downstream code ignores.

This is basic validation, but it catches a common failure mode: the model returns a plausible object with one field omitted, and a downstream default silently changes the meaning.

2. Enumerations and bounds

Assert allowed values, ranges, and formats. If severity has three values, test all three valid values and several near misses, including case changes and synonyms. If confidence is a number from zero to one, test negative values, values above one, NaN-like strings, and excessive precision.

Do not confuse a bound with calibration. A confidence of 0.91 is in range, but that does not establish that predictions marked 0.91 are correct 91 percent of the time. Range validation protects the interface. Calibration requires a separate evaluation set with known outcomes.

3. Cross-field invariants

Fields can be individually valid and jointly impossible. A response with severity: "low" and customer_impact: 50000 may violate a product rule. A response with next_action: "restart-service" and authorization: "read-only" is contradictory even if both strings are allowed.

Write these checks as invariants. For example:

if authorization == "read-only":
    assert next_action in ["inspect", "summarize", "escalate"]

The exact policy belongs to the product, not the model. The test should make the policy visible.

4. Provenance and claim support

A structured field is still a claim. If customer_impact came from a monitoring query, preserve the query result or trace reference. If no trusted source supports it, the valid response may need customer_impact: null and a reason such as insufficient-evidence.

This is an important negative test: provide a prompt that contains a confident but unsupported number, then verify that the assistant does not copy the number into a field whose contract requires evidence. A schema cannot test truth by itself. It can require a provenance field, after which a second assertion can verify that the provenance points to an allowed source.

5. Side-effect boundaries

Structured output often drives code. A field such as next_action may be treated as a command by an executor. Test that parsing and validation happen before execution, and that an invalid or unauthorized object causes zero side effects.

Inject a valid-looking action that is outside the user or service authority. The expected result is a refusal recorded in the trace, not a tool call followed by an apology. Also test duplicate delivery. If the same accepted object is processed twice, the executor should use an intent ID or an equivalent idempotency control.

A practical test matrix

For each output contract, keep a small fixture set with these cases:

• valid object with the minimum required fields
• valid object with every optional field
• missing field, wrong type, unknown field, and malformed format
• boundary values and one value just outside each boundary
• individually valid fields that violate a cross-field invariant
• unsupported claim with no provenance
• authorized action, refused action, and duplicate action

Record more than pass or fail. Capture the raw model output, parsed object, schema version, validation errors, authorization result, executor call count, and final user-visible claim. Without those fields, a failed test can tell you that something broke without telling you whether the model, validator, policy layer, or executor is responsible.

The limitation

Contract tests cannot prove that the schema describes the right product behavior. A perfectly enforced schema can encode a bad policy. Review the contract with the people who own the workflow, version it alongside the consumer, and add a regression fixture whenever a production incident exposes a missing rule.

The useful question is not "Did the model return JSON?" It is "What guarantees does this object make, and which system proves each guarantee?" That question gives QA a test boundary that survives model changes, prompt edits, and different wording in the final answer.