Back to the journal

LLM evaluation / 2 MIN READ

Valid JSON is not a valid decision

A response can satisfy a schema and still misunderstand the task.

Sai Sri Harsha Guddati ·

Structured output makes an AI response easier to pass into software. It does not make the response correct.

A payload can contain every required field and still refer to the wrong account, choose an unsupported action, or express confidence where the evidence is missing. Syntax is one layer of the contract. Meaning is another.

Validate in stages

I would separate parsing, structural validation, domain validation, and authorization. Parsing establishes whether the payload can be read. A schema establishes shape and types. Domain rules establish whether the values make sense together. Authorization establishes whether the requested operation is allowed.

Pydantic's model documentation describes validation and the resulting model guarantees; the application still needs rules appropriate to its domain. The models guide is useful for understanding that boundary.

Consider an illustrative scheduling response. A well-formed start time and end time are not enough if the end precedes the start. A legitimate project ID is not enough if it belongs to another team. A tool name is not enough if the user never authorized the action.

Return useful failures

A validation failure should help the caller distinguish a malformed response from a prohibited operation or an unresolved request. Those cases need different recovery behavior.

Blindly asking the model to try again can turn a clear authorization denial into a repeated attempt. Repairing formatting may be appropriate for a parse error, but it should not silently revise the user's intended action.

The system needs to preserve the original request and the validation reason through any repair path.

Evaluate the accepted mistakes

Tests often focus on invalid payloads because they are easy to reject. I also want examples that are structurally valid and semantically wrong.

These cases reveal whether the application is treating schema conformance as a substitute for understanding. They are especially important when structured output becomes input to another service rather than text reviewed by a person.

The practical goal is a layered boundary: make the output readable, make its values meaningful, then decide whether it may be used. JSON is a transport format. It should never become the entire argument for trusting an operation.

More from the journal