harshaAn evaluation score is useful only when its meaning survives a conversation with the person using the application. “Quality improved” needs a more concrete explanation: the assistant found the relevant passage more often, stopped inventing unsupported details, or produced output that the next system could actually accept.
My evaluation work spans retrieval relevance, groundedness, tool-call correctness, schema conformance, latency, and provider failures. I think of these as separate expectations rather than one interchangeable measure of intelligence.
Give each case a reason to exist
An illustrative test case should include an input, expected behavior, and a description of the failure it is meant to catch. A query about an absent fact might require an explicit statement that the evidence does not answer it. A tool request might require the correct argument values and a refusal to call an unauthorized operation.
This makes a failure actionable. The engineer can see which promise changed instead of trying to interpret a composite score.
I would include ordinary cases, boundary cases, and cases with insufficient evidence. A test set made entirely of easy successes encourages a system that appears capable until it reaches the first ambiguous request.
Version the environment
A result depends on more than the prompt. Document versions, retrieval configuration, tool schemas, model settings, and evaluation logic can all affect it.
Recording those inputs allows a comparison to explain what changed. Without that record, a regression can be indistinguishable from a different dataset or a moving dependency.
For probabilistic output, repeatability also needs an honest definition. A single favorable run is a weak basis for declaring a general improvement.
Keep judgment visible
Some checks are deterministic: valid schema, expected identifier, prohibited tool call. Others require human assessment or a carefully designed evaluator. Those categories should remain distinct in the report.
I want the evaluation suite to help a reviewer understand the application, not just approve a release. Its cases should read like a map of the promises the product makes.
The best scorecard therefore has a narrative behind it: these are the workflows we care about, these are the failures we refuse to hide, and this change improves these expectations under these conditions.