harshaA retrieval-augmented application can produce a confident answer while making its most important mistake earlier in the pipeline. The relevant document was never retrieved, the useful passage was split badly, or an access filter removed the wrong material.
The model then answers from the evidence it received. Fluency makes the result look complete, even when the evidence was incomplete.
Inspect the evidence separately
My work includes document pipelines, hybrid retrieval, reranking, and source attribution. I find those pieces easier to reason about when retrieval quality is evaluated before answer quality.
For an illustrative policy question, first identify the passages that should be available to an authorized user. Then inspect whether retrieval returns them, how highly they rank, and whether the returned text includes the necessary qualification.
Only after that should the generated answer be judged. Otherwise, a poor answer can send the investigation toward prompting when the real issue is extraction or indexing.
Preserve the question's shape
Some questions depend on exact terms: a course code, an identifier, a date, or a named exception. Others depend on meaning expressed differently from the document. A retrieval strategy needs to account for those differences.
A useful evaluation set includes both. It should also include questions whose answer is absent, questions requiring several passages, and questions that should return different evidence for different authorized users.
A system that always returns something can hide the difference between relevance and availability. Empty evidence is sometimes the correct result.
Keep a record of the decision
For debugging, I want a bounded record of the retrieval process: query configuration, document versions, filters, selected passage identifiers, and scores with their meaning documented. Sensitive document contents do not need to become unrestricted logs.
That record lets an engineer ask whether a failure came from stale data, poor candidate selection, access policy, or answer construction. Without it, changing the prompt becomes an attractive but poorly targeted experiment.
The application earns trust when it can distinguish “I found supporting evidence” from “I generated a plausible response.” Retrieval is not an invisible prelude to intelligence. It is the process that decides what the answer is allowed to know.