harshaA latency percentile is a compact summary of many requests. It can reveal a regression or help define a target. It can also flatten several different experiences into one number.
A streaming answer, a document import, and a dashboard refresh do not have the same definition of “done.” Treating their latency as interchangeable makes a metric easier to display and harder to use.
Name the interval
For a streaming workflow, I want to distinguish time to acceptance, time waiting for execution, time to first useful output, and time to completion. For a document pipeline, extraction and indexing deserve separate observations.
A fast first token can coexist with a slow complete answer. A quick API acknowledgment can hide a long queue. Neither measurement is wrong; each describes a different interval.
The metric name should carry that distinction. “Response time” is too broad when the system returns an acknowledgment and continues working elsewhere.
Explain the distribution
Percentiles are most useful alongside request volume, errors, and workload shape. A comparison between two periods may reflect a different mix of documents or model requests rather than a code improvement.
I would segment by meaningful workload characteristics before drawing a conclusion. That might include route, processing stage, or a bounded size category. It should not require exposing sensitive inputs or creating unbounded metric labels.
A trace can then explain an individual path through the system. OpenTelemetry describes traces as a representation of a request's journey through operations; that is a useful complement to an aggregate chart. Its tracing concepts explain the relationship.
Connect the number to a decision
An observation should suggest an action. If queue wait dominates, optimizing JSON serialization may not help. If retrieval is fast but generation is slow, changing the database index may solve the wrong problem.
I still value percentile improvements. I simply want them attached to a workflow, a measurement boundary, and a reason the change matters.
For a personal project or a production platform, the best performance story is not a lone number in large type. It is an explanation of what got faster, what remained expensive, and how that changed the experience of the person waiting.