Measure the p99, because the average hides your worst day
The claim An average response time is close to useless as a health metric, because it is dominated by the many fast requests and says nothing about the few slow ones — and the slow...
The claim
An average response time is close to useless as a health metric, because it is dominated by the many fast requests and says nothing about the few slow ones — and the slow ones are the customers you are losing. A dashboard reporting a healthy 120-millisecond average can coexist with 1% of requests taking eight seconds, and that 1% is not a rounding error: on a busy site it is thousands of people a day meeting a page that feels broken.
Why the average lies
Consider a hundred requests. Ninety-nine return in 80 milliseconds and one takes 5 seconds. The average is about 130 milliseconds — a number that looks fine and describes no actual request. Nobody experienced 130 milliseconds; ninety-nine people experienced 80 and one person experienced a five-second wait. The average has mathematically erased the only data point that represents a problem.
This matters because slow requests are not randomly distributed across your users. They cluster on the customers with the most data, the largest carts, the longest order histories — frequently your best customers. The average tells you the typical request is fast. It cannot tell you that your highest-value users are the ones hitting the tail.
What the percentiles mean
A percentile is a direct statement about a fraction of your traffic. The 95th percentile latency is the value that 95% of requests come in under and 5% exceed. The 99th is the value 99% beat. Read them as promises to your users:
| Metric | Says about your users |
|---|---|
| p50 (median) | Half of requests are faster than this |
| p95 | 19 in 20 requests are faster; 1 in 20 is slower |
| p99 | 1 in 100 requests is slower than this |
| p99.9 | 1 in 1000 — your worst regular experience |
The gap between p50 and p99 is the shape of your problem. A p50 of 80 milliseconds and a p99 of 200 milliseconds is a healthy, consistent system. A p50 of 80 milliseconds and a p99 of 6 seconds is a system that is fast for most and badly broken for a meaningful minority, and only the percentile reveals it.
The tail-amplification trap
There is a counterintuitive reason the tail matters more than its percentage suggests: a single page load often makes several backend requests, and the page is only as fast as its slowest one. If rendering a dashboard requires 10 backend calls and each has a 1% chance of being slow, the probability that at least one is slow is not 1% — it is about 10%. The per-request tail of 1% became a per-page tail of nearly one in ten. This is why systems that look fine at the request level feel sluggish at the page level, and why the p99 of your individual services governs the p90 of your user experience.
Measuring it without a heavy stack
You do not need an observability platform to start. Nginx can log request time, and a few lines of processing turn a log file into percentiles:
log_format perf '$request_time $request_uri';
# then, over a log window:
awk '{print $1}' access.log | sort -n | \
awk '{a[NR]=$1} END{print "p50", a[int(NR*0.50)];
print "p95", a[int(NR*0.95)];
print "p99", a[int(NR*0.99)]}'That is a genuine percentile report from data you already have. A proper metrics system computes these continuously and lets you break them down by route, which is the next thing you will want — because the p99 of your whole site is less actionable than the p99 of /checkout specifically.
Set targets on percentiles, not averages
Once you measure percentiles, set your objectives on them. "Average under 200 milliseconds" is a target you can hit while badly failing a fifth of requests. "p95 under 400 milliseconds and p99 under 1 second" is a target that actually constrains the experience, because it puts a ceiling on how bad the slow tail is allowed to get. Alert on the percentile crossing its threshold, not on the average moving, because by the time a slow tail is large enough to move the average, it has been hurting users for a while.
Where to start
Pull last week's access log and compute the p50, p95, and p99 for your three most important routes. If p99 is within about three times p50, your system is consistent and you can move on. If p99 is ten or more times p50, you have a tail problem that your average has been hiding, and the queries or code paths responsible are worth finding — because the users on the wrong side of that number are experiencing a different, much worse product than the one your average describes.
