ThinkKit Works
All notes
Performance Testing

Reading a Load Test: p95, p99, and Why Averages Lie

How to actually read load-test results — why the average response time hides your worst problems, and how percentiles, throughput, and error rate tell the real story.

2 min read
PerformancePercentilesMetricsLatency

The single most common mistake in performance testing is celebrating a good average response time. The average is the one number almost guaranteed to hide the problem you actually care about. Here is how to read a load test properly.

Why the average lies

Imagine 100 requests: 95 return in 50 ms, 5 return in 4 seconds. The average is about 250 ms — a number that looks fine and describes none of the actual experience. The 95 fast users don’t notice 250 ms, and the 5 slow users are having a terrible time the average completely erases.

Real traffic is full of these long tails: a cold cache, a lock, a slow query, a GC pause. Averages smooth them away. Percentiles expose them.

Percentiles, in plain terms

A percentile answers: “how slow was the request at this position in the sorted list?”

  • p50 (median) — half of requests were faster than this. The typical experience.
  • p95 — 95% were faster; 1 in 20 was slower. Where pain starts to show.
  • p99 — 99% were faster; 1 in 100 was slower. Your unlucky-but-real users.

If p50 is great but p99 is awful, you don’t have a fast system — you have a fast system that regularly fails a slice of your users.

Which percentile should you care about?

It depends on request volume. At 1,000 requests per second, p99 means 10 slow requests every second — not an edge case, a constant stream of unhappy users. High-traffic systems often set SLAs on p99 or even p99.9 for exactly this reason.

Set thresholds explicitly. In k6 this is a one-liner that turns “feels slow” into a pass/fail gate:

export const options = {
  thresholds: {
    http_req_duration: ['p(95)<400', 'p(99)<800'],
    http_req_failed: ['rate<0.01'],
  },
};

If a threshold is breached, the run fails — no interpretation needed.

Read three numbers together, never one

A load test result is only meaningful as a trio:

  • Latency percentiles — how slow (p50/p95/p99), not the average
  • Throughput — requests/sec actually served
  • Error rate — what fraction failed or timed out

The trap is reading them separately. Latency that looks great while the error rate is climbing usually means the system is shedding load — fast responses because half the requests are 500s. Always check error rate before trusting a latency number.

A quick reading checklist

When a report lands, I run down this list before drawing any conclusion:

  • Is the error rate near zero? (If not, latency numbers are suspect.)
  • What is p95 and p99 — not the average?
  • Did throughput actually reach the target load?
  • Did latency degrade as load ramped, or stay flat?
  • Is there a long tail (p99 ≫ p50) hiding behind a fine median?

The takeaway

An average response time is a marketing number. Percentiles, throughput, and error rate — read together — are an engineering number. Set your SLAs on the tail, gate your builds on the tail, and you will catch the problems your users would otherwise catch for you.

Related

More in Performance Testing