FoundationMethodFree Preview

Latency vs Throughput — P95, P99, Tail Latency Explained

Understand response time, work per second, p95/p99, and why averages hide user pain.

Latency is how long it takes for a single thing to happen. Throughput is how many things happen per second. They sound similar but they optimize differently and sometimes conflict. A database can have low latency (fast individual queries) but low throughput (can't handle many concurrent queries). A batch job can have high throughput (processes millions of records/hour) but high latency (takes 10 minutes to start).

Where most engineers go wrong is using average latency as their metric. Average is a lie. If 95% of your requests complete in 10ms but 5% take 5000ms, your average might look fine at 260ms — but 1 in 20 users is waiting 5 seconds. That's why p95, p99, and p999 matter: they show the worst experience real users actually have.

Tail latency is caused by things like GC pauses, lock contention, network jitter, and cache misses. It's worst in fan-out architectures — if a single user request fans out to 10 microservices, and each has a 1% chance of a 500ms slow response, the probability of hitting at least one is nearly 10%. This is the "long tail" problem: distributed systems amplify slowdowns.

Key concepts

p50/p95/p99queueingLittle's Lawhead-of-line blockingSLO budgets

Step-by-step approach

1
Define the latency target: interactive actions need <100ms; bulk operations <1s; background jobs <10s. Write it down before any design work.
2
Measure baseline p50, p95, p99, and p999 latency under realistic load — not single-thread benchmarks which hide concurrency effects.
3
Profile where each percentile of latency is spent: network, serialization, application logic, database query, downstream service calls.
4
Apply Little's Law: concurrency = throughput × latency. If p99 latency doubles, you need 2× the concurrent connections to maintain the same throughput.
5
Set error budgets: if your SLO is p99 < 200ms, a 10% error budget allows 1% of requests to exceed 200ms before burning the budget.

Key trade-offs

Throughput vs. latency

Batching increases throughput but adds latency. Streaming reduces latency but increases overhead per message. Choose based on what users actually feel.

Caching vs. freshness

A cache hit is fast but serves potentially stale data. Cache misses spike latency. Design your cache strategy around your latency SLO.

Fan-out vs. fan-in

Fan-out (one request → many services) amplifies tail latency. Fan-in (aggregate before responding) adds a join step but caps the tail.

Common pitfalls

⚠

Measuring average latency and calling it 'good performance' — averages hide the worst user experiences.

⚠

Not accounting for coordinated omission in load tests: if your test backs off when the system is slow, it measures latency under ideal conditions, not under load.

⚠

Ignoring head-of-line blocking: a slow request at the head of a queue blocks fast requests behind it, creating artificial latency spikes.

Interview questions on this topic

Your API has p50 of 10ms but p99 of 2 seconds. Walk me through how you'd diagnose and fix this.

Explain the trade-off between latency and throughput in the context of a write-heavy database.

Practice answering these with AI feedback → Start on CrackLab

Continue learning — explore 207 more topics on CrackLab DesignHub

From Classic HLD designs (Twitter, YouTube, Uber) to LLD patterns, distributed systems, databases, and company case studies.

Start Learning Free →See all 208 topics