Performance

Tail latency (p95 / p99 latency)

Tail latency is the response time of the slowest requests in a system, usually reported as the 95th percentile (p95) or 99th percentile (p99).

On this page

A p99 latency of 600 ms means 99% of requests finished in 600 ms or less and 1% took longer. For LLMs, tail latency applies to time to first token, inter-token latency, and end-to-end latency.

A percentile shows a rank that an average hides

A percentile ranks requests from fastest to slowest and reports the latency at one point in that ranking. The median p50 is the latency of the request in the middle, so it tracks typical performance. p95 tracks the edge of normal operation and is often used as the day-to-day target. p99 tracks rare worst cases and is often set as the hard ceiling, because at a million requests a day the slowest 1% is still 10,000 requests.

Latency is better measured as a distribution than as an average, because some requests are served quickly while others take much longer, giving latency a long right tail. An average hides that tail and any change in it, so a dashboard tracking average latency can stay flat while the slowest requests get much worse. Percentiles keep the tail visible and the gap between p50 and p99 shows how consistent latency is.

LLM tail latency comes from the whole request path

A slow LLM request can pick up delay at every stage of its path. For example, time to first token (TTFT) includes network and queueing time as well as prefill, so any of the three can push a request into the tail. Depending on the scheduler, long prefills can delay other requests, and insufficient KV-cache capacity can trigger preemption and recomputation.

Outside the GPU, extra gateway hops, a detour across the public internet, or failover to a replica that is still warming up each delay a small share of requests. Agents raise the stakes. If a workflow chains ten model calls and delays are independent, roughly one run in ten hits at least one call slower than p99.

What is a good p99 latency?

No single p99 fits every product, because the target comes from the delay the product can tolerate.

When we built sub-second inference for a real-time voice agent, responses had to arrive within 600 ms end-to-end. This set our target for end-to-end p99 latency. We split that 600 ms budget into 250 ms for model serving, 300 ms for the network and gateway path, and 50 ms of safety margin, then set a 300 to 400 ms p95 to keep normal traffic clear of that ceiling. That is one workload class's target, not a platform guarantee.

Test the budget against realistic traffic segmented by region, including bursts and failover, with the model and hardware held fixed. Send load on a fixed schedule, because a load generator that waits for each response skips the worst moments, an error called coordinated omission.

All terms