Performance

End-to-end latency (E2E latency)

End-to-end latency (E2E latency) is the time between sending a request to an LLM and receiving the last token of its response.

On this page

It measures how long a user or a calling program waits for the LLM’s complete answer. End-to-end latency includes network transit, queueing, prefill of the input, and generation of every output token, so it increases with the length of the response.

End-to-end latency is TTFT plus generation time

Time to first token (TTFT) covers the wait before the first token of the LLM’s response appears for the user. Generation time runs from the first token received to the last. To calculate it, multiply the number of tokens after the first by the time per output token (TPOT). TPOT is the average gap between tokens, and some tools call it inter-token latency (ITL).

NVIDIA's benchmarking guide breaks end-to-end latency into TTFT plus generation time. A request with a 400 ms TTFT that streams 300 more tokens at a TPOT of 20 ms has an end-to-end latency of 6.4 seconds, made up of 0.4 seconds of TTFT and 6 seconds of generation time.

NVIDIA counts end-to-end latency from the moment the client submits the query. This means that NVIDIA’s end-to-end latency figure includes network latency. vLLM's request latency metric starts when its frontend begins processing the request and ends when the frontend receives the final token. This means that vLLM’s request latency metric ignores network latency. Check how the LLM serving platform measures end-to-end latency before comparing it with your own.

End-to-end latency matters when the caller is waiting on a completed response

Agents, tool calls and inline code completion all wait on the complete response, so they need tight end-to-end latency. An agent waits for each response before starting its next step, so the latency of every call adds up across a task. Callers that need complete tool arguments or a validated JSON result care about completion time, even when partial parsing is possible.

For example, inline code completion must show the whole suggestion before the developer keeps typing. In this case, a request fails if the full response takes longer than the application is set to wait.

A deployment can report higher output-token throughput while agents and tool calls wait longer. Output-token throughput measures output tokens per second across requests. Larger batches can improve GPU utilization and throughput until saturation, but may increase the end-to-end latency of individual requests. Parasail's guide to increasing throughput without breaking your latency budget shows how to find the highest throughput a deployment can sustain within a latency target.

Output length, queueing and the network path drive end-to-end latency

Longer outputs usually take longer to generate. At an average 20 ms between tokens, 50-token and 500-token answers take about 1 and 10 seconds after the first token. Generation often dominates for long responses. Parasail's guide to prefill vs. decode explains which workloads spend most of their time in decode.

Queueing adds a wait before generation starts. A server runs a limited number of requests at once. When demand exceeds available serving capacity, new requests wait for a scheduling slot. Time spent in the queue adds directly to end-to-end latency.

The network path adds time in both directions. The request and the response both cross the network, and the delay grows with distance. Parasail reports roughly 180–220 ms of median round-trip delay for US-East–APAC traffic.

A good end-to-end target meets the application’s time limit under representative prompt lengths, response lengths and request load. To diagnose which stage is slowing your end-to-end latency, read Parasail's guide to reducing LLM latency in production.

All terms