Time to first token (TTFT)
Time to first token (TTFT) is the time between sending a request to an LLM and receiving the first generated token of its response.
On this page
It measures how long a user waits before a streamed answer starts. TTFT includes network transit, queueing, and prefill of the input prompt, so a slow first token can start anywhere on the serving path.
TTFT is measured at the client, from request sent to first token
To measure TTFT, send a streaming request and record two timestamps at the client: when the request leaves and when the first chunk containing an output token arrives. TTFT is the difference between them. Any chunk that doesn't directly contain output tokens counts as empty for TTFT, including chunks that carry only protocol data. The clock keeps running past those chunks until the first output token arrives.
NVIDIA's benchmarking guide measures TTFT from query submission to the first received token. The guide disregards initial responses with no content. Time to first byte is a separate metric. NVIDIA's AIPerf tool measures it from the moment the request finishes sending to the first byte of the response, including protocol data that arrives before any token. A non-streaming request can't separate TTFT from total response time, because the whole response arrives at once.
TTFT is reported as a percentile, such as p50 or p95, alongside the prompt length and concurrency that produced it. Server-side measurements leave out client network time and usually read lower than what users see. A 200-token prompt at low load and a 20,000-token retrieval request at peak traffic produce very different results.
TPOT, ITL and TPS answer different questions
Time per output token (TPOT) and inter-token latency (ITL) both describe how fast a response streams after the first token, but benchmark tools define them differently. vLLM calculates TPOT once per request and records ITL as every gap between streamed outputs, so ITL has its own distribution and tail.
NVIDIA's benchmarking guide treats ITL and TPOT as the same metric. Tools also disagree on the first token: NVIDIA's tools exclude TTFT from the ITL average, while LLMPerf includes it. Check each tool's formula before comparing published results.
NVIDIA separates TPS per user from TPS per system. TPS per user is one request's output tokens divided by its end-to-end latency. TPS per system is the total output tokens a deployment serves each second across all concurrent requests. As concurrency rises, the deployment batches more requests onto the same GPUs, and system TPS climbs. That growth stops at saturation. Saturation is the point where the GPUs have no spare capacity for another request. Past saturation, system TPS flattens or falls, and new requests wait in a queue. Latency rises across the whole range, and TPS per user drops with it.
Prompt length, queueing and network path drive TTFT
Prefill processes the entire input before the first token appears, so long prompts, retrieved documents, and conversation history all raise TTFT. Under concurrency, a large incoming prefill can hold up other requests waiting behind it. But prefill isn't the only source of delay.
Network travel adds to TTFT on both sides of the model. The request travels to the serving endpoint before prefill starts, and the first token travels back to the client once prefill finishes. Regional routing can reduce this delay when it shortens the complete request path. A nearby gateway doesn't help if the request still takes a long route from that gateway to the GPU.
Shared-prefix KV-cache reuse cuts TTFT by skipping prefill for the part of a prompt the cache already holds. Chunked prefill splits a long prompt into pieces so active streams keep moving. Its scheduling settings trade first-token latency against streaming smoothness. Disaggregated serving runs prefill and decode on separate hardware so each phase can be tuned and scaled on its own.
The right TTFT target depends on the product. Interactive chat may target a sub-second first token. In a typical voice agent, speech recognition runs before the LLM and speech synthesis runs after it. LLM TTFT is only one part of the delay before the user hears a reply, and the voice application has to budget for all three. Offline batch jobs usually prioritize throughput, cost, and completion deadlines over TTFT.
For more on how the network path and model serving shape TTFT in production, read how Parasail built sub-second LLM inference for global AI traffic.