LLM inference has two phases: prefill, when the model processes the input context before it can produce output, and decode, when it generates that output one token at a time. The distinction becomes visible in product behavior. A response may take too long to start, leaving a blank pause after a user sends a request, or it may begin promptly and arrive in slow, uneven bursts.
Prefill and decode put pressure on different parts of inference. A first-token delay and a slow stream can call for different measurements and serving choices; aggregate tokens per second can't show whether an individual request starts promptly, streams smoothly, or finishes within the time your product needs.
How prefill, decode, and the key-value (KV) cache work
Every autoregressive LLM request has a request path. Before generation can begin, the serving system assembles system instructions, retrieved context, conversation history, and the user’s prompt into the input token sequence sent to the model. During prefill, the model processes that input and builds the key-value (KV) cache: request state that helps it generate the response. During decode, the model produces the response one token at a time using that state.
The KV cache isn't the model’s “memory of the conversation.” It's serving state for the request context. It's created as the input is processed, then read as output tokens are generated. As the response grows, that state grows too.
The phases have different access patterns. Prefill can process prompt tokens in parallel, so it's typically compute-bound: the system has substantial model work to perform before it can emit the first token. Decode is sequential. The next token cannot be generated until the previous one is available, and each step repeatedly reads model and KV-cache state from memory. In common autoregressive serving, decode is therefore typically memory-bandwidth-bound as well as sequential. Redis’s overview of prefill and decode and WEKA’s technical guide describe the same broad distinction.
That doesn't mean every LLM, runtime, or request behaves identically. Model architecture, prompt shape, output length, queueing, and concurrency all affect what a user experiences. It does mean that a long wait before output and a slow stream deserve separate investigation.
How the two phases show up in latency and throughput
The right metrics answer different questions:
- Time to first token (TTFT) measures when a streamed response starts.
- Inter-token latency (ITL) measures the gap between output tokens and whether the stream feels smooth or stuttered.
- End-to-end latency captures total completion time.
- Throughput measures total work completed by the system over time.
TTFT is a useful front-of-request signal. Long prompts, large retrieval payloads, and extensive conversation history can all increase prefill work. But high TTFT doesn't prove prefill is the sole cause. Queueing, network overhead, and admission control can also delay the first token.
ITL is the corresponding streaming signal. A response can have a good TTFT and still feel poor once generation starts. Long outputs, code generation, and multi-step agent responses make token cadence and end-to-end perceived time especially important.
Throughput complements the latency measures; it answers a different, system-level question. A change that improves aggregate throughput can still hurt an individual user’s end-to-end perceived time. That trade-off shows up whenever a serving system packs more work onto shared resources.
The symptom is a starting point, not a final diagnosis. A long wait before streaming can involve prefill, queueing, or the network path. Uneven streaming can involve decode, cache pressure, or contention with other work.
For a broader framework on matching latency requirements and traffic shape to an inference mode, see How to choose the right managed inference architecture.
Start with the workload shape
Workload categories are useful starting points for a measurement plan. The same application can move between prefill and decode pressure as traffic, model choice, prompt construction, and concurrency change.
Long-context and retrieval-heavy requests should send you first to the input-token distribution, TTFT, and tail behavior. A RAG application may send a short user question alongside a large set of retrieved passages. A chat application may accumulate long conversation history. In both cases, the request can have far more input work than the UI makes obvious.
Long-output requests should send you to ITL, output-length distribution, and total completion time. Code generation, drafting workflows, and agents that explain their work can spend most of their perceived duration in decode.
Mixed traffic under concurrency requires both views. A large incoming prefill can block other requests in the queue, while a system tuned for one request shape can perform differently when the real distribution arrives. Measure p95 and p99 behavior with representative input lengths, output lengths, and arrival patterns—not one average prompt and one average response.
Serving patterns manage the trade-off
Once a team has identified where latency appears, the next decision is a trade-off among responsiveness, aggregate efficiency, cost, and operational complexity.
Scheduling and in-flight batching can improve how much work a system completes, but more packing can change per-request latency. The right policy depends on whether your product is more sensitive to TTFT, streaming smoothness, total completion time, or a combination.
Chunked prefill processes a large input in smaller pieces so active decode work can be interleaved. That can reduce interference between a large incoming prompt and an already-streaming response. Its effect depends on the request mix and scheduling policy.
KV-cache reuse and management affect memory pressure and concurrent capacity. The benefit depends on request similarity, cache behavior, and the serving implementation.
Speculative decoding addresses decode’s sequential path differently. A draft mechanism predicts candidate next tokens; the main model verifies them. When predictions are accepted, the system can advance multiple tokens without paying for the same number of sequential decode steps. It's worth evaluating for decode-heavy workloads; the outcome depends on the model pair, request pattern, and serving implementation.
Disaggregated serving is an industry architecture pattern that separates prefill and decode resources so they can be tuned and scaled independently. It can be relevant when sustained phase contention justifies the added data-transfer and operational complexity. Parasail doesn't currently support disaggregated serving, so this is a concept to understand when evaluating the wider serving landscape—not a statement of current product capability.
Phase separation can also allow the two pools to use different GPU profiles. This is a conceptual industry pattern, not a Parasail configuration: the workload and the cost of transferring KV-cache state have to justify the additional complexity.
For more context on batching, cache management, and the economics of keeping inference capacity ready, read The idle GPU tax.
Test the request path your users actually take
The benchmark that matters is the one that resembles your production request path. Before choosing an optimization, capture:
- prompt-length and output-length distributions;
- concurrency and burst behavior;
- streaming versus non-streaming product paths;
- model and configuration;
- cache behavior and request similarity; and
- TTFT, ITL, end-to-end latency, throughput, and p95/p99 values.
This is how to avoid treating a headline throughput number as a full diagnosis. A team with a slow first token needs a different next question from a team with smooth starts and uneven streams. A team whose metrics degrade only under load needs to understand the workload mix and queueing path before changing architecture.
Run the comparison against the request mixes your product actually receives. Keep the percentile, prompt length, output length, concurrency level, and measurement window together in the result. A TTFT number measured on short prompts at low concurrency can't settle a question about a retrieval-heavy workflow at peak traffic. The same applies to ITL: a smooth single stream doesn't establish how the system behaves when several long outputs are active.
This framing also keeps a useful distinction visible in planning. Per-stream measurements answer whether one user’s request feels responsive. Aggregate throughput answers how much work the system completes. Both matter, but they can move in different directions as batch size, scheduling, and contention change.
If no live user is waiting, that changes the evaluation too. Batch work can prioritize throughput and cost per useful result over interactive latency. If a user is waiting, start with the behavior they actually see, then trace it back through the request path.
Measure the phase before optimizing it
Prefill and decode are two parts of one inference request with separate performance consequences. Prefill helps determine when output can begin. The right serving decision follows from the constraint your workload actually exposes.
Bring your prompt and output distributions, concurrency pattern, target latency behavior, and model configuration to the conversation. Talk to us.
FAQ: What’s the difference between prefill and decode?
Prefill processes an LLM request’s input context and creates the KV-cache state used for generation. Decode then generates the output sequentially, one token at a time, using that state. Prefill is typically compute-bound; decode is typically memory-bandwidth-bound and constrained by each token’s dependency on the prior one. The distinction helps teams connect latency symptoms to the part of inference worth measuring first.