Inference throughput
Inference throughput is the amount of work an inference system completes per unit of time.
On this page
For large language models, teams commonly measure it in output tokens per second, completed requests per second, or records processed per hour. A useful throughput number names the work being counted and whether the measurement covers one request, one server, or an entire deployment.
Common inference throughput units
Choose a unit that reflects what your application needs to accomplish:
- Output tokens per second measures how much generated text the system produces over time.
- Completed requests per second measures how many requests finish successfully. Comparisons need similar prompt and response lengths, plus a consistent definition of a completed request.
- Records or jobs per hour measures completed work in an offline pipeline, such as documents classified.
For example, a pipeline that successfully processes 6,000 records in one hour averages 100 records per minute. That describes its throughput for that workload and measurement window.
Token-based measurements need another distinction: whether they count input tokens, output tokens, or both. A combined total can obscure differences between processing long prompts and generating long responses. Label the metric before comparing results.
System throughput and individual response speed
Aggregate throughput measures work completed across requests in the measured system over a defined time window, including concurrent requests. A per-user token rate measures how quickly one request produces output. Its value depends on whether the measurement includes the wait for the first token.
Increasing the number of concurrent requests can raise aggregate throughput while slowing token generation for each user. As the system reaches capacity, additional requests can spend longer waiting in a queue.
For interactive applications, read throughput alongside:
- Time to first token: how long the user waits for output to begin.
- Inter-token latency: the time between successive output tokens.
- End-to-end latency: how long the complete request takes.
Tail latency shows how long slower requests take, which an average can hide. For an offline pipeline, throughput helps determine whether the work will finish before its deadline.
What makes throughput numbers comparable?
Two results labeled “tokens per second” may measure different work under different conditions. Before using them to compare systems, check:
- Metric and scope: input, output, or total tokens; per-request or system-wide measurement.
- Model and infrastructure: model version, hardware type and count, numerical precision, and serving configuration.
- Workload and load: prompt and output lengths, request arrival rate, and concurrency.
- Measurement conditions: warmup, test duration, treatment of errors and retries, and latency requirements.
When latency requirements matter, goodput measures the rate of work meeting them—for example, successful requests per second that meet all specified latency limits. Raw throughput alone doesn’t establish response quality, reliability, or cost efficiency.
NVIDIA’s AIPerf metrics reference gives formula-level definitions for aggregate throughput, per-user token rate, requests per second, and goodput.
For ways to improve throughput while meeting response-time targets, see How to increase LLM inference throughput without breaking your latency budget.