GPU utilization
GPU utilization is a family of measurements that describe whether a GPU is active, how efficiently a workload uses its compute and memory, or how much paid capacity does useful work.
On this page
The percentage nvidia-smi reports measures the share of time at least one kernel was executing. It does not measure how much of the GPU’s compute capacity the workload used.
What the GPU-utilization percentage measures
The number most operators see is an activity metric. NVIDIA’s NVML reference defines it as the percentage of time one or more kernels were executing during the last sample period, which lasts between 1/6 of a second and 1 second depending on the GPU. A 90% reading means kernels ran for 90% of the sample period. It does not mean the workload used 90% of the GPU’s arithmetic throughput, Tensor Core capacity, or memory bandwidth.
The memory percentage NVIDIA reports next to GPU activity is also time-based. It measures how long device memory was being read or written during the sample period. Allocated VRAM is a separate measure, because a model can occupy memory while its compute work waits.
Activity still answers a useful question. Compared with request traffic, it shows idle periods, sustained demand, and capacity the workload isn’t using. Because the sample period varies by GPU and dashboards often average many periods together, every utilization number should name the resource and the time window behind it.
A busy GPU can still be inefficient for inference
A GPU can be active for every sample period and still use only part of its hardware. Model FLOPs Utilization (MFU) compares the model’s achieved computation rate with the hardware’s peak compute rate. Model Bandwidth Utilization (MBU) compares the model’s achieved memory bandwidth with the GPU’s peak memory bandwidth.
Whether MBU or MFU is more useful depends on what limits the workload. Decode at small batch sizes is often limited by memory bandwidth. Each sequence generates one token at a time, and every step reads the model weights and KV cache from device memory. A GPU running small-batch decode can show high activity and strong MBU while its MFU stays low. Larger decode batches can make decode compute-bound, and MFU then becomes the more useful ratio. During prefill, the GPU processes the whole prompt in parallel and compute often becomes the limit, so MFU tells you more.
Larger batches can share each weight read across more requests and raise aggregate throughput. They can also change execution latency and the time requests spend waiting. The added wait depends on how the scheduler forms batches. A latency-sensitive application may choose smaller batches and accept lower hardware efficiency so responses start or finish sooner.
There is no universal “good” GPU-utilization percentage
The right utilization level depends on the metric and the workload. Low activity can indicate wasteful overprovisioning, but it can also reflect spare capacity held for a traffic spike or latency target. Removing that headroom may improve the average while creating request queues when demand rises.
High activity needs the same scrutiny. It can reflect productive sustained demand or saturation that is already hurting tail latency. Daily and monthly averages can hide both short peaks and long quiet periods.
Autoscaling can remove idle replicas by matching allocated capacity to traffic. It can improve allocation utilization without guaranteeing that each active GPU computes more efficiently. Parasail’s guide to choosing a managed inference architecture shows how traffic shape, latency targets, and billing models change what a utilization number means.
For inference, read utilization with workload outcomes
Start by identifying whether a number describes device activity, compute or memory efficiency, or paid-capacity utilization. Then read it beside the workload signals that determine whether the system is healthy: request volume and traffic shape, throughput or queue depth, p95 and p99 latency, errors and retries, and effective cost per useful result.
Low activity can come from sparse traffic, a CPU or data bottleneck, or overprovisioning. Low arithmetic efficiency can come from a memory-bound workload, small batches chosen for latency, or a mismatch between model and hardware. A common mismatch is a matrix dimension that doesn’t fit evenly into the GPU’s compute tiles, which wastes work on partly filled tiles. Each cause calls for a different fix.
Under GPU-hour pricing, the team pays for provisioned capacity whether requests use it or not. Headroom the service needs still costs money, and a busy but inefficient GPU raises the cost of each useful result. The idle GPU tax breaks down what paid idle capacity costs.