GPU-hour cost
GPU-hour cost is the price of running one GPU for one hour.
On this page
Dedicated inference endpoints and rented cloud GPUs are typically priced by the GPU-hour. Providers bill the time a GPU runs by the minute or the second, and some stop billing when a deployment scales to zero.
GPU-hour cost runs from $0.50 for a T4 to $10 for an H100
Renting one GPU costs from $0.50 to $10 an hour on Hugging Face's Inference Endpoints pricing. The GPU-hour cost depends on the GPU model, the provider, and the service offered, such as GPU rental or managed inference. The GPU model sets the base GPU-hour cost. Hugging Face lists a T4 at $0.50 an hour on AWS and an A100 at $2.50 on AWS and GCP.
The provider changes the GPU-hour cost of the same GPU model. Parasail's comparison of inference providers for startups puts an on-demand H100 at $4.50 an hour from Nebius and $8 from Fireworks AI, which runs the deployment for you. Hugging Face lists an H100 at $10. The GPUs can also be billed in smaller units. Baseten bills per minute and stops compute charges after a deployment scales to zero, and Fireworks AI bills on-demand GPUs per second.
Per-token pricing bills for tokens and GPU-hour pricing bills for time
Per-token pricing charges a rate for each million input and output tokens the model processes. A per-token bill rises and falls with traffic and is zero when no requests arrive. A GPU-hour cost prices the GPU's time, so the bill is the same in a busy hour and a quiet one. The cost of GPU-hours that process no tokens is often referred to as the idle GPU tax.
To choose between per-token and GPU-hour pricing, first benchmark both options on a representative workload. Check whether each meets your latency and throughput requirements at the expected concurrency. Among options that meet those requirements, compare the per-token API bill with the total GPU-hour cost of serving the same input and output token volumes over the same period. Include idle time and all GPUs and replicas required to meet your performance targets.
If shared API cannot meet your application’s latency and throughput requirements, a dedicated endpoint may be a better fit. Dedicated capacity is often billed by GPU time, but some providers offer billing per token. For example, Parasail’s Elastic Endpoints offers a dedicated production endpoint with per-token billing for selected models. Parasail’s team can tune the deployment for your workload.
For each GPU-hour option that meets your requirements, divide the GPU-hour cost by the millions of tokens the GPU processes in that hour to get its cost per million tokens. A $4 GPU-hour that processes 2 million tokens costs $2 per million tokens. The same GPU-hour at 200,000 tokens costs $20 per million. The GPU-hour cost is fixed, so the cost per token falls as the GPU gets busier.
GPU-hour pricing can cost less than per-token pricing when the GPUs stay busy
Products with steady, high-volume traffic around the clock get the lowest cost per token from GPU-hour pricing. The GPUs behind these products run close to capacity in every paid hour.
Products with cyclical traffic pay GPU-hour pricing for hours they don’t always use. For example, meeting-notes apps and support chatbots peak in working hours and go quiet overnight and on weekends. An always-on GPU bills all 168 hours of the week, and a product that is quiet all weekend leaves 48 of those hours idle.
A product can also combine both billing models. A support chatbot with predictable business-hour demand could use dedicated GPUs billed by the hour during sustained busy periods, then route overnight and weekend requests to a per-token API.
Combining both billing models can reduce costs when daytime traffic keeps the GPUs sufficiently busy and GPU charges stop during quiet periods. Both deployments must support the required model and meet the chatbot’s performance requirements. Include startup costs and any minimum commitments when comparing this approach with using either billing model alone.
The break-even point for GPU-hour pricing depends on the model size, the GPU generation, your latency target and how much your traffic varies. To compare GPU-hour pricing with per-token pricing on your own traffic, read Parasail's guide to choosing a managed inference architecture.