Cold-start latency
Cold-start latency is the time a newly launched copy of a model takes to load and become ready to serve its first request.
On this page
Each running copy is called a replica, and the platform launches a new replica when traffic rises, when a replica fails, or when traffic returns to a deployment that had shut down. A replica that has finished starting is warm, and requests may wait on cold-start latency when no warm replica has spare capacity. The startup covers downloading the model weights, loading them into GPU memory and starting the inference engine.
Cold-start latency is measured from replica launch to ready
To measure cold-start latency, timestamp when a replica is launched and when it can accept its first request. Cold-start latency is the difference between the two timestamps. Parasail's snapshotting post reports baseline vLLM startup times for four model configurations on H200 GPUs: about 64 seconds for Qwen2.5-7B-Instruct at the low end and 186 seconds for Gemma-3-27B at the high end.
Where you choose to start measuring cold-start latency also changes the result. A study of vLLM startup counts from the start of engine initialization to the moment the engine is ready. That boundary leaves out container startup and downloading weights from remote storage. KServe's autoscaling guide counts the time to start the container and download the model. Check which stages a published figure includes before comparing it with your own.
Cold-start latency matters in scale-out, scale from zero and failure recovery
Requests wait on cold-start latency when traffic outgrows the number of warm replicas currently available, when a deployment scales from zero, and when a replica fails. Scale from zero happens when traffic returns to a deployment that shut down every replica during a quiet period, so the platform has to launch a first replica before it can serve anything.
During scale-out, the autoscaler adds replicas as load rises. Excess requests may queue while the new replicas start; longer startup prolongs overload when demand exceeds warm capacity.
When a deployment scales from zero, no warm replica is running, so the first request waits on the full cold start before its first token arrives. With GPU-hour billing, keeping replicas warm trades idle-compute cost for avoiding scale-from-zero delay. Comparing GPU-hour and per-token pricing against your traffic pattern shows whether that idle cost is worth paying.
After a failure, the replacement replica has to load the model before it takes traffic. Surviving replicas can take new or retried requests while the replacement starts, provided they have spare capacity. Failure detection, queueing on surviving replicas and replacement startup each add tail latency during recovery.
Weight download, weight loading and engine startup drive cold-start latency
Weight download adds to cold-start latency when the server has no local copy of the model files. The ServerlessLLM paper puts the download of a 130 GB checkpoint such as LLaMA-2-70B at a minimum of 26 seconds on a 5 GB/s network.
Weight loading moves the weights from local disk into GPU memory. The same paper measured 84 seconds to load LLaMA-2-70B into 8 GPUs with PyTorch. With comparable precision, hardware and cache state, larger weight files usually take longer to load.
Engine startup covers the engine's remaining work, such as torch compilation and CUDA graph capture. In the vLLM study, Llama 3.2 3B took 20.32 seconds in total to start on an H100 GPU. Four of the study's six startup steps are CPU-bound, so the server's CPU affects cold-start latency as well as the GPU.
Locally cached weights, optimized loaders and runtime snapshots can shorten startup. Read Parasail's guide to faster weight loading.