In modern production environments, cold-start latency is a major bottleneck for scaling AI services. When traffic spikes, the time it takes to initialize a new model instance can mean the difference between a seamless user experience and a frustrating delay. This is the second part in the 3-part series on reducing these latencies.
To understand the problem, think of it like a car engine: "cold starting" a vLLM process is like building and starting an engine from scratch, whereas our goal is to simply "wake up" a vehicle that is already warmed up, idling, and ready to drive. Model-weight loading is one major source of cold-start latency. In an earlier post, we covered how Parasail combines fastsafetensors, O_DIRECT, and io_uring to move weights from disk to GPU memory faster while preserving warm-start performance. This took us a long way.
However, starting a vLLM process involves quite a lot of state (buffers and data structures) and setting up this state makes up a significant portion of the startup time. What if there is a way to save the state of both the CPU and GPU so that we could seamlessly and instantaneously resume from this state? This is exactly what we set out to do with our model snapshotting. This blog post will delve into some of the details involved in making this work in production.
The core idea behind this is to serialize the states of the GPU and host CPU and then store them on a persistent device (disk/SSD). We will call these states “snapshots” because we are essentially capturing snapshots of a vLLM process. Now, when we want to autoscale to new replicas, recover from a crash/failure, clone a deployment, or resume from pausing a deployment, the snapshot can be read and the state can be recovered. Overall we observe a 3-5x times reduction in startup time when reading from the snapshots when compared to loading/starting from scratch.
What It Takes to Snapshot a Running vLLM Process
Once model weights can load efficiently, the remaining startup cost comes from reconstructing the running vLLM process itself: buffers, file descriptors, CUDA/NCCL state, readiness state, and other runtime structures. Snapshotting targets that layer.
The GPU snapshot preserves the expensive state, but the process also needs host-side runtime state to be valid when it comes back: open file descriptors, event loops, NCCL communication setup, readiness flags, and the bookkeeping vLLM uses to serve requests. If any of that state is stale or unsafe to restore, the process may come back with weights loaded but still unable to serve traffic correctly.
Saving process state across GPU and CPU layers involves three key steps:
- Snapshot the GPU state.
- Snapshot the host CPU state.
- Add vLLM plumbing for safe suspension and resumption.
These implementation details sit underneath a broader architecture tradeoff: production inference teams often want the latency and isolation of dedicated endpoints, without keeping excess GPU capacity warm forever. We covered that tradeoff in our guide to choosing between serverless, dedicated, dedicated serverless, and batch inference. Snapshotting is one of the system's primitives that makes faster scaling and recovery possible inside that architecture.
NVIDIA's cuda-checkpoint handles the GPU state by toggling a snapshot for a specific PID. While straightforward, the host CPU side is more difficult. Standard tools like CRIU (Checkpoint/Restore In Userspace) typically work for Linux applications, but fail here due to a fundamental incompatibility with IO_URING.
Challenge 1: CRIU <-> IO_URING incompatibility
To visualize this conflict, imagine a library where IO_URING has very strict rules about how books (file descriptors) are checked out to ensure they aren't lost. The copier (CRIU) only knows how to copy these books using an older checkout method that the library has explicitly banned to prevent errors. Because they can't agree on how to handle the books, the copying process fails.
The tricky part is that CRIU has to capture the process’s open file descriptors, but some of those descriptors are owned by IO_URING-backed libraries that do not allow CRIU’s usual transfer mechanism.
Aside from loading model weights, several vLLM dependencies (including uvloop, torch, grpc, and nccl) utilize IO_URING, making this incompatibility a critical blocker. The root of the issue lies in how CRIU captures file descriptors (fds). Traditionally, it injects a parasite code that uses SCM_RIGHTS over a Unix Domain Socket to transport fds. However, IO_URING maintainers have banned this method because it creates long-lived references that trigger reference cycle deadlocks. We circumvented this by implementing a modern, "safe" alternative: using pidfd_getfd() to access the necessary descriptors. This required a ~600LOC patch to CRIU, but successfully bypassed the deadlock-prone SCM_RIGHTS mechanism.
Challenge 2: vLLM Support for safe Suspension and Resumption
Achieving true "process hibernation" requires more than just pausing; it requires the vLLM process to be in a perfectly serializable state. Taking inspiration from community RFCs (https://github.com/vllm-project/vllm/issues/34303), we implemented specialized helpers to manage this transition. Sleeping keeps the process alive and GPU memory allocated. Suspension writes the restorable state to disk, so the process can be brought back later without paying the full startup path again.
The suspension process involves a thorough "tidying up" phase: /snapshot/suspend tears down the NCCL ring, closes sensitive /dev/nvidia* file descriptors, and flips the readiness flag, leaving a clean state for criu dump to capture. Upon resumption, the process undergoes a "rebuilding the infrastructure" phase: /snapshot/resume restores the GPU memory via cuda-checkpoint, rebuilds NCCL with fresh rendezvous URLs, and restores service readiness. Crucially, because weights are preserved in the GPU snapshot, the restore is sub-second for TP=1.
Snapshot Restore Cuts Startup Time by up to 5x
Here, we show a glimpse of results from benchmarking experiments we ran on several different models with different tensor parallelisms (TP). We ran our benchmarks on machines equipped with H200 GPUs.
This table compares the traditional "cold-start" baseline startup time against the optimized "restore-from-snapshot" time, highlighting the speedup factor for each model. The results show a clear improvement in startup efficiency across all tested configurations. Notably, the smaller Qwen2.5-7B model sees the most significant gains, with a 4.11x reduction in startup time, while larger models continue to show substantial speedups, validating our snapshotting strategy for diverse model architectures and tensor parallelism levels.
What This Means for Production Inference
In an earlier post, we covered how Parasail reduces model weight loading time with IO_URING and page-cache bypassing. Process snapshotting builds on that foundation: instead of starting a vLLM process from scratch, Parasail can restore CPU and GPU state from a saved snapshot and bring replicas online much faster.
By shifting our approach from rebuilding vLLM processes from scratch to resuming them from saved states, we have effectively removed the primary bottleneck in our cold-start latency. Through the implementation of pidfd_getfd to resolve the compatibility deadlock between CRIU and IO_URING, and by building the necessary hooks for true "Process Hibernation," we have enabled a snapshotting workflow that is both production-ready and highly efficient.
The impact of this work is tangible: we are consistently seeing a 3-5x reduction in startup time. This capability gives us the flexibility to autoscale replicas rapidly, recover near-instantaneously from failures, and pause or clone deployments without the overhead of reloading heavy model weights.
In practice, that changes what autoscaling can feel like for production inference workloads: fewer cold-start delays, faster recovery from failures, and more flexibility to pause, clone, or scale deployments without paying the full startup cost each time.
We have successfully turned the "cold start" problem from a persistent roadblock into an optimization challenge we can manage.
Acknowledgments
Thank you to fellow Parasail engineers Pooya Davoodi, Adam Lugowski, and Zifei Tong for their support in producing this analysis.