Prefill vs. decode in LLM inference
Understand how prefill and decode shape time to first token, streaming performance, KV-cache pressure, and LLM serving decisions.
GuidesFaster autoscaling for vLLM: Restoring from snapshots instead of starting cold
Cold-start latency is one of the biggest bottlenecks when scaling inference. Parasail's model snapshotting saves and restores CPU and GPU process state to bring vLLM replicas online 3-5x faster than rebuilding from scratch.
EngineeringMaking cold start latencies go brrrr: a multi-pronged approach (Part 1)
We walk through how we combined fastsafetensors, O_DIRECT, and io_uring to get fast cold-starts and fast warm-starts on the same stack.
Engineering