Batch inference wants a different shape from real-time serving. Real-time endpoints protect TTFT and ITL. Offline jobs optimize tokens cleared per GPU-hour. They also need fault tolerance and predictable idle-window fit.
The default stack in 2026 is vLLM V1 through the offline LLM API or Ray Data LLM. TensorRT-LLM can beat it on stable NVIDIA-only fleets. SGLang earns its place when many requests share prefixes.
Start with the KV cache and the per-iteration token budget. Then sort for prefix reuse and bucket by length when your workload has a long tail. Speculative decoding belongs at low concurrency. At high concurrency it burns compute you wanted for larger batches.
1. vLLM for batch
vLLM V1 is the right starting point for most offline workloads. The V1 engine became the default during the 2025 release line and is the standard baseline in 2026. The vLLM team reports up to 1.7× higher throughput than V0 without multi-step scheduling, mostly from lower CPU overhead in the execution path.
V1 isolates the EngineCore execution loop and uses a multiprocess/ZMQ architecture. It overlaps tokenization and multimodal preprocessing with model execution. It also overlaps detokenization and request streaming. vLLM maintainers changed the default preemption path: V1 uses RECOMPUTE and removes the old --swap-space path. It enables chunked prefill and prefix caching whenever the workload supports them.
For high-volume offline work, use the offline LLM class and LLM.generate() or LLM.chat(). That path processes a list of prompts and returns RequestOutput objects without HTTP server overhead. In vLLM issue #11488, one practitioner measured server inference at more than 2× slower than offline batch on the same data and model.
For distributed jobs, Ray Data LLM wraps vLLM’s async engine. It streams datasets larger than cluster memory and shards work automatically. It handles load balancing, autoscaling, and job-level checkpointing. Teams have reported 2× throughput over vLLM’s synchronous LLM engine. V1 also adds LLM.enqueue for submitting prompts without blocking.
The baseline batch config is simple:
A practitioner in vLLM issue #8513 used this recipe to generate 405B FP8 outputs for millions of prompts: --max-model-len 8192 --gpu_memory_utilization 0.93 --block-size 32 --max-num-seqs 512 plus --disable-log-requests.
Watch the KV cache. max_num_batched_tokens must be at least max_num_seqs, and it must be at least max_model_len when chunked prefill is off. The product max_num_seqs × max_model_len must fit the KV-cache budget. Frequent PreemptionMode.RECOMPUTE warnings mean the KV pool is too small. Raise gpu_memory_utilization or increase tensor parallelism. If that does not clear the warnings, reduce max_num_seqs, reduce the token budget, or quantize weights and KV.
PagedAttention is still the reason these large batches work. Kwon et al. found prior systems used only 20.4%-38.2% of allocated KV memory for actual token states, implying 61.8%-79.6% waste. vLLM's PagedAttention reduced KV waste to near zero in the paper and delivered 2-4× throughput over FasterTransformer and Orca at the same latency level. The original vLLM blog reported up to 24× over Hugging Face Transformers.
The baseline expectation is clear: vLLM V1, FP8 weights on Hopper or Blackwell, chunked prefill, prefix caching, gpu_memory_utilization near 0.95, a large sequence ceiling, and the offline token budget should put GPU utilization in the 75%-90% range for high-volume batch work.
2. How the other engines compare
Continuous batching has become table stakes. The engines in this section all support continuous or in-flight batching and paged KV cache. They also support chunked prefill and FP8 quantization. Teams should choose based on hardware constraints, startup cost, prefix sharing, and tuning budget.
Spheron’s March 23, 2026 benchmark used a single H100 SXM5 80GB, Llama 3.3 70B Instruct FP8, unique prompts, vLLM v0.18.0, TensorRT-LLM v1.2.0, and SGLang v0.5.9.
TensorRT-LLM led at every concurrency level once compiled. The gap was about 8% at one request and about 13% at 50 concurrent requests. SGLang sat between vLLM and TensorRT-LLM on unique prompts, where RadixAttention has little to reuse.
Use this decision frame:
TensorRT-LLM's main throughput controls are max_batch_size and max_num_tokens. NVIDIA documents defaults of 2048 and 8192. The scheduler enforces both. NVIDIA's tuning guide recommends powers-of-two sweeps and grid-searching combinations.
NVIDIA's Llama-3.3-70B on 4×H100 case study shows why sweeping matters. Moving max_batch_size from 64 to 512 raised throughput from 1,944.30 tok/s to 2,466.79 tok/s. Pushing to 2048 dropped throughput back to 2,044.26 tok/s. NVIDIA calls 512 the sweet spot for that workload, with almost 20% more throughput than 64 and negligible latency impact. Their cumulative tuning raised throughput from 1,564 tok/s to 2,474 tok/s and cut inter-token latency from 31.3 ms to 14.7 ms.
TensorRT-LLM’s in-flight batching mixes context and generation phases. Chunked prefill requires use_paged_context_fmha and can use dynamic chunk sizing to avoid manual max-input-length tuning. TensorRT-LLM 1.0 made the PyTorch-based architecture the stable default and stabilized the LLM API, so teams can run from tensorrt_llm import LLM on Hugging Face or quantized checkpoints without the old convert-to-build cycle for every path. The stable line in mid-2026 is around 1.2.x, with 1.3.0 in RC.
SGLang’s differentiator is RadixAttention. It stores KV cache in a radix tree and automatically matches previously seen prefixes. Its cache-aware scheduler sorts the waiting queue by matched prefix length. The SGLang paper proves longest-shared-prefix-first equals a DFS of the radix tree and maximizes cache hit rate in the offline case. Particula Tech reports about 29% higher throughput on H100 when prefixes repeat, at 16,200 tok/s versus 12,500 tok/s, and up to 6.4× on prefix-heavy RAG or multi-turn workloads. Particula also names vLLM as the better default for batch processing with unique prompts.
Researchers report higher ceilings in systems such as NanoFlow, BatchLLM, and BlendServe. NanoFlow uses nano-batching and operation-level pipelining inside one GPU, reporting 1.91× throughput over TensorRT-LLM and 2.66× over vLLM on LLaMA-2-70B. The authors argue that common end-to-end LLM serving workloads are compute-bound. Modular MAX and Aphrodite are emerging alternatives, but the batch-specific research systems to watch are BatchLLM and BlendServe.
3. Speculative decoding: worth it?
Probably not for high-concurrency offline batch. Speculative decoding helps when decode is memory-bandwidth-bound and the draft model’s extra FLOPs fit into idle compute. Once batch size rises and the GPU becomes compute-bound, draft and verification work compete with the target model. Throughput goes net-negative.
The clean practitioner rule: leave speculative decoding off for maximum-throughput batch jobs, or set vLLM’s --speculative-disable-by-batch-size around 32 and let the engine fall back to standard decoding.
Papers and practitioner reports point in the same direction:
Correct batch speculative decoding remains hard because sequences accept different numbers of draft tokens per step. That desynchronizes position IDs, attention masks, and KV-cache state. The arXiv 2510.22876 paper reports that BSP and DSD can corrupt outputs above batch size 1. Its EXSPEC method groups equal-length sequences to preserve correctness, reaches 3× over batch-1 at batch 8, and degrades as length diversity grows.
Newer methods push the boundary. vLLM V1 reimplemented speculative decoding and supports EAGLE-3. Red Hat reports up to 2.5×. vLLM skips tree decoding because it degrades past synchronous use. Spheron’s Eagle-3 guide recommends --speculative-disable-by-batch-size 32 and monitoring vllm:spec_decode_draft_acceptance_rate, with a target of at least 0.75.
vLLM v0.16.0+ integrates P-EAGLE, which drafts all K tokens in one forward pass. Its advantage over EAGLE-3 persists at higher concurrency but shrinks. On HumanEval with GPT-OSS-20B on B200, the reported speedup moves from 1.55× at concurrency 1 to 1.23× at concurrency 64. EAGLE 3.1 reports 2.03× at concurrency 1 and 1.66× at concurrency 16 on Kimi K2.6 on GB200.
How to check your stack: run the same batch twice, once with standard decoding and once with speculative decoding plus the auto-disable threshold. Track throughput and failure first: output tokens per second and OOM rate. Add energy and acceptance rate when available. If acceptance stays at or above 0.75 and effective batch size stays below roughly 16, speculation can still pay. Otherwise, spend the memory and compute budget on larger batches.
4. KV-cache reuse
Offline batch gives you one freedom that online serving usually cannot: you can reorder requests. Use that freedom to place shared prefixes next to each other.
vLLM has enabled Automatic Prefix Caching by default since v0.6.0. It hashes KV blocks by a chain hash: parent hash, block tokens, and optional LoRA, multimodal, or cache_salt keys. Exact prefix matches reuse prior KV. vLLM includes an automatic_prefix_caching_offline.py example for this path.
Hit rate depends on traffic shape. SwiftCache reports 90.7% and 81.8% prefix-cache hit rates for multi-turn conversation and QA, compared with 6.3% and 0.1% for summarization and code completion. Prefix caching does little when prefixes are unique.
The practical move is simple. Group first by the strongest shared key, such as system prompt or document. Then add few-shot template, tenant prompt wrapper, or another stable prefix when those keys explain the workload. Run the batch with prefix caching on and check vllm:gpu_prefix_cache_hits and _queries.
SGLang bakes this idea into the scheduler through longest-shared-prefix-first ordering. BatchLLM and BlendServe show how far the same idea can go when the whole workload is offline.
5. Length bucketing
Length bucketing is worth testing when output lengths have a long tail. It is less universal than prefix reuse, because it depends on how well you can predict or approximate generated length.
The waste pattern is familiar. Short sequences finish while long ones keep the batch alive. A single long request can drive head-of-line blocking. Frameworks that reserve KV for maximum possible output length also waste memory, which shrinks the effective batch.
The BucketServe authors built it on vLLM to group requests into sequence-length buckets and adjust batch size to GPU memory. On its mixed offline workload at high load, it reports up to 3.58× throughput over UELLM and 1.31× over DistServe. It also reports about 2× higher load capacity in several settings and less than 1% bucketing overhead.
Other schedulers attack the same shape with prediction and sorting. S³ and TetriInfer predict or approximate output length. SSJF and Learning to Rank use the same signal to sort toward shortest-job-first behavior. On NPUs, bucketing becomes mandatory because static-shape compilation targets predefined lengths such as 256, 512, 1024, and 2048.
For an offline job, run the experiment in two passes. First, sort by input length and compare tokens per second, preemptions, and tail completion time. Second, if your task has predictable output length, add output-length buckets. Prefix grouping and length bucketing can conflict. BlendServe’s resource-aware prefix tree exists because this conflict is common in multimodal and mixed-template workloads.
6. Multimodal batching
Multimodal batch inference is now a first-class vLLM path. The main levers are encoder-output caching and batch-level data parallelism for encoders. Queue construction also matters because modality cost varies by request.
vLLM V1’s EncoderCacheManager caches multimodal encoder outputs, such as vision embeddings, by a hash of the multimodal item. If the same image or video appears across requests or chunked-processing stages, vLLM can reuse the encoder output instead of rerunning the encoder.
The encoder parallelism setting matters at higher tensor-parallel sizes. With --mm-encoder-tp-mode data, vLLM replicates the lightweight vision encoder weights across GPUs and load-balances the image batch. That avoids the all-reduce cost from tensor-sharding small encoders. vLLM docs report about 10% throughput and TTFT improvement at TP=8, and up to 40% for vision encoders that use hardware-unoptimized Conv3D ops. The memory tradeoff is higher replicated encoder weight storage. Use it when tensor_parallel_size >= 4.
For omni models, vllm serve <model> --omni puts text, image, audio, and video behind one OpenAI-compatible API. Red Hat’s July 1, 2026 Qwen3-Omni writeup describes the pattern: autoregressive stages reuse PagedAttention, continuous batching, CUDA graphs, and the scheduler. Multimodal placeholder tokens become encoder embeddings during prefill. The KV cache stays modality-agnostic. Prefix caching carries over, with a separate cache for cross-stage hidden-state tensors.
For benchmarking, vllm bench throughput includes a flag to enqueue all requests before scheduling. That helps multimodal tests because image rendering and preprocessing can dominate benchmark setup time. Keep the benchmark harness from measuring client-side image work as model throughput.
Multimodal requests also make resource balancing harder. Some requests spend more time in encoders. Others consume more decoder KV. BlendServe targets this case by overlapping compute-heavy and memory-heavy requests while preserving prefix sharing where possible.
7. How this works in practice
Teams usually dedicate hardware to one concern. Dedicate each node to either real-time or batch tuning. A scheduler will not find a universal optimum across both goals on the same node.
The common pattern is fake-batching. A low-priority, timeout-free queue holds batch requests. During low-utilization windows, for example Sunday at 2 AM, operators drain that queue into a real-time endpoint that would otherwise sit idle. When real-time demand returns, the batch drain pauses or gets preempted. The endpoint remains RT-tuned, so the batch run leaves throughput on the table. The economics can still work because the marginal hardware cost during the idle window is near zero.
OpenAI, Anthropic, and Google have packaged that queue pattern as batch API tiers. OpenAI’s Batch API offers asynchronous groups of requests with 50% lower costs, a separate pool of higher rate limits, and a 24-hour turnaround time, with input files up to 50,000 requests. Anthropic’s Message Batches API charges all usage at 50% of standard API prices, with most batches finishing in less than one hour, and supports batches up to 100,000 requests or 256 MB. Google’s Gemini Batch Mode lets teams submit large jobs and retrieve results within 24 hours at a 50% discount compared with synchronous APIs.
Providers use the 24-hour window to choose when to spend capacity. Dodo Payments describes the model as dynamic pricing that fills idle compute capacity at a discount while preserving premium pricing for real-time requests.
Idle-GPU monetization products follow the same queue logic. FriendliAI launched InferenceSense in 2026 to run paid inference on neocloud operators’ otherwise-idle GPUs and preempt within seconds when the operator reclaims capacity. FriendliAI frames it plainly: instead of letting GPUs be idle, operators monetize them by running inference. FriendliAI also claims 2-3× vLLM throughput through a C++ engine. ScaleOps productizes policy-driven scheduling that lands batch jobs on available or spot capacity while meeting SLAs.
This explains the apparent contradiction. An RT-tuned config is suboptimal for batch, but it can still capture free savings during idle windows. A dedicated batch config with larger token budgets, FP8, prefix sorting, and length buckets can clear 2-4× more work per window. If the backlog always clears before the idle window closes, the RT endpoint path is good enough. If the backlog spills over, dedicate batch hardware or a batch-tuned endpoint and run the offline stack described above.
Use operational capacity as the threshold:
Benchmark numbers remain workload-specific. Spheron and Particula use particular models, prompt mixes, and hardware. NVIDIA’s Blackwell figures are first-party, and NVIDIA marks some as preliminary. Research systems such as BlendServe, BatchLLM, BucketServe, NanoFlow, and EXSPEC show the ceiling against specific baselines. Treat the research figures as experiment targets for your own stack.