Most companies dump batch work into a low-priority, timeout-free queue that gets drained when the real-time (RT) fleet hits a trough (like Sunday 2 AM). That queue runs through whatever config already serves real-time traffic. It's the wrong config for batch throughput, but next to a GPU sitting fully idle, running suboptimally feels like efficiency.
The problem is, the RT config is tuned for a different job. It protects tail latency with conservative token budgets and KV cache headroom held back for bursty traffic, none of which batch needs.
When your backlog is small, that inefficiency doesn't matter. But as your backlog grows, and it will, your idle window doesn't. That's when optimizing LLM inference for batch throughput comes into play:
TL;DR
For high-volume offline batch work, the single biggest lever is packing the GPU. Raise max_num_batched_tokens to 16,384 or higher and gpu_memory_utilization to around 0.95, use vLLM's V1 engine (the default as of 2026) through the offline LLM/llm.generate path or Ray Data LLM, and let continuous batching and chunked prefill do the rest.
vLLM is the right default engine, except for certain use cases. TensorRT-LLM wins on peak throughput for stable, NVIDIA-only deployments (10 to 20% faster at high concurrency), at the cost of a compile step measured in tens of minutes. SGLang wins when requests share prefixes.
Speculative decoding is a latency tool, not a throughput tool. At high batch sizes it's a wash at best and a net loss at worst, because the GPU is already compute-bound. The practical rule of thumb: disable it above a batch size in the low tens. vLLM exposes -speculative-disable-by-batch-size for exactly this.
Reordering is a free win in batch serving. Sorting requests so shared prefixes sit together (to hit the prefix cache or RadixAttention) and bucketing by similar length both raise throughput. Research systems built around this idea, BlendServe and BatchLLM, beat stock vLLM and SGLang by 1.1x to 1.44x. Multimodal batch inference is now first class in vLLM too, with encoder-output caching and batch-level data parallelism for vision encoders.
What makes offline batch inference different from real-time serving
Batch inference (also called offline inference) runs against a static dataset you already hold rather than serving a live stream of requests arriving one at a time. Because you can see the entire dataset before execution begins, you can reorder the queue however you want before you run it. There is no per-request latency target to hit, which unlocks a second efficiency play: you can tolerate a configuration that is suboptimal for any single request, because you are optimizing for the job's total throughput. Online inference has neither of these options, since each request arrives individually and must be served immediately.
This is the core difference between real-time and batch inference configuration. Real-time prioritizes latency on any single request while batch optimizes for throughput across the whole job.
vLLM V1 batching: Current state, config, and offline mode
vLLM's V1 engine became the default during vLLM's 2025 release cycle and remains the standard baseline through 2026 vLLM's alpha release V1 blog post puts the V0-to-V1 gain at up to 1.7x higher throughput without multi-step scheduling, with even larger gains for vision-language models.
In their September 2025 architecture writeup, they explain where that gain comes from: lower CPU overhead across the stack, an isolated EngineCore execution loop, a multiprocess and ZMQ architecture that overlaps tokenization, multimodal input processing, de-tokenization, and request streaming with model execution, and incremental, diff-based state updates between scheduler and workers.
V1 also switched the default preemption mode to RECOMPUTE instead of SWAP and dropped the old --swap-space flag entirely. Chunked prefill and prefix caching are on by default whenever possible.
Five core vLLM throughput knobs
Per vLLM’s optimization and tuning docs, these are the five knobs that have the biggest impact on throughput:
The three highest-impact throughput knobs are gpu_memory_utilization, max_num_batched_tokens, and max_num_seqs as they determine how to split the pool of GPU memory.
gpu_memory_utilization decides how much VRAM is available for the KV cache in the first place. max_num_batched_tokens and max_num_seqs then compete for that same pool from different angles. One caps tokens processed per scheduler pass, the other caps concurrent sequences. vLLM has to fit both inside whatever cache gpu_memory_utilization carved out. Raise the memory ceiling and there's more room for both. Raise either limit without raising the memory ceiling, and something else has to give.
None of this gets derived from a formula in practice. The real workflow is to push max_num_batched_tokens and max_num_seqs up, rerun, and wait for vLLM to throw a CUDA out of memory exception, then dial back one notch. The crash tells you exactly where your KV cache budget runs out for your model and hardware combination faster than multiplying batch size by sequence length by layer count by hand.
Constraints to know
max_num_batched_tokens has to be at least as large as both max_num_seqs and max_model_len when chunked prefill is off. The product of max_num_seqs and max_model_len also has to fit inside the KV cache budget.
Watch the Prometheus preemption counters. Frequent PreemptionMode.RECOMPUTE warnings mean the KV cache is too small. To fix this, raise gpu_memory_utilization, lower max_num_seqs or max_num_batched_tokens, or increase tensor_parallel_size.
Community example
One community ‘big batch offline’ recipe comes from a practitioner generating 405B FP8 outputs for millions of prompts (vLLM issue 8513): --max-model-len 8192 --gpu_memory_utilization 0.93 --block-size 32 --max-num-seqs 512, plus --disable-log-requests.
Offline batch inference mode
For batch specifically, use the offline LLM class (vllm/entrypoints/llm.py) as the explicit offline batch-inference entry point. LLM.generate() and LLM.chat() take a whole list of prompts and return RequestOutput objects. This offline mode skips the HTTP and API server overhead entirely. One practitioner measured the online server running more than 2x slower than offline batch on identical data and model (vLLM discussion 11488).
For distributed batch jobs, Ray Data LLM wraps vLLM's async engine and is the recommended path for large-scale offline inference. It adds streaming execution for datasets larger than cluster memory, automatic sharding and load balancing, and job-level checkpointing for fault tolerance, and reports 2x the throughput of vLLM's synchronous engine at production scale per Anyscale's own benchmarks.
A typical Ray Data LLM config sets engine_kwargs={"enable_chunked_prefill": True, "max_num_batched_tokens": 4096, "max_model_len": 16384} alongside a batch_size (64, for example) and a concurrency setting. V1 also adds LLM.enqueue, for queuing prompts without blocking.
Choosing an engine: vLLM vs. TensorRT-LLM vs. SGLang
vLLM is the right default for most use cases with the broadest hardware support (NVIDIA, AMD ROCm, Intel, TPU), an OpenAI-compatible server, and the largest community. Two engines beat it in specific situations.
- Reach for TensorRT-LLM when you're running a single stable model in long-term NVIDIA production and raw throughput is the deciding factor.
- Reach for SGLang when your workload has shared prefixes.
TensorRT-LLM's throughput edge in stable, NVIDIA-only deployments comes at the cost of a compile step that runs into the tens of minutes. Two knobs drive most of that throughput: max_batch_size (default 2,048) caps simultaneous requests, and max_num_tokens (default 8,192) caps the token budget per scheduler iteration, the same in-flight batching idea vLLM implements as max_num_seqs and max_num_batched_tokens.
NVIDIA's own tuning case study on a 4xH100 Llama 3.3 70B deployment found sweeping max_batch_size from 64 to 512 raised throughput from 1,944 to 2,467 tokens per second, with 2,048 overshooting and dropping it back to 2,044. Cumulative batch tuning took the same deployment from 1,564 to 2,474 tokens per second (a 58% gain) while cutting inter-token latency from 31.3 to 14.7 milliseconds.
SGLang wins when requests share prefixes. Its RadixAttention scheduler sorts the waiting queue by matched prefix length, storing KV cache in a radix tree and automatically reusing any previously seen prefix.
This delivers roughly 29% higher throughput on H100 with Llama 3.1 8B when prefixes repeat (16,200 vs. 12,500 tok/s) per Particula Tech. For prefix-heavy workloads like RAG or multi-turn chat, throughput can increase by as much as 6.4x. But for batch jobs with unique prompts: classifying a dataset, summarizing a document set, generating synthetic outputs from unique seeds, that advantage disappears.
Why KV cache is the real constraint
All three engines above are ultimately managing the same constraint: how much GPU memory the KV cache has to work with.
KV cache ≈ batchsize × seqlen × numlayers × 2 × hiddendim × precisionbytes
According to research by Kwon et al., existing systems waste 60 to 80% of memory due to fragmentation and over-reservation. PagedAttention, vLLM's paging-inspired KV cache memory manager, cuts that waste down to under 4% to achieve near optimal memory usage.The original vLLM blog post credited PagedAttention with up to 24x higher throughput than Hugging Face Transformers on the same hardware.
Each engine exposes a different set of levers to manage the KV cache constraint:
vLLM: gpu_memory_utilization sizes the KV pool, max_num_seqs and max_num_batched_tokens cap concurrent KV demand, max_model_len caps per-sequence KV, tensor_parallel_size or pipeline_parallel_size spreads weights across GPUs to free KV memory, and FP8 or AWQ quantization shrinks weights (or the KV cache itself) to make more room. Monitor preemption counters (RECOMPUTE in V1), this is the release valve when KV runs out. On modern accelerators, FP8 is the sweet spot right now. It frees memory, which means a bigger KV cache, more concurrent sequences, and higher throughput.
TensorRT-LLM: Paged KV cache with use_paged_context_fmha, FP8 KV cache, and max_num_tokens, with in-flight batching allocating KV dynamically.
SGLang: Radix-tree KV management with LRU eviction, plus hybrid or offloaded prefix cache pools (host memory or SSD) to raise hit rates.
Prefix caching and reordering to hit it
Prefix caching pays off when requests share a prefix such as a common system prompt, few-shot examples, or shared document or corpus.
vLLM's automatic prefix caching: enable_prefix_caching=True (turned on by default since v0.6.0) hashes KV blocks by a chain hash (parent hash block tokens optional LoRA/multimodal/cache_salt keys) and reuses them on exact prefix matches. vLLM ships a dedicated automatic_prefix_caching_offline.py example.
Hit rate is highly workload-dependent: multi-turn conversation and QA show 90.7% and 81.8% prefix-cache hit rates, while summarization and code completion show 6.3% and 0.1% (SwiftCache). For the latter, turning prefix caching off costs nothing.
The key insight: Because offline batch has no per-request latency to protect, you can sort requests to maximize cache reuse. This is the explicit thesis of two batch-first systems, on top of what SGLang already does natively:
SGLang: its cache-aware scheduler sorts the waiting queue longest-shared-prefix-first, proven equivalent to a depth-first search of the radix tree and optimal for cache hit rate in the offline case.
BatchLLM: identifies common prefixes globally ahead of time, instead of relying on a reactive LRU cache that may prematurely evict a prefix about to be reused, schedules requests sharing a prefix together, and applies throughput-oriented token batching on top, beating vLLM and SGLang by 1.3 to 10.8x (8.67 vs. 6.71 tok/s on an A100 test).
BlendServe, presented at MLSys and ASPLOS 2026: exploits relaxed offline latency to reorder and overlap compute- and memory-intensive requests via a resource-aware prefix tree, for up to 1.44 times over vLLM and SGLang, reaching 90% of optimal throughput.
On the vLLM side, issue 24394 notes stock vLLM doesn't reverse-release blocks across adjacent requests, causing dirty-cache eviction of prefixes about to be reused. The Echo paper shows the same LRU flushing hurts offline hit rates under mixed online and offline load.
The practical takeaway: Sort requests so those sharing a prefix are contiguous, group by system prompt, document, or few-shot template, keep prefix caching on, and consider SGLang if prefix sharing dominates. If prefixes are unique, prefix caching is a no-op: don't expect gains.
Length bucketing
Grouping requests of similar length reduces padding and bubble waste, and it's a documented throughput win. The problem is that in a batch, short sequences idle while waiting for the longest ones to finish.
Frameworks that reserve KV cache for the maximum possible output length waste memory and constrain batch size (a problem the Robust Length Prediction survey lays out in detail). Request length distributions are heavily long-tailed, most short and a few very long, so a single long request can cause head-of-line blocking.
Evidence and techniques:
BucketServe built on vLLM: Groups requests into size-homogeneous buckets by sequence length and dynamically adjusts batch size to GPU memory, for up to 3.58x higher throughput than UELLM and 2× greater load capacity vs. baselines while maintaining SLO.
Length-prediction sort schedulers: S³, TetriInfer (an OPT-125M classifier), SSJF (a BERT proxy for shortest-job-first), and Learning to Rank (NeurIPS 2024, predicts relative output-length ranks and sorts the batch accordingly) all use bucketing or sorting to approximate shortest-job-first and cut head-of-line blocking.
Bucketing is mandatory on NPUs: Static-shape compilation targets predefined lengths, for example 256, 512, 1024, and 2048, producing a step-function latency curve.
Practical takeaway: For offline batches, sort by input length and process in length-homogeneous buckets. Can you predict how long a request is? Probably, since you know your data. If you can, you should: sort by output length too. This reduces intra-batch variance so batch step time isn't dominated by the longest sequence, and it cuts wasted KV reservation. Combine carefully with prefix grouping: the two orderings can conflict, which is exactly what BlendServe's resource-aware prefix tree is designed to resolve.
Multimodal batching
Batching multimodal requests for images, audio, or video alongside text adds a wrinkle. Modality cost varies request to request, and encoding can dominate latency before generation even starts.
Optical Character Recognition (OCR) tasks like document parsing or scene text recognition benefit from multimodal batching because batching groups many visual inputs together so they can be processed more efficiently in one pass. vLLM V1 handles this through encoder-output caching, encoder parallelism, and unified omni-model serving:
Encoder-output caching: vLLM’s EncoderCacheManager caches multimodal encoder outputs, vision embeddings and the like, keyed by a hash of the multimodal item itself. If the same image or video shows up across requests, or across chunked-processing stages of the same request, vLLM reuses the encoder output instead of rerunning the encoder. That matters because image encoding can dominate TTFT and account for more than half of total latency on vision-language models with high-resolution inputs.
Batch-level data parallelism for encoders: The --mm-encoder-tp-mode data flag replicates the lightweight vision encoder's weights across GPUs and load-balances the image batch, instead of tensor-sharding the encoder and paying an all-reduce cost on every layer for a comparatively small piece of the model. vLLM's own numbers put the gain at roughly 10% for throughput and time to first token at tensor-parallel size 8, rising to 40% for vision encoders built on hardware-unoptimized Conv3D operations. The tradeoff is slightly higher memory: replicated encoder weights cost more than a sharded copy. This is worth enabling once tensor_parallel_size reaches 4 or higher.
Omni models: vllm serve <model> --omniRed Hat on Qwen3-Omni) brings text+image+audio+video behind one OpenAI-compatible API. The autoregressive stages reuse PagedAttention, continuous batching, CUDA graphs and the scheduler. Multimodal placeholder tokens are replaced by encoder embeddings during prefill, the KV cache is modality-agnostic, and prefix caching carries over plus a separate cache for cross-stage hidden-state tensors.
One benchmarking note: vllm bench throughput has a flag to enqueue every request before scheduling starts, specifically because image rendering and preprocessing can dominate a naive benchmark's setup time and make the engine look slower than it is.
Multimodal inputs also inflate KV and encoder-cache memory beyond what text-only requests need. Encoder caching and batch-level data parallelism are the two levers that do the most for throughput here, and BlendServe is designed exactly for this: workloads where compute and memory demands vary sharply by modality.
Speculative decoding: Skip it at batch scale
Speculative decoding speeds up low-batch, latency-sensitive serving. At high batch sizes it and its benefit decays toward, and past, break-even as batch size grows. At low batch sizes decode is memory-bandwidth-bound, so the extra draft/verify FLOPs are "free." As batch size grows, decode becomes compute-bound, and every speculative token adds a verification workload that competes with the target model for the same compute.
An energy efficiency study (a good proxy for utilization efficiency) found speculative decoding cut energy use by up to 29% at batch sizes of 16 or below, then reversed course, using roughly 26% more energy than plain autoregressive decoding at batch 128. A practitioner analysis places the throughput break-even around batch size 32
The rule of thumb: leave speculative decoding off for maximum-throughput batch jobs, or cap it with vLLM's --speculative-disable-by-batch-size around 32 and let the engine fall back automatically.
Inference economics: Why "good enough" beats "perfectly tuned"
Most companies approach batch inference with a ‘good enough’ mentality. The GPU is active when it would otherwise be idle and no one is waiting on a response, so the suboptimality of an RT-tuned config doesn't really matter. Batch API tiers monetize exactly this.
The 24 hour turnaround window is what lets providers optimize resource allocation instead of committing capacity up front. That turnaround window, not just the discount, is what lets providers optimize resource allocation instead of committing capacity up front.
The hidden catch: A batch API can advertise a generous file size and request count, but if the upload or download connection only stays open for a minute, the file-size ceiling only holds if the transfer finishes before that window closes.
OpenAI users have hit these connection timeouts in the wild, a 2.1 GB batch output download failing with a 504 timeout after 60 seconds, well under the documented file size ceiling. Few connections are fast enough to move a multi-gigabyte file in 60 seconds, so the real limit is whatever fits inside your transfer window at your actual bandwidth, not the number printed in the docs. It's a footgun: two generous-looking limits gated by a third, much smaller one that nobody advertises next to them (because it shoots the other two in the foot).
Beyond batch APIs
Idle-GPU monetization products follow the same logic. FriendliAI's InferenceSense runs paid inference on otherwise-idle GPU capacity and preempts within seconds when that capacity gets reclaimed, on a custom C++ engine FriendliAI claims runs 2 to 3 times the throughput of a standard vLLM deployment. ScaleOps does the same from the scheduling side: policy-driven placement of batch jobs onto available or spot capacity while still meeting SLAs.
The same logic shows up in how teams schedule batch work, not just how providers price it. Overnight and weekend follow-the-sun batch runs, timed to whichever region's real-time traffic is in its own trough, are a standard cost-optimization pattern. This happens alongside spot instances priced around 70% below on-demand for workloads that can tolerate interruption.
Queue-depth-based autoscaling rounds out the playbook: scale batch capacity to the backlog, not to a fixed schedule. Batch can always tolerate higher latency in exchange for better resource efficiency.
When perfectly tuned is worth it
When the marginal hardware cost is close to zero (e.g. idle capacity you've already paid for) even a throughput-suboptimal RT config captures most of the savings. The counterpoint is that a tuned batch configuration with prefix sorting, length bucketing, the baseline knob table, can clear 2 to 4x the tokens per idle window.
If your backlog already clears inside the idle window, the untuned configuration is good enough. If it doesn't clear in time, tuning is what closes that gap.
Recommendations
Stage 1 — Baseline (do this first). Use vLLM V1 via the offline LLM/llm.generate path (or Ray Data LLM for multi-node). Set gpu_memory_utilization=0.95, max_num_batched_tokens=16384, and a large max_num_seqs (start 256–512). Use FP8 weights on Hopper/Blackwell. Keep chunked prefill and prefix caching on (defaults). This alone should put GPU utilization in the 75–90% range. Benchmark with vllm bench throughput and watch preemption counters — frequent RECOMPUTE means shrink max_num_seqs or raise TP.
Stage 2 — Exploit reordering (free wins, offline only). Since you can reorder: (a) sort requests to group shared prefixes contiguously and confirm prefix-cache hits via vllm:gpu_prefix_cache_hits/_queries; if prefix sharing dominates, evaluate SGLang. (b) Bucket by sequence length to cut intra-batch variance and KV waste. If both matter and conflict, that's the BlendServe/BatchLLM problem — worth a prototype if throughput is business-critical.
Stage 3 — Turn OFF the latency tricks that hurt throughput. Disable speculative decoding for batch (or set --speculative-disable-by-batch-size 32). Don't tune for TTFT/ITL. Push max_num_batched_tokens toward 32k and measure diminishing returns.
Stage 4 — Consider a dedicated engine only if justified. If you have one stable model, are NVIDIA-only, and batch volume is large enough to amortize a 28-minute compile, benchmark TensorRT-LLM (expect 10–20% at high concurrency; tune max_batch_size/max_num_tokens via grid search). For multimodal at TP≥4, enable --mm-encoder-tp-mode data.
Thresholds that change the plan:
- If prefix-cache hit rate 10% → prefix grouping/SGLang won't help; skip it.
- If spec-decode acceptance ≥0.75 and your effective batch stays 16 → spec decode may still pay; otherwise off.
- If your batch backlog regularly exceeds trough/idle capacity → invest in a dedicated batch config (Stages 2–4); if backlog always clears within the idle window on the RT config → the author's "it doesn't matter" thesis holds and you can stop at Stage 1
- If preemptions persist at
gpu_memory_utilization=0.95→ raisetensor_parallel_sizeor quantize (weights and/or KV).
Caveats worth keeping in mind
Every benchmark cited here is workload-specific. The Spheron and Particula Tech H100 figures reflect one model, Llama 3.3 70B FP8, at one set of prompt lengths, on one GPU. NVIDIA's own tuning case study is first-party data from NVIDIA's documentation, not an independent benchmark. None of these numbers substitute for benchmarking your own model, hardware, and traffic pattern.
The research systems, BlendServe, BatchLLM, BucketServe, and the batch-speculative-decoding work covered here, are published papers, not turnkey production engines. Their reported gains are against specific baselines under specific conditions and may not reproduce on your stack without real engineering effort. SGLang's built-in cache-aware scheduler is the one production-ready version of the reordering idea available today; the rest are a ceiling to aim for, not a drop-in install.
Defaults drift fast in this space. The version-specific numbers in this piece, vLLM's max_num_seqs default of 1,024, TensorRT-LLM's max_batch_size default of 2,048, and the rest, reflect 2025 and mid-2026 documentation and change release to release. Confirm defaults against whatever version you deploy before trusting them.
And a structural note: There is no single dominant open-source engine built purely for offline batch. In practice, that means vLLM's offline path or Ray Data LLM, or SGLang if your workload shares prefixes, plus whatever reordering you implement yourself. BlendServe and BatchLLM show what's possible, but neither is broadly deployed in production the way vLLM, TensorRT-LLM, and SGLang are.
How this maps to Parasail
Our Batch Processing product runs the managed version this guide describes for you. We operate an offline engine using continuous batching on a multi-cloud GPU fleet, exposed through an API that's 100% compatible with OpenAI's batch format. That format supports up to 1 million requests per batch, a 1 GB job size, and a 60-second transfer window, whichever limit hits first.
Batch runs at 50% of Parasail's serverless per-token pricing. The savings beyond the batch discount depend on the open-weight model you can use for the task and the workload's input/output mix.
If you already have OpenAI batch pipelines, the third-party openai-batch-helper library applies directly. For high-volume, non-real-time inference on open-weight models, batch is the easiest win available on your inference economics, and Parasail makes sure it's deployed optimally.
If you're sizing a batch workload and weighing self-hosted vLLM against a managed engine, talk to one of our engineers. We'll work through the throughput and cost math with you.
Frequently asked questions
Does running batch inference locally or offline improve privacy and security?
Yes. Offline and self-hosted batch inference keeps prompts and outputs inside your own infrastructure, so sensitive data (PII, proprietary code, regulated records) never transits a third-party API. This is the primary reason regulated workloads (e.g. healthcare under HIPAA, EU data under GDPR) run offline batch on dedicated hardware rather than shared public endpoints.
The tradeoff is that you own the tuning, the KV cache budget, and the queueing yourself. A third-party batch API can still meet these needs if the provider offers data-isolation guarantees and no-retention terms, but self-hosting removes the trust dependency entirely.
When should I use a batch API instead of self-hosting vLLM offline?
Use a batch API when your volume doesn't justify owning GPU capacity and tuning, or when a 50% discount and variable latency fit your SLA for the workload. Self-host when you need data to stay in your own environment, when you want full config control over the engine knobs like chunked prefill size and KV cache allocation, or when your backlog regularly exceeds a provider's turnaround window. Neither option is universally better - when are they ever? - but the overwhelming majority of users are probably better off consuming batch through an API like Parasail’s, since you get to make use of our expertise (and otherwise idle fleet) at a huge discount.
How do I keep prompt data from leaking across batch requests that share a prefix cache?
Prefix caching reuses KV blocks only across requests within the same engine process and matches on exact prefix hashes, so cross-tenant leakage is not a concern in a single-tenant offline job. If you multiplex tenants through a shared engine, isolate them into separate engine instances or use vLLM's cache_salt parameter to prevent unintended cross-request cache hits. Treating tenant isolation as an engine-level concern rather than an application-level one is the safer default.
How do you enforce structured or JSON-formatted outputs during offline batch generation?
Pass a GuidedDecodingParams object to SamplingParams or use the convenience fields guided_json, guided_regex, guided_choice, or guided_grammar directly on SamplingParams. vLLM will route the constraint through the configured guided decoding backend; xgrammar is the default in current vLLM. The backend compiles the schema, regex, choice set, or context-free grammar into a token mask that is applied at each decode step, so only tokens that keep the output valid are sampable.
This guarantees parseable output structurally rather than relying on prompt engineering. For batch jobs where every request shares the same schema, grammar compilation is a one time cost amortized cleanly across the full static dataset. xgrammar keeps per-token masking overhead low enough that structured decoding runs close to free at batch scale, which is a meaningful improvement over older token-masking approaches that imposed non-trivial per-step latency.