Batch inference
Batch inference submits a bounded collection of model inputs for asynchronous processing.
On this page
The work is queued and scheduled outside the live request path, and results are retrieved or delivered later. It fits workloads with a completion window when no person or product is waiting for an immediate answer.
Batch inference runs a job outside the live request path
A batch job starts with a defined set of inputs. The platform validates the work, places it in a queue, schedules capacity, and processes the requests. The caller tracks the job state and retrieves the outputs later or receives them through a supported delivery method. Jobs may start on demand or run on a schedule.
Durable request IDs or another correlation mechanism keep outputs tied to inputs and isolate failed requests. Teams can then identify missing or failed outputs, catch mapping errors early, and keep invalid results out of downstream workflows. They still need visibility into queue time, job progress, partial failures, retries, and result integrity. Exact input formats, delivery methods, and recovery controls vary by provider.
How real-time, async, and engine-level batching differ
These terms describe different layers of an inference system, although vendors sometimes use them differently. Offline inference often names the broader workload category; a batch job is one way to execute it.
Real-time inference: The application holds a live request open for the result and prioritizes per-request responsiveness.
Batch inference: A bounded collection is processed asynchronously, with a focus on job completion and aggregate throughput.
Async inference: The caller receives an ID or later delivery instead of holding a connection open. The work may contain one request or many.
Dynamic or in-flight batching: The serving engine combines or admits requests internally to improve throughput within its latency limits.
A real-time endpoint or a batch service may use dynamic batching underneath. The engine mechanism does not determine whether a user is waiting for the result.
Choose batch for workloads with a completion window
Evaluations, embeddings, dataset generation, document enrichment, classification, research jobs, and backfills can all fit a batch workflow when their outputs are not needed immediately. The scheduler can then organize work around the full job. Job-level scheduling can improve aggregate throughput, capacity use, and cost efficiency, although the result still depends on workload shape, model, hardware, queue policy, and provider.
The tradeoff is delayed access to each result, so each job still needs a completion target that downstream work can plan around. Use real-time inference when a person or product needs the answer now; many systems use both modes for different paths.