Production operations

AI inference service level agreement (SLA)

AI inference service level agreement (SLA) is a contract in which an inference provider commits to a measured level of service and owes the customer a remedy for missing that level.

On this page

An SLA tells you what the provider is accountable for when your endpoint fails or slows down. The commitment usually covers uptime, and some SLAs add a target for latency or generation speed.

An AI inference SLA states a metric, a target and a remedy

An AI inference SLA names the metric the provider measures, the target the metric has to reach, and the remedy you receive when the provider misses the target.

The metric is defined in the SLA's own wording. Amazon Bedrock's SLA calculates uptime as the average availability across the 5-minute intervals in a billing cycle. Only a request that returns a 500 error code counts against availability. A slow response that completes still counts as available. Bedrock’s availability commitment excludes issues caused by customer actions, including provisioned throughput scaling and VPC configuration.

The target is the level the metric has to reach over a period. An SLA with a 99.9% monthly uptime target allows 0.1% of the 43,200 minutes in a 30-day month, which is about 43 minutes of downtime.

The remedy is usually a service credit on your bill. Bedrock's SLA starts at a 10% credit when monthly uptime falls below 99.9%, and you have to file a claim to receive the credit.

An SLI is the measurement, an SLO is the target and an SLA adds the consequence

A service level indicator (SLI) is a measurement, a service level objective (SLO) is a target for that measurement, and an SLA is a contract that attaches a consequence to the target. Google's Site Reliability Engineering book gives a test for telling an SLO from an SLA. Ask what happens when the target is missed. A target with no explicit consequence is an SLO.

An SLI is the number your monitoring records, such as the share of requests in a month that returned without a server error. The metric in an SLA is an SLI that the provider and the customer calculate the same way.

Parasail provides service level objectives for its Dedicated service, as outlined in Section 3.2 of its Terms of Service. These objectives express the service levels Parasail aims to deliver and help set expectations for production workloads. For Elastic Endpoints and Dedicated deployments, Parasail targets 99.9% uptime.

Latency SLAs come with reserved capacity

Whether an SLA covers latency depends on whether your endpoint runs on shared capacity or on capacity reserved for you.

Shared capacity leaves a provider little room to commit to latency in an SLA. Microsoft's guide to provisioned throughput for Azure OpenAI lists no latency SLA for standard deployments, where capacity is shared across customers and throughput can vary with demand.

Reserved capacity gives the provider a fixed amount of compute to measure a latency SLA against. The same guide lists a latency target for each model on provisioned deployments, which hold a fixed amount of processing capacity for one deployment.

Before relying on an SLA, confirm which deployments, regions and requests it covers, how its metrics are measured, and how to claim a credit. For latency, check whether the target concerns the first token, generation speed or the full response. Bedrock requires supporting request logs and a credit claim by the end of the second billing cycle after the incident.

To compare how shared and dedicated deployment modes handle latency requirements, read Parasail's guide to choosing a managed inference architecture.

All terms