Scaling and throughput best practices

This topic explains how throughput limits and scheduling work across Amazon Bedrock endpoints and provides best practices for scaling your generative AI applications.

Amazon Bedrock endpoints

Amazon Bedrock supports two endpoints for inference:

For more information about these endpoints and how to choose between them, see Endpoints supported by Amazon Bedrock.

Why the two endpoints behave differently

In many traditional multi-tenant services, the architecture is designed around per-account quotas to manage fair-share access to shared resources. This is the approach used with bedrock-runtime.

With bedrock-mantle, a different approach is used. This endpoint is architected with advanced scheduling and work-queueing mechanisms that deliver fair-share distribution while supporting higher initial throughput limits. This design also allows bedrock-mantle to host a broad set of models and deliver the full breadth of capabilities available across the model catalog. In most cases, requests are served immediately. In some cases, a request may be briefly queued while in-flight workloads complete and throughput becomes available. The sections below explain how to handle these scenarios.

bedrock-mantle endpoint: throughput and quotas

The throughput and quota behavior on the bedrock-mantle endpoint differs for Anthropic Claude versus other models, as shown in the following table.

* Your input TPM limit depends on your usage history with Amazon Bedrock. Check the Quotas page in the Amazon Bedrock console for your actual allocation.

bedrock-runtime endpoint: throughput and quotas

The following table summarizes the throughput and quotas for bedrock-runtime.

* Quotas for these models vary based on usage. Check the Quotas page in the Amazon Bedrock console for your allocations.

Understanding HTTP error responses

Recommended error handling

Transient errors (occasional 503 responses)

Implement exponential backoff with random jitter:

Most AWS SDKs and popular HTTP libraries provide built-in support for this pattern.

import boto3
from botocore.config import Config

config = Config(retries={"total_max_attempts": 6, "mode": "standard"})
client = boto3.client("bedrock-runtime", config=config)

from openai import OpenAI

client = OpenAI(
    api_key=api_key,
    base_url=f"https://bedrock-mantle.{region}.api.aws/v1",
    max_retries=6,
    timeout=60.0,
)

import anthropic

client = anthropic.Anthropic(
    api_key=api_key,
    base_url=f"https://bedrock-mantle.{region}.api.aws",
    max_retries=6,
    timeout=60.0,
)

Sustained errors (persistent 503 responses)

If you receive sustained 503 errors, retrying alone will not resolve the issue. Your request rate exceeds available throughput. Take the following steps:

Ramping up throughput

When consuming on-demand throughput on the bedrock-mantle endpoint, available throughput scales over time. Not all requests within your quota are guaranteed to succeed during periods of high demand, so ramping gradually is important.

Recommended ramp-up procedure

For example, if your target is 2,000 RPM but you receive 503 errors, reduce to 1,000 RPM. If errors persist, reduce to 500 RPM. Once requests succeed consistently at 500 RPM, hold for 15 minutes, then scale to 750, then 1,125, and so on.

Ramp rates are not adjustable. To request higher TPM quotas, use the Service Quotas console or contact your AWS account team.

Additional best practices

Regional availability and cross-Region inference

On-demand throughput is allocated at the Regional level and varies across Regions. If your workload targets a single Region, you may encounter 503 responses during periods of high demand. To maximize availability and if you are using bedrock-runtime, use Global cross-Region inference.

Getting help

Summary of recommendations