Recommended CloudWatch alarms for Amazon MSK Provisioned clusters - Amazon Managed Streaming for Apache Kafka
IAM access control alarms
Services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the AWS European Sovereign Cloud Region, see the AWS European Sovereign Cloud User Guide.

Recommended CloudWatch alarms for Amazon MSK Provisioned clusters

Monitor your Amazon MSK Provisioned clusters to detect issues before they affect your applications. CloudWatch alarms perform an action when a CloudWatch metric exceeds a specified value for some amount of time. For example, you might want to get an email notification if your partition count exceeds the recommended value for your broker instance size for longer than 15 minutes. The critical alarms in the following table are recommended, but they are not an exhaustive list of alarms you can create to monitor your cluster.

For more information about configuring alarms, see Creating Amazon CloudWatch alarms in the Amazon CloudWatch User Guide.

The following table lists alarms that apply to both Standard and Express brokers.

Alarm Issue

CPUUser + CPUSystem Average >= 60 for 5 minutes, 3 consecutive times

Dimensions: Cluster Name, Broker ID

One or more brokers have average CPUUser+CPUSystem over the recommended 60%. To learn more, see Monitor CPU usage.

PartitionCount Average >= X for 5 minutes, 3 consecutive times (X = Recommended partition count for broker instance size)

Dimensions: Cluster Name, Broker ID

One or more brokers have partitions higher than the recommended partition count limit. To learn more, see Right-size your cluster: Number of partitions per Standard broker and Express broker partition quota.

SumOffsetLag Average >= X for 5 minutes, 3 consecutive times (X is set for a consumer group and topic combination based on use case)

Dimensions: Cluster Name, Consumer Group, Topic

The aggregated offset lag for all the partitions in a topic is over X. For more information about the lag, you can use the partition-level Offset metric, which represents the lag for each partition, or use the Kafka command line tool to describe the consumer group. Review the processing speed of your consumer application relative to the producers to understand if the consumers are unable to keep up, and check if any consumer rebalances are slowing down the consumers.

The following table lists alarms that apply to Standard brokers only.

Alarm Issue

OfflinePartitionsCount Average >= 1 for 1 minute, 3 consecutive times

Dimensions: Cluster Name

One or more topic partitions are unavailable. When partitions are unavailable, produce and consume operations to those partitions fail. Offline partitions should not occur on a well-balanced, correctly sized, and correctly configured cluster. To learn more, see Build highly available clusters.

UnderMinIsrPartitionCount Average >= 1 for 1 minute, 3 consecutive times

Dimensions: Cluster Name, Broker ID

One or more topics have partitions under the minimum configured in-sync replica set (ISR). When partitions fall below the minimum ISR, produce operations fail (with producer acks=all). To learn more, see Build highly available clusters.

KafkaDataLogsDiskUsed Average >= 80 for 5 minutes, 3 consecutive times

Dimensions: Cluster Name, Broker ID

One or more brokers have data disk usage of 80% or more. To learn more, see Monitor disk space.

HeapMemoryAfterGC Average >= 60 for 5 minutes, 3 consecutive times

Dimensions: Cluster Name, Broker ID

One or more brokers have 60% or more of total heap memory in use after garbage collection. To learn more, see Monitor Apache Kafka memory.

(Sum(VolumeReadBytes) + Sum(VolumeWriteBytes)) / (5 * 60 * 1024 * 1024) >= X MiB for 5 minutes, 3 consecutive times (X = 80% of available volume throughput)

Dimensions: Cluster Name, Broker ID

One or more brokers have underlying volume read and write activity using up 80% of its available volume throughput. To learn more, see Provisioned storage throughput.

CPUCreditBalance Average <= 100 for 5 minutes, 3 consecutive times

Dimensions: Cluster Name, Broker ID

This is only relevant to the t3.small broker type. One or more brokers have depleted their CPU credit balance from a maximum of 576 to less than 100. When the balance reaches 0, the broker cannot exceed the 20% CPU baseline. To avoid CPU credit depletion, upgrade from a t3 broker instance type to an m7g instance type, which doesn't use CPU credits.

RequestHandlerAvgIdlePercent Average <= 0.3 for 5 minutes, 3 consecutive times

Dimensions: Cluster Name, Broker ID

One or more brokers are seeing activity congestion on the threadpool responsible for serving requests (threadpool is less than 30% idle). Saturation here indicates slow requests, which can cause client-side timeouts. Also, review whether your clients are generating excessive requests. For example, unauthorized clients might be aggressively retrying requests that the brokers are denying. To learn more about optimizing cluster throughput, see Optimize cluster throughput for m5.4xl, m7g.4xl or larger instances.

NetworkProcessorAvgIdlePercent Average <= 0.3 for 5 minutes, 3 consecutive times

Dimensions: Cluster Name, Broker ID

One or more brokers are seeing activity congestion on the network connection threadpool (threadpool is less than 30% idle). Saturation here can cause timeouts. Also, review whether your clients are generating excessive requests. For example, unauthorized clients might be aggressively retrying requests that the brokers are denying. To learn more about optimizing cluster throughput, see Optimize cluster throughput for m5.4xl, m7g.4xl or larger instances.

KafkaFileDescriptorsUsagePercent > 80% for 5 minutes, 3 consecutive times

Dimensions: Cluster Name, Broker ID

The percentage of file descriptors in use on the broker. At 100% exhaustion, the Kafka broker might not be able to start. The file descriptors count increases with the number of partitions, number of log segments in each partition, and number of client connections. Review whether you have topics with low segment.ms values resulting in frequent log rolls, and consider reducing the number of client connections.

KafkaMemoryMappedFilesUsagePercent > 80% for 5 minutes, 3 consecutive times

Dimensions: Cluster Name, Broker ID

The percentage of memory-mapped files in use on the broker. At 100% exhaustion, the Kafka broker might not be able to start. The memory-mapped files usage count increases with the number of partitions and number of log segments in each partition. Review whether you have topics with low segment.ms values resulting in frequent log rolls.

IAM access control alarms

In addition to the preceding alarms, we recommend creating alarms for the following metrics specific to IAM access control. These alarms apply to both Standard and Express brokers with IAM authentication enabled. Amazon MSK places logical limits on IAM connections to protect the broker from IAM connection request overload. Breaching any of these limits results in client connection timeouts, which will impact your workload.

Alarm Issue

ClientConnectionCount Sum >= X for 1 minute, 3 consecutive times (X = 80% of Maximum TCP connections per broker)

Dimensions: Cluster Name, Broker ID, Client Authentication

One or more brokers have connection count equal to 80% of the connection limit. The default Maximum TCP connections per broker for IAM access control is 3000. This value can be changed. For more information, see Amazon MSK Express broker quota for Express brokers and Amazon MSK Standard broker quota for Standard brokers.

ConnectionCreationRate Sum >= X for 1 minute, 3 consecutive times (X = 80% of the connection creation rate limit for your instance size)

Dimensions: Cluster Name, Broker ID

One or more brokers have clients creating IAM connections at a rate equal to 80% of its connection creation rate limit. The Maximum TCP connections rate per broker for IAM access control depends on instance size. For more information, see Amazon MSK Express broker quota for Express brokers and Amazon MSK Standard broker quota for Standard brokers.