Monitor replication from self-managed Kafka clusters - Amazon Managed Streaming for Apache Kafka
Services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the AWS European Sovereign Cloud Region, see the AWS European Sovereign Cloud User Guide.

Monitor replication from self-managed Kafka clusters

After you create an MSK Replicator to replicate data from an self-managed Apache Kafka cluster to Amazon MSK Express brokers, monitor the following Amazon CloudWatch metrics to track replication progress and determine when your applications are ready to migrate.

Monitor Replicator creation status

After calling the CreateReplicator API, the Replicator transitions through the following states: CREATING, RUNNING, or FAILED. Allow approximately 30 minutes.

aws kafka describe-replicator --replicator-arn <replicator-arn>
Monitor data replication

Monitor the following metrics to track data replication progress:

  • MessageLag — Monitor until it reaches 0, which indicates all data has been replicated.

  • ReplicationLatency — Track the time it takes for records to replicate from source to target.

  • ReplicationFailures — Should be 0. A non-zero value indicates replication issues.

Monitor consumer group offset synchronization

When synchroniseConsumerGroupOffsets is set to true, MSK Replicator periodically translates and synchronizes consumer group offsets from the source to the target cluster.

  • ConsumerGroupCount — Verify that this matches the expected number of consumer groups being synchronized.

  • ConsumerGroupOffsetSyncFailure — Should be 0. If greater than 0, check that consumer groups are active, verify Read and Describe permissions, and ensure topics exist on the target cluster.

  • OffsetLag (MSK) and OffsetLag (Non-MSK) — Compare partition-level consumer lag across both clusters to verify offsets are synchronized.

Determine migration readiness

Your applications are ready to migrate when all of the following conditions are met:

  • MessageLag = 0

  • ReplicationFailures = 0

  • ConsumerGroupOffsetSyncFailure = 0