Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters - Amazon Managed Streaming for Apache Kafka
Create an IAM execution roleConfigure SASL/SCRAM user and ACL permissionsConfigure SSL on self-managed clusterStore credentials in AWS Secrets ManagerConfigure network connectivityConfigure security groups
Services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the AWS European Sovereign Cloud Region, see the AWS European Sovereign Cloud User Guide.

Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters

Create an IAM execution role

Create an IAM role with a trust policy for kafka.amazonaws.com. Attach the AWSMSKReplicatorExecutionRole and AWSSecretsManagerClientReadOnlyAccess managed policies.

Example trust policy:

{
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "kafka.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}

Configure SASL/SCRAM user and ACL permissions

Create a dedicated SCRAM user on your self-managed Kafka cluster. The following ACL permissions are required:

  1. Read, Describe on all topics

  2. Read, Describe on all consumer groups

  3. Describe on cluster resource

Example kafka-acls.sh commands:

# Grant Read and Describe on all topics
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Read --operation Describe \
  --topic '*'

# Grant Read and Describe on all consumer groups
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Read --operation Describe \
  --group '*'

# Grant Describe on cluster
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Describe --cluster

Configure SSL on self-managed cluster

Configure SSL listeners on your brokers. For publicly trusted certificates, no additional configuration is required. For private or self-signed certificates, include the full CA certificate chain in the secret stored in AWS Secrets Manager.

Store credentials in AWS Secrets Manager

Create a secret of type Other (not RDS/Redshift) in AWS Secrets Manager with the following key-value pairs:

  1. username — SCRAM username for the self-managed cluster

  2. password — SCRAM password for the self-managed cluster

  3. certificate — CA certificate chain (PEM format; required for private/self-signed certs)

Configure network connectivity

MSK Replicator requires network connectivity to your self-managed Kafka cluster. Supported options:

  • AWS Site-to-Site VPN — Connect on-premises networks to your VPC over the internet.

  • AWS Direct Connect — Establish a dedicated private network connection from your premises to AWS.

Configure security groups

Ensure security groups allow traffic between MSK Replicator and the self-managed cluster on the SASL_SSL port (typically 9096). Update both inbound rules on VPC security groups and outbound rules on the self-managed cluster firewall.