Services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the AWS European Sovereign Cloud Region, see the AWS European Sovereign Cloud User Guide.

Using an external accounting database with SageMaker HyperPod Slurm clusters

Slurm stores job history, fairshare data, user associations, and quality of service (QOS) policy in a backing SQL database. By default, SageMaker HyperPod deploys MariaDB on your cluster's controller instance, so accounting works with no database for you to manage.

You can instead use an external MySQL-compatible database that you own, such as an Amazon RDS instance. Storing accounting data outside the controller instance means:

HyperPod does not create, host, or manage the database. Provisioning, sizing, patching, backups, and availability remain your responsibility.

How it works

You configure an external accounting database by adding an AccountingDatabase object to the Orchestrator.Slurm field of a CreateCluster or UpdateCluster request, or by using the Accounting database section of the SageMaker AI console. You provide two required values, the database endpoint and an AWS Secrets Manager secret Amazon Resource Name (ARN), and HyperPod writes the Slurm accounting configuration, connects the Slurm database daemon (slurmdbd) to your database, and validates the connection.

When you include AccountingDatabase in a CreateCluster request, HyperPod validates the configuration, then during provisioning retrieves the database credentials from Secrets Manager using the secret ARN you provided, writes the Slurm accounting configuration on the controller instance, tests connectivity to your database, and starts slurmdbd against it. The cluster reaches InService only after the accounting database connection succeeds.

Your database credentials are never passed through the API and are never stored by HyperPod. You pass only the secret ARN, and HyperPod persists only that ARN.

When you omit AccountingDatabase, the cluster uses the internal MariaDB database on the controller instance and behavior is unchanged.

If the database configuration fails, the cluster does not fall back to the internal MariaDB database. Falling back would mean that jobs run during the gap never record to your database, leaving a permanent gap in your sacct history and fairshare records. Instead, cluster creation fails and reports the reason. For more information, see Troubleshooting.

Responsibilities

Because the database is outside of the HyperPod control plane, two concerns remain yours:

Prerequisites

Complete the following before you create or update a cluster with an AccountingDatabase. Because these involve resources that you own, HyperPod cannot create them for you. For more information on the prerequisites, see Accounting Configuration on the SchedMD website.

IAM permissions

Add the following statements to your cluster execution role. Include the kms:Decrypt statement only if your secret is encrypted with a customer managed key.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "arn:aws:secretsmanager:us-west-2:111122223333:secret:slurm-db-credentials-AbCdEf"
        },
        {
            "Effect": "Allow",
            "Action": "kms:Decrypt",
            "Resource": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
        }
    ]
}

Ensure that SSL/TLS is enabled and enforced on the database instance to secure all connections between the application and the database. At minimum, set ssl-mode=REQUIRED. Additionally, add server-identity verification (SSL_CA + VERIFY_CA/VERIFY_IDENTITY) for any non-loopback or off-VPC endpoint, or gate remote endpoints on verified TLS. Do not allow silent plaintext fallback for endpoints outside the cluster VPC.

Enable the appropriate server-side SSL/TLS configuration and certificates according to the database platform (for example, Amazon RDS for MySQL, Amazon RDS for MariaDB, or Aurora MySQL). For secure connections for Amazon RDS, see the following:

If you configure the external accounting database in the SageMaker AI console, the IAM role or user that you use to access the console also needs the following permissions. These permissions are separate from the cluster execution role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:ListSecrets",
                "rds:DescribeDBInstances"
            ],
            "Resource": "*"
        }
    ]
}

Secret format

The secret must contain a JSON object with username and password keys. Additional keys are ignored.

{
    "username": "slurm",
    "password": "your-password-EXAMPLE"
}

Database user privileges

Grant the privileges on the accounting database (for example, slurm_acct_db) that you pre-created:

GRANT ALL ON slurm_acct_db.* TO 'username'@'%';

Alternatively, to let slurmdbd create the database itself, you can change your grant line to *.*. For more information, see MySQL Configuration on the SchedMD website.

GRANT ALL ON *.* TO 'username'@'%';

AccountingDatabase reference

AccountingDatabase is an optional object on the cluster-level Orchestrator.Slurm field. Include it in CreateCluster to create a cluster that uses an external accounting database, or in UpdateCluster to change the database on an existing cluster.

"Orchestrator": {
    "Slurm": {
        "AccountingDatabase": {
            "Endpoint": "my-cluster-db.abc123.us-west-2.rds.amazonaws.com",
            "Port": 3306,
            "Name": "slurm_acct_db",
            "SecretArn": "arn:aws:secretsmanager:us-west-2:111122223333:secret:slurm-db-credentials-AbCdEf"
        }
    }
}

Create a cluster with an external accounting database

Using the AWS CLI

In your create_cluster.json file, add an AccountingDatabase object to the cluster-level Orchestrator.Slurm field:

"Orchestrator": {
    "Slurm": {
        "AccountingDatabase": {
            "Endpoint": "my-cluster-db.abc123.us-west-2.rds.amazonaws.com",
            "SecretArn": "arn:aws:secretsmanager:us-west-2:111122223333:secret:slurm-db-credentials-AbCdEf"
        }
    }
}

Port and Name are omitted, so the cluster uses the defaults 3306 and slurm_acct_db_{cluster_id}. The rest of the request is unchanged. Set NodeProvisioningMode to Continuous, define Slurm topology through SlurmConfig on each instance group, and include VpcConfig with a security group and subnets that can reach your database. For a complete CreateCluster request, see Continuous provisioning for enhanced cluster operations with Slurm.

Using the console

Change the accounting database on an existing cluster

You can repoint a running cluster at a different database, or attach an external database to a cluster that is using the internal MariaDB database, with a single UpdateCluster call.

HyperPod retrieves the credentials from Secrets Manager, rewrites the Slurm accounting configuration, and restarts slurmdbd against the new database. Running jobs are not affected, but new accounting records for running jobs will be lost. We recommend that you stop running jobs and migrate the accounting data before you change the database.

Using the AWS CLI

Using the console

If the update fails

HyperPod validates the new configuration before it commits to it. If validation fails, the cluster keeps running against the previous configuration.

Failures are reported as cluster events. For more information, see SageMaker HyperPod Slurm cluster events.

Verify the configuration

To confirm which database a cluster is using, do one of the following:

To verify that accounting is recording jobs, connect to a login or controller instance and run a Slurm accounting command:

# Show the accounting configuration that slurmctld is using
scontrol show config | grep -i accounting

# List recent job records from the accounting database
sacct

For more information about connecting to cluster instances, see Getting started with SageMaker HyperPod using the AWS CLI.

Rotating database credentials

Rotating the database password is not an UpdateCluster operation. Rotate the secret in Secrets Manager and HyperPod applies the new password automatically, with no API call required. slurmdbd picks up the change automatically. Running jobs are not affected.

Keep the username and password keys in place when you rotate the secret. A secret that is missing either key causes the connection to fail.

Sharing one database across clusters

Multiple HyperPod clusters can record accounting data to a single database. Slurm gives each cluster its own set of tables, prefixed with the cluster name, and shares a small set of global tables. Point each cluster at the same endpoint and database name, and no further configuration is required.

Note the following constraints:

When a cluster starts against a database that already contains tables matching its name prefix, HyperPod emits an informational cluster event. The request still succeeds and the cluster reaches InService. This is expected when a controller instance is replaced, when you switch databases, or when you recreate a cluster with a name that was used before. Review the event to confirm that reusing the existing tables is what you intend.

Migrating accounting data

HyperPod changes the database connection. Moving the data is your responsibility. The following transitions are supported:

Migrating from the internal MariaDB database

Migrating the data preserves job history, associations, fairshare data, QOS, and usage records. The mysqldump utility and the mysql client are preinstalled on the HyperPod AMI. The username and password to connect to the internal MariaDB database can be found in /opt/slurm/etc/slurmdbd.conf.

If you skip the migration, the cluster still works and slurmdbd creates the tables it needs, but earlier job history and fairshare usage are not available through sacct. If AccountingStorageEnforce is configured on your cluster, you must also recreate user associations with sacctmgr.

Monitoring slurmdbd health

If slurmdbd stops or cannot reach your database, HyperPod emits a Warn cluster event—Slurm component slurmdbd is unhealthy. slurmdbd is not running. Job accounting records are not being written.—while the cluster stays InService and jobs keep running, though accounting records for that period are never written.

Troubleshooting

Accounting database errors are reported through cluster events for updates, and through ClusterStatus and FailureMessage for cluster creation. Accounting configuration logs are also delivered to Amazon CloudWatch for the life of the cluster.

Each error is terminal. Correct the underlying cause and submit a new request. The following table lists the error codes you might see.

The following validation errors are returned when you submit the request.

Database configuration logs are uploaded to the CloudWatch log group /aws/sagemaker/Clusters/{ClusterName}/{ClusterId}, stream SageMakerAccountingDatabaseEvents/{InstanceGroupName}/{instanceId}.

Related topics