Services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the AWS European Sovereign Cloud Region, see the AWS European Sovereign Cloud User Guide.

Help improve this page

To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.

Configure the Amazon EKS node monitoring agent

The EKS node monitoring agent is deployed as a DaemonSet. For the node health issues that the agent detects, see Detect node health issues with the EKS node monitoring agent.

The configuration values in this topic are Helm chart values, and you can set them whether you install the agent as an EKS add-on or manage it with Kubernetes tooling such as Helm. With an EKS add-on, you pass them as the add-on’s configuration values. The EKS add-on accepts the subset of chart values in its configuration schema and rejects any others. For the full set of values and their defaults, see the EKS node monitoring agent Helm chart.

Agent configuration values

You can customize the agent installation with the following configuration values.

You can pass any of the agent’s command-line arguments that don’t have a dedicated configuration value, such as hostname-override and verbosity, through nodeAgent.additionalArgs with EKS add-ons or a Helm installation.

aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name eks-node-monitoring-agent \
  --configuration-values '{"nodeAgent":{"additionalArgs":["--verbosity=4"]}}'

To change the health probe port, set nodeAgent.probePort rather than passing --probe-address through nodeAgent.additionalArgs. The chart always passes --probe-address and uses the same port for the agent’s liveness probe, so setting the port through nodeAgent.probePort keeps the two consistent.

NVIDIA DCGM configuration values

On NVIDIA GPU nodes, the EKS node monitoring agent collects GPU health from NVIDIA DCGM (Data Center GPU Manager). The agent connects as a DCGM client in standalone mode, so an nv-hostengine process must be reachable on the node. By default the chart provides one through the dcgm-server DaemonSet, which the agent reaches on localhost:5555. This component runs only on NVIDIA GPU instance types, as shown by the nodeAffinity in the agent’s Helm chart.

If your cluster already runs nv-hostengine on each GPU node, you can point the agent at it instead of running a second instance. For more information, see Use an existing NVIDIA DCGM host engine.

EKS Auto Mode includes the EKS node monitoring agent in the node AMI. There is no eks-node-monitoring-agent add-on to install and no separate dcgm-server Pod. On all other compute types, the add-on runs the DCGM host engine in a dcgm-server DaemonSet on NVIDIA GPU nodes.

The dcgm-server DaemonSet must be able to schedule on your GPU nodes. If it can’t, the EKS node monitoring agent can’t reach nv-hostengine and reports AcceleratedHardwareReady=False with the reason DCGMError. On EKS Auto Mode and self-managed Karpenter, automatic node repair acts on that condition and can replace otherwise healthy GPU nodes after each wait period. EKS managed node groups don’t repair DCGMError by default. For more information, see How the repair action is chosen.

In agent version v1.7.3 (add-on version v1.7.3-eksbuild.1) and later, dcgmAgent.tolerations defaults to a toleration for the nvidia.com/gpu taint key with any effect, so dcgm-server schedules on GPU nodes that carry the standard NVIDIA device plugin taint without additional configuration.

You can customize the NVIDIA DCGM installation with the following configuration values.

You can also set global.podLabels and global.podAnnotations to apply labels and annotations to every Pod that the chart deploys.

If you run an agent version earlier than v1.7.3, the following AWS CLI example updates the add-on so that dcgm-server tolerates the nvidia.com/gpu taint. If your GPU nodes carry a custom taint, replace the key value with the key of that taint.

aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name eks-node-monitoring-agent \
  --configuration-values '{"dcgmAgent":{"tolerations":[{"key":"nvidia.com/gpu","operator":"Exists"}]}}'

After the add-on update completes, verify that a dcgm-server Pod is running on each GPU node.

kubectl get pods -n kube-system -l k8s-app=dcgm-server -o wide

Use an existing NVIDIA DCGM host engine

If your cluster already runs an nv-hostengine process on each GPU node, you can configure the EKS node monitoring agent to use it instead of the bundled dcgm-server DaemonSet. One example is the NVIDIA GPU Operator with standalone DCGM enabled, which exposes a per-node host engine through a node-local nvidia-dcgm Service. NVIDIA recommends standalone mode when multiple clients interact with DCGM, so a single host engine per node is recommended over running two instances. For more information, see Standalone mode in the NVIDIA DCGM documentation.

This configuration requires agent version v1.7.1 (add-on version v1.7.1-eksbuild.1) or later. EKS Auto Mode doesn’t support it, because the agent is built into the node AMI and isn’t configurable.

To use an existing host engine, set the following configuration values.

The following example points the agent at the GPU Operator’s nvidia-dcgm Service.

{
  "dcgmAgent": {
    "enabled": false
  },
  "nodeAgent": {
    "dcgmAddress": "nvidia-dcgm.gpu-operator.svc:5555",
    "dnsPolicy": "ClusterFirstWithHostNet"
  }
}

Apply it to the EKS add-on with the following command.

aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name eks-node-monitoring-agent \
  --configuration-values '{"dcgmAgent":{"enabled":false},"nodeAgent":{"dcgmAddress":"nvidia-dcgm.gpu-operator.svc:5555","dnsPolicy":"ClusterFirstWithHostNet"}}'

After the update completes, confirm that the agent reports GPU health through the external host engine. Your GPU nodes should report AcceleratedHardwareReady=True, and no dcgm-server Pods should be running.

kubectl get pods -n kube-system -l k8s-app=dcgm-server
kubectl get nodes -o custom-columns='NAME:.metadata.name,ACCEL:.status.conditions[?(@.type=="AcceleratedHardwareReady")].status,REASON:.status.conditions[?(@.type=="AcceleratedHardwareReady")].reason'

To disable GPU health monitoring entirely rather than redirect it, disable the bundled server and the NVIDIA monitor together. Disabling the server alone leaves the monitor registered, which produces the DCGMError condition described previously.

{
  "dcgmAgent": {
    "enabled": false
  },
  "nodeAgent": {
    "monitors": {
      "nvidia": {
        "enabled": false
      }
    }
  }
}

Get add-on versions and the configuration schema

You can use the following AWS CLI commands to get useful information about the versions and schema for the EKS node monitoring agent EKS add-on.

Get the latest agent add-on version for your Kubernetes version. Replace 1.35 with your Kubernetes version.

aws eks describe-addon-versions \
  --addon-name eks-node-monitoring-agent \
  --kubernetes-version 1.35 \
  --query='addons[].addonVersions[].addonVersion'

Get the agent add-on schema supported in EKS add-ons. This returns the configuration values that the version you specify accepts. Replace v1.7.3-eksbuild.1 with your agent version.

aws eks describe-addon-configuration \
  --addon-name eks-node-monitoring-agent \
  --addon-version v1.7.3-eksbuild.1 \
  --query 'configurationSchema' --output text