Use time-slicing with NVIDIA GPUs on Amazon EKS - Amazon EKS
ConsiderationsConfiguration optionsUse GPU time-slicing with the NVIDIA DRA driverUse GPU time-slicing on Bottlerocket nodes with the NVIDIA device pluginUse GPU time-slicing on AL2023 nodes with the NVIDIA device pluginVerify that GPU time-slicing is active
Services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the AWS European Sovereign Cloud Region, see the AWS European Sovereign Cloud User Guide.

Help improve this page

To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.

Use time-slicing with NVIDIA GPUs on Amazon EKS

Time-slicing lets multiple Pods share a single physical NVIDIA GPU. The Kubernetes scheduler places multiple Pods on the same GPU, and the GPU’s CUDA scheduler time-multiplexes the work. Time-slicing is the simplest GPU-sharing strategy. It uses software only, requires no special hardware, and works on every NVIDIA GPU instance type on AWS. Time-slicing does not offer memory or compute isolation between Pods that share a GPU. It best suits workloads with low GPU utilization, such as inference services that idle between requests or development environments where several users share a GPU. For workloads that require hardware-level memory and compute isolation, use Multi-Instance GPU (MIG) instead.

On Amazon EKS, you can manage time-slicing with either the NVIDIA DRA driver or the NVIDIA device plugin.

Time-slicing can only be used with Karpenter if you are using static capacity provisioning. Time-slicing is not currently available in EKS Auto Mode.

Time-slicing is a good fit when:

  • GPU utilization is consistently low, for example latency-sensitive inference services that idle between requests.

  • Several developers share a single GPU node for notebook or development work.

  • Your nodes use a GPU instance type that does not support MIG, such as the g5, g6, or g6e families.

  • You can accept that a Pod is occasionally slower because another Pod on the same GPU is busy.

Consider a different approach when:

  • You need memory isolation. Pods on a time-sliced GPU share the same GPU memory, and one Pod can exhaust the memory that other Pods rely on. Use MIG for memory isolation.

  • You need predictable per-Pod latency or quality of service. The GPU scheduler shares compute across slots on a best-effort basis without guarantees.

  • You run training workloads. Time-slicing adds context-switching that reduces training efficiency. A checkpointed job that another Pod slows down must retry from its last checkpoint.

Considerations

Review the following considerations before you use time-slicing in production.

General considerations

  • No memory isolation: Pods that share a time-sliced GPU share its memory. One Pod can allocate memory that other Pods need, which can cause out-of-memory errors. Match the number of Pods that share each GPU to the memory footprint of your workloads. If your Pods regularly load large models, share fewer Pods per GPU or use MIG.

  • Best-effort compute sharing: The GPU scheduler shares compute across Pods on a best-effort basis and does not guarantee proportional compute to each Pod.

  • Time-slicing and MPS cannot share the same GPU: Time-slicing sets the GPU compute mode to DEFAULT, while NVIDIA Multi-Process Service (MPS) requires EXCLUSIVE_PROCESS. You can use both strategies in the same cluster, but not on the same physical GPU at the same time. Time-slicing also has no effect on a MIG instance. To share a single MIG instance across containers, use MPS instead.

  • Per-container metrics: NVIDIA Data Center GPU Manager (DCGM) cannot attribute metrics to individual containers when time-slicing is active. GPU-level metrics remain available, but you cannot identify which Pod consumed a given amount of GPU resources.

NVIDIA DRA driver considerations

  • Alpha feature: Time-slicing through the DRA driver requires the TimeSlicingSettings feature gate, which is an alpha feature disabled by default. For more information, see Use GPU time-slicing with the NVIDIA DRA driver.

  • User-mediated sharing only: Pods share a GPU only by referencing the same ResourceClaim or ResourceClaimTemplate, which is namespace-scoped, so sharing cannot cross namespaces. System-mediated sharing is a proposed future capability.

  • Disable the built-in device plugin on Bottlerocket: The DRA driver cannot run alongside the NVIDIA device plugin on the same node. On Bottlerocket, disable the built-in device plugin, which requires Bottlerocket version 1.63.0 or later. For more information, see Install the NVIDIA DRA driver.

  • Compute support: The NVIDIA DRA driver is supported with static capacity provisioning in Karpenter, EKS managed node groups, or self-managed nodes, and is not supported with EKS Auto Mode. For more information, see the Karpenter static NodePool documentation on the Karpenter website.

NVIDIA device plugin considerations

  • Best-effort compute sharing with slots: The device plugin advertises a fixed number of slots per GPU. If a single Pod requests more than one slot, it does not receive additional compute. Enable the fail-requests-greater-than-one option to reject Pods that request more than one slot.

  • Provisioning with Karpenter: Karpenter counts each nvidia.com/gpu request as a physical GPU even when time-slicing is enabled on the GPU. For more information, see Karpenter issue #2140 on GitHub.

  • No support on EKS Auto Mode: EKS Auto Mode manages the NVIDIA device plugin and does not expose its configuration. Because time-slicing requires device plugin configuration, you cannot apply a time-slicing configuration on EKS Auto Mode nodes.

  • Time-slicing configuration changes: The NVIDIA device plugin does not monitor the time-slicing ConfigMap for changes. Restart the device plugin after you update the configuration.

Configuration options

You can use time-slicing for NVIDIA GPUs with the EKS-optimized AL2023 and Bottlerocket NVIDIA AMIs. The time-slicing settings vary based on whether you are using the NVIDIA DRA driver or NVIDIA device plugin.

NVIDIA DRA driver

The EKS-optimized accelerated AMIs do not include the NVIDIA DRA driver. If you are using Bottlerocket, you must disable the built-in NVIDIA device plugin before you use the NVIDIA DRA driver by setting settings.kubelet-device-plugins.nvidia.enabled = false in the node user data, which requires Bottlerocket version 1.63.0 or later. For more information, see Install the NVIDIA DRA driver.

With the NVIDIA DRA driver, time-slicing is configured through ResourceClaimTemplates. The interval field controls the CUDA time-slice duration.

Interval Description

Default

Uses the NVIDIA GPU driver’s built-in default interval

Short

Shorter interval; contexts switch more frequently

Medium

Intermediate interval

Long

each context runs longer per turn before being preempted
NVIDIA device plugin

  • Bottlerocket – The AMI includes a pre-installed NVIDIA device plugin. You configure time-slicing through Bottlerocket settings, with no separate device plugin installation, Helm chart, or ConfigMap.

  • AL2023 – You install the NVIDIA device plugin as described in Install the NVIDIA Kubernetes device plugin, and supply the time-slicing configuration through a ConfigMap.

    The following options control how the NVIDIA device plugin advertises and manages time-sliced GPUs. The field names differ between the Bottlerocket settings and the AL2023 ConfigMap, as shown in the procedures that follow.

    Option Recommended value Description

    Replicas

    2–8

    The number of schedulable slots to advertise per physical GPU. Higher values allow more sharing but increase contention between Pods.

    Rename by default

    false

    When false, Pods request nvidia.com/gpu, which keeps compatibility with existing manifests. When true, the device plugin advertises the resource as nvidia.com/gpu.shared, and Pods must request that name. Set this to true when you run both shared and dedicated GPU node pools and want workloads to select the shared resource explicitly.

    Fail requests greater than one

    true

    Rejects Pods that request more than one time-sliced slot. A Pod that requests more than one slot does not receive proportional compute. Enable this setting to prevent a common misconfiguration.

    For the complete list of GPU sharing options and their default values, see the NVIDIA Kubernetes device plugin documentation on GitHub.

Use GPU time-slicing with the NVIDIA DRA driver

With the NVIDIA DRA driver, Pods share a GPU by referencing a common ResourceClaim that requests the gpu.nvidia.com DeviceClass with a time-slicing GpuConfig.

The NVIDIA DRA driver currently implements user-mediated time-slicing: a GPU is shared only among the Pods and containers that you explicitly point at the same claim. Because a ResourceClaim and a ResourceClaimTemplate are namespace-scoped, the Pods that share a GPU this way must be in the same namespace. To share a GPU across containers in a single Pod, have each container reference the same request name in the claim. Containers that reference different request names each receive a separate GPU. Create a separate ResourceClaimTemplate for each time-slice interval you need, and one in each namespace where you want to share GPUs.

System-mediated time-slicing is when the driver shares a GPU across independent claims (including across namespaces) based on criteria the system defines rather than a claim you configure. For more information about system-mediated time-slicing, see System-mediated time-slicing of GPUs (issue #659) and pull request #1257 on GitHub.

Important

Time-slicing through the DRA driver requires the TimeSlicingSettings feature gate, which is an alpha feature disabled by default. If you request the TimeSlicing sharing strategy without enabling this feature gate, the driver fails to prepare the device and the Pod stays in ContainerCreating with a FailedPrepareDynamicResources event that reports error validating GPU config: unknown GPU sharing strategy: TimeSlicing. Enable the feature gate only if you accept the risks of using an alpha capability.

Prerequisites

  • An Amazon EKS cluster running Kubernetes version 1.34 or later. The NVIDIA DRA driver is supported with static capacity provisioning in Karpenter, EKS managed node groups, or self-managed nodes.

  • Nodes with NVIDIA GPU instance types using the EKS-optimized AL2023 NVIDIA AMI.

  • The NVIDIA DRA driver installed as described in Install the NVIDIA DRA driver, with the TimeSlicingSettings feature gate enabled.

Procedure

  1. Create a ResourceClaim that requests a GPU with the TimeSlicing sharing strategy. Multiple Pods that reference this claim share the same physical GPU.

    cat <<EOF | kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: shared-timeslice-gpu
spec:
  devices:
    requests:
    - name: gpu
      exactly:
        deviceClassName: gpu.nvidia.com
        count: 1
    config:
    - requests: ["gpu"]
      opaque:
        driver: gpu.nvidia.com
        parameters:
          apiVersion: resource.nvidia.com/v1beta1
          kind: GpuConfig
          sharing:
            strategy: TimeSlicing
            timeSlicingConfig:
              interval: Long
EOF

  2. Deploy two or more Pods that reference the shared ResourceClaim by name. Each Pod references the claim through resourceClaims and resources.claims.

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: share-a
spec:
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: cuda
      image: nvidia/cuda:12.6.0-base-ubuntu22.04
      command: ["nvidia-smi", "-L"]
      resources:
        claims:
        - name: gpu
  resourceClaims:
    - name: gpu
      resourceClaimName: shared-timeslice-gpu
  restartPolicy: OnFailure
EOF

  3. Verify that the Pods share the same physical GPU. Run nvidia-smi -L in each Pod and confirm they report the same GPU UUID.

    kubectl logs share-a

    An example output is as follows. A second Pod that references the same claim reports an identical GPU UUID, which confirms that both Pods share one physical GPU.

    GPU 0: NVIDIA L4 (UUID: GPU-b41973ee-5d0a-cde8-6287-12b53f861f02)

Use GPU time-slicing on Bottlerocket nodes with the NVIDIA device plugin

On Bottlerocket, the EKS-optimized accelerated AMI includes the NVIDIA device plugin. You enable time-slicing through the settings.kubelet-device-plugins.nvidia settings, which Bottlerocket renders into the device plugin configuration when the node boots.

Prerequisites

  • An Amazon EKS cluster. The following procedure provisions NVIDIA GPU nodes with the EKS-optimized Bottlerocket NVIDIA AMI.

  • Karpenter installed and configured in your cluster, because the following procedure uses a Karpenter EC2NodeClass to supply the time-slicing settings in the Bottlerocket node user data. For more information, see the Getting Started with Karpenter on the Karpenter website.

  • kubectl configured to communicate with your cluster, see Install or update kubectl for more information.

Procedure

Add the time-slicing settings to the Bottlerocket user data for your GPU nodes. The following example configures four slots per GPU. The way you supply user data depends on how you provision nodes. The following example shows a Karpenter EC2NodeClass.

cat <<EOF | kubectl apply -f -
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-bottlerocket-timeslicing
spec:
  amiFamily: Bottlerocket
  amiSelectorTerms:
    - alias: bottlerocket@latest
  role: eksctl-KarpenterNodeRole-<cluster-name>
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: <cluster-name>
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: <cluster-name>
  userData: |
    [settings.kubelet-device-plugins.nvidia]
    device-sharing-strategy = "time-slicing"

    [settings.kubelet-device-plugins.nvidia.time-slicing]
    replicas = 4
    rename-by-default = false
    fail-requests-greater-than-one = true
EOF

When nodes provisioned with these settings join the cluster, the device plugin advertises four nvidia.com/gpu slots for each physical GPU. You request them in your workload specification with container resources requests or limits for the nvidia.com/gpu extended resource.

Use GPU time-slicing on AL2023 nodes with the NVIDIA device plugin

On AL2023, you install the NVIDIA device plugin as described in Install the NVIDIA Kubernetes device plugin, and supply the time-slicing configuration in a ConfigMap.

Prerequisites

  • An Amazon EKS cluster with nodes that use NVIDIA GPU instance types and the EKS-optimized AL2023 NVIDIA AMI.

  • Helm installed in your command-line environment, see the Setup Helm instructions for more information.

  • kubectl configured to communicate with your cluster, see Install or update kubectl for more information.

Procedure

  1. Create the time-slicing ConfigMap in the nvidia namespace, where you installed the device plugin in Install the NVIDIA Kubernetes device plugin. This example configures four slots per GPU.

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin-config
  namespace: nvidia
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: true
        resources:
          - name: nvidia.com/gpu
            replicas: 4
EOF

  2. Update the existing NVIDIA device plugin release to reference the ConfigMap with the config.name value. The --reuse-values flag preserves the values you set when you installed the device plugin in Install the NVIDIA Kubernetes device plugin.

    helm upgrade nvdp nvdp/nvidia-device-plugin \
    --namespace nvidia \
    --reuse-values \
    --set config.name=nvidia-device-plugin-config
Note

The device plugin does not reload automatically when you change the ConfigMap. After you update the time-slicing configuration, restart the device plugin Pods to apply the change.

Verify that GPU time-slicing is active

After your time-sliced nodes are Ready, confirm that the device plugin advertises the expected number of slots and that Pods share a physical GPU.

Note

The shared-UUID check in this procedure demonstrates time-slicing most clearly when the Pods land on the same physical GPU. On a node with a single GPU, the device plugin advertises four slots and all four Pods share that GPU, so they report the same UUID. On a node with multiple physical GPUs, the scheduler can place Pods on different GPUs. Those Pods report different UUIDs even though time-slicing is active. To demonstrate sharing on one GPU, schedule the workload onto a single-GPU instance type. For example, add a node selector such as node.kubernetes.io/instance-type: g6.2xlarge to the Pod specification.

  1. Confirm that the node advertises the configured number of GPU slots. With four slots per GPU, a node with one physical GPU reports 4.

    kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

    An example output is as follows.

    NAME                                           GPU
ip-192-168-11-225.us-west-2.compute.internal   4

  2. Create a deployment that runs four replicas, each requesting one GPU slot.

    cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: timeslicing-demo
spec:
  replicas: 4
  selector:
    matchLabels:
      app: timeslicing-demo
  template:
    metadata:
      labels:
        app: timeslicing-demo
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: cuda
          image: nvidia/cuda:12.6.0-base-ubuntu22.04
          command: ["bash", "-c", "nvidia-smi --query-gpu=uuid --format=csv,noheader; sleep infinity"]
          resources:
            limits:
              nvidia.com/gpu: 1
EOF

  3. Confirm that all four Pods are scheduled on the same node.

    kubectl get pods -l app=timeslicing-demo -o wide

  4. Confirm that all four Pods report the same GPU UUID. A single shared UUID across all four Pods confirms that one physical GPU is being time-multiplexed.

    kubectl logs -l app=timeslicing-demo --prefix

    An example output is as follows.

    [pod/timeslicing-demo-xxxxxxxxxx-aaaaa/cuda] GPU-c0583cce-87c5-c736-db7f-6d3128c84d03
[pod/timeslicing-demo-xxxxxxxxxx-bbbbb/cuda] GPU-c0583cce-87c5-c736-db7f-6d3128c84d03
[pod/timeslicing-demo-xxxxxxxxxx-ccccc/cuda] GPU-c0583cce-87c5-c736-db7f-6d3128c84d03
[pod/timeslicing-demo-xxxxxxxxxx-ddddd/cuda] GPU-c0583cce-87c5-c736-db7f-6d3128c84d03