Services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the AWS European Sovereign Cloud Region, see the AWS European Sovereign Cloud User Guide.

Inference Gateway troubleshooting guide

Overview: The HyperPod Inference Gateway routes traffic through three layers: the Body-Based Router (BBR), the gateway with HTTPRoute, and the Endpoint Picker (EPP). Misconfiguration at any layer can result in failed requests, traffic reaching the wrong model, or uneven load across model-serving pods. This section covers issues with the gateway, BBR, the gateway and HTTPRoute, InferencePool, and EPP, and any issues that arise from them.

Diagnose gateway state

Use the following commands to inspect the gateway and the resources it manages.

List every InferenceGatewayConfig resource across namespaces:

kubectl get inferencegatewayconfig -A

Show detailed status, per-scheduler rollout state, and condition messages for a specific gateway:

kubectl describe inferencegatewayconfig <name> -n <namespace>

Check the gateway controller and Body-Based Router pods:

kubectl get pods -n hyperpod-inference-system

Check the downstream routing resources generated by the controller:

kubectl get httproute,inferencepool,securitypolicy -A

Inspect status.conditions and each scheduler's rolloutState (Pending, Progressing, Available, or Degraded). Actionable failure reasons are on the corresponding condition message.

Add-on install issues

Problem: Gateway resources are missing, or the GatewayClass is not accepted after installing the HyperPod Inference Amazon EKS add-on.

Symptoms and resolution: kubectl get gatewayclass inference-gateway returns NotFound, or the resource shows ACCEPTED=False. This indicates that the add-on is not installed or that the installation did not complete. Reinstall or update the add-on:

aws eks update-addon --cluster-name $CLUSTER --region $REGION \
  --addon-name amazon-sagemaker-hyperpod-inference \
  --resolve-conflicts OVERWRITE

Then confirm the gateway controller is running:

kubectl rollout status deploy/inference-gateway-controller \
  -n hyperpod-inference-system --timeout=150s

InferenceGatewayConfig not becoming Ready

Problem: An InferenceGatewayConfig is created, but its status.conditions show Accepted=False or Ready=False, or the kubectl apply is rejected outright by validation.

Symptoms and resolution:

Per-scheduler failures

Problem: A specific scheduler's condition (BackendsReady, LoraSupported, or PoolReady) indicates that the scheduler is not fully ready.

Symptoms and resolution:

Scheduler rolloutState is Degraded

Problem: The Endpoint Picker Deployment for a scheduler is stuck and does not become Available.

Symptoms and resolution: The EPPReady condition on the scheduler carries the actionable reason. Common causes include:

Use the following commands to identify the failing scheduler and inspect its Deployment:

# List the schedulers reporting Degraded
kubectl get inferencegatewayconfig <name> -n <namespace> \
  -o jsonpath='{range .status.schedulers[?(@.rolloutState=="Degraded")]}{.name}{"\n"}{end}'

# Describe the scheduler's Endpoint Picker Deployment for pod events and container errors
kubectl describe deploy -n <namespace> \
  -l inference.sagemaker.aws.amazon.com/scheduler=<scheduler-name>

Stale endpoint after a model pod restart

Problem: After a model pod is deleted and a replacement pod becomes Ready, the Endpoint Picker continues to route to the deleted pod's IP address. Requests return HTTP 503 or connection refused, and the condition does not recover on its own.

Resolution: Add a readiness probe to the model pod so that Kubernetes marks the pod NotReady before its IP is removed from the pool, and only advertises the replacement pod once it is fully serving traffic. Set port to the scheduler's targetPort and path to your model server's health endpoint:

readinessProbe:
  httpGet:
    path: /health
    port: 8000

Workaround: If you cannot redeploy the model pod immediately, restart the scheduler's Endpoint Picker to force it to rebuild its endpoint list from the current pod set:

kubectl rollout restart deploy -n <namespace> \
  -l inference.sagemaker.aws.amazon.com/scheduler=<scheduler-name>

JWT authentication returns 401 or 403

Problem: spec.auth.jwt is configured and requests are rejected before reaching a model, or the gateway never becomes Ready after JWT authentication is enabled.

Symptoms and resolution:

Metrics missing from dashboards

Problem: Endpoint Picker or Body-Based Router metrics do not appear in your monitoring dashboard.

Symptoms and resolution: Metrics collection is enabled by default, so the OpenTelemetry Collector sidecar is normally present. Confirm whether the sidecar is running on both pod types, and whether metrics were explicitly disabled.

Check the sidecar on Body-Based Router pods:

kubectl -n hyperpod-inference-system get pods \
  -o jsonpath='{.items[*].spec.containers[*].name}' | tr ' ' '\n' | grep otel

Check the sidecar on Endpoint Picker pods:

kubectl -n <namespace> get pods \
  -o jsonpath='{.items[*].spec.containers[*].name}' | tr ' ' '\n' | grep otel

Check whether metrics were explicitly disabled:

kubectl get inferencegatewayconfig <name> -n <namespace> \
  -o jsonpath='{.spec.observability.metrics.enabled}'

Empty output from the last command means the field is not set, and metrics are enabled. Only an explicit false disables the sidecar. If the value is false, set it to true or remove the field, and the controller injects the sidecar on the next reconciliation.

Request failures

Problem: The gateway is Ready, but inference requests fail.

Symptoms and resolution:

Debugging endpoint selection

Problem: Traffic skews to a small number of model pods, or a LoRA request is routed to a pod that does not host the adapter.

Resolution: Temporarily raise the Endpoint Picker's log verbosity to inspect its scoring decisions. Set logLevel on the scheduler:

spec:
  schedulers:
    - name: <scheduler-name>
      logLevel: 4

Log level meanings:

Inspect the Endpoint Picker logs:

kubectl logs -n <namespace> -l app=<scheduler-name>-epp -c epp --tail=200 -f

Return logLevel to its default when the investigation is complete to avoid excess log volume.

Lifecycle and cleanup

Problem: Uninstalling or upgrading the HyperPod Inference Amazon EKS add-on leaves orphaned resources in the cluster, or blocks a subsequent install.

Resolution: Always delete every InferenceGatewayConfig resource before uninstalling or upgrading the add-on. Uninstalling the add-on while an InferenceGatewayConfig is still present removes the controller that owns the resource's finalizers, which leaves those resources stuck in Terminating.

kubectl delete inferencegatewayconfig --all -A
kubectl get inferencegatewayconfig -A

Confirm the second command returns no rows before continuing with the add-on operation.

After reinstalling the add-on, list the resources in the gateway namespace and remove anything that no longer maps to a live InferenceGatewayConfig:

kubectl get deploy,svc,httproute,inferencepool,gateway,configmap \
  -n hyperpod-inference-system

ACM certificates issued by the controller are not deleted by an add-on uninstall. To remove them, filter ACM certificates in the AWS Resource Groups Tagging API by the tag CreatedBy=HyperPodInference and delete the certificates you no longer need.

Collect logs

Use the following commands to retrieve logs from each gateway component:

# Gateway controller
kubectl logs -n hyperpod-inference-system deploy/inference-gateway-controller

# Body-Based Router (deployment name is <gateway-name>-bbr)
kubectl logs -n hyperpod-inference-system deploy/<gateway-name>-bbr -c bbr

# Endpoint Picker for a specific scheduler
kubectl logs -n <namespace> -l app=<scheduler-name>-epp -c epp