Migrate NGINX Ingress Controllers when enabling Amazon EKS Auto Mode - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsEpicsRelated resources
Services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the AWS European Sovereign Cloud Region, see the AWS European Sovereign Cloud User Guide.

Migrate NGINX Ingress Controllers when enabling Amazon EKS Auto Mode

Olawale Olaleye and Shamanth Devagari, Amazon Web Services

Summary

EKS Auto Mode for Amazon Elastic Kubernetes Service (Amazon EKS) can reduce the operational overhead of running your workloads on Kubernetes clusters. This mode allows AWS to also set up and manage the infrastructure on your behalf. When you enable EKS Auto Mode on an existing cluster, you must carefully plan the migration of NGINX Ingress Controller configurations. This is because the direct transfer of Network Load Balancers isn't possible.

You can use a blue/green deployment strategy to migrate an NGINX Ingress Controller instance when you enable EKS Auto Mode in an existing Amazon EKS cluster.

Prerequisites and limitations

Prerequisites

Architecture

A blue/green deployment is a deployment strategy in which you create two separate but identical environments. Blue/green deployments provide near-zero downtime release and rollback capabilities. The fundamental idea is to shift traffic between two identical environments that are running different versions of your application.

The following image shows the migration of Network Load Balancers from two different NGINX Ingress Controller instances when enabling EKS Auto Mode. You use a blue/green deployment to shift traffic between the two Network Load Balancers.

Using a blue/green deployment strategy to migrate NGINX Ingress Controller instances.

The original namespace is the blue namespace. This is where the original NGINX Ingress Controller service and instance run, before you enable EKS Auto Mode. The original service and instance connect to a Network Load Balancer that has a DNS name that is configured in Route 53. The AWS Load Balancer Controller deployed this Network Load Balancer in the target virtual private cloud (VPC).

The diagram shows the following workflow to set up an environment for a blue/green deployment:

  1. Install and configure another NGINX Ingress Controller instance in a different namespace, a green namespace.

  2. In Route 53, configure a DNS name for a new Network Load Balancer.

Tools

AWS services

  • Amazon Elastic Kubernetes Service (Amazon EKS) helps you run Kubernetes on AWS without needing to install or maintain your own Kubernetes control plane or nodes.

  • Elastic Load Balancing distributes incoming application or network traffic across multiple targets. For example, you can distribute traffic across Amazon Elastic Compute Cloud (Amazon EC2) instances, containers, and IP addresses in one or more Availability Zones.

  • Amazon Route 53 is a highly available and scalable DNS web service.

  • Amazon Virtual Private Cloud (Amazon VPC) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.

Other tools

  • Helm is an open source package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.

  • kubectl is a command-line interface that helps you run commands against Kubernetes clusters.

  • NGINX Ingress Controller connects Kubernetes apps and services with request handling, auth, self-service custom resources, and debugging.

Epics

TaskDescriptionSkills required

Confirm that the original NGINX Ingress Controller instance is operational.

Enter the following command to verify that the resources in the ingress-nginx namespace are operational. If you have deployed NGINX Ingress Controller in another namespace, update the namespace name in this command.

kubectl get all -n ingress-nginx

In the output, confirm that NGINX Ingress Controller pods are in running state. The following is an example output:

NAME                                           READY   STATUS      RESTARTS      AGE
pod/ingress-nginx-admission-create-xqn9d       0/1     Completed   0             88m
pod/ingress-nginx-admission-patch-lhk4j        0/1     Completed   1             88m
pod/ingress-nginx-controller-68f68f859-xrz74   1/1     Running     2 (10m ago)   72m

NAME                                         TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.100.67.255    k8s-ingressn-ingressn-abcdefg-12345.elb.eu-west-1.amazonaws.com   80:30330/TCP,443:31462/TCP   88m
service/ingress-nginx-controller-admission   ClusterIP      10.100.201.176   <none>                                                                          443/TCP                      88m

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   1/1     1            1           88m

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-68f68f859   1         1         1       72m
replicaset.apps/ingress-nginx-controller-d8c96cf68   0         0         0       88m

NAME                                       STATUS     COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   Complete   1/1           4s         88m
job.batch/ingress-nginx-admission-patch    Complete   1/1           5s         88m
DevOps engineer
TaskDescriptionSkills required

Create the Kubernetes resources.

Enter the following commands to create a sample Kubernetes deployment, service, and ingress:

kubectl create deployment demo --image=httpd --port=80
kubectl expose deployment demo
 kubectl create ingress demo --class=nginx \
  --rule nginxautomode.local.dev/=demo:80
DevOps engineer

Review the deployed resources.

Enter the following command to view a list of the deployed resources:

kubectl get all,ingress

In the output, confirm that the sample HTTPd pod is in a running state. The following is an example output:

NAME                        READY   STATUS    RESTARTS   AGE
pod/demo-7d94f8cb4f-q68wc   1/1     Running   0          59m

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/demo         ClusterIP   10.100.78.155   <none>        80/TCP    59m
service/kubernetes   ClusterIP   10.100.0.1      <none>        443/TCP   117m

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo   1/1     1            1           59m

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-7d94f8cb4f   1         1         1       59m

NAME                             CLASS   HOSTS                                  ADDRESS                                                                         PORTS   AGE
ingress.networking.k8s.io/demo   nginx   nginxautomode.local.dev                k8s-ingressn-ingressn-abcdefg-12345.elb.eu-west-1.amazonaws.com                 80      56m
DevOps engineer

Confirm the service is reachable.

Enter the following command to confirm that the service is reachable through the DNS name of the Network Load Balancer:

curl -H "Host: nginxautomode.local.dev" http://k8s-ingressn-ingressn-abcdefg-12345.elb.eu-west-1.amazonaws.com

The following is the expected output:

<html><body><h1>It works!</h1></body></html>
DevOps engineer

(Optional) Create a DNS record.

  1. Follow the instructions in Creating records by using the Amazon Route 53 console (Route 53 documentation) to create a DNS record for the configured domain.

  2. Enter the following command to confirm that the service is reachable through the configured domain name:

    curl "http://nginxautomode.local.dev/?[1-5]"

    The following is the expected output:

    <html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
DevOps engineer, AWS DevOps
TaskDescriptionSkills required

Enable EKS Auto Mode.

Follow the instructions in Enable EKS Auto Mode on an existing cluster (Amazon EKS documentation).

AWS DevOps
TaskDescriptionSkills required

Configure a new NGINX Ingress Controller instance.

  1. Download the deploy.yaml template.

  2. Open the deploy.yaml template in your preferred editor.

  3. In the kind: Namespace section, enter a unique name for the namespace, such as ingress-nginx-v2:

    apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  name: ingress-nginx-v2

  4. For each section, update the namespace value to the new name.

  5. In the kind: Deployment section, do the following:

    1. Enter a unique value for --controller-class, such as k8s.io/ingress-nginx-v2.

    2. Enter a unique value for --ingress-class, such as nginx-v2.

    apiVersion: apps/v1
kind: Deployment
  name: ingress-nginx-controller
  namespace: ingress-nginx-v2
...  
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
        - --election-id=ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx-v2
        - --ingress-class=nginx-v2

  6. In the kind: IngressClass section, enter the same values for --controller-class and --ingress-class that you used in the previous section:

    apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.12.0
  name: nginx-v2
spec:
  controller: k8s.io/ingress-nginx-v2

  7. In the following section, add loadBalancerClass: eks.amazonaws.com/nlb to provision a Network Load Balancer for the NGINX Ingress Controller instance:

    apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx-v2
spec:
...
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer
  loadBalancerClass: eks.amazonaws.com/nlb

  8. Save and close the deploy.yaml template.

DevOps engineer

Deploy the new NGINX Instance Controller instance.

Enter the following command to apply the modified manifest file:

kubectl apply -f deploy.yaml
DevOps engineer

Confirm successful deployment.

Enter the following command to verify that the resources in the ingress-nginx-v2 namespace are operational:

kubectl get all -n ingress-nginx-v2

In the output, confirm that NGINX Ingress Controller pods are in a running state. The following is an example output:

NAME                                            READY   STATUS      RESTARTS   AGE
pod/ingress-nginx-admission-create-7shrj        0/1     Completed   0          24s
pod/ingress-nginx-admission-patch-vkxr5         0/1     Completed   1          24s
pod/ingress-nginx-controller-757bfcbc6d-4fw52   1/1     Running     0          24s

NAME                                         TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.100.208.114   k8s-ingressn-ingressn-2e5e37fab6-848337cd9c9d520f.elb.eu-west-1.amazonaws.com   80:31469/TCP,443:30658/TCP   24s
service/ingress-nginx-controller-admission   ClusterIP      10.100.150.114   <none>                                                                          443/TCP                      24s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   1/1     1            1           24s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-757bfcbc6d   1         1         1       24s

NAME                                       STATUS     COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   Complete   1/1           4s         24s
job.batch/ingress-nginx-admission-patch    Complete   1/1           5s         24s
DevOps engineer

Create a new ingress for the sample HTTPd workload.

Enter the following command to create a new ingress for the existing sample HTTPd workload:

kubectl create ingress demo-new --class=nginx-v2 \
  --rule nginxautomode.local.dev/=demo:80
DevOps engineer

Confirm that the new ingress works.

Enter the following command to confirm that the new ingress works:

curl -H "Host: nginxautomode.local.dev" k8s-ingressn-ingressn-2e5e37fab6-848337cd9c9d520f.elb.eu-west-1.amazonaws.com

The following is the expected output:

<html><body><h1>It works!</h1></body></html>
DevOps engineer
TaskDescriptionSkills required

Cut over to the new namespace.

  1. (Optional) Follow the instructions in Editing records (Route 53 documentation) to update the DNS record.

  2. When you have confirmed that the new NGINX Ingress Controller instance is operating as expected, delete the original.

  3. Delete the self-managed AWS Load Balancer Controller. For instructions, see Migrate apps from deprecated ALB Ingress Controller (Amazon EKS documentation).

  4. Drain the managed node groups. For instructions, see Deleting and draining node groups (eksctl documentation).

AWS DevOps, DevOps engineer

Review the two ingresses.

Enter the following command to review the two ingresses that were created for the sample HTTPd workload:

kubectl get ingress

The following is an example output:

NAME       CLASS      HOSTS                                  ADDRESS                                                                         PORTS   AGE
demo       nginx      nginxautomode.local.dev   k8s-ingressn-ingressn-abcdefg-12345.elb.eu-west-1.amazonaws.com                              80      95m
demo-new   nginx-v2   nginxautomode.local.dev   k8s-ingressn-ingressn-2e5e37fab6-848337cd9c9d520f.elb.eu-west-1.amazonaws.com                80      33s
DevOps engineer

Related resources