Table of contents

  1. Notice
  2. What is Kubernetes (K8s)?
  3. What is Amazon Elastic Kubernetes Service (EKS)?
  4. How to create a cluster
  5. How to create a node group
  6. How to configure a cluster autoscaler (CA)
  7. How to install the AWS load balancer controller
  8. How to install a metric server
  9. How to apply a namespace
  10. How to apply a deployment
  11. How to apply a service
  12. How to apply a horizontal pod autoscaler (HPA)
  13. How to apply an ingress
  14. Useful kubectl commands
  15. How to fix error
  16. Reference

1. Notice

  • A guide for setting the Amazon Web Services (AWS) Cloud9
  • The name of region is Asia Pacific (Seoul) and the corresponding code is ap-northeast-2 [1].
  • I recommend that you should ignore the commented instructions with an octothorpe, #.
  • I recommend that you should fill in the appropriate letters between the parentheses, <>.

2. What is Kubernetes (K8s)?

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. The name Kubernetes originates from Greek, meaning helmsman or pilot. K8s as an abbreviation results from counting the eight letters between the “K” and the “s”. Google open-sourced the Kubernetes project in 2014. Kubernetes combines over 15 years of Google’s experience running production workloads at scale with best-of-breed ideas and practices from the community [2].

3. What is Amazon Elastic Kubernetes Service (EKS)?

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed Kubernetes service that makes it easy for you to run Kubernetes on AWS and on-premises. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Amazon EKS is certified Kubernetes-conformant, so existing applications that run on upstream Kubernetes are compatible with Amazon EKS [3].

1. Amazon EKS cluster

An Amazon EKS cluster, which calls cluster for short, consists of two primary components:

  • The Amazon EKS control plane
  • Amazon EKS nodes that are registered with the control plane

The Amazon EKS control plane consists of control plane nodes that run the Kubernetes software, such as etcd and the Kubernetes API server. The control plane runs in an account managed by AWS, and the Kubernetes API is exposed via the Amazon EKS endpoint associated with your cluster. Each Amazon EKS cluster control plane is single-tenant and unique, and runs on its own set of Amazon EC2 instances [4].

The kinds of the Amazon EKS cluster are public (non-private) and private cluster.

  • Public cluster (non-private cluster): All node groups in the cluster are in the public subnet.
  • Private cluster: All node groups in the cluster are in the private subnet. Thus, it enables that all pods (in nodes) in the node groups are only internally accessed.
2. Amazon EKS managed node group

A node group means a set of worker nodes which calls just nodes for short. A cluster consists of a number of node groups. Please note that a cluster can be composed of various types of the node groups (e.g. g4dn.xlarge and t5.large). There are two types of the node groups [5].

  • EKS managed node group
    • Flag: managedNodeGroups
    • When creating a worker node, it supports Amazon Linux based EKS optimized Amazon Linux 2 AMI.
    • It automates the provisioning and lifecycle management of nodes (Amazon EC2 instances) for Amazon EKS Kubernetes clusters [6].
  • Self managed nodes
    • Flag: nodeGroups
    • When creating a worker node, it supports other operating system (OS) which is not based on Amazon Linux (e.g. Windows).
3. Cluster Autosacler (CA)

The Kubernetes Cluster Autoscaler automatically adjusts the number of nodes in your cluster when pods fail or are rescheduled onto other nodes [7].

4. AWS Load Balancer Controller

The AWS Load Balancer Controller manages AWS Elastic Load Balancers for a Kubernetes cluster. Please note that The AWS Load Balancer Controller replaces the functionality of the AWS ALB Ingress Controller for Kubernetes [8].

5. Metrics Server

The Kubernetes Metrics Server is an aggregator of resource usage data in your cluster, and it is not deployed by default in Amazon EKS clusters. The Metrics Server is commonly used by other Kubernetes add ons, such as the Horizontal Pod Autoscaler (HPA) or the Kubernetes Dashboard [9].

6. Namespace

A namespace is a way to group services for an application. When you create a namespace, you specify how you want to discover service instances that you register with AWS Cloud Map: using API calls or using DNS queries. You also specify the name that you want your application to use to discover instances [10].

7. Deployment

A deployment provides declarative updates for pods and replicaset [11] In other words, the deployment manages a replicated application on the cluster. The pods are the smallest deployable units of computing that you can create and manage in Kubernetes [12]. The replicaset ensures that a specified number of pod replicas are running at any given time [13].

8. Service

An abstract way to expose an application running on a set of pods as a network service [14].

9. Horizontal Pod Autoscaling

In Kubernetes, a horizontal pod autoscaler (HPA) automatically updates a workload resource (such as a Deployment or StatefulSet), with the aim of automatically scaling the workload to match demand [15]. The statefulset is the workload API object used to manage stateful applications. it manages deployment and scaling of a set of pods, with durable storage and persistent identifier for each pod [16].

10. Ingress

Ingress is an Application Programming Interface (API) object that manages external access to the services in a cluster, typically Hypertext Transfer Protocol (HTTP) [17].

4. How to create a cluster

1. How to create a cluster

In about 40 minutes, you can create a cluster with a cluster.yaml. I recommend that you should check the field of the yaml file.

  • metadata:name: vujade-cluster
  • metadata:region: ap-northeast-2
  • metadata:version: “1.21”
  • vpc:subnets:private:
    • ap-northeast-2a: { id: subnet-012a3456b78cde9f0 }
    • ap-northeast-2c: { id: subnet-0123a4bc5d678e901 }
  • secretsEncryption:keyARN: arn:aws:kms:ap-northeast-2:123456789123:key/a0b1234c-de5f-6789-g01h-2i3456j789k0

In this case, the vpc:subnets:private is required because of the private cluster. The values of the field which can be obtained on VPC Dashboard correspond to the subnets of the data plane which are created in the Amazon Virtual Private Cloud (VPC) - Section 5. How to set up subnets. Also, as far as I mentioned in Amazon Virtual Private Cloud (VPC) - Section 5. How to set up subnets, the considered EC2 instance type, g4dn.xlarge is only supported ap-northeast-2a and ap-northeast-2c. The value of the secretsEncryption:keyARN is the MASTER_ARN which is created in the Amazon Web Services (AWS) Cloud9 - Section 8. How to create an AWS KMS custom managed key (CMK). Please note that any tags for the VPC are not required because the cluster version is 1.19+ as far as I mentioned in Amazon Virtual Private Cloud (VPC) - Section 4. How to create a VPC.



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15



 apiVersion: eksctl.io/v1alpha5
 kind: ClusterConfig
 
 metadata:
   name: vujade-cluster
   region: ap-northeast-2
   version: "1.21"
  
 vpc:
   subnets:
     private:
       ap-northeast-2a: { id: subnet-012a3456b78cde9f0 }
       ap-northeast-2c: { id: subnet-0123a4bc5d678e901 }
 secretsEncryption:
   keyARN: arn:aws:kms:ap-northeast-2:123456789123:key/a0b1234c-de5f-6789-g01h-2i3456j789k0



1 $ eksctl create cluster -f ./cluster.yaml

    2022-01-27 06:49:03 [ℹ]  eksctl version 0.80.0
    2022-01-27 06:49:03 [ℹ]  using region ap-northeast-2
    2022-01-27 06:49:03 [✔]  using existing VPC (vpc-012abcde34fg5678h) and subnets (private:map[ap-northeast-2a:{subnet-012a3456b78cde9f0 ap-northeast-2a 10.172.92.0/23} ap-northeast-2c:{subnet-0123a4bc5d678e901 ap-northeast-2c 10.172.94.0/23}] public:map[])
    2022-01-27 06:49:03 [!]  custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets
    2022-01-27 06:49:03 [ℹ]  using Kubernetes version 1.21
    2022-01-27 06:49:03 [ℹ]  creating EKS cluster "vujade-cluster" in "ap-northeast-2" region with 
    2022-01-27 06:49:03 [ℹ]  will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s)
    2022-01-27 06:49:03 [ℹ]  will create a CloudFormation stack for cluster itself and 0 managed nodegroup stack(s)
    2022-01-27 06:49:03 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-northeast-2 --cluster=vujade-cluster'
    2022-01-27 06:49:03 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "vujade-cluster" in "ap-northeast-2"
    2022-01-27 06:49:03 [ℹ]  CloudWatch logging will not be enabled for cluster "vujade-cluster" in "ap-northeast-2"
    2022-01-27 06:49:03 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=ap-northeast-2 --cluster=vujade-cluster'
    2022-01-27 06:49:03 [ℹ]  
    2 sequential tasks: { create cluster control plane "vujade-cluster", wait for control plane to become ready 
    }
    2022-01-27 06:49:03 [ℹ]  building cluster stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:49:04 [ℹ]  deploying stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:49:34 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:50:04 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:51:04 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:52:04 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:53:04 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:54:04 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:55:04 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:56:04 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:57:04 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:58:04 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 06:59:04 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 07:00:04 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-cluster"
    2022-01-27 07:02:05 [ℹ]  waiting for the control plane availability...
    2022-01-27 07:02:05 [✔]  saved kubeconfig as "/home/ubuntu/.kube/config"
    2022-01-27 07:02:05 [ℹ]  no tasks
    2022-01-27 07:02:05 [✔]  all EKS cluster resources for "vujade-cluster" have been created
    2022-01-27 07:02:06 [ℹ]  kubectl command should work with "/home/ubuntu/.kube/config", try 'kubectl get nodes'
    2022-01-27 07:02:06 [✔]  EKS cluster "vujade-cluster" in "ap-northeast-2" region is ready
2. How to check the created cluster

You can check the created cluster with three options.

  1. You can check the cluster on CloudFormation.
  2. You can check the cluster on Amazon Container Services - Clusters.
  3. You can check the cluster with an eksctl command. 
     1 $ eksctl get cluster

         2022-01-27 14:27:46 [ℹ]  eksctl version 0.80.0
     2022-01-27 14:27:46 [ℹ]  using region ap-northeast-2
     NAME            REGION          EKSCTL CREATED
     vujade-cluster  ap-northeast-2  True
3. How to connect a cluster

You can connect the created cluster as follows. Please note that the field, region and name are the AWS region and the cluster name, respectively.

1 $ aws eks --region ap-northeast-2 update-kubeconfig --name vujade-cluster

    Added new context arn:aws:eks:ap-northeast-2:123456789123:cluster/vujade-cluster to /home/ubuntu/.kube/config
4. How to delete a cluster

You can delete the created cluster as follows. Please note that the field, name is the cluster name.

1 $ eksctl delete cluster --name vujade-cluster

    2022-01-27 14:27:46 [ℹ]  eksctl version 0.79.0
    2022-01-27 14:27:46 [ℹ]  using region ap-northeast-2
    2022-01-27 14:27:46 [ℹ]  deleting EKS cluster "vujade-cluster"
    2022-01-27 14:27:46 [ℹ]  deleted 0 Fargate profile(s)
    2022-01-27 14:27:46 [✔]  kubeconfig has been updated
    2022-01-18 07:43:29 [ℹ]  cleaning up AWS load balancers created by Kubernetes objects of Kind Service or Ingress
    2022-01-27 14:27:30 [ℹ]  1 task: { delete cluster control plane "vujade-cluster" [async] }
    2022-01-27 14:27:30 [ℹ]  will delete stack "eksctl-vujade-cluster-cluster"
    2022-01-27 14:27:30 [✔]  all cluster resources were deleted

5. How to create a node group

1. How to create a node group

In about 20 minutes, you can create a node group with a nodegroup.yaml. Please note that the node group is based on the EKS managed node group in this case. I recommend that you should check the field of the yaml file.

  • metadata:name: vujade-cluster
  • metadata:region: ap-northeast-2
  • managedNodeGroups:name: ng-dl
  • managedNodeGroups:instanceType: g4dn.xlarge
  • managedNodeGroups:minSize: 1
  • managedNodeGroups:maxSize: 5
  • managedNodeGroups:desiredCapacity: 1
  • managedNodeGroups:volumeSize: 80
  • managedNodeGroups:privateNetworking: true
  • managedNodeGroups:ssh:allow: true
  • managedNodeGroups:ssh:publicKeyName: vujade-eks

The fields, managedNodeGroups:minSize, managedNodeGroups:maxSize and managedNodeGroups:desiredCapacity are parameters for the Cluster Autoscaler (CA) which scales up and down EC2 instances in node groups. the managedNodeGroups:volumeSize means the size of the Elastic Block Storage (EBS). The default value is 80. In this case, the managedNodeGroups:privateNetworking is true because the cluster is private cluster. The value of the field, managedNodeGroups:ssh:publicKeyName is the name of the SSH key obtained from the Amazon Web Services (AWS) Cloud9 - Section 7. How to create and import a SSH key.



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16



 apiVersion: eksctl.io/v1alpha5
 kind: ClusterConfig
 metadata:
   name: vujade-cluster
   region: ap-northeast-2
 managedNodeGroups:
   - name: ng-dl
     instanceType: g4dn.xlarge
     minSize: 1
     maxSize: 5
     desiredCapacity: 1
     volumeSize: 80
     privateNetworking: true
     ssh:
       allow: true
       publicKeyName: vujade-eks



1 $ eksctl create nodegroup -f ./eks_yaml/dev/nodegroup/nodegroup.yaml

    2022-01-27 07:11:22 [ℹ]  eksctl version 0.80.0
    2022-01-27 07:11:22 [ℹ]  using region ap-northeast-2
    2022-01-27 07:11:22 [ℹ]  will use version 1.21 for new nodegroup(s) based on control plane version
    2022-01-27 07:11:22 [ℹ]  nodegroup "ng-dl" will use "" [AmazonLinux2/1.21]
    2022-01-27 07:11:22 [ℹ]  using EC2 key pair "vujade-eks"
    2022-01-27 07:11:22 [ℹ]  1 nodegroup (ng-dl) was included (based on the include/exclude rules)
    2022-01-27 07:11:22 [ℹ]  will create a CloudFormation stack for each of 1 managed nodegroups in cluster "vuajde-cluster"
    2022-01-27 07:11:23 [ℹ]  
    2 sequential tasks: { fix cluster compatibility, 1 task: { 1 task: { create managed nodegroup "ng-dl" } } 
    }
    2022-01-27 07:11:23 [ℹ]  checking cluster stack for missing resources
    2022-01-27 07:11:23 [ℹ]  cluster stack has all required resources
    2022-01-27 07:11:23 [ℹ]  building managed nodegroup stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:11:23 [ℹ]  deploying stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:11:23 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:11:39 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:11:56 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:12:14 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:12:31 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:12:47 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:13:03 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:13:19 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:13:39 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:13:55 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:14:13 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:14:32 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:14:50 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:15:09 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:15:27 [ℹ]  waiting for CloudFormation stack "eksctl-vuajde-cluster-nodegroup-ng-dl"
    2022-01-27 07:15:27 [ℹ]  1 task: { install Nvidia device plugin }
    2022-01-27 07:15:27 [ℹ]  created "kube-system:DaemonSet.apps/nvidia-device-plugin-daemonset"
    2022-01-27 07:15:27 [ℹ]  as you are using the EKS-Optimized Accelerated AMI with a GPU-enabled instance type, the Nvidia Kubernetes device plugin was automatically installed.
            to skip installing it, use --install-nvidia-plugin=false.
    2022-01-27 07:15:27 [✔]  created 0 nodegroup(s) in cluster "vuajde-cluster"
    2022-01-27 07:15:27 [ℹ]  nodegroup "ng-dl" has 1 node(s)
    2022-01-27 07:15:27 [ℹ]  node "ip-10-172-95-87.ap-northeast-2.compute.internal" is ready
    2022-01-27 07:15:27 [ℹ]  waiting for at least 1 node(s) to become ready in "ng-dl"
    2022-01-27 07:15:27 [ℹ]  nodegroup "ng-dl" has 1 node(s)
    2022-01-27 07:15:27 [ℹ]  node "ip-10-172-95-87.ap-northeast-2.compute.internal" is ready
    2022-01-27 07:15:27 [✔]  created 1 managed nodegroup(s) in cluster "vuajde-cluster"
    2022-01-27 07:15:27 [ℹ]  checking security group configuration for all nodegroups
    2022-01-27 07:15:27 [ℹ]  all nodegroups have up-to-date cloudformation templates
2. How to check the created node group
  1. Check node groups.
    You can check node groups as follows. 
    1 $ eksctl get nodegroup --cluster vujade-cluster

     2022-01-28 00:47:46 [ℹ]  eksctl version 0.80.0
 2022-01-28 00:47:46 [ℹ]  using region ap-northeast-2
 CLUSTER         NODEGROUP       STATUS  CREATED                 MIN SIZE        MAX SIZE        DESIRED CAPACITY        INSTANCE TYPE   IMAGE ID        ASG NAME
 vujade-cluster  ng-dl           ACTIVE  2022-01-27T07:12:18Z    1               5               1                       g4dn.xlarge     AL2_x86_64_GPU  eks-ng-dl-0abc1d23-01a2-0a12-012a-a0bc1234d5e6
  2. Check nodes.
    You can also check nodes in the node groups. Please note that you should connect the created cluster as described in Section 5. How to create a cluster before listing nodes below. 
    1 $ kubectl get nodes --show-labels

     NAME                                              STATUS   ROLES    AGE   VERSION               LABELS
 ip-10-172-95-87.ap-northeast-2.compute.internal   Ready    <none>   17h   v1.21.5-eks-9017834   alpha.eksctl.io/cluster-name=vujade-cluster,alpha.eksctl.io/nodegroup-name=ng-dl,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=g4dn.xlarge,beta.kubernetes.io/os=linux,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup-image=ami-04a860d4e53598406,eks.amazonaws.com/nodegroup=ng-dl,eks.amazonaws.com/sourceLaunchTemplateId=lt-0fbc06a2d297490d2,eks.amazonaws.com/sourceLaunchTemplateVersion=1,failure-domain.beta.kubernetes.io/region=ap-northeast-2,failure-domain.beta.kubernetes.io/zone=ap-northeast-2c,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-172-95-87.ap-northeast-2.compute.internal,kubernetes.io/os=linux,node.kubernetes.io/instance-type=g4dn.xlarge,topology.kubernetes.io/region=ap-northeast-2,topology.kubernetes.io/zone=ap-northeast-2c
3. How to delete a node group

You can delete a node group as follows.

1 $ eksctl delete nodegroup --region=ap-northeast-2 --cluster=vujade-cluster --name=ng-dl

    2022-01-18 07:41:36 [ℹ]  eksctl version 0.79.0
    2022-01-18 07:41:36 [ℹ]  using region ap-northeast-2
    2022-01-18 07:41:36 [ℹ]  1 nodegroup (ng-dl) was included (based on the include/exclude rules)
    2022-01-18 07:41:36 [ℹ]  will drain 1 nodegroup(s) in cluster "vujade-cluster"
    2022-01-18 07:41:36 [!]  no nodes found in nodegroup "ng-dl" (label selector: "alpha.eksctl.io/nodegroup-name=ng-dl")
    2022-01-18 07:41:36 [ℹ]  will delete 1 nodegroups from cluster "vujade-cluster"
    2022-01-18 07:41:36 [ℹ]  1 task: { 1 task: { delete nodegroup "ng-dl" [async] } }
    2022-01-18 07:41:36 [ℹ]  will delete stack "eksctl-vujade-cluster-nodegroup-ng-dl"
    2022-01-18 07:41:36 [ℹ]  will delete 0 nodegroups from auth ConfigMap in cluster "vujade-cluster"
    2022-01-18 07:41:36 [✔]  deleted 1 nodegroup(s) from cluster "vujade-cluster"

6. How to configure a cluster autoscaler (CA)

You can configure a cluster autoscaler (CA) by executing following commands. You can get more information in the AWS EKS workshop guide, Configure Cluster Autoscaler (CA). Please note the CA only works when there is a node group at least.

I provide a bash script, bash_install_ca.sh to install the CA easily. Please note that you can get the ACCOUNT_ID in the Amazon Web Services (AWS) Cloud9 - Section 8. How to create an AWS KMS custom managed key (CMK).

1 $ bash ./bash_install_ca.sh NAME_CLUSTER ACCOUNT_ID
1. IAM roles for service accounts
  1. Enabling IAM roles for service accounts on your cluster (i.e. Creating IAM OIDC provider). 
    1 $ eksctl utils associate-iam-oidc-provider --cluster vujade-cluster --approve

     2022-01-28 08:41:31 [ℹ]  eksctl version 0.80.0
 2022-01-28 08:41:31 [ℹ]  using region ap-northeast-2
 2022-01-28 08:41:32 [ℹ]  will create IAM Open ID Connect provider for cluster "vujade-cluster" in "ap-northeast-2"
 2022-01-28 08:41:32 [✔]  created IAM Open ID Connect provider for cluster "vujade-cluster" in "ap-northeast-2"
  2. Creating an IAM policy for your service account that will allow your CA pod to interact with the autoscaling groups.
    You can create the IAM policy with k8s-asg-policy.json as below. Please note that you don’t need to create the customer managed IAM policy if it already exists as below. 
    1 $ aws iam create-policy --policy-name k8s-asg-policy --policy-document file://./k8s-asg-policy.json

     An error occurred (EntityAlreadyExists) when calling the CreatePolicy operation: A policy called k8s-asg-policy already exists. Duplicate names are not allowed.
  3. Finally, create an IAM role for the cluster-autoscaler Service Account in the kube-system namespace.
    You can obtain the policy ARN in IAM - Policies with k8s-asg-policy. 
    1 $ eksctl create iamserviceaccount \
2   --name cluster-autoscaler \
3   --namespace kube-system \
4   --cluster vujade-cluster \
5   --attach-policy-arn "arn:aws:iam::123456789123:policy/k8s-asg-policy" \
6   --approve \
7   --override-existing-serviceaccounts

     2022-01-28 09:15:51 [ℹ]  eksctl version 0.80.0
 2022-01-28 09:15:51 [ℹ]  using region ap-northeast-2
 2022-01-28 09:15:52 [ℹ]  1 iamserviceaccount (kube-system/cluster-autoscaler) was included (based on the include/exclude rules)
 2022-01-28 09:15:52 [!]  metadata of serviceaccounts that exist in Kubernetes will be updated, as --override-existing-serviceaccounts was set
 2022-01-28 09:15:52 [ℹ]  1 task: { 
     2 sequential sub-tasks: { 
         create IAM role for serviceaccount "kube-system/cluster-autoscaler",
         create serviceaccount "kube-system/cluster-autoscaler",
     } }2022-01-28 09:15:52 [ℹ]  building iamserviceaccount stack "eksctl-vujade-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
 2022-01-28 09:15:52 [ℹ]  deploying stack "eksctl-vujade-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
 2022-01-28 09:15:52 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
 2022-01-28 09:16:12 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
 2022-01-28 09:16:27 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
 2022-01-28 09:16:27 [ℹ]  created serviceaccount "kube-system/cluster-autoscaler"
  4. Make sure your service account with the ARN of the IAM role is annotated. 
    1 $ kubectl -n kube-system describe sa cluster-autoscaler 

     Name:                cluster-autoscaler
 Namespace:           kube-system
 Labels:              app.kubernetes.io/managed-by=eksctl
 Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::123456789123:role/eksctl-vujade-cluster-addon-iamserviceaccount-Role1-1DOGJCHJ8OE45
 Image pull secrets:  <none>
 Mountable secrets:   cluster-autoscaler-token-29s2h
 Tokens:              cluster-autoscaler-token-29s2h
 Events:              <none>
2. Deploy the Cluster Autoscaler (CA)
  1. Deploy the Cluster Autoscaler to your cluster with the following command.
    You can get a yaml file for deploying a cluster from official EKS workshop site or my GitHub. Please note that you should modify the cluster name to yours. 
    1 $ kubectl apply -f ./cluster-autoscaler-autodiscover.yaml

     clusterrole.rbac.authorization.k8s.io/cluster-autoscaler created
 role.rbac.authorization.k8s.io/cluster-autoscaler created
 clusterrolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
 rolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
 deployment.apps/cluster-autoscaler created
  2. To prevent CA from removing nodes where its own pod is running, we will add the cluster-autoscaler.kubernetes.io/safe-to-evict annotation to its deployment with the following command. 
    1 $ kubectl -n kube-system annotate deployment.apps/cluster-autoscaler cluster-autoscaler.kubernetes.io/safe-to-evict="false"

     deployment.apps/cluster-autoscaler annotated
  3. Finally let’s update the autoscaler image. 
    1 $ export K8S_VERSION=$(kubectl version --short | grep 'Server Version:' | sed 's/[^0-9.]*\([0-9.]*\).*/\1/' | cut -d. -f1,2)
2 $ export AUTOSCALER_VERSION=$(curl -s "https://api.github.com/repos/kubernetes/autoscaler/releases" | grep '"tag_name":' | sed -s 's/.*-\([0-9][0-9\.]*\).*/\1/' | grep -m1 ${K8S_VERSION})
3 $ kubectl -n kube-system set image deployment.apps/cluster-autoscaler cluster-autoscaler=us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v${AUTOSCALER_VERSION}

     deployment.apps/cluster-autoscaler image updated
  4. You can watch logs. 
    1 $ kubectl -n kube-system logs -f deployment/cluster-autoscaler
  5. Check the created cluster autoscaler. 
    1 $ kubectl get deploy -A 

     NAMESPACE     NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
 kube-system   cluster-autoscaler   1/1     1            1           7m39s
 kube-system   coredns              2/2     2            2           28h

7. How to install the AWS load balancer controller

You can install the AWS load balanacer controller by executing following commands. You can get more information in the AWS Load Balancer Controller, Ingress Controller and Install Kubernetes Tools.

I provide a bash script, bash_install_lbc.sh to install the AWS load balancer controller easily. Please note that you can get the ACCOUNT_ID and NAME_ROLE_AWS_LBC in the Amazon Web Services (AWS) Cloud9 - Section 8. How to create an AWS KMS custom managed key (CMK) and Step 7. Attach the IAM policy to the IAM role, respectively.

1 $ bash ./bash_install_lbc.sh NAME_CLUSTER ACCOUNT_ID NAME_ROLE_AWS_LBC
1. Deploy the AWS Load Balancer Controller
  1. We will verify if the AWS Load Balancer Controller version has been set. 
    1 $ echo 'export LBC_VERSION="v2.3.1"' >>  ~/.bash_profile
2 $ . ~/.bash_profile
3 $ if [ ! -x ${LBC_VERSION} ]
4     then
5       tput setaf 2; echo '${LBC_VERSION} has been set.'
6     else
7       tput setaf 1; echo '${LBC_VERSION} has NOT been set.'
8   fi

     ${LBC_VERSION} has been set.
  2. Create IAM OIDC provider.
    You can skip this step if you already execute the command in 7. How to configure a cluster autoscaler (CA) 
    1 $ eksctl utils associate-iam-oidc-provider --cluster vujade-cluster --approve
  3. Create an IAM policy, AWSLoadBalancerControllerIAMPolicy.
    Please note that you don’t need to create the customer managed IAM policy if it already exists as below. You can also download the iam_policy.json. 
    1 $ curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.3.1/docs/install/iam_policy.json
2 $ aws iam create-policy \
3   --policy-name AWSLoadBalancerControllerIAMPolicy \
4   --policy-document file://iam_policy.json

     An error occurred (EntityAlreadyExists) when calling the CreatePolicy operation: A policy called AWSLoadBalancerControllerIAMPolicy already exists. Duplicate names are not allowed.
  4. Create a IAM role and ServiceAccount.
    You can obtain the policy ARN in IAM - Policies with AWSLoadBalancerControllerIAMPolicy. 
    1 $ eksctl create iamserviceaccount \
2   --cluster vujade-cluster \
3   --namespace kube-system \
4   --name aws-load-balancer-controller \
5   --attach-policy-arn arn:aws:iam::123456789123:policy/AWSLoadBalancerControllerIAMPolicy \
6   --override-existing-serviceaccounts \
7   --approve

     2022-01-28 13:32:15 [ℹ]  eksctl version 0.80.0
 2022-01-28 13:32:15 [ℹ]  using region ap-northeast-2
 2022-01-28 13:32:16 [ℹ]  1 existing iamserviceaccount(s) (kube-system/cluster-autoscaler) will be excluded
 2022-01-28 13:32:16 [ℹ]  1 iamserviceaccount (kube-system/aws-load-balancer-controller) was included (based on the include/exclude rules)
 2022-01-28 13:32:16 [!]  metadata of serviceaccounts that exist in Kubernetes will be updated, as --override-existing-serviceaccounts was set
 2022-01-28 13:32:16 [ℹ]  1 task: { 
     2 sequential sub-tasks: { 
         create IAM role for serviceaccount "kube-system/aws-load-balancer-controller",
         create serviceaccount "kube-system/aws-load-balancer-controller",
     } }2022-01-28 13:32:16 [ℹ]  building iamserviceaccount stack "eksctl-vujade-cluster-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
 2022-01-28 13:32:16 [ℹ]  deploying stack "eksctl-vujade-cluster-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
 2022-01-28 13:32:16 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
 2022-01-28 13:32:34 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
 2022-01-28 13:32:53 [ℹ]  waiting for CloudFormation stack "eksctl-vujade-cluster-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
 2022-01-28 13:32:54 [ℹ]  created serviceaccount "kube-system/aws-load-balancer-controller"
  5. Download the IAM policy.
    You can also download the iam_policy_v1_to_v2_additional.json. 
    1 $ curl -o iam_policy_v1_to_v2_additional.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.3.1/docs/install/iam_policy_v1_to_v2_additional.json
  6. Create the IAM policy and note the ARN that is returned.
    Please note that you don’t need to create the customer managed IAM policy if it already exists as below. 
    1 $ aws iam create-policy \
2   --policy-name AWSLoadBalancerControllerAdditionalIAMPolicy \
3   --policy-document file://iam_policy_v1_to_v2_additional.json 

     An error occurred (EntityAlreadyExists) when calling the CreatePolicy operation: A policy called AWSLoadBalancerControllerAdditionalIAMPolicy already exists. Duplicate names are not allowed.
  7. Attach the IAM policy to the IAM role.
    You can find the role name as follows:
    • Open the AWS CloudFormation console
    • Select the eksctl-your-cluster-name-addon-iamserviceaccount-kube-system-aws-load-balancer-controller stack.
    • Click the Resources tab.
    • The role name is in the Physical ID column. bag
    1 $ aws iam attach-role-policy \
2   --role-name eksctl-vujade-cluster-addon-iamserviceaccount-Role1-1ABC2D3E4FGHI \
3   --policy-arn arn:aws:iam::123456789123:policy/AWSLoadBalancerControllerAdditionalIAMPolicy
  8. Add the eks-charts repository. 
    1 $ helm repo add eks https://aws.github.io/eks-charts

     "eks" has been added to your repositories
  9. Update your local repo to make sure that you have the most recent charts. 
    1 $ helm repo update

     Hang tight while we grab the latest from your chart repositories...
 ...Successfully got an update from the "eks" chart repository
 ...Successfully got an update from the "stable" chart repository
 Update Complete. ⎈Happy Helming!⎈
  10. Install the TargetGroupBinding CRDs. 
    1 $ kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master"
2 $ kubectl get crd 

    customresourcedefinition.apiextensions.k8s.io/ingressclassparams.elbv2.k8s.aws created
customresourcedefinition.apiextensions.k8s.io/targetgroupbindings.elbv2.k8s.aws created
    
NAME                                         CREATED AT
eniconfigs.crd.k8s.amazonaws.com             2022-01-27T06:57:16Z
ingressclassparams.elbv2.k8s.aws             2022-01-28T13:59:28Z
securitygrouppolicies.vpcresources.k8s.aws   2022-01-27T06:57:19Z
targetgroupbindings.elbv2.k8s.aws            2022-01-28T13:59:28Z
  11. Install the AWS Load Balancer Controller. 
    1 $ helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
2   -n kube-system \
3   --set clusterName=vujade-cluster \
4   --set serviceAccount.create=false \
5   --set serviceAccount.name=aws-load-balancer-controller

    NAME: aws-load-balancer-controller
LAST DEPLOYED: Fri Jan 28 14:02:37 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWS Load Balancer controller installed!
  12. Verify that the controller is installed. 
    1 $ kubectl get deployment -n kube-system aws-load-balancer-controller 

    NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   2/2     2            2           38s

8. How to install a metric server

You can install a metric server which is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines by executing following commands. You can get more information in the Configure Horizontal Pod Autoscaler (HPA).

1 $ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.0/components.yaml

    serviceaccount/metrics-server created
    clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
    clusterrole.rbac.authorization.k8s.io/system:metrics-server created
    rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
    clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
    clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
    service/metrics-server created
    deployment.apps/metrics-server created
    apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

1 $ kubectl get apiservice v1beta1.metrics.k8s.io -o json | jq '.status'

    {
      "conditions": [
        {
          "lastTransitionTime": "2022-01-28T14:15:04Z",
          "message": "all checks passed",
          "reason": "Passed",
          "status": "True",
          "type": "Available"
        }
      ]
    }

1 $ kubectl get pods -A

    NAMESPACE     NAME                                           READY   STATUS    RESTARTS   AGE
    kube-system   aws-load-balancer-controller-cb56f956d-cmvh7   1/1     Running   0          13m
    kube-system   aws-load-balancer-controller-cb56f956d-d76rf   1/1     Running   0          13m
    kube-system   aws-node-frdxk                                 1/1     Running   0          31h
    kube-system   cluster-autoscaler-77d7c6c9b8-8bzck            1/1     Running   0          155m
    kube-system   coredns-6dbb778559-2dhqq                       1/1     Running   0          31h
    kube-system   coredns-6dbb778559-t4twm                       1/1     Running   0          31h
    kube-system   kube-proxy-glbbl                               1/1     Running   0          31h
    kube-system   metrics-server-6dfddc5fb8-zq97v                1/1     Running   0          111s
    kube-system   nvidia-device-plugin-daemonset-q2fbv           1/1     Running   0          31h

9. How to apply a namespace

1. Preparations

You can check the namespace.yaml.

2. How to apply a namespace
1 $ kubectl apply -f namespace.yaml 

    namespace/ns-vujade created
3. How to check the applied namespace
1 $ kubectl get namespace

    NAME              STATUS   AGE
    default           Active   2d4h
    kube-node-lease   Active   2d4h
    kube-public       Active   2d4h
    kube-system       Active   2d4h
    ns-vujade         Active   50s
4. How to delete the applied namespace
1 $ kubectl delete -f namespace.yaml

    namespace "ns-vujade" deleted

10. How to apply a deployment

1. Preparations

You can check the dep_dl.yaml.

2. How to apply a deployment
1 $ kubectl apply -f deployment.yaml

    deployment.apps/dep-dl created
3. How to check the applied deployment
1 $ kubectl get deployment -A

    NAMESPACE     NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
    kube-system   aws-load-balancer-controller   2/2     2            2           21h
    kube-system   cluster-autoscaler             1/1     1            1           23h
    kube-system   coredns                        2/2     2            2           2d4h
    kube-system   metrics-server                 1/1     1            1           21h
    ns-vujade     dep-dl                         3/3     3            3           32s
4. How to delete the applied deployment
1 $ kubectl delete -f deployment.yaml

    deployment.apps "dep-dl" deleted

11. How to apply a service

1. Preparations

You can check the svc_dl.yaml.

2. How to apply a service
1 $ kubectl apply -f service.yaml

    service/svc-dl created
3. How to check the applied service
1 $ kubectl get service -A

    NAMESPACE     NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
    default       kubernetes                          ClusterIP   172.20.0.1       <none>        443/TCP           2d4h
    kube-system   aws-load-balancer-webhook-service   ClusterIP   172.20.122.144   <none>        443/TCP           21h
    kube-system   kube-dns                            ClusterIP   172.20.0.10      <none>        53/UDP,53/TCP     2d4h
    kube-system   metrics-server                      ClusterIP   172.20.46.230    <none>        443/TCP           21h
    ns-vujade     svc-dl                              NodePort    172.20.205.201   <none>        11001:31803/TCP   15s
4. How to delete the applied service
1 $ kubectl delete -f service.yaml

    service "svc-dl" deleted

12. How to apply a horizontal pod autoscaler (HPA)

1. Preparations

You can check the hpa_dl.yaml.

2. How to apply a HPA
1 $ kubectl apply -f hpa.yaml

    horizontalpodautoscaler.autoscaling/hpa-dl created
3. How to check the applied HPA
1 $ kubectl get hpa -A

    NAMESPACE   NAME     REFERENCE           TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
    ns-vujade   hpa-dl   Deployment/dep-dl   0%/65%    3         12        3          20s
4. How to delete the applied HPA
1 $ kubectl delete -f hpa.yaml

    horizontalpodautoscaler.autoscaling "hpa-dl" deleted
5. How does the HPA work?

The HPA is based on how many resources are being used compared to the requests allocated to the pod. In other words, the condition for scale-out of pods by the HPA as follows:

\begin{equation} \label{eq:cauchy-schwarz} averageUtilization \le 100 * \frac{CPU_{Total}}{N_{Pod} \times CPU_{Request}} \end{equation}

  • CPUTotal is the total central processing unit (CPU) usage in a worker node.
  • NPod is the number of pod in a worker node.
  • CPURequest is assigned in the deployment. It means minimum resources that the CPU must be guaranteed. Please note that the CPU usage is controlled in units of milli-cores (i.e m) by the Kubernetes. A single CPU is equivalent to 1m.

13. How to apply an ingress

1. Preparations

You can check the ingress.yaml.

2. How to apply an ingress
1 $ kubectl apply -f ingress.yaml

    ingress.extensions/alb-ing-eks-vujade created
3. How to check the applied ingress
1 $ kubectl get ingress -A

    NAMESPACE   NAME                  CLASS    HOSTS   ADDRESS                                                                                PORTS   AGE
    ns-vujade   alb-ing-eks-vujade   <none>   *       internal-k8s-nsvujade-albingea-1716e099f7-114012471.ap-northeast-2.elb.amazonaws.com   80      54s
4. How to delete the applied ingress
1 $ kubectl delete -f ingress.yaml

    ingress.extensions "alb-ing-eks-vujade" deleted

14. Useful kubectl commands

1. How to list nodes in details
1 $ kubectl get nodes -A -o wide
2. How to list pods in details
1 $ kubectl get pods -A -o wide
3. How to watch logs for a pod
1 $ kubectl logs -f --namespace ${NAMESPACE} ${NAME_POD}
4. How to execute a command for a pod
1 $ kubectl exec --namespace ${NAMESPACE} ${NAME_POD} ${COMMAND}

15. How to fix error

1. Cannot create a node group

When the node is deployed in a private subnet, the subnet must have a route to a NAT gateway where a public IP address is assigned. Amazon Virtual Private Cloud (VPC) - Section 10. How to set up a route table for the NAT gateway. The error messages are as follows.

    2022-01-18 08:32:14 [✖]  unexpected status "ROLLBACK_IN_PROGRESS" while waiting for CloudFormation stack "eksctl-vujade-cluster-nodegroup-ng-dl"
    2022-01-18 08:32:14 [ℹ]  fetching stack events in attempt to troubleshoot the root cause of the failure
    2022-01-18 08:32:14 [✖]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_FAILED – "Nodegroup ng-dl failed to stabilize: [{Code: NodeCreationFailure,Message: Instances failed to join the kubernetes cluster,ResourceIds: [i-0a12bc3d45e67fg8h]}]"
    2022-01-18 08:32:14 [ℹ]  1 error(s) occurred and nodegroups haven't been created properly, you may wish to check CloudFormation console

16. Reference

  1. Regions and Zones
  2. What is Kubernetes?
  3. Amazon EKS features
  4. Amazon EKS clusters
  5. Amazon EKS nodes
  6. Managed node groups
  7. Autoscaling
  8. AWS Load Balancer Controller
  9. Installing the Kubernetes Metrics Server
  10. Working with Namespaces
  11. Deployments
  12. Pods
  13. ReplicaSet
  14. Service
  15. Horizontal Pod Autoscaling
  16. StatefulSets
  17. Ingress