Amazon EKS Add-Ons

So the first thing is the introduction. What [are] the EKS add-ons? So, EKS add-ons [are] specifically made for the EKS, which is AWS’s managed Kubernetes cluster. As it’s a Kubernetes cluster, Kubernetes clusters require components and software for each and every work you want to do on it, like networking, compute storage, monitoring, [and] logging. 

For each and every aspect, you require a piece of software to achieve your goal. And EKS add-ons [are] the software to support you in operational capabilities. EKS add-ons are managed by Kubernetes communities, AWS themselves as well, and third-party vendors as well, like monitoring tools—for example, Datadog. Third-party vendors also create their own add-ons.

Let’s start [with] advantages of EKS add-ons. EKS add-ons are similar to Helm charts, which [are] packaged applications for Kubernetes. It’s very, very similar. But compared to Helm charts, it’s very easy to install. 

Another main advantage of add-ons is on the Helm charts: You need to find a proper version compatible with your Kubernetes version. And each and every three or four months, Kubernetes releases a version upgrade and releases a new version. So for each and every Kubernetes version, you need to find a compatible version for your software and for your Helm charts. Finding the compatibility, it’s very, very easy. In the Helm charts, there are lots of values and lots of configurations that you can do. The add-ons do not support that [many] configurations. But it does support some configurations, which I will show you. Compared to the Helm chart, it’s very easy to maintain. Right, so that is a main advantage, as well.

So now let’s move to when we should use EKS add-ons. As per my experience, only AWS-managed add-ons should be used. I do not suggest [or] recommend using third-party add-ons. The main reason for that is add-ons require pods to be deployed on worker nodes. So before deploying the add-ons, worker nodes should be deployed. 

If it’s not there for some scenarios, and for some reason, if the add-on is failing—for me, I tried more than 10 times [and] it fails me every time, even if I fix the issues, the add-on will still keep failing—so to make the add-on work again, delete the Kubernetes cluster and redeploy it again. 

So that is not a production case scenario for us—for the operation people. For that reason, it fails lots of times, and it’s not “kind of” production ready. So for third-party vendors and add-ons other than AWS managed, I do not recommend to use it. For that purpose, we should use AWS-managed add-ons only. The main add-on is the VPC CNI add-on which I will show in the next demo.

So, why? The reason we started using add-ons is there is a feature update after 14 December [2022], which allowed us to set up the advanced configurations and settings to the add-ons and the software. Before, it was not supported. So before December 2022, add-ons were completely useless. Add-ons were used just for a version upgrade of your software and if you don’t want to do any kind of configuration. Now, as it supports the configuration, I will show you where I started using it and how we can use it.

So this is the blog where Amazon launched this advanced configuration. If you want to go for it, I will provide a link later on. And this is the Terraform version, which is older than 4.46. Here, as you can see, it does not support any configuration values. And now I will show you. I will go to this latest version. And here, as you can see, it supports the configuration values. We can provide advanced configuration values to our add-ons. 

One of the reasons is [that], on the VPC CNI, if you want to enable different delegation and warm prefix targets, those are the configurations that are much needed on the production scenarios. Because without this, your EC2 instance will [limit and restrict] the pods to a number of 25 or 35 only. [By] enabling this feature, you can deploy pods of more than 110 or 125. On production, it will reduce your cost for the pods very, very [much]. It must have settings for the production. Mostly, we suggest configuration for production only. 

So, for this reason, if we wanted to use this configuration before, we needed to have deployed via Kubernetes manifest files, which is defined here. So if you want an upgraded version of this VPC CNI, you need to find a proper version for it. You need to generate this configuration file from scratch every time. So it’s a big hurdle just to update the environment variables only. So via the EKS add-ons and VPC CNI add-ons, we can very, very easily upgrade and add additional configurations to our add-ons. So basically, we require these two environment variables, and it’s very easy. So this is the main reason we started using the add-ons. Before, this was not supported. 

Another thing that I mentioned, right, is that it fails. EKS also supports add-ons for CoreDNS, cube proxy, and PVSCSI driver, but all other add-ons require pods to work. So for that, there is a workaround as well. I created this Terraform module and put my add-ons inside that module. And I provided explicit dependencies. If I don’t use this dependency, and if add-on creation has started before the node’s working, it will fail. So By providing this particular configuration, Terraform Core will make sure that, always, worker nodes will deploy first, and after that, the add-ons will be deployed. So that it will not fail and we don’t have to go through failure scenarios again and again.

Right. So, these are the basic commands that I will provide you in the link, as well. I will start with the deployment process of the add-ons. 

So, here we need to provide an add-on version as well, and I will show you how to find the proper add-on version for your EKS cluster. This is the first command, and in this first command, we will be using the EKS CTL CLI tool. The reason why we use this tool is [because] the basic AWS CLI tool does not support this functionality for a particular Kubernetes version. It requires that you have a Kubernetes cluster deployed, and then you can find the proper version for your Kubernetes cluster deployed. 

So first, you have to deploy a cluster only without any add-ons and worker nodes or anything. Then, via CLI, you need to find it. So this EKS CTL CLI supports us without deploying the Kubernetes cluster. So, I will run this command, which will list all available add-ons and provide it into the JSON file, which is this available add-ons JSON file, which is defined here. This file will have all the available add-ons for our Kubernetes version 1.27. I’m using 1.27, which is the latest version for EKS.

Here, I need to find which are the versions that my Kubernetes 1.27 EKS cluster supports, right? You have to find your EKS add-on, as well. I will just find VPC CNI. It’s here and 1.13.3 EKS 1. This is the latest version that I can utilize. As you can see, I’m using it, right? So, basically, just updating the version, it will not provide us with the configuration values that we need, right? We need to update an environment variable. How can we do that? How did I find that this is the JSON block that I need to use so that I can update the configuration? Other than E and V, I need to update the resource values, limitations, and everything. For that, there is another command EKS CTL describe add-on. And on that VPC CNI-specific version, we need to provide. And it will also provide us with a JSON output. So I will provide you with the details of how we can go through the JSON and understand what kind of configuration is supported.

So, in this JSON schema, only the properties are the values we can provide to EKS add-ons. As you can see, this is ENI config ENV. So there [are] no small ENVs available here. We need to scroll through and go through the whole JSON file. I am already used to it. So, in this resources section, you can provide the properties like remit and request. It’s a Kubernetes limit and request that you can provide. So, if I want to update my VPC CNI with the resources, this is the section that I need to look into, and it is referencing the capital resources. I need to go to the capital resources section, and inside that, the properties are only supported, which is our limits and the requests, right?

So this VPC CNI supports—as I have told you, I have used this ENV—it supports this livenessProbe, affinity, ENIconfig, [and] tolerations. And I will show you that if I want to use these resources, my add-on will look like this. These are the resources—we need to find the proper resource value for it. I will take this reference of limits, memory, and CPU from here. So this way, add-ons will provide resource limits on this memory. And we need to define the limit, as well. 

So this is the way you can go through this JSON file, read through this JSON file, understand it, and provide the proper configuration that you require.