Using Amazon CloudWatch Container Insights with Amazon EKS

What are container insights? As we all know, Amazon CloudWatch is used to monitor AWS resources and the applications running on AWS. And CloudWatch has a cool feature called container insights, which is used to collect aggregate metrics and logs from your containerized applications and microservices. We can use container insights with Amazon ECS, Amazon EKS, and Kubernetes running on EC2 and Fargate. Container insights collect aggregated metrics at the cluster level, node level, pod, and service level. And these can be observed in automatic dashboards created by CloudWatch. And also we can create alarms based on these metrics.

This is the architecture overview of the container insights. This is how container insights work. In this example, two worker nodes are registered with this EKS cluster. We have to install the fluent bit, which is a logging agent used to stream logs from our applications to the CloudWatch. And another component is the CloudWatch agent. We will install these components as demonstrated, which means that one pod related to each component will be run on each node. As you can see here, logs and metrics collected by these daemon sets are ingested to the CloudWatch, where we can use the container insights to view the metrics logs, and we can create alarms based on these metrics. Let us see how we can enable the container insights in our EKS Clusters.

Firstly, I have the EKS cluster up and running. And first of all, we need to attach an IAM policy to the IAM role of the worker nodes so that it can send the metrics to the CloudWatch. Let me open the IAM role which is attached to the worker nodes. Here I am going to add permission, which is a CloudWatch Server Agent policy. This is the policy. It will allow our worker nodes to send the metrics and logs to CloudWatch. Next, let’s install the container insights. To install the container insights, I’m going to use a quick setup method and I’m going to run this command. In this command, I am mentioning the cluster name here and the region in which our cluster is running. So, let me execute this. It will create the required components like fluent bit, a daemon set, and CloudWatch agent and cluster role and cluster role bindings for these components after executing this command. This will complete in a few seconds. Okay, let us verify if pods related to these components are in a running state. CloudWatch agent is in running state and fluent bit is in container creating state let us verify one more time. Okay, pods are now in a running state. Our installation is completed here.

Now let’s explore the CloudWatch console and see how we can use the container insights. On the left-hand side, we have to click on the container insights. It will take a few seconds to populate all that data from the EKS cluster to these dashboards.

This is the resource dashboard and we get all the resources that are running currently in our EKS cluster here along with the CPU utilization and the memory utilization. Below this, you will get the alarms that are created or we will create them in a few minutes. You can also click on any component here and you can filter these resources from the search bar. If I want to search the pods, I can type the pod here and we will get all the related pods here.

The second dashboard that is provided by the container insights is the container map. Here we get the map view of our EKS cluster that is running in our AWS environments. You can see all the related components we can see here like all the namespaces in the EKS cluster and all the pods which are running under the namespaces. If you hover your cursor over these icons, you will get detailed information about CPU utilization and memory utilization. Let me see if our cluster is showing here because it will take a few minutes to populate all that data to the CloudWatch. If you want to get the details, you can click on any component here and a small dashboard will open here. This will show you the CPU utilization of that cluster.

The third dashboard is the performance managing dashboard. Here you get the details and metrics about the cluster. You can get the details at the cluster level, the namespace level, or the pod levels. You can select the clusters which are running from here. This is our EKS cluster, which is named Friday demo. And you can see that at the cluster level, we get that details about the CPU utilization, memory utilization, cluster failures, and disk utilization. Also, we can enable the Application Insights for this EKS cluster.

Let me show you the pod-level metrics here. These are the metrics for the pods which are running currently in our cluster. You can filter the pod by selecting the names of the pods on the right-hand side here, and you will get the metrics for that specific pod. When you scroll down, you will also get the container performance metrics here, where you can see that in that specific pod, we have two containers running. You will also get the CPU and the memory utilization here. On the right-hand side, you can click on the actions and you get the application logs for that specific container. And you also get the performance logs. If you have ingested the extra traces to your application, then you will get an option to check all the extra traces for that related pod.

Let me show you the log groups that are provided by the container insights. Container insights provide us with four types of log groups – application, performance, data plane, and host. You can see all the logs related to our components here. Every application and the performance logs you can see in the log groups here. The next option is the log insights. Using log insights, we can query our log groups. We can get enhanced metrics to see what is going on behind our cluster.

Let me show you how we can run queries on the log insights. Firstly, we have to select the log group in which we have to run that query. I’m going to select the performance log group here. And for the query, let me copy this. We can paste this query here and run that query. And we will get the details about the pods like this query showing the requested number of pods and currently running pods in our EKS cluster. Also, we can create a dashboard on behalf of these queries from this console. I can create a dashboard from this page so that we can get an overview of all the critical metrics, which we get by running the queries on the log groups. We can add different types of metrics by running the other kind of queries to the log groups. We can get all the details about our cluster. We can create alarms based on these metrics from here. When we install those container insights, we get that custom namespace for the container insight. Using this container insight namespace, we can create alerts or alarms based on different metric names like you can create the alert on the pod memory utilization or CPU memory utilization. By using these container insights, we will get all the details like metrics and alerts in one AWS console using the container insights. So that is all from my side. Thank you, everyone.