nSights Talks

S3 Storage Lens

Tutorial Highlights & Transcript

00:00 - What is S3 Storage Lens

Hello, everyone. My name is Hamza. Today I’m presenting a demo about AWS S3 storage lens. Basically S3 storage lens is a service provided by Amazon which will help you to understand, analyze, and optimize your storage, activity metrics, and interactive dashboards. It also aggregate your data for your entire organization. When I say organization, I’m talking about AWS organization with multiple accounts. If you want to do it with specific accounts, you can do that with region base, bucket base, and all the prefixes. Everything is visible and accessible within your S3 Management Console. You can do it also via AWS CLI, or REST API, or you can also use SDKs as well to capture all that data. I prefer to do it with management console, because that gives me a better graphical interface.

How S3 Storage Lens works. It’s a storage lens that provides you a single view of usage and activity across your S3 storage. Let’s say if you are having 50 buckets in one account and 50 buckets in another account, and every account is a part of AWS organization. It will help you to get all the metrics out of your organization’s account or a specific account. You will see a single view where you can realize how much GB is present for that particular bucket. Whether you can optimize it or you can delete it or not depends upon the requirement you have. You can also use S3 Storage Lens to get summary insights, such as finding out how much storage you have across your entire organization, or which are fastest growing buckets. Identify the outliers in your storage metrics and drill down further to investigate the source of the spike in the usage or activity. It uses the best practices, you can also assess the best practices out of the storage lens such as analyzing the percentage of the bucket usage. If you have encryptions or if the object in the bucket is locked or not. All those best practices come into the picture. That’s where the storage lens will help you out.

01:33 - How S3 Storage Lens Works

How S3 Storage Lens works. It’s a storage lens that provides you a single view of usage and activity across your S3 storage. Let’s say if you are having 50 buckets in one account and 50 buckets in another account, and every account is a part of AWS organization. It will help you to get all the metrics out of your organization’s account or a specific account. You will see a single view where you can realize how much GB is present for that particular bucket. Whether you can optimize it or you can delete it or not depends upon the requirement you have. You can also use S3 Storage Lens to get summary insights, such as finding out how much storage you have across your entire organization, or which are fastest growing buckets. Identify the outliers in your storage metrics and drill down further to investigate the source of the spike in the usage or activity. It uses the best practices, you can also assess the best practices out of the storage lens such as analyzing the percentage of the bucket usage. If you have encryptions or if the object in the bucket is locked or not. All those best practices come into picture. That’s where the storage lens will help you out.

03:23 - S3 Storage Lens Diagram

Here’s the diagram which they have. They have S3 Storage Lens, it works to configure with organization, different accounts, specific accounts, based on your choice of regions and buckets. It aggregates the metrics by scanning internal S3 data you have and analyzes your metrics in the dashboard for a better view. Here are some use cases. It has summary insights, flag outliers, data protection, and cost efficiency.

04:11 - Demo - Using S3 Storage Lens

Let’s start the demo. Let’s go to the AWS Management Console. As you can see, under S3 console, I have four buckets, and I have created one, but it’s empty. Obviously I don’t have much data over here, but still I tried to show you what kind of size and everything is there. Here is the total storage with all those buckets. We have five objects account, average object size, and here’s the default dashboard they have. We can also make a custom dashboard. Let’s start. Basically it’s a default dashboard. You can also create a custom dashboard. Here you can filter out, select all the regions, I have only one. AWS regions, storage, classes, buckets. Here are the options.

In overview we have all that data, which previously I had mentioned. We have a total storage of 273 KB, five objects, we have active buckets, we have active account one. So here, we can check the summary, the cost efficiency of our buckets in the Data Protection. You can drill it with day by day, week by week, month by month.

Next, here are the trends and distribution. From here, we select the metrics. We have different metrics – schemetry metrics, primary metrics. Obviously we have no data there. It looks like this. Here we have storage class distribution. Region distribution, we use almost a standard class. Here are the graphs. Here are the top overview. In there we have selected with the last three top most three buckets here, you can change these with two with one. I’m going with three. Here we have one account and we’ve used two regions and also two active buckets. From accounts, we have to monitor these trends, distribution by Accounts mobile analytics from accounts. Here we can check trend of region which we have only two regions. We need to buy storage classes and here we can montior by buckets.

Hamza Chauhdary

DevOps Support Engineer

nClouds

Hamza is a DevOps Support Engineer at nClouds.