AWS Data Collection Prerequisites for an IAM Role

AWS Data Collection Prerequisites for an IAM Role

#410060

Overview

Densify collects resource utilization metrics (CloudWatch data) for your AWS services (e.g. EC2, RDS, ECS, etc.), analyzes the AWS data and then makes recommendations to save costs and reduce risks in your AWS environments.

You can also collect billing data and then use the Cloud Cost Intelligence module to analyze your billing data. You must create a second cloud connection to collect your billing data.

Collecting data via a cross-account IAM Role simplifies the process of connecting to multiple AWS accounts from Densify since the same role and external ID can be used across your multiple AWS accounts. As accounts are added or removed, you do not need to update the Densifycloud connection.

Note: Though the cloud connection wizard provides the option to use an IAM user and an access key, Densify recommends using the IAM Role.

The following list summarizes the prerequisites steps to be completed in your AWS account before you can create a Densify connection to collect CloudWatch utilization data.

  • In each linked and payer account, create an AWS role. Optionally, use the same name and external ID. You must enter the following Densify account that will assume the role:
  • 036437403198

    This is the AWS account that becomes the trusted entity.

  • Select and assign AWS's predefined, ReadOnlyAccess policy. This policy is an easy option for non-sensitive accounts. In most cases, you will need to create a permission policy to grant minimum access.
  • Once the AWS role has been created, copy the ARN and the External ID to create the connection.
  • Define your resource tags. The value of the resource tag is displayed and can be used for filtering within Densify. You must tag your resources appropriately and then map your AWS Resource tags to Densify attributes so the tags will be included in the analyses. Contact your Cloud Advisor for details.

When collecting billing data you must also complete the following additional steps:

Creating Connections from the Payer Account

In addition to creating connections from your linked accounts, you also need to create a Resource Utilization Metrics (CloudWatch) connection for your payer account. The payer account connection provides the account names and provides additional data when working with reserved instances and savings plans.

Working with Reserved Instances

When working with Reserved Instances you must collect both CloudWatch and billing data from all accounts in which you are using Reserved Instances. Since RIs can be shared between linked accounts they are often purchased through the payer account, so in order to see how the RI's are utilized, collect both CloudWatch and billing data from the payer account

Collecting Operating System Data

Operating System (OS) details are populated for AWS entities collected via the CloudWatch audit. OS details are not populated for entities discovered, exclusively through billing data collection.

Collecting Billing Data

Densify can collect a Cost and Usage report from a specified S3 bucket, analyze the data and generate various cost optimization reports. You can review these reports through the Cloud Cost Intelligence console. See Viewing Your Public Cloud Billing Reports.

There are additional prerequisites for collecting billing data. See Collecting Billing Data .

If you are planning to collect billing data, in addition to your CloudWatch data you may want to create the required cost and usage report and the associated policies first and then create the IAM Role.

In addition to populating the cloud cost reports, collecting the billing data provides the following added features to your Public Cloud optimization reports:

  • Calculation of reserved instance (RI) utilization details;
  • Insight into software utilization on each instance. i.e. your instance's installed OS, if it is an OS other than Windows.
  • Provides installed application details for non-Windows SQL installations.
  • Details of whether you are using bring-you-own license options, which can impact cost and effort estimates.

When working with Reserved Instances you must collect CloudWatch data from the payer account that is used to purchase the RIs, using the Resource Utilization Metrics connection type (also known as CloudWatch data collection).

Note: Reserved instance and savings plans do not span multiple payer accounts.

Using an IAM Role

To learn more watch the video:

When you create a role for cross-account access, you establish trust from the customer's account that owns the role (and the resources (trusting account) to the Densify account containing the user that will collect data (trusted account). You specify the trusted account number as the Principal in the role's trust policy when you create the role. This allows the Densify user in the trusted account to assume the role and collect utilization and billing.

In order to create an AWS connection in Densify to collect your AWS resource utilization (CloudWatch) data you need to create an IAM role for every linked or payer account.

When collecting billing data, you only need to create a connection to each payer account. The cost and usage report contains billing data from all of the accounts that are linked to the payer account.

Follow the process below to create and configure the IAM role for CloudWatch data collection.

Creating the IAM Role and Attaching a Permission Policy to Collect CloudWatch Data

This role allows you to collect CloudWatch data for the selected account. You need to attach a policy that allows the role to collect the required CloudWatch resource utilization metrics.

  1. Log into the AWS Management Console and navigate to Services > Security, Identity & Compliance > IAM. In the navigation tree on the left, click Roles.
  2. Click Create Role in the Roles dashboard.
  3. Select the Another AWS account type of trusted entity.
  4. Enter an Account ID. This is the Densify account that will assume the role. Enter the following Densify account ID: 036437403198.
  5. Select Require external ID and enter your external ID. This value is similar to a password and should be unique and difficult to guess. Densify recommends using a password generator to create a random, alphanumeric string (e.g. ae91ccf4) for the external ID.
  6. You will need this external ID later when creating the cloud connection from Densify.

  7. Click Next: Permissions.
  8. Attach the appropriate permission policy to the role. Select AWS's predefined ReadOnlyAccess policy. Use the filter to find the ReadOnlyAccess policy.
  9. Note: The ReadOnlyAccess policy is provided here as an easy option for non-sensitive accounts. In most cases, you will need to create a custom permission policy to grant Densify, the minimum permissions to collect only the required CloudWatch data . Refer to Creating an IAM Policy with Minimum Permissions for the CloudWatch Data Collection for details.

    For an IAM role that will be collecting both CloudWatch and billing data, you need to attach 2 policies:

    • ReadOnlyAccess or a customized minimum requirement policy that allows this role to collect CloudWatch data.
    • A policy granting the role access to the S3 bucket containing the cost and usage report. See Granting the IAM Role Access to the S3 Bucket for details on creating a custom policy for accessing billing data. This policy is in addition to the ReadOnlyAccess policy listed above.
  10. After selecting the permission policies for the role, click Next: Review.
  11. In the Review page, specify the Role name* and Role description. The role name can be any string used to identify and describe the role within the AWS account (e.g. DensifyCrossAccountRole).
  12. Click Create role. The new role is created.
  13. From the Roles page, click on the role name that you have just created, to view the role summary.
  14. Copy and save the Role ARN as you will need to paste this string into the Densify Cloud Connection wizard to create the connection.
  15. Continue with one of the following options:

Collecting Billing Data

Densify collects a Cost and Usage report from a specified S3 bucket, analyzes the data and generates various reports that allow you to view and analyze your billing data. If you have linked your AWS account to a payer account, billing data collection only needs to be performed on the payer account. If you have not linked your AWS accounts, billing data must be collected from each AWS account.

When working with Reserved Instances you must collect both CloudWatch and billing data from all accounts in which you are using Reserved Instances. Since RIs can be shared between linked accounts they are often purchased through the payer account, so in order to see how the RI's are utilized, collect both CloudWatch and billing data from the payer account

If you are planning to collect billing data you may want to create the required cost and usage report and the associated policies first and then create the IAM Role.

To learn more watch the video:

When collecting billing data the following additional configuration is required:

  1. Configuring a Cost and Usage Report;
  2. Granting the IAM Role Access to the S3 Bucket

Configuring a Cost and Usage Report

In order to collect billing data, Densify requires a Cost and Usage Report (CUR). You can use an existing report, or you can create a new report

Note: You need admin privileges, in order to create roles and to create and assign policies.

  1. Login to AWS management console of the payer account from which you want to collect data.
  2. Navigate to the AWS Cost and Usage Reports dashboard and determine if you have an existing cost and usage report (CUR) that satisfies Densify’s billing data collection requirements.

Required Cost and Usage Report Configuration

The report must have the following delivery options selected:

  • Time granularity—set to "Daily"
  • Additional report details, include "Resource IDs"
  • Compression type—GZIP.
  • Note: Do not select QuickSight or Athena, as these formats are not supported. Densify still only picks up the .CSV from the bucket.

  • TXT/CSV format

Optionally, enable loading of the report to RedShift.

Granting the IAM Role Access to the S3 Bucket

For AWS billing connections, you need to configure an IAM role in the payer account with a policy that grants the IAM role access to the S3 bucket to which the billing report is being stored. This policy is in addition to the policy appended to the bucket, enabling the CUR to be added to the bucket.

When working with savings plans you need to add an additional permission to the policy to enable Densify to collect the associated billing data:

  • ce:GetSavingsPlanUtilizationDetails

The following steps assume that have created the role and are on the Summary page.

  1. In the navigation tree on the left, click Policies
  2. Click Create policy. The Create Policy page opens.
  3. Click the JSON tab and enter the permission policy. You can copy this sample into the JSON tab and replace the highlighted string with your S3 bucket name.

Note: If you are using this sample to create your policy, you must provide the name of your specific bucket, as the resource. In the example below it is indicated as, "billing-report-for-mycompany".

The sample policy below contains sufficient permissions for collecting the cost and usage report from your S3 bucket. The additional Cost Explorer permission, is required for collecting savings plan cost data.

Copy

Example: AWS Minimum User Permission Policy for Billing

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                    "iam:GetUser",
                    "ce:GetSavingsPlansUtilizationDetails",
                    "cur:DescribeReportDefinitions",
                    "organizations:DescribeOrganization",
                    "organizations:ListAccounts"
            ],
            "Resource": [
                    "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action":[
                    "s3:Get*",
                    "s3:List*"
            ],
            "Resource":[
                    "arn:aws:s3:::billing-report-for-mycompany*"
            ]
        }
    ]
}
  1. Click Review Policy.
  2. Enter a name and description for the policy. i.e. "Grant_Role_Permission_to_Bucket"
  3. Click Create policy. You will be returned to the Policies page.
  4. In the navigation tree on the left, click Roles and then click on your newly created role.
  5. Click Attach policy.
  6. Use the filter to find the policy you created above. i.e. "Grant_Role_Permission_to_Bucket".
  7. Click Attach policy. You are returned to the Roles page.
  8. Click on your role and validate there are now two polices attached.
  9. You can now create AWS billing connection through the Densify Public Cloud Connection wizard. See Using the Public Cloud Connections Wizard.

Creating the Cloud Connection in Densify

Once all of the prerequisites are complete, you can create the cloud connection through the Cloud Connection wizard. See Using the Public Cloud Connections Wizard.

You can also use the Densify API. See Analysis: AWS.

Advanced Topics

The following sections contain detailed instructions, referenced in the procedures above.

Creating an IAM Policy with Minimum Permissions for the CloudWatch Data Collection

To simplify setup and maintenance of either an IAM user account or an IAM role for performing the Cloudwatch audit, Densify recommends attaching the AWS-managed “ReadOnlyAccess” policy to the user or role. This policy provides read-only access to your AWS services and resources and supports the requirements of the Densify Cloudwatch audit. As the Densify Cloudwatch audit continues to evolve and expand, you do not need to update permission policy to include newly added services and features.

Alternatively, if you must restrict the IAM user or role with the minimum permissions to perform the Cloudwatch audit, you can create a custom policy with only the required permissions, as shown below.

Note: This custom policy must be updated periodically as Densify’s standard audit requirements are updated to support additional AWS services and features.

Copy

Example: AWS Minimum User Permission Policy

{
            "Version": "2012-10-17",    
            "Statement": [
                    {
                        "Sid": "Stmt1499171905000",
                        "Effect": "Allow",    
                        "Action": [
                            "autoscaling:DescribeAutoScalingGroups",
                            "autoscaling:DescribeLaunchConfigurations",
                            "cloudformation:DescribeStackResource",
                            "cloudformation:ListStackInstances",
                            "cloudformation:ListStackResources",
                            "cloudwatch:GetMetricStatistics",    
                            "cloudwatch:ListMetrics",
                            "ec2:DescribeHosts",
                            "ec2:DescribeImages",
                            "ec2:DescribeInstances",
                            "ec2:DescribeRegions",
                            "ec2:DescribeReservedInstances",
                            "ec2:DescribeSnapshots",
                            "ec2:DescribeVolumes",
                            "ec2:DescribeSubnets",
                            "ec2:DescribeVpcs",
                            "ec2:DescribeLaunchTemplateVersions",
                            "ecs:DescribeClusters",    
                            "ecs:DescribeContainerInstances",    
                            "ecs:DescribeServices",    
                            "ecs:DescribeTaskDefinition",    
                            "ecs:ListClusters",    
                            "ecs:ListContainerInstances",    
                            "ecs:ListServices",    
                            "ecs:ListTaskDefinitions",    
                            "eks:DescribeCluster",
                            "eks:ListClusters",    
                            "elasticache:DescribeCacheClusters",
                            "elasticache:DescribeReplicationGroups",
                            "elasticache:ListTagsForResource",
                            "iam:ListAccountAliases",
                            "organizations:DescribeOrganization",
                            "organizations:ListAccounts",
                            "rds:DescribeDBInstances",    
                            "rds:DescribeReservedDBInstances",    
                            "rds:DescribeDBClusters",    
                            "rds:ListTagsForResource",
                            "savingsplans:DescribeSavingsPlans"
                            
                        ],
                        
                        "Resource": "*"
                     }
            ]
}

  1. Log into the AWS management console and navigate to Services > IAM.
  2. Select Policies and click Create policy.
  3. Click the JSON tab and enter the policy from the example above.
  4. Review the policy and enter a policy name (e.g. DensifyMinimumReadAccess) and a description (e.g. Minimum permissions required for Densify standard audit).

Creating a Cost and Usage Report for the Billing Data Collection

Use this procedure to create a cost and usage report. You can either reference an existing S3 bucket or create a new one.

To create an AWS Cost and Usage report:

  1. Login to the AWS management console of the payer AWS account. Billing data must be collected from the payer account.
  2. On the Billing & Cost Management Dashboard select Cost & Usage Reports from the sidebar.
  3. Click Create report. You can only have 10 different cost and usage reports. If the Create button is not available (grayed out) then you will need to delete one of the existing reports.
  4. Note: Consult your AWS administrator before deleting anything!

  5. Enter a valid name and select the following content and then click Next.
    • Include resource IDs—This setting is optional.
    • Data refresh settings—Leave the default setting to automatically update the report with any changes to past reports.
  6. Define the Delivery options for this report:
    • Configure S3 Bucket—See Step 6 below.
    • Report path prefix—This setting is optional. The default prefix is the name that you specify for the report and the date range for the report, in the following format: /report-name/date-range/.
    • Time granularity—Select "Daily".
    • Report versioning—Select "Overwrite existing report", to minimize the size of the bucket. Both options are supported.
    • Enable report data integration for—This is optional. Do not select Amazon QuickSight or Athena, as these formats are not supported. RedShift is a data warehouse. Densify still only picks up the .CSV from the bucket.
    • Compression type—This is optional. Do not select Parquet.
  7. Click Configure to define an S3 bucket for your report:
    • If you have a valid S3 bucket, then select an existing bucket from the list. Review and verify the policy. Click Next to return to the Delivery Options page.
    • If you need to create the bucket then enter a bucket name. Review the note at the bottom of the dialog box.
    1. Select another region, if necessary and then click Next.
    2. Review and verify the policy. Click Next to return to the Delivery Options page.
  8. On the Delivery options page, your S3 bucket will be listed and verified. Click Next.
  9. Review you settings and then click Review and Complete.

Enabling Collection of Memory Usage Metrics

Memory metrics are not collected by default, and they are not required to complete the Densify analyses. You can manually enable collection of memory and disk metrics.

Note: The CloudWatch Agent must be installed and configured on each instance for which you want to obtain memory and/or disk metrics. Refer to the AWS user documentation for details.

Once the CloudWatch Agent is installed and configured, Densify uses the default, CWAgent as the namespace for metrics collected by the CloudWatch agent.

Use the following information to configure the CloudWatch Agent (CWagent) via config.json to collect the metrics that Densify can use for analyses. Instructions are provided for both Linux and Windows instances.

Linux Configuration

For Linux instances, the default CWagent config.json file can be generated based on the following options:

  • Basic
  • Standard
  • Advanced

For all of the above options, the memory metric, “mem_used_percent” is collected by default, as specified in the config.json file. However, the metrics “mem_active” and “mem_used” should be added to the CWagent's settings, for Densify's recommendations.

Additionally, the disk “total” metric should be included if you want to analyze disk usage.

The following example shows the updated version of the Basic config.json file with the additional metrics highlighted:

Example: Basic CWAgent Configuration File

{

"agent": {

"metrics_collection_interval": 60,

"run_as_user": "root"

},

"metrics": {

"append_dimensions": {

"AutoScalingGroupName": "${aws:AutoScalingGroupName}",

"ImageId": "${aws:ImageId}",

"InstanceId": "${aws:InstanceId}",

"InstanceType": "${aws:InstanceType}"

},

"metrics_collected": {

"disk": {

"measurement": [

"total",

"used_percent"

],

"metrics_collection_interval": 60,

"resources": [

"*"

]

},

"mem": {

"measurement": [

"mem_used",

"mem_active",

"mem_used_percent"

],

"metrics_collection_interval": 60

}

}

}

}

Windows Configuration

For Windows instances, the default CWagent config.json file are the same as listed above, Basic, Standard and Advanced.

For all of the above options, the memory metric, "% Committed Bytes in Use" is collected by default, as specified in the config.json file. However, the metric “Available MBytes” should be added to the CWagent's settings, for Densify analysis.

The following example shows the updated version of the Basic config.json file with the additional metric highlighted:

Example: Basic CWAgent Configuration File

{

"agent": {

"metrics_collection_interval": 60,

"run_as_user": "root"

},

"metrics": {

"append_dimensions": {

"AutoScalingGroupName": "${aws:AutoScalingGroupName}",

"ImageId": "${aws:ImageId}",

"InstanceId": "${aws:InstanceId}",

"InstanceType": "${aws:InstanceType}"

},

"metrics_collected": {

"LogicalDisk": {

"measurement": [

"% Free Space"

],

"metrics_collection_interval": 60,

"resources": [

"*"

]

},

"Memory": {

"measurement": [

"Available MBytes",

"% Committed Bytes In Use"

],

"metrics_collection_interval": 60

}

}

}

}

If you are use a third-party application to collect memory metrics, the collected data can be loaded using the Receive Metrics API endpoint. See Importing Metrics for Existing Services.

Refer to the AWS user documentation for details on using the CloudWatch Agent to collect memory metrics.