View this page as a PDF

Densify also supports connecting to AWS using;

• An IAM Role and CloudFormation template—See AWS Data Collection Using a CloudFormation Template.

When you create a role for cross-account access, you establish trust from the customer's account that owns the role (and the resources (trusting account) to the Densify account containing the user that will collect data (trusted account). You specify the trusted account number as the Principal in the role's trust policy when you create the role. This allows the Densify user in the trusted account to assume the role and collect utilization data.

Creating the IAM Role and Attaching a Permission Policy to Collect CloudWatch Data

This role allows you to collect resource utilization data for the selected account. You need to attach a policy that allows the role to collect the required CloudWatch resource utilization metrics.

1. Log into the AWS Management Console and navigate to Identity and Access Management (IAM). In the navigation tree on the left, click Roles.
2. Click Create Role in the Roles dashboard.
3. Select AWS account as the of trusted entity type .
4. Enter an Account ID. This is the Densify account that will assume the role. Enter the following Densify account ID: 036437403198.
5. Select Require external ID and enter your external ID. This value is similar to a password and should be unique and difficult to guess. Densify recommends using a password generator to create a random, alphanumeric string (e.g. ae73mcf4ldjpet96) for the external ID.

You will need this external ID later, when creating the cloud connection in Densify.

6. Click Next.
Figure: Creating a Role (hover over image to expand)



7. Attach the appropriate permission policy to the role. Select AWS's predefined ReadOnlyAccess policy. Use the filter to find the ReadOnlyAccess policy. Even with the filter set you will need to go to the last page to find the "ReadOnlyAccess" policy.

Note:  The ReadOnlyAccess policy is provided here as an easy option for non-sensitive accounts. In most cases, you need to create a custom permission policy to grant Densify, the permissions to collect only the required CloudWatch data. Refer to Creating an IAM Policy with Minimum Permissions for the CloudWatch Data Collection for details.

Figure: Selecting the ReadOnlyAccess Policy (hover over image to expand)



8. After selecting the permission policy for the role, click Next.
9. In the Review page, specify the Role name and Role description. The role name can be any string used to identify and describe the role within the AWS account (e.g. DensifyCrossAccountRole).
10. Click Create role. The new role is created.
11. From the Roles page, click on the role name that you have just created, to view the role summary.
Figure: Role Summary (hover over image to expand)



12. Copy and save the Role ARN as you will need to paste this string into the Densify Cloud Connection wizard to create the connection.
13. You can now create an AWS connection through the Densify Public Cloud Connection wizard. See Using the Public Cloud Connections Wizard.

You can also use the Densify API.See Analysis: AWS.

Optional Configuration

The following sections contain detailed instructions for optional configuration. Some of this configuration is referenced in the procedures above.

• Creating an IAM Policy with Minimum Permissions for the CloudWatch Data Collection
• Enabling Collection of AWS Memory Usage Metrics

• Enabling the Collection of GPU Metrics

Creating an IAM Policy with Minimum Permissions for the CloudWatch Data Collection

To simplify setup and maintenance of either an IAM user account or an IAM role for performing the CloudWatch audit, Densify recommends attaching the AWS-managed “ReadOnlyAccess” policy to the user or role. This policy provides read-only access to your AWS services and resources and supports the requirements of the Densify CloudWatch audit. As the Densify CloudWatch audit continues to evolve and expand, you do not need to update permission policy to include newly added services and features.

Alternatively, if you must restrict the IAM user or role with the minimum permissions to perform the CloudWatch audit, you can create a custom policy with only the required permissions, as shown below.

Note:  This custom policy must be updated periodically as Densify’s standard audit requirements are updated to support additional AWS services and features.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
{
            "Version": "2012-10-17",    
            "Statement": [
                    {
                        "Sid": "Stmt1499171905000",
                        "Effect": "Allow",    
                        "Action": [
                            "autoscaling:DescribeAutoScalingGroups",
                            "autoscaling:DescribeLaunchConfigurations",
                            "autoscaling:DescribeScalingActivities",
                            "ce:GetCostAndUsage",
                            "cloudformation:DescribeStackResource",
                            "cloudformation:DescribeStackResources",
                            "cloudformation:ListStackInstances",
                            "cloudformation:ListStackResources",
                            "cloudwatch:GetMetricData",
                            "cloudwatch:GetMetricStatistics",    
                            "cloudwatch:ListMetrics",
                            "ec2:DescribeHosts",
                            "ec2:DescribeImages",
                            "ec2:DescribeInstances",
                            "ec2:DescribeLaunchTemplateVersions",
                            "ec2:DescribeRegions",
                            "ec2:DescribeSnapshots",
                            "ec2:DescribeVolumes",
                            "ec2:DescribeSubnets",
                            "ec2:DescribeSecurityGroupRules",
                            "ec2:DescribeSecurityGroups",                    
                            "ec2:DescribeVpcs",
                            "ecs:DescribeCapacityProviders",
                            "ecs:DescribeClusters",    
                            "ecs:DescribeContainerInstances",    
                            "ecs:DescribeServices",    
                            "ecs:DescribeTaskDefinition",    
                            "ecs:ListClusters",    
                            "ecs:ListContainerInstances",    
                            "ecs:ListServices",
                            "ecs:ListTagsForResource",    
                            "ecs:ListTaskDefinitions",    
                            "eks:DescribeCluster",
                            "eks:ListClusters",    
                            "iam:ListAccountAliases",
                            "organizations:DescribeOrganization",
                            "organizations:ListAccounts",
                            "rds:DescribeDBInstances",    
                            "rds:DescribeReservedDBInstances",    
                            "rds:DescribeDBClusters",    
                            "rds:ListTagsForResource"
                            
                        ],
                        
                        "Resource": "*"
                     }
            ]
}

Note:  The permissions related to CloudFormation are used for linking ASGs with ECS clusters. If these permissions are not included and the data is not available, linking the ASGs to ECS clusters may be done based on existing container instances. If the permissions are missing, Densify may not link some ASGs to their ECS clusters,

1. Log into the AWS management console and navigate to Services > IAM.
2. Select Policies and click Create policy.
3. Click the JSON tab and enter the policy from the example above.
4. Review the policy and enter a policy name (e.g. DensifyMinimumReadAccess) and a description (e.g. Minimum permissions required for Densify standard audit).

Enabling Collection of AWS Memory Usage Metrics

Memory metrics are not collected by default and they are not required to complete the Densify analyses.

You can manually enable collection of specific memory and disk metrics to improve the quality of your recommendations.

Note:  Additional metrics collected by the CloudWatch agent incur charges. See Amazon CloudWatch Pricing for details.

You need to configure the unified Cloudwatch agent for either Linux or Windows instances.

Note:  The CloudWatch Agent must be installed and configured on each instance for which you want to obtain memory and/or disk metrics. Refer to the AWS user documentation for details. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html

Once the CloudWatch Agent is installed and configured, Densify uses the default, CWAgent as the namespace for metrics collected by the CloudWatch agent.

Additionally, when working with ASGs, the ASG's EC2 members provide memory utilization data using the basic memory settings, but you need to specify "aggregation_dimensions" to collect memory, aggregated at the ASG level.

Use the following information to customize the default CloudWatch Agent (CWagent) to collect the metrics that Densify can use for analyses.

If you are using a third-party application to collect memory metrics, the collected data can be loaded using the Receive Metrics API endpoint. See Importing Metrics for Existing Services in the API documentation.

Refer to the AWS user documentation for details on using the CloudWatch Agent to collect memory metrics.

Enabling the Collection of GPU Metrics

To support collection of the NVIDIA® GPU data, the required metrics must be enabled through the CloudWatch agent.

Install the NVIDIA driver and CloudWatch agent on your instances. See Collecting NVIDIA GPU Metrics.

• Linux Servers—You need to add the section, nvidia_gpu inside the metrics_collected section of the CloudWatch agent configuration file. For more information, see: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-NVIDIA-GPU.html
• Window Servers—Configure and use Amazon Custom CloudWatch Metrics. See https://repost.aws/questions/QUcyzZfm3UR96Qj8JiadlCAA/how-to-monitor-and-collect-gpu-metrics-for-windows-ec2-instances-using-amazon-cloudwatch

Alternatively, you can use a third party tool such as Telegraf for metrics collection from Window Servers.

Densify requires the following GPU metrics:

Table: Required GPU Metrics

NVDIA Metric	CloudWatch Metric Name	Unit	Description
utilization_gpu	nvidia_smi_utilization_gpu	%	The percentage of time over the past sample period during which one or more kernals on the GPU was running.
utilization_memory	nvidia_smi_utilization_memory	%	The percentage of time over the past sample period during which global (device) memory was being read or written.
memory_used	nvidia_smi_memory_used	MB	Memory used.

The following new workloads, based on the above metrics are available in the metrics viewer:

• GPU_Utilization—GPU utilization in percent
• GPU_Mem_Utilization_As_Pct—GPU memory utilization as a percent of the total memory
• GPU_Mem_Used—GPU memory used, in MB