Continuously Optimize Your AWS Resources with CloudFormation

If you have discovered that your application demand changes over time, you’re probably wondering how you can continuously adjust your cloud capacity in accordance to application demand. If you use CloudFormation, then you’re in luck! This article walks through how you can update your template code to automatically implement infrastructure adjustments, periodically.

Considering Infrastructure as Code Best Practices

Before proceeding to enable this automatic optimization, we need to consider a key best practice when using infrastructure as code technologies like CloudFormation.

Configuration parameters that can potentially change in the future (also called transient parameters) should be surfaced out of the templates and stored in a centralized system of record. This system of record, typically implemented as a key-value store, holds configuration parameters that can be natively referenced in infrastructure as code.

Two key benefits to adopting this approach:

1. Future changes to those configuration parameters can be implemented without modification to the source templates
2. All changes can be tracked and audited

We are going to use AWS Parameter Store as our centralized database for holding specific infrastructure allocation parameters for CloudFormation. Specifically, EC2:InstanceType, RDS:DBInstanceClass, ASG:MinSize, and ASG:MaxSize.

How Should CloudFormation Reference Parameter Store?

CloudFormation templates can natively reference Parameter Store through parameter resources available as a feature in CloudFormation. Due to limitations in the number of definable parameter resources (200), it’s not practical to implement the reference mechanism in this way.

Instead, we can introduce a lookup function specified as an infrastructure resource into the template, which can act as a provider to extract optimization insights from Parameter Store. Now infrastructure resources (i.e. EC2, RDS, ASG, etc.) can then directly reference this lookup function for recommendations.

We can leverage CloudFormation CLI to build this resource as a resource provider which has a resource type specification and handlers that control API interactions with the underlying AWS or third-party services. These interactions may include create, read, update, delete and list (CRUDL) operations for resources. You would use resource providers to model and provision resources using CloudFormation.

The open source AWS CloudFormation CLI project will walk you through how to build a custom resource provider, which can be used as a platform to query any third party system (i.e. Parameter Store) for insights. These insights should be delivered back to CloudFormation using the supplied responseURL. We will refer to our resource provider as a lookup plugin from this point.

How to Generate Optimization Insights

The key to solving this problem lies beyond enabling CloudFormation to reference AWS Parameter Store. It is to have the ability to generate actionable insights based on a scientific analysis of your infrastructure performance. Unfortunately, a discussion on how to accurately generate such insights is beyond the scope of this article.

If you already have a mechanism to generate insights for the parameters I mentioned above, then you only need to look into ensuring those insights are delivered to AWS Parameter Store in a timely fashion.

If your looking for an engine to help you generate these insights, one option is Densify. We specialize in the analysis of cloud and container workloads to determine optimal supply allocations for performance and cost. We ultimately deliver these configuration settings to your preferred centralized system of record—for example, AWS Parameter Store.

Example Implementation

If you want to follow along, please ensure you have a Densify instance running and connected to your AWS environment. If you haven’t already, you can visit the Densify signup page to get access to a free, no-obligation trial.

Let’s consider an example where we have a CloudFormation template managing a set of EC2, RDS, and ASG resources. There are two steps to enabling Continuous Optimization:

Enable the Densify Lookup Plugin

You can enable this plugin by running this CloudFormation template in any account you wish to automate. Keep in mind that this account should be connected to Densify (as a prerequisite). CloudFormation is a regional technology and as such the plugin must be enabled in each region you wish to automate.

Start by running this template in any region. You will be presented with the following Parameters input screen:

1. Enter in the Densify login details. This will enable the lookup plugin to access the optimization insights stored in Densify.
2. Select how the lookup plugin will acquire these recommendations. Either directly from Densify or via a key-value store (i.e. Parameter Store).
3. Specify which regions to activate the provider in comma-delimited format

Modify Existing CloudFormation Templates (3 Steps)

This section details how to modify your existing CloudFormation templates manually. If you would like an automated approach, please skip to the next subsection.

Step 1: Add the Densify Lookup Plugin

In order for your existing CloudFormation templates to make use of the new lookup plugin, there needs to exist a mechanism for CloudFormation to reference that lookup plugin. This can be done by introducing the densify infrastructure resource into your template.

  DensifyResourceProvider:
    Type: Custom::DensifyResourceProvider
    Properties:
      ServiceToken: arn:aws:lambda:<Region>:<Account>:function:DensifyResourceProvider
      LastRefreshTime: !Ref 'DensifyLastUpdatedTimestamp'
      <logical_name_of_resource>.<parameter_to_optimize>: <default_value>

The areas that need customization are the highlighted text. The ServiceToken contains the ARN to the deployed lookup plugin. The region and account need to be updated accordingly.

For every resource that you would like to optimize you need to enter a row to specify the initial default sizes.

ASGInstance.MinSize: 1
ASGInstance.MaxSize: 3
ASGInstance.InstanceType: t2.micro

Step 2: Reference the Provider

For each of the individual resources you wish to optimize, we need to modify the line holding the parameter in the resource spec. For example for EC2, we are looking to update the Properties:InstanceType parameter. These parameters will be updated to a GetAtt with the following nomenclature, DensifyResourceProvider.<logical_name_of_resource>.<parameter_to_optimize>

Step 3: Add the Last Updated Timestamp Parameter

The LastUpdatedTimestamp parameter holds the last date/time a recommendation was generated by Densify for any resource it is currently managing. The parameter resource we will create in the CloudFormation template will directly reference Parameter Store for the value. The Parameter Store entry is initialized in Step 1, when we activated the lookup plugin.

  DensifyLastUpdatedTimestamp:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /densify/config/lastUpdatedTimestamp

Modify Existing CloudFormation Templates (Automatic): Alternative to Step 2

Densify also provides tooling to automatically modify existing CloudFormation templates. This can simplify the process greatly and eliminates any human error due to manual modification of templates.

Start by running this template in any region. You will be presented with the following Parameters input screen:

1. S3 Bucket Name: The name of the S3 bucket that this CloudFormation template will create to store all modified templates.
2. Stacks to Modify: A specification of which particular stacks in the environment we want to modify in the following format, <REGION>:<StackName>. Region where the stack exists and the name of the stack.
3. Execute Modified Stack: For stacks that have been modified, you can specify whether you want densify to auto update those stacks in place. This is achieved by issuing a CloudFormation stack-update for those stacks using the updated templates. The process will only execute against a stack if that stack does not contain any resources that do not support drift detection and the stack is not drifted.
4. Go to each of your existing templates and verify that they have been updated for automation. If there are some templates that weren’t modified, it’s usually because the associated stack is not currently in sync. For these situations, manual modification of the stack is required—please refer to the previous subsection.

That should be it!

Before running a CloudFormation stack-update, be sure to approve the relevant insights through the Densify console.

To learn more about enabling continuous optimization for your cloud resources via CloudFormation CLI, please visit our CloudFormation Optimization as Code GitHub repository or contact us at [email protected].