How to Adopt Continuous Optimization in AWS Using CloudFormation

Today, Amazon Web Services announced CloudFormation CLI and CloudFormation Registry. Densify has been working with AWS to provide this article, in addition to creating a resource provider with definitions, logic, and additional documentation that is available on GitHub for today’s launch.

Most of you are familiar with continuous integration/continuous delivery (CI/CD). At a high level, CI/CD provides the ability to automatically integrate, test, and deliver new iterations of a product and code with little to no human intervention. Continuous optimization (CO) is the additional concept of ensuring that with every iteration of the product or code, the optimal infrastructure is systemically selected.

This article discusses how you can enable continuous optimization within your DevOps pipeline to optimize infrastructure allocation for any application.

Continuous Optimization in Your DevOps Pipeline

Let’s start by looking at some of the basic components of a DevOps pipeline:

Code Generation
a set of tools designed to automate the code merge/build process. This can include repositories to hold templates and code and technologies to build code, for example, AWS Code Commit, GitHub, AWS Code Build, and others
Infrastructure Deployment
orchestration technologies like infrastructure as code that automate the allocation and configuration of infrastructure, for example, AWS CloudFormation, Terraform, Ansible, Chef, and others
Application Deployment
the final stage where the application or new code is deployed and configured on the procured infrastructure, for example, AWS Code Deploy, Lambda, and others
Three stages of a DevOps pipeline: code generation, infrastructure deployment, and application deployment

The highlighted box Infrastructure Deployment is where we will introduce continuous optimization, which relies on two critical technologies: 1) infrastructure as code, and 2) a centralized system of record.

  • Infrastructure as code is a collection of technologies that enable users to manage (create, update or destroy) cloud resources through templated descriptions of that infrastructure. Typically provided in either a JSON or YAML format.
  • A centralized system of record (persistent database) in the context of CO, is a centralized repository of configuration parameters that can be natively referenced and leveraged by the infrastructure as code technology.

As a best practice when using infrastructure as code, configuration parameters that can possibly change in the future should be surfaced out of the templates and stored in a centralized database. There are numerous benefits to this approach, one of which is all future changes to those elements being managed through infrastructure as code can be implemented without modification to the underlying templates.

For the purposes of this article, we will use CloudFormation (an AWS infrastructure as code technology) and Parameter Store (an AWS configuration and secrets store) to implement continuous optimization. This diagram describes the data flow that we will implement.

Automated data flow through Amazon services leveraging CloudFormation CLI

As an example, let’s consider an EC2 instance and specifically the instance type parameter which we will optimize for. This parameter controls what instance family and size CloudFormation should acquire, which directly impacts application performance and cost.

In order to practically enable this solution, we can leverage CloudFormation CLI to build a resource provider capable of referencing insights from Parameter Store. A resource provider has a resource type specification and handlers that control API interactions with the underlying AWS or third-party services. These interactions may include create, read, update, delete, and list (CRUDL) operations for resources. You would use resource providers to model and provision resources using CloudFormation. For example, AWS::EC2::Instance is a resource type by Amazon EC2 provider.

Making it Dynamic

At the crux of solving the continuous optimization challenge is to ensure you have automatable insights populated within Parameter Store. Densify specializes in the analysis of cloud and container workloads to determine optimal supply allocations for performance and cost and eventually delivers these configuration settings to your centralized system of record (an example being AWS Parameter Store). Densify also provides a resource provider available through the CloudFormation Registry that enables reference to insights directly within your CloudFormation templates.

If you would like to test this implementation you will need to have an account with Densify and ensure your environment is connected for analysis. If you don’t already have Densify, simply request a free, no-commitment, 14-day trial.

The final solution will include a Densify instance to analyze existing workloads and publish insights directly to Parameter Store.

Automated data flow through AWS using CloudFormation CLI to call continuous optimization

With a cloud environment connected to Densify, we can begin the process of augmenting your CloudFormation template with a reference to the Densify provider.

Example: Continuously Optimizing an EC2 Instance

Let’s pick a simple EC2 resource currently being managed through CloudFormation that you wish to optimize. We augment the existing template with the following code.

Adding the Densify Resource Provider

"<logical_name_of_resource>": {
  "Type": "Densify::Optimization::Recommendation",
  "Properties":{
    "ProvisioningID": "<unique_name_for_resource>",
    "InstanceType": "m4.large",
    "ForceUpdate": {"Ref":"RefreshTime"}
  }
},
Densify Resource Provider code snippet

The resource provider within your template will be referenced by the resource to extract the instance type insight for the EC2 resource. Best practice naming conventions for this resource should be an off-shoot of the logical name of the resource we’re optimizing. So, if the logical name of your EC2 instance is EC2eg000, then a good name choice would be EC2eg000Insight. The ProvisioningID should be set to a value that uniquely identifies the resource. The InstanceType is the default instance type that you would like for the resource being optimized.

Updating the EC2 Resource

"EC2eg000": {
  "Type": "AWS::EC2::Instance",
  "Properties": {
    "ImageID": "ami-00bb1234ccc789d00",
    "InstanceType": {
      "Fn::GetAtt": [
        "<logical_name_of_resource>",
        "InstanceType"
      ]
    },
    "Tags": [
      {
        "Key":"Provisioning ID",
        "Value":"<unique_name_for_resource>"
      }
    ]
  }
}
Updating the EC2 resource

The InstanceType attribute should be set to reference the Densify resource provider created in the first step. There should be a tag enabled on the resource containing the Provisioning ID set in the previous step.

Adding a New Parameter

We need to include a parameter reference, which will be used by the CloudFormation template to check whether new optimization insights exist. You should validate whether this parameter exists within Parameter Store by navigating to System Manager > Parameter Store and looking for a parameter named DensifyRefreshTime.

"RefreshTime": {
  "Type": "AWS::SSM::Parameter::Value<String>",
  "Default": "DensifyRefreshTime"
}
DensifyRefreshTime parameter

That should be it!

Before running a CloudFormation stack-update, be sure to approve the relevant insights through the Densify console.

To learn more about enabling continuous optimization for your cloud resources via CloudFormation CLI, please visit our CloudFormation Optimization as Code GitHub repository or contact us at answers@densify.com.