Today, Amazon Web Services announced CloudFormation CLI and CloudFormation Registry. Densify has been working with AWS to provide this article, in addition to creating a resource provider with definitions, logic, and additional documentation that is available on GitHub for today’s launch.
Most of you are familiar with continuous integration/continuous delivery (CI/CD). At a high level, CI/CD provides the ability to automatically integrate, test, and deliver new iterations of a product and code with little to no human intervention. Continuous optimization (CO) is the additional concept of ensuring that with every iteration of the product or code, the optimal infrastructure is systemically selected.
This article discusses how you can enable continuous optimization within your DevOps pipeline to optimize infrastructure allocation for any application.
Let’s start by looking at some of the basic components of a DevOps pipeline:
The highlighted box Infrastructure Deployment is where we will introduce continuous optimization, which relies on two critical technologies: 1) infrastructure as code, and 2) a centralized system of record.
As a best practice when using infrastructure as code, configuration parameters that can possibly change in the future should be surfaced out of the templates and stored in a centralized database. There are numerous benefits to this approach, one of which is all future changes to those elements being managed through infrastructure as code can be implemented without modification to the underlying templates.
For the purposes of this article, we will use CloudFormation (an AWS infrastructure as code technology) and Parameter Store (an AWS configuration and secrets store) to implement continuous optimization. This diagram describes the data flow that we will implement.
As an example, let’s consider an EC2 instance and specifically the instance type parameter which we will optimize for. This parameter controls what instance family and size CloudFormation should acquire, which directly impacts application performance and cost.
In order to practically enable this solution, we can leverage CloudFormation CLI to build a resource provider capable of referencing insights from Parameter Store. A resource provider has a resource type specification and handlers that control API interactions with the underlying AWS or third-party services. These interactions may include create, read, update, delete, and list (CRUDL) operations for resources. You would use resource providers to model and provision resources using CloudFormation. For example, AWS::EC2::Instance is a resource type by Amazon EC2 provider.
At the crux of solving the continuous optimization challenge is to ensure you have automatable insights populated within Parameter Store. Densify specializes in the analysis of cloud and container workloads to determine optimal supply allocations for performance and cost and eventually delivers these configuration settings to your centralized system of record (an example being AWS Parameter Store). Densify also provides a resource provider available through the CloudFormation Registry that enables reference to insights directly within your CloudFormation templates.
If you would like to test this implementation you will need to have an account with Densify and ensure your environment is connected for analysis. If you don’t already have Densify, simply request a free, no-commitment, 14-day trial.
The final solution will include a Densify instance to analyze existing workloads and publish insights directly to Parameter Store.
With a cloud environment connected to Densify, we can begin the process of augmenting your CloudFormation template with a reference to the Densify provider.
Let’s pick a simple EC2 resource currently being managed through CloudFormation that you wish to optimize. We augment the existing template with the following code.
"<logical_name_of_resource>": {
"Type": "Densify::Optimization::Recommendation",
"Properties":{
"ProvisioningID": "<unique_name_for_resource>",
"InstanceType": "m4.large",
"ForceUpdate": {"Ref":"RefreshTime"}
}
},
The resource provider within your template will be referenced by the resource to extract the instance type insight for the EC2 resource. Best practice naming conventions for this resource should be an off-shoot of the logical name of the resource we’re optimizing. So, if the logical name of your EC2 instance is EC2eg000, then a good name choice would be EC2eg000Insight. The ProvisioningID should be set to a value that uniquely identifies the resource. The InstanceType is the default instance type that you would like for the resource being optimized.
"EC2eg000": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageID": "ami-00bb1234ccc789d00",
"InstanceType": {
"Fn::GetAtt": [
"<logical_name_of_resource>",
"InstanceType"
]
},
"Tags": [
{
"Key":"Provisioning ID",
"Value":"<unique_name_for_resource>"
}
]
}
}
The InstanceType attribute should be set to reference the Densify resource provider created in the first step. There should be a tag enabled on the resource containing the Provisioning ID set in the previous step.
We need to include a parameter reference, which will be used by the CloudFormation template to check whether new optimization insights exist. You should validate whether this parameter exists within Parameter Store by navigating to System Manager > Parameter Store and looking for a parameter named DensifyRefreshTime.
"RefreshTime": {
"Type": "AWS::SSM::Parameter::Value<String>",
"Default": "DensifyRefreshTime"
}
That should be it!
Before running a CloudFormation stack-update, be sure to approve the relevant insights through the Densify console.
To learn more about enabling continuous optimization for your cloud resources via CloudFormation CLI, please visit our CloudFormation Optimization as Code GitHub repository or contact us at answers@densify.com.