What is EKS Anywhere?

EKS Anywhere is an AWS offering that lets users run Elastic Kubernetes Service (EKS) clusters in on-premises environments. Users can deploy a Kubernetes setup that is compatible with the managed cloud EKS offering without forcing users to migrate data or compute capacity to the cloud. AWS provides this functionality by packaging a collection of tools for deploying a Kubernetes distribution with an opinionated configuration. This feature provides an on-ramp for users interested in gradually shifting toward an EKS-based Kubernetes platform, integrating with other AWS services and benefiting from an AWS support plan.

Summary of key concepts

How does EKS Anywhere work?

EKS Anywhere includes several components that are designed to work together to orchestrate an end-to-end cluster deployment process. These components are mainly transparent to the user, who will primarily interface with EKS Anywhere using the eksctl tool:

• eksctl: This CLI tool is co-developed by Weaveworks and AWS and is used to provision, upgrade, and operate EKS clusters (both on-premises and managed cloud). The eksctl tool is the primary method by which users will deploy and upgrade EKS Anywhere clusters.
• EKS Distro: This is the Kubernetes distribution built by AWS that powers the managed EKS service. The distribution is open source and publicly provides all container images required to run an EKS-equivalent cluster, whether on self-managed cloud compute hosts or on-premises. The container images published for EKS Distro include all components needed to run a Kubernetes cluster, such as the Kube API Server, Kube Controller Manager, Kube Proxy, Kube Scheduler, etcd, CoreDNS, and so on. EKS Anywhere deploys the container images published by EKS Distro, which allows users to run an on-premises cluster with the same configuration as a managed cloud EKS cluster. AWS uses these same container images to deploy managed cloud EKS clusters.

• Tinkerbell: This open-source project enables functionality related to configuring operating systems in preparation for running Kubernetes workloads. Tinkerbell will set up a DHCP service, metadata service, and IP addressing functionality for the cluster.
• ClusterAPI: This open-source project is widely used for managing large fleets of Kubernetes clusters for both on-premises and managed cloud setups. This tool performs Kubernetes cluster bootstrapping actions such as configuring master nodes, etcd nodes, storage classes, and cluster networking.
• Flux: This open-source project implements a GitOps workflow for Kubernetes clusters. EKS Anywhere will (optionally) set up this tool and commit the cluster configuration to a Git repository. Storing cluster configurations in a Git repository allows for version control of all configuration resources, tracking historical changes and providing a centralized approach to deploying new changes to the cluster. New changes are simply pushed to the Git repository, and the EKS Anywhere cluster will automatically download and deploy any configuration changes observed in the repository.
• EKS Connector: This tool enables the exporting of on-premises cluster data to the AWS EKS web console. Users can implement this tool to gain visibility of their EKS Anywhere clusters directly from the AWS console. This functionality allows users to browse the configuration and status of their EKS Anywhere clusters in the same location where they can see their managed cloud EKS clusters.

While these are the key tools for managing an EKS Anywhere cluster lifecycle, there are other secondary tools curated by AWS that are designed to extend EKS Anywhere's functionality. These tools include MetalLB (a load balancer provisioner), Cluster Autoscaler (a node scaling tool), CertManager (a certificate manager for Kubernetes clusters), and many others. The tools are curated and tested by AWS, and they are deployable directly through the eksctl tool.

What are EKS Anywhere’s benefits?

There are multiple use cases where users benefit from deploying EKS Anywhere clusters for on-premises environments:

• Users can implement EKS Anywhere to validate that their workloads are compatible with cloud-managed EKS. Since EKS Anywhere uses the same control plane settings as cloud-managed EKS, implementing this project is a useful way to prepare and validate readiness for a migration from on-premises to cloud-managed EKS. 
• Users with hybrid cloud setups can maintain some resources with a cloud provider and keep some resources on-premises. Unifying the resource configuration between the various infrastructures will provide better operational stability, unify the skills required for management, and enable use cases like disaster recovery and failover.

• EKS Anywhere provides a vendor-supported approach to managing the Kubernetes cluster lifecycle. Users may find value in running tools with vendor support assurances, particularly for mission-critical workloads.

The benefits of EKS Anywhere for on-premises Kubernetes users are significant and can provide a helpful bridge between on-premises and cloud environments. Users can test out EKS Anywhere for free to validate that it suits their requirements.

Note: EKS Anywhere does not allow mixing worker nodes or control planes between on-premises setups and the AWS cloud environment. All cluster resources must be deployed within the same environment.

How do you scale EKS Anywhere nodes?

Users have the ability to horizontally or vertically scale an EKS Anywhere cluster’s worker and control plane nodes. This can be done via either the eksctl command-line tool or the Cluster Autoscaler project. In either case, the cluster will need information about the available hardware to create the new nodes. 

The cluster requires information such as the IP address, host name, gateway, and root filesystem device to correctly configure the hardware with an operating system and required software dependencies. This hardware information will need to be generated by the user and stored in a text file (hardware.csv). This file will be supplied to the EKS Anywhere cluster during either cluster creation or upgrade. The cluster will then have an awareness of what hardware is available in the on-premises environment for scaling operations.

Here’s an example of a hardware.csv file:

hostname,bmc_ip,bmc_username,bmc_password,mac,ip_address,netmask,gateway,nameservers,labels,disk
eksa-cp01,10.10.44.1,root,PrZ8W93i,CC:48:3A:00:00:01,10.10.50.2,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=cp,/dev/sda
eksa-cp02,10.10.44.2,root,Me9xQf93,CC:48:3A:00:00:02,10.10.50.3,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=cp,/dev/sda
eksa-cp03,10.10.44.3,root,Z8x2M6hl,CC:48:3A:00:00:03,10.10.50.4,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=cp,/dev/sda
eksa-wk01,10.10.44.4,root,B398xRTp,CC:48:3A:00:00:04,10.10.50.5,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=worker,/dev/sda
eksa-wk02,10.10.44.5,root,w7EenR94,CC:48:3A:00:00:05,10.10.50.6,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=worker,/dev/sda
            

Hardware information can be supplied during cluster creation:

eksctl anywhere create cluster --hardware-csv hardware.csv

It can also be supplied during cluster upgrades:

eksctl anywhere upgrade cluster --hardware-csv hardware.csv

The hardware information supplied to the cluster can be viewed with kubectl:

kubectl get hardware -n eksa-system
 NAME
 eksa-controlplane
 eksa-worker1
 eksa-worker2

The above confirms that some hardware resources have been registered to the cluster. The cluster can then be scaled manually by updating the cluster’s configuration file:

# cluster.yaml
kind: Cluster
metadata:
  name: test-cluster
spec:
  workerNodeGroupsConfiguration:
    count: 1           # Update from 1 -> 2 for horizontal scaling
                

We can then deploy the updated node count:

eksctl anywhere upgrade cluster -f cluster.yaml
            

For vertical scaling, users will register more powerful hardware in the hardware.csv file and then run the upgrade command above. This will trigger EKS Anywhere to replace smaller nodes with larger nodes with more hardware capacity. 

EKS Anywhere also supports dynamic scaling via the Cluster Autoscaler tool for bare metal hardware.

Support model and pricing

EKS Anywhere is an open-source project and can be deployed for free. However, users requiring ongoing support from AWS will need to acquire an AWS Enterprise Support subscription and an additional EKS Anywhere Support subscription.

The EKS Anywhere Support subscription enables users to request guidance on deployment and troubleshooting from AWS engineers. The subscription includes support related to all tools included with EKS Anywhere and integrations with curated packages.

The support subscription will be helpful to users running production-critical systems requiring immediate and direct access to vendor support. Users running EKS Anywhere for lower-priority workloads will be fine without the paid subscription. Since the EKS Anywhere components are all open-source, publicly available, and documented, many users can troubleshoot common problems without vendor support.

The cost of an EKS Anywhere Support subscription depends on how many clusters require support and for what duration. At the time this article was written, the baseline cost for one year of support for one EKS Anywhere cluster is $24,000; three-year contracts for one cluster will cost $54,000 ($18,000 per year). This pricing is in addition to the cost of AWS Enterprise Support, which is a requirement for enabling EKS Anywhere Support; AWS Enterprise Support starts at $15,000 per month.

Due to the significant costs associated with accessing EKS Anywhere Support, users will benefit from carefully evaluating whether their setups are mission-critical enough to justify the expenditures.

What are EKS Anywhere’s security considerations?

Maintaining the security of an EKS Anywhere cluster is a shared responsibility between the user and AWS. AWS provides assurances regarding patching vulnerabilities in the bundled software, but the user must regularly upgrade their clusters, secure RBAC configurations, restrict API endpoint access, and maintain the bare metal and virtual machine security posture. The user is responsible for all aspects of securing Kubernetes workloads in the EKS Anywhere cluster, the software dependencies, and the underlying compute hosts.

Users must carefully plan how to approach their security posture when deploying EKS Anywhere. The shared responsibility model is explained further in the EKS Anywhere FAQ.

How is EKS Anywhere deployed?

Users can test EKS Anywhere on their local machines to gain an understanding of what components are involved and how they are deployed. This tutorial involves using eksctl to deploy a cluster on a local machine (such as a laptop).

1. Install the eksctl command-line tool and the eksctl-anywhere plugin.

2. Set a cluster name as an environment variable.

CLUSTER_NAME=anywhere-cluster
            

3. Generate a cluster configuration. The following command will output a YAML file with a default configuration for the EKS Anywhere cluster.

eksctl anywhere generate clusterconfig $CLUSTER_NAME –provider docker > $CLUSTER_NAME.yaml
                

4. Now, let’s see the contents of the generated YAML file.

cat $CLUSTER_NAME.yaml
            

5. We can see the settings that will be applied when the cluster is created. Users may want to customize settings here, such as the Kubernetes cluster version, worker node count, and IP address ranges.

apiVersion: anywhere.eks.amazonaws.com/v1alpha1
 kind: Cluster
 metadata:
   name: anywhere-cluster
 spec:
   clusterNetwork:
     cniConfig:
       cilium: {}
     pods:
       cidrBlocks:
       - 192.168.0.0/16
     services:
       cidrBlocks:
       - 10.96.0.0/12
   controlPlaneConfiguration:
     count: 1
   datacenterRef:
     kind: DockerDatacenterConfig
     name: anywhere-cluster
   externalEtcdConfiguration:
     count: 1
   kubernetesVersion: "1.24"
   managementCluster:
     name: anywhere-cluster
   workerNodeGroupConfigurations:
   - count: 1
     name: md-0
 
 ---
 apiVersion: anywhere.eks.amazonaws.com/v1alpha1
 kind: DockerDatacenterConfig
 metadata:
   name: anywhere-cluster
 spec: {}
 ---
                

6. Once you are satisfied with the configuration, create the cluster.

eksctl anywhere create cluster -f $CLUSTER_NAME.yaml
            

7. The command-line output displays the actions executed for creating the cluster, installing tooling, and generating a Kubeconfig file.

✅ Docker Provider setup is valid
✅ Validate certificate for registry mirror
✅ Validate authentication for git provider
✅ Create preflight validations pass
Creating new bootstrap cluster
Provider specific pre-capi-install-setup on bootstrap cluster
Installing cluster-api providers on bootstrap cluster
Provider specific post-setup
Creating new workload cluster
Installing networking on workload cluster
Creating EKS-A namespace
Installing cluster-api providers on workload cluster
Installing EKS-A secrets on workload cluster
Installing resources on management cluster
Moving cluster management from bootstrap to workload cluster
Installing EKS-A custom components (CRD and controller) on workload cluster
Installing EKS-D components on workload cluster
Creating EKS-A CRDs instances on workload cluster
Installing GitOps Toolkit on workload cluster
GitOps field not specified, bootstrap flux skipped
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!
                    

8. The step above will generate a Kubeconfig file in the current working directory that contains credentials for connecting to the new Kubernetes cluster. Set an environment variable to ensure that Kubectl commands utilize this Kubeconfig file.

export KUBECONFIG=${CLUSTER_NAME}/${CLUSTER_NAME}-eks-a-cluster.kubeconfig
                

9. Now, verify access to the EKS Anywhere Kubernetes cluster by running a kubectl command. The output will show tools installed by EKS Anywhere for the cluster to function, such as CertManager, Cilium, and other controllers.

kubectl get pods  --all-namespaces
                

NAMESPACE                     	 
capd-system                   	 
capi-kubeadm-bootstrap-system 	 
capi-kubeadm-control-plane-system  
capi-system                   	 
cert-manager                  	 
cert-manager                  	 
cert-manager                  	 
eksa-packages                 	 
eksa-system                   	 
etcdadm-bootstrap-provider-system  
etcdadm-controller-system     	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system
                

NAME                                                    
capd-controller-manager-64849988b4-xtbk
capi-kubeadm-bootstrap-controller-manager
capi-kubeadm-control-plane-controller-manager-7966d44fc-7vnsq
capi-controller-manager-ffcf6f4f6-pdb4l                   
cert-manager-65b5449f6f-gv8l9      
cert-manager-cainjector-7dc55b8b4b-nn9fg
cert-manager-webhook-78c66c99c5-btzz8
eks-anywhere-packages-5bb99d74d-4xl8q
eksa-controller-manager-69d4dddccf-68xl2 
etcdadm-bootstrap-provider-controller-manager-6c6f479f84-npcj8
etcdadm-controller-controller-manager-9dbb4669b-czv
cilium-gn4xg
cilium-jv85s 
cilium-operator-8f857c-c27dg    
cilium-operator-8f857c-znsjd     
coredns-75f4dffff6-wlc6c 
coredns-75f4dffff6-zrxvt 	 
kube-apiserver-anywhere-cluster-wrv6v
kube-controller-manager-anywhere-cluster
kube-proxy-6jbcp   
kube-proxy-dwccp 
kube-scheduler-anywhere-cluster-wrv6v
                

You can now proceed with deploying your Kubernetes applications to the new EKS Anywhere cluster.

What are EKS Anywhere’s limitations?

While there are many benefits to deploying EKS Anywhere, some drawbacks also require consideration:

• EKS Anywhere is heavily opinionated. The configuration of all components (such as the Kubernetes control plane) is preconfigured in sync with the managed cloud EKS control plane configuration. While this has benefits, it may also restrict users requiring custom Kubernetes control plane configurations for specialized setups. Tools like Kops, Kubeadm, K3s, and KubeSpray offer more granular control over the cluster configuration and end-to-end cluster lifecycle management.
• The cost of EKS Anywhere support is high. The cost can be thousands per year and can be a roadblock for users on a budget. Obtaining support for cloud-managed EKS clusters is significantly cheaper: AWS Support subscriptions start at $29 per month; however, users will need to factor in the AWS cloud infrastructure cost as well.
• On-premises hardware/software setups need to be compatible with EKS Anywhere. It can be deployed to bare metal servers, CloudStack, and vSphere setups, but users running other types of virtualization software cannot run EKS Anywhere (unless support is released in the future).

Users must carefully evaluate the limitations of EKS Anywhere to ensure that it fits the requirements of their on-premises workloads.

Summary

EKS Anywhere is a valuable software package for deploying EKS clusters to on-premises environments and managing them. It provides a simple interface for deploying, upgrading, and configuring clusters with similar settings as a managed cloud EKS cluster. This enables users to quickly set up hybrid cloud environments, test the viability of migrating Kubernetes workloads to AWS, or set up disaster recovery between on-premises and cloud-based resources.

Users will benefit from carefully evaluating the limitations of EKS Anywhere, assessing the pricing of support plans, and determining whether the out-of-the-box configuration meets the requirements of their Kubernetes workloads. With EKS Anywhere clusters being free to deploy, testing and use case validation is straightforward.

Overall, EKS Anywhere can be a helpful tool for users managing Kubernetes workloads in on-premises environments. Further information can be found here.