EKS Anywhere is an AWS offering that lets users
run Elastic Kubernetes Service (EKS) clusters in on-premises environments. Users can deploy a Kubernetes
setup that is compatible with the managed cloud EKS offering without forcing users to migrate data or
compute capacity to the cloud. AWS provides this functionality by packaging a collection of tools for
deploying a Kubernetes distribution with an opinionated configuration. This feature provides an on-ramp
for users interested in gradually shifting toward an EKS-based Kubernetes platform, integrating with
other AWS services and benefiting from an AWS support plan.
How does EKS Anywhere work? | EKS Anywhere uses a collection of AWS-supported tools to deploy clusters and manage their life cycles in on-premises environments. |
What are EKS Anywhere’s benefits? | EKS Anywhere is useful for developing hybrid cloud setups, preparing for cloud migrations, or implementing tools to deploy and maintain on-premises clusters. |
How do you scale EKS Anywhere nodes? | Control plane and worker node counts can be scaled by registering hardware to the cluster and either updating the cluster configuration manually or configuring the Cluster Autoscaler tool. |
What is the support model and pricing? | EKS Anywhere is free to install, but obtaining ongoing developer support from AWS engineers involves a cost. Users pay for a support subscription on a per-cluster basis. |
What are EKS Anywhere’s security considerations? | Security for EKS Anywhere deployments is primarily the user’s responsibility. AWS supports vulnerability patches for relevant software components, but following best practices and securing on-premises hardware requires user involvement. |
How is EKS Anywhere deployed? | EKS Anywhere is deployed via the eksctl tool, which will leverage other software like ClusterAPI, EKS Distro, Flux, etc. |
What are EKS Anywhere’s limitations? | EKS Anywhere will restrict the control plane configuration settings to match the cloud-managed EKS settings. Customizing these settings isn’t allowed for the Anywhere project or for cloud-managed EKS. Use cases requiring customized control plane settings aren’t supported by EKS Anywhere and will be better served by alternative projects, such as Kubeadm. |
EKS Anywhere includes several components that are designed to work together to orchestrate an end-to-end
cluster deployment process. These components are mainly transparent to the user, who will primarily
interface with EKS Anywhere using the eksctl tool:
While these are the key tools for managing an EKS Anywhere cluster lifecycle, there are other secondary
tools curated by AWS that
are designed to extend EKS Anywhere’s functionality. These tools include MetalLB (a load balancer
provisioner), Cluster Autoscaler (a node scaling tool), CertManager (a certificate manager for
Kubernetes clusters), and many others. The tools are curated and tested by AWS, and they are deployable
directly through the eksctl tool.
Spend less time optimizing Kubernetes Resources. Rely on AI-powered Kubex - an automated Kubernetes optimization platform
Free 60-day TrialThere are multiple use cases where users benefit from deploying EKS Anywhere clusters for on-premises
environments:
The benefits of EKS Anywhere for on-premises Kubernetes users are significant and can provide a helpful
bridge between on-premises and cloud environments. Users can test out EKS Anywhere for free to validate
that it suits their requirements.
Note: EKS Anywhere does not allow mixing worker nodes or control planes between on-premises setups and
the AWS cloud environment. All cluster resources must be deployed within the same environment.
Users have the ability to horizontally or vertically scale an EKS Anywhere cluster’s worker and control
plane nodes. This can be done via either the eksctl
command-line tool or the Cluster
Autoscaler project. In either case, the cluster will need information about the available
hardware to create the new nodes.
The cluster requires information such as the IP address, host name, gateway, and root filesystem device
to correctly configure the hardware with an operating system and required software dependencies. This
hardware information will need to be generated by the user and stored in a text file (hardware.csv).
This file will be supplied to the EKS Anywhere cluster during either cluster creation or upgrade. The
cluster will then have an awareness of what hardware is available in the on-premises environment for
scaling operations.
Here’s an example of a hardware.csv file:
hostname,bmc_ip,bmc_username,bmc_password,mac,ip_address,netmask,gateway,nameservers,labels,disk
eksa-cp01,10.10.44.1,root,PrZ8W93i,CC:48:3A:00:00:01,10.10.50.2,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=cp,/dev/sda
eksa-cp02,10.10.44.2,root,Me9xQf93,CC:48:3A:00:00:02,10.10.50.3,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=cp,/dev/sda
eksa-cp03,10.10.44.3,root,Z8x2M6hl,CC:48:3A:00:00:03,10.10.50.4,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=cp,/dev/sda
eksa-wk01,10.10.44.4,root,B398xRTp,CC:48:3A:00:00:04,10.10.50.5,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=worker,/dev/sda
eksa-wk02,10.10.44.5,root,w7EenR94,CC:48:3A:00:00:05,10.10.50.6,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=worker,/dev/sda
Spend less time optimizing Kubernetes Resources. Rely on AI-powered Kubex - an automated Kubernetes optimization platform
Free 60-day TrialHardware information can be supplied during cluster creation:
eksctl anywhere create cluster --hardware-csv hardware.csv
It can also be supplied during cluster upgrades:
eksctl anywhere upgrade cluster --hardware-csv hardware.csv
The hardware information supplied to the cluster can be viewed with kubectl:
kubectl get hardware -n eksa-system
NAME
eksa-controlplane
eksa-worker1
eksa-worker2
The above confirms that some hardware resources have been registered to the cluster. The cluster can then
be scaled manually by updating the cluster’s configuration file:
# cluster.yaml
kind: Cluster
metadata:
name: test-cluster
spec:
workerNodeGroupsConfiguration:
count: 1 # Update from 1 -> 2 for horizontal scaling
We can then deploy the updated node count:
eksctl anywhere upgrade cluster -f cluster.yaml
For vertical scaling, users will register more powerful hardware in the hardware.csv file and then run
the upgrade command above. This will trigger EKS Anywhere to replace smaller nodes with larger nodes
with more hardware capacity.
EKS Anywhere also supports dynamic scaling via the Cluster
Autoscaler tool for bare metal hardware.
Spend less time optimizing Kubernetes Resources. Rely on AI-powered Kubex - an automated Kubernetes optimization platform
Free 60-day TrialEKS Anywhere is an open-source project and can be deployed for free. However, users requiring ongoing
support from AWS will need to acquire an AWS Enterprise Support subscription and an additional EKS Anywhere Support
subscription.
The EKS Anywhere Support subscription enables users to request guidance on deployment and troubleshooting
from AWS engineers. The subscription includes support related to all tools included with EKS Anywhere
and integrations with curated packages.
The support subscription will be helpful to users running production-critical systems requiring immediate
and direct access to vendor support. Users running EKS Anywhere for lower-priority workloads will be
fine without the paid subscription. Since the EKS Anywhere components are all open-source, publicly
available, and documented, many users can troubleshoot common problems without vendor support.
The cost of an EKS Anywhere Support
subscription depends on how many clusters require support and for what duration. At the time this
article was written, the baseline cost for one year of support for one EKS Anywhere cluster is $24,000;
three-year contracts for one cluster will cost $54,000 ($18,000 per year). This pricing is in addition
to the cost of AWS Enterprise Support, which is a requirement for enabling EKS Anywhere Support; AWS
Enterprise Support starts at $15,000 per month.
Due to the significant costs associated with accessing EKS Anywhere Support, users will benefit from
carefully evaluating whether their setups are mission-critical enough to justify the expenditures.
Maintaining the security of an EKS
Anywhere cluster is a shared responsibility between the user and AWS. AWS provides assurances
regarding patching vulnerabilities in the bundled software, but the user must regularly upgrade their
clusters, secure RBAC configurations, restrict API endpoint access, and maintain the bare metal and
virtual machine security posture. The user is responsible for all aspects of securing Kubernetes
workloads in the EKS Anywhere cluster, the software dependencies, and the underlying compute hosts.
Users must carefully plan how to approach their security posture when deploying EKS Anywhere. The shared
responsibility model is explained further in the EKS Anywhere FAQ.
Users can test EKS Anywhere on their local machines to gain an understanding of what components are
involved and how they are deployed. This tutorial
involves using eksctl to deploy a cluster on a local machine (such as a laptop).
1. Install the eksctl
command-line tool and the eksctl-anywhere plugin.
2. Set a cluster name as an environment variable.
CLUSTER_NAME=anywhere-cluster
3. Generate a cluster configuration. The following command will output a YAML file with a default
configuration for the EKS Anywhere cluster.
eksctl anywhere generate clusterconfig $CLUSTER_NAME –provider docker > $CLUSTER_NAME.yaml
4. Now, let’s see the contents of the generated YAML file.
cat $CLUSTER_NAME.yaml
5. We can see the settings that will be applied when the cluster is created. Users may want to customize
settings here, such as the Kubernetes cluster version, worker node count, and IP address ranges.
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: anywhere-cluster
spec:
clusterNetwork:
cniConfig:
cilium: {}
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 10.96.0.0/12
controlPlaneConfiguration:
count: 1
datacenterRef:
kind: DockerDatacenterConfig
name: anywhere-cluster
externalEtcdConfiguration:
count: 1
kubernetesVersion: "1.24"
managementCluster:
name: anywhere-cluster
workerNodeGroupConfigurations:
- count: 1
name: md-0
---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: DockerDatacenterConfig
metadata:
name: anywhere-cluster
spec: {}
---
6. Once you are satisfied with the configuration, create the cluster.
eksctl anywhere create cluster -f $CLUSTER_NAME.yaml
7. The command-line output displays the actions executed for creating the cluster, installing tooling,
and generating a Kubeconfig
file.
✅ Docker Provider setup is valid
✅ Validate certificate for registry mirror
✅ Validate authentication for git provider
✅ Create preflight validations pass
Creating new bootstrap cluster
Provider specific pre-capi-install-setup on bootstrap cluster
Installing cluster-api providers on bootstrap cluster
Provider specific post-setup
Creating new workload cluster
Installing networking on workload cluster
Creating EKS-A namespace
Installing cluster-api providers on workload cluster
Installing EKS-A secrets on workload cluster
Installing resources on management cluster
Moving cluster management from bootstrap to workload cluster
Installing EKS-A custom components (CRD and controller) on workload cluster
Installing EKS-D components on workload cluster
Creating EKS-A CRDs instances on workload cluster
Installing GitOps Toolkit on workload cluster
GitOps field not specified, bootstrap flux skipped
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!
8. The step above will generate a Kubeconfig file in the current working directory that contains
credentials for connecting to the new Kubernetes cluster. Set an environment variable to ensure that
Kubectl commands utilize this Kubeconfig file.
export KUBECONFIG=${CLUSTER_NAME}/${CLUSTER_NAME}-eks-a-cluster.kubeconfig
9. Now, verify access to the EKS Anywhere Kubernetes cluster by running a kubectl command. The output
will show tools installed by EKS Anywhere for the cluster to function, such as CertManager, Cilium, and
other controllers.
kubectl get pods --all-namespaces
NAMESPACE
capd-system
capi-kubeadm-bootstrap-system
capi-kubeadm-control-plane-system
capi-system
cert-manager
cert-manager
cert-manager
eksa-packages
eksa-system
etcdadm-bootstrap-provider-system
etcdadm-controller-system
kube-system
kube-system
kube-system
kube-system
kube-system
kube-system
kube-system
kube-system
kube-system
kube-system
kube-system
NAME
capd-controller-manager-64849988b4-xtbk
capi-kubeadm-bootstrap-controller-manager
capi-kubeadm-control-plane-controller-manager-7966d44fc-7vnsq
capi-controller-manager-ffcf6f4f6-pdb4l
cert-manager-65b5449f6f-gv8l9
cert-manager-cainjector-7dc55b8b4b-nn9fg
cert-manager-webhook-78c66c99c5-btzz8
eks-anywhere-packages-5bb99d74d-4xl8q
eksa-controller-manager-69d4dddccf-68xl2
etcdadm-bootstrap-provider-controller-manager-6c6f479f84-npcj8
etcdadm-controller-controller-manager-9dbb4669b-czv
cilium-gn4xg
cilium-jv85s
cilium-operator-8f857c-c27dg
cilium-operator-8f857c-znsjd
coredns-75f4dffff6-wlc6c
coredns-75f4dffff6-zrxvt
kube-apiserver-anywhere-cluster-wrv6v
kube-controller-manager-anywhere-cluster
kube-proxy-6jbcp
kube-proxy-dwccp
kube-scheduler-anywhere-cluster-wrv6v
You can now proceed with deploying your Kubernetes applications to the new EKS Anywhere cluster.
Spend less time optimizing Kubernetes Resources. Rely on AI-powered Kubex - an automated Kubernetes optimization platform
Free 60-day TrialWhile there are many benefits to deploying EKS Anywhere, some drawbacks also require consideration:
Users must carefully evaluate the limitations of EKS Anywhere to ensure that it fits the requirements of
their on-premises workloads.
EKS Anywhere is a valuable software package for deploying EKS clusters to on-premises environments and
managing them. It provides a simple interface for deploying, upgrading, and configuring clusters with
similar settings as a managed cloud EKS cluster. This enables users to quickly set up hybrid cloud
environments, test the viability of migrating Kubernetes workloads to AWS, or set up disaster recovery
between on-premises and cloud-based resources.
Users will benefit from carefully evaluating the limitations of EKS Anywhere, assessing the pricing of
support plans, and determining whether the out-of-the-box configuration meets the requirements of their
Kubernetes workloads. With EKS Anywhere clusters being free to deploy, testing and use case validation
is straightforward.
Overall, EKS Anywhere can be a helpful tool for users managing Kubernetes workloads in on-premises
environments. Further information can be found
here.