A Free Guide to Using EKS Anywhere

EKS Best Practices
calendar March 30, 2023

What is EKS Anywhere?

EKS Anywhere is an AWS offering that lets users
run Elastic Kubernetes Service (EKS) clusters in on-premises environments. Users can deploy a Kubernetes
setup that is compatible with the managed cloud EKS offering without forcing users to migrate data or
compute capacity to the cloud. AWS provides this functionality by packaging a collection of tools for
deploying a Kubernetes distribution with an opinionated configuration. This feature provides an on-ramp
for users interested in gradually shifting toward an EKS-based Kubernetes platform, integrating with
other AWS services and benefiting from an AWS support plan.

Summary of key concepts

How does EKS Anywhere work? EKS Anywhere uses a collection of AWS-supported tools to deploy clusters and manage
their life cycles in on-premises environments.
What are EKS Anywhere’s benefits? EKS Anywhere is useful for developing hybrid cloud setups, preparing for cloud
migrations, or implementing tools to deploy and maintain on-premises clusters.
How do you scale EKS Anywhere nodes? Control plane and worker node counts can be scaled by registering hardware to the
cluster and either updating the cluster configuration manually or configuring the
Cluster Autoscaler tool.
What is the support model and pricing? EKS Anywhere is free to install, but obtaining ongoing developer support from AWS
engineers involves a cost. Users pay for a support subscription on a per-cluster basis.
What are EKS Anywhere’s security considerations? Security for EKS Anywhere deployments is primarily the user’s responsibility. AWS
supports vulnerability patches for relevant software components, but following best
practices and securing on-premises hardware requires user involvement.
How is EKS Anywhere deployed? EKS Anywhere is deployed via the eksctl tool, which
will leverage other software like ClusterAPI, EKS Distro, Flux, etc.
What are EKS Anywhere’s limitations? EKS Anywhere will restrict the control plane configuration settings to match the
cloud-managed EKS settings. Customizing these settings isn’t allowed for the Anywhere
project or for cloud-managed EKS. Use cases requiring customized control plane settings
aren’t supported by EKS Anywhere and will be better served by alternative projects, such
as Kubeadm.

How does EKS Anywhere work?

EKS Anywhere includes several components that are designed to work together to orchestrate an end-to-end
cluster deployment process. These components are mainly transparent to the user, who will primarily
interface with EKS Anywhere using the eksctl tool:

  • eksctl:
    This CLI tool is co-developed by Weaveworks and AWS and is used to provision, upgrade, and operate
    EKS clusters (both on-premises and managed cloud). The eksctl tool is the primary method by which
    users will deploy and upgrade EKS Anywhere clusters.
  • EKS Distro: This
    is the Kubernetes distribution built by AWS that powers the managed EKS service. The distribution is
    open source and publicly provides all container images required to run an EKS-equivalent cluster,
    whether on self-managed cloud compute hosts or on-premises. The container images published for EKS
    Distro include all components needed to run a Kubernetes cluster, such as the Kube API Server, Kube
    Controller Manager, Kube Proxy, Kube Scheduler, etcd, CoreDNS, and so on. EKS Anywhere deploys the
    container images published by EKS Distro, which allows users to run an on-premises cluster with the
    same configuration as a managed cloud EKS cluster. AWS uses these same container images to deploy
    managed cloud EKS clusters.
  • Tinkerbell: This open-source project enables
    functionality related to configuring operating systems in preparation for running Kubernetes
    workloads. Tinkerbell will set up a DHCP service, metadata service, and IP addressing functionality
    for the cluster.
  • ClusterAPI: This open-source project
    is widely used for managing large fleets of Kubernetes clusters for both on-premises and managed
    cloud setups. This tool performs Kubernetes cluster bootstrapping actions such as configuring master
    nodes, etcd nodes, storage classes, and cluster networking.
  • Flux: This open-source project implements a GitOps workflow for Kubernetes clusters.
    EKS Anywhere will (optionally) set up this tool and commit the cluster configuration to a Git
    repository. Storing cluster configurations in a Git repository allows for version control of all
    configuration resources, tracking historical changes and providing a centralized approach to
    deploying new changes to the cluster. New changes are simply pushed to the Git repository, and the
    EKS Anywhere cluster will automatically download and deploy any configuration changes observed in
    the repository.
  • EKS
    Connector
    : This tool enables the exporting of on-premises cluster data to the
    AWS EKS web console. Users can implement this tool to gain visibility of their EKS Anywhere clusters
    directly from the AWS console. This functionality allows users to browse the configuration and
    status of their EKS Anywhere clusters in the same location where they can see their managed cloud
    EKS clusters.

While these are the key tools for managing an EKS Anywhere cluster lifecycle, there are other secondary
tools curated by AWS that
are designed to extend EKS Anywhere’s functionality. These tools include MetalLB (a load balancer
provisioner), Cluster Autoscaler (a node scaling tool), CertManager (a certificate manager for
Kubernetes clusters), and many others. The tools are curated and tested by AWS, and they are deployable
directly through the eksctl tool.

Spend less time optimizing Kubernetes Resources. Rely on AI-powered Kubex - an automated Kubernetes optimization platform

Free 60-day Trial

What are EKS Anywhere’s benefits?

There are multiple use cases where users benefit from deploying EKS Anywhere clusters for on-premises
environments:

  • Users can implement EKS Anywhere to validate that their workloads are compatible with cloud-managed
    EKS. Since EKS Anywhere uses the same control plane settings as cloud-managed EKS, implementing this
    project is a useful way to prepare and validate readiness for a migration from on-premises to
    cloud-managed EKS. 
  • Users with hybrid cloud setups can maintain some resources with a cloud provider and keep some
    resources on-premises. Unifying the resource configuration between the various infrastructures will
    provide better operational stability, unify the skills required for management, and enable use cases
    like disaster recovery and failover.
  • EKS Anywhere provides a vendor-supported approach to managing the Kubernetes cluster lifecycle.
    Users may find value in running tools with vendor support assurances, particularly for
    mission-critical workloads.

The benefits of EKS Anywhere for on-premises Kubernetes users are significant and can provide a helpful
bridge between on-premises and cloud environments. Users can test out EKS Anywhere for free to validate
that it suits their requirements.

Note: EKS Anywhere does not allow mixing worker nodes or control planes between on-premises setups and
the AWS cloud environment. All cluster resources must be deployed within the same environment.

How do you scale EKS Anywhere nodes?

Users have the ability to horizontally or vertically scale an EKS Anywhere cluster’s worker and control
plane nodes. This can be done via either the eksctl
command-line tool or the Cluster
Autoscaler
project. In either case, the cluster will need information about the available
hardware to create the new nodes. 

The cluster requires information such as the IP address, host name, gateway, and root filesystem device
to correctly configure the hardware with an operating system and required software dependencies. This
hardware information will need to be generated by the user and stored in a text file (hardware.csv).
This file will be supplied to the EKS Anywhere cluster during either cluster creation or upgrade. The
cluster will then have an awareness of what hardware is available in the on-premises environment for
scaling operations.

Here’s an example of a hardware.csv file:

hostname,bmc_ip,bmc_username,bmc_password,mac,ip_address,netmask,gateway,nameservers,labels,disk
eksa-cp01,10.10.44.1,root,PrZ8W93i,CC:48:3A:00:00:01,10.10.50.2,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=cp,/dev/sda
eksa-cp02,10.10.44.2,root,Me9xQf93,CC:48:3A:00:00:02,10.10.50.3,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=cp,/dev/sda
eksa-cp03,10.10.44.3,root,Z8x2M6hl,CC:48:3A:00:00:03,10.10.50.4,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=cp,/dev/sda
eksa-wk01,10.10.44.4,root,B398xRTp,CC:48:3A:00:00:04,10.10.50.5,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=worker,/dev/sda
eksa-wk02,10.10.44.5,root,w7EenR94,CC:48:3A:00:00:05,10.10.50.6,255.255.254.0,10.10.50.1,8.8.8.8|8.8.4.4,type=worker,/dev/sda
            

Spend less time optimizing Kubernetes Resources. Rely on AI-powered Kubex - an automated Kubernetes optimization platform

Free 60-day Trial

Hardware information can be supplied during cluster creation:

eksctl anywhere create cluster --hardware-csv hardware.csv

It can also be supplied during cluster upgrades:

eksctl anywhere upgrade cluster --hardware-csv hardware.csv

The hardware information supplied to the cluster can be viewed with kubectl:

kubectl get hardware -n eksa-system
 NAME
 eksa-controlplane
 eksa-worker1
 eksa-worker2

The above confirms that some hardware resources have been registered to the cluster. The cluster can then
be scaled manually by updating the cluster’s configuration file:

# cluster.yaml
kind: Cluster
metadata:
  name: test-cluster
spec:
  workerNodeGroupsConfiguration:
    count: 1           # Update from 1 -> 2 for horizontal scaling
                

We can then deploy the updated node count:

eksctl anywhere upgrade cluster -f cluster.yaml
            

For vertical scaling, users will register more powerful hardware in the hardware.csv file and then run
the upgrade command above. This will trigger EKS Anywhere to replace smaller nodes with larger nodes
with more hardware capacity. 

EKS Anywhere also supports dynamic scaling via the Cluster
Autoscaler tool
for bare metal hardware.

Spend less time optimizing Kubernetes Resources. Rely on AI-powered Kubex - an automated Kubernetes optimization platform

Free 60-day Trial

Support model and pricing

EKS Anywhere is an open-source project and can be deployed for free. However, users requiring ongoing
support from AWS will need to acquire an AWS Enterprise Support subscription and an additional EKS Anywhere Support
subscription
.

The EKS Anywhere Support subscription enables users to request guidance on deployment and troubleshooting
from AWS engineers. The subscription includes support related to all tools included with EKS Anywhere
and integrations with curated packages.

The support subscription will be helpful to users running production-critical systems requiring immediate
and direct access to vendor support. Users running EKS Anywhere for lower-priority workloads will be
fine without the paid subscription. Since the EKS Anywhere components are all open-source, publicly
available, and documented, many users can troubleshoot common problems without vendor support.

The cost of an EKS Anywhere Support
subscription depends on how many clusters require support and for what duration. At the time this
article was written, the baseline cost for one year of support for one EKS Anywhere cluster is $24,000;
three-year contracts for one cluster will cost $54,000 ($18,000 per year). This pricing is in addition
to the cost of AWS Enterprise Support, which is a requirement for enabling EKS Anywhere Support; AWS
Enterprise Support starts at $15,000 per month.

Due to the significant costs associated with accessing EKS Anywhere Support, users will benefit from
carefully evaluating whether their setups are mission-critical enough to justify the expenditures.

What are EKS Anywhere’s security considerations?

Maintaining the security of an EKS
Anywhere cluster
is a shared responsibility between the user and AWS. AWS provides assurances
regarding patching vulnerabilities in the bundled software, but the user must regularly upgrade their
clusters, secure RBAC configurations, restrict API endpoint access, and maintain the bare metal and
virtual machine security posture. The user is responsible for all aspects of securing Kubernetes
workloads in the EKS Anywhere cluster, the software dependencies, and the underlying compute hosts.

Users must carefully plan how to approach their security posture when deploying EKS Anywhere. The shared
responsibility model is explained further in the EKS Anywhere FAQ.

How is EKS Anywhere deployed?

Users can test EKS Anywhere on their local machines to gain an understanding of what components are
involved and how they are deployed. This tutorial
involves using eksctl to deploy a cluster on a local machine (such as a laptop).

1. Install the eksctl
command-line tool and the eksctl-anywhere plugin
.

2. Set a cluster name as an environment variable.

CLUSTER_NAME=anywhere-cluster
            

3. Generate a cluster configuration. The following command will output a YAML file with a default
configuration for the EKS Anywhere cluster.

eksctl anywhere generate clusterconfig $CLUSTER_NAME –provider docker > $CLUSTER_NAME.yaml
                

4. Now, let’s see the contents of the generated YAML file.

cat $CLUSTER_NAME.yaml
            

5. We can see the settings that will be applied when the cluster is created. Users may want to customize
settings here, such as the Kubernetes cluster version, worker node count, and IP address ranges.

apiVersion: anywhere.eks.amazonaws.com/v1alpha1
 kind: Cluster
 metadata:
   name: anywhere-cluster
 spec:
   clusterNetwork:
     cniConfig:
       cilium: {}
     pods:
       cidrBlocks:
       - 192.168.0.0/16
     services:
       cidrBlocks:
       - 10.96.0.0/12
   controlPlaneConfiguration:
     count: 1
   datacenterRef:
     kind: DockerDatacenterConfig
     name: anywhere-cluster
   externalEtcdConfiguration:
     count: 1
   kubernetesVersion: "1.24"
   managementCluster:
     name: anywhere-cluster
   workerNodeGroupConfigurations:
   - count: 1
     name: md-0
 
 ---
 apiVersion: anywhere.eks.amazonaws.com/v1alpha1
 kind: DockerDatacenterConfig
 metadata:
   name: anywhere-cluster
 spec: {}
 ---
                

6. Once you are satisfied with the configuration, create the cluster.

eksctl anywhere create cluster -f $CLUSTER_NAME.yaml
            

7. The command-line output displays the actions executed for creating the cluster, installing tooling,
and generating a Kubeconfig
file
.

✅ Docker Provider setup is valid
✅ Validate certificate for registry mirror
✅ Validate authentication for git provider
✅ Create preflight validations pass
Creating new bootstrap cluster
Provider specific pre-capi-install-setup on bootstrap cluster
Installing cluster-api providers on bootstrap cluster
Provider specific post-setup
Creating new workload cluster
Installing networking on workload cluster
Creating EKS-A namespace
Installing cluster-api providers on workload cluster
Installing EKS-A secrets on workload cluster
Installing resources on management cluster
Moving cluster management from bootstrap to workload cluster
Installing EKS-A custom components (CRD and controller) on workload cluster
Installing EKS-D components on workload cluster
Creating EKS-A CRDs instances on workload cluster
Installing GitOps Toolkit on workload cluster
GitOps field not specified, bootstrap flux skipped
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!
                    

8. The step above will generate a Kubeconfig file in the current working directory that contains
credentials for connecting to the new Kubernetes cluster. Set an environment variable to ensure that
Kubectl commands utilize this Kubeconfig file.

export KUBECONFIG=${CLUSTER_NAME}/${CLUSTER_NAME}-eks-a-cluster.kubeconfig
                

9. Now, verify access to the EKS Anywhere Kubernetes cluster by running a kubectl command. The output
will show tools installed by EKS Anywhere for the cluster to function, such as CertManager, Cilium, and
other controllers.

kubectl get pods  --all-namespaces
                
NAMESPACE                     	 
capd-system                   	 
capi-kubeadm-bootstrap-system 	 
capi-kubeadm-control-plane-system  
capi-system                   	 
cert-manager                  	 
cert-manager                  	 
cert-manager                  	 
eksa-packages                 	 
eksa-system                   	 
etcdadm-bootstrap-provider-system  
etcdadm-controller-system     	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system                   	 
kube-system
                
NAME                                                    
capd-controller-manager-64849988b4-xtbk
capi-kubeadm-bootstrap-controller-manager
capi-kubeadm-control-plane-controller-manager-7966d44fc-7vnsq
capi-controller-manager-ffcf6f4f6-pdb4l                   
cert-manager-65b5449f6f-gv8l9      
cert-manager-cainjector-7dc55b8b4b-nn9fg
cert-manager-webhook-78c66c99c5-btzz8
eks-anywhere-packages-5bb99d74d-4xl8q
eksa-controller-manager-69d4dddccf-68xl2 
etcdadm-bootstrap-provider-controller-manager-6c6f479f84-npcj8
etcdadm-controller-controller-manager-9dbb4669b-czv
cilium-gn4xg
cilium-jv85s 
cilium-operator-8f857c-c27dg    
cilium-operator-8f857c-znsjd     
coredns-75f4dffff6-wlc6c 
coredns-75f4dffff6-zrxvt 	 
kube-apiserver-anywhere-cluster-wrv6v
kube-controller-manager-anywhere-cluster
kube-proxy-6jbcp   
kube-proxy-dwccp 
kube-scheduler-anywhere-cluster-wrv6v
                

You can now proceed with deploying your Kubernetes applications to the new EKS Anywhere cluster.

Spend less time optimizing Kubernetes Resources. Rely on AI-powered Kubex - an automated Kubernetes optimization platform

Free 60-day Trial

What are EKS Anywhere’s limitations?

While there are many benefits to deploying EKS Anywhere, some drawbacks also require consideration:

  • EKS Anywhere is heavily opinionated. The configuration of all components (such as
    the Kubernetes control plane) is preconfigured in sync with the managed cloud EKS control plane
    configuration. While this has benefits, it may also restrict users requiring custom Kubernetes
    control plane configurations for specialized setups. Tools like Kops, Kubeadm, K3s, and KubeSpray
    offer more granular control over the cluster configuration and end-to-end cluster lifecycle
    management.
  • The cost of EKS Anywhere support is high. The cost can be thousands per year and
    can be a roadblock for users on a budget. Obtaining support for cloud-managed EKS clusters is
    significantly cheaper: AWS Support
    subscriptions
    start at $29 per month; however, users will need to factor in the AWS cloud
    infrastructure cost as well.
  • On-premises hardware/software setups need to be compatible with EKS Anywhere. It
    can be deployed to bare metal servers, CloudStack, and vSphere setups, but users running other types
    of virtualization software cannot run EKS Anywhere (unless support is released in the future).

Users must carefully evaluate the limitations of EKS Anywhere to ensure that it fits the requirements of
their on-premises workloads.

Summary

EKS Anywhere is a valuable software package for deploying EKS clusters to on-premises environments and
managing them. It provides a simple interface for deploying, upgrading, and configuring clusters with
similar settings as a managed cloud EKS cluster. This enables users to quickly set up hybrid cloud
environments, test the viability of migrating Kubernetes workloads to AWS, or set up disaster recovery
between on-premises and cloud-based resources.

Users will benefit from carefully evaluating the limitations of EKS Anywhere, assessing the pricing of
support plans, and determining whether the out-of-the-box configuration meets the requirements of their
Kubernetes workloads. With EKS Anywhere clusters being free to deploy, testing and use case validation
is straightforward.

Overall, EKS Anywhere can be a helpful tool for users managing Kubernetes workloads in on-premises
environments. Further information can be found
here
.

$100

Amazon Gift Card for Kubex Demo

If you are involved in a K8s environment with 500 or more containers, spend 20 minutes seeing a demo and we will send you $100.

Book it