Fundamentals of EKS Logging

Introduction

Users should log several components of an Elastic Kubernetes Service (EKS) cluster to ensure easy operations, maintenance, and troubleshooting. An effective logging strategy will involve selecting the appropriate tools and validating that they meet user requirements. 

This article will discuss how to enable logging for each component of an EKS cluster, what tools are available, and how they can be implemented to improve the operational readiness of EKS clusters.

Key concepts about EKS logging

Why is logging important for EKS clusters?

Logging is helpful for any environment hosting critical applications. Log data provides insight into the performance of applications and the underlying infrastructure. The data is essential for analyzing performance, identifying bottlenecks, troubleshooting bugs and unexpected behavior, detecting security breaches, maintaining uptime, and monitoring the environment’s health.

In the context of an EKS cluster, users will benefit from logs providing insights into the control plane, worker nodes, system pods, application pods, and surrounding AWS-related resources. Logging these components will give users deep insights into how the entire cluster behaves, ensuring that they can manage their clusters effectively.

What components of EKS can be logged?

EKS clusters contain several different components that support logging. Users operating production clusters should consider configuring logging for each of these.

Control plane logs

As discussed further below, EKS provides a range of log streams generated by master node components (like the API Server and Kube Scheduler). Enabling these logs using the steps below will provide insight into cluster operations, including performance and security posture.

Worker node logs

The worker nodes in a Kubernetes cluster will generate system logs based on the operating system in use. Collecting the system logs will provide insight into the host’s performance and data, which is helpful in troubleshooting host issues.

Kubernetes-specific logs are generated by the kubelet agent, which is the agent responsible for communicating between the worker node and the control plane. The kubelet logs are typically stored alongside the operating system logs (e.g., the SystemD Journal on Linux) and should be collected to enable analysis of the kubelet’s behavior.

Pod logs

Each pod running in a Kubernetes cluster can produce log output; the contents of the output will depend on the application containers deployed. The application’s developer controls which log messages are output to stdout, and Kubernetes will fetch the logs accordingly when running kubectl logs <pod_name>. Users can collect these logs to enable application-level troubleshooting and diagnostics capabilities.

Machine learning for Kubernetes sizing

Learn More

	Visualize Utilization Metrics	Set Resource Requests & Limits	Set Requests & Limits with Machine Learning	Identify mis-sized containers at a glance & automate resizing	Get Optimal Node Configuration Recommendations
Kubernetes
Kubernetes + Densify

CloudTrail logs

EKS clusters run on AWS infrastructure, and most AWS services will generate CloudTrail log event output. CloudTrail provides insight into API calls made to AWS services (e.g., EC2:RunInstance) and is a valuable tool for troubleshooting problems, auditing AWS resource access, and verifying the operational stability of AWS resources. A production EKS setup should include enabling CloudTrail logging to ensure that all AWS resources implemented for the cluster are logged. The log data is helpful for administrators managing EKS clusters.

Maintaining an effective logging setup for every aspect of an EKS cluster may appear daunting and complex to many users due to the number of components involved. However, implementing an appropriate logging setup will help improve the operational readiness of an EKS cluster, enable easier troubleshooting for future issues, allow more straightforward forensic analysis, and provide greater insight into performance bottlenecks. The payoff for proper logging is significant and will benefit many users.

How can I enable EKS control plane logs?

EKS provides the ability to forward control plane logs automatically to the CloudWatch Logs service. Logs from the control plane are disabled by default to mitigate unnecessary costs, but it is highly recommended that users enable the logs for production clusters where analyzing cluster operations is vital for administrators.

Control plane logs are enabled via the AWS web console or the AWS CLI tool.

Enabling logs via the CLI is done with the following AWS CLI command:

aws eks update-cluster-config --region region-code --name my-cluster --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'
    

Users have a choice of which control plane logs to enable. Generally, users will enable all logs in a production environment, but those optimizing for cost may selectively enable the logs most relevant to user requirements.

The EKS control plane provides log data from several master node components:

• API Server: This is the primary entry point for requests sent to the EKS control plane. Log data from this component provides insight into request latency and volume, authorization history details, and helpful error messages.
• Audit: The audit log is generated by the API Server component but has a separate log stream for easier organization and configuration. The audit log provides insight into exactly what objects are being viewed or modified in the cluster and by which user. This data is helpful for examining changes to the cluster over time, enabling forensic analysis, and troubleshooting problems.
• Controller Manager: The Kube Controller Manager component will log data related to all control loops managing the cluster’s resources; for example, events associated with generating new pods due to applying a new deployment resource are logged. Log data from this component provides insight into many operations constantly occurring within an EKS cluster.
• Authenticator: This log stream is unique to EKS and not present in other Kubernetes distributions. The Authenticator log is generated by the IAM Authenticator component running in the EKS control plane and logs data for each IAM principal’s authentication request. Cluster users will authenticate to EKS clusters via IAM users and IAM roles, and the Authenticator is responsible for mapping the IAM principal to a Kubernetes RBAC resource (users, groups, or service accounts). The log data from this component provides insight into which IAM principals are accessing the EKS cluster and what RBAC resources they are utilizing.
• Scheduler: The Kube Scheduler component assigns pods to worker nodes in a Kubernetes cluster. The Scheduler will log data on how the algorithm calculates which nodes to utilize depending on factors like available capacity, taints, and affinity rules. Log data from this component provides insight into scheduling decisions made by the Kube Scheduler and is relevant for troubleshooting issues related to unexpected scheduling decisions.

The EKS control plane log data provides extensive insight into the cluster’s activities. The data is crucial for administrative operations like troubleshooting issues and forensic analysis, so all users should enable these logs when operating production clusters.

How can I query EKS control plane logs?

EKS control plane logs are exported to the CloudWatch Logs service. CloudWatch Logs supports a tool called Insights for querying and analyzing log data, which is used for querying the EKS control plane logs.

To query the control plane logs for a particular EKS cluster, users will need to open the Insights tool:

1. Open the CloudWatch web console. 
2. Select Logs -> Log Insights in the navigation menu.
3. You will see many log groups listed here, including one for your EKS cluster. Open the log group containing your EKS cluster’s name.
4. Run one of the queries below to analyze data from your EKS cluster.

Query examples

Discovering useful log queries is very helpful for investigations, troubleshooting, and diagnostics purposes. Storing successful queries in a document is a common practice for users who use CloudWatch Log Insights often. Saving important queries saves time in the future and makes it easier to customize existing queries for new purposes.

The CloudWatch Log Insights query syntax can be found here.

Query which IAM principals have accessed the “kubernetes-admin” RBAC user

This user has unlimited permissions to modify the cluster, and its access is heavily restricted as a best practice. Monitoring its actions is useful for investigating potential security incidents. The query can be modified to investigate which other RBAC users are being accessed by IAM principals.

fields @logStream, @timestamp, @message
| sort @timestamp desc
| filter @logStream like /authenticator/
| filter @message like "username=kubernetes-admin"
| limit 50
                    

Query what actions were performed by the “kubernetes-admin” RBAC user

Following on from the above, investigating exactly what actions were performed by this privileged user in the EKS cluster is useful for securing a cluster.

fields @logStream, @timestamp, @message
| filter @logStream like /^kube-apiserver-audit/
| filter strcontains(user.username,"kubernetes-admin")
| sort @timestamp desc
| limit 50
                    

Query which API Server requests resulted in 5XX errors

Analyzing this data is useful for troubleshooting potential issues occurring in the EKS control plane or misconfigured requests being performed by clients.

fields @logStream, @timestamp, responseStatus.code, @message
| filter @logStream like /^kube-apiserver-audit/
| filter responseStatus.code >= 500
| limit 50
                    

Query which RBAC User deleted a particular pod

Queries like this are useful for determining which users accessed or modified a particular resource. Simply modify the “verb” and “requestURI” to perform a range of useful queries related to auditing and root cause analysis.

fields @logStream, @timestamp, @message
| filter @logStream like /^kube-apiserver-audit/
| filter verb == "delete" and requestURI like "/api/v1/namespaces/default/pods/my-app"
| sort @timestamp desc
| limit 10
                    

There are many more types of CloudWatch Log Insights queries available for users. Learning the basic syntax for performing queries will yield many benefits for cluster administrators, especially for the troubleshooting and analysis of cluster behavior.

How can I enable logs for EKS worker nodes and pods?

EKS supports various logging tools provided by AWS, third-party companies, and open-source communities. These tools typically run as software installed on the worker nodes, allowing them access to collect and export node and pod logs to the destination log storage service.

Open-source solutions

All open-source tools enabling log collection for Kubernetes clusters are supported by EKS. Examples of open-source logging tools include Grafana Loki, Logstash, and FluentD. These are all big projects with large user bases and developer communities.

Open-source projects provide a high degree of flexibility and many features at low cost. A key drawback of open-source logging tools is the operational overhead of configuring log storage, though. Proprietary solutions typically manage the operational burden of supplying redundant, highly available, and scalable storage backends for log storage. Open-source projects like Grafana Loki involve self-hosting the log storage, which requires additional operational overhead for the user.

Users will need to evaluate their use cases to determine whether self-hosting their logs is a worthwhile trade-off for an open-source project’s improved flexibility and community support.

Proprietary logging solutions

Companies like SumoLogic, DataDog, Splunk, and New Relic provide managed solutions for log streaming and storage. Users already implementing these types of services in other environments may choose to adopt the same solutions in their EKS clusters for consistency. Managed solutions will cost more than open-source equivalents and may provide reduced feature sets. However, they will handle the operational overhead of log storage, redundancy, and scaling.

AWS

The in-house cluster logging solution provided by AWS is called CloudWatch Container Insights. AWS’s approach to enabling cluster logging involves exporting logs to AWS CloudWatch, a service allowing the storage and analysis of logs and metrics.

Making the right choice

Determining which of the logging solutions above to implement will depend on various factors. Users will have to take into account the following:

• What tools are they comfortable and familiar with using?
• What is their tooling budget?
• How much operational overhead or engineering time are they willing to invest in maintaining a logging setup?
• What specific features of a logging setup does their use case require? For example, are there requirements related to throughput, log volume, storage redundancy, compliance, etc.?

These questions will help users narrow down the ideal logging solution for their use cases. As with all tooling, the best way to find the appropriate one is to test and validate various options. Experimentation will provide data on which tools fit requirements and which are inadequate.

AWS users with simple requirements and no exceptional use cases typically default to implementing CloudWatch Container Insights. This is appropriate for users who are already comfortable using CloudWatch and may already be using it to store logs/metrics from other AWS services. Container Insights is a good starting point for a logging solution, and it is easy to migrate away from if a user eventually decides to switch the logging solution to another provider.

AWS provides a Quickstart Solution that contains all the relevant manifests for enabling Container Insights and deploys the following setup:

• FluentD: A Kubernetes DaemonSet containing FluentD pods is deployed. These pods are configured to collect logs from other resources in the cluster. FluentD can be configured via a Kubernetes ConfigMap to control specifically which logs to aggregate, how to label them, what to exclude, and the log format. FluentD will default to collecting logs from all pods, worker nodes, and the kubelet agents.
• CloudWatch agent: The CloudWatch agent reads the logs collected by FluentD and exports these logs to the CloudWatch service. The agent is installed as a DaemonSet, just like FluentD, which means a pod runs on every worker node.
• IAM Policy: The CloudWatch agent needs IAM permissions to push logs to the CloudWatch service.

The complete installation procedure is located here.

Users will benefit from experimenting with various logging solutions to gather data on which ones meet their requirements.

How can I enable logging for EKS Fargate?

EKS Fargate is a serverless compute feature available for EKS users. Deploying pods to Fargate allows users to delegate the management of worker node compute hosts to AWS. This enables users to mitigate the operational overhead of managing a fleet of EC2 instances.

Pods deployed to Fargate are still capable of exporting log data. However, since AWS manages the underlying compute host, there are fewer options available for configuring log streaming.

The two options for enabling logging for EKS Fargate pods are sidecar containers and the Fluent Bit log router.

Sidecar containers

A sidecar container is a secondary container defined in the pod schema. A pod can define multiple containers to run collectively on the same host.

Implementing sidecar containers helps enable additional functionality for the primary application pods. For example, they can be used to include network proxies, service meshes, and log routers. A typical pattern for EKS Fargate users is to include a sidecar with a logging solution like Fluent Bit or DataDog to capture logs from the primary container and forward them to a destination service. This pattern provides the flexibility of using almost any logging agent (open-source, AWS native, or third-party) for Fargate pods. However, there is added complexity in modifying every Fargate pod’s schema to include an additional container with the logging agent. This can result in significant complexity and overhead in clusters with large numbers of Fargate pods.

Fargate Fluent Bit log router

The sidecar container solution used to be standard for users to implement on Fargate. However, based on user feedback regarding the complexity of managing sidecar logging containers, AWS provided an alternative solution to simplify the logging setup for users.

EKS Fargate nodes now include a built-in log router based on the open-source Fluent Bit project. The log router is transparently installed by default on the underlying Fargate node, so users do not need to include a sidecar container for log routing. Users let the Fluent Bit router manage log streaming automatically for their application pods. The Fluent Bit log router deployed by AWS is capable of streaming Fargate pod logs to a variety of AWS services, including:

• CloudWatch Logs
• OpenSearch
• S3
• Kinesis

Streaming logs to third-party providers like SumoLogic and DataDog is not supported. Use cases requiring third-party log providers are better suited to EC2 worker nodes than Fargate.

Three key steps are required to enable Fluent Bit logging for Fargate nodes:

1. A ConfigMap called aws-logging is created to define the logging configuration. Parameters such as the destination AWS service, log filters, retention time, and log group name are defined here.
2. The ConfigMap must be deployed to a fixed Kubernetes namespace called aws-observability. This specific namespace is where Fargate will search for the aws-logging ConfigMap.
3. The pod running on Fargate must be configured with an IAM role capable of accessing the destination log service. For example, if the log stream destination is CloudWatch Logs, the IAM role must have an IAM policy allowing access to CloudWatch Logs.

The setup process for Fargate logging via Fluent Bit can be found here.

How can I optimize EKS logging costs?

Implementing logging for EKS clusters will incur varying costs depending on the configuration setup. Logs will typically involve expenses for data transfer from the EC2 instances (worker nodes), storage, and log queries. The exact cost will depend on what log service is implemented.

Users can prevent high and unexpected costs by configuring their logging setups appropriately. Log configuration details that will have a significant impact on costs include the following:

• Which pods are selected for logging: Large clusters with high volumes of running pods will produce a significant log volume. Logging agents can be configured to selectively log relevant pods only; for example, pods deployed for testing purposes may not need to be exported to external services since kubectl logs <pod_name> may be enough for testing purposes.
• Which logs are filtered out: Not all log messages generated by pods are helpful for future analysis, so log agents typically include features for ignoring log messages containing specific keywords or labels. Implementing this type of filtering will help reduce noise from irrelevant logs and reduce ingestion costs accordingly.
• Duration of log storage: Storing data always costs money, and keeping high-volume log data with unlimited retention may be unnecessary. Reducing log retention time to only what is necessary will help mitigate costs.
• Scoping log queries appropriately: Log queries are often priced based on how many logs were scanned for a result. Performing queries without specific filters and timeframes will result in higher costs than performing more granular searches on specific log subsets.

For CloudWatch Container Insights, pricing details for log ingestion, storage, and querying can be found here.

Summary of key concepts

Configuring a thorough logging setup is crucial to operating a production EKS cluster. Log data enables users to investigate every aspect of cluster behavior, troubleshoot problems, analyze performance, diagnose security issues, and optimize operations.

Log data is readily available from every component of an EKS cluster, including the control plane, worker nodes, pods, and AWS API events. Collecting and storing this data will be beneficial in the long term. However, the initial configuration will require time and effort to accurately determine what approach will suit the user’s use case.

Identifying the appropriate logging strategy will require testing and validation. Experimenting with various tools and services will provide greater confidence in logging strategy choices and help with validating the selected approach.