Kubernetes Labels in kube-state-metrics

In kube-state-metrics v2.x features have been added to improve performance of both kube-state-metrics, itself and the Prometheus server that collects the resulting data.

One of these improvements affects the usage of Kubernetes object labels and annotations as Prometheus labels of the kube-state-metrics datapoints. In v2.x kube-state-metric, the default settings no longer include the collection of the Kubernetes object labels nor annotations and you need to configure the collection of these items using the command-line options.

Densify Container Data Collection and Kubernetes Labels

Though Densify's container data collection will work without the Kubernetes object labels as kube-state-metrics labels, you may want to enable the kube-state-metrics labels for the following use cases:

• Node group data collection requires the node labels.
• Data collection of other Kubernetes objects attempts to collect labels and annotations. These can be used to sort and filter containers in the UI or API and to create customized reports.

Node Groups

Node groups are not a Kubernetes feature, but rather are implemented by the public cloud provider's Kubernetes solution (e.g. AWS EKS, GCP GKE, Azure AKS). They are also used by 3rd party tools to provision the Kubernetes cluster (e.g. eksctl, kops).

Collecting node group data is only meaningful if you are able to match it to the public cloud provider's node group data (e.g. AWS ASG). In this case you need to enable node group data collection with kube-state-metrics version v2.x or higher.

1. Add the following command-line argument to the kube-state-metrics container:

["--metric-labels-allowlist=nodes=[*]"]

2. You can replace the wildcard (*) with a comma-separated list of specific node labels. This requires specific knowledge of the available node labels in the cluster, which depends on the cloud provider's Kubernetes solution and/or the 3rd party tool used to provision the cluster and their versions. You can do this if the performance of kube-state-metrics and/or Prometheus is a consideration.

Labels of Other Kubernetes Objects

In addition to node labels, Densify attempts to collect the following data, which can be further used as sort/filter criteria and to generate custom reports:

Table: Optional Node Labels and Annotations

Labels		Annotations
namespaces pods deployments replicasets daemonsets	statefulsets jobs cronjobs horizontalpodautoscalers	namespaces

1. If you want to collect this data with kube-state-metrics v2.x or higher, add the following command-line arguments to the kube-state-metrics container:

["--metric-labels-allowlist=nodes=[*],namespaces=[*],pods=[*],deployments=[*],replicasets=[*],daemonsets=[*],statefulsets=[*],jobs=[*],cronjobs=[*],horizontalpodautoscalers=[*]", "--metric-annotations-allowlist=namespaces=[*]"]

2. Optionally, you can specify only the Kubernetes object labels that you need. Contact [email protected] for details.

Legacy Kube-State-Metrics

Both Kubernetes and kube-state-metrics have changed over the years, including addition and removal of features. Feature changes result in changes to kube-state-metrics, including removal of, or replacement by new metrics.

Densify's container data collection supports kube-state-metrics, version 1.5 or higher. If your monitoring stack is running an older version, some metrics listed on Densify's Github page are absent. In this case, Densify collects the older metrics. The table below summarizes the older versions of the metrics that have been deprecated, and their replacements.

Table: Legacy Kube-State-Metrics

Deprecated Metric	Replaced by Metric
kube_hpa_labels	kube_horizontalpodautoscaler_labels
kube_hpa_spec_max_replicas	kube_horizontalpodautoscaler_spec_max_replicas
kube_hpa_spec_min_replicas	kube_horizontalpodautoscaler_spec_min_replicas
kube_hpa_status_condition	kube_horizontalpodautoscaler_status_condition
kube_hpa_status_current_replicas	kube_horizontalpodautoscaler_status_current_replicas
kube_node_status_allocatable_cpu_cores	kube_node_status_allocatable
kube_node_status_allocatable_memory_bytes
kube_node_status_allocatable_pods
kube_node_status_capacity_cpu_cores	kube_node_status_capacity
kube_node_status_capacity_cpu_cores
kube_node_status_capacity_pods
kube_pod_container_resource_limits_cpu_cores	kube_pod_container_resource_limits
kube_pod_container_resource_limits_memory_bytes
kube_pod_container_resource_requests_cpu_cores	kube_pod_container_resource_requests
kube_pod_container_resource_requests_memory_bytes