Containers Data Forwarder with Densify Operator

Containers Data Forwarder with Densify Operator

#410220

The Data Forwarder is an on-demand container that collects your container environment data from Kubernetes via Prometheus and forwards that data to Densify for analysis.

This topic shows you how to configure and deploy the Data Forwarder using the Densify Operator, which is currently supported on Red Hat Openshift. If you do not want to use the Densify Operator on Openshift, you can still deploy the Data Forwarder with YAML files. See Container Data Forwarder with Authenticated Prometheus for YAML file deployment instructions.

Contact your [email protected] for Densify credentials and to enable your Densify instance with container optimization after the Data Forwarder is deployed.

Prerequisites

The following software is required for Densify container data collection and optimization.

  1. Densify account—Contact Densify for details of your subscription or sign up for a free trial.
  2. Kubernetes or OpenShift must be deployed.
    • Running cAdvisor as part of the kubelet provides the workload and configuration data required by Densify.
  3. kube-state-metrics—This service monitors the Kubernetes API server and generates metrics from the various objects inside the individual Kubernetes components. This service provides orchestration and cluster level metrics such as deployments, pod metrics, resource reservation, etc. The collected metrics allow Densify to get a complete picture of how your containers are setup i.e. Replica Sets, Deployments, Pod and Container Labels.
  4. Prometheus or other supported observability platform—Collects metrics from configured targets at given intervals. It provides the monitoring/data aggregation layer. It must be deployed and configured to collect kube-state-metrics and cAdvisor/kubelet metrics. See
  5. Node Exporter—This is an agent deployed on every node to collect data about the nodes, on which the containers are running. This provides the required host-related metrics such as CPU, mem, network, etc.

The following item is not mandatory but provides additional environment information for Densify's container optimization analysis.

  1. Openshift-state-metrics—Expands upon kube-state-metrics by adding metrics for OpenShift-specific resources and provides additional details such as Cluster Resource Quotas (CRQ).

When deploying Prometheus and kube-state-metrics using a standard operator, some of the metrics that Densify needs for analysis may be excluded (i.e. on a deny list). Refer to Prometheus-Data.md for details of the required metrics.

Contact [email protected] for configuration details.

Additional Configuration for kube-state-metrics v2.x and Higher

Kubernetes Labels in kube-state-metrics

In kube-state-metrics v2.x features have been added to improve performance of both kube-state-metrics, itself and the Prometheus server that collects the resulting data.

One of these improvements affects the usage of Kubernetes object labels and annotations as Prometheus labels of the kube-state-metrics datapoints. In v2.x kube-state-metric, the default settings no longer include the collection of the Kubernetes object labels nor annotations and you need to configure the collection of these items using the command-line options.

Densify Container Data Collection and Kubernetes Labels

Though Densify's container data collection will work without the Kubernetes object labels as kube-state-metrics labels, you may want to enable the kube-state-metrics labels for the following use cases:

  • Node group data collection requires the node labels.
  • Data collection of other Kubernetes objects attempts to collect labels and annotations. These can be used to sort and filter containers in the UI or API and to create customized reports.

Node Groups

Node groups are not a Kubernetes feature, but rather are implemented by the public cloud provider's Kubernetes solution (e.g. AWS EKS, GCP GKE, Azure AKS). They are also used by 3rd party tools to provision the Kubernetes cluster (e.g. eksctl, kops).

Collecting node group data is only meaningful if you are able to match it to the public cloud provider's node group data (e.g. AWS ASG). In this case you need to enable node group data collection with kube-state-metrics version v2.x or higher.

  1. Add the following command-line argument to the kube-state-metrics container:

    2. ["--metric-labels-allowlist=nodes=[*]"]

  2. You can replace the wildcard (*) with a comma-separated list of specific node labels. This requires specific knowledge of the available node labels in the cluster, which depends on the cloud provider's Kubernetes solution and/or the 3rd party tool used to provision the cluster and their versions. You can do this if the performance of kube-state-metrics and/or Prometheus is a consideration.

Labels of Other Kubernetes Objects

In addition to node labels, Densify attempts to collect the following data, which can be further used as sort/filter criteria and to generate custom reports:

Table: Optional Node Labels and Annotations

Labels

Annotations
  • namespaces
  • pods
  • deployments
  • replicasets
  • daemonsets
  • statefulsets
  • jobs
  • cronjobs
  • horizontalpodautoscalers
  • namespaces
  1. If you want to collect this data with kube-state-metrics v2.x or higher, add the following command-line arguments to the kube-state-metrics container:

    2. ["--metric-labels-allowlist=nodes=[*],namespaces=[*],pods=[*],deployments=[*],replicasets=[*],daemonsets=[*],statefulsets=[*],jobs=[*],cronjobs=[*],horizontalpodautoscalers=[*]", "--metric-annotations-allowlist=namespaces=[*]"]

  2. Optionally, you can specify only the Kubernetes object labels that you need. Contact [email protected] for details.

Note:  The Data Forwarder is only supported on Linux OS and AMD64 architecture.

Deploying the Data Forwarder

To deploy the Data Forwarder, you need to do the following from the Red Hat Openshift Container Platform console:

  1. Install the Densify Operator as an admin user. See Installing the Densify Operator.

  2. Create the Densify Data Forwarder instance. See Deploying the Data Forwarder.

  3. Review the Data Forwarder pod processes, logs, cron job, and ensure that container metric data files were sent to your Densify instance. See Reviewing the Data Forwarder Pod.

Installing the Densify Operator

To install an Operator, you will need admin privileges to the Red Hat Openshift Container Platform. Perform the following steps as an admin user:

  1. From the Red Hat Openshift Container Platform console, navigate to Operators > OperatorHub > Monitoring from the left menu.
  2. Select the Densify Operator tile. The Densify Operator details panel is displayed.
  3. Click Install. The Install Operator page is displayed.
  4. Select the following configuration values on the Install Operator page:

    5. Table: Install Operator Settings

    Configuration

    Select Value

    Upgrade Channel

    v1.0

    Install Mode

    All namespace on the cluster (default)

    Installed Namespace

    Select the namespace, or leave as:

    openshift-operators

    Approve Strategy

    Automatic

  5. Click Install. The Installed Operators page is displayed with the Densify Operator installation status.

Note:  After the Densify Operator is installed, other users with operator access can see the Densify Operator under the Installed Operators page.

Deploying the Data Forwarder

Red Hat Openshift Container Platform users are able to deploy the Densify Data Forwarder once the Densify Operator is installed.

  1. From the Red Hat Openshift Container Platform console, navigate to Operators > Installed Operators in the left menu.
  2. Click the Densify Operator link.

    3. From the Densify Operator's Details tab, a description of the operator, prerequisites, basic installation instructions, configurations, contact and documentation links, and cluster version details are displayed.

  3. From the Densify Operator tile in the Details tab, click Create Instance.

    4. The Create Densify page is displayed.

  4. Configure the Densify Operator (i.e. Densify Data Forwarder) parameters in either the Form View or the YAML View. Both views have identical configuration parameters. Review the Table: Required Connectivity Parameters for the Data Forwarder table for a list of parameter descriptions required. If you want to change the default value of additional parameters, add the additional parameters in the Form View.

  5. Click Create.

    6. This action deploys and schedules the Densify Data Forwarder pod for data collection. This operation takes you back to the Densify Operator page, where the operator object is displayed.

  6. Click the operator object link. The Densify Operator Overview page is displayed with conditions and pod status.

    7. You can click on the Resources tab to view the config map used for this pod.

Reviewing the Data Forwarder Pod

Once the Densify Operator instance has been created, you can review the Data Forwarder pod status and check if your Densify instance has received the container metric data files.

  1. From the Red Hat Openshift Container Platform console, navigate to Workloads > Pods in the left menu.

    2. Details and status of pods are listed on this page. The name of the pod is a combination of name specified in the Create Instance page and the name identified in the config map.

  2. Click on the Densify pod link to review further details.
    • Click on the Details tab to review pod memory, CPU, disk, and network I/O usage.
    • Click on the YAML tab to see the yaml files used for this pod.
    • Click on the Logs tab to review the pod processing logs. The number of metric data files sent to your Densify instance should be displayed in the logs.

      • {"level":"info","pkg":"default","time":1682521649018741,"caller":"src/densify.com/forwarderv2/files.go:88","goid":1,"message":"zipping mycrc.zip, contents: cluster - 21 files; container - 16 files; node - 17 files; node_group - 0 files; hpa - 4 files; rq - 0 files; crq - 7 files; total - 65 files"}

      {"level":"info","pkg":"default","file":"data/mycrc.zip","time":1682521649264663,"caller":"src/densify.com/forwarderv2/main.go:49","goid":1,"message":"file uploaded successfully"}

      The number of compressed and uploaded files (displayed near the end of the log) is dependent on the types of pods you are running in your cluster and if you have enabled the optional node-exporter component. An indication of a successful data transfer is determined when the number of uploaded files range from 20 to 70.

      If you see anything less than 20 files uploaded, then this indicates a transfer problem. You need to review the log to determine the issue.

  3. From the Red Hat Openshift Container Platform console, navigate to Workloads > Cron Jobs in the left menu.

    4. Details and schedule of cron jobs are listed on this page. The name of the Data Forwarder cron job displayed on this page is a combination of the name specified in the Create Instance page and the name identified in the config map.

When the Data Forwarder pod completes successfully, contact [email protected] to enable your Densify instance with container optimization.