Data Collection for Containers

Data Collection for Containers

#410140

Densify can collect and analyze your container configuration and utilization data and then provide recommendations for optimizing your container-based applications.

To learn more watch the following video:

Densify's container data collection consists of:

  • Deploying the Densify Data Forwarder container, in your current container environment, to collect deployment objects and utilization metrics from Prometheus and;
  • Configuring a connection between the Data Forwarder and Prometheus;
  • Forwarding the collected data to your Densify instance.

After the Data Forwarder sends the data to your Densify instance, it terminates. Your Densify instance then performs the optimization analysis on the collected data and generates optimization recommendations. The Data Forwarder is scheduled to run periodically during the day, sending the latest container metrics to your Densify instance for analysis.

Before you begin, refer to the prerequisites, listed below, for a list of container data collection requirements.

Densify's container data collection process consists of:

  1. Deploy the Densify Data Forwarder using any one of the following methods:

  2. Once the Data Forwarder is able to send the collected container metrics, Densify will enable the container optimization analysis and reports.

Prerequisites

The following items are necessary for Densify container data collection and optimization.

  1. Densify account— Contact Densify for details of your subscription or sign up for a free trial.
  2. Kubernetes or OpenShift must be deployed.
    • kubelet with cAdvisor is required to provide workload data to Densify
  3. kube-state-metrics—This service monitors the Kubernetes API server and generates metrics from the various objects inside the individual Kubernetes components. This service provides orchestration and cluster level metrics such as deployments, pod metrics, resource reservation, etc. The collected metrics allow Densify to get a complete picture of how your containers are setup. i.e. Replica Sets, Deployments, Pod and Container Labels.
  4. Prometheus—Provides the monitoring/data aggregation layer. It must be deployed and configured to collect kube-state-metrics and cAdvisor/kubelet metrics.

The following items are not mandatory but provide additional environment information for Densify's container optimization analysis.

  1. Node Exporter—Collects data about the nodes, on which the containers are running. This provides the classical host-related metrics such as CPU, mem, network, etc.
  2. openshift-state-metrics—These metrics provide additional details for OpenShift-specific items such as Cluster Resource Quotas (CRQ).

Contact [email protected] for configuration details.

Other Considerations

When deploying the forwarder, you need to ensure that the same version of the forwarder is deployed for all of your clusters. The forwarder has the following 2 settings:

  • image: densify/container-optimization-data-forwarder
  • imagePullPolicy: Always

If you change that second setting to "IfNotPresent" or "Never" then K8s will never update the image or will only pull the image if it is missing.

If, for example, you select "IfNotPresent" and you have a cluster with 20 nodes and the forwarder has only ever run on nodes 1-10, those nodes all have the version 2.2.1.

When Densify releases the new 2.3.0 version and since the Forwarder does not specify its version by default, and the Forwarder runs on node #15 for an hour, since you may have stopped some services and #15 is now the best node on which to run, but this node does not have the image so it pulls and gets v2.3.0. Then you run more pods so during the next hour the Forwarder is running on node #8 that already has version 2.2.1 so it uses that version, as the policy says don’t look for a new image unless it is missing.

The sets of data from the various hours will not match and will result in Densify loading errors.