Data Collection for Containers
Data Collection for Containers
Densify can collect and analyze your container configuration and utilization data and then provide recommendations for optimizing your container-based applications.
Densify's container data collection consists of:
- Deploying the Densify Data Forwarder container, in your current container environment, to collect deployment objects and utilization metrics from Prometheus and;
- Configuring a connection between the Data Forwarder and Prometheus;
- Forwarding the collected data to your Densify instance.
After the Data Forwarder sends the data to your Densify instance, it terminates. Your Densify instance then performs the optimization analysis on the collected data and generates optimization recommendations. The Data Forwarder is scheduled to run periodically during the day.
Densify's container data collection process consists of:
Deploy the Densify Data Forwarder using any one of the following methods:
- For Data Forwarder setup instructions using YAML files with a non-authenticated Prometheus, see Containers Data Forwarder with Prometheus.
- For Data Forwarder setup instructions using YAML files with an authenticated Prometheus, see Containers Data Forwarder with Authenticated Prometheus.
- For Data Forwarder setup instructions using a Helm chart, see Containers Data Forwarder Using a Helm Chart.
- If you are using Red Hat OpenShift and want to set up Data Forwarder using an Operator, see Containers Data Forwarder with Densify Operator.
- Once the Data Forwarder is able to send the collected container metrics,
Densify will enable the container optimization analysis and reports.
Before you begin, refer to the prerequisites, listed below, for a list of container data collection requirements.
- Densify account—Contact Densify for details of your subscription or sign up for a free trial.
- Running cAdvisor as part of the kubelet that by default, provides the workload and configuration data required by Densify.
- Requires v1.5.0 or later
The following items are not mandatory but provide additional environment information for Densify's container optimization analysis.
- Node Exporter—This is an agent deployed on every node to collects data about the nodes, on which the containers are running. This provides the required host-related metrics such as CPU, mem, network, etc.
When deploying Prometheus and kube-state-metrics using a standard operator, some of the metrics that Densify needs for analysis may be excluded (i.e. on a deny list). Refer to Prometheus-Data.md for details of the required metrics.
Contact [email protected] for configuration details.
When deploying the forwarder, you need to ensure that the same version of the forwarder is deployed for all of your clusters. The forwarder has the following 2 settings:
- image: densify/container-optimization-data-forwarder
- imagePullPolicy: Always
If you change that second setting to "IfNotPresent" or "Never" then K8s will never update the image or will only pull the image if it is missing.
If, for example, you select "IfNotPresent" and you have a cluster with 20 nodes and the forwarder has only ever run on nodes 1-10, those nodes all have the version 2.2.1.
When Densify releases the new 2.4.0 version and since the Forwarder does not specify its version by default, and the Forwarder runs on node #15 for an hour, since you may have stopped some services and #15 is now the best node on which to run, but this node does not have the image so it pulls and gets v2.4.0. Then you run more pods so during the next hour the Forwarder is running on node #8 that already has version 2.2.1 so it uses that version, as the policy says don’t look for a new image unless it is missing.
The sets of data from the various hours will not match and will result in Densify loading errors.