Prerequisites

The following software is required for Densify container data collection and optimization.

1. Densify account—Contact Densify for details of your subscription or sign up for a free trial.
• See www.densify.com/product/trial.
2. Kubernetes or OpenShift must be deployed.
• Running cAdvisor as part of the kubelet provides the workload and configuration data required by Densify.
3. kube-state-metrics—This service monitors the Kubernetes API server and generates metrics from the various objects inside the individual Kubernetes components. This service provides orchestration and cluster level metrics such as deployments, pod metrics, resource reservation, etc. The collected metrics allow Densify to get a complete picture of how your containers are setup. i.e. Replica Sets, Deployments, Pod and Container Labels.
• Requires v1.5.0 or later. There are additional considerations when using v2.x. See below.
• https://github.com/kubernetes/kube-state-metrics
4. Prometheus—Collects metrics from configured targets at given intervals. It provides the monitoring/data aggregation layer. It must be deployed and configured to collect kube-state-metrics and cAdvisor/kubelet metrics.
• https://prometheus.io
5. Node Exporter—This is an agent deployed on every node to collect data about the nodes, on which the containers are running. This provides the required host-related metrics such as CPU, mem, network, etc.
• https://github.com/prometheus/node_exporter

The following item is not mandatory but provides additional environment information for Densify's container optimization analysis.

1. Openshift-state-metrics—Expands upon kube-state-metrics by adding metrics for OpenShift-specific resources and provide additional details such as Cluster Resource Quotas (CRQ).
• https://github.com/openshift/openshift-state-metrics

When deploying Prometheus and kube-state-metrics using a standard operator, some of the metrics that Densify needs for analysis may be excluded (i.e. on a deny list). Refer to Prometheus-Data.md for details of the required metrics.

Contact [email protected] for configuration details.

Additional Configuration for kube-state-metrics v2.x and Higher

To learn more watch the following video:

Testing the Data Forwarder

Once configmap.yml has been updated with your connection parameters, you can create and schedule the pod. A test pod is created via the pod.yml file. This pod contains all of the utilities to connect to Prometheus, collect metrics, compress them and send the compressed data to Densify.

1. Open a terminal session to run kubectl commands for your Kubernetes cluster.
2. Optionally, you can create a namespace for the data forwarder and related .YML files. For example, to create the namespace, "collector", execute the following command:

kubectl create namespace collector

Remember to create all your data collection pods in this namespace to organize your running pods (i.e. append the -n collector option to the commands below).

3. Create the configmap in the "collector" namespace using the following command:

kubectl create -f configmap.yml

Note: If you are using a namespace with your data collection pods, add the -n <namespace> option to your kubectl commands. For example: 
kubectl create -f configmap.yml -n collector


4. Create the pod using the following command:

kubectl create -f pod.yml -n collector

This command creates the pod according to the details provided by configmap.yml. The jobs inside the pod collect data, compress and transfer data to your Densify instance. Once the data has been collected and sent to your Densify instance, the pod terminates.

5. Run the following command to review the pod status:

kubectl get pods -n collector

The command should return the details of the pod. The status will be similar to:

densify 0/1 Completed 0 1m

6. You can use the pod ID to review the log file for the job execution.

kubectl logs densify - n collector

You can review the content of the log file to verify successful completion of the data collection and transfer of the compressed data file to your Densify instance.

Figure: Sample Log File

{"level":"info","pkg":"default","time":1682517791260344,"caller":"src/densify.com/forwarderv2/files.go:88","goid":1,"message":"zipping mycluster.zip, contents: cluster - 21 files; container - 16 files; node - 17 files; node_group - 0 files; hpa - 6 files; rq - 0 files; crq - 0 files; total - 60 files"}

{"level":"info","pkg":"default","file":"mycluster.zip","time":1682517791462827,"caller":"src/densify.com/forwarderv2/main.go:49","goid":1,"message":"file uploaded successfully"}

The number of compressed and uploaded files (displayed at the end of the log) is dependent on the types of pods you are running in your cluster and if you have enabled the optional node-exporter component. An indication of a successful data transfer is determined when the number of uploaded files range from 20 to 70.

If you see anything less than 20 files uploaded, then this indicates a transfer problem. You need to review the log to determine the issue.

7. Optionally, once you have reviewed the logs you can remove the pod, as it is no longer required:

kubectl delete -f pod.yml

Editing configmap.yml

You can make changes to the configmap.yml file as required. After editing the file you must delete and recreate the config map to pick up any changes:

kubectl delete -f configmap.yml -n <namespace>

kubectl create -f configmap.yml -n <namespace>

You do not need to recreate the job or pod files. The next, scheduled run will start sending the updated set of data. If you want to test your changes you can run the pod.yml immediately:

kubectl create -f pod.yml-n <namespace>

Scheduling the Data Forwarder

Schedule the pod to run at the same interval you are using for data collection, as defined in the configmap.yml.

1. Create the CronJob object using the following command:

kubectl create -f cronjob.yml

Note: If you created and used a namespace for the configmap.yml, then you need to use the same namespace for the cronjob.yml file. For example: 
kubectl create -f cronjob.yml -n collector


Similar to pod.yml, this command creates the pod according to the details provided by configmap.yml. Additionally it creates the cron job that runs the pod hourly, on the hour.

2. You can use the get pods command to review the last 3 jobs that were executed successfully. If there is a failure for any reason, the pod history is retained.

You can now let your [email protected] know that data forwarder is running.

Figure: kubectl get pods Command