Containers Data Forwarder Using a Helm Chart

Containers Data Forwarder Using a Helm Chart

#410210

The Data Forwarder is an on-demand container that collects your container environment data from Kubernetes via Prometheus and forwards that data to Densify for analysis.

This topic shows you how to configure and deploy the Data Forwarder for data collection using a Helm chart.

After deploying the Data Forwarder, contact your Cloud Advisor to enable your Densify instance with container optimization.

Prerequisites

The following items are necessary for Densify container data collection and optimization.

  1. Densify account— Contact Densify for details of your subscription or sign up for a free trial.
  2. Kubernetes or OpenShift must be deployed.
    • kubelet with cAdvisor is required to provide workload data to Densify
  3. kube-state-metrics—This is a service that monitors the Kubernetes API server and generates metrics from the various objects inside the individual Kubernetes components, such as deployments, nodes and pods. The collected metrics allow Densify to get a complete picture of how your containers are setup. i.e. Replica Sets, Deployments, Pod and Container Labels.
  4. Prometheus—Provides the monitoring/data aggregation layer. It must be deployed and configured to collect kube-state-metrics and cAdvisor/kubelet metrics.
  5. Helm—To deploy Data Forwarder using a Helm chart, install the latest Helm 3 version.

The following item is not mandatory but does provide additional environment information for Densify container optimization.

  1. Node Exporter—Collects data about the nodes, on which the containers are running.

Contact your Cloud Advisor for configuration details.

Note: The Data Forwarder is only supported on Linux OS and AMD64 architecture.

Deploying Data Forwarder for Data Collection

To configure and deploy the Data Forwarder, you need to:

  1. Setup the repo using Helm from Densify's GitHub page. See Setting Up the Densify Repo Using Helm.
  2. Update the required parameters in the values.yaml file. See Configuring the Data Forwarder.

  3. Deploy and schedule the Data Forwarder pod to run regularly. See Deploying and Scheduling the Data Forwarder.

Setting Up the Densify Repo Using Helm

The Densify Helm repo can be obtained from https://densify-dev.github.io/helm-charts/. For details on the Data Forwarder Helm chart, see https://github.com/densify-dev/helm-charts/tree/master/charts/container-optimization-data-forwarder. Follow the steps below to add and explore the repo:

  1. From your working Helm environment CLI, add the Densify Helm repo:
  2. helm repo add densify https://densify-dev.github.io/helm-charts

  3. To view details about charts within the Densify repo, use Helm search:
  4. helm search repo densify

  5. To view the parameter values of the Densify Data Forwarder chart, type the following command:
  6. helm show values densify/container-optimization-data-forwarder

    Alternatively, you can download a copy of the values.yaml file from https://raw.githubusercontent.com/densify-dev/helm-charts/master/charts/container-optimization-data-forwarder/values.yaml.

  7. Edit the chart parameters using the Helm CLI, or edit and save the downloaded values.yaml file according to the parameters described in the next section. See Configuring the Data Forwarder.

Configuring the Data Forwarder

To configure the Data Forwarder for your environment, you need to edit the values.yaml file with the following connectivity parameters, listed in the table below. For a list of all the available parameters, refer to https://github.com/densify-dev/helm-charts/tree/master/charts/container-optimization-data-forwarder#configuration.

Table: Required Connectivity Parameters for the Data Forwarder

Parameter

Description

Default

config.densify.hostname

Specify the Densify instance hostname. You may need to specify a fully qualified domain name.

<instance>.densify.com

config.densify.protocol

Specify the Densify connection protocol: http or https.

<http|https>

config.densify.port

Specify the Densify connection port.

443

config.densify.user

Specify the Densify user account. You must specify either a password or an epassword along with this parameter. This user must already exist in your Densify instance and have API access privileges.

Alternatively, you can use config.densify.UserSecretName for authentication instead of user/(e)password combination (see config.densify.UserSecretName below).

nil

config.densify.password

Specify the password for the Densify user.

Only specify a password or an encrypted password (not both). If you specify a password , then disable the config.densify.epassword parameter.

nil

config.densify.epassword

Specify the encrypted password (i.e. epassword) for the Densify user.

If you specify an epassword, then disable the config.densify.password parameter.

nil

config.densify.UserSecretName

Specify the secret name used to store the Densify user and epassword (the keys must contain username and epassword).

If this parameter is used, then disable the following parameters: config.densify.username, config.densify.epassword, config.densify.password.

nil

config.prometheus.hostname

Specify the Prometheus address.

Using the internal service name is recommended:

<service name>.<namespace>.svc

nil

config.prometheus.protocol

Specify the Prometheus connection protocol: http or https.

<http|https>

config.prometheus.port

Specify the Prometheus service connection port.

9090

config.prometheus.clustername

Optional, but recommended: Specify a name by which to identify this cluster within Densify. If this parameter is disabled, then the specified Prometheus hostname is used to identify the cluster in the Densify Console.

nil

config.prometheus.oauth_token

Specify the path to the OAuth token used to authenticate with a secured Prometheus server.

/var/run/secrets/kubernetes.io/ serviceaccount/token

config.prometheus.ca_certificate

Specify the CA certificate used to commuicate with a secured Prometheus server.

/var/run/secrets/kubernetes.io/ serviceaccount/service-ca.crt

config.zipname

Specify the name of the zipfile to send to Densify.

data/<zipfilename>

Deploying and Scheduling the Data Forwarder

After customizing the values.yaml file for your environment, deploy the pod using the command below. The pod is automatically scheduled to run at the top of each hour, which is based on the configuration in the values.yaml file.

  1. Install Data Forwarder helm chart with the updated values.yaml file, which contains your connectivity parameters.
  2. helm install densify-helm densify/container-optimization-data-forwarder -f values.yaml

    This example uses the default namespace, however, you should use an appropriate namespace for the Densify pod. The output should be similar to the following:

    Copy
    NAME: densify-helm
    LAST DEPLOYED: Wed Dec 1 15:00:00 2020
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
  3. Review your logs to ensure that the pod completed successfully and metric data files were sent to Densify. The name of the config map and cronjob is in the format: <helm-chart-name>-densify-forwarder. In the example above, the config map and cron job would be labeled as densify-helm-densify-forwarder.
  4. Note: You can customize the "densify-forwarder" part of the name by editing the nameOverride parameter in values.yaml. See https://github.com/densify-dev/helm-charts/tree/master/charts/container-optimization-data-forwarder#configuration.

When the Data Forwarder pod completes successfully, contact your Cloud Advisor to enable your Densify instance with container optimization.