Google Cloud Platform Data Collection Prerequisites

Google Cloud Platform Data Collection Prerequisites


The following prerequisites are required for each Google Cloud Platform (GCP) project in order to connect to and collect compute engine instance data from GCP:

  1. Google Cloud Platform Service Account
    • Create a service account with Project role set to "Viewer. Alternatively, you can create a role with the minimum permissions required for data collection.
    • Obtain JSON key type credentials;
    • Optionally, link this service account to other GCP projects from which you want to collect data.
  2. Stackdriver Account
    • If you do not have an existing Stackdriver account, create one to monitor your GCP project;
    • Densify supports both Basic and Premium tier accounts.
  3. Data collection APIs
    • Enable Cloud Resource Manager API;
    • Enable Stackdriver Monitoring API.

To learn more watch the video,

Creating a GCP Service Account and Credentials

To connect Densify to your GCP project, you will need a GCP service account and corresponding credentials in JSON format. You need to repeat this process for each of your GCP projects and save the credential JSON files in an encrypted .ZIP file. You will then import this file to create a connection to collect data from your GCP projects.

Use the following instructions to create and configure the GCP service account:

  1. Login to the GCP console and select your project.
  2. From the main menu, select IAM & Admin > Service accounts.
  3. From the Service Accounts page, click on Create Service Account.
  4. In the Create service account pane:
    1. Specify Service account name. As a best practice, use the display name to keep track of what the account will be used for and what permissions are assigned. (e.g. Densify_Connection_Viewer)
    2. Provide a description of the service. This step is optional, but may help you link this account to other projects later on.
    3. Click Create to create the service account.
  5. Select Role > Project and set the role for the whole project to "Viewer".
  6. Click CREATE KEY and choose the JSON as the key type. The key is created and automatically downloaded to your local system. The key file will be located in the /download directory.
  7. Click DONE.
  8. Retain the generated keys for creating the Densify cloud connection.
  9. Note: Do not check these service account keys into your source code or leave them in the download directory.

Creating a Role with Minimum Permissions for Data Collection

Densify recommends using the viewer role, as indicated above, to simplify setup and maintenance of data collection. The viewer role provides read-only access to your GCP services and resources and supports the requirements of the Densify data collection and analysis. As the Densify data collection continues to evolve and expand, you do not need to update this permission policy to include newly added services and features.

Alternatively, you create a custom role and grant the minimum required permissions to the service account to be used Densify for data collection.

You must have the iam.roles.create permission, to create the custom role. The custom role must be created at the Organization level.

Note: This custom role must be updated periodically as Densify’s standard data collection requirements are updated to support additional services and features.

The custom role must have the following permissions to perform the GCP data collection:

  • compute.disks.get
  • compute.disks.list
  • compute.images.list
  • compute.instances.get
  • compute.instances.list
  • compute.licenses.get
  • compute.machineTypes.get
  • compute.machineTypes.list
  • compute.projects.get
  • compute.regions.list
  • compute.zones.list
  • monitoring.metricDescriptors.list
  • monitoring.timeSeries.list
  • resourcemanager.projects.get
  • resourcemanager.projects.list

Use the following instructions to create the custom role:

  1. Login to the GCP console and navigate to Roles in the main menu.
  2. Select your organization from the Organization drop-down.
  3. From the Roles page, click on Create Role.
  4. Enter a Title, Description for the role.
  5. Roles have both an ID and a title. The role ID is a unique identifier for the role and can be auto-generated. The role title appears in the list of roles in the Cloud Console.

    The Role Launch Stage property is optional for this use case.

  6. Click Add Permissions. See the above list and select all 15 permission for your custom role.
  7. Click Add to grant the selected permissions to your custom role.
  8. Click CREATE to create the role. You will now see your new role in the list on the Roles page.
  9. When you create the service account, select this custom role as follows:

Refer to Creating a GCP Service Account and Credentials, above for details.

Using a Single GCP Service Account to Access Multiple Projects

In order to simplify data collection you can grant a single GCP service account access to other projects. This can be done by adding the service account ID from one project to another, through the account's IAM & admin dashboard.

You would then use this master service account to create your Densify cloud connection.

  1. Login to the GCP console and select the project that contains the service account that will become the master account.
  2. From the GCP console main menu, select IAM & admin > Service accounts.
  3. Select the Service account ID to be used as the master account, from the list of service accounts. Copy the Service account ID onto the clipboard, as you cannot search from the Add Members function (in the steps below).
  4. Select the project to which you want to add the service account.
  5. From the IAM & admin menu, select IAM.
  6. Click the ADD in the toolbar.
  7. The Add Members dialog box opens. Paste the Service Account ID into the New Members field,
  8. Select the account’s role in the project. Select the Project > Viewer role for the newly added member.
  9. Click Add to complete and save the change. You can now use the private key previously generated from the service account to access the selected project.
  10. You can repeat these steps for your other projects.

Creating a Stackdriver Account

In order for Densify to collect compute engine instance data, you need a Stackdriver account associated with your GCP project. Your Stackdriver account can be either of the following options:.

  • With a basic tier Stackdriver account, standard workload metrics (CPU, Disk and Network I/O) are automatically collected.
  • With a Premium tier Stackdriver account, in addition to the standard workload metrics, Densify also collects memory metrics.

If you do not have an existing Stackdriver account, use the following instructions to create a Stackdriver account for monitoring:

  1. Login to the GCP console and select your project.
  2. Scroll down the main menu, and select StackdriverMonitoring. The Stackdriver console is displayed.
  3. From the <GCP Project> is not in a Stackdriver account dialog box, select Create a new Stackdriver account and click Continue.
  4. If you do not see this dialog box and the Stackdriver Monitor console is displayed, then your GCP project is already enabled for Stackdriver; you can skip the next step.

  5. From the Create your Stackdriver account dialog box, click Create Account to create a Stackdriver account for your GCP project.
  6. Only add the currently selected user.

Ensuring Data Collection APIs are Enabled

  1. Login to the GCP console and select your project.
  2. From the GCP console main menu, navigate to APIs & services > Dashboard.
  3. From the APIs & services Dashboard, ensure that the following APIs are enabled and listed in the Enabled APIs and services list:
    • Cloud Resource Manager API
    • Stackdriver Monitoring API
  4. If the APIs are not listed, click on ENABLE APIS AND SERVICES at the top of the dashboard. Search for the following APIs from the Library and click ENABLE to enable them:
    • Cloud Resource Manager API
    • Stackdriver Monitoring API

Creating the Cloud Connection in Densify

Once all of the prerequisites are complete, you can create the cloud connection through the Cloud Connection wizard. See Using the Public Cloud Connections Wizard.

Modifying Your Google Cloud Connection

When you create the GCP cloud connection for the first time, Densify discovers all of the projects, associated with the role or service account. Upon saving the connection Densify will schedule data collection from each of the discovered and selected projects.

If subsequently, projects are added, they will not be included in data collection. Additionally, projects that are removed will continue to be included, resulting in wasted time and resources. To add new projects or remove old ones, edit the cloud connection. See Reviewing and Editing a Connection.