icon-cloud-connector.svg
Cloud & Branch Connector

Troubleshooting Cloud Connector with HashiCorp Vault for Google Cloud Platform

This article provides troubleshooting information and guidelines about Cloud Connector deployment in Google Cloud Platform (GCP) with HashiCorp Vault, described in Storing Your Secret Credentials in HashiCorp Vault for Google Cloud Platform-Based Cloud Connectors and Deploying Zscaler Cloud Connector on the Google Cloud Platform. You can learn about different issues, potential causes, and corresponding solutions for the deployed Zscaler Cloud Connectors.

Prerequisites

To troubleshoot your Cloud Connector, ensure that you have the following:

  • In GCP:
    • Access to the GCP console: The ability to view and modify settings and IAM service accounts and roles in GCP.
    • Vault/GCP sync service account: A service account with the secrets JSON file exported from GCP and uploaded to HashiCorp with the Service Account Key Admin GCP role.
    • Cloud Connector virtual machine (VM) service account: A service account assigned to the VM at deployment and referenced in HashiCorp as bound_service_accounts with the Service Account Token Creator GCP role.
  • In HashiCorp:
    • Access to the Vault CLI: The ability to view and modify settings in the Vault CLI.
    • Vault address: The URL of your Vault instance.
    • Secret path: The path within the Vault where secrets are stored.
    • GCP IAM authentication: Ensure that the GCP auth method is enabled.
    • GCP IAM role: The GCP IAM type authentication role.

Troubleshooting

The following is a step-by-step troubleshooting guide for Cloud Connector deployment in GCP with HashiCorp:

  • To verify your service account configuration:

    • Check the service account attached to the VM:
      1. Sign in to the GCP console.
      2. In the top-left corner of the console, select Navigation menu (.).
      3. From the left-side navigation, select Compute Engine > VM instances.
      4. On the VM instances page, select your VM instance.
      5. On the Details tab, under API and identity management, verify that the correct Service account is displayed.
    • Verify the service account permissions:
      1. Sign in to the GCP console.
      2. In the top-left corner of the console, select Navigation menu (.).
      3. From the left-side navigation, select IAM & Admin > Service Accounts
      4. On the Service accounts page, select your service account.
      5. On the service account page, select the Permissions tab.
      6. On the Permissions tab, ensure that the Service Account Token Creator role is correctly assigned.
    • Confirm the service account in the HashiCorp Vault's bound_service_accounts by contacting your Vault administrator to verify that the service account email is listed in the bound_service_accounts for the IAM authentication role.
    Close
  • To verify your VM configuration, check your VM metadata:

    1. Sign in to the GCP console.
    2. In the top-left corner of the console, select Navigation menu (.).
    3. From the left-side navigation, select Compute Engine > VM instances.
    4. On the VM instances page, select your VM instance.
    5. On the Details tab, locate the Custom metadata section.
    6. Verify that the metadata keys for Vault Address, Secret Path, and IAM Auth Role are correctly set per your Vault configuration.

      ZSCALER
       hcp_vault_addr: "https://vault-cluster-public-vault-example.z1.hashicorp.cloud:8200"
       hcp_vault_secret_path: "admin/secret/data/example-zscaler-secret"
       hcp_vault_role_name: "example-iam-auth-role"
    Close
  • To verify your Vault configuration:

    • Confirm your Vault address and accessibility:
      • Ensure that the Vault address, protocol, and port number (i.e., 8200) are correct.
      • Ensure that the Vault UI or API is reachable from your machine by testing the accessibility.
    • Verify your GCP IAM authentication role in the Vault by confirming the following with your administrator:
      • The GCP IAM authentication role exists and is named correctly.
      • The type is set to iam.
      • The appropriate policies are attached.
      • The max_jwt_exp is set to "60m".
      • The bound_service_accounts include your VM service account.
    • Verify your Vault GCP auth credentials:
      • Configure the GCP auth method to use the VaultServiceAccountKey.json credentials. Ensure that you successfully created a service account with the Service Account Key Admin permissions and that the service account credentials are exported and used for auth/gcp/config (i.e., VaultServiceAccountKey.json).
      • HashiCorp CLI:

        vault write auth/gcp/config \

        credentials=@VaultServiceAccountKey.json

    Close
  • To test your network connectivity:

    1. Verify that your network tags and firewall rules are correctly set:
      1. Sign in to the GCP console.
      2. In the top-left corner of the console, select Navigation menu (.).
      3. From the left-side navigation, select Compute Engine > VM instances.
      4. On the VM instances page, select your VM instance.
      5. On the Details tab, in the Network tags section, note the tags assigned to your VM.
      6. In the top-left corner of the console, select Navigation menu (.).
      7. From the left-side navigation, select VPC Network > Firewall.
      8. On the Firewall policies page, ensure that your outbound traffic to the Vault address on port 8200 is permitted.
    2. Secure Socket Shell (SSH) into your VM instance.
    3. Replace <vault-address> with your Vault's address. Omit https:// or http://.

      sudo bash
      nc -zv vault-cluster-public-vault-abc.xyz.z1.hashicorp.cloud 8200
      Connection to vault-cluster-public-vault-abc.xyz.z1.hashicorp.cloud 8200 port [tcp/*] succeeded!
    4. Test the connection using curl.

      curl -v <vault-address>:8200/v1/sys/health
      A JSON response indicating Vault's health should be returned.
    Close

The following table displays common issues and resolutions:

Potential CausesTroubleshooting Action
Permission is denied or authentication fails.

Verify the following:

  • The service account assigned to the Cloud Connector VMs has the Service Account Token Creator role.
  • The service account is included in the Vault's bound_service_accounts.
  • max_jwt_exp is correctly set in the Vault.
  • Check your GCP jwt: https://jwt.ms/
The Vault is unreachable from the VM, causing network connectivity issues.

Verify the following:

  • Firewall rules allow outbound traffic to the Vault on port 8200.
  • The Vault address and DNS resolution are correct from the Cloud Connector.
  • There are no network outages or issues with connectivity or routing.
Your access is denied to the secret path after authentication succeeds.

Verify the following:

  • The policies attached to the IAM authentication role include "read and list" permissions for the secret path.
  • The secret exists at the specified path.
  • There are no incorrectly spelled words or incorrect paths.
Related Articles
Enabling Remote Assistance Enabling Remote AccessTroubleshooting Cloud Connector with Amazon Web ServicesTroubleshooting Cloud Connector with Microsoft AzureTroubleshooting Cloud Connector with Google Cloud PlatformTroubleshooting Cloud Connector with HashiCorp Vault for Google Cloud Platform