Skip to main content

Create a network connection between Astro and GCP

Use this document to learn how you can grant an Astro cluster and its Deployments access to your external Google Cloud Platform (GCP) resources.

Publicly accessible endpoints allow you to quickly connect your Astro clusters or Deployments to GCP through an Airflow connection. If your cloud restricts IP addresses, you can add the external IPs of your Deployment or cluster to an GCP resource's allowlist.

If you have stricter security requirements, you can create a private connection to GCP in a few different ways.

After you create a connection from your cluster to GCP, you might also need to individually authorize Deployments to access specific resources. See Authorize your Deployment using workload identity.

Standard and dedicated cluster support for GCP networking​

Standard clusters have different connection options than dedicated clusters.

Standard clusters can connect to GCP in the following ways:

Dedicated clusters can use all of the same connection options as standard clusters. Additionally, they support a number of private connectivity options including:

  • VPC peering

If you require a private connection between Astro and GCP, Astronomer recommends configuring a dedicated cluster. See Create a dedicated cluster.

Access a public GCP endpoint​

All Astro clusters include a set of external IP addresses that persist for the lifetime of the cluster. When you create a Deployment in your workspace, Astro assigns it one of these external IP addresses. To facilitate communication between Astro and your cloud, you can allowlist these external IPs in your cloud. If you have no other security restrictions, this means that any cluster with an allowlisted external IP address can access your GCP resources through a valid Airflow connection.

Allowlist a Deployment's external IP addresses on GCP​

  1. In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.
  2. Select the Details tab.
  3. In the Other section, you can find the External IPs associated with the Deployment.
  4. Add the IP addresses to the allowlist of any external services that you want your Deployment to access.

When you use publicly accessible endpoints to connect to GCP, traffic moves directly between your Astro cluster and the GCP API endpoint. Data in this traffic never reaches the Astronomer managed control plane. Note that you still might also need to authorize your Deployment to some resources before it can access them. For example, you can Authorize deployments to your cloud with workload identity so that you can avoid adding passwords or other access credentials to your Airflow connections.

Dedicated cluster external IP addresses

If you use Dedicated clusters and want to allowlist external IP addresses at the cluster level instead of at the Deployment level, you can find the list cluster-level external IP addresses in your Organization's Clusters page.

  1. In the Organization section of the Astro UI, click Clusters, then select a cluster.
  2. In the Details page, copy the IP addresses listed under External IPs.
  3. Add the IP addresses to the allowlist of any external services that you want your cluster to access. You can also access these IP addresses from the Details page of any Deployment in the cluster.

After you allowlist a cluster's IP addresses, all Deployments in that cluster have network connectivity to GCP.

Create a private connection between Astro and GCP​

Choose one of the following setups based on the security requirements of your company and your existing infrastructure.

info

This connection option is available only for dedicated Astro Hosted clusters and Astro Hybrid.

VPC peering ensures private and secure connectivity, reduces network transit costs, and simplifies network layouts. Because Astro uses source network address translation (SNAT) that performs many-to-one IP address translations for connections to your data sources, to minimize the risk and concern with IP overlap and exhaustion with dedicated GCP clusters, you might need to confirm that the default Astro subnet and peering ranges do not overlap with the ranges used by your target resource. See create a dedicated GCP cluster for more information about default ranges and alternative configurations.

To create a VPC peering connection between an Astro VPC and a GCP VPC:

  1. Contact Astronomer support and provide the following information:

    • Astro cluster ID and name.
    • Google Cloud project ID of the target VPC.
    • VPC NAME of the target VPC.
    • Classless Inter-Domain Routing (CIDR) block of the target VPC.

    After receiving your request, Astronomer support will create a VPC peering connection from your Astro VPC to your target VPC. The support team will then provide you with your Astro cluster GCP project ID and VPC name.

  2. Using the information provided by Astronomer support, create a peering connection from your target VPC to your Astro cluster VPC. For example, you can use the following gcloud CLI command to create the connection:

    gcloud compute networks peerings create <choose-any-name> --network=<your-target-vpc-network-name>  --peer-project=<your-cluster-project-id> --peer-network=<your-cluster-vpc-name>

After both VPC peering connections have been created, the connection becomes active.

Hostname resolution options​

Securely connect Astro to resources running in other VPCs or on-premises through a resolving service.

As most flexible and reliable solution Astronomer recommends using Domain Name System (DNS) forwarding. In case of small mount of records and unmutable IP addresses, support team can create a Private zone with DNS records, pointed to customer's resources.

Use Domain Name System (DNS) forwarding to allow Astro to resolve DNS queries for resources running in other VPCs or on-premises. You have access to internal resources through private names. All changes in zone will be available for Astro environment immediatelly.

To use this solution, make sure Astro can connect to the DNS server using a VPC peering or VPN connection and then submit a request to Astronomer support. With your request, include the following information:

  • The domain name for forwarding requests
  • The IP address of the DNS server where requests are forwarded

(Optional) Create an Airflow connection to confirm connectivity​

After Astronomer support confirms that DNS forwarding was successfully set up, you can confirm that it works by creating an Airflow connection to a resource running in a VPC or on-premises. See Managing Connections.

See Also​

Was this page helpful?