Skip to main content

Create a network connection between Astro and Azure

Use this document to learn how you can grant an Astro cluster and its Deployments access to your external Azure resources.

Publicly accessible endpoints allow you to quickly connect your Astro clusters or Deployments to Azure through an Airflow connection. If your cloud restricts IP addresses, you can add the external IPs of your Deployment or cluster to an Azure resource's allowlist.

If you have stricter security requirements, you can create a private connection to Azure in a few different ways.

After you create a connection from your cluster to Azure, you might also need to individually authorize Deployments to access specific resources. See Authorize your Deployment using workload identity.

Standard and dedicated cluster support for Azure networking​

Standard clusters have different connection options than dedicated clusters.

Standard clusters can connect to Azure in the following ways:

Dedicated clusters can also connect to Azure using static IP addresses. Additionally, they support a number of private connectivity options including:

  • VNet peering
  • Azure Private Link

If you require a private connection between Astro and Azure, Astronomer recommends configuring a dedicated cluster. See Create a dedicated cluster.

Access a public Azure endpoint​

All Astro clusters include a set of external IP addresses that persist for the lifetime of the cluster. When you create a Deployment in your workspace, Astro assigns it one of these external IP addresses. To facilitate communication between Astro and your cloud, you can allowlist these external IPs in your cloud. If you have no other security restrictions, this means that any cluster with an allowlisted external IP address can access your Azure resources through a valid Airflow connection.

Allowlist a Deployment's external IP addresses on Azure​

  1. In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.
  2. Select the Details tab.
  3. In the Other section, you can find the External IPs associated with the Deployment.
  4. Add the IP addresses to the allowlist of any external services that you want your Deployment to access.

When you use publicly accessible endpoints to connect to Azure, traffic moves directly between your Astro cluster and the Azure API endpoint. Data in this traffic never reaches the Astronomer managed control plane. Note that you still might also need to authorize your Deployment to some resources before it can access them. For example, you can Authorize deployments to your cloud with workload identity so that you can avoid adding passwords or other access credentials to your Airflow connections.

Dedicated cluster external IP addresses

If you use Dedicated clusters and want to allowlist external IP addresses at the cluster level instead of at the Deployment level, you can find the list of cluster-level external IP addresses in your Organization's Clusters.

  1. In the Organization section of the Astro UI, click Clusters, then select a cluster.
  2. In the Details page, copy the IP addresses listed under External IPs.
  3. Add the IP addresses to the allowlist of any external services that you want your cluster to access. You can also access these IP addresses from the Details page of any Deployment in the cluster.

After you allowlist a cluster's IP addresses, all Deployments in that cluster have network connectivity to Azure.

Create a private connection between Astro and Azure​

The option that you choose is determined by the security requirements of your company and your existing infrastructure.

info

This connection option is only available for dedicated Astro Hosted clusters and Astro Hybrid.

To set up a private connection between an Astro Virtual Network (VNet) and an Azure VNet, you can create a VNet peering connection. VNet peering ensures private and secure connectivity, reduces network transit costs, and simplifies network layouts.

  1. Retrieve the following information from the target Azure environment that you want to connect with:

    • Azure Tenant ID and Subscription ID.
    • VNet ID.
    • Resource Group ID.

  2. Prepare the astro-vnet-peering-creator-role.json JSON file with the following permissions. Replace {customer-subscription-id} with your value:

    {
        "Name": "Astro VNET Peering Contributor",
        "IsCustom": true,
        "Description": "Can create VNET peering with Astro.",
        "Actions": [
            "Microsoft.Resources/subscriptions/resourceGroups/read",
            "Microsoft.Resources/subscriptions/read",
            "Microsoft.Network/virtualNetworks/read",
            "Microsoft.Network/virtualNetworks/write",
            "Microsoft.Network/virtualNetworks/peer/action",
            "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write",
            "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read"
        ],
        "NotActions": [

        ],
        "AssignableScopes": [
          "/subscriptions/{customer-subscription-id}"
        ]
    }

  3. Run the following Azure CLI commands to give Astronomer support temporary permissions to establish a VNet peering connection:

    # Add Astronomer Service Principal
    az ad sp create --id a67e6057-7138-4f78-bbaf-fd9db7b8aab0

    # Create a Custom role with permissions prepared in previous step
    az role definition create --role-definition ~/astro-vnet-peering-creator-role.json

    # Assign Custom role to the Astronomer Service Principal ({customer-subscription-id} has to be replaced with your value)
    az role assignment create \
    --assignee a67e6057-7138-4f78-bbaf-fd9db7b8aab0 \
    --role "Astro VNET Peering Contributor" \
    --scope "/subscriptions/{customer-subscription-id}"

    # Verify an assignment
    az role assignment list --assignee a67e6057-7138-4f78-bbaf-fd9db7b8aab0 --all -o table

  4. Contact Astronomer support to tell them that you granted permissions to the Astronomer Service Principal. In addition, provide the following details in your request:

    • Astro Cluster ID
    • Azure Tenant ID and Subscription ID of the target VNet
    • Resource group ID
    • VNet ID for the peering

After receiving your request, Astronomer support creates a VNet peering connection between the two VNets. No other actions are required from you. Astronomer support will notify you when the connection is ready to use.

When the network connection is confirmed, you can delete the temporary roles you created using the following command. Replace {customer-subscription-id} with your value:

az role assignment delete --assignee a67e6057-7138-4f78-bbaf-fd9db7b8aab0 --role "Astro VNET Peering Contributor" --scope "/subscriptions/{customer-subscription-id}"

Hostname resolution options​

Securely connect Astro to resources running in other VNets or on-premises through a resolving service.

Astronomer recommends using Domain Name System (DNS) forwarding as the most flexible and reliable solution. If you have a small number of records and immutable IP addresses, the Astronomer support team can create a Private zone with DNS records, pointed to your resources.

Use Domain Name System (DNS) forwarding to allow Astro to resolve DNS queries for resources running in other VPCs or on-premises. You have access to internal resources through private names. All changes in zone will be available for Astro environment immediatelly.

To use this solution, make sure Astro can connect to the DNS server using a VNet or VHub peering connection and then submit a request to Astronomer support. With your request, include the following information:

  • The domain name for forwarding requests
  • The IP address of the DNS server where requests are forwarded

(Optional) Create an Airflow connection to confirm connectivity​

After Astronomer support confirms that DNS forwarding was successfully set up, you can confirm that it works by creating an Airflow connection to a resource running in a VPC or on-premises. See Managing Connections.

See Also​

Was this page helpful?