Skip to main content

Authorize an Astro Deployment to cloud resources using workload identity

When you create an Airflow connection from a Deployment to access cloud resources, Airflow uses your connection details to access those services. You can add credentials to your Airflow connections to authenticate, but it can be risky to add secrets like passwords to your Airflow environment.

To avoid adding secrets to your Airflow connection, you can directly authorize your Astro Deployment to access AWS or GCP cloud services using workload identity. Astronomer recommends using a workload identity in most cases to improve security and avoid managing credentials across your Deployments. If you have less strict security requirements, you can still use any of the methods described in Airflow connection guides to manage your connection authorization.

This guide explains how to authorize your Deployment to a cloud using workload identity. For each Deployment, you will:

  • Authorize your Deployment to your cloud services.
  • Create an Airflow connection to access your cloud services.
tip

Watch the Astro Academy Customer Workload Managed Identity Learning Byte video to learn more about managed identities and how to set up passwordless authentication for GCP.

Prerequisites​

The Astro cluster running your Deployment must be connected to your cloud's network. See Networking overview.

What is workload identity?​

A workload identity is a Kubernetes service account that provides an identity to your Deployment. The Deployment can use this identity to authenticate to a cloud's API server, and the cloud can use this identity to authorize the Deployment to access different resources.

Setup​

Attach an IAM role to your Deployment​

This feature is available only on Astro Hosted.

You can attach an AWS IAM role to your Deployment to grant the Deployment all of the role's permissions.

Using IAM roles provides the greatest amount of flexibility for authorizing Deployments to your cloud. For example, you can use existing IAM roles on new Deployments, or your can attach a single IAM role to multiple Deployments that all require the same level of access to your cloud.

Prerequisites​

  • Minimum Astro Runtime version:
    • 9.15.0
    • 10.9.0
    • 11.5.0
  • A new or existing IAM role in your data sources with the required permissions you want your Deployment to have.
  • If using AWS CloudShell, the required CLIs are enabled by default.
  • If you use a local terminal, the following CLIs are required:

Step 1: Authorize the Deployment to your IAM role​

To authorize your Deployment, create an IAM role to assign as your Deployment's workload identity:

  1. Create an IAM role to delegate permissions to in an AWS service. Grant the role any permission that the Deployment will need in your AWS account. Copy the IAM role ARN to use later in this setup.
  2. In the Astro UI, select your Deployment and then click Details. In the Advanced section, click Edit.
  3. In the Workload Identity menu, select Customer Managed Identity.
  4. Enter your IAM role ARN when prompted, then copy and run the provided CLI command. Click Save Configuration to save the IAM role as a selectable configuration.
  5. Click Update Deployment to apply the selected IAM role to the Deployment.
  6. (Optional) Repeat these steps for each Astro Deployment that needs to access your AWS resources. Or, you can edit the <DeploymentNamespace> value in Condition when setting up the Workload Identity for one of the following scenarios to apply to multiple Deployments.
Specify Kubernetes service accounts

Available for both Standard and Dedicated clusters. If your organization does not allow you to use a wildcards in your IAM Trust Policies, change the <DeploymentNamespace> value in Condition to specify the Kubernetes service accounts. The following shows an example:

{
    "Condition": {
        "StringLike": {
            "<clusterOIDCIssuerUrl>:aud": "sts.amazonaws.com",
            "<clusterOIDCIssuerUrl>:sub": "system:serviceaccount:<DeploymentNamespace>:<DeploymentNamespace>-kpo"
            "<clusterOIDCIssuerUrl>:sub": "system:serviceaccount:<DeploymentNamespace>:<DeploymentNamespace>-dag-processor-serviceaccount"
            "<clusterOIDCIssuerUrl>:sub": "system:serviceaccount:<DeploymentNamespace>:<DeploymentNamespace>-scheduler-serviceaccount"
            "<clusterOIDCIssuerUrl>:sub": "system:serviceaccount:<DeploymentNamespace>:<DeploymentNamespace>-triggerer-serviceaccount"
            "<clusterOIDCIssuerUrl>:sub": "system:serviceaccount:<DeploymentNamespace>:<DeploymentNamespace>-webserver-serviceaccount"
            "<clusterOIDCIssuerUrl>:sub": "system:serviceaccount:<DeploymentNamespace>:<DeploymentNamespace>-worker-serviceaccount"
        }
    }
}
Dedicated clusters only: Share or re-use a managed identity using a wildcard

If you want to share or re-use the same customer managed identity on static or ephemeral Deployments for dedicated clusters, without having to update your Trust Policy in your AWS account for every net new Deployment, change the <DeploymentNamespace> value in Condition to include a wildcard. You should only use a wildcard in dedicated clusters for security purposes. The following shows an example:

{
    "Condition": {
        "StringLike": {
            "<clusterOIDCIssuerUrl>:aud": "sts.amazonaws.com",
            "<clusterOIDCIssuerUrl>:sub": "system:serviceaccount:*:*"
        }
    }
}

Step 2: Create an Airflow connection​

Now that your Deployment is authorized, you can connect it to your cloud using an Airflow connection. Create an Amazon Web Services connection in either the Astro UI or the Airflow UI for your Deployment and specify the following fields:

  • Connection Id: Enter a name for the connection.

If you don't see Amazon Web Services as a connection type in the Airflow UI, ensure you have installed its provider package in your Astro project's requirements.txt file. See Use Provider in the Astronomer Registry for the latest package.

tip

If you use a mix of strategies for managing connections and define the same connection in multiple ways, Airflow uses the following order of precedence:

  • Secrets Backend
  • Environment Manager
  • Environment Variables
  • Airflow UI using the Airflow metadata database

Alternative setup: Authorize your Deployment with AWS IAM roles​

Step 1: Authorize the Deployment in your cloud​

To grant a Deployment access to a service that is running in an AWS account not managed by Astronomer, use AWS IAM roles to authorize your Deployment's workload identity. IAM roles on AWS are often used to manage the level of access a specific user, object, or group of users has to a resource, such as Amazon S3 buckets, Redshift instances, and secrets backends.

To authorize your Deployment, create an IAM role that is assumed by the Deployment's workload identity:

  1. In the Astro UI, select your Deployment and then click Details. Copy the Deployment's Workload Identity.

  2. In the AWS account that contains your AWS service, create an IAM role. See Creating a role to delegate permissions to an AWS service.

  3. In the AWS Management Console, go to the Identity and Access Management (IAM) dashboard.

  4. Click Roles and in the Role name column, select the role you created in Step 2.

  5. Click Trust relationships.

  6. Click Edit trust policy and paste the workload identity you copied from Step 1 in the trust policy. Your policy should look like the following:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": ["<workload-identity-role>"]
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }

  7. Click Update policy.

Repeat these steps for each Astro Deployment that needs to access your AWS resources.

Step 2: Create an Airflow connection​

Now that your Deployment is authorized, you can connect it to your cloud using an Airflow connection. Either create an Amazon Web Services connection in the Astro UI or the Airflow UI for your Deployment and specify the following fields:

  • Connection Id: Enter a name for the connection.

  • Extra:

    {
      "role_arn": "<your-role-arn>",
      "region_name": "<your-region>"
    }

If you don't see Amazon Web Services as a connection type in the Airflow UI, ensure you have installed its provider package in your Astro project's requirements.txt file. See Use Provider in the Astronomer Registry for the latest package.

tip

If you use a mix of strategies for managing connections, if you define the same connection in multiple ways, Airflow uses the following order of precedence:

  • Secrets Backend
  • Environment Manager
  • Environment Variables
  • Airflow UI using the Airflow metadata database

See also​

Was this page helpful?