Skip to main content

Configure a secrets backend

Apache Airflow variables and connections often contain sensitive information about your external systems that you need to keep in a secrets backend tool, which stores secrets in a secure and centralized location. Unlike other management strategies, such as using Environment Variables or working with connections and variables in the Airflow UI, secrets backends require a third-party secrets manager. This means that you can use a secrets manager administered by your organization for existing security protocols, or you need to choose and set up a secrets backend.

This document explains the available secrets backend integrations supported by Astro and how Airflow finds connections and variables if you use multiple strategies to manage them.

See Manage connections and variables to learn more about your available options and decide whether using a secrets backend complies with your organization's security requirements.

Available integrations​

Secrets backend integrations can be configured individually with each Astro Deployment by someone with Workspace Operator permissions.

Using secrets to set Airflow connections requires knowledge of how to generate Airflow connections in URI or JSON format. See Import and export Airflow connections and variables for guidance on how to export your connections and variables based on where they are stored.

Astro integrates with the following secrets backend tools:

How Airflow finds connections or variables​

tip

If you need to access your secrets backend from your local Airflow, you can mount your user credentials to a local Airflow environment. While this implementation is not recommended for Astro Deployments, it lets you quickly test pipelines with data hosted in your cloud. See Authenticate to cloud services.

If you configure a secrets backend on Astro, you can still continue to define Airflow variables and connections as environment variables, with the Astro Environment Manager or in the Airflow UI. The order of precedence for connections is:

  1. Secrets Backend
  2. Astro Environment Manager
  3. Environment Variables
  4. Airflow's metadata database (Airflow UI)

Was this page helpful?