Astronomer Certified Image Architecture

danger

No versions of Astronomer Certified (AC) are currently supported by Astronomer. Astronomer stopped releasing new versions of AC with the release of Apache Airflow 2.4. Astronomer recommends creating all new Deployments with Astro Runtime, as well as migrating existing Deployments from AC to Astro Runtime as soon as your organization is ready. See Migrate to Runtime and Runtime image architecture.

Overview

The Astronomer Certified Docker image for Apache Airflow extends the community-developed Airflow image in a way that makes running Airflow more secure, reliable, and extensible. It is the default image for Airflow Deployments on Astronomer.

This guide provides reference information for the building blocks of Astronomer Certified, as well as information on its release and distribution.

Distribution

Astronomer Certified is distributed both as a Python wheel and a Debian-based Docker image. These distributions vary slightly in scope and dependencies.

The Python wheel and Docker image are functionally identical to open source Apache Airflow. Additionally, they both include performance and stability improvements, including critical bug fixes and security patches.

The Astronomer Certified Docker image is built from the Python wheel and incorporates additional logic that makes it easier for users to both get started and run Airflow at scale. This includes:

• A robust testing suite that covers performance benchmarking and end-to-end functionality and upgrade testing for certified environments.
• A built-in directory of example DAGs that leverage Airflow's most powerful features.
• A collection of pre-installed Airflow provider packages.
• Full compatibility with the Astronomer Platform.

Every supported version of the Astronomer Certified Python wheel is available at pip.astronomer.io. The Dockerfiles for all supported Astronomer Certified images can be found in Astronomer's ap-airflow GitHub repository.

Image Requirements

Running Airflow with the Astronomer Certified Docker image requires specific versions for key system components.

• Python: 3.6, 3.7, 3.8, 3.9
• Database: PostgreSQL (11, 12), MySQL (5.7, 8.0+)
• System Distribution: Debian 10 (Buster)

Note: While the Astronomer Certified Python Wheel supports Python versions 3.6, 3.7, and 3.8, Astronomer Certified Docker images have been tested and built only with Python 3.7. These requirements are slightly different for running only the Python wheel. For the Python wheel, you can use:

• Python: 3.6, 3.7, 3.8, 3.9
• Database: PostgreSQL (9.6, 10, 11, 12, 13), MySQL (5.7, 8+), SQLite (3.15.0+)
• System Distribution: Debian 10 (Buster)

For more information on running a Python wheel installation of Astronomer Certified, read Install on a Virtual Machine.

Environment Variables

When an Airflow service is started, it checks a file for runtime environment variables. These are equivalent to values defined in Airflow's airflow.cfg file.

If you run the Astronomer Certified Docker image without the Astronomer platform, environment variables are defined in your Dockerfile. They can be overwritten with a runtime command, such as docker run.

If you're running the Astronomer Certified Docker image with the Astronomer platform, there are a few ways you can configure environment variables. For more information, read Environment Variables.

Astronomer Certified supports the same environment variables as Apache Airflow. For a list of all configurable environment variables, read the Apache Airflow documentation.

The following table lists the essential environment variables used when running Airflow with Astronomer Certified. Only environment variables explicitly defined in the Dockerfile are listed here; all other environment variables have the same default value as they have in OSS Airflow.

Variable	Description	Default Value
AIRFLOW_PIP_VERSION	The version of pip that is used to install Python packages and Airflow itself.	19.3.1
AIRFLOW_HOME	Filepath for your Airflow project directory.	usr/local/airflow
AIRFLOW__WEBSERVER__BASE_URL	The URL used to access the Airflow UI.	http://localhost:8080
ASTRONOMER_USER	The username for your Airflow user.	astro
ASTRONOMER_UID	The ID for your Airflow user.	5000
PIP_NO_CACHE_DIR	Specifies whether to maintain copies of source files when installing via pip.	True
PYTHON_MAJOR_MINOR_VERSION	The version of Python to use for Airflow.	3.9

Provider Packages

Starting in version 2.0.0, the Astronomer Certified image includes provider packages that are utilized in some background processes, as well as packages which are commonly used by the Airflow community. The following table contains version information for each provider package installed as part of Astronomer Certified:

Astronomer Certified	amazon	azure	celery	cncf.kubernetes	elasticsearch	ftp	google	http	imap	mysql	postgres	redis	slack	sqlite	ssh
2.0.0	1.0.0	1.1.0	1.0.0	1.0.1	1.0.4	1.0.0	1.0.0	1.0.0	1.0.0	1.0.0	1.0.0	1.0.0	1.0.0	1.0.0	1.0.0
2.0.2	1.3.0	1.3.0	1.0.1	1.2.0	1.0.4	1.0.1	2.2.0	1.1.1	1.0.1	1.1.0	1.0.1	1.0.1	3.0.0	1.0.2	1.3.0
2.1.0	1.4.0	2.0.0	1.0.1	1!1.2.1	1.0.4	1.1.0	3.0.0	2.0.0	1.0.1	1.1.0	1.0.2	1.0.1	3.0.0	1.0.2	1.3.0
2.1.1	1!2.0.0	1!3.0.0	1!2.0.0	1!2.0.0	1!2.0.1	1!2.0.0	1!4.0.0	1!2.0.0	1!2.0.0	1!2.0.0	1!2.0.0	1!2.0.0	1!4.0.0	1!2.0.0	1!2.0.0
2.1.3	1!2.1.0	1!3.1.0	1!2.0.0	1!2.0.2	1!2.0.2	1!2.0.0	1!5.0.0	1!2.0.0	1!2.0.0	1!2.1.0	1!2.0.0	1!2.0.0	1!4.0.0	1!2.0.0	1!2.1.0
2.1.4	1!2.2.0	1!3.1.1	1!2.1.0	1!2.0.2	1!2.0.3	1!2.0.1	1!5.1.0	1!2.0.1	1!2.0.1	1!2.1.1	1!2.2.0	1!2.0.1	1!4.0.1	1!2.0.1	1!2.1.1
2.2.0	1!2.2.0	1!3.2.0	1!2.1.0	1!2.0.3	1!2.0.3	1!2.0.1	1!6.0.0	1!2.0.1	1!2.0.1	1!2.1.1	1!2.3.0	1!2.0.1	1!4.1.0	1!2.0.1	1!2.2.0
2.2.1	1!2.2.0	1!3.2.0	1!2.1.0	1!2.0.3	1!2.0.3	1!2.0.1	1!6.0.0	1!2.0.1	1!2.0.1	1!2.1.1	1!2.3.0	1!2.0.1	1!4.1.0	1!2.0.1	1!2.2.0
2.2.2	1!2.3.0	1!3.3.0	1!2.1.0	1!2.1.0	1!2.1.0	1!2.0.1	1!6.1.0	1!2.0.1	1!2.0.1	1!2.1.1	1!2.3.0	1!2.0.1	1!4.1.0	1!2.0.1	1!2.3.0
2.2.3	1!2.3.0	1!3.3.0	1!2.1.0	1!2.1.0	1!2.1.0	1!2.0.1	1!6.1.0	1!2.0.1	1!2.0.1	1!2.1.1	1!2.3.0	1!2.0.1	1!4.1.0	1!2.0.1	1!2.3.0
2.2.4	1!3.0.0	1!3.6.0	1!2.1.0	1!3.0.2	1!2.2.0	1!2.0.1	1!6.4.0	1!2.0.3	1!2.2.0	1!2.2.0	1!3.0.0	1!2.0.1	1!4.2.0	1!2.1.0	1!2.4.0

System Dependencies

The Astronomer Certified Docker image includes a number of OS-level dependencies for running basic system processes. These dependencies can be installed in the Python Wheel as described in Install Packages.

• apt-utils
• build-essential
• curl
• build-essential
• default-libmysqlclient-dev
• libmariadb3
• freetds-bin
• gosu
• libffi6
• libffi-dev
• libkrb5-3
• libkrb5-dev
• libpq5
• libpq-dev
• libsasl2-2
• libsasl2-dev
• libsasl2-modules
• libssl1.1
• libssl-dev
• locales
• netcat
• rsync
• sasl2-bin
• sudo
• tini

Extras

Astronomer Certified includes a few packages that don't have a corresponding provider. These packages are used for basic system functions or optional Airflow functionality. The following list contains all extra packages built into Astronomer Certified by default:

• async: Provides asynchronous workers for Gunicorn
• password: Adds support for user password hashing
• statsd: Adds support for sending metrics to StatsD
• virtualenv: Adds support for running Python tasks in local virtual environments