Skip to main content

Troubleshoot a local Airflow environment

Use the following topics to resolve common issues with running an Astro project in a local Airflow environment.

Troubleshoot KubernetesPodOperator issues​

View local Kubernetes logs to troubleshoot issues with Pods that are created by the KubernetesPodOPerator. See Test and Troubleshoot the KubernetesPodOperator Locally.

Troubleshoot dependency errors​

When dependency errors occur, the error message that is returned often doesn't contain enough information to help you resolve the error. To retrieve additional error information, you can review individual operating system or python package dependencies inside your local Docker containers.

For example, if your packages.txt file contains several packages and you receive build errors after running astro dev start, you can enter the container and install the packages manually to review additional information about the errors.

  1. Open your Astro project packages.txt file and remove the references to the packages that are returning error messages.

  2. Run the following command to build your Astro project into a Docker image and start a local Docker container for each Airflow component:

    astro dev start

  3. Run the following command to open a bash terminal in your scheduler container:

    astro dev bash --scheduler

  4. In the bash terminal for your container, run the following command to install a package and review any error messages that are returned:

    apt-get install <package-name>

    For example, to install the GNU Compiler Collection (GCC) compiler, you would run:

    apt-get install gcc

  5. Open your Astro project packages.txt file and add the package references you removed in Step 1 individually until you find the package that is the source of the error.

New DAGs aren't visible in the Airflow UI​

Make sure that no DAGs have duplicate dag_ids. When two DAGs use the same dag_id, the newest DAG won't appear in the Airflow UI and you won't receive an error message.

By default, the Airflow scheduler scans the dags directory of your Astro project for new files every 300 seconds (5 minutes). For this reason, it might take a few minutes for new DAGs to appear in the Airflow UI. Changes to existing DAGs appear immediately.

To have the scheduler check for new DAGs more frequently, you can set the AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL environment variable to less than 300 seconds. If you have less than 200 DAGs in a Deployment, it's safe to set AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL to 30 (30 seconds). See Set environment variables for how to set this on Astro.

In Astro Runtime 7.0 and later, the Airflow UI Code page includes a Parsed at value which shows when a DAG was last parsed. This value can help you determine when a DAG was last rendered in the Airflow UI. To view the Parsed at value in the Airflow UI, click DAGs, select a DAG, and then click Code. The Parsed at value appears at the top of the DAG code pane.

DAGs running slowly​

If your Astro project contains many DAGs or tasks, then you might experience performance issues in your local Airflow environment.

To improve the performance of your environment, you can:

  • Adjust CPU and memory resource allocation in your Docker configuration. Be aware that increasing Docker resource allocation might decrease the performance of your computer.
  • Modify Airflow-level environment variables, including concurrency and parallelism. See Scaling out Airflow.

Generating DAGs dynamically can also decrease the performance of your local Airflow environment, though it's a common authoring pattern for advanced use cases. For more information, see Dynamically Generating DAGs in Airflow. If your DAGs continue to run slowly and you can't scale Docker or Airflow any further, Astronomer recommends pushing your project to a Deployment on Astro that's dedicated to testing.

tip

If you don't have enough Docker resources allocated to your local Airflow environment, you might see tasks fail and exit with this error:

Task exited with return code Negsignal.SIGKILL

If you see this error, increase the CPU and memory allocated to Docker. If you're using Docker Desktop, you can do this by opening Docker Desktop and going to Preferences > Resources > Advanced. See Change Docker Desktop preferences on Mac.

If you are using Podman, you can run podman machine set --cpus 4 --memory 4096. See Podman commands reference for more details.

Astro project won't load after running astro dev start​

If you're running the Astro CLI on a Mac computer that's built with the Apple M1 chip, your Astro project might take more than 5 mins to start after running astro dev start. This is a current limitation of Astro Runtime and the Astro CLI.

If your project won't load, it might also be because your webserver or scheduler is unhealthy. In this case, you might need to debug your containers.

  1. After running astro dev start, retrieve a list of running containers by running astro dev ps.

  2. If the webserver and scheduler containers exist but are unhealthy, check their logs by running:

    $ astro dev logs --webserver
    $ astro dev logs --scheduler

  3. (Optional) Run the following command to prune all unused Docker objects including volumes and free disk space:

    docker system prune --volumes

    See docker system prune for more information about pruning. These logs should help you understand why your webserver or scheduler is unhealthy. Possible reasons why these containers might be unhealthy include:

  • Not enough Docker resources.
  • A failed Airflow or Astro Runtime version upgrade.
  • Misconfigured Dockerfile or Docker override file.
  • Misconfigured Airflow settings including packages.txt or requirements.txt.

Ports are not available for my local Airflow webserver​

By default, the Astro CLI uses port 8080 for the Airflow webserver and port 5432 for the Airflow metadata database in a local Airflow environment. If these ports are already in use on your local computer, an error message similar to the following appears:

Error: error building, (re)creating or starting project containers: Error response from daemon: Ports are not available: exposing port TCP 0.0.0.0:5432 → 0.0.0.0:0: listen tcp 0.0.0.0:5432: bind: address already in use

To resolve a port availability error, you have the following options:

  • Stop all running Docker containers and restart your local environment using astro dev restart.
  • Change the default ports for these components. For example, you can use astro config set webserver.port 8081 for the webserver and astro config set postgres.port 5433 for Postgres. See Configure CLI for all available configurations.

Stop all running Docker containers​

  1. Run docker ps to identify the Docker containers running on your computer.

  2. Copy the values in the CONTAINER ID column.

  3. Select one of the following options:

    • Run docker stop <container_id> to stop a specific Docker container. Replace <container_id> with one of the values you copied in step 2.
    • Run docker stop $(docker ps -q) to stop all running Docker containers.

Change the default port assignment​

If port 8080 or 5432 are in use on your machine by other services, the Airflow webserver and metadata database won't be able to start. To run these components on different ports, run the following commands in your Astro project:

astro config set webserver.port <available-port>
astro config set postgres.port <available-port>

For example, to use 8081 for your webserver port and 5435 for your database port, you would run the following commands:

astro config set webserver.port 8081
astro config set postgres.port 5435

Was this page helpful?