Skip to main content

Astro CI/CD templates for CircleCI

CircleCI is a continuous integration and continuous delivery platform that can be used to implement DevOps practices. This document provides sample CI/CD templates to automate deploying Apache Airflow DAGs from a GitHub repository to Astro using CircleCI.

If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation. If your team builds custom Docker images, use the custom image implementation.

Refer to Template overview to see generic templates expressed as simple shell scripts or configure your own. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.

Prerequisites​

Image deploy templates​

Image deploy templates build a Docker image and push it to Astro whenever you update any file in your Astro project.

To automate code deploys to a Deployment using CircleCI for a single branch implementation, complete the following setup in a Git-based repository that hosts an Astro project:

Configuration requirements​

  • You have a main branch of an Astro project hosted in a single GitHub repository.
  • You have a production Deployment on Astro where you want to deploy your main GitHub branch.
  • You have a production CircleCI context that stores environment variables for your CI/CD workflows.

Implementation​

  1. Set the following environment variables in a CircleCI context:

    • ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
    • ASTRO_DEPLOYMENT_ID: The ID for your Deployment.

  2. Create a new YAML file in .circleci/config.yml that includes the following configuration:

    # Use the latest CircleCI pipeline process engine version.
    # See: https://circleci.com/docs/2.0/configuration-reference
    version: 2.1

    orbs:
      docker: circleci/docker@2.0.1
      github-cli: circleci/github-cli@2.0.0

    # Define a job to be invoked later in a workflow.
    # See: https://circleci.com/docs/2.0/configuration-reference/#jobs
    jobs:

      build_image_and_deploy:
        docker:
          - image: cimg/base:stable
        # Add steps to the job
        # See: https://circleci.com/docs/2.0/configuration-reference/#steps
        steps:
          - setup_remote_docker:
              version: 20.10.11
          - checkout
          - run:
              name: "Deploy to Astro"
              command: |
                curl -sSL install.astronomer.io | sudo bash -s
                astro deploy ${ASTRO_DEPLOYMENT_ID} -f

    # Invoke jobs with workflows
    # See: https://circleci.com/docs/2.0/configuration-reference/#workflows
    workflows:
      version: 2.1
      wf-build-and-deploy:
        jobs:
          - build_image_and_deploy:
              context:
                - <YOUR-CIRCLE-CI-CONTEXT>
              filters:
                branches:
                  only:
                    - <YOUR-BRANCH-NAME>

DAG deploy templates​

A DAG deploy template uses the --dags flag in the astro deploy command in the Astro CLI to push only DAGs to your Deployment.

This CI/CD pipeline deploys your DAGs to Astro when one or more files in your dags folder are modified. It deploys the rest of your Astro project as a Docker image when other files or directories are also modified. For more information about the benefits of this workflow, see Deploy DAGs only.

Configuration requirements​

For each Deployment that you use with DAG deploy templates, you must enable DAG deploys.

info

If you stage multiple commits to DAG files and push them all at once to your remote branch, the template only deploys DAG code changes from the most recent commit. It will miss any code changes made in previous commits.

To avoid this, either push commits individually or configure your repository to Squash commits for pull requests that merge multiple commits simultaneously.

Single branch implementation​

To automate code deploys to a Deployment using CircleCI, complete the following setup in a Git-based repository that hosts an Astro project:

  1. Set the following environment variables in a CircleCI context:

    • ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
    • ASTRO_DEPLOYMENT_ID: The ID for your Deployment.

  2. In your project repository, create a new YAML file in .circleci/config.yml that includes the following configuration:

    # Use the latest CircleCI pipeline process engine version.
    # See: https://circleci.com/docs/2.0/configuration-reference
    version: 2.1

    orbs:
      docker: circleci/docker@2.0.1
      github-cli: circleci/github-cli@2.0.0

    # Define a job to be invoked later in a workflow.
    # See: https://circleci.com/docs/2.0/configuration-reference/#jobs
    jobs:

      build_image_and_deploy:
        docker:
          - image: cimg/base:stable
        # Add steps to the job
        # See: https://circleci.com/docs/2.0/configuration-reference/#steps
        steps:
          - setup_remote_docker:
              version: 20.10.11
          - checkout
          - run:
              name: "Build image and deploy"
              command: |
                curl -sSL install.astronomer.io | sudo bash -s
                files=($(git diff-tree HEAD --name-only --no-commit-id))
                echo ${files}
                find="dags"
                if [[ ${files[*]} =~ (^|[[:space:]])"$find"($|[[:space:]]) && ${#files[@]} -eq 1 ]]; then
                  echo "only deploying dags"
                  astro deploy ${ASTRO_DEPLOYMENT_ID} --dags -f;
                else
                  echo "image deploy"
                  astro deploy ${ASTRO_DEPLOYMENT_ID} -f;
                fi

    # Invoke jobs with workflows
    # See: https://circleci.com/docs/2.0/configuration-reference/#workflows
    workflows:
      version: 2.1
      wf-build-and-deploy:
        jobs:
          - build_image_and_deploy:
              context:
                - <YOUR-CIRCLE-CI-CONTEXT>
              filters:
                branches:
                  only:
                    - <YOUR-BRANCH-NAME>

This script checks the diff between your current commit and the HEAD of your branch to which you are pushing the changes to. If the changes are only in dags then it executes a dag-only deploy. Otherwise, it executes an image-based deploy. Make sure to customize the script to use your specific branch and context.

You can customize this script to work for multiple branches as shown in the image-based multi-branch deploy template by creating separate job and workflow for each branch.

Was this page helpful?