Astro CI/CD templates for GitLab

Use the following CI/CD templates to automate deploys to Astro from a GitLab repository.

Read the following sections to choose the right template for your project. The templates for GitLab include the image deploy templates and DAG deploy templates.

If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation. If you want your CI/CD process to automatically decide which deploy strategy to choose, see DAG deploy templates.

To learn more about CI/CD on Astro, see Choose a CI/CD strategy.

Prerequisites

• An Astro project hosted in a GitLab repository.
• An Astro Deployment.
• A Deployment API token, Workspace API token, or Organization API token.

Each CI/CD template implementation might have additional requirements.

Image deploy templates

Single branch

Use this template to push code to from a GitLab repository to Astro.

1. Set the following environment variables in your GitLab project:

   • ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
   • DEPLOYMENT_ID: The ID of your Astro Deployment. You can copy the ID from your Deployment's home page in the Astro UI.

   Astronomer recommends that you always mask your API token to prevent it from being accessible in plain text. You can also set the API token as an external secret for an extra layer of security.

2. Go to Build > Pipeline Editor and commit the following:

astro_deploy:
  stage: deploy
  image: docker:latest
  services:
    - docker:dind

  variables:
      ASTRO_API_TOKEN: ${ASTRO_API_TOKEN}
      DEPLOYMENT_ID: ${DEPLOYMENT_ID}

  before_script:
    - apk add --update curl && rm -rf /var/cache/apk/*
    - apk add bash
  script:
    - (curl -sSL install.astronomer.io | bash -s)
    - astro deploy -f $DEPLOYMENT_ID
  only:
    - main

Multiple branch

Use this template to push code to a development and a production Deployment in Astro based on your GitLab project's branch name.

1. Set the following environment variables in your GitLab project:

   • PROD_ASTRO_API_TOKEN: The value of your production Workspace or Organization API token.
   • PROD_DEPLOYMENT_ID: The ID of your Astro Deployment. You can copy the ID from your production Deployment's page in the Astro UI.
   • DEV_ASTRO_API_TOKEN: The value of your development Workspace or Organization API token.
   • DEV_DEPLOYMENT_ID: The ID of your Astro Deployment. You can copy the ID from your development Deployment's page in the Astro UI.

   Astronomer recommends that you always mask your API token to prevent it from being accessible in plain text. You can also set the API token as an external secret for an extra layer of security.

tip

When you create a CI/CD variable that will be used in multiple branches, you might want to protect the variable so that it can only be accessed from the relevant branches.

2. Go to the Editor option in your project's CI/CD section and commit the following:

astro_deploy_dev:
  stage: deploy
  image: docker:latest
  services:
    - docker:dind
  variables:
      ASTRO_API_TOKEN: ${DEV_ASTRO_API_TOKEN}
      DEPLOYMENT_ID: ${DEV_DEPLOYMENT_ID}
  before_script:
    - apk add --update curl && rm -rf /var/cache/apk/*
    - apk add bash
  script:
    - (curl -sSL install.astronomer.io | bash -s)
    - astro deploy -f $DEPLOYMENT_ID
  only:
    - dev

astro_deploy_prod:
  stage: deploy
  image: docker:latest
  services:
    - docker:dind
  variables:
      ASTRO_API_TOKEN: ${PROD_ASTRO_API_TOKEN}
      DEPLOYMENT_ID: ${PROD_DEPLOYMENT_ID}
  before_script:
    - apk add --update curl && rm -rf /var/cache/apk/*
    - apk add bash
    - apk add jq
  script:
    - (curl -sSL install.astronomer.io | bash -s)
    - astro deploy -f $DEPLOYMENT_ID
  only:
    - main

DAG deploy templates

The DAG deploy template uses the --dags flag in the Astro CLI to push DAG changes to Astro. These CI/CD pipelines deploy your DAGs only when files in your dags folder are modified, and they deploy the rest of your Astro project as a Docker image when other files or directories are modified. For more information about the benefits of this workflow, see Deploy DAGs only.

info

If you stage multiple commits to DAG files and push them all at once to your remote branch, the template only deploys DAG code changes from the most recent commit. It will miss any code changes made in previous commits.

To avoid this, either push commits individually or configure your repository to Squash commits for pull requests that merge multiple commits simultaneously.

Single branch implementation

Use this template to push code to from a GitLab repository to Astro.

1. Set the following environment variables in your GitLab project:

   • ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
   • DEPLOYMENT_ID: The ID of your Astro Deployment. You can copy the ID from your Deployment's page in the Astro UI.

   Astronomer recommends that you always mask your API token to prevent it from being accessible in plain text. You can also set the API token as an external secret for an extra layer of security.

2. Go to the Editor option in your project's CI/CD section and commit the following:

   astro_smart_deploy:
     stage: deploy
     image: docker:latest
     services:
       - docker:dind
     variables:
       ASTRO_API_TOKEN: ${ASTRO_API_TOKEN}
       DAG_FOLDER: "dags"
       DEPLOYMENT_ID: ${DEPLOYMENT_ID}
     before_script:
       - apk add --update curl && rm -rf /var/cache/apk/*
       - apk add git
       - apk add bash
     script:
       - (curl -sSL install.astronomer.io | bash -s)
       - files=$(git diff --name-only $(git rev-parse HEAD~1) -- .)
       - dags_only=1
       - echo "$DAG_FOLDER"
       - echo "$files"
       - for file in $files; do
       -   echo "$file"
       -   if [[ "$file" != "$DAG_FOLDER"* ]]; then
       -     echo "$file is not a dag, triggering a full image build"
       -     dags_only=0
       -     break
       -   else
       -     echo "just a DAG"
       -   fi
       - done
       - if [[ $dags_only == 1 ]]; then
       -   echo "doing dag-only deploy"
       -   astro deploy --dags $DEPLOYMENT_ID
       - elif [[ $dags_only == 0 ]]; then
       -   echo "doing image deploy"
       -   astro deploy -f $DEPLOYMENT_ID
       - fi
     only:
       - main