Dagster Cloud CI/CD GitHub Action#

The Dagster Cloud CI/CD GitHub Action lets you automatically update Dagster Cloud code locations when pipeline code is updated. The action builds a Docker image, pushes it to a Docker repository, and uses the Dagster Cloud API to tell your agent to add the built image to your workspace.

Quickstart Template Repo#

We provide a quickstart template repo which you can use to get CI/CD for your Cloud instance up and running quickly.

Usage#

To use the Dagster Cloud CI/CD GitHub Action, you will first need to create a new GitHub Actions Workflow. The job must first clone your Git repository using the actions/checkout action, and then enable access to your Docker registry. The Workflow included in the quickstart template already has this workflow set up for you.

Next, you must set up a locations.yaml file which describes each of the Dagster Cloud repo locations to be built and updated. If this locations.yaml file is not located at the repo root, it must be specified with the location-file input.

Example Job#

This example uses the docker/login-action action to set up Docker registry access. ECR users may want to use the aws-actions/amazon-ecr-login action instead. To speed up Docker builds, you may also use the satackey/action-docker-layer-caching action.

on:
  push:
    branches:
      - main

jobs:
  dagster-cloud:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v2

      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build images & update Dagster Cloud
        uses: dagster-io/dagster-cloud-cicd-action/deploy@v0.2.4
        with:
          dagit-url: https://hooli.dagster.cloud/prod
          api-token: ${{ secrets.DAGSTER_AGENT_TOKEN }}

Example locations.yaml#

This locations file indicates that two locations, foo and bar, should be built. These locations have Dockerfiles located at /foo_src/Dockerfile and /bar_src/Dockerfile, and are pushed to the dagster-io/foo and dagster-io/bar registries, respectively.

locations:
  # Location name
  foo:
    # Path to build directory, which must contain a Dockerfile or
    # requirements.txt file, relative to the locations.yaml folder
    build: ./foo_src

    # The base Docker image to use, if providing only a requirements.txt
    # file and no Dockerfile
    base_image: python:3.8-slim

    # Docker registry to push the built Docker image to
    registry: dagster-io/foo

    # Python file containing the job repo
    # Can alternatively supply package_name, as below
    python_file: repo.py

  bar:
    build: ./bar_src
    registry: dagster-io/bar
    package_name: bar

Customization#

Inputs#

NameDescription
dagit-url(Required) URL to your Dagster Cloud instance, including the deployment path.
api-token(Required) Dagster Cloud Agent API token.
location-filePath to the locations.yaml file defining the repo locations to update. Defaults to /locations.yaml in the repo root.
image-tagTag for the built Docker images, defaults to the first 6 chars of git hash.
parallelWhether to build and push Docker images in parallel. Defaults to true.

locations.yaml Properties#

Each location specified in the locations.yaml can have the following properties:

NameDescription
build(Required) Path to the build directory relative to the locations.yaml's folder. Build directory must contain a Dockerfile or requirements.txt file.
registry(Required) Docker registry to push the built Docker image to.
package_namePython module containing the Dagster Repository. Can alternatively use python_file below.
python_filePython file containing the Dagster Repository. Can alternatively use package_name above.
base_imageIf the build directory only contains a requirements.txt file and no Dockerfile, specifies the base Docker image to use to build the code location.
targetIf providing a multistage Dockerfile, can be used to specify the target stage to build.