Dagster Cloud CI/CD GitHub Action#

The Dagster Cloud CI/CD GitHub Action lets you automatically update Dagster Cloud code locations when pipeline code is updated. The action builds a Docker image, pushes it to a Docker repository, and uses the Dagster Cloud API to tell your agent to add the built image to your workspace.

Quickstart Template Repo#

We provide a quickstart template repo which you can use to get CI/CD for your Cloud instance up and running quickly.

Usage#

To use the Dagster Cloud CI/CD GitHub Action, you will first need to create a new GitHub Actions Workflow. The job must first clone your Git repository using the actions/checkout action, and then enable access to your Docker registry. The Workflow included in the quickstart template already has this workflow set up for you.

Next, you must set up a locations.yaml file which describes each of the Dagster Cloud repo locations to be built and updated. If this locations.yaml file is not located at the repo root, it must be specified with the location-file input.

Example Job#

This example uses the docker/login-action action to set up Docker registry access. ECR users may want to use the aws-actions/amazon-ecr-login action instead. To speed up Docker builds, you may also use the satackey/action-docker-layer-caching action.

on:
  push:
    branches:
      - main

jobs:
  dagster-cloud:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v2

      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build images & update Dagster Cloud
        uses: dagster-io/dagster-cloud-cicd-action/deploy@v0.2.4
        with:
          dagit-url: https://hooli.dagster.cloud/prod
          api-token: ${{ secrets.DAGSTER_AGENT_TOKEN }}

Example `locations.yaml`#

This locations file indicates that two locations, foo and bar, should be built. These locations have Dockerfiles located at /foo_src/Dockerfile and /bar_src/Dockerfile, and are pushed to the dagster-io/foo and dagster-io/bar registries, respectively.

locations:
  # Location name
  foo:
    # Path to build directory, which must contain a Dockerfile or
    # requirements.txt file, relative to the locations.yaml folder
    build: ./foo_src

    # The base Docker image to use, if providing only a requirements.txt
    # file and no Dockerfile
    base_image: python:3.8-slim

    # Docker registry to push the built Docker image to
    registry: dagster-io/foo

    # Python file containing the job repo
    # Can alternatively supply package_name, as below
    python_file: repo.py

  bar:
    build: ./bar_src
    registry: dagster-io/bar
    package_name: bar

Customization#

Inputs#

Name	Description
`dagit-url`	(Required) URL to your Dagster Cloud instance, including the deployment path.
`api-token`	(Required) Dagster Cloud Agent API token.
`location-file`	Path to the `locations.yaml` file defining the repo locations to update. Defaults to `/locations.yaml` in the repo root.
`image-tag`	Tag for the built Docker images, defaults to the first 6 chars of git hash.
`parallel`	Whether to build and push Docker images in parallel. Defaults to `true`.

`locations.yaml` Properties#

Each location specified in the locations.yaml can have the following properties:

Name	Description
`build`	(Required) Path to the build directory relative to the `locations.yaml`'s folder. Build directory must contain a `Dockerfile` or `requirements.txt` file.
`registry`	(Required) Docker registry to push the built Docker image to.
`package_name`	Python module containing the Dagster Repository. Can alternatively use `python_file` below.
`python_file`	Python file containing the Dagster Repository. Can alternatively use `package_name` above.
`base_image`	If the build directory only contains a `requirements.txt` file and no `Dockerfile`, specifies the base Docker image to use to build the code location.
`target`	If providing a multistage `Dockerfile`, can be used to specify the target stage to build.