Developing Dagster code with Dagster Cloud#

Because of Dagster Cloud's hybrid architecture, you can use Dagster Cloud anywhere you can run an agent. This includes your computer. One popular strategy for developing your code with Dagster Cloud is to run the Docker agent locally:

  1. Install the dagster-cloud package (with additional dependencies for running in Docker):
pip install "dagster-cloud[docker]"
  1. Choose a deployment that doesn't already have an agent running. We recommend creating a deployment specifically for development. In this example, we use the deployment staging.

  2. Generate the agent token that your Dagster Cloud agent will use to authenticate to the agent API (See Managing user and agent tokens).

  3. Run a Docker agent:

dagster-cloud agent run \
  --deployment staging \
  --agent-token "agent:hooli:lbpsyhtauqqfu24ae2brz2fsbxmjwr6y" \
  --user-code-launcher "dagster_cloud.workspace.docker.DockerUserCodeLauncher"

This will run a Docker agent on your local machine. If you already have code locations configured, your agent will pull their images and launch new containers from them. To change your code, rebuild your image.

Developing without rebuilding an image#

Rebuilding an image every time your code changes can lead to a slow development cycle. One way to shorten this loop is to mount your code as a volume in the container your Docker agent will spin up. For example:

dagster-cloud agent run \
  --deployment staging \
  --agent-token "agent:hooli:lbpsyhtauqqfu24ae2brz2fsbxmjwr6y" \
  --user-code-launcher "dagster_cloud.workspace.docker.DockerUserCodeLauncher" \
  --user-code-launcher-config '{"container_kwargs": {"volumes": {"$HOME/dagster/my_location": {"bind": "/app", "mode": "rw"}}}}'

This will mount the directories at $HOME/dagster/my_location into the /app directory inside any container your Docker agent launches.

Now, you won't need to wait to rebuild between each change. If your changes alter the shape or metadata of your graph, you should first click "Re-deploy" on the Workspace tab in Dagster Cloud. If your changes only alter business logic, they will be immediately available without clicking "Re-deploy."

Customizing your environment#

You can bind environment variables from your local machine to the containers your Docker agent launches. This is useful if you use environment variables to configure your jobs and you want to pass a different set of configuraton during development than you do in production. For example, your production deployment might connect to your production database while your development deployment might connect to a database running on your computer.

To pass environment variables:

dagster-cloud agent run \
  --deployment staging \
  --agent-token "agent:hooli:lbpsyhtauqqfu24ae2brz2fsbxmjwr6y" \
  --user-code-launcher "dagster_cloud.workspace.docker.DockerUserCodeLauncher" \
  --user-code-launcher-config '{"env_vars": ["MY_VAR", "MY_VAR_2"]}'

This will bind $MY_VAR and $MY_VAR_2 from your computer into every container your Docker agent runs.