Managing Deployments#

Dagster Cloud Deployments allow you to execute Dagster application code in separate compute environments and with separately managed permissions.

Execution for each deployment is managed by a separate agent, so if you want to use multiple deployments, you will need to run one instance of the agent per deployment.

The default deployment (present when your Dagster Cloud organization is first created) is prod.

Switching Deployments#

When viewing the Dagster Cloud web interface, you are able to toggle between any of your deployments through the deployment switcher drop-down located on the top bar.

Managing Deployments#

Users who have the Organization Admin role are able to create and delete existing deployments. If you have this permission, you will see a "Manage" option in the deployment switcher dropdown.

Deployment Switcher with the Manage Deployments Option

The deployment management interface allows you to create new deployments or delete existing deployments.

Deleting a Deployment#

To delete a deployment, click the "Delete" button and confirm the operation. Deleting deployments will delete all of their associated data, including their code locations, jobs, schedules, and sensors.

Creating a New Deployment#

To create a deployment, enter a name and then choose whether to begin the new deployment with an empty set of permissions, or to duplicate these permissions from an existing deployment. Deployments with empty permissions will only be able to be managed by Organization Admins until other users are granted Admin or Editor permissions.

Once a new deployment is created, permissions for users can be managed via the user settings page.

Deployment Settings#

Each of your Dagster Cloud deployments can be configured with these settings.

Run Queue Settings#

Name	Description	Default
`run_queue.max_concurrent_runs`	The maximum number of runs that will execute concurrently on this deployment. This can be configured up to 500. Reach out if you would like to increase that limit.	10
`run_queue.tag_concurrency_limits`	A set of concurrency limits that are applied to runs with particular tags.	[]

The schema for these fields matches the configuration in open source Dagster.

Run Monitoring Settings#

Name	Description	Default
`run_monitoring.start_timeout_seconds`	The number of seconds that Dagster Cloud will wait after a run is launched for the process or container to start executing. After the timeout the run will fail. This prevents runs from hanging in `STARTING` indefinitely when the process or container don't start.	300 (5 minutes)

Configuring the Settings#

Note: This requires that you have dagster-cloud 0.13.14 or later installed for running the CLI. You don't need to upgrade your Agent or Job code.

Create a file with the settings you'd like to configure.

# my-settings.yaml

run_queue:
  max_concurrent_runs: 10
  tag_concurrency_limits:
    - key: "special-runs"
      limit: 5

run_monitoring:
  start_timeout_seconds: 300

run_retries: # Automatically re-execute runs from failure
  max_retries: 0 # default number of retry attempts

Use the dagster-cloud CLI to upload the settings.

dagster-cloud deployment settings set-from-file my-settings.yaml

This will replace all of your configured settings. Any that are not specified will resort to their default values. You also use the CLI to read your current settings, including the default values.

dagster-cloud deployment settings get

These will soon be available to configure within Dagit.