Getting Started#

Requirements#

Dagster Cloud requires that your Dagster application code be written using version 0.13.2 or higher of Dagster.

Architectural overview#

The agent model#

Dagster Cloud uses an agent that runs in your infrastructure to ensure that your code, data, and secrets remain private. The agent is responsible for executing your code in your infrastructure. The agent streams metadata about code execution over HTTPS to Dagster Cloud’s agent API. For details, please see The Dagster Cloud Architecture.

Dagster Cloud Architecture Diagram (Summary)

Code locations#

Each Dagster Cloud agent can load user code from arbitrarily many code locations. You'll use the dagster-cloud CLI tool to tell Dagster Cloud about code locations that are available to your agent.

Multiple deployments#

Dagster Cloud deployments allow you to execute Dagster application code in separate compute environments with separately managed permissions.

Execution for each deployment is managed by a separate agent, so if you want to use multiple deployments, you will need to run one instance of the agent per deployment.

Your default deployment (present when your Dagster Cloud organization is first created) is prod.

Choosing an agent implementation#

The local agent#

The local agent can run directly on bare metal (such as your laptop), on a VM, or in a container. The local agent executes user code in isolated OS processes. Its ability to scale is limited by the capacity of the node on which it runs. This agent is appropriate for test or for deployments whose load is known to be less than the capacity of a single node. To use the local agent, see Setting up the Local Agent.

The Docker agent#

The Docker agent executes user code in Docker containers on the machine in which the agent runs. It can run in any environment in which Docker is installed, including bare metal (such as your laptop), a VM, or in a container. Using the Docker agent allows you to take advantage of automated CI/CD with our GitHub Action, as well as run jobs in multiple code locations using Python environments. To use the Docker agent, see Setting up the Docker Agent.

The Kubernetes agent#

The Kubernetes agent is intended to run as a Kubernetes Deployment. The Kubernetes agent executes user code in Kubernetes Jobs and Services. Its ability to scale is limited by the capacity of your Kubernetes cluster. This agent is appropriate for scaled production deployments or for organizations that are already standardized on Kubernetes. To use the Kubernetes agent, see Setting up the Kubernetes Agent.

The ECS agent#

The ECS agent executes user code on AWS Fargate. The agent scales without a need to manage your own servers. This agent is appropriate for scaled production deployments, for organizations that are already standardized on ECS, and for teams that don't want to manage their own infrastructure. To use the ECS agent, see Setting up the ECS Agent.