GitHub iconSlack iconTwitter iconStack Overflow icon

Setting up the ECS Agent#

Requirements#

ECS is a managed container service. To use the ECS cluster, you must first configure several resources in your AWS account. This guide provides instructions for how to spin up an ECS agent in a new VPC using CloudFormation.

Provisioning Using CloudFormation#

You can use our CloudFormation template to quickly spin up the ECS agent stack. For more information, see the AWS documentation on CloudFormation.

The CloudFormation template in this guide sets up an ECS agent from scratch, creating a new VPC and ECS cluster for the agent to run in. The template code can be found here. The Advanced ECS Setup guide includes additional instructions for deploying a Dagster Cloud agent in an existing VPC.

Prerequisites#

  1. Your Dagster Cloud organization name and deployment name.
  2. A Dagster Cloud agent token (See managing user and agent tokens).
  3. Your AWS account must be under its VPC quota limit in the region in which you're spinning up the agent. By default, AWS allows 5 VPCs per region - if you're already at your limit, see the AWS docs for information on how to request a quota increase.
  4. ECS requires a special IAM role that is automatically created for you the first time you create an ECS cluster in the ECS console, but is not created when creating ECS clusters using CloudFormation. This can cause the CloudFormation template to fail if setting up your Dagster Cloud agent is the first time you've ever created an ECS cluster in your account. If you've never created an ECS cluster before, either go through the ECS first run wizard to create your first ECS cluster before running the CloudFormation template, or follow the instructions in the AWS docs to create the ECS service-linked role using the AWS CLI.
Example Configuration for the ECS Agent CloudFormation Template

Installing the Stack#

Press the "Launch Stack" button below to install the CloudFormation stack in your AWS account.

Creating the CloudFormation stack may take a few minutes; refreshing the AWS console Stacks page will show its status.

After the CloudFormation template installs the stack, if you navigate to the Agents tab on the Status page in Dagster Cloud, you should now see one running agent.

Instance Status

Next steps#

If you set this agent up as part of the Dagster Cloud tutorial, the next step of the tutorial shows you how to add an example code location using your new agent.

To use your agent to load your own Dagster code, see the Adding Code to Dagster Cloud guide.

See Advanced ECS Setup for information about deploying the ECS Agent within an existing VPC.