Configuring and running a Kubernetes agent#

In this guide, you'll configure and run a Kubernetes agent. Kubernetes agents are used to launch your code in Kubernetes Jobs and Services.

Prerequisites#

To complete the steps in this guide, you'll need:

• Permissions in Dagster Cloud that allow you to manage agent tokens. Refer to the User permissions documentation for more info.

• A Kubernetes cluster into which you can deploy the agent. This can be a self-hosted Kubernetes cluster or a managed offering like Amazon EKS, Azure AKS, or Google GKE.

• Access to a container registry to which you can push images and from which pods in the Kubernetes cluster can pull images. This can be:

  • A self-hosted registry,
  • A public registry such as DockerHub, or
  • A managed offering such as Amazon ECR, Azure ACR, or Google GCR

• To have Helm installed. Refer to the Helm installation documentation for more info.

Step 1: Generate a Dagster Cloud agent token#

In this step, you'll generate a token for the Dagster Cloud agent. The Dagster Cloud agent will use this to authenticate to the agent API.

1. Sign in to your Dagster Cloud instance.
2. Click the user menu (your icon) > Organization Settings.
3. In the Organization Settings page, click the Tokens tab.
4. Click the + Create agent token button.
5. After the token has been created, click Reveal token.

Keep the token somewhere handy - you'll need it to complete the setup.

Step 2: Configure the agent#

In this step, you'll create a Kubernetes namespace for your Dagster Cloud resources. You'll also add the agent token to the Kubernetes cluster.

1. Create a Kubernetes namespace for your Dagster Cloud resources:

   kubectl create namespace <namespace>
   

   For example, if the namespace is dagster-cloud:

   kubectl create namespace dagster-cloud
   

2. Add the agent token you created in Step 1 as a secret in the Kubernetes cluster:

   kubectl create secret generic dagster-cloud-agent-token \
       --from-literal=DAGSTER_CLOUD_AGENT_TOKEN=<your_agent_token> \
       --namespace dagster-cloud
   

Step 3: Start the agent#

In this step, you'll spin up the agent with Helm.

1. Add the agent chart repository:

   helm repo add dagster-cloud https://dagster-io.github.io/helm-user-cloud
   helm repo update
   

2. Install the agent chart, specifying:

   • The namespace into which to install the chart
   • The Dagster Cloud deployment the agent will be responsible for. Note: When your Dagster Cloud organization is first created, the default deployment is prod:

   helm upgrade \
      --install user-cloud dagster-cloud/dagster-cloud-agent \
      --namespace dagster-cloud \
      --set dagsterCloud.deployment=prod
   

After the Helm chart is installed into the Kubernetes cluster, you can view the agent in Dagster Cloud.

In Dagster Cloud, navigate to the Status page and click the Agents tab. You should see the agent running in the Agent statuses section:

Next steps#

Now that you've got your agent running, what's next?

• If you're getting Dagster Cloud set up, the next step is to add a code location using the agent.

• If you're ready to load your Dagster code, refer to the Adding Code to Dagster Cloud guide for more info.

You can also further configure the jobs the agent runs, including adding secrets, mounting volumes, and setting resource limits.