Ask AI

Configuring and running a local agent#

This guide is applicable to Dagster Cloud.

In this guide, you'll configure and run a local agent. Local agents are used to launch user code in operating system subprocesses.

Considering running the agent in production? Running the local agent in production can be a good choice if:

  • Load is relatively light (guaranteed to fit on a single node)
  • Jobs aren’t computationally intensive and don’t use much memory
  • A Kubernetes cluster is prohibitively difficult to set up
  • You can restart the agent when you update your code

Keep in mind that the local agent’s ability to run jobs is limited by the capacity of the single node on which it’s running.

If you’re running the local agent in production, make sure you’ve set up a supervisor to automatically restart the agent process if it crashes. You’ll also want a system in place to alert you if the VM or container dies, or to automatically restart it.


Prerequisites#

To complete the steps in this guide, you'll need:

  • Permissions in Dagster Cloud that allow you to manage agent tokens. Refer to the User permissions documentation for more info.

  • To install the dagster-cloud CLI. This should be in the same environment where the agent will run.

    Note: Your Dagster application code and its Python and system requirements must also be installed in this environment. We recommend using Python virtual environments to accomplish this.


Step 1: Generate a Dagster Cloud agent token#

In this step, you'll generate a token for the Dagster Cloud agent. The Dagster Cloud agent will use this to authenticate to the agent API.

  1. Sign in to your Dagster Cloud instance.
  2. Click the user menu (your icon) > Organization Settings.
  3. In the Organization Settings page, click the Tokens tab.
  4. Click the + Create agent token button.
  5. After the token has been created, click Reveal token.

Keep the token somewhere handy - you'll need it to complete the setup.


Step 2: Configure the agent#

  1. Create a directory on disk to act as your Dagster home. We'll use ~/dagster_home in our examples, but this directory can be located wherever you want.

  2. In the directory you created, create a dagster.yaml file like the following:

    # ~/dagster_home/dagster.yaml
    
    instance_class:
      module: dagster_cloud.instance
      class: DagsterCloudAgentInstance
    
    dagster_cloud_api:
      agent_token: <YOUR_AGENT_TOKEN>
      deployment: prod
    
    user_code_launcher:
      module: dagster_cloud.workspace.user_code_launcher
      class: ProcessUserCodeLauncher
    
  3. In the file, fill in the following:

    • agent_token - Add the agent token you created in Step 1. This specifies that the agent will launch work in local processes, reading the API URL and authentication token from environment variables.

    • deployment - Enter the deployment associated with this instance of the agent.

      In the above example, we specified prod as the deployment. This is present when Dagster Cloud organizations are first created.

  4. Save the file.


Step 3: Run the agent#

Next, run the process agent by pointing at the home directory you created:

dagster-cloud agent run ~/dagster_home/

To view the agent in Dagster Cloud, click the Dagster icon in the top left to navigate to the Status page and click the Agents tab. You should see the agent running in the Agent statuses section:

Instance Status

Next steps#

Now that you've got your agent running, what's next?