Monitoring and Alerting#

Alert policies define which jobs will trigger an alert, the conditions under which an alert will be sent, and how the alert will be sent.

An alert policy includes a set of configured tags. If an alert policy has no configured tags, all jobs will be eligible for that alert. Otherwise, only jobs that contain all the tags for a given alert policy are eligible for that alert. Additionally, an alert policy includes a set of conditions under which an alert will be triggered. For example, the alert can be triggered to fire on job failure, job success, or both events. Finally, an alert policy specifies where the alert will be sent. Currently, we support Slack and email as targets for an alert.

Setting up alerts#

Slack#

To enable Slack notifications in your workspace, install the Dagster Cloud Slack application.

The Slack application can be installed by navigating to https://<ORGANIZATION_NAME>.dagster.cloud/slack/install. This will initiate a flow to install the application into your workspace.

After the Slack application is installed in your workspace, invite the bot user to your desired channel.

Email#

No additional setup is required for email alerts.

Using Alert Policies#

Configuring Alerts in Dagit#

You can manage your alerts from the Alerts tab on the Cloud Settings page:

Click on "Create Alert" to configure your alert.

An email alert can be configured to email multiple recipients:

A Slack alert can only be configured to message one channel:

After saving these alerts they will show up in your alerts dashboard where you can disable, edit or delete them.

Configuring Alerts with the CLI#

A list of alert policies can be defined in a single YAML file. After declaring your policies, you can set them for your Dagster Cloud deployment using the following command:

dagster-cloud deployment alert-policies sync -a <ALERT_POLICIES_PATH>

To see the policies that you've currently configured on your deployment, you can list them using the following command:

dagster-cloud deployment alert-policies list

Email alert policies

Suppose we have a job named important_job with tag {"level": "critical"}. We would like to configure an email alert whenever this job fails.

@op
def important_computation():
    ...


@job(tags={"level": "critical"})
def important_job():
    important_computation()

Here, we define an email alert policy that listens to jobs with the tag {"level": "critical"}. If any of these jobs fail, and email will be sent to both richard.hendricks@hooli.com and nelson.bighetti@hooli.com.

alert_policies:
  - name: "email-alert-policy"
    description: "An alert policy to email company executives during job failure."
    tags:
      - key: "level"
        value: "critical"
    event_types:
      - "JOB_FAILURE"
    notification_service:
      email:
        email_addresses:
          - "richard.hendricks@hooli.com"
          - "nelson.bighetti@hooli.com"

Slack alert policies

Suppose we have another job named sales_job with tag {"team", "sales"}. We would like to configure a Slack notification whenever a run of this job succeeds or fails.

@op
def sales_computation():
    ...


@job(tags={"team": "sales"})
def sales_job():
    sales_computation()

Here we define a slack alert policy to send a notification to sales-notifications whenever the job succeeds or fails.

alert_policies:
  - name: "slack-alert-policy"
    description: "An alert policy to send a Slack notification to sales on job failure or success."
    tags:
      - key: "team"
        value: "sales"
    event_types:
      - "JOB_SUCCESS"
      - "JOB_FAILURE"
    notification_service:
      slack:
        slack_workspace_name: "hooli"
        slack_channel_name: "sales-notifications"

Using system tags#

For a job, alert emails can be configured by setting the dagster-cloud/alert_emails tag on a job. Then, when a job run has failed, a notification will be sent to the alert emails.

In this example, we define two alert emails for our job important_job: richard.hendricks@hooli.com and nelson.bighetti@hooli.com. On run failure, these two emails will be sent a notification.

from dagster import job, op
from dagster_cloud import ALERT_EMAILS_TAG


@op
def important_computation():
    ...


@job(
    tags={
        ALERT_EMAILS_TAG: [
            "richard.hendricks@hooli.com",
            "nelson.bighetti@hooli.com",
        ]
    }
)
def important_job():
    important_computation()