How We Made It Easy for Anyone at Our Company to Trigger Alerts
Scale relies on Mode Analytics for internal SQL and Python based reporting and
analytics, helping power our operations, insights and business decisions. Many
of these reports we set to run on a schedule, so that we can always go to a
report and see data that’s no more than fifteen minutes to an hour out of
date. However, Mode doesn’t natively support a method of notifying people when
something needs attention, e.g. when our margins on a project take a sharp
dive or many new tasks are suddenly created. We’ve set up a system to utilize
these reports to easily alert us when something’s unusual, so that someone can
take action.
Alert generation
The flow for alerting goes as follows: when a report is run (either manually
or on a schedule), it sends a webhook to an AWS Lambda function, which then
pulls the report data from the Mode API, and retrieves a list of alert
definitions from our database. It then looks through the alert definitions and
sees if any are applicable to the current report being run, and if so, creates
an alert for each row in the report that satisfies conditions set out in the
matching alert definition. These alerts are then sent to a Slack webhook,
which generates an alert in a channel specified in the alert definition.
Defining alerts
Alerts are defined in our database, and are easily viewable/creatable from our
corp dashboard.
Alerts are simply defined with a handful of fields:
- Report URL: the URL of the Mode report that this alert can
- fire for.
- Alert Title
- Alert Condition: Python expression that is evaluated with
- every row of a report to determine if that row is alert-worthy.
- Filter-max Column: name of a column to only take rows where
- the value of that column is maximal. Useful for reports with time-series
- data, usually you only want to alert on rows with the most recent data.
- Alert Text Format String: Python format string, used to
- generate the message text in the Slack alert.
- Color: sidebar color of the alert message
- Alert Channel: Slack channel to post to
- Enabled
The combination of the easy-to-use dashboards, and the simple definitions of
the alerts, makes creating alerts based on reports very easy to non-technical
people—all that’s required to make fully-featured custom alerts is very basic
Python knowledge—essentially basic math operations—and SQL if one is creating
their own reports from scratch.