Incident Workflows

Understanding the root cause and knowing how to address it are essential to incident response. Some parts of the investigation involve steps that can be automated to achieve faster resolution times. Incident Workflows do just that. It aims to eliminate the need for manual involvement in aspects of the response that can be structurally defined as a solution.

Incident Workflows allow you to perform actions sequentially and automatically. The execution of the actions can be conditioned on certain incident values or outcome of prior actions executed in the flow itself. It introduces the core idea of if/else logic from programming into incident response. It is tailored to enable anyone with or without programming knowledge to build a chain of commands that can be triggered automatically after events or manually with a click of a button.

Pricing Plans

Incident Workflows are only available on the Business and Digital Operations pricing plans.

Incident Workflows replace Response Sets

Incident Workflows precedes Response Sets, which have been decommissioned in December 2023. It introduces more powerful capabilities and conditional logic on top of the basic actions that were allowed in Response Sets.

Triggers

Triggers define how the workflow should be run – manually or automatically. Both options can be selected together as well.

Workflow Triggers

  • Manually: If you select manually, then the workflow can be run from the "Run Workflow" option under the "More" dropdown button on the incident details page.
  • Automatically: If you select automatically, then you will be prompted to also select the events after which it will be run:
    • After an incident is created
    • After an incident is acknowledged
    • After the urgency is amended
Workflow

The actual workflow is almost like a computer program. You tell it what to do directly or you can tell it what to do if certain conditions are met. There are two basic statements available:

  • Action statement to define the action to perform.
  • If/Else statement to define what to do if some conditions are met and what else to do otherwise. They can be nested inside other If/Else statements along with Action statements.

These two statements give users the flexibility needed to define any type of workflow. We have put together an example to help you understand how Action statements and If/Else Statements can be combined to form an execution chain.

Workflow Example

In the above example, there are two main steps, where the second step is conditional and contains multiple options and sub-actions. The instructions are executed as explained below:

  1. Add a note.
  2. The second step is conditional. If the Payload Field source_payload.key equals datadog-eu, then
    1. Run the custom action Restart EU Server.
    2. If the server is restarted successfully and responds with a HTTP status of 200, then resolve the incident. Otherwise,
      1. Add more responders to the incident.
      2. Post a status update.
      3. Add a conference bridge.

If/Else Statements

If/Else statements consist of three parts – (IF) if conditions, (THEN) actions to take if the conditions are met and (ELSE) actions to take if the defined conditions are not met.

  • IF: Under the IF component you should define all the conditions that must apply for that step. Any type of logic can be formed. You can group certain conditions together and use logical operators to tie them together to form the complete logic.

    For example, the condition ((Service equals Apache Server and Urgency equals Medium) or (Service equals CloudFront and Payload Field tags contains us-region-2)) can be represented by:

    Workflow Conditions

  • THEN: Here you define the actions that must be performed if the condition defined before is met. Here you add Action statements or more If/Else statements.
  • ELSE (optional): This is optional. Only if you need actions to be performed if the conditions described in the IF section are not met. Both Action and more If/Else statements can be added here as well.
Conditional Workflows

If a workflow is conditional (meaning it has conditions), then even when it is run manually, those conditions must be met before the ensuing actions can be performed.

Condition Types

Any value from the payload of an incident can be used to form the conditions. Values from actions already performed in the workflow can also be referred. The list of available condition types are:

  • Service: Service the incident occurred on.
  • Status: Status of the incident.
  • Urgency: Urgency level of the incident.
  • Email From: If the incident is created by an email integration, then the email address of the sender.
  • Email To: If the incident is created by email integration, then the (TaskCall) email address it was sent to.
  • Payload Field: Refer to any field from the source payload of the incident. Besides just the top layer keys, you can refer to nested keys inside the payload as well by connecting the parent key and the child key by a dot. For example, if the json is {"a": {"b": 2}} you can get to "b" by setting the value to "a.b". You can view the source payload from Automation > Conditional Routing.
  • Workflow Not Run Already: If this condition type is selected, then it will be checked that the workflow has not been run for this incident already. This is to avoid running the workflow multiple times on the same incident.
  • Previous Action HTTP Status: The HTTP status of the very last action that was performed (e.g. 200, 201, 400, 403, etc). It will only apply for custom actions that require TaskCall to send an outbound request to an external service. For example, if you run an action that creates a Jira Issue and if that action fails you want to add a note, then after the Create Jira Issue action, you can include an If/Else statement with if Previous Action HTTP Status not equals 201 then Add Note.
  • Previous Action JSON: The JSON response received from the last executed action. It will only apply for the custom actions that require TaskCall to make outbound requests just like the Previous Action HTTP Status condition type. In the example above, where we define an action to create a Jira Issue, you can refer to a field in the JSON response received back from Jira.
Condition Evaluation

Conditions can be evaluated using the following 8 comparators:

  • Contains (could be used for lists and texts)
  • Equals
  • Exists (could be used to check if a key exists)
  • Matches (could be used when the value is a regular expression)
  • Not Contains
  • Not Equals
  • Not Exists
  • Not Matches
Workflow Actions

Workflows support the following actions:

Internal Field References

Field references can be made to dynamically insert incident data into actions that utilize a text filed like posting a status update, sending an email, etc. This will allow the actions to be more contextualized.

Field references can be made by putting the desired in between starting and ending double curly braces {{ field }}. The following internal fields are supported at the moment:

  • {{ incident_id }}: ID of the incident.
  • {{ title }}: title of the incident.
  • {{ description }}: description/text details of the incident
  • {{ status }}: status of the incident
Field References with Regular Expressions

Field references can be made even more dynamic by using regular expressions to capture specific portions of the field itself. This can be done by adding the regular expression after the field name preceded by "|".

{{ field | regular-expression }}

You should test the regular expressions beforehand to make sure they generate the correct outcome.

Field References from Actions

Field references can also be made to the output of custom actions (actions that are executed on external systems and receive a response back). References to action outputs can be made as:

{{ step["action name"].field["field"] }}

If the same action was run multiple times, then you can refer to a specific run by its number. Numbering starts from 0. To add the run number, add a hyphen right after the action name followed by the run number (without any spaces).

For example: {{ step["Create Jira Issue-1"].field["self"] }}

Regular expressions can also be added just like in internal field references as:

{{ step["action name"].field["field"] | regular-expression }}

Permissions

Since workflows can be used to run intricate actions on your system, permissions can be imposed to restrict access to it. Permissions can be managed from the Permissions tab on the workflow details page. Two types of permissions can be imposed:

Workflow Permissions

Permissions to Run Workflow

Define which users will be able to run the workflow:

  • All incident responders, or
  • Responders of a specific team

This permission check will only be done when the workflow is run manually.

Permissions to Edit Workflow

Define which users will be able to edit and manage the workflow:

  • All admins of the organization, or
  • Managers of a specific team

Only the Account Owner will always be able to edit the workflow even when it is restricted to a specific team.

On this page