How can I set up outgoing problem-notifications using a webhook?

Dynatrace offers several out-of-the-box integrations that automatically push problem notifications to third-party incident-management and ChatOps systems. If however your third-party system isn’t supported with an out-of-the-box integration, you can easily set up a customizable webhook integration. Using this approach, whenever Dynatrace detects a problem in your environment that affects real users, a webhook triggers an HTTP POST request to a target URL that you specify. The payload message of the HTTP POST request is completely customizable, as long as it represents valid JSON syntax. You can include information placeholders, such as {ProblemTitle} and {State}, to fill the custom JSON with the dynamic information of each detected problem.

To integrate problem-notifications using a custom webhook, navigate to Dynatrace > Settings > Integration > Problem notifications, click Set up notifications, and select Custom integration, as shown below.

The Set up custom integration page appears.

Type the following information to set up a webhook integration

  • Name User-defined display name that distinguises this unique integration from other integrations.
  • Webhook URL Target URL where the HTTP POST should push the JSON payload. This URL can contain HTTP parameters such as an authentication token, in case the destination system works with authentication tokens instead of basic authentication.
  • Username (optional) Optional username that is used as an HTTP POST basic authentication header field in case the target system requires a basic authentication method.
  • Password (optional) Optional password that is used as an HTTP POST basic authentication header field in case the target system requires a basic authentication method.
  • Custom JSON payload
    Once a problem is detected or resolved, this customizable JSON payload is pushed through an HTTP POST to the target system. Use specific placeholders to dynamically provide problem-related information such as problem state or problem title.

Here is a valid JSON web hook problem-notification definition example.

{
   "State" : "{State}",
   "ProblemID" : "{ProblemID}",
   "ProblemTitle" : "{ProblemTitle}"
}

Once a problem is detected, the placeholders are populated with the actual values and results, as shown in this example payload:

{
   "State" : "OPEN",
   "ProblemID" : "512",
   "ProblemTitle" : "OPEN Problem 912: Elastic load balancer has a high backend failure rate on Elastic Load Balancer help-LB"
}

Available placeholders

  • {State}: Problem state. Possible values are OPEN and RESOLVED.
  • {ProblemID}: ID number of the reported problem.
  • {PID}: Unique system identifier of the reported problem.
  • {ProblemImpact}: Impact level of the problem. Possible values are APPLICATION, SERVICE, and INFRASTRUCTURE.
  • {ProblemTitle}: Short description of the problem.
  • {ProblemDetails}: All problem event details, including root cause, as an HTML-formatted string.
  • {ProblemURL}: URL of the details page of the corresponding problem in Dynatrace.
  • {ImpactedEntity}: Entities impacted by the problem (or the term “multiple” when more than two entities are impacted).
  • {Tags}: Comma-separated values list of tags that have been defined for all impacted entities.

Finalize, send a test notification, and save

Use the Notification level switches to control the types of problem notifications that are sent to the target system (problems with Applications, Services, and/or Infrastructure).

Finally, before you save your custom integration setup, click Send test notification to perform a test run and ensure that the destination address is available.