Datadog

Connect Datadog to an Oppex service so that a triggered Datadog monitor creates an incident and its recovery resolves it. Datadog's Webhooks integration posts to the Oppex Generic Webhook, with headers and a payload that you set up once on this page.

Before you start

You need the following:

• An Oppex API key from the workspace that owns the service. Open Settings, and in your workspace's section click API Keys. If your organization has only one workspace, click Organization → API Keys instead. See API Keys to create one.
• The service's integration key. Open Services, click Integration Details in the service's row, select Generic Webhook, and copy the key from the sample request in Test your integration. The API Integration option shows the same key as Integration Key.
• Permission to manage integrations in Datadog.

Step 1: Create the webhook in Datadog

1. In Datadog, open the Webhooks integration tile and add a new webhook.

2. Enter a name, for example oppex-paper-supply. You use this name to notify the webhook from monitors.

3. Set URL to the Oppex Generic Webhook URL:

   https://api.oppex.ai/integrations/webhook/incident/generic

4. Set Payload to the JSON in step 2.

5. Set Custom Headers to the JSON in step 3.

6. Save the webhook.

Step 2: Set the payload

Paste this payload, and replace YOUR_SERVICE_KEY with the service's integration key. Datadog replaces each $ variable with the monitor's value when it sends the alert.

{
  "title": "$EVENT_TITLE",
  "status": "$ALERT_TRANSITION",
  "group": "$ALERT_CYCLE_KEY",
  "serviceKey": "YOUR_SERVICE_KEY",
  "srcTimestamp": "$DATE",
  "externalID": "$ALERT_ID",
  "component": "$HOSTNAME",
  "type": "$ALERT_TYPE",
  "link": "$LINK",
  "tags": "$TAGS",
  "datadogPriority": "$ALERT_PRIORITY"
}

Field	Why it is there
title	Becomes the incident title. Required
status	Datadog's alert transition, such as Triggered or Recovered. The headers in step 3 decide which values create and resolve incidents
group	Datadog's alert cycle key, which links a trigger to its recovery. Oppex resolves the incident that has the same group
serviceKey	Sends the incident to this service
srcTimestamp	When the alert happened, in epoch milliseconds
link, tags, datadogPriority	Oppex keeps fields it does not recognise in the incident details, so responders can follow the link back to Datadog

Put the service key in the payload, not the URL

Oppex reads the service key from the payload for this integration. A serviceKey added to the URL is not used, and the incident goes to the service that your workspace's service routing chooses.

Why the priority field is called datadogPriority

Datadog priorities look like P1. A field named priority must hold a number from 1 to 5 in Oppex, so the payload keeps Datadog's value under a different name, where it appears in the incident details.

Step 3: Set the custom headers

Datadog takes custom headers as a JSON object. Paste this object, and replace YOUR_API_KEY with your Oppex API key.

{
  "X-GE-API-KEY": "YOUR_API_KEY",
  "X-GE-SOURCE": "Generic",
  "X-GE-AUTORESOLVE": "true",
  "X-GE-STATUS-TRIGGERED": "Triggered,Re-Triggered,Warn,Re-Warn",
  "X-GE-STATUS-RESOLVED": "Recovered",
  "X-GE-STATUS-SEVERITY": "{\"Triggered\":1,\"Re-Triggered\":1,\"Warn\":3,\"Re-Warn\":3}"
}

Header	What it does
X-GE-API-KEY	Authenticates the request with your Oppex API key
X-GE-SOURCE	Sets the incident's source label. Keep it Generic, as explained below
X-GE-AUTORESOLVE	Lets a Recovered alert resolve the incident
X-GE-STATUS-TRIGGERED	The transitions that create incidents. Matching ignores letter case
X-GE-STATUS-RESOLVED	The transitions that resolve incidents
X-GE-STATUS-SEVERITY	Maps each transition to an Oppex severity. The inner quotes are escaped because the value is itself JSON

Keep X-GE-SOURCE set to Generic

Oppex links a recovery to its incident through the group field only for sources it recognises, and Generic is one of them. With a custom label such as Datadog, Oppex drops the group, so Recovered alerts cannot find the incident to resolve, and repeated alerts open new incidents. To show Datadog as the source and keep auto-resolve, contact Oppex support.

Always set X-GE-STATUS-TRIGGERED

Without this header, every message that does not resolve an incident creates one, including No Data and Renotify transitions.

With these headers, No Data, Re-No Data and Renotify transitions are ignored. Add them to X-GE-STATUS-TRIGGERED if you want incidents for them. A transition missing from X-GE-STATUS-SEVERITY creates a Severity 1 incident.

Step 4: Notify the webhook from your monitors

1. Open a Datadog monitor, or create one.
2. In the monitor's message, mention the webhook: @webhook-oppex-paper-supply.
3. Save the monitor.

Datadog posts to Oppex each time the monitor changes state.

What happens when Datadog sends an alert

Alert transition	What Oppex does
Triggered, Re-Triggered	Creates a Severity 1 incident
Warn, Re-Warn	Creates a Severity 3 incident
Recovered	Resolves the incidents with the same alert cycle key
Anything else, such as No Data	Ignores the alert

While an incident is open, a Re-Triggered alert from the same alert cycle attaches to it instead of opening a new incident. See Grouping strategy.

Worked example

The monitor Paper Supply API error rate notifies @webhook-oppex-paper-supply.

1. The error rate crosses the alert threshold. Datadog sends Triggered with the title [Triggered] Paper Supply API error rate. Oppex creates a Severity 1 incident with that title.
2. The error rate stays high and Datadog sends Re-Triggered with the same alert cycle key. Oppex attaches it to the open incident.
3. The error rate recovers. Datadog sends Recovered with the same alert cycle key, and Oppex resolves the incident.

Test the integration

Trigger a monitor that notifies the webhook, for example by temporarily lowering its threshold. The incident appears on the Incidents page within a few seconds.

If no incident appears, the reply Oppex sent to Datadog explains the cause:

Reply	Cause
Status not in triggered or resolved states — ignored	The transition is not in X-GE-STATUS-TRIGGERED or X-GE-STATUS-RESOLVED
Field 'title' is required in payload	The payload has no title
X-GE-API-KEY header is required	The custom headers are missing or not valid JSON
Invalid Service Key	The serviceKey in the payload does not match a service
No open incidents found for group	A recovery arrived with no matching open incident. Check that X-GE-SOURCE is Generic
Unexpected processing error	The API key is not a valid Oppex API key

In-product guide

Oppex has no Datadog-specific guide in the product. The Generic Webhook guide covers the same headers and payload fields: open Services, click Integration Details in the service's row, and select Generic Webhook. See also Generic Webhook.

FAQ

Why does the incident source say Generic instead of Datadog?

The source label must be one that Oppex recognises for recoveries to resolve incidents. Contact Oppex support to show Datadog as the source while keeping auto-resolve.

Can I send Datadog alerts to several services?

Yes. Create one Datadog webhook per service, each with that service's key in the payload, and mention the right webhook in each monitor.

Do warnings create incidents?

Yes, with the headers on this page. Warn and Re-Warn create Severity 3 incidents. Remove them from X-GE-STATUS-TRIGGERED to ignore warnings.

Does a recovery for one alert resolve another?

Only if both share an alert cycle key. Oppex resolves the incidents whose group matches the recovery's alert cycle key.