Splunk

Connect Splunk Observability Cloud to an Oppex service so that a detector alert creates an incident and resolves it when the alert clears. A Splunk webhook integration posts to the Oppex Generic Webhook, with the custom headers and payload described on this page.

Before you start

You need the following:

• An Oppex API key from the workspace that owns the service. Open Settings, and in your workspace's section click API Keys. If your organization has only one workspace, click Organization → API Keys instead. See API Keys to create one.
• The service's integration key. Open Services, click Integration Details in the service's row, select API Integration, and copy the Integration Key.
• Administrator access in Splunk Observability Cloud, which creating an integration requires.

Step 1: Create the webhook integration in Splunk

1. In Splunk Observability Cloud, go to Data Management and open the available integrations.
2. Search for Webhook, select it, and click New Integration.
3. On the Configure connection page, enter:
   • Name: for example Oppex - Pied Piper.
   • URL: https://api.oppex.ai/integrations/webhook/incident/generic
   • Method: POST.
   • Shared secret: leave it empty. Oppex does not use it.
   • Headers: the headers in step 2.
4. Click Next. On the Customize message page, replace the payload with the payload in step 3.
5. Click Next, review the integration, and click Save.

Step 2: Add the custom headers

Add each header as a separate name and value.

Header	Value
X-GE-API-KEY	Your Oppex API key
X-GE-SOURCE	Generic
X-GE-AUTORESOLVE	true
X-GE-STATUS-TRIGGERED	anomalous
X-GE-STATUS-RESOLVED	ok,manually resolved,stopped
X-GE-STATUS-SEVERITY	{"anomalous":2}

• X-GE-STATUS-TRIGGERED and X-GE-STATUS-RESOLVED list the Splunk alert statuses that create and resolve incidents. Matching ignores letter case.
• X-GE-STATUS-SEVERITY gives triggered alerts Severity 2. Without it, they get Severity 1.

Keep X-GE-SOURCE set to Generic

Oppex links a cleared alert to its incident through the group field only for sources it recognises, and Generic is one of them. With a custom label such as Splunk, Oppex drops the group, so cleared alerts cannot find the incident to resolve, and repeated alerts open new incidents. To show Splunk as the source and keep auto-resolve, contact Oppex support.

Step 3: Set the payload

Replace the default message with this payload, and replace YOUR_SERVICE_KEY with the service's integration key. Splunk fills in each {{{variable}}} when it sends the alert.

{
  "title": "{{{messageTitle}}}",
  "status": "{{{status}}}",
  "group": "{{{incidentId}}}",
  "externalID": "{{{incidentId}}}",
  "serviceKey": "YOUR_SERVICE_KEY",
  "detector": "{{{detectorName}}}",
  "detectorUrl": "{{{detectorUrl}}}",
  "splunkSeverity": "{{{severity}}}"
}

Field	Why it is there
title	Becomes the incident title. Required
status	The alert status, such as anomalous or ok. The headers decide which values create and resolve incidents
group	Splunk's incident ID, which stays the same from trigger to clear. Oppex resolves the incident that has the same group
serviceKey	Sends the incident to this service
detector, detectorUrl, splunkSeverity	Oppex keeps fields it does not recognise in the incident details, so responders can open the detector

Put the service key in the payload, not the URL

Oppex reads the service key from the payload for this integration. A serviceKey added to the URL is not used, and the incident goes to the service that your workspace's service routing chooses.

Check the variable names against your Splunk template

The Customize message page shows Splunk's default template with the variables your account offers. If a value arrives empty in Oppex, compare the variable names in this payload with that template. Splunk replaces a variable it cannot fill with an empty string.

Do not map Splunk's timestamp to srcTimestamp. Splunk sends an ISO-8601 time, and this field accepts epoch milliseconds only. Without it, Oppex records the time it received the alert.

Step 4: Add the webhook to your detectors

1. Open a detector, or create one.
2. In the Alert recipients step, click Add Recipient.
3. Select Webhook, then the integration from step 1.
4. Activate and save the detector.

What happens when Splunk sends an alert

Alert status	What Oppex does
anomalous	Creates a Severity 2 incident
ok, manually resolved, stopped	Resolves the incidents with the same Splunk incident ID
Any other status	Ignores the alert

While an incident is open, another anomalous alert for the same Splunk incident attaches to it instead of opening a new one. See Grouping strategy.

Worked example

The detector Middle-Out Compression Latency sends to the Oppex - Pied Piper integration.

1. Latency breaches the threshold and Splunk sends anomalous. Oppex creates a Severity 2 incident titled with the alert's message title.
2. Latency returns to normal and Splunk sends ok with the same incident ID. Oppex resolves the incident.

Test the integration

In Splunk, go to Data Management, open the deployed webhook integration, and click Run test. Then trigger a detector, for example by temporarily lowering its threshold, and confirm that an incident appears on the Incidents page with the expected title.

If no incident appears, the reply Oppex sent explains the cause:

Reply	Cause
Status not in triggered or resolved states — ignored	The status is not in X-GE-STATUS-TRIGGERED or X-GE-STATUS-RESOLVED
Field 'title' is required in payload	The payload has no title, or the title variable is empty
X-GE-API-KEY header is required	The X-GE-API-KEY header is missing
Invalid Service Key	The serviceKey in the payload does not match a service
No open incidents found for group	A clear arrived with no matching open incident. Check that X-GE-SOURCE is Generic
Unexpected processing error	The API key is not a valid Oppex API key

Use Splunk's default payload instead

Oppex support can set up your account to read Splunk's default webhook payload without changes. In that setup, you keep the default message on the Customize message page, send the headers X-GE-API-KEY and X-GE-SOURCE: Splunk, and add ?serviceKey=YOUR_SERVICE_KEY to the URL. Contact Oppex support to arrange it.

In-product guide

Oppex has no Splunk-specific guide in the product. The Generic Webhook guide covers the same headers and payload fields: open Services, click Integration Details in the service's row, and select Generic Webhook. See also Generic Webhook.

FAQ

Can I send alerts from Splunk Enterprise?

Not with its built-in webhook alert action, which cannot send the API key header Oppex requires. Splunk Observability Cloud's webhook integration can.

Why does the incident source say Generic instead of Splunk?

The source label must be one that Oppex recognises for cleared alerts to resolve incidents. Contact Oppex support to show Splunk as the source while keeping auto-resolve.

Can Splunk's severity set the Oppex severity?

Not with this payload. X-GE-STATUS-SEVERITY sets one severity for every triggered alert, and Splunk's own severity appears in the incident details as splunkSeverity.

Does a cleared alert resolve other incidents?

Only incidents with the same Splunk incident ID. Oppex resolves the incidents whose group matches the cleared alert's incident ID.