Google Cloud Monitoring

Connect Google Cloud Monitoring to an Oppex service so that an alert policy incident creates an Oppex incident. A webhook notification channel sends each incident to Oppex when it opens, and again when it closes, at which point Oppex resolves its incident.

Before you start

You need the following:

• An Oppex API key from the workspace that owns the service. Open Settings, and in your workspace's section click API Keys. If your organization has only one workspace, click Organization → API Keys instead. See API Keys to create one.
• The service's integration key. Open Services, click Integration Details in the service's row, select API Integration, and copy the Integration Key.
• Permission to edit notification channels and alert policies in the Google Cloud project.

Step 1: Build the webhook URL

Google Cloud sends alerts to a URL. Put the service's integration key into this URL:

https://api.oppex.ai/integrations/webhook/incident/generic?source=GCP&serviceKey=YOUR_SERVICE_KEY

• source=GCP is required. It tells Oppex to read the message as a Google Cloud Monitoring incident.
• serviceKey sends every incident to that service. Leave it out to let the workspace's service routing choose the service; the workspace then needs a routing rule that matches, or a default service.

Do not put the API key in the URL

Oppex does not read an API key from the URL. Google Cloud sends the API key with HTTP Basic authentication, as described in step 2.

Step 2: Create the webhook notification channel

1. In the Google Cloud console, go to Monitoring > Alerting and click Edit notification channels.
2. In the Webhooks section, click Add new.
3. Enter a display name, for example Oppex.
4. Paste the URL from step 1 as the endpoint URL.
5. Select Use HTTP Basic Auth.
6. Enter any username, for example oppex. Oppex ignores it.
7. Enter your Oppex API key as the password.
8. Click Test connection, then Save.

Oppex replies to the test notification with a success response and does not create an incident for it.

A successful test does not prove the API key is right

Oppex accepts the test notification before it checks the password. Confirm the setup with a real alert, as described in "Test the integration".

Step 3: Add the channel to your alert policies

1. Open an alert policy, or create one.
2. In the policy's notifications, add the Oppex channel.
3. Select Notify on incident closure, so that Google Cloud tells Oppex when the incident closes.
4. Set the policy's severity level, which decides the Oppex severity.
5. Save the policy.

Turn on notifications for closed incidents

Without Notify on incident closure, Google Cloud does not send the closure, and the Oppex incident stays open until someone resolves it.

What happens when Google Cloud sends an incident

Google Cloud incident state	What Oppex does
open	Creates an incident
closed	Resolves the Oppex incidents for that Google Cloud incident
Any other state	Ignores the message

The alert policy's severity level sets the Oppex severity:

Alert policy severity	Oppex severity
Critical	1
Error	2
Warning	3
No severity, or any other value	3

The incident contains the following:

• Title: the alert policy's name. When the message has no policy name, Oppex uses the incident summary, and then the condition name.
• Source: GCP.
• Details: the complete message that Google Cloud sends.

Oppex links each message to its Google Cloud incident ID. If Google Cloud sends another message for an incident that is still open in Oppex, it attaches to that incident instead of opening a new one.

Worked example

Suppose an alert policy named Paddock Fence Power Lost, with the severity level Critical, sends to the channel whose URL carries the integration key of the Raptor Paddock API service.

1. The fence power metric breaches the condition, and Google Cloud opens an incident. Oppex creates a Severity 1 incident titled Paddock Fence Power Lost on Raptor Paddock API.
2. Power is restored, and Google Cloud closes its incident. Because Notify on incident closure is on, Google Cloud sends the closure, and Oppex resolves the incident.

Test the integration

Trigger a test incident in Google Cloud, for example by temporarily lowering a condition's threshold. An incident appears on the Incidents page within a few seconds. When the condition clears, the incident is resolved.

If no incident appears, check the reply Oppex sent:

Reply	Cause
Invalid API key	The basic authentication password is not a valid, enabled Oppex API key
X-GE-API-KEY header is required	Use HTTP Basic Auth is not selected, so no API key was sent
Invalid Service Key	The serviceKey in the URL does not match a service
No open incidents found for group	A closure arrived for an incident that is already resolved in Oppex

In-product guide

Oppex has no in-app setup guide for Google Cloud Monitoring. This page is the reference. The service's integration key is in Services → Integration Details → API Integration.

FAQ

Why does Google Cloud need basic authentication?

Google Cloud webhook channels cannot send custom headers. Oppex reads the API key from the basic authentication password instead.

What severity does an alert policy without a severity level get?

Severity 3. Set the policy's severity level to Critical or Error to create Severity 1 or Severity 2 incidents.

Why did my incident not resolve when Google Cloud closed it?

The alert policy must have Notify on incident closure selected. Without it, Google Cloud does not send the closure to Oppex.

Can one channel send to several services?

Yes. Leave serviceKey out of the URL, and Oppex chooses the service for each incident with the workspace's service routing.