How to Set Up Signed Webhook Alerts From a Monitoring Tool

Webhook alerts are the integration glue that connects an uptime monitor to everything else: PagerDuty, Opsgenie, internal incident systems, ticket trackers, custom Slack bots. The basic mechanic is simple (your monitor POSTs JSON to your endpoint when an incident fires). The details (signature verification, retry behaviour, idempotency) are where the implementation gets interesting. This guide covers the practical setup.

Why webhooks instead of email or Slack

Email and Slack work fine for humans. Webhooks work for systems. Three use cases justify the extra setup.

Routing to a paging tool that runs your on-call rotation (PagerDuty, Opsgenie). Driving an internal incident-management system that opens tickets, creates Slack threads, and runs runbooks automatically. Triggering automated remediation (restart a worker, scale up a cluster, failover to a backup region). All three need a structured payload, retry-safe delivery, and authentication. Webhooks provide all three; email and Slack provide none.

The payload shape to expect

Most monitoring tools send a similar JSON payload on incident events. The exact field names vary, but four sections are standard.

• event: 'monitor.down' or 'monitor.up', so your endpoint can route opens and closes.
• monitor: id, name, type, target, last status. Enough context to identify what is broken.
• check: status code, response time, error message. The diagnostic detail.
• timestamp: ISO 8601, UTC. Important for ordering and deduplication.

Verifying the HMAC signature

Any webhook endpoint exposed to the public internet needs signature verification. Without it, an attacker who guesses your URL can fake a 'monitor down' event and trigger your remediation, your paging, and your support response. With HMAC verification, only your monitoring tool can sign valid payloads.

The pattern: the monitoring tool sends an X-Monitorah-Signature header with the HMAC-SHA256 of the raw request body, keyed with a shared secret. Your endpoint recomputes the HMAC of the body using the same secret, compares (in constant time) to the header value, rejects on mismatch. Use the raw bytes of the body for the HMAC, not the parsed JSON: a single whitespace difference produces a different signature.

Retry and idempotency

Webhook delivery is best-effort, not guaranteed. Two patterns make your endpoint robust to retries and partial failures.

• Honour retries gracefully. Most monitors will retry a 5xx response with exponential backoff for up to 24 hours. Return a 5xx if you cannot process the event; return a 2xx if you can.
• Treat the event id as an idempotency key. The same incident open can fire the webhook twice during a network blip. Deduplicate by storing the event id and skipping if you have already processed it.
• Respond quickly. Take less than 5 seconds to return a 2xx, even if your downstream processing takes longer. Use a job queue if needed.

Integrating with PagerDuty

PagerDuty accepts a generic webhook through its Events API v2. The flow: your monitoring tool POSTs a custom JSON payload to your own handler. Your handler verifies the HMAC, maps the payload into PagerDuty's expected schema, and POSTs to PagerDuty's events endpoint with your routing key.

The translation layer is the right place to add deduplication, escalation rules, and per-monitor routing keys. PagerDuty's UI handles the rotation once the event has been ingested. Keep the translator simple. The cleanest version is fewer than a hundred lines of code.

A starter webhook handler in 20 lines

The pattern: receive POST, extract X-Monitorah-Signature header, HMAC-SHA256 the raw body with your shared secret, constant-time compare with the header. If they match, parse JSON, deduplicate by event id, dispatch. If they do not match, return 401 without processing. This handler is the building block for every integration on top: PagerDuty, Opsgenie, Slack bots, internal incident systems. Once it is in place, every new integration is a few lines of dispatch logic and nothing more.