Semaphor
Observability

Webhook Destinations

Configure webhook endpoints to receive real-time failure notifications from Semaphor

Webhook destinations let your organization receive HTTP POST notifications when failures occur in Semaphor. Route these to incident management tools (PagerDuty, OpsGenie), messaging platforms (Slack or Teams via middleware), or custom logging endpoints.

Admin role required

Webhook destination management requires the Admin role. Non-admin users can view that a webhook is configured but cannot change settings.

Self-hosted deployments

Set ENTERPRISE_TELEMETRY_ENABLED=true on your Semaphor container to enable webhook delivery. Without this variable, the telemetry settings UI shows webhooks as unavailable.


Setting Up a Webhook

  1. Navigate to Organization Settings.
  2. Scroll to the Telemetry Webhook section.
  3. Enter a webhook URL. The URL must use HTTPS.
  4. Toggle the status to Active.
  5. (Optional) Enter a Signing Secret for HMAC payload verification.
  6. Open the Events dropdown and select which event types to receive.
  7. Click Save Settings.

One destination per organization

The current version supports one webhook destination per organization. If you need to fan out events to multiple systems, use a webhook relay service as your destination.


Event Subscriptions

By default, all nine event types are subscribed. Use the Events dropdown to narrow which types your endpoint receives.

Event TypeDescription
query_failedA data query failed to execute
dashboard_load_failedA dashboard failed to load its template
token_generation_failedAn embed or API token could not be generated
assistant_request_failedAn AI assistant request failed
assistant_request_abortedAn AI assistant request was cancelled
assistant_tool_failedA tool invocation within an assistant session failed
assistant_provider_errorThe upstream AI model provider returned an error
assistant_dynamic_visual_failedThe assistant failed to generate a dynamic visualization
export_failedA PDF or CSV data export failed

At least one event type must remain selected. See Event Reference for full payload details.


Signing Secret and Payload Verification

Setting a Signing Secret

When you provide a signing secret, Semaphor signs every outgoing payload with HMAC-SHA256. The signature is included in the X-Semaphor-Signature header as sha256=<hex_digest>.

Verifying Signatures

To verify a webhook payload, compute the HMAC-SHA256 of the raw request body using your shared secret and compare the result to the X-Semaphor-Signature header value.

verify-webhook.ts
import crypto from 'crypto';

function verifySignature(body: string, signature: string, secret: string): boolean {
  const expected = crypto.createHmac('sha256', secret).update(body).digest('hex');
  return signature === `sha256=${expected}`;
}

// In your webhook handler:
const signature = req.headers['x-semaphor-signature'];
const isValid = verifySignature(rawBody, signature, SIGNING_SECRET);
verify_webhook.py
import hashlib, hmac

def verify_signature(body: bytes, signature: str, secret: str) -> bool:
    expected = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    return signature == f"sha256={expected}"

Always verify against the raw request body, not a re-serialized version. Re-serialization can change key ordering or whitespace, which invalidates the signature.


Testing Your Webhook

Click the Test button in the Telemetry Webhook settings to send a webhook_test event to your endpoint. The test uses the current form values (URL and signing secret), not the last saved configuration. This lets you verify a new URL before saving.

A successful test confirms URL reachability, TLS handshake, and signature verification (if a secret is set). If you don't have an endpoint ready yet, you can use webhook.site to generate a temporary URL and inspect incoming payloads.

Sample test event payload:

webhook_test payload
{
  "event_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "event_type": "webhook_test",
  "occurred_at": "2025-06-15T14:32:08.491Z",
  "status": "test",
  "category": "webhook",
  "component": "telemetry-webhook-delivery",
  "message": "This is a test event from Semaphor.",
  "org_id": "org_abc123",
  "actor_type": "org_user",
  "actor_id": "usr_def456",
  "org_user_id": "usr_def456",
  "operation": "webhook_test",
  "stage": "webhook_test",
  "error_code": "test_event",
  "error_message": "This is a test event sent from the telemetry settings page.",
  "error_details": "{\"destinationId\":\"dest_ghi789\",\"destinationUrl\":\"https://example.com/webhooks\"}"
}

Delivery Behavior

PropertyValue
HTTP methodPOST
Content-Typeapplication/json
User-AgentSemaphor-Enterprise-Telemetry/1.0
Signature headerX-Semaphor-Signature (when a signing secret is set)
Max delivery attempts2
Timeout per attempt10 seconds
Retry backoffExponential (250ms, 500ms)
Delivery guaranteeBest-effort (in-memory queue)

Monitoring Destination Health

Each destination record tracks two health indicators:

  • lastError -- The error message from the most recent delivery failure.
  • lastFailureAt -- The timestamp of the most recent delivery failure.

Both fields are cleared on the next successful delivery. If lastError persists, your endpoint is consistently unreachable or returning non-2xx responses. These values are visible in the organization settings UI.


Troubleshooting

ProblemCauseSolution
Test returns an errorURL unreachable or returns non-2xxCheck the URL, firewall rules, and TLS certificate
Signature verification failsSecret mismatch or re-serialized bodyVerify against the raw request body with the correct secret
Events stop arrivingDestination disabled or returning errorsCheck the destination status and lastError field
"Webhook delivery is not available"Self-hosted without the env varAdd ENTERPRISE_TELEMETRY_ENABLED=true to your environment
Missing some event typesSubscription filter narrowedOpen the Events dropdown and enable the missing types
Duplicate events after restartQueue was flushed before shutdownExpected behavior -- design your handler to be idempotent using event_id

On this page