Observability
Monitor failures and debug issues in your Semaphor deployment with structured logging and webhook notifications
Semaphor tracks server-side failures across query execution, dashboard loading, token generation, AI assistant interactions, and data exports. When something goes wrong, the platform emits structured telemetry events that help you identify, diagnose, and resolve issues. The current version focuses on failure events.
Why Observability Matters
- Troubleshooting -- Structured events include resource IDs, actor context, and error details so you can pinpoint root causes without searching through raw logs.
- Monitoring -- Forward failure events to PagerDuty, Slack, or your alerting tool of choice for proactive notification before users report problems.
- Compliance -- Maintain an audit trail of system failures, including which user and resource were involved.
Two Modes of Operation
Semaphor provides two ways to consume telemetry events, depending on your deployment model.
Webhook Telemetry (Cloud and Self-Hosted)
For organizations on Semaphor Cloud, webhook telemetry is the primary way to receive failure notifications. Configure a webhook endpoint and Semaphor sends structured JSON events to it via HTTP POST whenever a failure occurs. This is also available in self-hosted deployments.
- One webhook destination per organization
- Requires the Admin role to configure
- Supports HMAC-SHA256 request signing for payload verification
- Asynchronous and non-blocking -- never affects request performance
See Webhook Destinations for setup instructions.
Structured Console Logging (Self-Hosted)
For self-hosted deployments, every failure event is also written as a single-line JSON object to stdout (for warnings) or stderr (for errors). This is always active with no configuration required.
Pipe your container output to any log aggregator -- Datadog, ELK, CloudWatch, Fluent Bit -- and search or alert on the structured fields.
{
"timestamp": "2025-06-15T14:32:08.491Z",
"level": "error",
"component": "query-execution",
"category": "query",
"event": "query_failed",
"message": "Query execution failed: relation \"orders\" does not exist",
"event_id": "evt_a1b2c3d4",
"request_id": "req_x9y8z7",
"org_id": "org_abc123",
"project_id": "prj_def456",
"dashboard_id": "d_ghi789",
"card_id": "card_jkl012",
"connection_id": "conn_mno345",
"actor_type": "tenant_user",
"actor_id": "tu_pqr678",
"tenant_id": "t_stu901",
"query_mode": "config",
"query_text_truncated": false,
"error_code": "query_execution_error",
"error_message": "relation \"orders\" does not exist"
}What Gets Monitored
| Event Type | Category | Description |
|---|---|---|
query_failed | Query | A data query failed to execute against the connected database |
dashboard_load_failed | Dashboard | A dashboard failed to load its template or configuration |
token_generation_failed | Token | An embed or API token could not be generated |
assistant_request_failed | Assistant | An AI assistant request failed to complete |
assistant_request_aborted | Assistant | An AI assistant request was cancelled before completing |
assistant_tool_failed | Assistant | A tool invocation within an AI assistant session failed |
assistant_provider_error | Assistant | The upstream AI model provider returned an error |
assistant_dynamic_visual_failed | Assistant | The AI assistant failed to generate a dynamic visualization |
export_failed | Export | A PDF or CSV data export failed |
See Event Reference for full payload details and error codes.
Data Privacy
What is never included in telemetry events
Telemetry events are designed to aid debugging without exposing sensitive data:
- Raw JWT tokens
- Database connection strings or passwords
- AI assistant transcripts or chain-of-thought reasoning
- Full security policy definitions (only policy names and parameter keys)
Query text is included to help diagnose query failures, but is truncated at 16 KB.
Self-Hosted Configuration
Enabling webhooks in self-hosted deployments
Structured console logging is always active and requires no configuration.
To enable webhook delivery, set the following environment variable on your Semaphor container:
ENTERPRISE_TELEMETRY_ENABLED=trueSee the Self-Hosted guide for full deployment configuration.