Deploy on AWS EC2

Step-by-step guide to deploy Semaphor on a single EC2 instance with nginx and SSL

Deploy Semaphor on a single AWS EC2 instance with nginx as a reverse proxy and Let's Encrypt SSL certificates. This setup is suitable for small to medium teams.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        Internet                             │
└─────────────────────────┬───────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────┐
│                    EC2 Instance                             │
│  ┌───────────────────────────────────────────────────────┐  │
│  │                   nginx (ports 80, 443)               │  │
│  │                   SSL termination                     │  │
│  │                   Let's Encrypt certs                 │  │
│  └───────────────────────────┬───────────────────────────┘  │
│                              │                              │
│                              ▼                              │
│  ┌───────────────────────────────────────────────────────┐  │
│  │                   Docker Compose                      │  │
│  │  ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐  │  │
│  │  │ API Service │ │Data Service │ │ Sidecar Service │  │  │
│  │  │  (port 3000)│ │ (port 8080) │ │   (port 8081)   │  │  │
│  │  └─────────────┘ └─────────────┘ └─────────────────┘  │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

Prerequisites

Before you begin, ensure you have:

AWS Account with permissions to create EC2 instances, security groups, and Elastic IPs
Domain name with access to DNS settings (e.g., semaphor.yourdomain.com)
SSH key pair created in your target AWS region
Kinde account with Client ID, Client Secret, and Issuer URL (setup guide)
Semaphor license key from support.semaphor.cloud

Request a free 60-day evaluation license to test Semaphor before purchasing.

Semaphor uses Kinde for authenticating console users only. Before deploying, you'll need to set up a Kinde account and configure an application. See the Kinde setup guide for instructions.

Step 1: Create Security Group

First, create a security group that allows SSH, HTTP, and HTTPS traffic.

Using AWS CLI:

# Create security group
aws ec2 create-security-group \
  --group-name semaphor-sg \
  --description "Security group for Semaphor"
 
# Allow SSH (port 22) - restrict to your IP for security
aws ec2 authorize-security-group-ingress \
  --group-name semaphor-sg \
  --protocol tcp \
  --port 22 \
  --cidr 0.0.0.0/0
 
# Allow HTTP (port 80) - needed for Let's Encrypt validation
aws ec2 authorize-security-group-ingress \
  --group-name semaphor-sg \
  --protocol tcp \
  --port 80 \
  --cidr 0.0.0.0/0
 
# Allow HTTPS (port 443)
aws ec2 authorize-security-group-ingress \
  --group-name semaphor-sg \
  --protocol tcp \
  --port 443 \
  --cidr 0.0.0.0/0

Using AWS Console:

Go to EC2 > Security Groups > Create security group
Add inbound rules:

Port	Protocol	Source	Purpose
22	TCP	Your IP	SSH access
80	TCP	0.0.0.0/0	HTTP (Let's Encrypt)
443	TCP	0.0.0.0/0	HTTPS

For production, restrict SSH access to your IP address or VPN range.

Step 2: Launch EC2 Instance

Using AWS CLI:

# Get the latest Amazon Linux 2023 AMI ID for your region
AMI_ID=$(aws ec2 describe-images \
  --owners amazon \
  --filters "Name=name,Values=al2023-ami-2023*-x86_64" \
  --query 'sort_by(Images, &CreationDate)[-1].ImageId' \
  --output text)
 
echo "Using AMI: $AMI_ID"
 
# Launch the instance
aws ec2 run-instances \
  --image-id $AMI_ID \
  --instance-type t3.large \
  --key-name your-key-pair-name \
  --security-groups semaphor-sg \
  --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"VolumeSize":50,"VolumeType":"gp3"}}]' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=Semaphor}]' \
  --query 'Instances[0].InstanceId' \
  --output text

Save the Instance ID from the output - you'll need it for the next steps.

Using AWS Console:

Open the EC2 Console and click Launch Instance
Configure:
- Name: Semaphor
- AMI: Amazon Linux 2023
- Instance type: t3.large (2 vCPU, 8GB RAM)
- Key pair: Select your existing key pair
- Security group: Select semaphor-sg
- Storage: 50 GB gp3

For production workloads with many concurrent users, consider t3.xlarge or m5.large instances.

Step 3: Allocate Elastic IP

An Elastic IP ensures your instance keeps the same public IP address even after restarts.

Using AWS CLI:

# Allocate an Elastic IP
ALLOCATION_ID=$(aws ec2 allocate-address \
  --domain vpc \
  --query 'AllocationId' \
  --output text)
 
echo "Allocation ID: $ALLOCATION_ID"
 
# Get your instance ID (if you don't have it)
INSTANCE_ID=$(aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=Semaphor" "Name=instance-state-name,Values=running" \
  --query 'Reservations[0].Instances[0].InstanceId' \
  --output text)
 
echo "Instance ID: $INSTANCE_ID"
 
# Associate the Elastic IP with your instance
aws ec2 associate-address \
  --instance-id $INSTANCE_ID \
  --allocation-id $ALLOCATION_ID
 
# Get the Elastic IP address
ELASTIC_IP=$(aws ec2 describe-addresses \
  --allocation-ids $ALLOCATION_ID \
  --query 'Addresses[0].PublicIp' \
  --output text)
 
echo "Elastic IP: $ELASTIC_IP"

Using AWS Console:

Go to EC2 > Elastic IPs
Click Allocate Elastic IP address
Click Allocate
Select the new Elastic IP, click Actions > Associate Elastic IP address
Choose your Semaphor instance and click Associate

Note the Elastic IP address - you'll need it for DNS configuration.

Step 4: Configure DNS

Your domain must point to your EC2 instance's Elastic IP address. This is required for:

Accessing Semaphor via your domain name
SSL certificate generation (Let's Encrypt validates domain ownership)
Kinde OAuth callbacks (must match your configured domain exactly)

DNS misconfiguration is a common source of issues. The domain must resolve to your Elastic IP before you can set up SSL in Step 10.

Option A: AWS Route 53

If your domain is hosted in Route 53:

Using AWS CLI:

# Step 1: Find your hosted zone ID
aws route53 list-hosted-zones \
  --query 'HostedZones[*].{Name:Name,Id:Id}' \
  --output table
 
# Step 2: Create the A record
# Replace these values with your own:
HOSTED_ZONE_ID="Z0123456789EXAMPLE"  # From step 1 (just the ID part, e.g., Z0123...)
DOMAIN="semaphor.yourdomain.com"      # Your full subdomain
ELASTIC_IP="54.82.8.111"              # Your Elastic IP from Step 3
 
aws route53 change-resource-record-sets \
  --hosted-zone-id $HOSTED_ZONE_ID \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "'"$DOMAIN"'",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [{"Value": "'"$ELASTIC_IP"'"}]
      }
    }]
  }'

Using AWS Console:

Go to Route 53 > Hosted zones
Click on your domain (e.g., yourdomain.com)
Click Create record
Configure:
- Record name: semaphor (or your chosen subdomain)
- Record type: A
- Value: Your Elastic IP address
- TTL: 300 (5 minutes)
Click Create records

Option B: Other DNS Providers

For Cloudflare, GoDaddy, Namecheap, or other providers:

Log into your DNS provider's dashboard
Navigate to DNS management for your domain
Create a new A record:

Setting	Value
Type	A
Name/Host	`semaphor` (subdomain) or `@` (root domain)
Value/Points to	Your Elastic IP (e.g., `54.82.8.111`)
TTL	300 or "Auto"

If using Cloudflare, disable the orange cloud (proxy) initially. You can enable it later after Semaphor is working.

Verify DNS Configuration

Wait for propagation, then verify the DNS record is working:

# Check if DNS resolves to your IP
dig +short semaphor.yourdomain.com
 
# Expected output: your Elastic IP
# 54.82.8.111

If you get no output or a different IP, the DNS hasn't propagated yet. You can also check from multiple locations:

# Use Google's DNS to verify
dig +short semaphor.yourdomain.com @8.8.8.8

DNS Troubleshooting

Problem	Solution
`dig` returns nothing	DNS not propagated yet - wait 5-15 minutes
Wrong IP returned	Check for conflicting A records in your DNS provider
SSL setup fails later	Ensure DNS is resolving before running Certbot
Works locally but not elsewhere	Local DNS cache - try `dig @8.8.8.8` to test

Do not proceed to Step 10 (SSL setup) until DNS is working. Certbot will fail if the domain doesn't resolve to your server.

Step 5: Connect via SSH

ssh -i your-key.pem ec2-user@<your-elastic-ip>

Replace your-key.pem with your key file path and <your-elastic-ip> with your Elastic IP.

Step 6: Install Dependencies

Run these commands to install Docker, Docker Compose, nginx, and Certbot.

Amazon Linux 2023 uses dnf instead of yum. The commands below are for AL2023 - if using Amazon Linux 2, replace dnf with yum and use amazon-linux-extras for nginx.

# Update system packages
sudo dnf update -y
 
# Install Docker
sudo dnf install -y docker
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker ec2-user
 
# Install Docker Compose
sudo mkdir -p /usr/local/lib/docker/cli-plugins
sudo curl -SL "https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64" \
  -o /usr/local/lib/docker/cli-plugins/docker-compose
sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
sudo ln -sf /usr/local/lib/docker/cli-plugins/docker-compose /usr/local/bin/docker-compose
 
# Install nginx
sudo dnf install -y nginx
sudo systemctl enable nginx
 
# Install Certbot for SSL (via pip on Amazon Linux 2023)
sudo dnf install -y python3-pip augeas-libs
sudo pip3 install certbot certbot-nginx

Log out and back in for Docker group membership to take effect:

exit
ssh -i your-key.pem ec2-user@<your-elastic-ip>

Verify installations:

docker --version          # Should show Docker version 25+
docker-compose --version  # Should show Docker Compose version 5+
nginx -v                  # Should show nginx version 1.28+
certbot --version         # Should show certbot 4+

Step 7: Set Up Semaphor

Create the Semaphor directory and configuration files:

sudo mkdir -p /opt/semaphor
cd /opt/semaphor

Create docker-compose.yml:

sudo tee /opt/semaphor/docker-compose.yml > /dev/null << 'EOF'
services:
  api-service:
    image: semaphorstack/api-service:latest
    env_file:
      - ./semaphor.env
    volumes:
      - ./semaphor.env:/app/.env
      - semaphor-data:/data
    ports:
      - '127.0.0.1:3000:3000'
    healthcheck:
      test: ['CMD-SHELL', 'node -e "http=require(''http''); http.get(''http://localhost:3000/api/health'', r=>process.exit(r.statusCode===200?0:1)).on(''error'', ()=>process.exit(1))"']
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    restart: unless-stopped
 
  data-service:
    image: semaphorstack/data-service:latest
    depends_on:
      api-service:
        condition: service_healthy
    environment:
      PORT: 80
      SIDECAR_URL: http://data-service-sidecar:8080
    ports:
      - '127.0.0.1:8080:80'
    healthcheck:
      test: ['CMD', 'python', '-c', "import urllib.request,sys; r=urllib.request.urlopen('http://localhost:80/health', timeout=2); sys.exit(0 if r.getcode()==200 else 1)"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    restart: unless-stopped
 
  data-service-sidecar:
    image: semaphorstack/data-service-sidecar:latest
    depends_on:
      data-service:
        condition: service_healthy
    environment:
      PORT: 8080
    ports:
      - '127.0.0.1:8081:8080'
    healthcheck:
      test: ['CMD', 'python', '-c', "import urllib.request,sys; r=urllib.request.urlopen('http://localhost:8080/health', timeout=2); sys.exit(0 if r.getcode()==200 else 1)"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s
    restart: unless-stopped
 
volumes:
  semaphor-data:
 
networks:
  default:
    name: semaphor-network
    driver: bridge
EOF

Create semaphor.env with your credentials:

sudo tee /opt/semaphor/semaphor.env > /dev/null << 'EOF'
# ===== REQUIRED: License Key =====
LICENSE_KEY=SEMA-your-license-key-here
 
# ===== REQUIRED: Authentication (from Kinde) =====
KINDE_CLIENT_ID=your_kinde_client_id
KINDE_CLIENT_SECRET=your_kinde_client_secret
KINDE_ISSUER_URL=https://your-subdomain.kinde.com
KINDE_SITE_URL=https://semaphor.yourdomain.com
KINDE_POST_LOGOUT_REDIRECT_URL=https://semaphor.yourdomain.com
KINDE_POST_LOGIN_REDIRECT_URL=https://semaphor.yourdomain.com/project
 
# ===== REQUIRED: App URL =====
# APP_URL: Your external domain (MUST change this)
# This URL is embedded in authentication tokens and used for all browser API calls
APP_URL=https://semaphor.yourdomain.com
 
# DATA_API_URL: Internal Docker network URL (do NOT change this)
# This is how api-service communicates with data-service inside Docker
DATA_API_URL=http://data-service
 
# ===== App Secrets (generate random 32-character strings) =====
SYMMETRIC_ENCRYPTION_KEY=your-random-32-character-string1
ACCESS_TOKEN_SECRET=your-random-32-character-string2
 
# ===== OPTIONAL: External Services =====
# Leave blank to use embedded PostgreSQL and Redis
 
# Database (embedded if blank)
DATABASE_URL=
 
# Cache (embedded if blank)
REDIS_URL=
 
# Email notifications
RESEND_API_KEY=
 
# AI Features
OPENAI_API_KEY=
EOF

Edit the file with your actual values:

sudo nano /opt/semaphor/semaphor.env

Replace:

SEMA-your-license-key-here with your Semaphor license key
your_kinde_client_id, your_kinde_client_secret, your-subdomain with your Kinde credentials
semaphor.yourdomain.com with your actual domain name
Generate random 32-character strings for the encryption keys

Generate random strings with: openssl rand -hex 16

Understanding the URL variables:

Variable	Type	What to set
`APP_URL`	External	Your domain with `https://` (e.g., `https://semaphor.yourdomain.com`)
`DATA_API_URL`	Internal	Keep as `http://data-service` - this is Docker's internal DNS
`KINDE_*_URL`	External	Same domain as `APP_URL`

The APP_URL is critical - it gets embedded in authentication tokens at container startup. If set incorrectly, your browser will try to call localhost:3000 instead of your actual domain.

Step 8: Configure Kinde Callback URLs

In your Kinde dashboard, update your application settings:

Go to Settings > Applications > Your App > Authentication
Add these callback URLs:

Setting	Value
Allowed callback URLs	`https://semaphor.yourdomain.com/api/auth/kinde_callback`
Allowed logout redirect URLs	`https://semaphor.yourdomain.com`

Replace semaphor.yourdomain.com with your actual domain.

Step 9: Configure nginx

Create the initial nginx configuration:

sudo tee /etc/nginx/conf.d/semaphor.conf > /dev/null << 'EOF'
server {
    listen 80;
    listen [::]:80;
    server_name semaphor.yourdomain.com;
 
    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }
 
    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
        proxy_read_timeout 300s;
        client_max_body_size 100M;
 
        # IMPORTANT: Increase buffer sizes for Kinde auth headers
        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;
    }
}
EOF

The proxy_buffer_size settings are critical. Without them, you'll get 502 Bad Gateway errors when Kinde authentication sends large cookies/headers.

Edit the file and replace semaphor.yourdomain.com with your domain:

sudo nano /etc/nginx/conf.d/semaphor.conf

Create the Certbot webroot and start nginx:

sudo mkdir -p /var/www/certbot
sudo nginx -t
sudo systemctl start nginx

Step 10: Set Up SSL with Certbot

Verify your DNS is resolving correctly:

dig +short semaphor.yourdomain.com

This should return your Elastic IP. If not, wait for DNS propagation.

Request the SSL certificate:

sudo certbot --nginx -d semaphor.yourdomain.com --non-interactive --agree-tos --email your-email@example.com --redirect

Replace:

semaphor.yourdomain.com with your domain
your-email@example.com with your email for certificate notifications

Set up automatic certificate renewal:

echo "0 3 * * * root certbot renew --quiet --post-hook 'systemctl reload nginx'" | sudo tee /etc/cron.d/certbot-renew
sudo chmod 644 /etc/cron.d/certbot-renew

Step 11: Start Semaphor

Pull the Docker images and start Semaphor:

cd /opt/semaphor
sudo docker-compose pull
sudo docker-compose up -d

Wait about 60 seconds for all services to start. Check the status:

sudo docker-compose ps

All three services should show as "healthy" or "running".

Step 12: Verify Deployment

Check that all components are working:

# Check API service health
curl -s http://127.0.0.1:3000/api/health
 
# Check data service health
curl -s http://127.0.0.1:8080/health
 
# Check nginx is proxying correctly
curl -s -o /dev/null -w "%{http_code}" https://semaphor.yourdomain.com/api/health

Open your browser and navigate to:

https://semaphor.yourdomain.com/project

You should see the Semaphor login page.

Maintenance

Update Semaphor

Before upgrading, back up your data and review the Upgrade Guide for important information about database schema changes.

To update to a new version:

cd /opt/semaphor
sudo docker-compose pull
sudo docker-compose up -d

For the full upgrade procedure including backup, rollback, and production database considerations, see the Upgrade Guide.

View Logs

# All services
sudo docker-compose logs -f
 
# Specific service
sudo docker-compose logs -f api-service
 
# nginx
sudo tail -f /var/log/nginx/semaphor-access.log
sudo tail -f /var/log/nginx/semaphor-error.log

Backup Data

Create a backup of the Semaphor data volume:

cd /opt/semaphor
 
# Stop services
sudo docker-compose stop
 
# Backup the data volume
sudo docker run --rm \
  -v semaphor-data:/data:ro \
  -v /opt/semaphor/backups:/backup \
  alpine tar czf /backup/semaphor-backup-$(date +%Y%m%d).tar.gz -C /data .
 
# Backup configuration
sudo cp /opt/semaphor/semaphor.env /opt/semaphor/backups/semaphor.env.$(date +%Y%m%d)
 
# Restart services
sudo docker-compose up -d

Restore from Backup

cd /opt/semaphor
sudo docker-compose down
sudo docker volume rm semaphor-data
sudo docker volume create semaphor-data
 
sudo docker run --rm \
  -v semaphor-data:/data \
  -v /opt/semaphor/backups:/backup \
  alpine sh -c 'cd /data && tar xzf /backup/semaphor-backup-YYYYMMDD.tar.gz'
 
sudo cp /opt/semaphor/backups/semaphor.env.YYYYMMDD /opt/semaphor/semaphor.env
sudo docker-compose up -d

Replace YYYYMMDD with your backup date.

SSL Certificate Renewal

Certificates renew automatically via cron. To test renewal:

sudo certbot renew --dry-run

To manually renew:

sudo certbot renew
sudo systemctl reload nginx

Troubleshooting

502 Bad Gateway Error

This is the most common issue and is usually caused by one of these:

1. nginx buffer sizes too small (most common after login)

Kinde authentication sends large cookies/headers that exceed nginx's default buffer size. You'll see this in /var/log/nginx/error.log:

upstream sent too big header while reading response header from upstream

Fix: Ensure your nginx config includes these buffer settings:

# Check current nginx config
cat /etc/nginx/conf.d/semaphor.conf
 
# If missing proxy_buffer_size settings, update the config:
sudo nano /etc/nginx/conf.d/semaphor.conf

Add these lines inside the location / block:

proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

Then reload nginx:

sudo nginx -t && sudo systemctl reload nginx

2. Semaphor containers not running

# Check container status
cd /opt/semaphor
sudo docker-compose ps
 
# If containers show "Restarting", check logs
sudo docker-compose logs api-service --tail=50

3. nginx not running

sudo systemctl status nginx
sudo systemctl start nginx

Container Keeps Restarting (Exit Code 92)

This usually indicates a license key issue:

# Check logs for the specific error
sudo docker-compose logs api-service --tail=20

If you see License check failed: License expired or License invalid:

Get a new license key from support.semaphor.cloud
Update the license in semaphor.env:

sudo nano /opt/semaphor/semaphor.env
# Update the LICENSE_KEY line

Restart the containers:

cd /opt/semaphor
sudo docker-compose down
sudo docker-compose up -d

SSL Certificate Fails to Issue

Check DNS first:

dig +short semaphor.yourdomain.com
# Should return your Elastic IP

If DNS isn't resolving:

Wait 5-15 minutes for propagation
Verify the A record in your DNS provider
Try with Google's DNS: dig +short semaphor.yourdomain.com @8.8.8.8

Check port 80 is accessible:

# From your local machine
curl -I http://semaphor.yourdomain.com

If connection refused:

Verify security group has port 80 open
Check nginx is running: sudo systemctl status nginx

Check nginx error logs:

sudo tail -50 /var/log/nginx/error.log

Cannot Access Semaphor After Setup

Run this diagnostic script:

echo "=== Diagnostic Check ==="
 
echo -e "\n1. Container Status:"
cd /opt/semaphor && sudo docker-compose ps
 
echo -e "\n2. API Health (localhost):"
curl -s http://127.0.0.1:3000/api/health
 
echo -e "\n3. Data Service Health:"
curl -s http://127.0.0.1:8080/health
 
echo -e "\n4. nginx Status:"
sudo systemctl status nginx --no-pager | head -5
 
echo -e "\n5. nginx Config Test:"
sudo nginx -t
 
echo -e "\n6. Recent nginx Errors:"
sudo tail -10 /var/log/nginx/error.log

Browser Making API Calls to localhost:3000

Symptoms: After login, the browser makes API calls to http://localhost:3000/api/... instead of your domain. You'll see errors in the browser console.

Cause: The APP_URL environment variable is not set correctly, or the containers weren't restarted after changing it.

How it works: At container startup, APP_URL is injected into the application. The server embeds this URL into JWT tokens. The browser decodes the token and uses that URL for all API calls.

Fix:

Verify the env var is set correctly:

grep APP_URL /opt/semaphor/semaphor.env
# Should show: APP_URL=https://yourdomain.com

Restart containers (required after any env var change):

cd /opt/semaphor
sudo docker-compose down
sudo docker-compose up -d

Clear browser cache and reload the page (or use incognito mode)

Advanced debugging: Decode the JWT token to verify the URL:

# Get the token from browser dev tools (Network tab > any API request > Authorization header)
# Then decode the payload (middle part of JWT):
echo "PASTE_JWT_TOKEN_HERE" | cut -d'.' -f2 | base64 -d 2>/dev/null | jq .apiServiceUrl

Authentication Redirect Fails

Symptoms: Login redirects to Kinde but returns with an error, or you get stuck in a redirect loop.

Check Kinde callback URLs:

In your Kinde dashboard, verify:

Callback URL: https://semaphor.yourdomain.com/api/auth/kinde_callback (exact match!)
Logout redirect: https://semaphor.yourdomain.com

Check semaphor.env URLs:

grep KINDE /opt/semaphor/semaphor.env

All URLs must:

Use https:// (not http://)
Match your actual domain exactly
Have no trailing slashes

Common mistakes:

http:// instead of https://
Typo in domain name
Missing /api/auth/kinde_callback in Kinde callback URL
Trailing slash mismatch

Services Start But Health Checks Fail

Wait longer: The API service needs ~60 seconds to fully initialize (embedded PostgreSQL and Redis start inside the container).

# Watch the logs
sudo docker-compose logs -f api-service

Look for:

Database connected
Mode: DB=embedded, Redis=embedded
App: starting on http://0.0.0.0:3000

Check resource usage:

# Memory - need at least 4GB
free -h
 
# Disk space
df -h /
 
# Docker disk usage
docker system df

Important Notes

Amazon Linux 2023 vs Amazon Linux 2

Task	Amazon Linux 2023	Amazon Linux 2
Package manager	`dnf`	`yum`
Install nginx	`sudo dnf install nginx`	`sudo amazon-linux-extras install nginx1`
Install Certbot	`sudo pip3 install certbot certbot-nginx`	`sudo amazon-linux-extras install epel && sudo yum install certbot`

Critical nginx Configuration

The proxy_buffer_size settings are required for Kinde authentication to work. Without them, you'll get 502 errors after login:

proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

After Certbot Runs

Certbot modifies your nginx config to add SSL settings. You must re-add the proxy buffer settings if they get removed:

# Check if buffer settings are present
grep proxy_buffer /etc/nginx/conf.d/semaphor.conf
 
# If not, edit and add them back
sudo nano /etc/nginx/conf.d/semaphor.conf
sudo nginx -t && sudo systemctl reload nginx

License Key Expiration

License keys have expiration dates. If your containers keep restarting with exit code 92, check the logs:

sudo docker-compose logs api-service | grep -i license

Docker Ports

The docker-compose.yml binds ports to 127.0.0.1 only (not 0.0.0.0). This is intentional for security - all external traffic goes through nginx.

ports:
  - '127.0.0.1:3000:3000'  # Only accessible from localhost

Environment Variable URLs

All URLs in semaphor.env must:

Use https:// in production
Match your domain exactly
Be consistent across all *_URL variables

# Quick check for URL consistency
grep -E "URL|SITE" /opt/semaphor/semaphor.env

Container Restart Required After Env Changes

Environment variables are read when containers start. If you change semaphor.env, you must restart the containers for changes to take effect:

cd /opt/semaphor
sudo docker-compose down
sudo docker-compose up -d

This is especially important for APP_URL since it's injected into the application at container startup and embedded in JWT tokens.

Internal vs External URLs

URL Type	Example	Purpose
Internal	`http://data-service`	Docker container-to-container communication
External	`https://semaphor.yourdomain.com`	Browser-to-server, OAuth callbacks, email links

DATA_API_URL should stay as http://data-service (Docker DNS resolves this internally)
All other URLs should use your external domain with https://

Useful Commands Reference

Container Management

cd /opt/semaphor
 
# Start services
sudo docker-compose up -d
 
# Stop services
sudo docker-compose down
 
# Restart services
sudo docker-compose restart
 
# View status
sudo docker-compose ps
 
# View logs (all services)
sudo docker-compose logs -f
 
# View logs (specific service)
sudo docker-compose logs -f api-service
 
# Pull latest images
sudo docker-compose pull

Health Checks

# API service
curl -s http://127.0.0.1:3000/api/health | jq
 
# Data service
curl -s http://127.0.0.1:8080/health
 
# Via HTTPS (external)
curl -s https://yourdomain.com/api/health

nginx Commands

# Test configuration
sudo nginx -t
 
# Reload (apply config changes)
sudo systemctl reload nginx
 
# Restart
sudo systemctl restart nginx
 
# View error logs
sudo tail -f /var/log/nginx/error.log
 
# View access logs
sudo tail -f /var/log/nginx/access.log

SSL/Certbot Commands

# View certificates
sudo certbot certificates
 
# Test renewal
sudo certbot renew --dry-run
 
# Force renewal
sudo certbot renew --force-renewal
sudo systemctl reload nginx

System Diagnostics

# Memory usage
free -h
 
# Disk usage
df -h
 
# Docker disk usage
docker system df
 
# Running processes
htop  # or top

Next Steps

Configure data sources to connect your databases
Set up report scheduling for automated reports
Enable AI features by adding your OpenAI API key

Deploy on AWS EC2

On this page