Auto-Starting Services with Supervisord on Reboot

When running services on Linux servers, ensuring they automatically restart after a system reboot is critical for production reliability. Supervisord provides a robust process control system that handles automatic restarts, logging, and process monitoring.

Supervisord Configuration

Basic Service Configuration

Create or modify the configuration file at /etc/supervisor/conf.d/default.conf:

[program:api-service]
command=java -XX:+UseStringDeduplication \
    -Dspring.profiles.active=production \
    -Xms512m -Xmx2048m \
    -jar /opt/services/api-service.jar
directory=/opt/services
autostart=true
autorestart=true
startretries=3
stderr_logfile=/var/log/api-service/error.log
stdout_logfile=/var/log/api-service/output.log
user=appuser
environment=JAVA_HOME="/usr/lib/jvm/java-17-openjdk"

Configuration Parameters Explained

┌────────────────────────────────────────────────────────────┐
│              SUPERVISORD CONFIGURATION OPTIONS              │
├────────────────────────────────────────────────────────────┤
│                                                            │
│  autostart=true     → Start when supervisord starts        │
│  autorestart=true   → Restart if process exits             │
│  startretries=3     → Retry attempts before giving up      │
│  startsecs=10       → Seconds to consider "started"        │
│  stopwaitsecs=30    → Wait time for graceful shutdown      │
│  stopsignal=TERM    → Signal to send for stop              │
│                                                            │
└────────────────────────────────────────────────────────────┘

Multiple Services Configuration

For managing multiple Java services on a single server:

; /etc/supervisor/conf.d/services.conf

[program:u-service]
command=java -XX:+UseStringDeduplication \
    -Dspring.profiles.active=%(ENV_ENVIRONMENT)s \
    -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 \
    -jar /opt/services/u-service.jar
directory=/opt/services
autostart=true
autorestart=true
startretries=3
stderr_logfile=/var/log/u-service/error.log
stdout_logfile=/var/log/u-service/output.log
user=ubuntu
priority=100

[program:n-service]
command=java -Dspring.profiles.active=%(ENV_ENVIRONMENT)s \
    -Xms256m -Xmx1024m \
    -jar /opt/services/n-service.jar
directory=/opt/services
autostart=true
autorestart=true
startretries=3
stderr_logfile=/var/log/n-service/error.log
stdout_logfile=/var/log/n-service/output.log
user=ubuntu
priority=200

[program:s-service]
command=java -Dspring.profiles.active=%(ENV_ENVIRONMENT)s \
    -jar /opt/services/s-service.jar
directory=/opt/services
autostart=true
autorestart=true
startretries=3
stderr_logfile=/var/log/s-service/error.log
stdout_logfile=/var/log/s-service/output.log
user=ubuntu
priority=300

Service Startup Order

┌─────────────────────────────────────────────────────────────┐
│                  SERVICE STARTUP SEQUENCE                    │
│                    (Lower priority = First)                  │
└─────────────────────────────────────────────────────────────┘

     priority=100          priority=200          priority=300
          │                     │                     │
          ▼                     ▼                     ▼
    ┌──────────┐          ┌──────────┐          ┌──────────┐
    │  Uxxx    │    →     │ Nxxxx.   │    →     │Sxxxxxxxx │
    │ Service  │          │ Service  │          │ Service  │
    └──────────┘          └──────────┘          └──────────┘
         │                     │                     │
         └─────────────────────┴─────────────────────┘
                              │
                    All services running

Ensuring Supervisord Starts on Boot

Check Supervisor Status

# Check if supervisor service exists
systemctl status supervisor

# Check if it's enabled for boot
systemctl is-enabled supervisor

Enable Supervisor at Boot

# Enable supervisor to start on system boot
sudo systemctl enable supervisor

# Start supervisor now if not running
sudo systemctl start supervisor

# Verify status
systemctl | grep supervisor

Managing Services with Supervisorctl

Common Commands

# List all managed processes
sudo supervisorctl status

# Start a specific service
sudo supervisorctl start api-service

# Stop a specific service
sudo supervisorctl stop api-service

# Restart a specific service
sudo supervisorctl restart api-service

# Reload configuration (after changes)
sudo supervisorctl reread
sudo supervisorctl update

# Restart all services
sudo supervisorctl restart all

Interactive Mode

$ sudo supervisorctl
supervisor> status
api-service                      RUNNING   pid 1234, uptime 2:30:45
n-service             RUNNING   pid 1235, uptime 2:30:42
s-service                RUNNING   pid 1236, uptime 2:30:40

supervisor> tail -f api-service stdout
[Following log output...]

supervisor> stop api-service
api-service: stopped

supervisor> start api-service
api-service: started

supervisor> exit

Log Rotation Configuration

Prevent log files from consuming all disk space:

[program:api-service]
command=java -jar /opt/services/api-service.jar
stdout_logfile=/var/log/api-service/output.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stderr_logfile=/var/log/api-service/error.log
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10

Process Groups

Group related services for batch operations:

[group:backend-services]
programs=u-service,n-service,s-service
priority=999

[group:frontend-services]
programs=web-server,api-gateway
priority=100

# Restart all backend services
sudo supervisorctl restart backend-services:*

# Stop all frontend services
sudo supervisorctl stop frontend-services:*

Health Check Script

Create a monitoring script for service health:

#!/bin/bash
# /opt/scripts/check_services.sh

SERVICES=("u-service" "n-service" "s-service")
ALERT_THRESHOLD=3

for service in "${SERVICES[@]}"; do
    status=$(sudo supervisorctl status "$service" | awk '{print $2}')
    
    if [ "$status" != "RUNNING" ]; then
        echo "ALERT: $service is $status"
        
        # Attempt restart
        sudo supervisorctl restart "$service"
        
        # Wait and recheck
        sleep 10
        new_status=$(sudo supervisorctl status "$service" | awk '{print $2}')
        
        if [ "$new_status" != "RUNNING" ]; then
            echo "CRITICAL: $service failed to restart"
            # Send alert to monitoring system
        fi
    fi
done

Comparison: Supervisord vs Systemd

Best Practices

┌─────────────────────────────────────────────────────────────┐
│                    SUPERVISORD BEST PRACTICES                │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ✓ Always set autostart=true for production services        │
│  ✓ Configure appropriate startretries (3-5 recommended)     │
│  ✓ Set startsecs high enough for JVM warmup                 │
│  ✓ Use priority to control startup order                    │
│  ✓ Enable log rotation to prevent disk exhaustion           │
│  ✓ Run services as non-root user when possible              │
│  ✓ Set environment variables in config, not globally        │
│  ✓ Test configuration changes in staging first              │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Supervisord provides a reliable, easy-to-configure process management solution that ensures your Java services survive reboots and unexpected crashes, making it an essential tool for production deployments.