foxhunt/systemd

Foxhunt HFT SystemD Services

Production-ready SystemD service units for the Foxhunt High-Frequency Trading system.

Service Architecture

PostgreSQL Database
        |
        v
foxhunt-trading.service (Cores 0-1, High Priority)
        |
        v  
foxhunt-backtesting.service (Cores 2-3, Normal Priority)
        |
        v
foxhunt-tli.service (Cores 4-5, Low Priority)

Services Overview

Service	Purpose	CPU Cores	Priority	Memory Limit
foxhunt-trading	Core trading engine	0-1	High (-10)	2GB
foxhunt-backtesting	Strategy testing	2-3	Normal (0)	4GB
foxhunt-tli	Terminal interface	4-5	Low (+5)	1GB

Installation

1. Create Service User

sudo useradd -r -s /bin/false foxhunt
sudo mkdir -p /opt/foxhunt/{bin,data,logs,config,tli_config,backtesting_results}
sudo chown -R foxhunt:foxhunt /opt/foxhunt

2. Install Service Files

sudo cp systemd/*.service /etc/systemd/system/
sudo cp systemd/*.target /etc/systemd/system/
sudo chmod 644 /etc/systemd/system/foxhunt*
sudo systemctl daemon-reload

3. Enable Services

sudo systemctl enable foxhunt.target
sudo systemctl enable foxhunt-trading.service
sudo systemctl enable foxhunt-backtesting.service
sudo systemctl enable foxhunt-tli.service

Usage

Start/Stop All Services

# Start entire system
sudo systemctl start foxhunt.target

# Stop entire system
sudo systemctl stop foxhunt.target

# Check system status
sudo systemctl status foxhunt.target

Individual Service Management

# Start individual service
sudo systemctl start foxhunt-trading

# Check service status
sudo systemctl status foxhunt-trading

# View service logs
sudo journalctl -u foxhunt-trading -f

Monitoring Commands

# View all service dependencies
sudo systemctl list-dependencies foxhunt.target

# Check if all services are active
sudo systemctl is-active foxhunt-trading foxhunt-backtesting foxhunt-tli

# View combined logs
sudo journalctl -u foxhunt-trading -u foxhunt-backtesting -u foxhunt-tli -f

Performance Configuration

CPU Affinity

• Trading Service: Cores 0-1 (dedicated for ultra-low latency)
• Backtesting Service: Cores 2-3 (isolated from trading)
• TLI Service: Cores 4-5 (UI responsiveness)

Scheduling Priorities

• Trading: Nice -10 (highest priority)
• Backtesting: Nice 0 (normal priority)
• TLI: Nice +5 (lower priority)

Memory Limits

• Trading: 2GB (conservative for stability)
• Backtesting: 4GB (data-intensive operations)
• TLI: 1GB (lightweight client interface)

Security Features

Process Isolation

• Dedicated service user with minimal privileges
• No new privileges allowed
• Protected system directories
• Restricted network access (localhost only)

Resource Limits

• File descriptor limits (65536)
• Process limits
• Memory limits per service
• Network address family restrictions

Troubleshooting

Common Issues

1. Service fails to start

   sudo journalctl -u foxhunt-trading --no-pager
   sudo systemctl status foxhunt-trading -l
   

2. Database connection issues

   sudo systemctl status postgresql
   sudo -u postgres psql -c "\l" # List databases
   

3. CPU affinity problems

   # Check CPU assignment
   ps -o pid,psr,comm -C trading_service
   

4. Memory issues

   # Check memory usage
   sudo systemctl status foxhunt-trading
   

Log Locations

• SystemD logs: journalctl -u <service-name>
• Application logs: /opt/foxhunt/logs/
• System logs: /var/log/syslog

Production Considerations

Before Deployment

1. Set up PostgreSQL database and user
2. Configure firewall rules for gRPC ports
3. Set up monitoring and alerting
4. Test service restart behavior
5. Validate CPU affinity assignments

Monitoring Setup

1. CPU usage per service and assigned cores
2. Memory consumption tracking
3. Database connection health
4. Service restart frequency
5. gRPC connection status between services

Backup Strategy

1. Database backups (PostgreSQL)
2. Configuration file backups
3. Service state snapshots
4. Log rotation and archival

Security Checklist

• Service user created with minimal privileges
• File permissions set correctly (644 for service files)
• Network access restricted to localhost
• System directories protected
• Resource limits configured
• Audit trails enabled via systemd logging

Support

For issues and questions:

1. Check service logs: sudo journalctl -u <service-name> -f
2. Verify service status: sudo systemctl status <service-name>
3. Review configuration files for syntax errors
4. Check PostgreSQL connectivity and permissions