foxhunt/systemd/README.md

# Foxhunt HFT SystemD Services

Production-ready SystemD service units for the Foxhunt High-Frequency Trading system.

## Service Architecture

```
PostgreSQL Database
        |
        v
foxhunt-trading.service (Cores 0-1, High Priority)
        |
        v  
foxhunt-backtesting.service (Cores 2-3, Normal Priority)
        |
        v
foxhunt-tli.service (Cores 4-5, Low Priority)
```

## Services Overview

| Service | Purpose | CPU Cores | Priority | Memory Limit |
|---------|---------|-----------|----------|--------------|
| foxhunt-trading | Core trading engine | 0-1 | High (-10) | 2GB |
| foxhunt-backtesting | Strategy testing | 2-3 | Normal (0) | 4GB |
| foxhunt-tli | Terminal interface | 4-5 | Low (+5) | 1GB |

## Installation

### 1. Create Service User
```bash
sudo useradd -r -s /bin/false foxhunt
sudo mkdir -p /opt/foxhunt/{bin,data,logs,config,tli_config,backtesting_results}
sudo chown -R foxhunt:foxhunt /opt/foxhunt
```

### 2. Install Service Files
```bash
sudo cp systemd/*.service /etc/systemd/system/
sudo cp systemd/*.target /etc/systemd/system/
sudo chmod 644 /etc/systemd/system/foxhunt*
sudo systemctl daemon-reload
```

### 3. Enable Services
```bash
sudo systemctl enable foxhunt.target
sudo systemctl enable foxhunt-trading.service
sudo systemctl enable foxhunt-backtesting.service
sudo systemctl enable foxhunt-tli.service
```

## Usage

### Start/Stop All Services
```bash
# Start entire system
sudo systemctl start foxhunt.target

# Stop entire system
sudo systemctl stop foxhunt.target

# Check system status
sudo systemctl status foxhunt.target
```

### Individual Service Management
```bash
# Start individual service
sudo systemctl start foxhunt-trading

# Check service status
sudo systemctl status foxhunt-trading

# View service logs
sudo journalctl -u foxhunt-trading -f
```

### Monitoring Commands
```bash
# View all service dependencies
sudo systemctl list-dependencies foxhunt.target

# Check if all services are active
sudo systemctl is-active foxhunt-trading foxhunt-backtesting foxhunt-tli

# View combined logs
sudo journalctl -u foxhunt-trading -u foxhunt-backtesting -u foxhunt-tli -f
```

## Performance Configuration

### CPU Affinity
- **Trading Service**: Cores 0-1 (dedicated for ultra-low latency)
- **Backtesting Service**: Cores 2-3 (isolated from trading)
- **TLI Service**: Cores 4-5 (UI responsiveness)

### Scheduling Priorities
- **Trading**: Nice -10 (highest priority)
- **Backtesting**: Nice 0 (normal priority)
- **TLI**: Nice +5 (lower priority)

### Memory Limits
- **Trading**: 2GB (conservative for stability)
- **Backtesting**: 4GB (data-intensive operations)
- **TLI**: 1GB (lightweight client interface)

## Security Features

### Process Isolation
- Dedicated service user with minimal privileges
- No new privileges allowed
- Protected system directories
- Restricted network access (localhost only)

### Resource Limits
- File descriptor limits (65536)
- Process limits
- Memory limits per service
- Network address family restrictions

## Troubleshooting

### Common Issues

1. **Service fails to start**
   ```bash
   sudo journalctl -u foxhunt-trading --no-pager
   sudo systemctl status foxhunt-trading -l
   ```

2. **Database connection issues**
   ```bash
   sudo systemctl status postgresql
   sudo -u postgres psql -c "\l" # List databases
   ```

3. **CPU affinity problems**
   ```bash
   # Check CPU assignment
   ps -o pid,psr,comm -C trading_service
   ```

4. **Memory issues**
   ```bash
   # Check memory usage
   sudo systemctl status foxhunt-trading
   ```

### Log Locations
- SystemD logs: `journalctl -u <service-name>`
- Application logs: `/opt/foxhunt/logs/`
- System logs: `/var/log/syslog`

## Production Considerations

### Before Deployment
1. Set up PostgreSQL database and user
2. Configure firewall rules for gRPC ports
3. Set up monitoring and alerting
4. Test service restart behavior
5. Validate CPU affinity assignments

### Monitoring Setup
1. CPU usage per service and assigned cores
2. Memory consumption tracking
3. Database connection health
4. Service restart frequency
5. gRPC connection status between services

### Backup Strategy
1. Database backups (PostgreSQL)
2. Configuration file backups
3. Service state snapshots
4. Log rotation and archival

## Security Checklist

- [ ] Service user created with minimal privileges
- [ ] File permissions set correctly (644 for service files)
- [ ] Network access restricted to localhost
- [ ] System directories protected
- [ ] Resource limits configured
- [ ] Audit trails enabled via systemd logging

## Support

For issues and questions:
1. Check service logs: `sudo journalctl -u <service-name> -f`
2. Verify service status: `sudo systemctl status <service-name>`
3. Review configuration files for syntax errors
4. Check PostgreSQL connectivity and permissions