foxhunt/MAINTENANCE_CHECKLIST.md

# Maintenance Checklist - Foxhunt HFT Trading System

**Version**: 1.0.0
**Last Updated**: 2025-10-07
**Purpose**: Regular maintenance tasks to ensure system reliability and performance

---

## 📅 Daily Tasks (15-20 minutes)

### Morning System Health Check

**Time**: 08:00 UTC (before market open)
**Duration**: 5 minutes
**Responsible**: On-call engineer

```bash
# 1. Run comprehensive health check
./deployment/health_check.sh --mode comprehensive

# 2. Check all services running
docker-compose ps  # Docker
sudo systemctl status foxhunt-*  # Bare-metal

# 3. Verify metrics collection
for port in 9091 9092 9093 9094; do
  curl -s http://localhost:$port/metrics | grep -q "foxhunt_" || echo "⚠️ Port $port metrics missing"
done

# 4. Check database connectivity
psql $DATABASE_URL -c "SELECT count(*) FROM orders WHERE created_at > NOW() - INTERVAL '24 hours';"

# 5. Check GPU status (ML services)
nvidia-smi  # Bare-metal
docker-compose exec ml_training_service nvidia-smi  # Docker
```

**Success Criteria**:
- [ ] All services status: HEALTHY
- [ ] All metrics endpoints responding
- [ ] Database connected, <100ms query time
- [ ] GPU detected and accessible
- [ ] No ERROR logs in last 24 hours

---

### Log Review

**Time**: 09:00 UTC
**Duration**: 10 minutes
**Responsible**: Operations team

```bash
# 1. Check for errors in last 24 hours
journalctl -u foxhunt-* --since "24 hours ago" | grep -i error > /tmp/daily-errors.log

# 2. Review error count
ERROR_COUNT=$(wc -l < /tmp/daily-errors.log)
echo "Errors in last 24h: $ERROR_COUNT"

# 3. Check for panics/crashes
journalctl -u foxhunt-* --since "24 hours ago" | grep -i panic

# 4. Review audit logs
psql $DATABASE_URL -c "SELECT action, count(*) FROM audit_logs WHERE created_at > NOW() - INTERVAL '24 hours' GROUP BY action ORDER BY count DESC LIMIT 10;"

# 5. Check for unauthorized access attempts
psql $DATABASE_URL -c "SELECT * FROM audit_logs WHERE action = 'login_failed' AND created_at > NOW() - INTERVAL '24 hours';"
```

**Action Items**:
- If ERROR_COUNT > 10: Investigate root cause
- If panics detected: File incident report
- If unauthorized access: Escalate to security team

---

### Backup Verification

**Time**: 10:00 UTC
**Duration**: 5 minutes
**Responsible**: Database admin

```bash
# 1. Verify last backup completed
aws s3 ls s3://foxhunt-backups/postgres/ | tail -5

# 2. Check backup age
LATEST_BACKUP=$(aws s3 ls s3://foxhunt-backups/postgres/ | tail -1 | awk '{print $4}')
echo "Latest backup: $LATEST_BACKUP"

# 3. Verify backup integrity
aws s3 cp s3://foxhunt-backups/postgres/$LATEST_BACKUP /tmp/verify.dump
pg_restore --list /tmp/verify.dump > /dev/null && echo "✅ Backup valid" || echo "❌ Backup corrupted"
rm /tmp/verify.dump

# 4. Check configuration backups
aws s3 ls s3://foxhunt-backups/config/ | tail -1

# 5. Verify model backups
aws s3 ls s3://foxhunt-models/ --recursive | wc -l
```

**Success Criteria**:
- [ ] Latest backup <24 hours old
- [ ] Backup integrity verified
- [ ] Configuration backup exists
- [ ] Model backups accessible

---

## 📅 Weekly Tasks (1-2 hours)

### Sunday Maintenance Window

**Time**: Sunday 02:00-04:00 UTC (low trading volume)
**Duration**: 1-2 hours
**Responsible**: DevOps team

#### 1. Database Maintenance (30 minutes)

```bash
# 1. Vacuum and analyze
psql $DATABASE_URL <<EOF
VACUUM ANALYZE orders;
VACUUM ANALYZE positions;
VACUUM ANALYZE market_data;
VACUUM ANALYZE audit_logs;
REINDEX TABLE orders;
REINDEX TABLE positions;
EOF

# 2. Check for bloat
psql $DATABASE_URL -c "SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size FROM pg_tables WHERE schemaname = 'public' ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC LIMIT 10;"

# 3. Review slow queries
psql $DATABASE_URL -c "SELECT query, calls, mean_exec_time, max_exec_time FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 20;"

# 4. Check replication lag (if replicas configured)
psql $DATABASE_URL -c "SELECT client_addr, state, sync_state, replay_lag FROM pg_stat_replication;"

# 5. Archive old audit logs (>90 days)
psql $DATABASE_URL -c "DELETE FROM audit_logs WHERE created_at < NOW() - INTERVAL '90 days';"
```

#### 2. Log Rotation and Cleanup (15 minutes)

```bash
# 1. Archive logs older than 30 days
find /var/log/foxhunt -name "*.log" -mtime +30 -exec gzip {} \;

# 2. Upload archived logs to S3
aws s3 sync /var/log/foxhunt s3://foxhunt-backups/logs/ --exclude "*" --include "*.log.gz"

# 3. Delete logs older than 90 days
find /var/log/foxhunt -name "*.log.gz" -mtime +90 -delete

# 4. Clean journal logs
sudo journalctl --vacuum-time=30d

# 5. Docker log cleanup (if using Docker)
docker system prune -af --volumes --filter "until=720h"
```

#### 3. Performance Analysis (30 minutes)

```bash
# 1. Generate weekly performance report
./deployment/scripts/weekly-performance-report.sh

# 2. Review key metrics
curl http://localhost:9092/metrics | grep -E "(order_latency|throughput|error_rate)" > /tmp/weekly-metrics.txt

# 3. Compare with baseline
./deployment/scripts/compare-metrics.sh /tmp/weekly-metrics.txt /var/log/foxhunt/baseline-metrics.txt

# 4. Check for performance degradation
psql $DATABASE_URL -c "SELECT
  DATE(created_at) as date,
  percentile_cont(0.99) WITHIN GROUP (ORDER BY processing_time_ms) as p99_latency
FROM orders
WHERE created_at > NOW() - INTERVAL '7 days'
GROUP BY DATE(created_at)
ORDER BY date DESC;"

# 5. Update performance baselines (if improved)
cp /tmp/weekly-metrics.txt /var/log/foxhunt/baseline-metrics.txt
```

#### 4. Security Review (15 minutes)

```bash
# 1. Review failed login attempts
psql $DATABASE_URL -c "SELECT user_id, ip_address, count(*) FROM audit_logs WHERE action = 'login_failed' AND created_at > NOW() - INTERVAL '7 days' GROUP BY user_id, ip_address HAVING count(*) > 5;"

# 2. Check for unusual API usage
psql $DATABASE_URL -c "SELECT api_key, count(*) FROM api_requests WHERE created_at > NOW() - INTERVAL '7 days' GROUP BY api_key ORDER BY count DESC LIMIT 20;"

# 3. Review Vault access logs
vault audit list
vault read sys/audit/file/log | grep "$(date -d '7 days ago' +%Y-%m-%d)"

# 4. Check for expired secrets
vault list secret/foxhunt/ | while read secret; do
  vault read "secret/foxhunt/$secret" | grep ttl
done

# 5. Run security scan
./deployment/scripts/security-scan.sh
```

---

## 📅 Monthly Tasks (2-4 hours)

### First Sunday of Month

**Time**: Sunday 02:00-06:00 UTC
**Duration**: 2-4 hours
**Responsible**: DevOps team + Database admin

#### 1. System Updates (1 hour)

```bash
# 1. Check for security updates
sudo apt-get update
sudo apt-get upgrade -s | grep -i security

# 2. Apply security patches (requires downtime planning)
# Schedule maintenance window
# Update all systems
sudo apt-get upgrade -y

# 3. Update Rust dependencies
cargo update
cargo audit

# 4. Update Docker images
docker-compose pull

# 5. Update Kubernetes components (if applicable)
kubectl version
helm repo update
```

#### 2. Database Deep Maintenance (1 hour)

```bash
# 1. Full database backup
pg_dump -h localhost -U foxhunt -Fc foxhunt > /tmp/monthly-backup-$(date +%Y%m).dump

# 2. Upload to S3
aws s3 cp /tmp/monthly-backup-$(date +%Y%m).dump s3://foxhunt-backups/postgres/monthly/

# 3. Test restore (to separate instance)
createdb foxhunt_test
pg_restore -h localhost -U foxhunt -d foxhunt_test /tmp/monthly-backup-$(date +%Y%m).dump

# 4. Verify data integrity
psql foxhunt_test -c "SELECT count(*) FROM orders;" > /tmp/test-count.txt
psql foxhunt -c "SELECT count(*) FROM orders;" > /tmp/prod-count.txt
diff /tmp/test-count.txt /tmp/prod-count.txt

# 5. Cleanup test database
dropdb foxhunt_test
rm /tmp/monthly-backup-$(date +%Y%m).dump
```

#### 3. Capacity Planning (30 minutes)

```bash
# 1. Review storage growth
df -h
du -sh /var/lib/postgresql
du -sh /opt/foxhunt

# 2. Review memory trends
./deployment/scripts/memory-trend-analysis.sh --days 30

# 3. Review CPU utilization
./deployment/scripts/cpu-trend-analysis.sh --days 30

# 4. Review database size growth
psql $DATABASE_URL -c "SELECT
  schemaname,
  tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename) -
    LAG(pg_total_relation_size(schemaname||'.'||tablename)) OVER (PARTITION BY tablename ORDER BY NOW())) AS growth
FROM pg_tables
WHERE schemaname = 'public';"

# 5. Forecast capacity needs (next 3 months)
./deployment/scripts/capacity-forecast.sh
```

#### 4. Disaster Recovery Test (1 hour)

```bash
# 1. Test database failover (staging only!)
./deployment/scripts/test-database-failover.sh

# 2. Test service crash recovery
./deployment/scripts/test-service-crash-recovery.sh

# 3. Test network partition handling
./deployment/scripts/test-network-partition.sh

# 4. Test backup restoration
./deployment/scripts/test-backup-restore.sh

# 5. Document results
./deployment/scripts/generate-dr-report.sh > /var/log/foxhunt/dr-test-$(date +%Y%m).txt
```

#### 5. Certificate Renewal (15 minutes)

```bash
# 1. Check certificate expiration
echo | openssl s_client -servername api.foxhunt.trading -connect api.foxhunt.trading:443 2>/dev/null | openssl x509 -noout -dates

# 2. Renew if expiring in <30 days
sudo certbot renew --dry-run
sudo certbot renew

# 3. Reload services
sudo systemctl reload nginx

# 4. Verify new certificate
echo | openssl s_client -servername api.foxhunt.trading -connect api.foxhunt.trading:443 2>/dev/null | openssl x509 -noout -text | grep -A 2 Validity
```

---

## 📅 Quarterly Tasks (4-8 hours)

### First Sunday of Quarter

**Time**: Sunday 00:00-08:00 UTC
**Duration**: 4-8 hours
**Responsible**: Full DevOps team

#### 1. Major Version Updates (2-3 hours)

```bash
# 1. Plan upgrade path
# Review release notes for:
# - Rust
# - PostgreSQL
# - Redis
# - Docker
# - Kubernetes (if applicable)

# 2. Test in staging
# Deploy updates to staging environment
# Run full test suite
./tests/integration_tests/run_all.sh

# 3. Schedule production upgrade
# Create maintenance window
# Notify stakeholders

# 4. Backup everything
./deployment/scripts/full-system-backup.sh

# 5. Execute upgrade
./deployment/scripts/quarterly-upgrade.sh
```

#### 2. Security Audit (2 hours)

```bash
# 1. Dependency vulnerability scan
cargo audit
npm audit (if any JS components)

# 2. Container security scan
docker scan foxhunt-trading-service:latest
docker scan foxhunt-backtesting-service:latest
docker scan foxhunt-ml-training-service:latest
docker scan foxhunt-api-gateway:latest

# 3. Penetration testing (external vendor)
# Schedule pen test
# Review findings
# Implement fixes

# 4. Compliance review
# SOX compliance check
# MiFID II compliance check
# GDPR compliance check

# 5. Update security policies
vim /opt/foxhunt/docs/security-policy.md
```

#### 3. Performance Optimization (2 hours)

```bash
# 1. Profile critical paths
./deployment/scripts/profile-trading-service.sh
./deployment/scripts/profile-ml-service.sh

# 2. Identify bottlenecks
./deployment/scripts/identify-bottlenecks.sh

# 3. Database query optimization
psql $DATABASE_URL -c "SELECT query, mean_exec_time, calls FROM pg_stat_statements WHERE mean_exec_time > 100 ORDER BY mean_exec_time DESC LIMIT 50;"

# 4. Implement optimizations
# Review and apply performance improvements

# 5. Benchmark improvements
./tests/performance/benchmark-suite.sh
```

#### 4. Disaster Recovery Drill (1 hour)

```bash
# 1. Full DR scenario test
./deployment/scripts/dr-drill-full.sh

# 2. Test complete system restore from backup
./deployment/scripts/test-full-restore.sh

# 3. Measure RTO/RPO
./deployment/scripts/measure-rto-rpo.sh

# 4. Update DR documentation
vim /opt/foxhunt/docs/disaster-recovery.md

# 5. Review and update runbooks
vim /opt/foxhunt/PRODUCTION_DEPLOYMENT_RUNBOOK.md
```

---

## 📅 Annual Tasks (1-2 days)

### First Week of January

**Duration**: 1-2 days
**Responsible**: Entire engineering team

#### 1. Architecture Review

- [ ] Review system architecture
- [ ] Identify technical debt
- [ ] Plan major refactoring
- [ ] Update architecture diagrams
- [ ] Document lessons learned

#### 2. Security Assessment

- [ ] Full security audit (external vendor)
- [ ] Compliance certification renewal
- [ ] Update security policies
- [ ] Security training for team
- [ ] Incident response drill

#### 3. Capacity Planning

- [ ] Review 12-month growth trends
- [ ] Forecast infrastructure needs
- [ ] Budget planning for infrastructure
- [ ] Evaluate new technologies
- [ ] Plan scaling strategy

#### 4. Documentation Update

- [ ] Update all runbooks
- [ ] Update architecture documentation
- [ ] Update API documentation
- [ ] Update deployment guides
- [ ] Create training materials

---

## 📊 Maintenance Tracking

### Log Template

```bash
# /var/log/foxhunt/maintenance/YYYY-MM-DD.md

# Maintenance Log - YYYY-MM-DD

**Type**: Daily/Weekly/Monthly/Quarterly/Annual
**Start Time**: HH:MM UTC
**End Time**: HH:MM UTC
**Performed By**: [Name]

## Tasks Completed

- [ ] Task 1: Description
- [ ] Task 2: Description
- [ ] Task 3: Description

## Issues Found

1. Issue 1: Description and resolution
2. Issue 2: Description and resolution

## Metrics

- Service uptime: X%
- Average latency: Xμs
- Error rate: X%
- Database size: XGB

## Action Items

- [ ] Action 1: Owner, Due Date
- [ ] Action 2: Owner, Due Date

## Notes

Any additional observations or comments.
```

---

## 🚨 Maintenance Alerts

### Configure Monitoring for Maintenance Tasks

```yaml
# config/prometheus/rules/maintenance.yml
groups:
- name: maintenance_alerts
  rules:
  - alert: BackupOld
    expr: time() - foxhunt_last_backup_timestamp > 86400
    for: 1h
    labels:
      severity: warning
    annotations:
      summary: "Database backup is older than 24 hours"

  - alert: LogDiskFull
    expr: node_filesystem_free_bytes{mountpoint="/var/log"} / node_filesystem_size_bytes < 0.1
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Log disk space critical (<10% free)"

  - alert: CertificateExpiring
    expr: (ssl_certificate_expiry_seconds - time()) / 86400 < 30
    for: 1h
    labels:
      severity: warning
    annotations:
      summary: "SSL certificate expires in {{ $value }} days"
```

---

## 📞 Maintenance Contacts

| Task Category | Primary Contact | Backup Contact |
|---------------|----------------|----------------|
| Daily Health Checks | On-Call Engineer | [CONFIGURE] |
| Database Maintenance | Database Admin | [CONFIGURE] |
| Security Reviews | Security Lead | [CONFIGURE] |
| Performance Tuning | DevOps Lead | [CONFIGURE] |
| DR Testing | Infrastructure Lead | [CONFIGURE] |

---

## 📚 Related Documentation

- **Main Runbook**: `PRODUCTION_DEPLOYMENT_RUNBOOK.md`
- **Emergency Procedures**: `EMERGENCY_PROCEDURES.md`
- **Quick Start**: `QUICK_START_PRODUCTION.md`
- **Architecture**: `CLAUDE.md`

---

**Last Updated**: 2025-10-07
**Version**: 1.0.0
**Next Review**: 2025-11-07