foxhunt/docs/archive/historical/MAINTENANCE_CHECKLIST.md

Maintenance Checklist - Foxhunt HFT Trading System

Version: 1.0.0 Last Updated: 2025-10-07 Purpose: Regular maintenance tasks to ensure system reliability and performance

---

📅 Daily Tasks (15-20 minutes)

Morning System Health Check

Time: 08:00 UTC (before market open) Duration: 5 minutes Responsible: On-call engineer

# 1. Run comprehensive health check
./deployment/health_check.sh --mode comprehensive

# 2. Check all services running
docker-compose ps  # Docker
sudo systemctl status foxhunt-*  # Bare-metal

# 3. Verify metrics collection
for port in 9091 9092 9093 9094; do
  curl -s http://localhost:$port/metrics | grep -q "foxhunt_" || echo "⚠️ Port $port metrics missing"
done

# 4. Check database connectivity
psql $DATABASE_URL -c "SELECT count(*) FROM orders WHERE created_at > NOW() - INTERVAL '24 hours';"

# 5. Check GPU status (ML services)
nvidia-smi  # Bare-metal
docker-compose exec ml_training_service nvidia-smi  # Docker

Success Criteria:

• All services status: HEALTHY
• All metrics endpoints responding
• Database connected, <100ms query time
• GPU detected and accessible
• No ERROR logs in last 24 hours

---

Log Review

Time: 09:00 UTC Duration: 10 minutes Responsible: Operations team

# 1. Check for errors in last 24 hours
journalctl -u foxhunt-* --since "24 hours ago" | grep -i error > /tmp/daily-errors.log

# 2. Review error count
ERROR_COUNT=$(wc -l < /tmp/daily-errors.log)
echo "Errors in last 24h: $ERROR_COUNT"

# 3. Check for panics/crashes
journalctl -u foxhunt-* --since "24 hours ago" | grep -i panic

# 4. Review audit logs
psql $DATABASE_URL -c "SELECT action, count(*) FROM audit_logs WHERE created_at > NOW() - INTERVAL '24 hours' GROUP BY action ORDER BY count DESC LIMIT 10;"

# 5. Check for unauthorized access attempts
psql $DATABASE_URL -c "SELECT * FROM audit_logs WHERE action = 'login_failed' AND created_at > NOW() - INTERVAL '24 hours';"

Action Items:

• If ERROR_COUNT > 10: Investigate root cause
• If panics detected: File incident report
• If unauthorized access: Escalate to security team

---

Backup Verification

Time: 10:00 UTC Duration: 5 minutes Responsible: Database admin

# 1. Verify last backup completed
aws s3 ls s3://foxhunt-backups/postgres/ | tail -5

# 2. Check backup age
LATEST_BACKUP=$(aws s3 ls s3://foxhunt-backups/postgres/ | tail -1 | awk '{print $4}')
echo "Latest backup: $LATEST_BACKUP"

# 3. Verify backup integrity
aws s3 cp s3://foxhunt-backups/postgres/$LATEST_BACKUP /tmp/verify.dump
pg_restore --list /tmp/verify.dump > /dev/null && echo "✅ Backup valid" || echo "❌ Backup corrupted"
rm /tmp/verify.dump

# 4. Check configuration backups
aws s3 ls s3://foxhunt-backups/config/ | tail -1

# 5. Verify model backups
aws s3 ls s3://foxhunt-models/ --recursive | wc -l

Success Criteria:

• Latest backup <24 hours old
• Backup integrity verified
• Configuration backup exists
• Model backups accessible

---

📅 Weekly Tasks (1-2 hours)

Sunday Maintenance Window

Time: Sunday 02:00-04:00 UTC (low trading volume) Duration: 1-2 hours Responsible: DevOps team

1. Database Maintenance (30 minutes)

# 1. Vacuum and analyze
psql $DATABASE_URL <<EOF
VACUUM ANALYZE orders;
VACUUM ANALYZE positions;
VACUUM ANALYZE market_data;
VACUUM ANALYZE audit_logs;
REINDEX TABLE orders;
REINDEX TABLE positions;
EOF

# 2. Check for bloat
psql $DATABASE_URL -c "SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size FROM pg_tables WHERE schemaname = 'public' ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC LIMIT 10;"

# 3. Review slow queries
psql $DATABASE_URL -c "SELECT query, calls, mean_exec_time, max_exec_time FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 20;"

# 4. Check replication lag (if replicas configured)
psql $DATABASE_URL -c "SELECT client_addr, state, sync_state, replay_lag FROM pg_stat_replication;"

# 5. Archive old audit logs (>90 days)
psql $DATABASE_URL -c "DELETE FROM audit_logs WHERE created_at < NOW() - INTERVAL '90 days';"

2. Log Rotation and Cleanup (15 minutes)

# 1. Archive logs older than 30 days
find /var/log/foxhunt -name "*.log" -mtime +30 -exec gzip {} \;

# 2. Upload archived logs to S3
aws s3 sync /var/log/foxhunt s3://foxhunt-backups/logs/ --exclude "*" --include "*.log.gz"

# 3. Delete logs older than 90 days
find /var/log/foxhunt -name "*.log.gz" -mtime +90 -delete

# 4. Clean journal logs
sudo journalctl --vacuum-time=30d

# 5. Docker log cleanup (if using Docker)
docker system prune -af --volumes --filter "until=720h"

3. Performance Analysis (30 minutes)

# 1. Generate weekly performance report
./deployment/scripts/weekly-performance-report.sh

# 2. Review key metrics
curl http://localhost:9092/metrics | grep -E "(order_latency|throughput|error_rate)" > /tmp/weekly-metrics.txt

# 3. Compare with baseline
./deployment/scripts/compare-metrics.sh /tmp/weekly-metrics.txt /var/log/foxhunt/baseline-metrics.txt

# 4. Check for performance degradation
psql $DATABASE_URL -c "SELECT
  DATE(created_at) as date,
  percentile_cont(0.99) WITHIN GROUP (ORDER BY processing_time_ms) as p99_latency
FROM orders
WHERE created_at > NOW() - INTERVAL '7 days'
GROUP BY DATE(created_at)
ORDER BY date DESC;"

# 5. Update performance baselines (if improved)
cp /tmp/weekly-metrics.txt /var/log/foxhunt/baseline-metrics.txt

4. Security Review (15 minutes)

# 1. Review failed login attempts
psql $DATABASE_URL -c "SELECT user_id, ip_address, count(*) FROM audit_logs WHERE action = 'login_failed' AND created_at > NOW() - INTERVAL '7 days' GROUP BY user_id, ip_address HAVING count(*) > 5;"

# 2. Check for unusual API usage
psql $DATABASE_URL -c "SELECT api_key, count(*) FROM api_requests WHERE created_at > NOW() - INTERVAL '7 days' GROUP BY api_key ORDER BY count DESC LIMIT 20;"

# 3. Review Vault access logs
vault audit list
vault read sys/audit/file/log | grep "$(date -d '7 days ago' +%Y-%m-%d)"

# 4. Check for expired secrets
vault list secret/foxhunt/ | while read secret; do
  vault read "secret/foxhunt/$secret" | grep ttl
done

# 5. Run security scan
./deployment/scripts/security-scan.sh

---

📅 Monthly Tasks (2-4 hours)

First Sunday of Month

Time: Sunday 02:00-06:00 UTC Duration: 2-4 hours Responsible: DevOps team + Database admin

1. System Updates (1 hour)

# 1. Check for security updates
sudo apt-get update
sudo apt-get upgrade -s | grep -i security

# 2. Apply security patches (requires downtime planning)
# Schedule maintenance window
# Update all systems
sudo apt-get upgrade -y

# 3. Update Rust dependencies
cargo update
cargo audit

# 4. Update Docker images
docker-compose pull

# 5. Update Kubernetes components (if applicable)
kubectl version
helm repo update

2. Database Deep Maintenance (1 hour)

# 1. Full database backup
pg_dump -h localhost -U foxhunt -Fc foxhunt > /tmp/monthly-backup-$(date +%Y%m).dump

# 2. Upload to S3
aws s3 cp /tmp/monthly-backup-$(date +%Y%m).dump s3://foxhunt-backups/postgres/monthly/

# 3. Test restore (to separate instance)
createdb foxhunt_test
pg_restore -h localhost -U foxhunt -d foxhunt_test /tmp/monthly-backup-$(date +%Y%m).dump

# 4. Verify data integrity
psql foxhunt_test -c "SELECT count(*) FROM orders;" > /tmp/test-count.txt
psql foxhunt -c "SELECT count(*) FROM orders;" > /tmp/prod-count.txt
diff /tmp/test-count.txt /tmp/prod-count.txt

# 5. Cleanup test database
dropdb foxhunt_test
rm /tmp/monthly-backup-$(date +%Y%m).dump

3. Capacity Planning (30 minutes)

# 1. Review storage growth
df -h
du -sh /var/lib/postgresql
du -sh /opt/foxhunt

# 2. Review memory trends
./deployment/scripts/memory-trend-analysis.sh --days 30

# 3. Review CPU utilization
./deployment/scripts/cpu-trend-analysis.sh --days 30

# 4. Review database size growth
psql $DATABASE_URL -c "SELECT
  schemaname,
  tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename) -
    LAG(pg_total_relation_size(schemaname||'.'||tablename)) OVER (PARTITION BY tablename ORDER BY NOW())) AS growth
FROM pg_tables
WHERE schemaname = 'public';"

# 5. Forecast capacity needs (next 3 months)
./deployment/scripts/capacity-forecast.sh

4. Disaster Recovery Test (1 hour)

# 1. Test database failover (staging only!)
./deployment/scripts/test-database-failover.sh

# 2. Test service crash recovery
./deployment/scripts/test-service-crash-recovery.sh

# 3. Test network partition handling
./deployment/scripts/test-network-partition.sh

# 4. Test backup restoration
./deployment/scripts/test-backup-restore.sh

# 5. Document results
./deployment/scripts/generate-dr-report.sh > /var/log/foxhunt/dr-test-$(date +%Y%m).txt

5. Certificate Renewal (15 minutes)

# 1. Check certificate expiration
echo | openssl s_client -servername api.foxhunt.trading -connect api.foxhunt.trading:443 2>/dev/null | openssl x509 -noout -dates

# 2. Renew if expiring in <30 days
sudo certbot renew --dry-run
sudo certbot renew

# 3. Reload services
sudo systemctl reload nginx

# 4. Verify new certificate
echo | openssl s_client -servername api.foxhunt.trading -connect api.foxhunt.trading:443 2>/dev/null | openssl x509 -noout -text | grep -A 2 Validity

---

📅 Quarterly Tasks (4-8 hours)

First Sunday of Quarter

Time: Sunday 00:00-08:00 UTC Duration: 4-8 hours Responsible: Full DevOps team

1. Major Version Updates (2-3 hours)

# 1. Plan upgrade path
# Review release notes for:
# - Rust
# - PostgreSQL
# - Redis
# - Docker
# - Kubernetes (if applicable)

# 2. Test in staging
# Deploy updates to staging environment
# Run full test suite
./tests/integration_tests/run_all.sh

# 3. Schedule production upgrade
# Create maintenance window
# Notify stakeholders

# 4. Backup everything
./deployment/scripts/full-system-backup.sh

# 5. Execute upgrade
./deployment/scripts/quarterly-upgrade.sh

2. Security Audit (2 hours)

# 1. Dependency vulnerability scan
cargo audit
npm audit (if any JS components)

# 2. Container security scan
docker scan foxhunt-trading-service:latest
docker scan foxhunt-backtesting-service:latest
docker scan foxhunt-ml-training-service:latest
docker scan foxhunt-api-gateway:latest

# 3. Penetration testing (external vendor)
# Schedule pen test
# Review findings
# Implement fixes

# 4. Compliance review
# SOX compliance check
# MiFID II compliance check
# GDPR compliance check

# 5. Update security policies
vim /opt/foxhunt/docs/security-policy.md

3. Performance Optimization (2 hours)

# 1. Profile critical paths
./deployment/scripts/profile-trading-service.sh
./deployment/scripts/profile-ml-service.sh

# 2. Identify bottlenecks
./deployment/scripts/identify-bottlenecks.sh

# 3. Database query optimization
psql $DATABASE_URL -c "SELECT query, mean_exec_time, calls FROM pg_stat_statements WHERE mean_exec_time > 100 ORDER BY mean_exec_time DESC LIMIT 50;"

# 4. Implement optimizations
# Review and apply performance improvements

# 5. Benchmark improvements
./tests/performance/benchmark-suite.sh

4. Disaster Recovery Drill (1 hour)

# 1. Full DR scenario test
./deployment/scripts/dr-drill-full.sh

# 2. Test complete system restore from backup
./deployment/scripts/test-full-restore.sh

# 3. Measure RTO/RPO
./deployment/scripts/measure-rto-rpo.sh

# 4. Update DR documentation
vim /opt/foxhunt/docs/disaster-recovery.md

# 5. Review and update runbooks
vim /opt/foxhunt/PRODUCTION_DEPLOYMENT_RUNBOOK.md

---

📅 Annual Tasks (1-2 days)

First Week of January

Duration: 1-2 days Responsible: Entire engineering team

1. Architecture Review

• Review system architecture
• Identify technical debt
• Plan major refactoring
• Update architecture diagrams
• Document lessons learned

2. Security Assessment

• Full security audit (external vendor)
• Compliance certification renewal
• Update security policies
• Security training for team
• Incident response drill

3. Capacity Planning

• Review 12-month growth trends
• Forecast infrastructure needs
• Budget planning for infrastructure
• Evaluate new technologies
• Plan scaling strategy

4. Documentation Update

• Update all runbooks
• Update architecture documentation
• Update API documentation
• Update deployment guides
• Create training materials

---

📊 Maintenance Tracking

Log Template

# /var/log/foxhunt/maintenance/YYYY-MM-DD.md

# Maintenance Log - YYYY-MM-DD

**Type**: Daily/Weekly/Monthly/Quarterly/Annual
**Start Time**: HH:MM UTC
**End Time**: HH:MM UTC
**Performed By**: [Name]

## Tasks Completed

- [ ] Task 1: Description
- [ ] Task 2: Description
- [ ] Task 3: Description

## Issues Found

1. Issue 1: Description and resolution
2. Issue 2: Description and resolution

## Metrics

- Service uptime: X%
- Average latency: Xμs
- Error rate: X%
- Database size: XGB

## Action Items

- [ ] Action 1: Owner, Due Date
- [ ] Action 2: Owner, Due Date

## Notes

Any additional observations or comments.

---

🚨 Maintenance Alerts

Configure Monitoring for Maintenance Tasks

# config/prometheus/rules/maintenance.yml
groups:
- name: maintenance_alerts
  rules:
  - alert: BackupOld
    expr: time() - foxhunt_last_backup_timestamp > 86400
    for: 1h
    labels:
      severity: warning
    annotations:
      summary: "Database backup is older than 24 hours"

  - alert: LogDiskFull
    expr: node_filesystem_free_bytes{mountpoint="/var/log"} / node_filesystem_size_bytes < 0.1
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Log disk space critical (<10% free)"

  - alert: CertificateExpiring
    expr: (ssl_certificate_expiry_seconds - time()) / 86400 < 30
    for: 1h
    labels:
      severity: warning
    annotations:
      summary: "SSL certificate expires in {{ $value }} days"

---

📞 Maintenance Contacts

Task Category	Primary Contact	Backup Contact
Daily Health Checks	On-Call Engineer	[CONFIGURE]
Database Maintenance	Database Admin	[CONFIGURE]
Security Reviews	Security Lead	[CONFIGURE]
Performance Tuning	DevOps Lead	[CONFIGURE]
DR Testing	Infrastructure Lead	[CONFIGURE]

---

📚 Related Documentation

• Main Runbook: PRODUCTION_DEPLOYMENT_RUNBOOK.md
• Emergency Procedures: EMERGENCY_PROCEDURES.md
• Quick Start: QUICK_START_PRODUCTION.md
• Architecture: CLAUDE.md

---

Last Updated: 2025-10-07 Version: 1.0.0 Next Review: 2025-11-07