foxhunt/docs/DEPLOYMENT_QUICK_REFERENCE.md

# Foxhunt HFT Production Deployment - Quick Reference Card

**Version**: 3.0.0 | **Wave**: 75 | **Date**: 2025-10-03

---

## 🚀 Deployment Checklist (30-Second View)

### Pre-Deployment
- [ ] All tests passing (1,919/1,919)
- [ ] Security scan clean (CVSS 0.0)
- [ ] Secrets generated (JWT, DB, Redis, Vault)
- [ ] TLS certificates ready
- [ ] Database migrations ready (12 files)
- [ ] Backups taken
- [ ] Rollback plan ready

### Deployment Steps
1. **Infrastructure** (60 min): PostgreSQL + Redis + Vault + S3
2. **Migrations** (30 min): Apply 12 SQL migrations
3. **Certificates** (45 min): Generate CA + service certs
4. **Services** (90 min): Deploy 3 services
5. **Validation** (60 min): Health checks + smoke tests
6. **Tuning** (60 min): Performance baselines

**Total**: 4-6 hours (first deployment)

---

## 📍 Critical File Locations

```
/opt/foxhunt/
├── bin/                           # Service binaries
│   ├── trading_service
│   ├── backtesting_service
│   ├── ml_training_service
│   ├── health_check.sh           # Run this!
│   ├── backup.sh                 # Daily at 2 AM
│   └── rollback.sh               # Emergency only
├── config/                        # Configuration files
├── logs/                          # Application logs
├── data/                          # Runtime data
└── backups/                       # Backup archives

/etc/foxhunt/
├── certs/                         # TLS certificates
│   ├── ca.crt
│   └── *.{key,crt}
├── trading_service/.env           # Service config
├── backtesting_service/.env
└── ml_training_service/.env

/var/log/foxhunt/                  # Log files
/cache/models/                     # ML model cache
/secrets/                          # Vault secrets
```

---

## 🔧 Service Quick Commands

### Service Management
```bash
# Status
sudo systemctl status foxhunt-{trading,backtesting,ml-training}

# Start
sudo systemctl start foxhunt-{trading,backtesting,ml-training}

# Stop
sudo systemctl stop foxhunt-{trading,backtesting,ml-training}

# Restart
sudo systemctl restart foxhunt-{trading,backtesting,ml-training}

# Logs (live)
sudo journalctl -u foxhunt-trading -f

# Logs (last hour)
sudo journalctl -u foxhunt-trading --since "1 hour ago"
```

### Health Checks
```bash
# Comprehensive health check (12 checks)
/opt/foxhunt/bin/health_check.sh

# Individual service checks
curl http://localhost:8080/health  # Trading Service
curl http://localhost:8082/health  # Backtesting Service
curl http://localhost:8083/health  # ML Training Service

# Check ports
nc -z localhost 50051  # Trading
nc -z localhost 50052  # Backtesting
nc -z localhost 50053  # ML Training
```

### Emergency Procedures
```bash
# Emergency kill switch (halt all trading)
echo "HALT" | nc -U /var/run/foxhunt/kill_switch.sock

# Emergency rollback (last backup)
/opt/foxhunt/bin/rollback.sh /opt/foxhunt/backups/$(ls -t /opt/foxhunt/backups | head -1)

# Emergency restart (20-second downtime)
sudo systemctl restart foxhunt-{trading,backtesting,ml-training}
```

---

## 📊 Service Ports Reference

| Service | gRPC | Health | Metrics | Notes |
|---------|------|--------|---------|-------|
| **Trading Service** | 50051 | 8080 | 9092 | Critical path |
| **Backtesting Service** | 50052 | 8082 | 9093 | Strategy testing |
| **ML Training Service** | 50053 | 8083 | 9094 | Model training |
| **PostgreSQL** | 5432 | - | - | Primary database |
| **Redis** | 6379 | - | - | Kill switch, cache |
| **Vault** | 8200 | - | - | Secrets management |
| **Prometheus** | 9090 | - | - | Metrics collection |
| **Grafana** | 3000 | - | - | Dashboards |

---

## 🔐 Security Quick Reference

### JWT Authentication
```bash
# Generate production JWT secret (512-bit)
openssl rand -base64 64 > /secrets/jwt_secret.key
chmod 600 /secrets/jwt_secret.key

# Test authentication
curl -X POST http://localhost:8080/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"PASSWORD"}'
```

### TLS Certificates
```bash
# Check certificate expiry
openssl x509 -in /etc/foxhunt/certs/trading_service.crt -noout -dates

# Verify certificate chain
openssl verify -CAfile /etc/foxhunt/certs/ca.crt /etc/foxhunt/certs/trading_service.crt
```

### Database Access
```bash
# Connect to production database
psql postgresql://foxhunt_user:PASSWORD@localhost:5432/foxhunt_production

# Quick audit trail check
psql $DATABASE_URL -c "SELECT COUNT(*) FROM transaction_audit_events;"
```

---

## 📈 Performance Targets

| Metric | Target | Critical Threshold |
|--------|--------|-------------------|
| **Trading Latency (avg)** | <50μs | 100μs |
| **Trading Latency (p99)** | <100μs | 200μs |
| **JWT Auth Overhead** | <10μs | 50μs |
| **JWT Revocation Check** | <100μs | 500μs |
| **Database Query (hot)** | <5ms | 50ms |
| **ML Model Inference** | <100ms | 500ms |
| **Audit Trail Write** | <5ms | 50ms |
| **Health Check Response** | <1ms | 10ms |
| **Memory Usage** | <80% | 90% |
| **Disk Usage** | <85% | 90% |
| **CPU Load** | <cores | 2x cores |
| **Time Sync Offset** | <100μs | 1ms |

---

## 🔥 Common Issues & Quick Fixes

### Service Won't Start
```bash
# Check logs
sudo journalctl -u foxhunt-trading -n 50

# Check if port is in use
sudo netstat -tlnp | grep 50051

# Verify configuration
cat /etc/foxhunt/trading_service/.env

# Check dependencies
systemctl status postgresql redis
```

### High Latency
```bash
# Check system load
uptime

# Check network latency
ping -c 10 exchange.example.com

# Check database performance
psql $DATABASE_URL -c "SELECT * FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 5;"

# Check Redis latency
redis-cli --latency
```

### Database Connection Errors
```bash
# Check PostgreSQL status
systemctl status postgresql

# Check connection count
sudo -u postgres psql -c "SELECT count(*) FROM pg_stat_activity;"

# Test connection
psql $DATABASE_URL -c "SELECT 1;"
```

### Kill Switch Not Responding
```bash
# Check socket exists
ls -l /var/run/foxhunt/kill_switch.sock

# Check Redis fallback
redis-cli GET foxhunt:kill_switch:enabled

# Restart trading service
sudo systemctl restart foxhunt-trading
```

---

## 📞 Emergency Contacts (Template)

| Role | Name | Phone | Email | PagerDuty |
|------|------|-------|-------|-----------|
| **System Admin** | [Name] | [Phone] | [Email] | [ID] |
| **Lead Developer** | [Name] | [Phone] | [Email] | [ID] |
| **DevOps Engineer** | [Name] | [Phone] | [Email] | [ID] |
| **DBA** | [Name] | [Phone] | [Email] | [ID] |
| **Trading Ops** | [Name] | [Phone] | [Email] | [ID] |
| **Risk Management** | [Name] | [Phone] | [Email] | [ID] |
| **Compliance** | [Name] | [Phone] | [Email] | [ID] |

---

## 🎯 Incident Response (P0 - System Down)

**Timeline**: 0-60 minutes

```
0-5 min:   HALT TRADING (kill switch)
           Page on-call engineer
           Open incident war room

5-15 min:  Check logs: journalctl -u foxhunt-* --since "30 min ago"
           Check infrastructure: PostgreSQL, Redis, network
           Identify root cause

15-30 min: Apply emergency fix OR rollback
           Verify health checks pass
           Monitor for 10 minutes

30-60 min: Resume trading (deactivate kill switch)
           Document incident timeline
           Schedule postmortem
```

---

## 📦 Backup & Recovery

### Daily Backup
```bash
# Run manual backup
/opt/foxhunt/bin/backup.sh

# Check last backup
ls -lht /opt/foxhunt/backups | head -5

# Verify backup integrity
cat /opt/foxhunt/backups/LATEST/manifest.txt
```

### Emergency Rollback
```bash
# List available backups
ls -1dt /opt/foxhunt/backups/*/

# Rollback to specific backup
/opt/foxhunt/bin/rollback.sh /opt/foxhunt/backups/20251003_120000

# Rollback to last backup
/opt/foxhunt/bin/rollback.sh /opt/foxhunt/backups/$(ls -t /opt/foxhunt/backups | head -1)
```

---

## 🔍 Monitoring Dashboard URLs

| Dashboard | URL | Purpose |
|-----------|-----|---------|
| **Grafana** | http://localhost:3000 | Metrics visualization |
| **Prometheus** | http://localhost:9090 | Metrics collection |
| **Trading Health** | http://localhost:8080/health | Service health |
| **Backtesting Health** | http://localhost:8082/health | Service health |
| **ML Training Health** | http://localhost:8083/health | Service health |

**Default Credentials** (CHANGE IN PRODUCTION):
- Grafana: admin / admin
- Prometheus: No auth (internal only)

---

## 📚 Full Documentation

**Comprehensive Runbook**: `/home/jgrusewski/Work/foxhunt/docs/PRODUCTION_DEPLOYMENT_RUNBOOK_V3.md` (2,082 lines)

**Wave 75 Summary**: `/home/jgrusewski/Work/foxhunt/docs/WAVE75_AGENT9_DEPLOYMENT_GUIDE.md` (698 lines)

**Deployment Scripts**:
- Health Check: `/home/jgrusewski/Work/foxhunt/deployment/scripts/health_check.sh`
- Rollback: `/home/jgrusewski/Work/foxhunt/deployment/scripts/rollback.sh`
- Backup: `/home/jgrusewski/Work/foxhunt/deployment/scripts/backup.sh`

---

**Last Updated**: 2025-10-03
**Version**: 3.0.0
**Status**: ✅ Production Ready

*Keep this card accessible during deployments and incidents*