foxhunt/docs/DEPLOYMENT_QUICK_REFERENCE.md

Foxhunt HFT Production Deployment - Quick Reference Card

Version: 3.0.0 | Wave: 75 | Date: 2025-10-03

---

🚀 Deployment Checklist (30-Second View)

Pre-Deployment

• All tests passing (1,919/1,919)
• Security scan clean (CVSS 0.0)
• Secrets generated (JWT, DB, Redis, Vault)
• TLS certificates ready
• Database migrations ready (12 files)
• Backups taken
• Rollback plan ready

Deployment Steps

1. Infrastructure (60 min): PostgreSQL + Redis + Vault + S3
2. Migrations (30 min): Apply 12 SQL migrations
3. Certificates (45 min): Generate CA + service certs
4. Services (90 min): Deploy 3 services
5. Validation (60 min): Health checks + smoke tests
6. Tuning (60 min): Performance baselines

Total: 4-6 hours (first deployment)

---

📍 Critical File Locations

/opt/foxhunt/
├── bin/                           # Service binaries
│   ├── trading_service
│   ├── backtesting_service
│   ├── ml_training_service
│   ├── health_check.sh           # Run this!
│   ├── backup.sh                 # Daily at 2 AM
│   └── rollback.sh               # Emergency only
├── config/                        # Configuration files
├── logs/                          # Application logs
├── data/                          # Runtime data
└── backups/                       # Backup archives

/etc/foxhunt/
├── certs/                         # TLS certificates
│   ├── ca.crt
│   └── *.{key,crt}
├── trading_service/.env           # Service config
├── backtesting_service/.env
└── ml_training_service/.env

/var/log/foxhunt/                  # Log files
/cache/models/                     # ML model cache
/secrets/                          # Vault secrets

---

🔧 Service Quick Commands

Service Management

# Status
sudo systemctl status foxhunt-{trading,backtesting,ml-training}

# Start
sudo systemctl start foxhunt-{trading,backtesting,ml-training}

# Stop
sudo systemctl stop foxhunt-{trading,backtesting,ml-training}

# Restart
sudo systemctl restart foxhunt-{trading,backtesting,ml-training}

# Logs (live)
sudo journalctl -u foxhunt-trading -f

# Logs (last hour)
sudo journalctl -u foxhunt-trading --since "1 hour ago"

Health Checks

# Comprehensive health check (12 checks)
/opt/foxhunt/bin/health_check.sh

# Individual service checks
curl http://localhost:8080/health  # Trading Service
curl http://localhost:8082/health  # Backtesting Service
curl http://localhost:8083/health  # ML Training Service

# Check ports
nc -z localhost 50051  # Trading
nc -z localhost 50052  # Backtesting
nc -z localhost 50053  # ML Training

Emergency Procedures

# Emergency kill switch (halt all trading)
echo "HALT" | nc -U /var/run/foxhunt/kill_switch.sock

# Emergency rollback (last backup)
/opt/foxhunt/bin/rollback.sh /opt/foxhunt/backups/$(ls -t /opt/foxhunt/backups | head -1)

# Emergency restart (20-second downtime)
sudo systemctl restart foxhunt-{trading,backtesting,ml-training}

---

📊 Service Ports Reference

Service	gRPC	Health	Metrics	Notes
Trading Service	50051	8080	9092	Critical path
Backtesting Service	50052	8082	9093	Strategy testing
ML Training Service	50053	8083	9094	Model training
PostgreSQL	5432	-	-	Primary database
Redis	6379	-	-	Kill switch, cache
Vault	8200	-	-	Secrets management
Prometheus	9090	-	-	Metrics collection
Grafana	3000	-	-	Dashboards

---

🔐 Security Quick Reference

JWT Authentication

# Generate production JWT secret (512-bit)
openssl rand -base64 64 > /secrets/jwt_secret.key
chmod 600 /secrets/jwt_secret.key

# Test authentication
curl -X POST http://localhost:8080/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"PASSWORD"}'

TLS Certificates

# Check certificate expiry
openssl x509 -in /etc/foxhunt/certs/trading_service.crt -noout -dates

# Verify certificate chain
openssl verify -CAfile /etc/foxhunt/certs/ca.crt /etc/foxhunt/certs/trading_service.crt

Database Access

# Connect to production database
psql postgresql://foxhunt_user:PASSWORD@localhost:5432/foxhunt_production

# Quick audit trail check
psql $DATABASE_URL -c "SELECT COUNT(*) FROM transaction_audit_events;"

---

📈 Performance Targets

Metric	Target	Critical Threshold
Trading Latency (avg)	<50μs	100μs
Trading Latency (p99)	<100μs	200μs
JWT Auth Overhead	<10μs	50μs
JWT Revocation Check	<100μs	500μs
Database Query (hot)	<5ms	50ms
ML Model Inference	<100ms	500ms
Audit Trail Write	<5ms	50ms
Health Check Response	<1ms	10ms
Memory Usage	<80%	90%
Disk Usage	<85%	90%
CPU Load	<cores	2x cores
Time Sync Offset	<100μs	1ms

---

🔥 Common Issues & Quick Fixes

Service Won't Start

# Check logs
sudo journalctl -u foxhunt-trading -n 50

# Check if port is in use
sudo netstat -tlnp | grep 50051

# Verify configuration
cat /etc/foxhunt/trading_service/.env

# Check dependencies
systemctl status postgresql redis

High Latency

# Check system load
uptime

# Check network latency
ping -c 10 exchange.example.com

# Check database performance
psql $DATABASE_URL -c "SELECT * FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 5;"

# Check Redis latency
redis-cli --latency

Database Connection Errors

# Check PostgreSQL status
systemctl status postgresql

# Check connection count
sudo -u postgres psql -c "SELECT count(*) FROM pg_stat_activity;"

# Test connection
psql $DATABASE_URL -c "SELECT 1;"

Kill Switch Not Responding

# Check socket exists
ls -l /var/run/foxhunt/kill_switch.sock

# Check Redis fallback
redis-cli GET foxhunt:kill_switch:enabled

# Restart trading service
sudo systemctl restart foxhunt-trading

---

📞 Emergency Contacts (Template)

Role	Name	Phone	Email	PagerDuty
System Admin	[Name]	[Phone]	[Email]	[ID]
Lead Developer	[Name]	[Phone]	[Email]	[ID]
DevOps Engineer	[Name]	[Phone]	[Email]	[ID]
DBA	[Name]	[Phone]	[Email]	[ID]
Trading Ops	[Name]	[Phone]	[Email]	[ID]
Risk Management	[Name]	[Phone]	[Email]	[ID]
Compliance	[Name]	[Phone]	[Email]	[ID]

---

🎯 Incident Response (P0 - System Down)

Timeline: 0-60 minutes

0-5 min:   HALT TRADING (kill switch)
           Page on-call engineer
           Open incident war room

5-15 min:  Check logs: journalctl -u foxhunt-* --since "30 min ago"
           Check infrastructure: PostgreSQL, Redis, network
           Identify root cause

15-30 min: Apply emergency fix OR rollback
           Verify health checks pass
           Monitor for 10 minutes

30-60 min: Resume trading (deactivate kill switch)
           Document incident timeline
           Schedule postmortem

---

📦 Backup & Recovery

Daily Backup

# Run manual backup
/opt/foxhunt/bin/backup.sh

# Check last backup
ls -lht /opt/foxhunt/backups | head -5

# Verify backup integrity
cat /opt/foxhunt/backups/LATEST/manifest.txt

Emergency Rollback

# List available backups
ls -1dt /opt/foxhunt/backups/*/

# Rollback to specific backup
/opt/foxhunt/bin/rollback.sh /opt/foxhunt/backups/20251003_120000

# Rollback to last backup
/opt/foxhunt/bin/rollback.sh /opt/foxhunt/backups/$(ls -t /opt/foxhunt/backups | head -1)

---

🔍 Monitoring Dashboard URLs

Dashboard	URL	Purpose
Grafana	http://localhost:3000	Metrics visualization
Prometheus	http://localhost:9090	Metrics collection
Trading Health	http://localhost:8080/health	Service health
Backtesting Health	http://localhost:8082/health	Service health
ML Training Health	http://localhost:8083/health	Service health

Default Credentials (CHANGE IN PRODUCTION):

• Grafana: admin / admin
• Prometheus: No auth (internal only)

---

📚 Full Documentation

Comprehensive Runbook: /home/jgrusewski/Work/foxhunt/docs/PRODUCTION_DEPLOYMENT_RUNBOOK_V3.md (2,082 lines)

Wave 75 Summary: /home/jgrusewski/Work/foxhunt/docs/WAVE75_AGENT9_DEPLOYMENT_GUIDE.md (698 lines)

Deployment Scripts:

• Health Check: /home/jgrusewski/Work/foxhunt/deployment/scripts/health_check.sh
• Rollback: /home/jgrusewski/Work/foxhunt/deployment/scripts/rollback.sh
• Backup: /home/jgrusewski/Work/foxhunt/deployment/scripts/backup.sh

---

Last Updated: 2025-10-03 Version: 3.0.0 Status: ✅ Production Ready

Keep this card accessible during deployments and incidents