foxhunt/QUICK_START_PRODUCTION.md

# Quick Start Production Deployment - Foxhunt HFT

**Time Required**: 15-20 minutes (Docker), 30-45 minutes (Bare-Metal)
**Prerequisites**: All infrastructure ready (PostgreSQL, Redis, Vault)

---

## 🚀 Docker Deployment (Recommended for Testing/Staging)

### Step 1: Fix Agent 96 Issues (REQUIRED - 5 minutes)

```bash
cd /opt/foxhunt

# Fix 1: Update Dockerfile paths (all 4 services)
find services -name Dockerfile -exec sed -i 's|COPY crates/config|COPY config|g' {} \;

# Fix 2: Add ML Training Service entry point
echo 'CMD ["ml_training_service", "serve"]' >> services/ml_training_service/Dockerfile

# Fix 3: Set Benzinga API key
export BENZINGA_API_KEY="your-benzinga-pro-api-key"
echo "BENZINGA_API_KEY=$BENZINGA_API_KEY" >> .env
```

### Step 2: Configure Environment (2 minutes)

```bash
# Generate required secrets
export JWT_SECRET=$(openssl rand -base64 64 | tr -d '\n')
export KILL_SWITCH_TOKEN=$(openssl rand -base64 32)

# Create .env file
cat > .env <<EOF
DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@postgres:5432/foxhunt
REDIS_URL=redis://redis:6379
VAULT_ADDR=http://vault:8200
VAULT_TOKEN=foxhunt-dev-root
JWT_SECRET=$JWT_SECRET
BENZINGA_API_KEY=$BENZINGA_API_KEY
KILL_SWITCH_MASTER_TOKEN=$KILL_SWITCH_TOKEN
AWS_ACCESS_KEY_ID=your_aws_key
AWS_SECRET_ACCESS_KEY=your_aws_secret
AWS_REGION=us-east-1
RUST_LOG=info
EOF
```

### Step 3: Build Images (5-10 minutes)

```bash
# Build all service images
docker-compose build

# Or build individually (parallel)
docker-compose build trading_service &
docker-compose build backtesting_service &
docker-compose build ml_training_service &
docker-compose build api_gateway &
wait

# Verify images
docker images | grep foxhunt
```

### Step 4: Start Infrastructure (2 minutes)

```bash
# Start database and cache layers
docker-compose up -d postgres redis vault influxdb

# Wait for health checks
sleep 30
docker-compose ps | grep healthy
```

### Step 5: Run Migrations (1 minute)

```bash
# Apply database migrations
cargo sqlx migrate run

# Or from Docker
docker-compose exec postgres psql -U foxhunt -d foxhunt -c '\dt'
```

### Step 6: Start Services (2 minutes)

```bash
# Start monitoring
docker-compose up -d prometheus grafana

# Start backend services
docker-compose up -d trading_service backtesting_service ml_training_service

# Wait for startup (ML service takes 60s for GPU init)
sleep 60

# Start API Gateway
docker-compose up -d api_gateway
```

### Step 7: Verify Deployment (2 minutes)

```bash
# Check all services healthy
docker-compose ps

# Test health endpoints
curl -s http://localhost:9092/metrics | grep foxhunt_orders_total

# View logs
docker-compose logs --tail=50 trading_service
docker-compose logs --tail=50 api_gateway

# Access Grafana
open http://localhost:3000
# Login: admin / foxhunt123
```

### Success Checklist

- [ ] All 10 containers running (6 infrastructure + 4 application)
- [ ] No errors in `docker-compose logs`
- [ ] Metrics endpoints responding (ports 9091-9094)
- [ ] Grafana accessible at http://localhost:3000
- [ ] GPU detected: `docker-compose exec ml_training_service nvidia-smi`

---

## 🔧 Bare-Metal Deployment (Recommended for Production)

### Step 1: Build Binaries (10-15 minutes)

```bash
cd /opt/foxhunt

# Build with CPU-specific optimizations
export RUSTFLAGS="-C target-cpu=native -C opt-level=3"
cargo build --release --workspace

# Verify
ls -lh target/release/ | grep -E "(trading|backtesting|ml_training|api_gateway)"
```

### Step 2: Install Binaries (2 minutes)

```bash
# Create directories
sudo mkdir -p /opt/foxhunt/{bin,config,logs,data,models,checkpoints}

# Install binaries
sudo cp target/release/{trading_service,backtesting_service,ml_training_service,api_gateway} /opt/foxhunt/bin/

# Set permissions
sudo chown -R foxhunt:foxhunt /opt/foxhunt
sudo chmod +x /opt/foxhunt/bin/*
```

### Step 3: Configure SystemD (5 minutes)

```bash
# Generate SystemD services
./deployment/create_systemd_services.sh

# Reload
sudo systemctl daemon-reload

# Enable auto-start
sudo systemctl enable foxhunt-{trading,backtesting,ml-training,api-gateway}
```

### Step 4: Setup Database (3 minutes)

```bash
# Create database
sudo -u postgres psql <<EOF
CREATE USER foxhunt WITH PASSWORD 'secure_password';
CREATE DATABASE foxhunt OWNER foxhunt;
\c foxhunt
CREATE EXTENSION IF NOT EXISTS timescaledb;
EOF

# Run migrations
export DATABASE_URL=postgresql://foxhunt:secure_password@localhost:5432/foxhunt
cargo sqlx migrate run
```

### Step 5: Configure Secrets (3 minutes)

```bash
# Start Vault
sudo systemctl start vault

# Store secrets
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=foxhunt-dev-root

vault kv put secret/foxhunt/jwt secret="$(openssl rand -base64 64)"
vault kv put secret/foxhunt/kill_switch master_token="$(openssl rand -base64 32)"
vault kv put secret/foxhunt/api_keys benzinga="$BENZINGA_API_KEY"
vault kv put secret/foxhunt/aws access_key_id="$AWS_ACCESS_KEY_ID" secret_access_key="$AWS_SECRET_ACCESS_KEY"
```

### Step 6: Start Services (2 minutes)

```bash
# Start infrastructure
sudo systemctl start redis vault

# Start backend services (parallel)
sudo systemctl start foxhunt-trading &
sudo systemctl start foxhunt-backtesting &
sudo systemctl start foxhunt-ml-training &
wait

# Wait for health checks
sleep 30

# Start API Gateway
sudo systemctl start foxhunt-api-gateway
```

### Step 7: Verify Deployment (2 minutes)

```bash
# Check service status
sudo systemctl status foxhunt-*

# Test health
./deployment/health_check.sh --mode comprehensive

# View logs
sudo journalctl -u foxhunt-trading -n 50
sudo journalctl -u foxhunt-api-gateway -n 50

# Check metrics
curl http://localhost:9092/metrics | grep foxhunt_
```

### Success Checklist

- [ ] All 4 services active: `systemctl status foxhunt-*`
- [ ] No errors in logs: `journalctl -u foxhunt-* --since "10 minutes ago"`
- [ ] Metrics endpoints responding
- [ ] GPU detected: `nvidia-smi`
- [ ] Database connected: `psql $DATABASE_URL -c "SELECT 1;"`

---

## 🎯 Post-Deployment Smoke Tests

### Test 1: Service Health

```bash
# Docker
for service in trading backtesting ml_training api_gateway; do
  echo "Testing $service..."
  docker-compose logs --tail=1 ${service}_service | grep -q "started" && echo "✅ $service OK" || echo "❌ $service FAILED"
done

# Bare-metal
for service in trading backtesting ml-training api-gateway; do
  sudo systemctl is-active foxhunt-$service && echo "✅ $service OK" || echo "❌ $service FAILED"
done
```

### Test 2: Metrics Collection

```bash
# Test each metrics endpoint
for port in 9091 9092 9093 9094; do
  curl -s http://localhost:$port/metrics | grep -q "foxhunt_" && echo "✅ Port $port OK" || echo "❌ Port $port FAILED"
done
```

### Test 3: Database Connectivity

```bash
# Test database from each service
psql $DATABASE_URL -c "SELECT count(*) FROM information_schema.tables;"
redis-cli ping
```

### Test 4: GPU Access (ML Services)

```bash
# Docker
docker-compose exec ml_training_service nvidia-smi

# Bare-metal
nvidia-smi
```

### Test 5: End-to-End (Optional)

```bash
# Requires TLI or gRPC client
# Test API Gateway → Trading Service flow
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check

# Expected: {"status": "SERVING"}
```

---

## 🚨 Troubleshooting Quick Fixes

### Issue: Service Won't Start

```bash
# Check logs
docker-compose logs <service>  # Docker
sudo journalctl -u foxhunt-<service> -n 50  # Bare-metal

# Common fix: Restart
docker-compose restart <service>
sudo systemctl restart foxhunt-<service>
```

### Issue: Missing Environment Variable

```bash
# Docker: Add to .env
echo "MISSING_VAR=value" >> .env
docker-compose up -d <service>

# Bare-metal: Add to SystemD
sudo systemctl edit foxhunt-<service>
# Add: Environment="MISSING_VAR=value"
sudo systemctl restart foxhunt-<service>
```

### Issue: GPU Not Detected

```bash
# Docker: Install nvidia-docker2
sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker

# Verify
docker run --rm --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi
```

### Issue: Port Already in Use

```bash
# Find process
sudo lsof -i :<port>

# Kill process
sudo kill -9 <PID>

# Restart service
docker-compose up -d <service>
```

### Issue: Database Connection Failed

```bash
# Verify PostgreSQL running
docker-compose ps postgres  # Docker
sudo systemctl status postgresql  # Bare-metal

# Test connection
psql $DATABASE_URL -c "SELECT 1;"

# Restart database
docker-compose restart postgres
sudo systemctl restart postgresql
```

---

## 📊 Monitoring Access

- **Grafana**: http://localhost:3000 (admin / foxhunt123)
- **Prometheus**: http://localhost:9090
- **Metrics**:
  - Trading: http://localhost:9092/metrics
  - Backtesting: http://localhost:9093/metrics
  - ML Training: http://localhost:9094/metrics
  - API Gateway: http://localhost:9091/metrics

---

## 📞 Next Steps

1. **Configure monitoring alerts**: Edit `config/prometheus/rules/alerts.yml`
2. **Set up backups**: Configure daily PostgreSQL backups to S3
3. **Enable TLS**: Configure SSL/TLS certificates for production
4. **Review security**: Rotate default passwords and tokens
5. **Load testing**: Run `services/load_tests` to validate performance

---

## 📚 Full Documentation

- **Complete Runbook**: `PRODUCTION_DEPLOYMENT_RUNBOOK.md`
- **Emergency Procedures**: `EMERGENCY_PROCEDURES.md`
- **Maintenance Checklist**: `MAINTENANCE_CHECKLIST.md`
- **Architecture**: `CLAUDE.md`

---

**Last Updated**: 2025-10-07
**Version**: 1.0.0
**Estimated Time**: 15-20 minutes (Docker), 30-45 minutes (Bare-Metal)