foxhunt/QUICK_START_PRODUCTION.md

Quick Start Production Deployment - Foxhunt HFT

Time Required: 15-20 minutes (Docker), 30-45 minutes (Bare-Metal) Prerequisites: All infrastructure ready (PostgreSQL, Redis, Vault)

---

🚀 Docker Deployment (Recommended for Testing/Staging)

Step 1: Fix Agent 96 Issues (REQUIRED - 5 minutes)

cd /opt/foxhunt

# Fix 1: Update Dockerfile paths (all 4 services)
find services -name Dockerfile -exec sed -i 's|COPY crates/config|COPY config|g' {} \;

# Fix 2: Add ML Training Service entry point
echo 'CMD ["ml_training_service", "serve"]' >> services/ml_training_service/Dockerfile

# Fix 3: Set Benzinga API key
export BENZINGA_API_KEY="your-benzinga-pro-api-key"
echo "BENZINGA_API_KEY=$BENZINGA_API_KEY" >> .env

Step 2: Configure Environment (2 minutes)

# Generate required secrets
export JWT_SECRET=$(openssl rand -base64 64 | tr -d '\n')
export KILL_SWITCH_TOKEN=$(openssl rand -base64 32)

# Create .env file
cat > .env <<EOF
DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@postgres:5432/foxhunt
REDIS_URL=redis://redis:6379
VAULT_ADDR=http://vault:8200
VAULT_TOKEN=foxhunt-dev-root
JWT_SECRET=$JWT_SECRET
BENZINGA_API_KEY=$BENZINGA_API_KEY
KILL_SWITCH_MASTER_TOKEN=$KILL_SWITCH_TOKEN
AWS_ACCESS_KEY_ID=your_aws_key
AWS_SECRET_ACCESS_KEY=your_aws_secret
AWS_REGION=us-east-1
RUST_LOG=info
EOF

Step 3: Build Images (5-10 minutes)

# Build all service images
docker-compose build

# Or build individually (parallel)
docker-compose build trading_service &
docker-compose build backtesting_service &
docker-compose build ml_training_service &
docker-compose build api_gateway &
wait

# Verify images
docker images | grep foxhunt

Step 4: Start Infrastructure (2 minutes)

# Start database and cache layers
docker-compose up -d postgres redis vault influxdb

# Wait for health checks
sleep 30
docker-compose ps | grep healthy

Step 5: Run Migrations (1 minute)

# Apply database migrations
cargo sqlx migrate run

# Or from Docker
docker-compose exec postgres psql -U foxhunt -d foxhunt -c '\dt'

Step 6: Start Services (2 minutes)

# Start monitoring
docker-compose up -d prometheus grafana

# Start backend services
docker-compose up -d trading_service backtesting_service ml_training_service

# Wait for startup (ML service takes 60s for GPU init)
sleep 60

# Start API Gateway
docker-compose up -d api_gateway

Step 7: Verify Deployment (2 minutes)

# Check all services healthy
docker-compose ps

# Test health endpoints
curl -s http://localhost:9092/metrics | grep foxhunt_orders_total

# View logs
docker-compose logs --tail=50 trading_service
docker-compose logs --tail=50 api_gateway

# Access Grafana
open http://localhost:3000
# Login: admin / foxhunt123

Success Checklist

• All 10 containers running (6 infrastructure + 4 application)
• No errors in docker-compose logs
• Metrics endpoints responding (ports 9091-9094)
• Grafana accessible at http://localhost:3000
• GPU detected: docker-compose exec ml_training_service nvidia-smi

---

🔧 Bare-Metal Deployment (Recommended for Production)

Step 1: Build Binaries (10-15 minutes)

cd /opt/foxhunt

# Build with CPU-specific optimizations
export RUSTFLAGS="-C target-cpu=native -C opt-level=3"
cargo build --release --workspace

# Verify
ls -lh target/release/ | grep -E "(trading|backtesting|ml_training|api_gateway)"

Step 2: Install Binaries (2 minutes)

# Create directories
sudo mkdir -p /opt/foxhunt/{bin,config,logs,data,models,checkpoints}

# Install binaries
sudo cp target/release/{trading_service,backtesting_service,ml_training_service,api_gateway} /opt/foxhunt/bin/

# Set permissions
sudo chown -R foxhunt:foxhunt /opt/foxhunt
sudo chmod +x /opt/foxhunt/bin/*

Step 3: Configure SystemD (5 minutes)

# Generate SystemD services
./deployment/create_systemd_services.sh

# Reload
sudo systemctl daemon-reload

# Enable auto-start
sudo systemctl enable foxhunt-{trading,backtesting,ml-training,api-gateway}

Step 4: Setup Database (3 minutes)

# Create database
sudo -u postgres psql <<EOF
CREATE USER foxhunt WITH PASSWORD 'secure_password';
CREATE DATABASE foxhunt OWNER foxhunt;
\c foxhunt
CREATE EXTENSION IF NOT EXISTS timescaledb;
EOF

# Run migrations
export DATABASE_URL=postgresql://foxhunt:secure_password@localhost:5432/foxhunt
cargo sqlx migrate run

Step 5: Configure Secrets (3 minutes)

# Start Vault
sudo systemctl start vault

# Store secrets
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=foxhunt-dev-root

vault kv put secret/foxhunt/jwt secret="$(openssl rand -base64 64)"
vault kv put secret/foxhunt/kill_switch master_token="$(openssl rand -base64 32)"
vault kv put secret/foxhunt/api_keys benzinga="$BENZINGA_API_KEY"
vault kv put secret/foxhunt/aws access_key_id="$AWS_ACCESS_KEY_ID" secret_access_key="$AWS_SECRET_ACCESS_KEY"

Step 6: Start Services (2 minutes)

# Start infrastructure
sudo systemctl start redis vault

# Start backend services (parallel)
sudo systemctl start foxhunt-trading &
sudo systemctl start foxhunt-backtesting &
sudo systemctl start foxhunt-ml-training &
wait

# Wait for health checks
sleep 30

# Start API Gateway
sudo systemctl start foxhunt-api-gateway

Step 7: Verify Deployment (2 minutes)

# Check service status
sudo systemctl status foxhunt-*

# Test health
./deployment/health_check.sh --mode comprehensive

# View logs
sudo journalctl -u foxhunt-trading -n 50
sudo journalctl -u foxhunt-api-gateway -n 50

# Check metrics
curl http://localhost:9092/metrics | grep foxhunt_

Success Checklist

• All 4 services active: systemctl status foxhunt-*
• No errors in logs: journalctl -u foxhunt-* --since "10 minutes ago"
• Metrics endpoints responding
• GPU detected: nvidia-smi
• Database connected: psql $DATABASE_URL -c "SELECT 1;"

---

🎯 Post-Deployment Smoke Tests

Test 1: Service Health

# Docker
for service in trading backtesting ml_training api_gateway; do
  echo "Testing $service..."
  docker-compose logs --tail=1 ${service}_service | grep -q "started" && echo "✅ $service OK" || echo "❌ $service FAILED"
done

# Bare-metal
for service in trading backtesting ml-training api-gateway; do
  sudo systemctl is-active foxhunt-$service && echo "✅ $service OK" || echo "❌ $service FAILED"
done

Test 2: Metrics Collection

# Test each metrics endpoint
for port in 9091 9092 9093 9094; do
  curl -s http://localhost:$port/metrics | grep -q "foxhunt_" && echo "✅ Port $port OK" || echo "❌ Port $port FAILED"
done

Test 3: Database Connectivity

# Test database from each service
psql $DATABASE_URL -c "SELECT count(*) FROM information_schema.tables;"
redis-cli ping

Test 4: GPU Access (ML Services)

# Docker
docker-compose exec ml_training_service nvidia-smi

# Bare-metal
nvidia-smi

Test 5: End-to-End (Optional)

# Requires TLI or gRPC client
# Test API Gateway → Trading Service flow
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check

# Expected: {"status": "SERVING"}

---

🚨 Troubleshooting Quick Fixes

Issue: Service Won't Start

# Check logs
docker-compose logs <service>  # Docker
sudo journalctl -u foxhunt-<service> -n 50  # Bare-metal

# Common fix: Restart
docker-compose restart <service>
sudo systemctl restart foxhunt-<service>

Issue: Missing Environment Variable

# Docker: Add to .env
echo "MISSING_VAR=value" >> .env
docker-compose up -d <service>

# Bare-metal: Add to SystemD
sudo systemctl edit foxhunt-<service>
# Add: Environment="MISSING_VAR=value"
sudo systemctl restart foxhunt-<service>

Issue: GPU Not Detected

# Docker: Install nvidia-docker2
sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker

# Verify
docker run --rm --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi

Issue: Port Already in Use

# Find process
sudo lsof -i :<port>

# Kill process
sudo kill -9 <PID>

# Restart service
docker-compose up -d <service>

Issue: Database Connection Failed

# Verify PostgreSQL running
docker-compose ps postgres  # Docker
sudo systemctl status postgresql  # Bare-metal

# Test connection
psql $DATABASE_URL -c "SELECT 1;"

# Restart database
docker-compose restart postgres
sudo systemctl restart postgresql

---

📊 Monitoring Access

• Grafana: http://localhost:3000 (admin / foxhunt123)
• Prometheus: http://localhost:9090
• Metrics:
  • Trading: http://localhost:9092/metrics
  • Backtesting: http://localhost:9093/metrics
  • ML Training: http://localhost:9094/metrics
  • API Gateway: http://localhost:9091/metrics

---

📞 Next Steps

1. Configure monitoring alerts: Edit config/prometheus/rules/alerts.yml
2. Set up backups: Configure daily PostgreSQL backups to S3
3. Enable TLS: Configure SSL/TLS certificates for production
4. Review security: Rotate default passwords and tokens
5. Load testing: Run services/load_tests to validate performance

---

📚 Full Documentation

• Complete Runbook: PRODUCTION_DEPLOYMENT_RUNBOOK.md
• Emergency Procedures: EMERGENCY_PROCEDURES.md
• Maintenance Checklist: MAINTENANCE_CHECKLIST.md
• Architecture: CLAUDE.md

---

Last Updated: 2025-10-07 Version: 1.0.0 Estimated Time: 15-20 minutes (Docker), 30-45 minutes (Bare-Metal)