foxhunt/PRODUCTION_DEPLOYMENT_CHECKLIST.md

# Production Deployment Checklist
# 6-Model Ensemble System

**Document Version**: 1.0
**Date**: 2025-10-14
**Target System**: Foxhunt HFT Trading System
**Models**: DQN (2 variants), PPO (2 variants), MAMBA-2, TFT
**Status**: Pre-Deployment Validation

---

## Pre-Deployment Validation (Phase 0)

### 1. Model Checkpoint Verification

#### 1.1 DQN Checkpoints (2 variants)
- [ ] **DQN Epoch 10** checkpoint exists and is valid SafeTensors format
  - Path: `/home/jgrusewski/Work/foxhunt/ml/trained_models/production/dqn_real_data/dqn_epoch_10.safetensors`
  - Expected size: 75,628 bytes (74KB)
  - Verification: `hexdump -C dqn_epoch_10.safetensors | head -3` (check JSON header)
  - Expected Q-value: ~19.76 (high trading activity: 90%)

- [ ] **DQN Epoch 380** checkpoint exists and is valid SafeTensors format
  - Path: `/home/jgrusewski/Work/foxhunt/ml/trained_models/production/dqn_real_data/dqn_epoch_380.safetensors`
  - Expected size: 75,628 bytes (74KB)
  - Verification: SafeTensors header validation
  - Expected Q-value: ~2.81 (conservative: 75% activity)

**Command to verify DQN checkpoints**:
```bash
cd /home/jgrusewski/Work/foxhunt
ls -lh ml/trained_models/production/dqn_real_data/dqn_epoch_{10,380}.safetensors
hexdump -C ml/trained_models/production/dqn_real_data/dqn_epoch_10.safetensors | head -5
```

**Expected output**:
```
00000000  58 02 00 00 00 00 00 00  7b 22 6c 61 79 65 72 5f  |X.......{"layer_|
00000010  30 2e 62 69 61 73 22 3a  7b 22 64 74 79 70 65 22  |0.bias":{"dtype"|
```

#### 1.2 PPO Checkpoints (2 variants)
- [ ] **PPO Epoch 380** checkpoint exists and is valid SafeTensors format
  - Path: `/home/jgrusewski/Work/foxhunt/ml/trained_models/production/ppo_real_data/ppo_actor_epoch_380.safetensors`
  - Expected size: ~42,000 bytes (42KB)
  - Verification: SafeTensors header validation
  - Expected explained variance: 0.4469 (BEST checkpoint)

- [ ] **PPO Epoch 500** checkpoint exists and is valid SafeTensors format
  - Path: `/home/jgrusewski/Work/foxhunt/ml/trained_models/production/ppo_real_data/ppo_actor_epoch_500.safetensors`
  - Expected size: ~42,000 bytes (42KB)
  - Verification: SafeTensors header validation
  - Expected explained variance: 0.4386 (final model)

**Command to verify PPO checkpoints**:
```bash
cd /home/jgrusewski/Work/foxhunt
ls -lh ml/trained_models/production/ppo_real_data/ppo_actor_epoch_{380,500}.safetensors
file ml/trained_models/production/ppo_real_data/ppo_actor_epoch_380.safetensors
```

**Expected file type**: `data` (binary SafeTensors format)

#### 1.3 MAMBA-2 Checkpoint
- [ ] **MAMBA-2 Best** checkpoint exists and is valid SafeTensors format
  - Path: `/home/jgrusewski/Work/foxhunt/ml/trained_models/production/mamba2_real_data/mamba2_best.safetensors`
  - Expected size: 150-500 MB (transformer architecture)
  - Verification: SafeTensors header validation
  - Training status: ✅ COMPLETE (from Wave 160)

**Command to verify MAMBA-2 checkpoint**:
```bash
cd /home/jgrusewski/Work/foxhunt
ls -lh ml/trained_models/production/mamba2_real_data/mamba2_best.safetensors
du -h ml/trained_models/production/mamba2_real_data/mamba2_best.safetensors
```

#### 1.4 TFT Checkpoint
- [ ] **TFT Best** checkpoint exists and is valid SafeTensors format
  - Path: `/home/jgrusewski/Work/foxhunt/ml/trained_models/production/tft_real_data/tft_best.safetensors`
  - Expected size: 1.5-2.5 GB (temporal fusion transformer)
  - Verification: SafeTensors header validation
  - Training status: ✅ COMPLETE (from Wave 160)

**Command to verify TFT checkpoint**:
```bash
cd /home/jgrusewski/Work/foxhunt
ls -lh ml/trained_models/production/tft_real_data/tft_best.safetensors
du -h ml/trained_models/production/tft_real_data/tft_best.safetensors
```

### 2. Test Coverage Validation

#### 2.1 Unit Tests
- [ ] All ML library tests passing: `cargo test -p ml --lib`
  - Expected: 574/575 tests passing (99.8%)
  - Known failure: 1 TLOB test (inference-only, not blocking)

- [ ] All trading_service tests passing: `cargo test -p trading_service`
  - Expected: 100% pass rate
  - Critical: Ensemble coordinator tests

- [ ] All integration tests passing: `cargo test --test '*'`
  - Expected: 22/22 E2E tests passing (100%)
  - Critical: Backtesting integration tests

**Command to run comprehensive test suite**:
```bash
cd /home/jgrusewski/Work/foxhunt
cargo test -p ml --lib 2>&1 | tee test_ml_library.log
cargo test -p trading_service 2>&1 | tee test_trading_service.log
cargo test --test '*' 2>&1 | tee test_integration.log
```

**Success criteria**:
- Zero test failures in critical paths (ensemble, inference, backtesting)
- No panics or segfaults
- All async tests complete within timeout (60s)

#### 2.2 Model Inference Tests
- [ ] DQN inference latency < 50μs P99
  - Test: Load checkpoint, run 10,000 predictions with real market data
  - Command: `cargo run -p ml --example test_dqn_inference --release`

- [ ] PPO inference latency < 50μs P99
  - Test: Load checkpoint, run 10,000 predictions with real market data
  - Command: `cargo run -p ml --example test_ppo_inference --release`

- [ ] MAMBA-2 inference latency < 100μs P99
  - Test: Load checkpoint, run 10,000 predictions (larger model)
  - Command: `cargo run -p ml --example test_mamba2_inference --release`

- [ ] TFT inference latency < 150μs P99
  - Test: Load checkpoint, run 10,000 predictions (largest model)
  - Command: `cargo run -p ml --example test_tft_inference --release`

**Expected benchmark results**:
```
DQN Epoch 10:   P50=12μs, P95=35μs, P99=48μs ✅
DQN Epoch 380:  P50=11μs, P95=34μs, P99=47μs ✅
PPO Epoch 380:  P50=13μs, P95=37μs, P99=49μs ✅
PPO Epoch 500:  P50=12μs, P95=36μs, P99=48μs ✅
MAMBA-2:        P50=42μs, P95=85μs, P99=98μs ✅
TFT:            P50=78μs, P95=128μs, P99=147μs ✅
```

### 3. Database Migration Applied

#### 3.1 Ensemble Predictions Table
- [ ] Migration 022: `ensemble_predictions` table exists
  - Command: `psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c '\d ensemble_predictions'`
  - Expected columns:
    - `id` (UUID)
    - `timestamp` (TIMESTAMPTZ)
    - `symbol` (VARCHAR)
    - `ensemble_action` (VARCHAR)
    - `ensemble_confidence` (DOUBLE PRECISION)
    - `disagreement_rate` (DOUBLE PRECISION)
    - Per-model votes (dqn_signal, ppo_signal, mamba2_signal, tft_signal)
    - Per-model confidence (dqn_confidence, ppo_confidence, ...)
    - Per-model weights (dqn_weight, ppo_weight, ...)
    - Execution tracking (order_id, executed_price, pnl)
    - A/B testing (ab_test_id, ab_group)

- [ ] Hypertable created for time-series optimization
  - Command: `psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT * FROM timescaledb_information.hypertables WHERE hypertable_name='ensemble_predictions';"`
  - Expected: 1 row (hypertable active)

#### 3.2 Model Performance Attribution Table
- [ ] Migration 023: `model_performance_attribution` table exists
  - Command: `psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c '\d model_performance_attribution'`
  - Expected columns:
    - `id` (UUID)
    - `timestamp` (TIMESTAMPTZ)
    - `model_id` (VARCHAR)
    - `symbol` (VARCHAR)
    - Performance metrics (total_predictions, correct_predictions, accuracy, total_pnl, sharpe_ratio)
    - Ensemble contribution (avg_weight, avg_confidence)
    - Rolling window (window_hours: 1, 24, 168)

- [ ] Hypertable created for time-series optimization
  - Command: `psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT * FROM timescaledb_information.hypertables WHERE hypertable_name='model_performance_attribution';"`
  - Expected: 1 row (hypertable active)

**Command to apply migrations**:
```bash
cd /home/jgrusewski/Work/foxhunt
cargo sqlx migrate run
```

**Rollback procedure** (if needed):
```bash
cargo sqlx migrate revert --target-version 021
```

### 4. Metrics Collection Operational

#### 4.1 Prometheus Metrics Available
- [ ] Ensemble aggregation latency metric registered
  - Metric: `ensemble_aggregation_latency_microseconds`
  - Type: Histogram
  - Labels: `aggregation_method` (weighted_average, majority_vote)
  - Buckets: [1.0, 5.0, 10.0, 25.0, 50.0, 100.0]

- [ ] Ensemble confidence metric registered
  - Metric: `ensemble_confidence_score`
  - Type: Gauge
  - Labels: `symbol`
  - Range: 0.0 to 1.0

- [ ] Ensemble disagreement metric registered
  - Metric: `ensemble_disagreement_rate`
  - Type: Gauge
  - Labels: `symbol`
  - Range: 0.0 to 1.0

- [ ] Ensemble predictions counter registered
  - Metric: `ensemble_predictions_total`
  - Type: Counter
  - Labels: `action` (buy, sell, hold), `symbol`

- [ ] Per-model weight metric registered
  - Metric: `ensemble_model_weight`
  - Type: Gauge
  - Labels: `model_id`, `symbol`

- [ ] High disagreement counter registered
  - Metric: `ensemble_high_disagreement_total`
  - Type: Counter
  - Labels: `symbol`, `threshold`

- [ ] Per-model P&L contribution metric registered
  - Metric: `ensemble_model_pnl_contribution_dollars`
  - Type: Histogram
  - Labels: `model_id`, `symbol`

- [ ] Checkpoint swap counter registered
  - Metric: `checkpoint_swaps_total`
  - Type: Counter
  - Labels: `model_id`, `status` (success, failed, rollback)

- [ ] A/B test assignment counter registered
  - Metric: `ab_test_assignments_total`
  - Type: Counter
  - Labels: `test_id`, `group` (control, treatment)

- [ ] A/B test metric difference gauge registered
  - Metric: `ab_test_metric_difference`
  - Type: Gauge
  - Labels: `test_id`, `metric` (sharpe_ratio, win_rate, pnl)

**Command to verify metrics registration**:
```bash
curl http://localhost:9092/metrics | grep ensemble
curl http://localhost:9092/metrics | grep checkpoint_swaps
curl http://localhost:9092/metrics | grep ab_test
```

**Expected output** (sample):
```
# HELP ensemble_aggregation_latency_microseconds Time to aggregate signals from all models
# TYPE ensemble_aggregation_latency_microseconds histogram
ensemble_aggregation_latency_microseconds_bucket{aggregation_method="weighted_average",le="1"} 0
ensemble_aggregation_latency_microseconds_bucket{aggregation_method="weighted_average",le="5"} 0
...
```

#### 4.2 Grafana Dashboard Imported
- [ ] Dashboard JSON file created: `/home/jgrusewski/Work/foxhunt/grafana/dashboards/ensemble_production_monitoring.json`
- [ ] Dashboard imported into Grafana: http://localhost:3000
- [ ] 7 panels visible:
  1. Ensemble Confidence & Disagreement (line graphs)
  2. Model Weights (stacked area chart)
  3. Per-Model P&L Attribution (heatmap)
  4. Aggregation Latency (histogram)
  5. High Disagreement Events (counter)
  6. Checkpoint Swap Health (counters)
  7. A/B Test Progress (gauge + counters)

**Command to import dashboard**:
```bash
curl -X POST -H "Content-Type: application/json" \
  -d @grafana/dashboards/ensemble_production_monitoring.json \
  http://admin:foxhunt123@localhost:3000/api/dashboards/db
```

**Verification**:
```bash
curl -s http://admin:foxhunt123@localhost:3000/api/search?query=Ensemble | jq '.[].title'
```

**Expected output**: `"Ensemble ML Production Monitoring"`

### 5. Alerting Configured

#### 5.1 PagerDuty Integration
- [ ] PagerDuty integration key configured in environment
  - Variable: `PAGERDUTY_INTEGRATION_KEY`
  - Location: `/home/jgrusewski/Work/foxhunt/.env` (gitignored)
  - Verification: `grep PAGERDUTY_INTEGRATION_KEY .env`

- [ ] Critical alerts configured:
  - [ ] Checkpoint rollback triggered
  - [ ] Circuit breaker triggered (consecutive losses)
  - [ ] Circuit breaker triggered (daily drawdown)
  - [ ] Circuit breaker triggered (latency spike)
  - [ ] Cascading model failures (all models erroring)

**Alert test procedure**:
```bash
# Test PagerDuty alert (staging only)
curl -X POST https://events.pagerduty.com/v2/enqueue \
  -H 'Content-Type: application/json' \
  -d '{
    "routing_key": "'"$PAGERDUTY_INTEGRATION_KEY"'",
    "event_action": "trigger",
    "payload": {
      "summary": "TEST: Ensemble deployment pre-flight check",
      "severity": "info",
      "source": "foxhunt-trading-service"
    }
  }'
```

**Expected**: Alert visible in PagerDuty dashboard within 30 seconds

#### 5.2 Slack Integration
- [ ] Slack webhook URL configured in environment
  - Variable: `SLACK_WEBHOOK_URL`
  - Channel: `#trading-alerts`
  - Verification: `grep SLACK_WEBHOOK_URL .env`

- [ ] Warning alerts configured:
  - [ ] High disagreement rate (> 60%)
  - [ ] Latency spike (P99 > 75μs)
  - [ ] Accuracy drop (> 5%)
  - [ ] Model weight drift (> 20% change in 1 hour)

**Alert test procedure**:
```bash
# Test Slack alert (staging only)
curl -X POST "$SLACK_WEBHOOK_URL" \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "TEST: Ensemble deployment pre-flight check - All systems operational"
  }'
```

**Expected**: Message visible in `#trading-alerts` Slack channel within 10 seconds

#### 5.3 Email Alerts
- [ ] Email SMTP configuration verified
  - SMTP server: Configured in environment
  - From address: `alerts@foxhunt.trading`
  - To addresses: Engineering team distribution list

- [ ] Daily summary email configured:
  - [ ] P&L attribution per model
  - [ ] A/B test progress
  - [ ] Checkpoint swap health
  - [ ] System uptime and error rate

**Email test procedure**:
```bash
# Test email alert (staging only)
cargo run -p trading_service --bin send_test_alert
```

**Expected**: Test email received within 5 minutes

### 6. Service Health Checks

#### 6.1 Trading Service Health
- [ ] Trading service running: `ps aux | grep trading_service`
- [ ] gRPC port listening: `lsof -i :50052`
- [ ] Health check endpoint responding: `curl http://localhost:8081/health`
  - Expected: `{"status":"healthy","timestamp":"..."}`
- [ ] Metrics endpoint responding: `curl http://localhost:9092/metrics`
  - Expected: Prometheus metrics text format

**Command to verify Trading Service health**:
```bash
grpc_health_probe -addr=localhost:50052
```

**Expected output**: `status: SERVING`

#### 6.2 API Gateway Health
- [ ] API Gateway running: `ps aux | grep api_gateway`
- [ ] gRPC port listening: `lsof -i :50051`
- [ ] Health check endpoint responding: `curl http://localhost:8080/health`
  - Expected: `{"status":"healthy","timestamp":"..."}`

**Command to verify API Gateway health**:
```bash
grpc_health_probe -addr=localhost:50051
```

**Expected output**: `status: SERVING`

#### 6.3 ML Training Service Health
- [ ] ML Training Service running: `ps aux | grep ml_training_service`
- [ ] gRPC port listening: `lsof -i :50054`
- [ ] Health check endpoint responding: `curl http://localhost:8095/health`
  - Expected: `{"status":"healthy","timestamp":"..."}`

**Command to verify ML Training Service health**:
```bash
grpc_health_probe -addr=localhost:50054
```

**Expected output**: `status: SERVING`

#### 6.4 Database Health
- [ ] PostgreSQL accessible: `psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c 'SELECT 1;'`
  - Expected: `?column? \n----------\n        1\n(1 row)`

- [ ] TimescaleDB extension loaded: `psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c '\dx timescaledb'`
  - Expected: `timescaledb | 2.x.x | ...`

- [ ] All migrations applied: `psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT version FROM _sqlx_migrations ORDER BY version DESC LIMIT 1;"`
  - Expected: `023` (or latest migration version)

#### 6.5 Redis Health
- [ ] Redis accessible: `redis-cli -h localhost -p 6379 PING`
  - Expected: `PONG`

- [ ] Redis memory usage: `redis-cli -h localhost -p 6379 INFO memory | grep used_memory_human`
  - Expected: < 1GB (normal operational usage)

#### 6.6 Prometheus & Grafana Health
- [ ] Prometheus targets up: `curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health=="up") | .labels.job'`
  - Expected: `"api_gateway", "trading_service", "backtesting_service", "ml_training_service"`

- [ ] Grafana accessible: `curl -s http://localhost:3000/api/health`
  - Expected: `{"commit":"...","database":"ok","version":"..."}`

### 7. GPU Availability (for inference)

#### 7.1 CUDA Check
- [ ] CUDA toolkit available: `nvcc --version`
  - Expected: `cuda_12.1.r12.1` (or later)

- [ ] NVIDIA driver loaded: `nvidia-smi`
  - Expected: GPU info displayed (RTX 3050 Ti)

- [ ] VRAM available: `nvidia-smi --query-gpu=memory.free --format=csv,noheader`
  - Expected: > 3000 MiB (3GB free for inference)

**Command to verify GPU health**:
```bash
nvidia-smi --query-gpu=name,driver_version,memory.total,memory.free --format=csv
```

**Expected output**:
```
name, driver_version, memory.total [MiB], memory.free [MiB]
NVIDIA GeForce RTX 3050 Ti Laptop GPU, 535.183.01, 4096, 3800
```

#### 7.2 Candle CUDA Backend
- [ ] Candle CUDA build enabled: `cargo build -p ml --features cuda --release 2>&1 | grep -i cuda`
  - Expected: No errors, CUDA feature compiled

- [ ] Candle CUDA test: `cargo test -p ml --features cuda test_cuda_device`
  - Expected: Test passes, CUDA device available

**Fallback plan**: If GPU unavailable, system falls back to CPU (10-50x slower, still functional)

---

## Deployment Prerequisites Checklist Summary

### Critical (Must Pass)
- ✅ All 6 model checkpoints exist and are valid SafeTensors format
- ✅ All unit tests passing (99.8%+)
- ✅ All integration tests passing (100%)
- ✅ Database migrations applied (migrations 022, 023)
- ✅ Prometheus metrics registered (10 ensemble metrics)
- ✅ Trading Service healthy (gRPC + HTTP)
- ✅ PostgreSQL healthy (all migrations applied)

### Important (Should Pass)
- ⚠️ Grafana dashboard imported and visible
- ⚠️ PagerDuty integration configured and tested
- ⚠️ Slack integration configured and tested
- ⚠️ GPU available (CUDA 12.1+, RTX 3050 Ti)

### Optional (Nice to Have)
- 🔵 Email alerts configured
- 🔵 Redis caching operational (optional performance boost)

---

## Pre-Deployment Validation Script

Create automated validation script: `/home/jgrusewski/Work/foxhunt/scripts/pre_deployment_validation.sh`

```bash
#!/bin/bash
set -e

echo "=== Foxhunt Ensemble Pre-Deployment Validation ==="
echo "Date: $(date)"
echo ""

# Colors
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

PASSED=0
FAILED=0
WARNINGS=0

# Function to check and report
check() {
    local name="$1"
    local command="$2"

    echo -n "Checking $name... "
    if eval "$command" > /dev/null 2>&1; then
        echo -e "${GREEN}PASS${NC}"
        ((PASSED++))
        return 0
    else
        echo -e "${RED}FAIL${NC}"
        ((FAILED++))
        return 1
    fi
}

warn() {
    local name="$1"
    local command="$2"

    echo -n "Checking $name... "
    if eval "$command" > /dev/null 2>&1; then
        echo -e "${GREEN}PASS${NC}"
        ((PASSED++))
        return 0
    else
        echo -e "${YELLOW}WARN${NC}"
        ((WARNINGS++))
        return 1
    fi
}

echo "=== 1. Model Checkpoint Verification ==="
check "DQN Epoch 10" "test -f ml/trained_models/production/dqn_real_data/dqn_epoch_10.safetensors"
check "DQN Epoch 380" "test -f ml/trained_models/production/dqn_real_data/dqn_epoch_380.safetensors"
check "PPO Epoch 380" "test -f ml/trained_models/production/ppo_real_data/ppo_actor_epoch_380.safetensors"
check "PPO Epoch 500" "test -f ml/trained_models/production/ppo_real_data/ppo_actor_epoch_500.safetensors"
check "MAMBA-2 Best" "test -f ml/trained_models/production/mamba2_real_data/mamba2_best.safetensors"
check "TFT Best" "test -f ml/trained_models/production/tft_real_data/tft_best.safetensors"

echo ""
echo "=== 2. Service Health Checks ==="
check "Trading Service gRPC" "grpc_health_probe -addr=localhost:50052"
check "API Gateway gRPC" "grpc_health_probe -addr=localhost:50051"
check "ML Training Service gRPC" "grpc_health_probe -addr=localhost:50054"
check "PostgreSQL" "psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c 'SELECT 1;'"
check "Redis" "redis-cli -h localhost -p 6379 PING"

echo ""
echo "=== 3. Database Schema Verification ==="
check "ensemble_predictions table" "psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c '\d ensemble_predictions'"
check "model_performance_attribution table" "psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c '\d model_performance_attribution'"
check "TimescaleDB hypertables" "psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c \"SELECT * FROM timescaledb_information.hypertables WHERE hypertable_name IN ('ensemble_predictions', 'model_performance_attribution');\""

echo ""
echo "=== 4. Metrics Collection Verification ==="
check "Ensemble metrics endpoint" "curl -s http://localhost:9092/metrics | grep ensemble_aggregation_latency"
check "Checkpoint swap metrics" "curl -s http://localhost:9092/metrics | grep checkpoint_swaps_total"
check "A/B test metrics" "curl -s http://localhost:9092/metrics | grep ab_test_assignments_total"

echo ""
echo "=== 5. GPU Availability (Optional) ==="
warn "CUDA Toolkit" "nvcc --version"
warn "NVIDIA Driver" "nvidia-smi"
warn "GPU VRAM > 3GB" "nvidia-smi --query-gpu=memory.free --format=csv,noheader | awk '{if(\$1>3000) exit 0; else exit 1}'"

echo ""
echo "=== Validation Summary ==="
echo -e "${GREEN}PASSED: $PASSED${NC}"
echo -e "${RED}FAILED: $FAILED${NC}"
echo -e "${YELLOW}WARNINGS: $WARNINGS${NC}"

if [ $FAILED -eq 0 ]; then
    echo ""
    echo -e "${GREEN}✅ PRE-DEPLOYMENT VALIDATION SUCCESSFUL${NC}"
    echo "System is ready for Phase 0 (Pre-Production) deployment"
    exit 0
else
    echo ""
    echo -e "${RED}❌ PRE-DEPLOYMENT VALIDATION FAILED${NC}"
    echo "Fix all FAILED checks before proceeding to deployment"
    exit 1
fi
```

**Usage**:
```bash
cd /home/jgrusewski/Work/foxhunt
chmod +x scripts/pre_deployment_validation.sh
./scripts/pre_deployment_validation.sh
```

**Expected output** (all checks pass):
```
=== Foxhunt Ensemble Pre-Deployment Validation ===
Date: 2025-10-14 16:30:00

=== 1. Model Checkpoint Verification ===
Checking DQN Epoch 10... PASS
Checking DQN Epoch 380... PASS
Checking PPO Epoch 380... PASS
Checking PPO Epoch 500... PASS
Checking MAMBA-2 Best... PASS
Checking TFT Best... PASS

=== 2. Service Health Checks ===
Checking Trading Service gRPC... PASS
Checking API Gateway gRPC... PASS
Checking ML Training Service gRPC... PASS
Checking PostgreSQL... PASS
Checking Redis... PASS

=== 3. Database Schema Verification ===
Checking ensemble_predictions table... PASS
Checking model_performance_attribution table... PASS
Checking TimescaleDB hypertables... PASS

=== 4. Metrics Collection Verification ===
Checking Ensemble metrics endpoint... PASS
Checking Checkpoint swap metrics... PASS
Checking A/B test metrics... PASS

=== 5. GPU Availability (Optional) ===
Checking CUDA Toolkit... PASS
Checking NVIDIA Driver... PASS
Checking GPU VRAM > 3GB... PASS

=== Validation Summary ===
PASSED: 19
FAILED: 0
WARNINGS: 0

✅ PRE-DEPLOYMENT VALIDATION SUCCESSFUL
System is ready for Phase 0 (Pre-Production) deployment
```

---

## Phase-Specific Checklists

### Phase 0: Pre-Production (1-2 days)
- [ ] All pre-deployment validation checks passed
- [ ] Staging environment deployed
- [ ] 10,000 predictions executed with zero errors
- [ ] Ensemble coordinator loads all 6 models successfully
- [ ] Inference latency targets met (< 50μs P99 for DQN/PPO)
- [ ] Hot-swap mechanism tested with dummy checkpoints
- [ ] Prometheus metrics reporting correctly
- [ ] PostgreSQL audit logs persisting

### Phase 1: Paper Trading (7 days)
- [ ] Shadow mode enabled (0% real capital)
- [ ] Ensemble predictions logged alongside baseline
- [ ] Compare P&L: ensemble vs current production strategy
- [ ] Monitor disagreement rate (expect 20-40% in volatile markets)
- [ ] A/B testing infrastructure verified
- [ ] Exit criteria met:
  - Sharpe ratio > 1.5
  - Win rate > 52%
  - Zero critical errors or rollbacks

### Phase 2: Small Position (7 days, 1% capital)
- [ ] Real execution enabled with $50K capital
- [ ] Risk limits configured:
  - Max position: $10K per symbol
  - Max daily loss: $5K
  - Circuit breaker: 3 consecutive losses
- [ ] Real P&L positive after transaction costs
- [ ] Slippage within acceptable range (< 5 bps)
- [ ] No execution errors or order rejections
- [ ] Exit criteria met:
  - Sharpe ratio > 1.5
  - Total P&L > $5K
  - Max drawdown < 10%

### Phase 3: Medium Position (14 days, 10% capital)
- [ ] Capital scaled to $500K
- [ ] Risk limits configured:
  - Max position: $100K per symbol
  - Max daily loss: $50K
  - Dynamic position sizing based on confidence
- [ ] Consistent profitability across multiple symbols
- [ ] Ensemble outperforms baseline (A/B test)
- [ ] Model weights stabilize (no wild swings)
- [ ] Exit criteria met:
  - Sharpe ratio > 1.8 (2 weeks)
  - A/B test statistically significant (p < 0.05)
  - Zero checkpoint rollbacks

### Phase 4: Full Deployment (Ongoing, 100% capital)
- [ ] Capital scaled to $5M (full allocation)
- [ ] Risk limits configured:
  - Max position: $1M per symbol
  - Max daily loss: $250K
  - VaR-based position sizing
- [ ] Daily P&L attribution per model
- [ ] Weekly checkpoint updates with hot-swapping
- [ ] Monthly A/B tests for new model variants
- [ ] Quarterly retraining with latest market data

---

## Emergency Rollback Checklist

### Trigger Conditions
- 3 consecutive losses
- Daily drawdown > 5%
- Disagreement rate > 70% for 1 hour
- Latency P99 > 100μs for 5 minutes
- Any model error rate > 5%

### Rollback Procedure
1. [ ] Circuit breaker triggers automatically
2. [ ] Trading halted within 5 seconds
3. [ ] PagerDuty alert sent to on-call engineer
4. [ ] Slack notification posted to `#trading-alerts`
5. [ ] Hot-swap to previous known-good checkpoints
6. [ ] 5-minute canary validation
7. [ ] Resume trading after validation passes
8. [ ] Post-mortem incident report filed

**Command to manually trigger rollback**:
```bash
tli rollout rollback --reason "High disagreement rate detected"
```

---

## Sign-Off

### Pre-Deployment Sign-Off
- [ ] **Engineering Lead**: Verified all technical checks passed
  - Name: ________________
  - Date: ________________

- [ ] **Trading Desk**: Approved rollout plan and risk limits
  - Name: ________________
  - Date: ________________

- [ ] **Risk Management**: Approved circuit breaker thresholds
  - Name: ________________
  - Date: ________________

### Phase Advancement Sign-Off
- [ ] **Phase 0 → Phase 1**: Pre-production validation successful
  - Approved by: ________________
  - Date: ________________

- [ ] **Phase 1 → Phase 2**: Paper trading exit criteria met
  - Approved by: ________________
  - Date: ________________

- [ ] **Phase 2 → Phase 3**: Small position exit criteria met
  - Approved by: ________________
  - Date: ________________

- [ ] **Phase 3 → Phase 4**: Medium position exit criteria met
  - Approved by: ________________
  - Date: ________________

---

**Document Status**: ✅ READY FOR EXECUTION
**Next Action**: Run pre-deployment validation script
**Estimated Timeline**: Phase 0 start within 24 hours (after sign-off)