foxhunt/PAPER_TRADING_DEPLOYMENT_GUIDE.md

# Paper Trading Deployment Guide

**Document Version**: 1.0  
**Last Updated**: 2025-10-14  
**Status**: PRODUCTION READY  
**Target System**: Foxhunt HFT Ensemble ML System  

---

## Executive Summary

This guide provides complete instructions for deploying the 3-model ensemble (DQN epoch 30 + PPO epoch 130 + PPO epoch 420) to paper trading, followed by a gradual 5-phase rollout to 100% real capital.

**Ensemble Configuration**:
- **DQN epoch 30**: Sharpe 1.63, explained_variance 0.84, weight 0.4 (40%)
- **PPO epoch 130**: Sharpe 1.59, explained_variance 0.69, weight 0.4 (40%)
- **PPO epoch 420**: Sharpe 1.48, explained_variance 0.67, weight 0.2 (20%)

**Total Timeline**: 8-10 weeks (1 week paper trading + 7-9 weeks real capital)

**Key Features**:
- Zero-downtime hot-swapping (dual-buffer architecture)
- A/B testing framework (ensemble vs baseline)
- Comprehensive monitoring (10 Prometheus metrics + Grafana dashboard)
- Risk mitigation (circuit breakers, dynamic position sizing)
- 5-phase rollout (paper → 1% → 10% → 50% → 100%)

---

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Quick Start (Copy-Paste Script)](#quick-start)
3. [Monitoring Instructions](#monitoring)
4. [5-Phase Rollout Plan](#rollout-plan)
5. [Rollback Procedures](#rollback)
6. [Troubleshooting](#troubleshooting)
7. [FAQ](#faq)

---

## Prerequisites

### Required Infrastructure

✅ **Services Running**:
```bash
docker-compose ps
# Expected: All services "Up (healthy)"
# - foxhunt-api-gateway
# - foxhunt-trading-service
# - foxhunt-postgres
# - foxhunt-redis
# - foxhunt-prometheus
# - foxhunt-grafana
```

✅ **Model Checkpoints Available**:
```bash
ls -lh ml/trained_models/production/dqn/dqn_epoch_30.safetensors
ls -lh ml/trained_models/production/ppo/ppo_*_epoch_130.safetensors
ls -lh ml/trained_models/production/ppo/ppo_*_epoch_420.safetensors
# Expected: 5 files (1 DQN + 2x2 PPO)
```

✅ **Database Migration Applied**:
```bash
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "\dt ensemble*"
# Expected: ensemble_predictions, model_performance_attribution, ab_test_experiments
```

If tables don't exist, run migration:
```bash
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -f migrations/022_create_ensemble_tables.sql
```

✅ **Monitoring Dashboards**:
- Grafana running: http://localhost:3000
- Prometheus scraping: http://localhost:9090/targets
- Trading Service metrics: http://localhost:9092/metrics

---

## Quick Start (Copy-Paste Script)

### Step 1: Run Smoke Test (2-3 minutes)

```bash
# Verify infrastructure is ready
bash tests/paper_trading_smoke_test.sh

# Expected output:
# Tests Passed: 10/10
# ✅ ALL TESTS PASSED
```

### Step 2: Deploy Paper Trading (5 minutes)

```bash
# Deploy ensemble configuration
bash scripts/deploy_paper_trading.sh

# Expected output:
# ✅ All checkpoints verified (5 files)
# ✅ All services healthy
# ✅ Database tables ready
# 🎉 Deployment Ready
```

### Step 3: Verify Deployment (1 minute)

```bash
# Check Trading Service is using ensemble
docker logs foxhunt-trading-service --tail 50 | grep -i ensemble

# Expected output:
# INFO Loaded 3 ensemble models (DQN, PPO_130, PPO_420)
# INFO Ensemble coordinator initialized successfully

# Check Prometheus metrics
curl -s http://localhost:9092/metrics | grep ensemble | head -10

# Expected output:
# ensemble_confidence_score{symbol="ES.FUT"} 0.72
# ensemble_disagreement_rate{symbol="ES.FUT"} 0.18
# ...
```

### Step 4: Monitor First Predictions (Real-Time)

```bash
# Watch PostgreSQL predictions (updates every 10 seconds)
watch -n 10 'psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT COUNT(*) AS total_predictions, AVG(ensemble_confidence)::NUMERIC(5,3) AS avg_confidence, AVG(disagreement_rate)::NUMERIC(5,3) AS avg_disagreement FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '"'"'1 hour'"'"'"'

# Expected output (after a few minutes):
# total_predictions | avg_confidence | avg_disagreement
#-----------------+----------------+------------------
#              42 |          0.680 |            0.245
```

---

## Monitoring Instructions

### Daily Monitoring (5-10 minutes/day)

#### 1. Grafana Dashboard (Primary Monitoring)

**URL**: http://localhost:3000/d/ensemble-ml-prod

**Panels to Check**:
- **Panel 1**: Ensemble confidence & disagreement (should be stable)
  - Confidence: 0.6-0.8 (healthy)
  - Disagreement: <0.4 (healthy), >0.7 (alert)
  
- **Panel 2**: Model weights (dynamic contribution)
  - Weights should not change drastically (>20% swing = concern)
  
- **Panel 3**: Per-model P&L attribution
  - All 3 models should be profitable
  - If one model is consistently negative → investigate

- **Panel 4**: Aggregation latency
  - P99 < 50μs (target), <100μs (acceptable)
  - >100μs = performance issue

- **Panel 5**: High disagreement events
  - <10 events/hour (normal)
  - >50 events/hour (market regime shift or model issue)

- **Panel 6**: Checkpoint swap health
  - Success rate should be >95%
  - >10% rollback rate = checkpoint quality issue

#### 2. PostgreSQL Daily Queries

**Daily P&L Summary**:
```sql
SELECT
    symbol,
    COUNT(*) AS predictions,
    SUM(pnl) / 100.0 AS total_pnl_dollars,  -- Convert cents to dollars
    AVG(ensemble_confidence)::NUMERIC(5,3) AS avg_confidence,
    AVG(disagreement_rate)::NUMERIC(5,3) AS avg_disagreement,
    COUNT(CASE WHEN pnl > 0 THEN 1 END)::FLOAT / NULLIF(COUNT(CASE WHEN pnl IS NOT NULL THEN 1 END), 0) AS win_rate
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '24 hours'
GROUP BY symbol
ORDER BY total_pnl_dollars DESC;
```

**Top Performing Models (Last 24 Hours)**:
```sql
SELECT * FROM get_top_models_24h() ORDER BY sharpe_ratio DESC;
```

**High Disagreement Events (Possible Regime Shifts)**:
```sql
SELECT * FROM get_high_disagreement_events_24h() LIMIT 20;
```

#### 3. Service Health Checks

```bash
# Quick health check (all services)
docker-compose ps | grep -E "Up.*healthy"

# Trading Service logs (last 100 lines)
docker logs foxhunt-trading-service --tail 100

# Check for errors
docker logs foxhunt-trading-service --tail 500 | grep -i error

# Prometheus targets (should all be UP)
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
```

---

## 5-Phase Rollout Plan

### Phase 1: Paper Trading (7 Days) 🟢 **CURRENT PHASE**

**Objective**: Validate ensemble with zero risk (simulated execution only)

**Configuration**:
- Capital: $100K virtual (no real money)
- Symbols: ES.FUT, NQ.FUT
- Max position: $10K per symbol
- Max daily loss: $2K (circuit breaker)

**Success Criteria** (ALL must pass):
- ✅ Sharpe ratio > 1.5 (7 days)
- ✅ Win rate > 52%
- ✅ Max drawdown < 10%
- ✅ Simulated P&L > $10,000
- ✅ Zero model errors
- ✅ Latency P99 < 50μs
- ✅ 100% uptime

**Monitoring**:
- Daily Grafana dashboard review
- Daily PostgreSQL P&L check
- Daily disagreement rate review

**Exit Criteria**: All success criteria met for 7 consecutive days

---

### Phase 2: 1% Capital (7 Days)

**Objective**: First real capital deployment (conservative allocation)

**Configuration**:
- Capital: 1% of total ($50K if total = $5M)
- Symbols: ES.FUT, NQ.FUT
- Max position: $10K per symbol
- Max daily loss: $5K (circuit breaker)

**Deployment**:
```bash
# Advance to Phase 2 (if TLI integrated)
tli rollout advance --phase small-position --capital-pct 1
# Confirm: "Allocate 1% capital ($50,000) to ensemble? [y/N]" -> y
```

**Success Criteria**:
- ✅ Sharpe ratio > 1.5 (real trading)
- ✅ Real P&L > $5,000 (after costs)
- ✅ Slippage < 5 bps
- ✅ Zero order rejections

**Rollback Trigger**: Daily loss > $5K OR 3 consecutive losing days

---

### Phase 3: 10% Capital (14 Days)

**Objective**: Scale to meaningful allocation + run A/B test

**Configuration**:
- Capital: 10% of total ($500K if total = $5M)
- Symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT
- Max position: $100K per symbol
- Max daily loss: $50K (circuit breaker)

**A/B Test** (Optional):
```bash
# Compare ensemble vs DQN-only baseline (50/50 split)
tli ab start --control DQN_epoch30 --treatment Ensemble --split 50/50 --duration 14d
```

**Success Criteria**:
- ✅ Sharpe ratio > 1.8 (2 weeks)
- ✅ Real P&L > $50,000
- ✅ Max drawdown < 15%
- ✅ A/B test: Ensemble outperforms baseline (p < 0.05)

**Rollback Trigger**: Sharpe drops below 1.2 OR drawdown > 15%

---

### Phase 4: 50% Capital (21 Days)

**Objective**: Major capital allocation + test hot-swapping

**Configuration**:
- Capital: 50% of total ($2.5M if total = $5M)
- Symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT, CL.FUT, GC.FUT
- Max position: $500K per symbol
- Max daily loss: $100K (circuit breaker)

**Hot-Swap Test** (Week 2):
```bash
# Test zero-downtime checkpoint update
tli checkpoint swap --model DQN --checkpoint dqn_epoch_40.safetensors
# System will auto-rollback if latency/accuracy degrades during 5-minute canary
```

**Success Criteria**:
- ✅ Sharpe ratio > 1.8 (3 weeks)
- ✅ Real P&L > $250,000
- ✅ Hot-swap success rate > 95%
- ✅ Model weights stabilize

**Rollback Trigger**: Sharpe drops below 1.3 OR hot-swap failure > 10%

---

### Phase 5: 100% Capital (Ongoing)

**Objective**: Full production deployment

**Configuration**:
- Capital: 100% of total ($5M example)
- Symbols: All futures (ES, NQ, ZN, 6E, CL, GC, YM, RTY)
- Max position: $1M per symbol
- Max daily loss: $250K (circuit breaker)

**Maintenance**:
- Weekly checkpoint updates (hot-swapping)
- Monthly retraining with latest data
- Quarterly A/B tests for new model variants

**Success Criteria** (Ongoing):
- ✅ Sharpe ratio > 1.8 (rolling 90 days)
- ✅ Annualized return > 40%
- ✅ Max drawdown < 20% (rolling 1 year)
- ✅ Uptime > 99.9%

---

## Rollback Procedures

### Scenario 1: Daily Loss Exceeds Threshold

**Trigger**: Daily P&L < -$2,000 (Phase 1) or exceeds phase-specific limit

**Action**:
```bash
# Emergency halt
tli trading emergency-stop --reason "Daily loss threshold exceeded"

# Check recent predictions
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c \
  "SELECT symbol, ensemble_action, pnl FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '4 hours' ORDER BY pnl ASC LIMIT 20"

# Investigate: Are all models losing? Or just one?
# If all models: Market event (stop trading, manual review)
# If one model: Disable that model, continue with 2 models
```

---

### Scenario 2: High Disagreement (>70% for 1 hour)

**Trigger**: `ensemble_disagreement_rate` > 0.70 for 60+ minutes

**Action**:
```bash
# Reduce position sizes by 50% (automatic via config)
# System continues trading but with reduced exposure

# Check disagreement events
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c \
  "SELECT * FROM get_high_disagreement_events_24h() LIMIT 50"

# Investigate: Market regime shift or model issue?
# - Check news/events (Fed announcement, earnings, etc.)
# - Compare model signals: Are they correlated or random?
```

**No rollback needed** (graceful degradation)

---

### Scenario 3: Single Model Failure

**Trigger**: One model has >3 consecutive prediction errors

**Action**:
```bash
# System automatically disables failed model (circuit breaker)
# Ensemble continues with 2 remaining models

# Investigate failed model
docker logs foxhunt-trading-service --tail 500 | grep -i "DQN\|PPO" | grep -i error

# Possible causes:
# - Checkpoint corruption (reload checkpoint)
# - GPU memory issue (restart service)
# - Feature extraction failure (check data pipeline)

# After fix, re-enable model
tli ensemble enable-model --model DQN
```

---

### Scenario 4: Complete Ensemble Failure

**Trigger**: All 3 models fail OR latency P99 > 200μs

**Action**:
```bash
# CRITICAL: Immediate halt
tli trading emergency-stop --reason "Ensemble catastrophic failure"

# Close all positions (aggressive execution)
tli trading close-all --aggressive --max-slippage 10

# Disable ensemble coordinator
curl -X POST http://localhost:8081/api/v1/ensemble/disable

# Fallback to single-model baseline (DQN only)
curl -X POST http://localhost:8081/api/v1/config/revert-baseline

# Trigger PagerDuty incident
# (Automated via alert system)

# Full diagnostic
docker logs foxhunt-trading-service > ensemble_failure_logs.txt
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c \
  "SELECT * FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '2 hours' ORDER BY timestamp DESC" > recent_predictions.csv
```

**Recovery Time Objective**: < 5 minutes

---

## Troubleshooting

### Issue: No Predictions in Database

**Symptom**: `SELECT COUNT(*) FROM ensemble_predictions` returns 0 after 10+ minutes

**Diagnosis**:
```bash
# Check Trading Service logs
docker logs foxhunt-trading-service --tail 100 | grep -i ensemble

# Possible causes:
# 1. Ensemble not initialized
# 2. Market data not flowing
# 3. Feature extraction failure
# 4. Database connection issue
```

**Fix**:
```bash
# Restart Trading Service
docker-compose restart foxhunt-trading-service

# Wait 2 minutes, check again
sleep 120
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT COUNT(*) FROM ensemble_predictions"
```

---

### Issue: Latency P99 > 100μs

**Symptom**: Grafana dashboard shows aggregation latency > 100μs

**Diagnosis**:
```bash
# Check GPU utilization
nvidia-smi

# Check CPU load
top -b -n 1 | head -20

# Check model checkpoint sizes
ls -lh ml/trained_models/production/{dqn,ppo}/*.safetensors
```

**Fix**:
- If GPU memory full: Restart service to clear cache
- If CPU overloaded: Reduce concurrent predictions (config adjustment)
- If checkpoint too large: Use smaller model variant

---

### Issue: High Disagreement Rate (>70%)

**Symptom**: Models constantly disagreeing on direction

**Diagnosis**:
```sql
-- Check model correlation
SELECT * FROM calculate_model_correlation_7d();

-- Expected: Correlation 0.5-0.8 (healthy diversity)
-- Concern: Correlation < 0.2 (models completely uncorrelated)
-- Concern: Correlation > 0.95 (models too similar)
```

**Action**:
- If correlation < 0.2: Possible regime shift (reduce positions, not a bug)
- If correlation > 0.95: Models too similar (need more diversity)

---

### Issue: One Model Always Wrong

**Symptom**: DQN consistently loses money while PPO models profit

**Diagnosis**:
```sql
-- Check per-model P&L attribution
SELECT
    model_id,
    total_pnl / 100.0 AS pnl_dollars,
    accuracy,
    sharpe_ratio
FROM model_performance_attribution
WHERE window_hours = 24
  AND timestamp > NOW() - INTERVAL '7 days'
ORDER BY model_id, timestamp DESC
LIMIT 21;  -- 7 days * 3 models
```

**Fix**:
```bash
# If one model consistently underperforms:
# 1. Reduce its weight (adaptive weighting should do this automatically)
# 2. Retrain model with latest data
# 3. If still bad after retraining → disable model permanently

tli ensemble set-weight --model DQN --weight 0.1  # Reduce to 10%
```

---

## FAQ

### Q1: How long does paper trading take?

**A**: 7 days minimum. Success criteria must be met for 7 consecutive days before advancing to Phase 2.

---

### Q2: What if paper trading fails on day 6?

**A**: Reset the 7-day clock. Fix the issue, restart paper trading from day 1. Do NOT skip to Phase 2 with incomplete validation.

---

### Q3: Can I skip Phase 2 and go directly to 10%?

**A**: **NO**. The gradual rollout is designed to minimize capital risk. Each phase validates the ensemble at increasing scale. Skipping phases violates risk management policy.

---

### Q4: How often should I retrain models?

**A**: 
- **Phase 1-3**: No retraining (use current checkpoints)
- **Phase 4-5**: Monthly retraining with latest 90 days of data
- **Ad-hoc**: Retrain if Sharpe drops by >50% for 2+ weeks

---

### Q5: What's the expected Sharpe ratio?

**A**: 
- **Phase 1 (paper trading)**: > 1.5 (minimum target)
- **Phase 3-5 (real trading)**: > 1.8 (production target)
- **Best case**: 2.0-2.5 (ensemble at peak performance)

---

### Q6: How do I update to new checkpoints?

**A**: Use hot-swapping mechanism (zero downtime):

```bash
# Stage new checkpoint in shadow buffer
tli checkpoint stage --model DQN --checkpoint dqn_epoch_45.safetensors

# Validate with 1,000 test predictions
tli checkpoint validate --model DQN --iterations 1000

# If validation passes, swap atomically (< 1μs)
tli checkpoint swap --model DQN

# System monitors for 5 minutes (canary period)
# Auto-rollback if latency/accuracy degrades
```

---

### Q7: What if Grafana dashboard is down?

**A**: Use PostgreSQL queries as backup monitoring:

```sql
-- Daily summary (equivalent to Grafana dashboard)
SELECT
    symbol,
    COUNT(*) AS predictions,
    AVG(ensemble_confidence)::NUMERIC(5,3) AS avg_confidence,
    AVG(disagreement_rate)::NUMERIC(5,3) AS avg_disagreement,
    AVG(inference_latency_us) AS avg_latency_us,
    SUM(pnl) / 100.0 AS total_pnl_dollars
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '24 hours'
GROUP BY symbol;
```

---

### Q8: How do I enable A/B testing?

**A**: Edit `config/paper_trading_config.yaml`:

```yaml
ab_testing:
  enabled: true  # Change from false to true
  test_id: "ab_test_$(date +%Y%m%d)"  # Generate unique ID
  control_model: DQN_epoch30  # Baseline
  treatment_model: ensemble  # Ensemble
  traffic_split: 0.5  # 50/50 split
  min_sample_size: 1000
```

Then restart Trading Service: `docker-compose restart foxhunt-trading-service`

---

### Q9: What's the emergency stop procedure?

**A**: See [Scenario 4: Complete Ensemble Failure](#scenario-4-complete-ensemble-failure) in Rollback Procedures section.

**Summary**:
1. `tli trading emergency-stop`
2. Close all positions
3. Disable ensemble
4. Revert to baseline
5. Trigger incident

---

### Q10: Who do I contact for help?

**A**:
- **PagerDuty**: ml-team@foxhunt.trading (24/7 on-call)
- **Slack**: #trading-alerts (warnings, questions)
- **Email**: risk-management@foxhunt.trading (risk concerns)
- **Escalation**: CTO (critical failures only)

---

## Appendix: Configuration File Reference

### Config File Location

`/home/jgrusewski/Work/foxhunt/config/paper_trading_config.yaml`

### Key Sections

**Paper Trading Settings**:
- `mode`: `paper` (DO NOT change to `live` until Phase 5)
- `initial_capital`: $100,000 (virtual)
- `max_position_size`: $10,000 per symbol
- `max_daily_loss`: $2,000 (circuit breaker)

**Ensemble Models**:
- 3 models: DQN epoch 30 (40%), PPO epoch 130 (40%), PPO epoch 420 (20%)
- `voting_strategy`: `weighted` (weighted average of signals)
- `consensus_threshold`: 0.6 (60% agreement required)

**Risk Management**:
- `per_model_circuit_breaker`: Enabled (disable model after 3 errors)
- `cascade_failure_threshold`: 2 (halt if 2+ models fail)
- `confidence_based_sizing`: Enabled (scale positions by confidence)

**Monitoring**:
- `log_all_predictions`: true (full audit trail)
- `prometheus_metrics`: Enabled (export to port 9092)
- `grafana_dashboard`: http://localhost:3000/d/ensemble-ml-prod

---

## Appendix: Database Schema Reference

### Table: `ensemble_predictions`

**Purpose**: Audit log of every ensemble prediction with per-model attribution

**Key Columns**:
- `ensemble_action`: BUY, SELL, HOLD
- `ensemble_confidence`: 0.0-1.0 (prediction confidence)
- `disagreement_rate`: 0.0-1.0 (model disagreement)
- `dqn_signal`, `ppo_signal`, `mamba2_signal`, `tft_signal`: Per-model votes
- `pnl`: Profit/loss in cents (populated after trade closes)

**Useful Queries**:
```sql
-- Daily P&L by symbol
SELECT symbol, SUM(pnl) / 100.0 AS pnl_dollars FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '24 hours' GROUP BY symbol;

-- Win rate
SELECT COUNT(CASE WHEN pnl > 0 THEN 1 END)::FLOAT / COUNT(*) AS win_rate FROM ensemble_predictions WHERE pnl IS NOT NULL;
```

---

### Table: `model_performance_attribution`

**Purpose**: Rolling performance metrics per model for adaptive weighting

**Key Columns**:
- `model_id`: DQN, PPO, MAMBA2, TFT
- `window_hours`: 1, 24, 168 (rolling window)
- `sharpe_ratio`: Risk-adjusted returns
- `avg_weight`: Average weight in ensemble

**Useful Queries**:
```sql
-- Top performing models (last 24h)
SELECT * FROM get_top_models_24h();

-- Model correlation
SELECT * FROM calculate_model_correlation_7d();
```

---

## Document Change Log

| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0 | 2025-10-14 | ML Team | Initial release |

---

**End of Guide**

**Last Updated**: 2025-10-14  
**Next Review**: After Phase 1 completion (7 days)  
**Approvals Required**: ML Lead, Trading Desk, Risk Management, CTO