foxhunt/PAPER_TRADING_DEPLOYMENT_CHECKLIST.md

# Paper Trading Deployment Checklist

**Document Version**: 1.0
**Last Updated**: 2025-10-14
**Status**: Ready for Execution
**Target System**: Foxhunt HFT Ensemble ML System

---

## Executive Summary

This checklist provides a step-by-step guide for deploying the 3-model ensemble (DQN + 2x PPO) to paper trading, followed by a 5-phase gradual rollout to 100% real capital. Each phase has clear success criteria and rollback procedures.

**Total Timeline**: 8-10 weeks (1 week paper trading + 7-9 weeks real capital ramp)

**Ensemble Configuration**:
- DQN epoch 30 (Sharpe 1.63, weight 0.4)
- PPO epoch 130 (Sharpe 1.59, weight 0.4)
- PPO epoch 420 (Sharpe 1.48, weight 0.2)

---

## Pre-Deployment Prerequisites (Before Phase 1)

### ✅ Infrastructure Verification

- [ ] **PostgreSQL Tables Created**
  - [ ] `ensemble_predictions` table exists
  - [ ] `model_performance_attribution` table exists
  - [ ] `ab_test_experiments` table exists
  - [ ] Verify: `psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "\dt ensemble*"`

- [ ] **Model Checkpoints Available**
  - [ ] DQN epoch 30: `ml/trained_models/production/dqn/dqn_epoch_30.safetensors`
  - [ ] PPO epoch 130 actor: `ml/trained_models/production/ppo/ppo_actor_epoch_130.safetensors`
  - [ ] PPO epoch 130 critic: `ml/trained_models/production/ppo/ppo_critic_epoch_130.safetensors`
  - [ ] PPO epoch 420 actor: `ml/trained_models/production/ppo/ppo_actor_epoch_420.safetensors`
  - [ ] PPO epoch 420 critic: `ml/trained_models/production/ppo/ppo_critic_epoch_420.safetensors`
  - [ ] Verify: `ls -lh ml/trained_models/production/{dqn,ppo}/*.safetensors`

- [ ] **Services Running & Healthy**
  - [ ] API Gateway (port 50051): `grpc_health_probe -addr=localhost:50051`
  - [ ] Trading Service (port 50052): `grpc_health_probe -addr=localhost:50052`
  - [ ] ML Training Service (port 50054): `grpc_health_probe -addr=localhost:50054`
  - [ ] PostgreSQL: `psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT 1"`
  - [ ] Redis: `redis-cli -h localhost -p 6379 PING`
  - [ ] Prometheus: `curl http://localhost:9090/api/v1/targets | jq`
  - [ ] Grafana: `curl http://localhost:3000/api/health`

- [ ] **Monitoring Dashboards Configured**
  - [ ] Grafana dashboard imported: `monitoring/grafana/ensemble_ml_production.json`
  - [ ] Prometheus scraping Trading Service metrics (port 9092)
  - [ ] Verify metrics: `curl http://localhost:9092/metrics | grep ensemble`

- [ ] **Configuration Files Loaded**
  - [ ] `config/paper_trading_config.yaml` exists and validated
  - [ ] Trading Service can read config (no syntax errors)

### ✅ Smoke Test (Run Before Deployment)

```bash
# Run smoke test script
bash tests/paper_trading_smoke_test.sh

# Expected output:
# ✅ All 3 models load successfully
# ✅ Inference latency < 50μs P99
# ✅ Ensemble prediction generates valid output
# ✅ PostgreSQL audit log working
# ✅ Prometheus metrics exported
```

---

## Phase 1: Paper Trading (Duration: 7 days)

### Objectives
- Validate ensemble predictions with real market data
- Zero risk (simulated execution only)
- Compare ensemble P&L vs existing production strategy

### Risk Limits
- **Capital**: $100K virtual (no real money)
- **Max Position Size**: $10K per symbol
- **Max Daily Loss**: $2K (circuit breaker)
- **Symbols**: ES.FUT, NQ.FUT only

### Deployment Steps

1. **Load Configuration**
   ```bash
   # Copy paper trading config to production location
   cp config/paper_trading_config.yaml /opt/foxhunt/config/active/

   # Reload Trading Service configuration
   curl -X POST http://localhost:8081/api/v1/config/reload
   ```

2. **Start Paper Trading Mode**
   ```bash
   # Via TLI (if integrated)
   tli trading start --mode paper --capital 100000 --symbols ES.FUT,NQ.FUT

   # OR via gRPC (if TLI not integrated)
   grpcurl -plaintext -d '{
     "mode": "paper",
     "initial_capital": 100000,
     "symbols": ["ES.FUT", "NQ.FUT"]
   }' localhost:50052 TradingService/StartPaperTrading
   ```

3. **Verify Paper Trading Active**
   ```bash
   # Check Trading Service logs
   docker logs foxhunt-trading-service --tail 50 | grep "Paper trading"

   # Expected output:
   # INFO Paper trading mode enabled (capital: $100,000)
   # INFO Loaded 3 ensemble models (DQN, PPO_130, PPO_420)
   # INFO Ensemble coordinator initialized successfully
   ```

4. **Monitor for 7 Days**
   - Daily check Grafana dashboard: http://localhost:3000/d/ensemble-ml-prod
   - Daily check PostgreSQL predictions: `SELECT COUNT(*), AVG(ensemble_confidence) FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '24 hours'`
   - Daily check P&L: `SELECT SUM(pnl) FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '24 hours'`

### Success Criteria (All Must Pass)

- [ ] **Performance Metrics**
  - [ ] Sharpe ratio > 1.5 (over 7 days)
  - [ ] Win rate > 52%
  - [ ] Max drawdown < 10%
  - [ ] Total simulated P&L > $10,000

- [ ] **Technical Metrics**
  - [ ] Inference latency P99 < 50μs
  - [ ] Zero model prediction errors
  - [ ] Zero circuit breaker triggers
  - [ ] Disagreement rate < 40% (average)

- [ ] **Operational Metrics**
  - [ ] 100% uptime (no crashes or restarts)
  - [ ] All predictions logged to PostgreSQL
  - [ ] Prometheus metrics exported without gaps

### Rollback Procedure (If Failure)

```bash
# Stop paper trading immediately
tli trading stop --mode paper

# Disable ensemble coordinator
curl -X POST http://localhost:8081/api/v1/ensemble/disable

# Revert to single-model baseline (DQN only)
curl -X POST http://localhost:8081/api/v1/config/revert-baseline

# Investigate failure
docker logs foxhunt-trading-service --tail 500 > paper_trading_failure_logs.txt
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c \
  "SELECT * FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '1 hour' ORDER BY timestamp DESC LIMIT 100"
```

---

## Phase 2: Small Position (1% Capital) - Duration: 7 days

### Objectives
- First real capital deployment (conservative)
- Validate execution with real slippage/commission
- Confirm profitability after transaction costs

### Risk Limits
- **Capital**: 1% of total ($50K if total capital = $5M)
- **Max Position Size**: $10K per symbol
- **Max Daily Loss**: $5K (circuit breaker)
- **Symbols**: ES.FUT, NQ.FUT

### Deployment Steps

1. **Advance to Phase 2**
   ```bash
   tli rollout advance --phase small-position --capital-pct 1
   # Confirm: "Allocate 1% capital ($50,000) to ensemble? [y/N]" -> y
   ```

2. **Monitor Real Execution**
   - Track actual fills: `SELECT * FROM orders WHERE ensemble_order = true`
   - Track slippage: `SELECT AVG(slippage_bps) FROM ensemble_predictions WHERE executed_price IS NOT NULL`
   - Track commission: `SELECT SUM(commission) FROM ensemble_predictions WHERE executed_price IS NOT NULL`

### Success Criteria (All Must Pass)

- [ ] **Performance Metrics**
  - [ ] Sharpe ratio > 1.5 (real trading, 7 days)
  - [ ] Real P&L > $5,000 (after costs)
  - [ ] Max drawdown < 10%
  - [ ] Slippage < 5 basis points (average)

- [ ] **Technical Metrics**
  - [ ] Zero order rejections
  - [ ] Zero execution errors
  - [ ] Latency P99 < 50μs

- [ ] **Operational Metrics**
  - [ ] 100% uptime
  - [ ] No rollbacks or emergency stops

### Rollback Procedure (If Failure)

```bash
# Immediately halt trading
tli rollout rollback --reason "Phase 2 failure: [specific reason]"

# Close all open positions
tli trading close-all --reason "Rollback to baseline"

# Revert to baseline strategy
curl -X POST http://localhost:8081/api/v1/config/revert-baseline

# Incident report
tli trading report --phase phase2 --output phase2_incident_report.json
```

---

## Phase 3: Medium Position (10% Capital) - Duration: 14 days

### Objectives
- Scale to meaningful capital allocation
- Validate consistency across multiple symbols
- Run A/B test (ensemble vs single-model baseline)

### Risk Limits
- **Capital**: 10% of total ($500K if total = $5M)
- **Max Position Size**: $100K per symbol
- **Max Daily Loss**: $50K (circuit breaker)
- **Symbols**: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT

### Deployment Steps

1. **Advance to Phase 3**
   ```bash
   tli rollout advance --phase medium-position --capital-pct 10
   ```

2. **Start A/B Test (Optional)**
   ```bash
   # Compare ensemble vs DQN-only baseline
   tli ab start --control DQN_epoch30 --treatment Ensemble --split 50/50 --duration 14d
   ```

3. **Monitor Multi-Symbol Performance**
   - Daily P&L per symbol
   - Model weight adjustments (adaptive weighting)
   - Disagreement patterns per symbol

### Success Criteria (All Must Pass)

- [ ] **Performance Metrics**
  - [ ] Sharpe ratio > 1.8 (2 weeks, real trading)
  - [ ] Real P&L > $50,000 (after costs)
  - [ ] Max drawdown < 15%
  - [ ] Win rate > 54%

- [ ] **A/B Test Results (If Running)**
  - [ ] Ensemble outperforms baseline (p < 0.05)
  - [ ] Sharpe lift > 10% (statistical significance)
  - [ ] No degradation in worst-case scenarios

- [ ] **Technical Metrics**
  - [ ] Zero checkpoint rollbacks
  - [ ] Model weights stabilize (no wild swings)

### Rollback Procedure (If Failure)

```bash
# Halt trading and rollback
tli rollout rollback --phase phase3 --reason "[specific reason]"

# Stop A/B test if running
tli ab stop --test-id [uuid]

# Close all positions
tli trading close-all

# Generate failure report
tli trading report --phase phase3 --output phase3_failure_report.json
```

---

## Phase 4: Large Position (50% Capital) - Duration: 21 days

### Objectives
- Major capital allocation (production scale)
- Validate performance under large position sizes
- Test hot-swapping mechanism (weekly checkpoint updates)

### Risk Limits
- **Capital**: 50% of total ($2.5M if total = $5M)
- **Max Position Size**: $500K per symbol
- **Max Daily Loss**: $100K (circuit breaker)
- **Symbols**: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT, CL.FUT, GC.FUT

### Deployment Steps

1. **Advance to Phase 4**
   ```bash
   tli rollout advance --phase large-position --capital-pct 50
   ```

2. **Test Hot-Swapping (Week 2)**
   ```bash
   # Upload new checkpoint to MinIO (from ML training)
   mc cp ml/trained_models/dqn_epoch_40.safetensors foxhunt/checkpoints/dqn/

   # Trigger hot-swap
   tli checkpoint swap --model DQN --checkpoint dqn_epoch_40.safetensors

   # Monitor canary period (5 minutes)
   # System will auto-rollback if latency/accuracy degrades
   ```

3. **Monitor Large Positions**
   - Track market impact (price movement from trades)
   - Validate VaR-based position sizing
   - Monitor correlation with baseline strategy

### Success Criteria (All Must Pass)

- [ ] **Performance Metrics**
  - [ ] Sharpe ratio > 1.8 (3 weeks, real trading)
  - [ ] Real P&L > $250,000 (after costs)
  - [ ] Max drawdown < 15%
  - [ ] Annualized return > 40%

- [ ] **Technical Metrics**
  - [ ] Hot-swap success rate > 95%
  - [ ] Zero unplanned downtime
  - [ ] Latency P99 < 50μs (even under large positions)

- [ ] **Operational Metrics**
  - [ ] Weekly checkpoint updates without issues
  - [ ] Model weights adapt to changing markets

### Rollback Procedure (If Failure)

```bash
# Emergency halt
tli rollout rollback --phase phase4 --reason "[specific reason]"

# Close all large positions (may take time)
tli trading close-all --aggressive

# Reduce to Phase 3 allocation (10%)
tli rollout revert --phase medium-position

# Full incident analysis
tli trading report --phase phase4 --full-analysis --output phase4_incident.json
```

---

## Phase 5: Full Deployment (100% Capital) - Ongoing

### Objectives
- Full production deployment (all capital managed by ensemble)
- Continuous monitoring and adaptation
- Regular checkpoint updates (weekly)

### Risk Limits
- **Capital**: 100% of total ($5M example)
- **Max Position Size**: $1M per symbol
- **Max Daily Loss**: $250K (circuit breaker)
- **Symbols**: All futures (ES, NQ, ZN, 6E, CL, GC, YM, RTY)

### Deployment Steps

1. **Advance to Phase 5**
   ```bash
   tli rollout advance --phase full-deployment --capital-pct 100
   # Requires manual confirmation + risk management approval
   ```

2. **Continuous Monitoring (Daily)**
   - Grafana dashboard: http://localhost:3000/d/ensemble-ml-prod
   - Daily P&L report (automated email)
   - Weekly risk review meeting

3. **Regular Maintenance**
   - Weekly checkpoint updates (hot-swapping)
   - Monthly retraining with latest data
   - Quarterly A/B tests for new model variants

### Success Criteria (Ongoing)

- [ ] **Performance Metrics**
  - [ ] Sharpe ratio > 1.8 (rolling 90 days)
  - [ ] Annualized return > 40%
  - [ ] Max drawdown < 20% (rolling 1 year)
  - [ ] Win rate > 54%

- [ ] **Technical Metrics**
  - [ ] Uptime > 99.9%
  - [ ] Latency P99 < 50μs
  - [ ] Zero data loss (all predictions logged)

- [ ] **Operational Metrics**
  - [ ] Hot-swap success rate > 95%
  - [ ] Alert noise < 10 false positives/week
  - [ ] Time to rollback < 5 minutes

### Emergency Rollback Procedure

```bash
# CRITICAL FAILURE - Immediate action required

# 1. Halt all trading (< 1 second)
tli trading emergency-stop --reason "CRITICAL: [reason]"

# 2. Close all positions (aggressive execution)
tli trading close-all --aggressive --max-slippage 10

# 3. Disable ensemble coordinator
curl -X POST http://localhost:8081/api/v1/ensemble/disable

# 4. Fallback to single-model baseline
curl -X POST http://localhost:8081/api/v1/config/revert-baseline

# 5. Trigger PagerDuty incident
tli alert trigger --severity critical --message "Ensemble emergency stop"

# 6. Full system diagnostic
tli system diagnose --output emergency_diagnostic.json

# 7. Notify trading desk + risk management
# (Automated via PagerDuty/Slack)
```

---

## Post-Deployment Monitoring Checklist (Daily)

### Daily Tasks (5-10 minutes)

- [ ] Check Grafana dashboard for anomalies
  - [ ] Ensemble confidence trend
  - [ ] Disagreement rate (alert if >70%)
  - [ ] Latency P99 (alert if >100μs)
  - [ ] Per-model P&L attribution

- [ ] Review PostgreSQL audit logs
  ```sql
  -- Daily P&L summary
  SELECT
    symbol,
    COUNT(*) AS predictions,
    SUM(pnl) AS total_pnl,
    AVG(ensemble_confidence) AS avg_confidence,
    AVG(disagreement_rate) AS avg_disagreement
  FROM ensemble_predictions
  WHERE timestamp > NOW() - INTERVAL '24 hours'
  GROUP BY symbol;
  ```

- [ ] Check for circuit breaker triggers
  ```bash
  # Query Prometheus for circuit breaker alerts
  curl http://localhost:9090/api/v1/query?query='circuit_breaker_triggered_total' | jq
  ```

- [ ] Review model performance attribution
  ```sql
  -- Top performing models (last 24 hours)
  SELECT * FROM get_top_models_24h() ORDER BY sharpe_ratio DESC;
  ```

### Weekly Tasks (30-60 minutes)

- [ ] Generate weekly performance report
  ```bash
  tli trading report --period week --output weekly_report_$(date +%Y%m%d).pdf
  ```

- [ ] Review model weight adjustments (adaptive weighting)
  ```sql
  SELECT
    model_id,
    AVG(avg_weight) AS avg_weight_7d,
    AVG(sharpe_ratio) AS avg_sharpe_7d
  FROM model_performance_attribution
  WHERE window_hours = 24 AND timestamp > NOW() - INTERVAL '7 days'
  GROUP BY model_id;
  ```

- [ ] Check for high disagreement events (regime shifts)
  ```sql
  SELECT * FROM get_high_disagreement_events_24h() LIMIT 20;
  ```

- [ ] Hot-swap checkpoint (if new checkpoint available)
  ```bash
  # Test new checkpoint in shadow mode
  tli checkpoint stage --model DQN --checkpoint dqn_epoch_45.safetensors
  tli checkpoint validate --model DQN --iterations 1000
  tli checkpoint swap --model DQN  # Atomic swap
  ```

### Monthly Tasks (2-4 hours)

- [ ] Comprehensive performance review
  - [ ] 30-day Sharpe ratio trend
  - [ ] Drawdown analysis
  - [ ] Model correlation matrix
  - [ ] A/B test results (if running)

- [ ] Retrain models with latest data
  ```bash
  # Trigger model retraining (ML Training Service)
  tli train start --model DQN --epochs 50 --data-range 90d
  ```

- [ ] Review alert thresholds (adjust if needed)
  - [ ] Too many false positives? Increase thresholds
  - [ ] Missed critical events? Decrease thresholds

- [ ] Compliance audit
  - [ ] Verify all trades logged
  - [ ] Check feature snapshots for reproducibility
  - [ ] Validate audit trail completeness

---

## Rollback Decision Matrix

| Failure Type | Severity | Rollback Action | Recovery Time |
|--------------|----------|-----------------|---------------|
| **Single model prediction error** | Low | Disable model, continue with 2 models | 1 hour |
| **High disagreement (>70% for 1 hour)** | Medium | Reduce position sizes by 50% | No rollback |
| **Latency spike (P99 >100μs)** | High | Halt trading, investigate | 5 minutes |
| **Circuit breaker triggered (2+ models)** | Critical | Emergency stop, close positions | Immediate |
| **Daily loss exceeds threshold** | Critical | Halt trading, risk review | 1 day |
| **Checkpoint rollback failure** | Critical | Emergency stop, revert to baseline | 5 minutes |
| **Database connection loss** | Critical | Emergency stop, cache to Redis | Immediate |
| **GPU failure (all models down)** | Critical | Fallback to CPU, reduce positions | 10 minutes |

---

## Success Metrics Summary

| Phase | Duration | Capital | Sharpe | Win Rate | Max DD | P&L Target |
|-------|----------|---------|--------|----------|--------|------------|
| **Phase 1: Paper Trading** | 7 days | $100K (virtual) | >1.5 | >52% | <10% | >$10K (sim) |
| **Phase 2: 1% Capital** | 7 days | $50K | >1.5 | >52% | <10% | >$5K |
| **Phase 3: 10% Capital** | 14 days | $500K | >1.8 | >54% | <15% | >$50K |
| **Phase 4: 50% Capital** | 21 days | $2.5M | >1.8 | >54% | <15% | >$250K |
| **Phase 5: 100% Capital** | Ongoing | $5M | >1.8 | >54% | <20% | >$1M/year |

---

## Contacts & Escalation

### On-Call Engineers
- **Primary**: ml-team@foxhunt.trading (PagerDuty)
- **Secondary**: risk-management@foxhunt.trading
- **Escalation**: CTO (for critical failures only)

### Alert Channels
- **PagerDuty**: Critical alerts (circuit breaker, emergency stop)
- **Slack**: #trading-alerts (warnings, high disagreement)
- **Email**: Daily summary reports

### Documentation
- **This Checklist**: Paper trading deployment steps
- **Ensemble Strategy**: ENSEMBLE_PRODUCTION_DEPLOYMENT_STRATEGY.md
- **Runbooks**: /docs/runbooks/ensemble_troubleshooting.md

---

## Approval Signatures

| Role | Name | Signature | Date |
|------|------|-----------|------|
| **ML Engineering Lead** | _____________ | _____________ | _______ |
| **Trading Desk Manager** | _____________ | _____________ | _______ |
| **Risk Management** | _____________ | _____________ | _______ |
| **CTO** | _____________ | _____________ | _______ |

---

**End of Checklist**

**Last Updated**: 2025-10-14
**Next Review**: After Phase 1 completion (7 days)