foxhunt/PAPER_TRADING_DEPLOYMENT_GUIDE.md

Paper Trading Deployment Guide

Document Version: 1.0
Last Updated: 2025-10-14
Status: PRODUCTION READY
Target System: Foxhunt HFT Ensemble ML System

---

Executive Summary

This guide provides complete instructions for deploying the 3-model ensemble (DQN epoch 30 + PPO epoch 130 + PPO epoch 420) to paper trading, followed by a gradual 5-phase rollout to 100% real capital.

Ensemble Configuration:

• DQN epoch 30: Sharpe 1.63, explained_variance 0.84, weight 0.4 (40%)
• PPO epoch 130: Sharpe 1.59, explained_variance 0.69, weight 0.4 (40%)
• PPO epoch 420: Sharpe 1.48, explained_variance 0.67, weight 0.2 (20%)

Total Timeline: 8-10 weeks (1 week paper trading + 7-9 weeks real capital)

Key Features:

• Zero-downtime hot-swapping (dual-buffer architecture)
• A/B testing framework (ensemble vs baseline)
• Comprehensive monitoring (10 Prometheus metrics + Grafana dashboard)
• Risk mitigation (circuit breakers, dynamic position sizing)
• 5-phase rollout (paper → 1% → 10% → 50% → 100%)

---

Table of Contents

1. Prerequisites
2. Quick Start (Copy-Paste Script)
3. Monitoring Instructions
4. 5-Phase Rollout Plan
5. Rollback Procedures
6. Troubleshooting
7. FAQ

---

Prerequisites

Required Infrastructure

✅ Services Running:

docker-compose ps
# Expected: All services "Up (healthy)"
# - foxhunt-api-gateway
# - foxhunt-trading-service
# - foxhunt-postgres
# - foxhunt-redis
# - foxhunt-prometheus
# - foxhunt-grafana

✅ Model Checkpoints Available:

ls -lh ml/trained_models/production/dqn/dqn_epoch_30.safetensors
ls -lh ml/trained_models/production/ppo/ppo_*_epoch_130.safetensors
ls -lh ml/trained_models/production/ppo/ppo_*_epoch_420.safetensors
# Expected: 5 files (1 DQN + 2x2 PPO)

✅ Database Migration Applied:

psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "\dt ensemble*"
# Expected: ensemble_predictions, model_performance_attribution, ab_test_experiments

If tables don't exist, run migration:

psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -f migrations/022_create_ensemble_tables.sql

✅ Monitoring Dashboards:

• Grafana running: http://localhost:3000
• Prometheus scraping: http://localhost:9090/targets
• Trading Service metrics: http://localhost:9092/metrics

---

Quick Start (Copy-Paste Script)

Step 1: Run Smoke Test (2-3 minutes)

# Verify infrastructure is ready
bash tests/paper_trading_smoke_test.sh

# Expected output:
# Tests Passed: 10/10
# ✅ ALL TESTS PASSED

Step 2: Deploy Paper Trading (5 minutes)

# Deploy ensemble configuration
bash scripts/deploy_paper_trading.sh

# Expected output:
# ✅ All checkpoints verified (5 files)
# ✅ All services healthy
# ✅ Database tables ready
# 🎉 Deployment Ready

Step 3: Verify Deployment (1 minute)

# Check Trading Service is using ensemble
docker logs foxhunt-trading-service --tail 50 | grep -i ensemble

# Expected output:
# INFO Loaded 3 ensemble models (DQN, PPO_130, PPO_420)
# INFO Ensemble coordinator initialized successfully

# Check Prometheus metrics
curl -s http://localhost:9092/metrics | grep ensemble | head -10

# Expected output:
# ensemble_confidence_score{symbol="ES.FUT"} 0.72
# ensemble_disagreement_rate{symbol="ES.FUT"} 0.18
# ...

Step 4: Monitor First Predictions (Real-Time)

# Watch PostgreSQL predictions (updates every 10 seconds)
watch -n 10 'psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT COUNT(*) AS total_predictions, AVG(ensemble_confidence)::NUMERIC(5,3) AS avg_confidence, AVG(disagreement_rate)::NUMERIC(5,3) AS avg_disagreement FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '"'"'1 hour'"'"'"'

# Expected output (after a few minutes):
# total_predictions | avg_confidence | avg_disagreement
#-----------------+----------------+------------------
#              42 |          0.680 |            0.245

---

Monitoring Instructions

Daily Monitoring (5-10 minutes/day)

1. Grafana Dashboard (Primary Monitoring)

URL: http://localhost:3000/d/ensemble-ml-prod

Panels to Check:

• Panel 1: Ensemble confidence & disagreement (should be stable)

  • Confidence: 0.6-0.8 (healthy)
  • Disagreement: <0.4 (healthy), >0.7 (alert)

• Panel 2: Model weights (dynamic contribution)

  • Weights should not change drastically (>20% swing = concern)

• Panel 3: Per-model P&L attribution

  • All 3 models should be profitable
  • If one model is consistently negative → investigate

• Panel 4: Aggregation latency

  • P99 < 50μs (target), <100μs (acceptable)

  • 100μs = performance issue

• Panel 5: High disagreement events

  • <10 events/hour (normal)

  • 50 events/hour (market regime shift or model issue)

• Panel 6: Checkpoint swap health

  • Success rate should be >95%

  • 10% rollback rate = checkpoint quality issue

2. PostgreSQL Daily Queries

Daily P&L Summary:

SELECT
    symbol,
    COUNT(*) AS predictions,
    SUM(pnl) / 100.0 AS total_pnl_dollars,  -- Convert cents to dollars
    AVG(ensemble_confidence)::NUMERIC(5,3) AS avg_confidence,
    AVG(disagreement_rate)::NUMERIC(5,3) AS avg_disagreement,
    COUNT(CASE WHEN pnl > 0 THEN 1 END)::FLOAT / NULLIF(COUNT(CASE WHEN pnl IS NOT NULL THEN 1 END), 0) AS win_rate
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '24 hours'
GROUP BY symbol
ORDER BY total_pnl_dollars DESC;

Top Performing Models (Last 24 Hours):

SELECT * FROM get_top_models_24h() ORDER BY sharpe_ratio DESC;

High Disagreement Events (Possible Regime Shifts):

SELECT * FROM get_high_disagreement_events_24h() LIMIT 20;

3. Service Health Checks

# Quick health check (all services)
docker-compose ps | grep -E "Up.*healthy"

# Trading Service logs (last 100 lines)
docker logs foxhunt-trading-service --tail 100

# Check for errors
docker logs foxhunt-trading-service --tail 500 | grep -i error

# Prometheus targets (should all be UP)
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

---

5-Phase Rollout Plan

Phase 1: Paper Trading (7 Days) 🟢 CURRENT PHASE

Objective: Validate ensemble with zero risk (simulated execution only)

Configuration:

• Capital: $100K virtual (no real money)
• Symbols: ES.FUT, NQ.FUT
• Max position: $10K per symbol
• Max daily loss: $2K (circuit breaker)

Success Criteria (ALL must pass):

• ✅ Sharpe ratio > 1.5 (7 days)
• ✅ Win rate > 52%
• ✅ Max drawdown < 10%
• ✅ Simulated P&L > $10,000
• ✅ Zero model errors
• ✅ Latency P99 < 50μs
• ✅ 100% uptime

Monitoring:

• Daily Grafana dashboard review
• Daily PostgreSQL P&L check
• Daily disagreement rate review

Exit Criteria: All success criteria met for 7 consecutive days

---

Phase 2: 1% Capital (7 Days)

Objective: First real capital deployment (conservative allocation)

Configuration:

• Capital: 1% of total ($50K if total = $5M)
• Symbols: ES.FUT, NQ.FUT
• Max position: $10K per symbol
• Max daily loss: $5K (circuit breaker)

Deployment:

# Advance to Phase 2 (if TLI integrated)
tli rollout advance --phase small-position --capital-pct 1
# Confirm: "Allocate 1% capital ($50,000) to ensemble? [y/N]" -> y

Success Criteria:

• ✅ Sharpe ratio > 1.5 (real trading)
• ✅ Real P&L > $5,000 (after costs)
• ✅ Slippage < 5 bps
• ✅ Zero order rejections

Rollback Trigger: Daily loss > $5K OR 3 consecutive losing days

---

Phase 3: 10% Capital (14 Days)

Objective: Scale to meaningful allocation + run A/B test

Configuration:

• Capital: 10% of total ($500K if total = $5M)
• Symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT
• Max position: $100K per symbol
• Max daily loss: $50K (circuit breaker)

A/B Test (Optional):

# Compare ensemble vs DQN-only baseline (50/50 split)
tli ab start --control DQN_epoch30 --treatment Ensemble --split 50/50 --duration 14d

Success Criteria:

• ✅ Sharpe ratio > 1.8 (2 weeks)
• ✅ Real P&L > $50,000
• ✅ Max drawdown < 15%
• ✅ A/B test: Ensemble outperforms baseline (p < 0.05)

Rollback Trigger: Sharpe drops below 1.2 OR drawdown > 15%

---

Phase 4: 50% Capital (21 Days)

Objective: Major capital allocation + test hot-swapping

Configuration:

• Capital: 50% of total ($2.5M if total = $5M)
• Symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT, CL.FUT, GC.FUT
• Max position: $500K per symbol
• Max daily loss: $100K (circuit breaker)

Hot-Swap Test (Week 2):

# Test zero-downtime checkpoint update
tli checkpoint swap --model DQN --checkpoint dqn_epoch_40.safetensors
# System will auto-rollback if latency/accuracy degrades during 5-minute canary

Success Criteria:

• ✅ Sharpe ratio > 1.8 (3 weeks)
• ✅ Real P&L > $250,000
• ✅ Hot-swap success rate > 95%
• ✅ Model weights stabilize

Rollback Trigger: Sharpe drops below 1.3 OR hot-swap failure > 10%

---

Phase 5: 100% Capital (Ongoing)

Objective: Full production deployment

Configuration:

• Capital: 100% of total ($5M example)
• Symbols: All futures (ES, NQ, ZN, 6E, CL, GC, YM, RTY)
• Max position: $1M per symbol
• Max daily loss: $250K (circuit breaker)

Maintenance:

• Weekly checkpoint updates (hot-swapping)
• Monthly retraining with latest data
• Quarterly A/B tests for new model variants

Success Criteria (Ongoing):

• ✅ Sharpe ratio > 1.8 (rolling 90 days)
• ✅ Annualized return > 40%
• ✅ Max drawdown < 20% (rolling 1 year)
• ✅ Uptime > 99.9%

---

Rollback Procedures

Scenario 1: Daily Loss Exceeds Threshold

Trigger: Daily P&L < -$2,000 (Phase 1) or exceeds phase-specific limit

Action:

# Emergency halt
tli trading emergency-stop --reason "Daily loss threshold exceeded"

# Check recent predictions
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c \
  "SELECT symbol, ensemble_action, pnl FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '4 hours' ORDER BY pnl ASC LIMIT 20"

# Investigate: Are all models losing? Or just one?
# If all models: Market event (stop trading, manual review)
# If one model: Disable that model, continue with 2 models

---

Scenario 2: High Disagreement (>70% for 1 hour)

Trigger: ensemble_disagreement_rate > 0.70 for 60+ minutes

Action:

# Reduce position sizes by 50% (automatic via config)
# System continues trading but with reduced exposure

# Check disagreement events
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c \
  "SELECT * FROM get_high_disagreement_events_24h() LIMIT 50"

# Investigate: Market regime shift or model issue?
# - Check news/events (Fed announcement, earnings, etc.)
# - Compare model signals: Are they correlated or random?

No rollback needed (graceful degradation)

---

Scenario 3: Single Model Failure

Trigger: One model has >3 consecutive prediction errors

Action:

# System automatically disables failed model (circuit breaker)
# Ensemble continues with 2 remaining models

# Investigate failed model
docker logs foxhunt-trading-service --tail 500 | grep -i "DQN\|PPO" | grep -i error

# Possible causes:
# - Checkpoint corruption (reload checkpoint)
# - GPU memory issue (restart service)
# - Feature extraction failure (check data pipeline)

# After fix, re-enable model
tli ensemble enable-model --model DQN

---

Scenario 4: Complete Ensemble Failure

Trigger: All 3 models fail OR latency P99 > 200μs

Action:

# CRITICAL: Immediate halt
tli trading emergency-stop --reason "Ensemble catastrophic failure"

# Close all positions (aggressive execution)
tli trading close-all --aggressive --max-slippage 10

# Disable ensemble coordinator
curl -X POST http://localhost:8081/api/v1/ensemble/disable

# Fallback to single-model baseline (DQN only)
curl -X POST http://localhost:8081/api/v1/config/revert-baseline

# Trigger PagerDuty incident
# (Automated via alert system)

# Full diagnostic
docker logs foxhunt-trading-service > ensemble_failure_logs.txt
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c \
  "SELECT * FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '2 hours' ORDER BY timestamp DESC" > recent_predictions.csv

Recovery Time Objective: < 5 minutes

---

Troubleshooting

Issue: No Predictions in Database

Symptom: SELECT COUNT(*) FROM ensemble_predictions returns 0 after 10+ minutes

Diagnosis:

# Check Trading Service logs
docker logs foxhunt-trading-service --tail 100 | grep -i ensemble

# Possible causes:
# 1. Ensemble not initialized
# 2. Market data not flowing
# 3. Feature extraction failure
# 4. Database connection issue

Fix:

# Restart Trading Service
docker-compose restart foxhunt-trading-service

# Wait 2 minutes, check again
sleep 120
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT COUNT(*) FROM ensemble_predictions"

---

Issue: Latency P99 > 100μs

Symptom: Grafana dashboard shows aggregation latency > 100μs

Diagnosis:

# Check GPU utilization
nvidia-smi

# Check CPU load
top -b -n 1 | head -20

# Check model checkpoint sizes
ls -lh ml/trained_models/production/{dqn,ppo}/*.safetensors

Fix:

• If GPU memory full: Restart service to clear cache
• If CPU overloaded: Reduce concurrent predictions (config adjustment)
• If checkpoint too large: Use smaller model variant

---

Issue: High Disagreement Rate (>70%)

Symptom: Models constantly disagreeing on direction

Diagnosis:

-- Check model correlation
SELECT * FROM calculate_model_correlation_7d();

-- Expected: Correlation 0.5-0.8 (healthy diversity)
-- Concern: Correlation < 0.2 (models completely uncorrelated)
-- Concern: Correlation > 0.95 (models too similar)

Action:

• If correlation < 0.2: Possible regime shift (reduce positions, not a bug)
• If correlation > 0.95: Models too similar (need more diversity)

---

Issue: One Model Always Wrong

Symptom: DQN consistently loses money while PPO models profit

Diagnosis:

-- Check per-model P&L attribution
SELECT
    model_id,
    total_pnl / 100.0 AS pnl_dollars,
    accuracy,
    sharpe_ratio
FROM model_performance_attribution
WHERE window_hours = 24
  AND timestamp > NOW() - INTERVAL '7 days'
ORDER BY model_id, timestamp DESC
LIMIT 21;  -- 7 days * 3 models

Fix:

# If one model consistently underperforms:
# 1. Reduce its weight (adaptive weighting should do this automatically)
# 2. Retrain model with latest data
# 3. If still bad after retraining → disable model permanently

tli ensemble set-weight --model DQN --weight 0.1  # Reduce to 10%

---

FAQ

Q1: How long does paper trading take?

A: 7 days minimum. Success criteria must be met for 7 consecutive days before advancing to Phase 2.

---

Q2: What if paper trading fails on day 6?

A: Reset the 7-day clock. Fix the issue, restart paper trading from day 1. Do NOT skip to Phase 2 with incomplete validation.

---

Q3: Can I skip Phase 2 and go directly to 10%?

A: NO. The gradual rollout is designed to minimize capital risk. Each phase validates the ensemble at increasing scale. Skipping phases violates risk management policy.

---

Q4: How often should I retrain models?

A:

• Phase 1-3: No retraining (use current checkpoints)
• Phase 4-5: Monthly retraining with latest 90 days of data
• Ad-hoc: Retrain if Sharpe drops by >50% for 2+ weeks

---

Q5: What's the expected Sharpe ratio?

A:

• Phase 1 (paper trading): > 1.5 (minimum target)
• Phase 3-5 (real trading): > 1.8 (production target)
• Best case: 2.0-2.5 (ensemble at peak performance)

---

Q6: How do I update to new checkpoints?

A: Use hot-swapping mechanism (zero downtime):

# Stage new checkpoint in shadow buffer
tli checkpoint stage --model DQN --checkpoint dqn_epoch_45.safetensors

# Validate with 1,000 test predictions
tli checkpoint validate --model DQN --iterations 1000

# If validation passes, swap atomically (< 1μs)
tli checkpoint swap --model DQN

# System monitors for 5 minutes (canary period)
# Auto-rollback if latency/accuracy degrades

---

Q7: What if Grafana dashboard is down?

A: Use PostgreSQL queries as backup monitoring:

-- Daily summary (equivalent to Grafana dashboard)
SELECT
    symbol,
    COUNT(*) AS predictions,
    AVG(ensemble_confidence)::NUMERIC(5,3) AS avg_confidence,
    AVG(disagreement_rate)::NUMERIC(5,3) AS avg_disagreement,
    AVG(inference_latency_us) AS avg_latency_us,
    SUM(pnl) / 100.0 AS total_pnl_dollars
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '24 hours'
GROUP BY symbol;

---

Q8: How do I enable A/B testing?

A: Edit config/paper_trading_config.yaml:

ab_testing:
  enabled: true  # Change from false to true
  test_id: "ab_test_$(date +%Y%m%d)"  # Generate unique ID
  control_model: DQN_epoch30  # Baseline
  treatment_model: ensemble  # Ensemble
  traffic_split: 0.5  # 50/50 split
  min_sample_size: 1000

Then restart Trading Service: docker-compose restart foxhunt-trading-service

---

Q9: What's the emergency stop procedure?

A: See Scenario 4: Complete Ensemble Failure in Rollback Procedures section.

Summary:

1. tli trading emergency-stop
2. Close all positions
3. Disable ensemble
4. Revert to baseline
5. Trigger incident

---

Q10: Who do I contact for help?

A:

• PagerDuty: ml-team@foxhunt.trading (24/7 on-call)
• Slack: #trading-alerts (warnings, questions)
• Email: risk-management@foxhunt.trading (risk concerns)
• Escalation: CTO (critical failures only)

---

Appendix: Configuration File Reference

Config File Location

/home/jgrusewski/Work/foxhunt/config/paper_trading_config.yaml

Key Sections

Paper Trading Settings:

• mode: paper (DO NOT change to live until Phase 5)
• initial_capital: $100,000 (virtual)
• max_position_size: $10,000 per symbol
• max_daily_loss: $2,000 (circuit breaker)

Ensemble Models:

• 3 models: DQN epoch 30 (40%), PPO epoch 130 (40%), PPO epoch 420 (20%)
• voting_strategy: weighted (weighted average of signals)
• consensus_threshold: 0.6 (60% agreement required)

Risk Management:

• per_model_circuit_breaker: Enabled (disable model after 3 errors)
• cascade_failure_threshold: 2 (halt if 2+ models fail)
• confidence_based_sizing: Enabled (scale positions by confidence)

Monitoring:

• log_all_predictions: true (full audit trail)
• prometheus_metrics: Enabled (export to port 9092)
• grafana_dashboard: http://localhost:3000/d/ensemble-ml-prod

---

Appendix: Database Schema Reference

Table: ensemble_predictions

Purpose: Audit log of every ensemble prediction with per-model attribution

Key Columns:

• ensemble_action: BUY, SELL, HOLD
• ensemble_confidence: 0.0-1.0 (prediction confidence)
• disagreement_rate: 0.0-1.0 (model disagreement)
• dqn_signal, ppo_signal, mamba2_signal, tft_signal: Per-model votes
• pnl: Profit/loss in cents (populated after trade closes)

Useful Queries:

-- Daily P&L by symbol
SELECT symbol, SUM(pnl) / 100.0 AS pnl_dollars FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '24 hours' GROUP BY symbol;

-- Win rate
SELECT COUNT(CASE WHEN pnl > 0 THEN 1 END)::FLOAT / COUNT(*) AS win_rate FROM ensemble_predictions WHERE pnl IS NOT NULL;

---

Table: model_performance_attribution

Purpose: Rolling performance metrics per model for adaptive weighting

Key Columns:

• model_id: DQN, PPO, MAMBA2, TFT
• window_hours: 1, 24, 168 (rolling window)
• sharpe_ratio: Risk-adjusted returns
• avg_weight: Average weight in ensemble

Useful Queries:

-- Top performing models (last 24h)
SELECT * FROM get_top_models_24h();

-- Model correlation
SELECT * FROM calculate_model_correlation_7d();

---

Document Change Log

Version	Date	Author	Changes
1.0	2025-10-14	ML Team	Initial release

---

End of Guide

Last Updated: 2025-10-14
Next Review: After Phase 1 completion (7 days)
Approvals Required: ML Lead, Trading Desk, Risk Management, CTO