foxhunt/PAPER_TRADING_DEPLOYMENT_CHECKLIST.md

Paper Trading Deployment Checklist

Document Version: 1.0 Last Updated: 2025-10-14 Status: Ready for Execution Target System: Foxhunt HFT Ensemble ML System

---

Executive Summary

This checklist provides a step-by-step guide for deploying the 3-model ensemble (DQN + 2x PPO) to paper trading, followed by a 5-phase gradual rollout to 100% real capital. Each phase has clear success criteria and rollback procedures.

Total Timeline: 8-10 weeks (1 week paper trading + 7-9 weeks real capital ramp)

Ensemble Configuration:

• DQN epoch 30 (Sharpe 1.63, weight 0.4)
• PPO epoch 130 (Sharpe 1.59, weight 0.4)
• PPO epoch 420 (Sharpe 1.48, weight 0.2)

---

Pre-Deployment Prerequisites (Before Phase 1)

✅ Infrastructure Verification

• PostgreSQL Tables Created

  • ensemble_predictions table exists
  • model_performance_attribution table exists
  • ab_test_experiments table exists
  • Verify: psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "\dt ensemble*"

• Model Checkpoints Available

  • DQN epoch 30: ml/trained_models/production/dqn/dqn_epoch_30.safetensors
  • PPO epoch 130 actor: ml/trained_models/production/ppo/ppo_actor_epoch_130.safetensors
  • PPO epoch 130 critic: ml/trained_models/production/ppo/ppo_critic_epoch_130.safetensors
  • PPO epoch 420 actor: ml/trained_models/production/ppo/ppo_actor_epoch_420.safetensors
  • PPO epoch 420 critic: ml/trained_models/production/ppo/ppo_critic_epoch_420.safetensors
  • Verify: ls -lh ml/trained_models/production/{dqn,ppo}/*.safetensors

• Services Running & Healthy

  • API Gateway (port 50051): grpc_health_probe -addr=localhost:50051
  • Trading Service (port 50052): grpc_health_probe -addr=localhost:50052
  • ML Training Service (port 50054): grpc_health_probe -addr=localhost:50054
  • PostgreSQL: psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT 1"
  • Redis: redis-cli -h localhost -p 6379 PING
  • Prometheus: curl http://localhost:9090/api/v1/targets | jq
  • Grafana: curl http://localhost:3000/api/health

• Monitoring Dashboards Configured

  • Grafana dashboard imported: monitoring/grafana/ensemble_ml_production.json
  • Prometheus scraping Trading Service metrics (port 9092)
  • Verify metrics: curl http://localhost:9092/metrics | grep ensemble

• Configuration Files Loaded

  • config/paper_trading_config.yaml exists and validated
  • Trading Service can read config (no syntax errors)

✅ Smoke Test (Run Before Deployment)

# Run smoke test script
bash tests/paper_trading_smoke_test.sh

# Expected output:
# ✅ All 3 models load successfully
# ✅ Inference latency < 50μs P99
# ✅ Ensemble prediction generates valid output
# ✅ PostgreSQL audit log working
# ✅ Prometheus metrics exported

---

Phase 1: Paper Trading (Duration: 7 days)

Objectives

• Validate ensemble predictions with real market data
• Zero risk (simulated execution only)
• Compare ensemble P&L vs existing production strategy

Risk Limits

• Capital: $100K virtual (no real money)
• Max Position Size: $10K per symbol
• Max Daily Loss: $2K (circuit breaker)
• Symbols: ES.FUT, NQ.FUT only

Deployment Steps

1. Load Configuration

   # Copy paper trading config to production location
   cp config/paper_trading_config.yaml /opt/foxhunt/config/active/
   
   # Reload Trading Service configuration
   curl -X POST http://localhost:8081/api/v1/config/reload
   

2. Start Paper Trading Mode

   # Via TLI (if integrated)
   tli trading start --mode paper --capital 100000 --symbols ES.FUT,NQ.FUT
   
   # OR via gRPC (if TLI not integrated)
   grpcurl -plaintext -d '{
     "mode": "paper",
     "initial_capital": 100000,
     "symbols": ["ES.FUT", "NQ.FUT"]
   }' localhost:50052 TradingService/StartPaperTrading
   

3. Verify Paper Trading Active

   # Check Trading Service logs
   docker logs foxhunt-trading-service --tail 50 | grep "Paper trading"
   
   # Expected output:
   # INFO Paper trading mode enabled (capital: $100,000)
   # INFO Loaded 3 ensemble models (DQN, PPO_130, PPO_420)
   # INFO Ensemble coordinator initialized successfully
   

4. Monitor for 7 Days

   • Daily check Grafana dashboard: http://localhost:3000/d/ensemble-ml-prod
   • Daily check PostgreSQL predictions: SELECT COUNT(*), AVG(ensemble_confidence) FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '24 hours'
   • Daily check P&L: SELECT SUM(pnl) FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '24 hours'

Success Criteria (All Must Pass)

• Performance Metrics

  • Sharpe ratio > 1.5 (over 7 days)
  • Win rate > 52%
  • Max drawdown < 10%
  • Total simulated P&L > $10,000

• Technical Metrics

  • Inference latency P99 < 50μs
  • Zero model prediction errors
  • Zero circuit breaker triggers
  • Disagreement rate < 40% (average)

• Operational Metrics

  • 100% uptime (no crashes or restarts)
  • All predictions logged to PostgreSQL
  • Prometheus metrics exported without gaps

Rollback Procedure (If Failure)

# Stop paper trading immediately
tli trading stop --mode paper

# Disable ensemble coordinator
curl -X POST http://localhost:8081/api/v1/ensemble/disable

# Revert to single-model baseline (DQN only)
curl -X POST http://localhost:8081/api/v1/config/revert-baseline

# Investigate failure
docker logs foxhunt-trading-service --tail 500 > paper_trading_failure_logs.txt
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c \
  "SELECT * FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '1 hour' ORDER BY timestamp DESC LIMIT 100"

---

Phase 2: Small Position (1% Capital) - Duration: 7 days

Objectives

• First real capital deployment (conservative)
• Validate execution with real slippage/commission
• Confirm profitability after transaction costs

Risk Limits

• Capital: 1% of total ($50K if total capital = $5M)
• Max Position Size: $10K per symbol
• Max Daily Loss: $5K (circuit breaker)
• Symbols: ES.FUT, NQ.FUT

Deployment Steps

1. Advance to Phase 2

   tli rollout advance --phase small-position --capital-pct 1
   # Confirm: "Allocate 1% capital ($50,000) to ensemble? [y/N]" -> y
   

2. Monitor Real Execution

   • Track actual fills: SELECT * FROM orders WHERE ensemble_order = true
   • Track slippage: SELECT AVG(slippage_bps) FROM ensemble_predictions WHERE executed_price IS NOT NULL
   • Track commission: SELECT SUM(commission) FROM ensemble_predictions WHERE executed_price IS NOT NULL

Success Criteria (All Must Pass)

• Performance Metrics

  • Sharpe ratio > 1.5 (real trading, 7 days)
  • Real P&L > $5,000 (after costs)
  • Max drawdown < 10%
  • Slippage < 5 basis points (average)

• Technical Metrics

  • Zero order rejections
  • Zero execution errors
  • Latency P99 < 50μs

• Operational Metrics

  • 100% uptime
  • No rollbacks or emergency stops

Rollback Procedure (If Failure)

# Immediately halt trading
tli rollout rollback --reason "Phase 2 failure: [specific reason]"

# Close all open positions
tli trading close-all --reason "Rollback to baseline"

# Revert to baseline strategy
curl -X POST http://localhost:8081/api/v1/config/revert-baseline

# Incident report
tli trading report --phase phase2 --output phase2_incident_report.json

---

Phase 3: Medium Position (10% Capital) - Duration: 14 days

Objectives

• Scale to meaningful capital allocation
• Validate consistency across multiple symbols
• Run A/B test (ensemble vs single-model baseline)

Risk Limits

• Capital: 10% of total ($500K if total = $5M)
• Max Position Size: $100K per symbol
• Max Daily Loss: $50K (circuit breaker)
• Symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT

Deployment Steps

1. Advance to Phase 3

   tli rollout advance --phase medium-position --capital-pct 10
   

2. Start A/B Test (Optional)

   # Compare ensemble vs DQN-only baseline
   tli ab start --control DQN_epoch30 --treatment Ensemble --split 50/50 --duration 14d
   

3. Monitor Multi-Symbol Performance

   • Daily P&L per symbol
   • Model weight adjustments (adaptive weighting)
   • Disagreement patterns per symbol

Success Criteria (All Must Pass)

• Performance Metrics

  • Sharpe ratio > 1.8 (2 weeks, real trading)
  • Real P&L > $50,000 (after costs)
  • Max drawdown < 15%
  • Win rate > 54%

• A/B Test Results (If Running)

  • Ensemble outperforms baseline (p < 0.05)
  • Sharpe lift > 10% (statistical significance)
  • No degradation in worst-case scenarios

• Technical Metrics

  • Zero checkpoint rollbacks
  • Model weights stabilize (no wild swings)

Rollback Procedure (If Failure)

# Halt trading and rollback
tli rollout rollback --phase phase3 --reason "[specific reason]"

# Stop A/B test if running
tli ab stop --test-id [uuid]

# Close all positions
tli trading close-all

# Generate failure report
tli trading report --phase phase3 --output phase3_failure_report.json

---

Phase 4: Large Position (50% Capital) - Duration: 21 days

Objectives

• Major capital allocation (production scale)
• Validate performance under large position sizes
• Test hot-swapping mechanism (weekly checkpoint updates)

Risk Limits

• Capital: 50% of total ($2.5M if total = $5M)
• Max Position Size: $500K per symbol
• Max Daily Loss: $100K (circuit breaker)
• Symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT, CL.FUT, GC.FUT

Deployment Steps

1. Advance to Phase 4

   tli rollout advance --phase large-position --capital-pct 50
   

2. Test Hot-Swapping (Week 2)

   # Upload new checkpoint to MinIO (from ML training)
   mc cp ml/trained_models/dqn_epoch_40.safetensors foxhunt/checkpoints/dqn/
   
   # Trigger hot-swap
   tli checkpoint swap --model DQN --checkpoint dqn_epoch_40.safetensors
   
   # Monitor canary period (5 minutes)
   # System will auto-rollback if latency/accuracy degrades
   

3. Monitor Large Positions

   • Track market impact (price movement from trades)
   • Validate VaR-based position sizing
   • Monitor correlation with baseline strategy

Success Criteria (All Must Pass)

• Performance Metrics

  • Sharpe ratio > 1.8 (3 weeks, real trading)
  • Real P&L > $250,000 (after costs)
  • Max drawdown < 15%
  • Annualized return > 40%

• Technical Metrics

  • Hot-swap success rate > 95%
  • Zero unplanned downtime
  • Latency P99 < 50μs (even under large positions)

• Operational Metrics

  • Weekly checkpoint updates without issues
  • Model weights adapt to changing markets

Rollback Procedure (If Failure)

# Emergency halt
tli rollout rollback --phase phase4 --reason "[specific reason]"

# Close all large positions (may take time)
tli trading close-all --aggressive

# Reduce to Phase 3 allocation (10%)
tli rollout revert --phase medium-position

# Full incident analysis
tli trading report --phase phase4 --full-analysis --output phase4_incident.json

---

Phase 5: Full Deployment (100% Capital) - Ongoing

Objectives

• Full production deployment (all capital managed by ensemble)
• Continuous monitoring and adaptation
• Regular checkpoint updates (weekly)

Risk Limits

• Capital: 100% of total ($5M example)
• Max Position Size: $1M per symbol
• Max Daily Loss: $250K (circuit breaker)
• Symbols: All futures (ES, NQ, ZN, 6E, CL, GC, YM, RTY)

Deployment Steps

1. Advance to Phase 5

   tli rollout advance --phase full-deployment --capital-pct 100
   # Requires manual confirmation + risk management approval
   

2. Continuous Monitoring (Daily)

   • Grafana dashboard: http://localhost:3000/d/ensemble-ml-prod
   • Daily P&L report (automated email)
   • Weekly risk review meeting

3. Regular Maintenance

   • Weekly checkpoint updates (hot-swapping)
   • Monthly retraining with latest data
   • Quarterly A/B tests for new model variants

Success Criteria (Ongoing)

• Performance Metrics

  • Sharpe ratio > 1.8 (rolling 90 days)
  • Annualized return > 40%
  • Max drawdown < 20% (rolling 1 year)
  • Win rate > 54%

• Technical Metrics

  • Uptime > 99.9%
  • Latency P99 < 50μs
  • Zero data loss (all predictions logged)

• Operational Metrics

  • Hot-swap success rate > 95%
  • Alert noise < 10 false positives/week
  • Time to rollback < 5 minutes

Emergency Rollback Procedure

# CRITICAL FAILURE - Immediate action required

# 1. Halt all trading (< 1 second)
tli trading emergency-stop --reason "CRITICAL: [reason]"

# 2. Close all positions (aggressive execution)
tli trading close-all --aggressive --max-slippage 10

# 3. Disable ensemble coordinator
curl -X POST http://localhost:8081/api/v1/ensemble/disable

# 4. Fallback to single-model baseline
curl -X POST http://localhost:8081/api/v1/config/revert-baseline

# 5. Trigger PagerDuty incident
tli alert trigger --severity critical --message "Ensemble emergency stop"

# 6. Full system diagnostic
tli system diagnose --output emergency_diagnostic.json

# 7. Notify trading desk + risk management
# (Automated via PagerDuty/Slack)

---

Post-Deployment Monitoring Checklist (Daily)

Daily Tasks (5-10 minutes)

• Check Grafana dashboard for anomalies

  • Ensemble confidence trend
  • Disagreement rate (alert if >70%)
  • Latency P99 (alert if >100μs)
  • Per-model P&L attribution

• Review PostgreSQL audit logs

  -- Daily P&L summary
  SELECT
    symbol,
    COUNT(*) AS predictions,
    SUM(pnl) AS total_pnl,
    AVG(ensemble_confidence) AS avg_confidence,
    AVG(disagreement_rate) AS avg_disagreement
  FROM ensemble_predictions
  WHERE timestamp > NOW() - INTERVAL '24 hours'
  GROUP BY symbol;
  

• Check for circuit breaker triggers

  # Query Prometheus for circuit breaker alerts
  curl http://localhost:9090/api/v1/query?query='circuit_breaker_triggered_total' | jq
  

• Review model performance attribution

  -- Top performing models (last 24 hours)
  SELECT * FROM get_top_models_24h() ORDER BY sharpe_ratio DESC;
  

Weekly Tasks (30-60 minutes)

• Generate weekly performance report

  tli trading report --period week --output weekly_report_$(date +%Y%m%d).pdf
  

• Review model weight adjustments (adaptive weighting)

  SELECT
    model_id,
    AVG(avg_weight) AS avg_weight_7d,
    AVG(sharpe_ratio) AS avg_sharpe_7d
  FROM model_performance_attribution
  WHERE window_hours = 24 AND timestamp > NOW() - INTERVAL '7 days'
  GROUP BY model_id;
  

• Check for high disagreement events (regime shifts)

  SELECT * FROM get_high_disagreement_events_24h() LIMIT 20;
  

• Hot-swap checkpoint (if new checkpoint available)

  # Test new checkpoint in shadow mode
  tli checkpoint stage --model DQN --checkpoint dqn_epoch_45.safetensors
  tli checkpoint validate --model DQN --iterations 1000
  tli checkpoint swap --model DQN  # Atomic swap
  

Monthly Tasks (2-4 hours)

• Comprehensive performance review

  • 30-day Sharpe ratio trend
  • Drawdown analysis
  • Model correlation matrix
  • A/B test results (if running)

• Retrain models with latest data

  # Trigger model retraining (ML Training Service)
  tli train start --model DQN --epochs 50 --data-range 90d
  

• Review alert thresholds (adjust if needed)

  • Too many false positives? Increase thresholds
  • Missed critical events? Decrease thresholds

• Compliance audit

  • Verify all trades logged
  • Check feature snapshots for reproducibility
  • Validate audit trail completeness

---

Rollback Decision Matrix

Failure Type	Severity	Rollback Action	Recovery Time
Single model prediction error	Low	Disable model, continue with 2 models	1 hour
High disagreement (>70% for 1 hour)	Medium	Reduce position sizes by 50%	No rollback
Latency spike (P99 >100μs)	High	Halt trading, investigate	5 minutes
Circuit breaker triggered (2+ models)	Critical	Emergency stop, close positions	Immediate
Daily loss exceeds threshold	Critical	Halt trading, risk review	1 day
Checkpoint rollback failure	Critical	Emergency stop, revert to baseline	5 minutes
Database connection loss	Critical	Emergency stop, cache to Redis	Immediate
GPU failure (all models down)	Critical	Fallback to CPU, reduce positions	10 minutes

---

Success Metrics Summary

Phase	Duration	Capital	Sharpe	Win Rate	Max DD	P&L Target
Phase 1: Paper Trading	7 days	$100K (virtual)	>1.5	>52%	<10%	>$10K (sim)
Phase 2: 1% Capital	7 days	$50K	>1.5	>52%	<10%	>$5K
Phase 3: 10% Capital	14 days	$500K	>1.8	>54%	<15%	>$50K
Phase 4: 50% Capital	21 days	$2.5M	>1.8	>54%	<15%	>$250K
Phase 5: 100% Capital	Ongoing	$5M	>1.8	>54%	<20%	>$1M/year

---

Contacts & Escalation

On-Call Engineers

• Primary: ml-team@foxhunt.trading (PagerDuty)
• Secondary: risk-management@foxhunt.trading
• Escalation: CTO (for critical failures only)

Alert Channels

• PagerDuty: Critical alerts (circuit breaker, emergency stop)
• Slack: #trading-alerts (warnings, high disagreement)
• Email: Daily summary reports

Documentation

• This Checklist: Paper trading deployment steps
• Ensemble Strategy: ENSEMBLE_PRODUCTION_DEPLOYMENT_STRATEGY.md
• Runbooks: /docs/runbooks/ensemble_troubleshooting.md

---

Approval Signatures

Role	Name	Signature	Date
ML Engineering Lead	_____________	_____________	_______
Trading Desk Manager	_____________	_____________	_______
Risk Management	_____________	_____________	_______
CTO	_____________	_____________	_______

---

End of Checklist

Last Updated: 2025-10-14 Next Review: After Phase 1 completion (7 days)