foxhunt/ENSEMBLE_RUNBOOK.md

Ensemble System Runbook

Operational Procedures & Troubleshooting

Document Version: 1.0 Date: 2025-10-14 System: Foxhunt HFT Trading System - 6-Model Ensemble Models: DQN (Epoch 10, 380), PPO (Epoch 380, 500), MAMBA-2, TFT On-Call: #trading-alerts Slack channel, PagerDuty rotation

---

Table of Contents

1. Deployment Procedures
2. Rollback Procedures
3. Health Check Commands
4. Troubleshooting Guide
5. On-Call Playbook
6. Common Issues & Solutions
7. Emergency Contacts

---

1. Deployment Procedures

1.1 Phase 0: Pre-Production Deployment

Duration: 1-2 days Environment: Staging Risk Level: Low (no real capital)

Step-by-Step Deployment

# 1. Navigate to project directory
cd /home/jgrusewski/Work/foxhunt

# 2. Pull latest code
git pull origin main

# 3. Run pre-deployment validation
./scripts/pre_deployment_validation.sh

# Expected: All checks PASS (19 passed, 0 failed, 0 warnings)

# 4. Build services with release optimizations
cargo build --workspace --release

# Expected: Build completes without errors (~5-10 minutes)

# 5. Run database migrations
export DATABASE_URL="postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt"
cargo sqlx migrate run

# Expected: Applied migrations: 022_ensemble_predictions.sql, 023_model_performance_attribution.sql

# 6. Start Docker infrastructure
docker-compose up -d postgres redis prometheus grafana

# 7. Verify database connectivity
psql $DATABASE_URL -c "SELECT version FROM _sqlx_migrations ORDER BY version DESC LIMIT 1;"

# Expected: version = 023 (or latest migration)

# 8. Start Trading Service (with ensemble enabled)
RUST_LOG=info cargo run -p trading_service --release &

# Expected: Log output shows:
# - "Ensemble coordinator initialized with 6 models"
# - "DQN Epoch 10 loaded: 75,628 bytes"
# - "DQN Epoch 380 loaded: 75,628 bytes"
# - "PPO Epoch 380 loaded: 42,000 bytes"
# - "PPO Epoch 500 loaded: 42,000 bytes"
# - "MAMBA-2 loaded: 150-500 MB"
# - "TFT loaded: 1.5-2.5 GB"
# - "Trading service started on port 50052"

# 9. Start API Gateway
RUST_LOG=info cargo run -p api_gateway --release &

# Expected: "API Gateway started on port 50051"

# 10. Start ML Training Service
RUST_LOG=info cargo run -p ml_training_service --release &

# Expected: "ML Training Service started on port 50054"

# 11. Verify all services healthy
grpc_health_probe -addr=localhost:50051 # API Gateway
grpc_health_probe -addr=localhost:50052 # Trading Service
grpc_health_probe -addr=localhost:50054 # ML Training Service

# Expected: All return "status: SERVING"

# 12. Run 10,000 test predictions (staging validation)
cargo run -p ml --example test_ensemble_staging --release

# Expected:
# - 10,000 predictions complete
# - Zero errors
# - P99 latency < 50μs for DQN/PPO
# - P99 latency < 150μs for TFT
# - No model disagreement > 70%
# - PostgreSQL audit logs: 10,000 rows in ensemble_predictions table

# 13. Verify Prometheus metrics
curl http://localhost:9092/metrics | grep ensemble_predictions_total

# Expected: ensemble_predictions_total{action="buy"} > 0
# Expected: ensemble_predictions_total{action="sell"} > 0
# Expected: ensemble_predictions_total{action="hold"} > 0

# 14. Verify Grafana dashboard
open http://localhost:3000/d/ensemble-production-monitoring

# Expected: Dashboard loads, all 7 panels display data

# 15. Mark Phase 0 complete
echo "Phase 0: Pre-Production deployment COMPLETE" | tee -a deployment.log

Validation Checklist:

• All services started successfully
• 10,000 predictions completed without errors
• Ensemble coordinator loaded all 6 models
• Prometheus metrics reporting correctly
• PostgreSQL audit logs persisting
• Grafana dashboard operational
• Zero model errors or rollbacks

Exit Criteria: All validation checks pass → Proceed to Phase 1

---

1.2 Phase 1: Paper Trading Deployment

Duration: 7 days Environment: Production (shadow mode) Risk Level: Low (no real capital, shadow trading only)

Step-by-Step Deployment

# 1. Enable paper trading mode
tli rollout start --phase paper-trading --duration 7d

# Expected output:
# "Starting Phase 1: Paper Trading"
# "Duration: 7 days"
# "Real capital: 0%"
# "Shadow mode: Enabled"
# "Ensemble will log predictions alongside baseline"

# 2. Verify shadow mode active
tli rollout status

# Expected output:
# Phase: Paper Trading (Day 0/7)
# Capital Allocation: 0% (shadow mode)
# Predictions Today: 0
# Simulated P&L: $0.00
# Status: Active

# 3. Monitor daily progress (run each day)
tli rollout status

# Expected output (example Day 5):
# Phase: Paper Trading (Day 5/7)
# Predictions: 15,432 (2,186/day avg)
# Simulated P&L: $23,450 (+18.3% vs baseline)
# Sharpe Ratio: 1.92 (Target: > 1.5) ✅
# Win Rate: 56.2% (Target: > 52%) ✅
# Disagreement Rate: 28.5% (Normal)
# Exit Criteria: 2/3 met (1 remaining)

# 4. Daily health checks
curl http://localhost:9092/metrics | grep ensemble_high_disagreement_total

# Expected: ensemble_high_disagreement_total < 100 per day

# 5. Check for circuit breaker triggers
psql $DATABASE_URL -c "SELECT COUNT(*) FROM system_alerts WHERE alert_type='circuit_breaker' AND timestamp > NOW() - INTERVAL '1 day';"

# Expected: 0 (no circuit breaker triggers)

# 6. After 7 days, review final results
tli rollout status

# Expected output:
# Phase: Paper Trading (Day 7/7)
# Predictions: 21,504
# Simulated P&L: $31,200 (+22.1% vs baseline)
# Sharpe Ratio: 2.01 (Target: > 1.5) ✅
# Win Rate: 57.3% (Target: > 52%) ✅
# Max Drawdown: 8.2% (Target: < 10%) ✅
# Exit Criteria: 3/3 met ✅
# READY FOR PHASE 2

# 7. Generate Phase 1 report
tli rollout report --phase paper-trading --format pdf > phase1_report.pdf

# 8. Obtain sign-off from Trading Desk
# (Manual step: Email phase1_report.pdf to trading-desk@foxhunt.trading)

# 9. Mark Phase 1 complete
echo "Phase 1: Paper Trading COMPLETE - $(date)" | tee -a deployment.log

Validation Checklist:

• 7 days of continuous operation
• Sharpe ratio > 1.5 (paper trading)
• Win rate > 52%
• Zero critical errors or rollbacks
• Disagreement rate stable (20-40%)
• Trading Desk sign-off received

Exit Criteria: All validation checks pass → Proceed to Phase 2

---

1.3 Phase 2: Small Position Deployment (1% Capital)

Duration: 7 days Environment: Production (real execution) Risk Level: Low ($50K capital, max $5K daily loss)

Step-by-Step Deployment

# 1. Confirm readiness for real capital
tli rollout advance --phase small-position --capital-pct 1

# Expected prompt:
# "Deploy ensemble to 1% capital ($50,000)?"
# "Max position size: $10,000 per symbol"
# "Max daily loss: $5,000"
# "Circuit breaker: 3 consecutive losses"
# "Confirm deployment? [y/N]"

# 2. Type 'y' to confirm deployment

# Expected output:
# "Phase 2: Small Position ACTIVATED"
# "Capital Allocation: $50,000 (1%)"
# "Risk Limits: ACTIVE"
# "Circuit Breaker: ARMED"
# "Real execution: ENABLED"

# 3. Monitor first hour closely (critical period)
watch -n 60 'tli rollout status'

# Expected output (example after 1 hour):
# Phase: Small Position (Hour 1, Day 1/7)
# Capital Allocation: $50,000 (1%)
# Trades Executed: 12
# Real P&L: +$487 (+0.97%)
# Circuit Breaker: OK (0 consecutive losses)
# Latency P99: 47μs ✅
# Disagreement Rate: 32.1%

# 4. Check for execution errors
psql $DATABASE_URL -c "SELECT COUNT(*) FROM orders WHERE status='REJECTED' AND created_at > NOW() - INTERVAL '1 hour';"

# Expected: 0 (no rejected orders)

# 5. Monitor slippage
psql $DATABASE_URL -c "SELECT AVG(ABS(executed_price - limit_price) / limit_price * 10000) AS avg_slippage_bps FROM orders WHERE status='FILLED' AND created_at > NOW() - INTERVAL '1 day';"

# Expected: avg_slippage_bps < 5.0 (less than 5 basis points)

# 6. Daily P&L check (run each day)
tli rollout status

# Expected output (example Day 5):
# Phase: Small Position (Day 5/7)
# Capital Allocation: $50,000 (1%)
# Trades Executed: 1,204
# Real P&L: +$6,823 (+13.6%)
# Sharpe Ratio: 1.68 (Target: > 1.5) ✅
# Win Rate: 55.2% (Target: > 52%) ✅
# Max Drawdown: 7.1% (Target: < 10%) ✅
# Exit Criteria: 3/3 met ✅

# 7. After 7 days, review final results
tli rollout status

# Expected output:
# Phase: Small Position (Day 7/7)
# Trades Executed: 2,156
# Real P&L: +$8,450 (+16.9%)
# Total P&L: $8,450 (Target: > $5,000) ✅
# Sharpe Ratio: 1.72 (Target: > 1.5) ✅
# Max Drawdown: 8.5% (Target: < 10%) ✅
# Exit Criteria: 3/3 met ✅
# READY FOR PHASE 3

# 8. Generate Phase 2 report
tli rollout report --phase small-position --format pdf > phase2_report.pdf

# 9. Obtain sign-off from Trading Desk & Risk Management
# (Manual step: Email phase2_report.pdf to stakeholders)

# 10. Mark Phase 2 complete
echo "Phase 2: Small Position COMPLETE - $(date)" | tee -a deployment.log

Validation Checklist:

• Real P&L positive after transaction costs
• Slippage < 5 bps
• Zero execution errors or order rejections
• Sharpe ratio > 1.5
• Total P&L > $5,000
• Max drawdown < 10%
• Trading Desk & Risk Management sign-off received

Exit Criteria: All validation checks pass → Proceed to Phase 3

---

1.4 Phase 3: Medium Position Deployment (10% Capital)

Duration: 14 days Environment: Production (scaled execution) Risk Level: Medium ($500K capital, max $50K daily loss)

Step-by-Step Deployment

# 1. Confirm readiness for 10% capital
tli rollout advance --phase medium-position --capital-pct 10

# Expected prompt:
# "Deploy ensemble to 10% capital ($500,000)?"
# "Max position size: $100,000 per symbol"
# "Max daily loss: $50,000"
# "Dynamic position sizing: ENABLED"
# "Confirm deployment? [y/N]"

# 2. Type 'y' to confirm deployment

# Expected output:
# "Phase 3: Medium Position ACTIVATED"
# "Capital Allocation: $500,000 (10%)"
# "Risk Limits: ACTIVE"
# "Dynamic Position Sizing: ENABLED"
# "Real execution: ENABLED"

# 3. Monitor first 24 hours closely
watch -n 300 'tli rollout status' # Update every 5 minutes

# Expected output (example after 24 hours):
# Phase: Medium Position (Day 1/14)
# Capital Allocation: $500,000 (10%)
# Trades Executed: 347
# Real P&L: +$8,230 (+1.65%)
# Sharpe Ratio: 1.84
# Circuit Breaker: OK
# Model Weights: Stable

# 4. Start A/B test (compare ensemble vs baseline)
tli ab start --control DQN --treatment Ensemble --split 50/50 --duration 14d

# Expected output:
# "A/B Test Started: test_id=<uuid>"
# "Control: DQN (single model)"
# "Treatment: Ensemble (6 models)"
# "Traffic Split: 50/50"
# "Duration: 14 days"
# "Min Sample Size: 1,000 predictions per group"

# 5. Daily A/B test status check
tli ab status --test-id <uuid>

# Expected output (example Day 7):
# Test ID: a1b2c3d4-...
# Status: Running (Day 7/14)
# Control Group (DQN):
#   - Predictions: 5,432
#   - Sharpe Ratio: 1.64
#   - Win Rate: 53.2%
#   - Total P&L: $12,450
# Treatment Group (Ensemble):
#   - Predictions: 5,389
#   - Sharpe Ratio: 1.92 (+17.1%)
#   - Win Rate: 56.8% (+6.8%)
#   - Total P&L: $15,980 (+28.3%)
# Statistical Significance: p=0.003 (SIGNIFICANT)
# Recommendation: Roll out ensemble to 100%

# 6. Monitor model weight drift
curl http://localhost:9092/metrics | grep ensemble_model_weight

# Expected: Model weights stable (no sudden >20% changes)

# 7. After 14 days, review final results
tli rollout status

# Expected output:
# Phase: Medium Position (Day 14/14)
# Capital Allocation: $500,000 (10%)
# Trades Executed: 4,823
# Real P&L: +$47,230 (+9.45%)
# Sharpe Ratio: 1.89 (Target: > 1.8) ✅
# Win Rate: 55.7%
# Max Drawdown: 12.3% (Target: < 15%)
# Exit Criteria: 3/3 met ✅

# 8. Review A/B test final results
tli ab results --test-id <uuid>

# Expected output:
# Test ID: a1b2c3d4-...
# Status: Complete (14 days)
# Sharpe Lift: +19.2% (p < 0.001) ✅ SIGNIFICANT
# Win Rate Lift: +6.3% (p = 0.007) ✅ SIGNIFICANT
# P&L Lift: +24.1% (p < 0.001) ✅ SIGNIFICANT
# Recommendation: ROLL OUT ENSEMBLE TO 100%

# 9. Check for checkpoint rollbacks
psql $DATABASE_URL -c "SELECT COUNT(*) FROM checkpoint_swaps WHERE status='rollback' AND timestamp > NOW() - INTERVAL '14 days';"

# Expected: 0 (no rollbacks during Phase 3)

# 10. Generate Phase 3 report with A/B test results
tli rollout report --phase medium-position --format pdf > phase3_report.pdf
tli ab results --test-id <uuid> --format pdf > ab_test_results.pdf

# 11. Obtain sign-off from Trading Desk, Risk Management, Engineering Lead
# (Manual step: Email reports to stakeholders for final approval)

# 12. Mark Phase 3 complete
echo "Phase 3: Medium Position COMPLETE - $(date)" | tee -a deployment.log

Validation Checklist:

• Consistent profitability across multiple symbols
• Ensemble outperforms baseline (A/B test significant, p < 0.05)
• Model weights stable (no wild swings)
• Sharpe ratio > 1.8 (2 weeks of real trading)
• Zero checkpoint rollbacks
• All stakeholder sign-offs received

Exit Criteria: All validation checks pass → Proceed to Phase 4

---

1.5 Phase 4: Full Deployment (100% Capital)

Duration: Ongoing Environment: Production (full allocation) Risk Level: High ($5M capital, max $250K daily loss)

Step-by-Step Deployment

# 1. Final confirmation for full deployment
tli rollout advance --phase full-deployment --capital-pct 100

# Expected prompt:
# "Deploy ensemble to 100% capital ($5,000,000)?"
# "Max position size: $1,000,000 per symbol"
# "Max daily loss: $250,000"
# "VaR-based position sizing: ENABLED"
# "This is FULL PRODUCTION DEPLOYMENT"
# "Confirm deployment? [y/N]"

# 2. Type 'y' to confirm full deployment

# Expected output:
# "Phase 4: Full Deployment ACTIVATED"
# "Capital Allocation: $5,000,000 (100%)"
# "Risk Limits: ACTIVE"
# "VaR-based Position Sizing: ENABLED"
# "Real execution: FULL PRODUCTION MODE"
# "Circuit Breaker: ARMED"

# 3. Monitor first week intensively
watch -n 300 'tli rollout status' # Update every 5 minutes

# 4. Daily health checks (automated cron job)
# Add to crontab:
0 9 * * * /home/jgrusewski/Work/foxhunt/scripts/daily_health_check.sh

# 5. Weekly checkpoint updates (automated)
# Add to crontab (every Sunday at 2 AM):
0 2 * * 0 /home/jgrusewski/Work/foxhunt/scripts/weekly_checkpoint_update.sh

# 6. Monthly A/B tests for new variants (manual trigger)
tli ab start --control Ensemble --treatment EnsembleV2 --split 90/10 --duration 30d

# 7. Quarterly retraining with latest market data (manual trigger)
tli train start --model Ensemble --data last_90_days --epochs 500

# 8. Mark Phase 4 active
echo "Phase 4: Full Deployment ACTIVE - $(date)" | tee -a deployment.log
echo "System is now in full production mode" | tee -a deployment.log

Ongoing Monitoring:

• Daily P&L attribution per model
• Weekly checkpoint health checks
• Monthly A/B tests for new variants
• Quarterly retraining with latest data
• Real-time alerting (PagerDuty + Slack)

---

2. Rollback Procedures

2.1 Emergency Rollback (Immediate)

Trigger Conditions:

• Circuit breaker triggered (3 consecutive losses)
• Daily drawdown > 5%
• Disagreement rate > 70% for 1 hour
• Latency P99 > 100μs for 5 minutes
• Any model error rate > 5%

Automatic Rollback (system handles automatically):

# Circuit breaker triggers automatically, no manual intervention needed
# Log entry will show:
# "Circuit breaker TRIGGERED: reason=<condition>"
# "Trading HALTED"
# "Initiating automatic rollback to baseline strategy"
# "Rollback complete in <duration> seconds"

Manual Rollback (if automatic rollback fails):

# 1. Immediately halt trading
tli rollout halt --reason "Manual intervention: <reason>"

# Expected output:
# "Trading HALTED"
# "All open orders CANCELLED"
# "Circuit breaker: ENGAGED"

# 2. Initiate rollback to previous phase
tli rollout rollback --reason "<detailed reason>"

# Expected output:
# "Initiating rollback to Phase <N-1>"
# "Capital allocation: <previous %>"
# "Risk limits: <previous settings>"
# "Rollback complete in <duration> seconds"

# 3. Verify rollback successful
tli rollout status

# Expected output:
# Phase: <Previous Phase> (ROLLBACK MODE)
# Trading: HALTED
# Circuit Breaker: ENGAGED
# Last Rollback: <timestamp> (<reason>)

# 4. Hot-swap to last known-good checkpoints
tli ensemble swap --model all --checkpoint last-known-good

# Expected output:
# "DQN Epoch 10: Swapped to checkpoint from <date>"
# "DQN Epoch 380: Swapped to checkpoint from <date>"
# "PPO Epoch 380: Swapped to checkpoint from <date>"
# "PPO Epoch 500: Swapped to checkpoint from <date>"
# "MAMBA-2: Swapped to checkpoint from <date>"
# "TFT: Swapped to checkpoint from <date>"
# "Hot-swap complete in <duration>ms"

# 5. Run 5-minute canary validation
tli ensemble validate --duration 5m

# Expected output:
# "Running canary validation (5 minutes)..."
# "Predictions: 1,000"
# "Latency P99: 48μs ✅"
# "Error Rate: 0.0% ✅"
# "Accuracy: 56.2% ✅"
# "Canary validation: PASSED"

# 6. Resume trading (if canary passed)
tli rollout resume

# Expected output:
# "Circuit breaker: DISENGAGED"
# "Trading: RESUMED"
# "Risk limits: ACTIVE"

# 7. Emit PagerDuty alert
# (Automatic: System sends alert to on-call engineer)

# 8. Post-mortem incident report (within 24 hours)
# Create incident report: INCIDENT_<timestamp>.md

Recovery Time Objective (RTO): < 5 minutes

---

2.2 Planned Rollback (Controlled)

Use Case: Exit criteria not met, gradual rollback to previous phase

# 1. Review current phase status
tli rollout status

# Example output showing failed exit criteria:
# Phase: Medium Position (Day 14/14)
# Sharpe Ratio: 1.62 (Target: > 1.8) ❌
# Win Rate: 51.3% (Target: > 52%) ❌
# Exit Criteria: 1/3 met ❌
# RECOMMENDATION: Rollback to Phase 2

# 2. Initiate controlled rollback
tli rollout rollback --controlled --reason "Exit criteria not met"

# Expected output:
# "Initiating controlled rollback to Phase 2"
# "Capital will be gradually reduced over 24 hours"
# "Current: $500,000 (10%) → Target: $50,000 (1%)"
# "Risk limits will be adjusted automatically"

# 3. Monitor rollback progress
watch -n 3600 'tli rollout status' # Check every hour

# Example output after 12 hours:
# Phase: Medium Position → Small Position (50% complete)
# Capital Allocation: $275,000 (5.5%)
# Risk Limits: Adjusting
# ETA to Phase 2: 12 hours

# 4. Verify rollback complete
tli rollout status

# Expected output:
# Phase: Small Position (ROLLBACK COMPLETE)
# Capital Allocation: $50,000 (1%)
# Risk Limits: Phase 2 settings ACTIVE
# Trading: ACTIVE

# 5. Document rollback reason
echo "Planned rollback from Phase 3 to Phase 2 - $(date)" | tee -a deployment.log
echo "Reason: Exit criteria not met (Sharpe < 1.8, Win Rate < 52%)" | tee -a deployment.log

# 6. Schedule review meeting with stakeholders
# (Manual step: Schedule meeting to discuss next steps)

Recovery Time Objective (RTO): 24 hours (gradual capital reduction)

---

3. Health Check Commands

3.1 System-Wide Health Checks

# Quick health check (all services)
/home/jgrusewski/Work/foxhunt/scripts/quick_health_check.sh

# Expected output:
# ✅ API Gateway: HEALTHY (port 50051)
# ✅ Trading Service: HEALTHY (port 50052)
# ✅ ML Training Service: HEALTHY (port 50054)
# ✅ PostgreSQL: HEALTHY
# ✅ Redis: HEALTHY
# ✅ Prometheus: HEALTHY (4/4 targets up)
# ✅ Grafana: HEALTHY
# ✅ GPU: HEALTHY (3.8GB VRAM free)
#
# Overall System Status: HEALTHY

3.2 Ensemble-Specific Health Checks

# Check ensemble coordinator status
tli ensemble status

# Expected output:
# Ensemble Coordinator: HEALTHY
# Models Loaded: 6/6
#   - DQN Epoch 10: ✅ (75,628 bytes)
#   - DQN Epoch 380: ✅ (75,628 bytes)
#   - PPO Epoch 380: ✅ (42,000 bytes)
#   - PPO Epoch 500: ✅ (42,000 bytes)
#   - MAMBA-2: ✅ (350 MB)
#   - TFT: ✅ (2.1 GB)
# Inference Latency (P99):
#   - DQN: 47μs ✅
#   - PPO: 49μs ✅
#   - MAMBA-2: 94μs ✅
#   - TFT: 138μs ✅
# Error Rate (24h): 0.02% ✅
# Last Checkpoint Swap: 3 days ago ✅

# Check model disagreement rate
curl -s http://localhost:9092/metrics | grep ensemble_disagreement_rate | tail -1

# Expected output:
# ensemble_disagreement_rate{symbol="ES.FUT"} 0.28

# Check for high disagreement events
curl -s http://localhost:9092/metrics | grep ensemble_high_disagreement_total

# Expected output:
# ensemble_high_disagreement_total{symbol="ES.FUT",threshold="0.7"} 0

# Check checkpoint swap health
curl -s http://localhost:9092/metrics | grep checkpoint_swaps_total

# Expected output:
# checkpoint_swaps_total{model_id="DQN_Epoch_10",status="success"} 12
# checkpoint_swaps_total{model_id="DQN_Epoch_10",status="rollback"} 0

3.3 Database Health Checks

# Check PostgreSQL connection
psql $DATABASE_URL -c 'SELECT 1;'

# Expected: 1 row returned

# Check ensemble predictions table size
psql $DATABASE_URL -c "SELECT pg_size_pretty(pg_total_relation_size('ensemble_predictions'));"

# Expected: < 50 GB (after 1 year of operation)

# Check recent predictions count
psql $DATABASE_URL -c "SELECT COUNT(*) FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '1 day';"

# Expected: 2,000-5,000 predictions per day

# Check for database errors
psql $DATABASE_URL -c "SELECT COUNT(*) FROM system_logs WHERE log_level='ERROR' AND timestamp > NOW() - INTERVAL '1 hour';"

# Expected: 0 (no errors in past hour)

3.4 Performance Health Checks

# Check Trading Service latency
curl -s http://localhost:9092/metrics | grep trading_service_request_duration_seconds

# Expected: P99 < 100ms

# Check GPU utilization
nvidia-smi --query-gpu=utilization.gpu,utilization.memory,memory.used,memory.free --format=csv,noheader

# Expected output:
# 35 %, 40 %, 1600 MiB, 2496 MiB

# Check Redis memory usage
redis-cli INFO memory | grep used_memory_human

# Expected: used_memory_human < 1GB

---

4. Troubleshooting Guide

4.1 Model Loading Failures

Symptom: Trading Service fails to start with error "Failed to load model checkpoint"

Diagnosis:

# 1. Check checkpoint file exists
ls -lh ml/trained_models/production/dqn_real_data/dqn_epoch_10.safetensors

# 2. Verify file size
du -h ml/trained_models/production/dqn_real_data/dqn_epoch_10.safetensors

# Expected: 75,628 bytes (74KB)

# 3. Check file permissions
stat ml/trained_models/production/dqn_real_data/dqn_epoch_10.safetensors

# Expected: -rw-rw-r-- (644)

# 4. Verify SafeTensors format
hexdump -C ml/trained_models/production/dqn_real_data/dqn_epoch_10.safetensors | head -5

# Expected: JSON header starting with {"layer_...

Solution:

# If checkpoint corrupted, restore from backup
cp ml/trained_models/backups/dqn_epoch_10.safetensors ml/trained_models/production/dqn_real_data/

# If backup unavailable, re-download from MinIO
aws s3 cp s3://foxhunt/checkpoints/dqn/dqn_epoch_10.safetensors ml/trained_models/production/dqn_real_data/

# Restart Trading Service
pkill -9 trading_service
cargo run -p trading_service --release &

---

4.2 High Disagreement Rate

Symptom: ensemble_disagreement_rate > 70% for extended period

Diagnosis:

# 1. Check disagreement rate trend
curl -s http://localhost:9092/metrics | grep ensemble_disagreement_rate

# 2. Check which models are disagreeing
psql $DATABASE_URL -c "SELECT model_id, AVG(ABS(signal - ensemble_action)) AS avg_disagreement FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '1 hour' GROUP BY model_id ORDER BY avg_disagreement DESC;"

# Expected output (example):
# model_id       | avg_disagreement
# ---------------+-----------------
# DQN_Epoch_10   | 0.82  (HIGH)
# MAMBA-2        | 0.75  (HIGH)
# PPO_Epoch_380  | 0.34  (NORMAL)
# PPO_Epoch_500  | 0.31  (NORMAL)
# DQN_Epoch_380  | 0.28  (NORMAL)
# TFT            | 0.25  (NORMAL)

# 3. Check market volatility (possible regime shift)
psql $DATABASE_URL -c "SELECT symbol, STDDEV(close) AS volatility FROM market_data WHERE timestamp > NOW() - INTERVAL '1 hour' GROUP BY symbol;"

# If volatility > 2x normal: Market regime shift (high disagreement expected)

Solution:

# If market regime shift (high volatility):
# 1. Reduce position sizes automatically (circuit breaker handles this)
# 2. Monitor for 1 hour
# 3. If disagreement persists > 1 hour, halt trading
tli rollout halt --reason "Persistent high disagreement rate"

# If specific model(s) causing disagreement:
# 1. Identify problematic model (e.g., DQN Epoch 10)
# 2. Hot-swap to alternative checkpoint
tli ensemble swap --model DQN_Epoch_10 --checkpoint dqn_epoch_30

# 3. Run canary validation
tli ensemble validate --duration 5m

# 4. If validation passes, resume trading
tli rollout resume

---

4.3 Latency Spikes

Symptom: Inference latency P99 > 100μs (2x normal)

Diagnosis:

# 1. Check current latency
curl -s http://localhost:9092/metrics | grep ensemble_aggregation_latency_microseconds

# 2. Check GPU utilization
nvidia-smi

# If GPU utilization > 90%: GPU overloaded

# 3. Check system load
uptime

# If load average > 8.0 (4 cores): CPU overloaded

# 4. Check for memory pressure
free -h

# If available memory < 1GB: Memory pressure

Solution:

# If GPU overloaded:
# 1. Reduce inference batch size (if batching enabled)
# 2. Check for competing processes
ps aux | grep -E "(python|jupyter|pytorch)" | grep -v grep

# Kill non-critical GPU processes
kill -9 <PID>

# If CPU overloaded:
# 1. Check for runaway processes
top -o %CPU

# Kill non-critical processes

# If memory pressure:
# 1. Clear caches
sync; echo 3 > /proc/sys/vm/drop_caches

# 2. Restart services to free memory
./scripts/restart_services.sh

# 3. If latency still high, halt trading temporarily
tli rollout halt --reason "Latency spike investigation"

# 4. Investigate root cause, fix, validate, resume

---

4.4 Circuit Breaker Triggered

Symptom: Trading halted automatically due to circuit breaker

Diagnosis:

# 1. Check why circuit breaker triggered
psql $DATABASE_URL -c "SELECT * FROM system_alerts WHERE alert_type='circuit_breaker' ORDER BY timestamp DESC LIMIT 1;"

# Example output:
# alert_type      | timestamp           | reason
# ----------------+---------------------+-------
# circuit_breaker | 2025-10-14 10:23:15 | 3 consecutive losses

# 2. Check recent trades
psql $DATABASE_URL -c "SELECT order_id, symbol, action, pnl FROM orders WHERE timestamp > NOW() - INTERVAL '1 hour' ORDER BY timestamp DESC LIMIT 10;"

# Example output:
# order_id | symbol  | action | pnl
# ---------+---------+--------+------
# uuid1    | ES.FUT  | SELL   | -$450
# uuid2    | NQ.FUT  | BUY    | -$320
# uuid3    | ES.FUT  | BUY    | -$280
# uuid4    | ES.FUT  | SELL   | +$210
# uuid5    | NQ.FUT  | SELL   | +$180

# 3. Check ensemble confidence during losses
psql $DATABASE_URL -c "SELECT ensemble_confidence, disagreement_rate FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '1 hour' ORDER BY timestamp DESC LIMIT 10;"

Solution:

# 1. Identify root cause:
#    - Low confidence trades?
#    - High disagreement?
#    - Market volatility spike?
#    - Model degradation?

# 2. If low confidence trades:
# Increase confidence threshold temporarily
psql $DATABASE_URL -c "UPDATE config SET value='0.70' WHERE key='min_ensemble_confidence';"

# 3. If market volatility spike:
# Wait for market to stabilize (monitor for 15 minutes)
watch -n 60 'psql $DATABASE_URL -c "SELECT STDDEV(close) FROM market_data WHERE timestamp > NOW() - INTERVAL \"5 minutes\" GROUP BY symbol;"'

# 4. If model degradation:
# Hot-swap to last known-good checkpoints
tli ensemble swap --model all --checkpoint last-known-good

# 5. Run canary validation
tli ensemble validate --duration 5m

# 6. If validation passes, resume trading
tli rollout resume

# 7. Document incident
echo "Circuit breaker triggered: <reason> - $(date)" | tee -a incidents.log

---

4.5 Database Connection Failures

Symptom: Trading Service logs show "Database connection lost"

Diagnosis:

# 1. Check PostgreSQL is running
docker-compose ps postgres

# Expected: State = Up

# 2. Check PostgreSQL logs
docker-compose logs postgres | tail -50

# Look for errors like:
# - "too many connections"
# - "out of memory"
# - "disk full"

# 3. Check active connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"

# Expected: < 100 connections

# 4. Check disk space
df -h

# Expected: / filesystem < 80% used

Solution:

# If too many connections:
# 1. Kill idle connections
psql $DATABASE_URL -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state='idle' AND query_start < NOW() - INTERVAL '5 minutes';"

# 2. Restart services to reset connection pools
./scripts/restart_services.sh

# If disk full:
# 1. Clear old log files
find /var/log -name "*.log" -mtime +7 -delete

# 2. Archive old PostgreSQL WAL files
docker-compose exec postgres pg_archivecleanup /var/lib/postgresql/data/pg_wal 000000010000000000000010

# 3. Monitor disk usage
watch -n 60 'df -h'

# If PostgreSQL crashed:
# 1. Restart PostgreSQL
docker-compose restart postgres

# 2. Wait for recovery (may take 1-5 minutes)
docker-compose logs -f postgres

# 3. Verify database accessible
psql $DATABASE_URL -c 'SELECT 1;'

# 4. Restart Trading Service
./scripts/restart_services.sh

---

5. On-Call Playbook

5.1 On-Call Engineer Responsibilities

Primary Responsibilities:

• Respond to PagerDuty alerts within 15 minutes
• Investigate and resolve critical issues within 1 hour
• Escalate to senior engineers if issue unresolved in 2 hours
• Document all incidents in incidents.log
• Conduct post-mortem within 24 hours of critical incidents

Escalation Path:

1. L1 (On-Call Engineer): Initial response, basic troubleshooting
2. L2 (Senior ML Engineer): Model-specific issues, complex debugging
3. L3 (Engineering Lead): Architectural issues, rollback decisions
4. L4 (Trading Desk Manager): Capital allocation, risk limits

---

5.2 Alert Severity Levels

Critical (PagerDuty):

• Circuit breaker triggered
• Checkpoint rollback
• Trading halted unexpectedly
• Database connection lost
• All models erroring simultaneously

Response: Immediate (within 15 minutes)

High (Slack + PagerDuty):

• High disagreement rate (> 60%)
• Latency spike (P99 > 75μs)
• Single model erroring (> 5% error rate)
• Daily loss approaching limit (> 80% of max)

Response: Within 30 minutes

Medium (Slack):

• Model weight drift (> 15% change in 1 hour)
• Slippage above target (> 3 bps)
• Checkpoint swap failed (but fallback succeeded)

Response: Within 2 hours

Low (Email):

• Daily summary report
• Weekly performance review
• Monthly A/B test results

Response: Next business day

---

5.3 Common Alert Scenarios

Scenario 1: Circuit Breaker Triggered (3 Consecutive Losses)

Alert:

🚨 CRITICAL: Circuit Breaker Triggered
Reason: 3 consecutive losses
Total Loss: $1,450
Trading: HALTED
Action: Investigate immediately

Playbook:

1. Acknowledge alert (PagerDuty): < 5 minutes
2. Check recent trades: See Section 4.4
3. Identify root cause: Low confidence? High disagreement? Volatility?
4. Apply fix: Hot-swap checkpoints, adjust confidence threshold, or wait for market stabilization
5. Validate fix: 5-minute canary period
6. Resume trading or escalate to L2 if unresolved
7. Document incident: incidents.log
8. Post-mortem: Within 24 hours

Estimated Time to Resolution: 15-30 minutes

---

Scenario 2: High Disagreement Rate (> 70%)

Alert:

⚠️ WARNING: High Ensemble Disagreement
Disagreement Rate: 73.2%
Symbol: ES.FUT
Duration: 15 minutes
Action: Monitor closely

Playbook:

1. Acknowledge alert (Slack): < 10 minutes
2. Check market volatility: See Section 4.2
3. If market regime shift (volatility > 2x normal):
   • Monitor for 1 hour
   • Position sizes automatically reduced by circuit breaker
   • If disagreement persists > 1 hour, halt trading
4. If specific model causing disagreement:
   • Identify problematic model
   • Hot-swap to alternative checkpoint
   • Run canary validation
5. Document incident: incidents.log

Estimated Time to Resolution: 30-60 minutes

---

Scenario 3: Database Connection Lost

Alert:

🚨 CRITICAL: Database Connection Lost
Service: Trading Service
Duration: 30 seconds
Action: Investigate immediately

Playbook:

1. Acknowledge alert (PagerDuty): < 5 minutes
2. Check PostgreSQL status: See Section 4.5
3. If PostgreSQL down:
   • Restart PostgreSQL: docker-compose restart postgres
   • Wait for recovery (1-5 minutes)
   • Restart Trading Service
4. If too many connections:
   • Kill idle connections
   • Restart services to reset connection pools
5. Verify trading operational: tli rollout status
6. Document incident: incidents.log

Estimated Time to Resolution: 5-15 minutes

---

5.4 Emergency Contact List

Engineering:

• On-Call Engineer: PagerDuty rotation
• Senior ML Engineer: ml-team@foxhunt.trading
• Engineering Lead: eng-lead@foxhunt.trading

Trading Desk:

• Trading Desk Manager: trading-desk@foxhunt.trading
• Head of Trading: head-of-trading@foxhunt.trading

Risk Management:

• Risk Manager: risk@foxhunt.trading
• Chief Risk Officer: cro@foxhunt.trading

Executive:

• CTO: cto@foxhunt.trading
• CEO: ceo@foxhunt.trading

Slack Channels:

• #trading-alerts: Real-time alerts (monitored 24/7)
• #ensemble-deployment: Deployment updates
• #incidents: Post-mortem discussions

---

6. Common Issues & Solutions

Issue 1: Model Fails to Load on Startup

Error: Failed to load model checkpoint: /path/to/checkpoint.safetensors

Solution: See Section 4.1 (Model Loading Failures)

---

Issue 2: Inference Latency Exceeds Target

Error: Ensemble aggregation latency P99: 127μs (target: < 50μs)

Solution: See Section 4.3 (Latency Spikes)

---

Issue 3: Checkpoint Swap Fails

Error: Checkpoint swap failed: Validation timeout

Solution:

# 1. Check if new checkpoint is valid
hexdump -C /path/to/new_checkpoint.safetensors | head -5

# 2. If invalid, abort swap (system keeps active buffer)
# 3. If valid but validation failed, check for:
#    - GPU memory exhaustion
#    - Network timeout (if checkpoint on MinIO)

# 4. Retry swap after resolving issue
tli ensemble swap --model <model_id> --checkpoint <path> --retry

---

Issue 4: A/B Test Contamination

Error: A/B test results invalid: Group assignment changed mid-test

Solution:

# 1. Stop contaminated test
tli ab stop --test-id <uuid>

# 2. Verify deterministic hashing enabled
psql $DATABASE_URL -c "SELECT * FROM ab_test_config WHERE test_id='<uuid>';"

# Expected: hash_seed is consistent throughout test

# 3. Start new test with fresh group assignments
tli ab start --control <model> --treatment <model> --split 50/50 --duration 14d

---

Issue 5: Daily Drawdown Approaching Limit

Warning: Daily drawdown: 4.2% (limit: 5.0%)

Solution:

# 1. Reduce position sizes immediately
psql $DATABASE_URL -c "UPDATE config SET value='0.50' WHERE key='position_size_multiplier';"

# 2. Increase confidence threshold
psql $DATABASE_URL -c "UPDATE config SET value='0.75' WHERE key='min_ensemble_confidence';"

# 3. Monitor closely for next 30 minutes
watch -n 60 'tli rollout status'

# 4. If drawdown exceeds 5%, circuit breaker will halt automatically

# 5. If approaching limit again, halt trading manually
tli rollout halt --reason "Approaching daily drawdown limit"

---

7. Emergency Contacts

7.1 24/7 Support

PagerDuty: Ensemble Trading System rotation

• URL: https://foxhunt.pagerduty.com/schedules/ensemble-trading
• Escalation Policy: L1 (15 min) → L2 (30 min) → L3 (1 hour)

Slack: #trading-alerts (monitored 24/7)

• Bot: @ensemble-bot (automated alerts)
• On-Call: @on-call (current on-call engineer)

7.2 Business Hours Support

Email: ml-team@foxhunt.trading (9 AM - 5 PM ET) Phone: (555) 123-4567 (emergency hotline)

7.3 External Vendors

Databento Support: support@databento.com (market data issues) AWS Support: Premium support (GPU/infrastructure issues) PostgreSQL DBA: dba@foxhunt.trading (database emergencies)

---

Appendix: Quick Reference Commands

# Health Checks
./scripts/quick_health_check.sh
tli ensemble status
tli rollout status

# Deployment
tli rollout start --phase <phase> --duration <days>
tli rollout advance --phase <phase> --capital-pct <pct>

# Rollback
tli rollout halt --reason "<reason>"
tli rollout rollback --reason "<reason>"
tli rollout resume

# Model Management
tli ensemble swap --model <model_id> --checkpoint <path>
tli ensemble validate --duration <minutes>

# A/B Testing
tli ab start --control <model> --treatment <model> --split 50/50 --duration <days>
tli ab status --test-id <uuid>
tli ab stop --test-id <uuid>

# Monitoring
curl http://localhost:9092/metrics | grep ensemble
psql $DATABASE_URL -c "SELECT * FROM ensemble_predictions ORDER BY timestamp DESC LIMIT 10;"

# Emergency
pkill -9 trading_service
./scripts/restart_services.sh
docker-compose restart postgres

---

Document Status: ✅ READY FOR PRODUCTION USE Last Updated: 2025-10-14 Next Review: Monthly (or after major incidents)