# Ensemble System Runbook
# Operational Procedures & Troubleshooting

**Document Version**: 1.0
**Date**: 2025-10-14
**System**: Foxhunt HFT Trading System - 6-Model Ensemble
**Models**: DQN (Epoch 10, 380), PPO (Epoch 380, 500), MAMBA-2, TFT
**On-Call**: #trading-alerts Slack channel, PagerDuty rotation

---

## Table of Contents

1. [Deployment Procedures](#1-deployment-procedures)
2. [Rollback Procedures](#2-rollback-procedures)
3. [Health Check Commands](#3-health-check-commands)
4. [Troubleshooting Guide](#4-troubleshooting-guide)
5. [On-Call Playbook](#5-on-call-playbook)
6. [Common Issues & Solutions](#6-common-issues--solutions)
7. [Emergency Contacts](#7-emergency-contacts)

---

## 1. Deployment Procedures

### 1.1 Phase 0: Pre-Production Deployment

**Duration**: 1-2 days
**Environment**: Staging
**Risk Level**: Low (no real capital)

#### Step-by-Step Deployment

```bash
# 1. Navigate to project directory
cd /home/jgrusewski/Work/foxhunt

# 2. Pull latest code
git pull origin main

# 3. Run pre-deployment validation
./scripts/pre_deployment_validation.sh

# Expected: All checks PASS (19 passed, 0 failed, 0 warnings)

# 4. Build services with release optimizations
cargo build --workspace --release

# Expected: Build completes without errors (~5-10 minutes)

# 5. Run database migrations
export DATABASE_URL="postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt"
cargo sqlx migrate run

# Expected: Applied migrations: 022_ensemble_predictions.sql, 023_model_performance_attribution.sql

# 6. Start Docker infrastructure
docker-compose up -d postgres redis prometheus grafana

# 7. Verify database connectivity
psql $DATABASE_URL -c "SELECT version FROM _sqlx_migrations ORDER BY version DESC LIMIT 1;"

# Expected: version = 023 (or latest migration)

# 8. Start Trading Service (with ensemble enabled)
RUST_LOG=info cargo run -p trading_service --release &

# Expected: Log output shows:
# - "Ensemble coordinator initialized with 6 models"
# - "DQN Epoch 10 loaded: 75,628 bytes"
# - "DQN Epoch 380 loaded: 75,628 bytes"
# - "PPO Epoch 380 loaded: 42,000 bytes"
# - "PPO Epoch 500 loaded: 42,000 bytes"
# - "MAMBA-2 loaded: 150-500 MB"
# - "TFT loaded: 1.5-2.5 GB"
# - "Trading service started on port 50052"

# 9. Start API Gateway
RUST_LOG=info cargo run -p api_gateway --release &

# Expected: "API Gateway started on port 50051"

# 10. Start ML Training Service
RUST_LOG=info cargo run -p ml_training_service --release &

# Expected: "ML Training Service started on port 50054"

# 11. Verify all services healthy
grpc_health_probe -addr=localhost:50051 # API Gateway
grpc_health_probe -addr=localhost:50052 # Trading Service
grpc_health_probe -addr=localhost:50054 # ML Training Service

# Expected: All return "status: SERVING"

# 12. Run 10,000 test predictions (staging validation)
cargo run -p ml --example test_ensemble_staging --release

# Expected:
# - 10,000 predictions complete
# - Zero errors
# - P99 latency < 50μs for DQN/PPO
# - P99 latency < 150μs for TFT
# - No model disagreement > 70%
# - PostgreSQL audit logs: 10,000 rows in ensemble_predictions table

# 13. Verify Prometheus metrics
curl http://localhost:9092/metrics | grep ensemble_predictions_total

# Expected: ensemble_predictions_total{action="buy"} > 0
# Expected: ensemble_predictions_total{action="sell"} > 0
# Expected: ensemble_predictions_total{action="hold"} > 0

# 14. Verify Grafana dashboard
open http://localhost:3000/d/ensemble-production-monitoring

# Expected: Dashboard loads, all 7 panels display data

# 15. Mark Phase 0 complete
echo "Phase 0: Pre-Production deployment COMPLETE" | tee -a deployment.log
```

**Validation Checklist**:
- [ ] All services started successfully
- [ ] 10,000 predictions completed without errors
- [ ] Ensemble coordinator loaded all 6 models
- [ ] Prometheus metrics reporting correctly
- [ ] PostgreSQL audit logs persisting
- [ ] Grafana dashboard operational
- [ ] Zero model errors or rollbacks

**Exit Criteria**: All validation checks pass → Proceed to Phase 1

---

### 1.2 Phase 1: Paper Trading Deployment

**Duration**: 7 days
**Environment**: Production (shadow mode)
**Risk Level**: Low (no real capital, shadow trading only)

#### Step-by-Step Deployment

```bash
# 1. Enable paper trading mode
tli rollout start --phase paper-trading --duration 7d

# Expected output:
# "Starting Phase 1: Paper Trading"
# "Duration: 7 days"
# "Real capital: 0%"
# "Shadow mode: Enabled"
# "Ensemble will log predictions alongside baseline"

# 2. Verify shadow mode active
tli rollout status

# Expected output:
# Phase: Paper Trading (Day 0/7)
# Capital Allocation: 0% (shadow mode)
# Predictions Today: 0
# Simulated P&L: $0.00
# Status: Active

# 3. Monitor daily progress (run each day)
tli rollout status

# Expected output (example Day 5):
# Phase: Paper Trading (Day 5/7)
# Predictions: 15,432 (2,186/day avg)
# Simulated P&L: $23,450 (+18.3% vs baseline)
# Sharpe Ratio: 1.92 (Target: > 1.5) ✅
# Win Rate: 56.2% (Target: > 52%) ✅
# Disagreement Rate: 28.5% (Normal)
# Exit Criteria: 2/3 met (1 remaining)

# 4. Daily health checks
curl http://localhost:9092/metrics | grep ensemble_high_disagreement_total

# Expected: ensemble_high_disagreement_total < 100 per day

# 5. Check for circuit breaker triggers
psql $DATABASE_URL -c "SELECT COUNT(*) FROM system_alerts WHERE alert_type='circuit_breaker' AND timestamp > NOW() - INTERVAL '1 day';"

# Expected: 0 (no circuit breaker triggers)

# 6. After 7 days, review final results
tli rollout status

# Expected output:
# Phase: Paper Trading (Day 7/7)
# Predictions: 21,504
# Simulated P&L: $31,200 (+22.1% vs baseline)
# Sharpe Ratio: 2.01 (Target: > 1.5) ✅
# Win Rate: 57.3% (Target: > 52%) ✅
# Max Drawdown: 8.2% (Target: < 10%) ✅
# Exit Criteria: 3/3 met ✅
# READY FOR PHASE 2

# 7. Generate Phase 1 report
tli rollout report --phase paper-trading --format pdf > phase1_report.pdf

# 8. Obtain sign-off from Trading Desk
# (Manual step: Email phase1_report.pdf to trading-desk@foxhunt.trading)

# 9. Mark Phase 1 complete
echo "Phase 1: Paper Trading COMPLETE - $(date)" | tee -a deployment.log
```

**Validation Checklist**:
- [ ] 7 days of continuous operation
- [ ] Sharpe ratio > 1.5 (paper trading)
- [ ] Win rate > 52%
- [ ] Zero critical errors or rollbacks
- [ ] Disagreement rate stable (20-40%)
- [ ] Trading Desk sign-off received

**Exit Criteria**: All validation checks pass → Proceed to Phase 2

---

### 1.3 Phase 2: Small Position Deployment (1% Capital)

**Duration**: 7 days
**Environment**: Production (real execution)
**Risk Level**: Low ($50K capital, max $5K daily loss)

#### Step-by-Step Deployment

```bash
# 1. Confirm readiness for real capital
tli rollout advance --phase small-position --capital-pct 1

# Expected prompt:
# "Deploy ensemble to 1% capital ($50,000)?"
# "Max position size: $10,000 per symbol"
# "Max daily loss: $5,000"
# "Circuit breaker: 3 consecutive losses"
# "Confirm deployment? [y/N]"

# 2. Type 'y' to confirm deployment

# Expected output:
# "Phase 2: Small Position ACTIVATED"
# "Capital Allocation: $50,000 (1%)"
# "Risk Limits: ACTIVE"
# "Circuit Breaker: ARMED"
# "Real execution: ENABLED"

# 3. Monitor first hour closely (critical period)
watch -n 60 'tli rollout status'

# Expected output (example after 1 hour):
# Phase: Small Position (Hour 1, Day 1/7)
# Capital Allocation: $50,000 (1%)
# Trades Executed: 12
# Real P&L: +$487 (+0.97%)
# Circuit Breaker: OK (0 consecutive losses)
# Latency P99: 47μs ✅
# Disagreement Rate: 32.1%

# 4. Check for execution errors
psql $DATABASE_URL -c "SELECT COUNT(*) FROM orders WHERE status='REJECTED' AND created_at > NOW() - INTERVAL '1 hour';"

# Expected: 0 (no rejected orders)

# 5. Monitor slippage
psql $DATABASE_URL -c "SELECT AVG(ABS(executed_price - limit_price) / limit_price * 10000) AS avg_slippage_bps FROM orders WHERE status='FILLED' AND created_at > NOW() - INTERVAL '1 day';"

# Expected: avg_slippage_bps < 5.0 (less than 5 basis points)

# 6. Daily P&L check (run each day)
tli rollout status

# Expected output (example Day 5):
# Phase: Small Position (Day 5/7)
# Capital Allocation: $50,000 (1%)
# Trades Executed: 1,204
# Real P&L: +$6,823 (+13.6%)
# Sharpe Ratio: 1.68 (Target: > 1.5) ✅
# Win Rate: 55.2% (Target: > 52%) ✅
# Max Drawdown: 7.1% (Target: < 10%) ✅
# Exit Criteria: 3/3 met ✅

# 7. After 7 days, review final results
tli rollout status

# Expected output:
# Phase: Small Position (Day 7/7)
# Trades Executed: 2,156
# Real P&L: +$8,450 (+16.9%)
# Total P&L: $8,450 (Target: > $5,000) ✅
# Sharpe Ratio: 1.72 (Target: > 1.5) ✅
# Max Drawdown: 8.5% (Target: < 10%) ✅
# Exit Criteria: 3/3 met ✅
# READY FOR PHASE 3

# 8. Generate Phase 2 report
tli rollout report --phase small-position --format pdf > phase2_report.pdf

# 9. Obtain sign-off from Trading Desk & Risk Management
# (Manual step: Email phase2_report.pdf to stakeholders)

# 10. Mark Phase 2 complete
echo "Phase 2: Small Position COMPLETE - $(date)" | tee -a deployment.log
```

**Validation Checklist**:
- [ ] Real P&L positive after transaction costs
- [ ] Slippage < 5 bps
- [ ] Zero execution errors or order rejections
- [ ] Sharpe ratio > 1.5
- [ ] Total P&L > $5,000
- [ ] Max drawdown < 10%
- [ ] Trading Desk & Risk Management sign-off received

**Exit Criteria**: All validation checks pass → Proceed to Phase 3

---

### 1.4 Phase 3: Medium Position Deployment (10% Capital)

**Duration**: 14 days
**Environment**: Production (scaled execution)
**Risk Level**: Medium ($500K capital, max $50K daily loss)

#### Step-by-Step Deployment

```bash
# 1. Confirm readiness for 10% capital
tli rollout advance --phase medium-position --capital-pct 10

# Expected prompt:
# "Deploy ensemble to 10% capital ($500,000)?"
# "Max position size: $100,000 per symbol"
# "Max daily loss: $50,000"
# "Dynamic position sizing: ENABLED"
# "Confirm deployment? [y/N]"

# 2. Type 'y' to confirm deployment

# Expected output:
# "Phase 3: Medium Position ACTIVATED"
# "Capital Allocation: $500,000 (10%)"
# "Risk Limits: ACTIVE"
# "Dynamic Position Sizing: ENABLED"
# "Real execution: ENABLED"

# 3. Monitor first 24 hours closely
watch -n 300 'tli rollout status' # Update every 5 minutes

# Expected output (example after 24 hours):
# Phase: Medium Position (Day 1/14)
# Capital Allocation: $500,000 (10%)
# Trades Executed: 347
# Real P&L: +$8,230 (+1.65%)
# Sharpe Ratio: 1.84
# Circuit Breaker: OK
# Model Weights: Stable

# 4. Start A/B test (compare ensemble vs baseline)
tli ab start --control DQN --treatment Ensemble --split 50/50 --duration 14d

# Expected output:
# "A/B Test Started: test_id=<uuid>"
# "Control: DQN (single model)"
# "Treatment: Ensemble (6 models)"
# "Traffic Split: 50/50"
# "Duration: 14 days"
# "Min Sample Size: 1,000 predictions per group"

# 5. Daily A/B test status check
tli ab status --test-id <uuid>

# Expected output (example Day 7):
# Test ID: a1b2c3d4-...
# Status: Running (Day 7/14)
# Control Group (DQN):
#   - Predictions: 5,432
#   - Sharpe Ratio: 1.64
#   - Win Rate: 53.2%
#   - Total P&L: $12,450
# Treatment Group (Ensemble):
#   - Predictions: 5,389
#   - Sharpe Ratio: 1.92 (+17.1%)
#   - Win Rate: 56.8% (+6.8%)
#   - Total P&L: $15,980 (+28.3%)
# Statistical Significance: p=0.003 (SIGNIFICANT)
# Recommendation: Roll out ensemble to 100%

# 6. Monitor model weight drift
curl http://localhost:9092/metrics | grep ensemble_model_weight

# Expected: Model weights stable (no sudden >20% changes)

# 7. After 14 days, review final results
tli rollout status

# Expected output:
# Phase: Medium Position (Day 14/14)
# Capital Allocation: $500,000 (10%)
# Trades Executed: 4,823
# Real P&L: +$47,230 (+9.45%)
# Sharpe Ratio: 1.89 (Target: > 1.8) ✅
# Win Rate: 55.7%
# Max Drawdown: 12.3% (Target: < 15%)
# Exit Criteria: 3/3 met ✅

# 8. Review A/B test final results
tli ab results --test-id <uuid>

# Expected output:
# Test ID: a1b2c3d4-...
# Status: Complete (14 days)
# Sharpe Lift: +19.2% (p < 0.001) ✅ SIGNIFICANT
# Win Rate Lift: +6.3% (p = 0.007) ✅ SIGNIFICANT
# P&L Lift: +24.1% (p < 0.001) ✅ SIGNIFICANT
# Recommendation: ROLL OUT ENSEMBLE TO 100%

# 9. Check for checkpoint rollbacks
psql $DATABASE_URL -c "SELECT COUNT(*) FROM checkpoint_swaps WHERE status='rollback' AND timestamp > NOW() - INTERVAL '14 days';"

# Expected: 0 (no rollbacks during Phase 3)

# 10. Generate Phase 3 report with A/B test results
tli rollout report --phase medium-position --format pdf > phase3_report.pdf
tli ab results --test-id <uuid> --format pdf > ab_test_results.pdf

# 11. Obtain sign-off from Trading Desk, Risk Management, Engineering Lead
# (Manual step: Email reports to stakeholders for final approval)

# 12. Mark Phase 3 complete
echo "Phase 3: Medium Position COMPLETE - $(date)" | tee -a deployment.log
```

**Validation Checklist**:
- [ ] Consistent profitability across multiple symbols
- [ ] Ensemble outperforms baseline (A/B test significant, p < 0.05)
- [ ] Model weights stable (no wild swings)
- [ ] Sharpe ratio > 1.8 (2 weeks of real trading)
- [ ] Zero checkpoint rollbacks
- [ ] All stakeholder sign-offs received

**Exit Criteria**: All validation checks pass → Proceed to Phase 4

---

### 1.5 Phase 4: Full Deployment (100% Capital)

**Duration**: Ongoing
**Environment**: Production (full allocation)
**Risk Level**: High ($5M capital, max $250K daily loss)

#### Step-by-Step Deployment

```bash
# 1. Final confirmation for full deployment
tli rollout advance --phase full-deployment --capital-pct 100

# Expected prompt:
# "Deploy ensemble to 100% capital ($5,000,000)?"
# "Max position size: $1,000,000 per symbol"
# "Max daily loss: $250,000"
# "VaR-based position sizing: ENABLED"
# "This is FULL PRODUCTION DEPLOYMENT"
# "Confirm deployment? [y/N]"

# 2. Type 'y' to confirm full deployment

# Expected output:
# "Phase 4: Full Deployment ACTIVATED"
# "Capital Allocation: $5,000,000 (100%)"
# "Risk Limits: ACTIVE"
# "VaR-based Position Sizing: ENABLED"
# "Real execution: FULL PRODUCTION MODE"
# "Circuit Breaker: ARMED"

# 3. Monitor first week intensively
watch -n 300 'tli rollout status' # Update every 5 minutes

# 4. Daily health checks (automated cron job)
# Add to crontab:
0 9 * * * /home/jgrusewski/Work/foxhunt/scripts/daily_health_check.sh

# 5. Weekly checkpoint updates (automated)
# Add to crontab (every Sunday at 2 AM):
0 2 * * 0 /home/jgrusewski/Work/foxhunt/scripts/weekly_checkpoint_update.sh

# 6. Monthly A/B tests for new variants (manual trigger)
tli ab start --control Ensemble --treatment EnsembleV2 --split 90/10 --duration 30d

# 7. Quarterly retraining with latest market data (manual trigger)
tli train start --model Ensemble --data last_90_days --epochs 500

# 8. Mark Phase 4 active
echo "Phase 4: Full Deployment ACTIVE - $(date)" | tee -a deployment.log
echo "System is now in full production mode" | tee -a deployment.log
```

**Ongoing Monitoring**:
- Daily P&L attribution per model
- Weekly checkpoint health checks
- Monthly A/B tests for new variants
- Quarterly retraining with latest data
- Real-time alerting (PagerDuty + Slack)

---

## 2. Rollback Procedures

### 2.1 Emergency Rollback (Immediate)

**Trigger Conditions**:
- Circuit breaker triggered (3 consecutive losses)
- Daily drawdown > 5%
- Disagreement rate > 70% for 1 hour
- Latency P99 > 100μs for 5 minutes
- Any model error rate > 5%

**Automatic Rollback** (system handles automatically):
```bash
# Circuit breaker triggers automatically, no manual intervention needed
# Log entry will show:
# "Circuit breaker TRIGGERED: reason=<condition>"
# "Trading HALTED"
# "Initiating automatic rollback to baseline strategy"
# "Rollback complete in <duration> seconds"
```

**Manual Rollback** (if automatic rollback fails):
```bash
# 1. Immediately halt trading
tli rollout halt --reason "Manual intervention: <reason>"

# Expected output:
# "Trading HALTED"
# "All open orders CANCELLED"
# "Circuit breaker: ENGAGED"

# 2. Initiate rollback to previous phase
tli rollout rollback --reason "<detailed reason>"

# Expected output:
# "Initiating rollback to Phase <N-1>"
# "Capital allocation: <previous %>"
# "Risk limits: <previous settings>"
# "Rollback complete in <duration> seconds"

# 3. Verify rollback successful
tli rollout status

# Expected output:
# Phase: <Previous Phase> (ROLLBACK MODE)
# Trading: HALTED
# Circuit Breaker: ENGAGED
# Last Rollback: <timestamp> (<reason>)

# 4. Hot-swap to last known-good checkpoints
tli ensemble swap --model all --checkpoint last-known-good

# Expected output:
# "DQN Epoch 10: Swapped to checkpoint from <date>"
# "DQN Epoch 380: Swapped to checkpoint from <date>"
# "PPO Epoch 380: Swapped to checkpoint from <date>"
# "PPO Epoch 500: Swapped to checkpoint from <date>"
# "MAMBA-2: Swapped to checkpoint from <date>"
# "TFT: Swapped to checkpoint from <date>"
# "Hot-swap complete in <duration>ms"

# 5. Run 5-minute canary validation
tli ensemble validate --duration 5m

# Expected output:
# "Running canary validation (5 minutes)..."
# "Predictions: 1,000"
# "Latency P99: 48μs ✅"
# "Error Rate: 0.0% ✅"
# "Accuracy: 56.2% ✅"
# "Canary validation: PASSED"

# 6. Resume trading (if canary passed)
tli rollout resume

# Expected output:
# "Circuit breaker: DISENGAGED"
# "Trading: RESUMED"
# "Risk limits: ACTIVE"

# 7. Emit PagerDuty alert
# (Automatic: System sends alert to on-call engineer)

# 8. Post-mortem incident report (within 24 hours)
# Create incident report: INCIDENT_<timestamp>.md
```

**Recovery Time Objective (RTO)**: < 5 minutes

---

### 2.2 Planned Rollback (Controlled)

**Use Case**: Exit criteria not met, gradual rollback to previous phase

```bash
# 1. Review current phase status
tli rollout status

# Example output showing failed exit criteria:
# Phase: Medium Position (Day 14/14)
# Sharpe Ratio: 1.62 (Target: > 1.8) ❌
# Win Rate: 51.3% (Target: > 52%) ❌
# Exit Criteria: 1/3 met ❌
# RECOMMENDATION: Rollback to Phase 2

# 2. Initiate controlled rollback
tli rollout rollback --controlled --reason "Exit criteria not met"

# Expected output:
# "Initiating controlled rollback to Phase 2"
# "Capital will be gradually reduced over 24 hours"
# "Current: $500,000 (10%) → Target: $50,000 (1%)"
# "Risk limits will be adjusted automatically"

# 3. Monitor rollback progress
watch -n 3600 'tli rollout status' # Check every hour

# Example output after 12 hours:
# Phase: Medium Position → Small Position (50% complete)
# Capital Allocation: $275,000 (5.5%)
# Risk Limits: Adjusting
# ETA to Phase 2: 12 hours

# 4. Verify rollback complete
tli rollout status

# Expected output:
# Phase: Small Position (ROLLBACK COMPLETE)
# Capital Allocation: $50,000 (1%)
# Risk Limits: Phase 2 settings ACTIVE
# Trading: ACTIVE

# 5. Document rollback reason
echo "Planned rollback from Phase 3 to Phase 2 - $(date)" | tee -a deployment.log
echo "Reason: Exit criteria not met (Sharpe < 1.8, Win Rate < 52%)" | tee -a deployment.log

# 6. Schedule review meeting with stakeholders
# (Manual step: Schedule meeting to discuss next steps)
```

**Recovery Time Objective (RTO)**: 24 hours (gradual capital reduction)

---

## 3. Health Check Commands

### 3.1 System-Wide Health Checks

```bash
# Quick health check (all services)
/home/jgrusewski/Work/foxhunt/scripts/quick_health_check.sh

# Expected output:
# ✅ API Gateway: HEALTHY (port 50051)
# ✅ Trading Service: HEALTHY (port 50052)
# ✅ ML Training Service: HEALTHY (port 50054)
# ✅ PostgreSQL: HEALTHY
# ✅ Redis: HEALTHY
# ✅ Prometheus: HEALTHY (4/4 targets up)
# ✅ Grafana: HEALTHY
# ✅ GPU: HEALTHY (3.8GB VRAM free)
#
# Overall System Status: HEALTHY
```

### 3.2 Ensemble-Specific Health Checks

```bash
# Check ensemble coordinator status
tli ensemble status

# Expected output:
# Ensemble Coordinator: HEALTHY
# Models Loaded: 6/6
#   - DQN Epoch 10: ✅ (75,628 bytes)
#   - DQN Epoch 380: ✅ (75,628 bytes)
#   - PPO Epoch 380: ✅ (42,000 bytes)
#   - PPO Epoch 500: ✅ (42,000 bytes)
#   - MAMBA-2: ✅ (350 MB)
#   - TFT: ✅ (2.1 GB)
# Inference Latency (P99):
#   - DQN: 47μs ✅
#   - PPO: 49μs ✅
#   - MAMBA-2: 94μs ✅
#   - TFT: 138μs ✅
# Error Rate (24h): 0.02% ✅
# Last Checkpoint Swap: 3 days ago ✅

# Check model disagreement rate
curl -s http://localhost:9092/metrics | grep ensemble_disagreement_rate | tail -1

# Expected output:
# ensemble_disagreement_rate{symbol="ES.FUT"} 0.28

# Check for high disagreement events
curl -s http://localhost:9092/metrics | grep ensemble_high_disagreement_total

# Expected output:
# ensemble_high_disagreement_total{symbol="ES.FUT",threshold="0.7"} 0

# Check checkpoint swap health
curl -s http://localhost:9092/metrics | grep checkpoint_swaps_total

# Expected output:
# checkpoint_swaps_total{model_id="DQN_Epoch_10",status="success"} 12
# checkpoint_swaps_total{model_id="DQN_Epoch_10",status="rollback"} 0
```

### 3.3 Database Health Checks

```bash
# Check PostgreSQL connection
psql $DATABASE_URL -c 'SELECT 1;'

# Expected: 1 row returned

# Check ensemble predictions table size
psql $DATABASE_URL -c "SELECT pg_size_pretty(pg_total_relation_size('ensemble_predictions'));"

# Expected: < 50 GB (after 1 year of operation)

# Check recent predictions count
psql $DATABASE_URL -c "SELECT COUNT(*) FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '1 day';"

# Expected: 2,000-5,000 predictions per day

# Check for database errors
psql $DATABASE_URL -c "SELECT COUNT(*) FROM system_logs WHERE log_level='ERROR' AND timestamp > NOW() - INTERVAL '1 hour';"

# Expected: 0 (no errors in past hour)
```

### 3.4 Performance Health Checks

```bash
# Check Trading Service latency
curl -s http://localhost:9092/metrics | grep trading_service_request_duration_seconds

# Expected: P99 < 100ms

# Check GPU utilization
nvidia-smi --query-gpu=utilization.gpu,utilization.memory,memory.used,memory.free --format=csv,noheader

# Expected output:
# 35 %, 40 %, 1600 MiB, 2496 MiB

# Check Redis memory usage
redis-cli INFO memory | grep used_memory_human

# Expected: used_memory_human < 1GB
```

---

## 4. Troubleshooting Guide

### 4.1 Model Loading Failures

**Symptom**: Trading Service fails to start with error "Failed to load model checkpoint"

**Diagnosis**:
```bash
# 1. Check checkpoint file exists
ls -lh ml/trained_models/production/dqn_real_data/dqn_epoch_10.safetensors

# 2. Verify file size
du -h ml/trained_models/production/dqn_real_data/dqn_epoch_10.safetensors

# Expected: 75,628 bytes (74KB)

# 3. Check file permissions
stat ml/trained_models/production/dqn_real_data/dqn_epoch_10.safetensors

# Expected: -rw-rw-r-- (644)

# 4. Verify SafeTensors format
hexdump -C ml/trained_models/production/dqn_real_data/dqn_epoch_10.safetensors | head -5

# Expected: JSON header starting with {"layer_...
```

**Solution**:
```bash
# If checkpoint corrupted, restore from backup
cp ml/trained_models/backups/dqn_epoch_10.safetensors ml/trained_models/production/dqn_real_data/

# If backup unavailable, re-download from MinIO
aws s3 cp s3://foxhunt/checkpoints/dqn/dqn_epoch_10.safetensors ml/trained_models/production/dqn_real_data/

# Restart Trading Service
pkill -9 trading_service
cargo run -p trading_service --release &
```

---

### 4.2 High Disagreement Rate

**Symptom**: `ensemble_disagreement_rate` > 70% for extended period

**Diagnosis**:
```bash
# 1. Check disagreement rate trend
curl -s http://localhost:9092/metrics | grep ensemble_disagreement_rate

# 2. Check which models are disagreeing
psql $DATABASE_URL -c "SELECT model_id, AVG(ABS(signal - ensemble_action)) AS avg_disagreement FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '1 hour' GROUP BY model_id ORDER BY avg_disagreement DESC;"

# Expected output (example):
# model_id       | avg_disagreement
# ---------------+-----------------
# DQN_Epoch_10   | 0.82  (HIGH)
# MAMBA-2        | 0.75  (HIGH)
# PPO_Epoch_380  | 0.34  (NORMAL)
# PPO_Epoch_500  | 0.31  (NORMAL)
# DQN_Epoch_380  | 0.28  (NORMAL)
# TFT            | 0.25  (NORMAL)

# 3. Check market volatility (possible regime shift)
psql $DATABASE_URL -c "SELECT symbol, STDDEV(close) AS volatility FROM market_data WHERE timestamp > NOW() - INTERVAL '1 hour' GROUP BY symbol;"

# If volatility > 2x normal: Market regime shift (high disagreement expected)
```

**Solution**:
```bash
# If market regime shift (high volatility):
# 1. Reduce position sizes automatically (circuit breaker handles this)
# 2. Monitor for 1 hour
# 3. If disagreement persists > 1 hour, halt trading
tli rollout halt --reason "Persistent high disagreement rate"

# If specific model(s) causing disagreement:
# 1. Identify problematic model (e.g., DQN Epoch 10)
# 2. Hot-swap to alternative checkpoint
tli ensemble swap --model DQN_Epoch_10 --checkpoint dqn_epoch_30

# 3. Run canary validation
tli ensemble validate --duration 5m

# 4. If validation passes, resume trading
tli rollout resume
```

---

### 4.3 Latency Spikes

**Symptom**: Inference latency P99 > 100μs (2x normal)

**Diagnosis**:
```bash
# 1. Check current latency
curl -s http://localhost:9092/metrics | grep ensemble_aggregation_latency_microseconds

# 2. Check GPU utilization
nvidia-smi

# If GPU utilization > 90%: GPU overloaded

# 3. Check system load
uptime

# If load average > 8.0 (4 cores): CPU overloaded

# 4. Check for memory pressure
free -h

# If available memory < 1GB: Memory pressure
```

**Solution**:
```bash
# If GPU overloaded:
# 1. Reduce inference batch size (if batching enabled)
# 2. Check for competing processes
ps aux | grep -E "(python|jupyter|pytorch)" | grep -v grep

# Kill non-critical GPU processes
kill -9 <PID>

# If CPU overloaded:
# 1. Check for runaway processes
top -o %CPU

# Kill non-critical processes

# If memory pressure:
# 1. Clear caches
sync; echo 3 > /proc/sys/vm/drop_caches

# 2. Restart services to free memory
./scripts/restart_services.sh

# 3. If latency still high, halt trading temporarily
tli rollout halt --reason "Latency spike investigation"

# 4. Investigate root cause, fix, validate, resume
```

---

### 4.4 Circuit Breaker Triggered

**Symptom**: Trading halted automatically due to circuit breaker

**Diagnosis**:
```bash
# 1. Check why circuit breaker triggered
psql $DATABASE_URL -c "SELECT * FROM system_alerts WHERE alert_type='circuit_breaker' ORDER BY timestamp DESC LIMIT 1;"

# Example output:
# alert_type      | timestamp           | reason
# ----------------+---------------------+-------
# circuit_breaker | 2025-10-14 10:23:15 | 3 consecutive losses

# 2. Check recent trades
psql $DATABASE_URL -c "SELECT order_id, symbol, action, pnl FROM orders WHERE timestamp > NOW() - INTERVAL '1 hour' ORDER BY timestamp DESC LIMIT 10;"

# Example output:
# order_id | symbol  | action | pnl
# ---------+---------+--------+------
# uuid1    | ES.FUT  | SELL   | -$450
# uuid2    | NQ.FUT  | BUY    | -$320
# uuid3    | ES.FUT  | BUY    | -$280
# uuid4    | ES.FUT  | SELL   | +$210
# uuid5    | NQ.FUT  | SELL   | +$180

# 3. Check ensemble confidence during losses
psql $DATABASE_URL -c "SELECT ensemble_confidence, disagreement_rate FROM ensemble_predictions WHERE timestamp > NOW() - INTERVAL '1 hour' ORDER BY timestamp DESC LIMIT 10;"
```

**Solution**:
```bash
# 1. Identify root cause:
#    - Low confidence trades?
#    - High disagreement?
#    - Market volatility spike?
#    - Model degradation?

# 2. If low confidence trades:
# Increase confidence threshold temporarily
psql $DATABASE_URL -c "UPDATE config SET value='0.70' WHERE key='min_ensemble_confidence';"

# 3. If market volatility spike:
# Wait for market to stabilize (monitor for 15 minutes)
watch -n 60 'psql $DATABASE_URL -c "SELECT STDDEV(close) FROM market_data WHERE timestamp > NOW() - INTERVAL \"5 minutes\" GROUP BY symbol;"'

# 4. If model degradation:
# Hot-swap to last known-good checkpoints
tli ensemble swap --model all --checkpoint last-known-good

# 5. Run canary validation
tli ensemble validate --duration 5m

# 6. If validation passes, resume trading
tli rollout resume

# 7. Document incident
echo "Circuit breaker triggered: <reason> - $(date)" | tee -a incidents.log
```

---

### 4.5 Database Connection Failures

**Symptom**: Trading Service logs show "Database connection lost"

**Diagnosis**:
```bash
# 1. Check PostgreSQL is running
docker-compose ps postgres

# Expected: State = Up

# 2. Check PostgreSQL logs
docker-compose logs postgres | tail -50

# Look for errors like:
# - "too many connections"
# - "out of memory"
# - "disk full"

# 3. Check active connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"

# Expected: < 100 connections

# 4. Check disk space
df -h

# Expected: / filesystem < 80% used
```

**Solution**:
```bash
# If too many connections:
# 1. Kill idle connections
psql $DATABASE_URL -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state='idle' AND query_start < NOW() - INTERVAL '5 minutes';"

# 2. Restart services to reset connection pools
./scripts/restart_services.sh

# If disk full:
# 1. Clear old log files
find /var/log -name "*.log" -mtime +7 -delete

# 2. Archive old PostgreSQL WAL files
docker-compose exec postgres pg_archivecleanup /var/lib/postgresql/data/pg_wal 000000010000000000000010

# 3. Monitor disk usage
watch -n 60 'df -h'

# If PostgreSQL crashed:
# 1. Restart PostgreSQL
docker-compose restart postgres

# 2. Wait for recovery (may take 1-5 minutes)
docker-compose logs -f postgres

# 3. Verify database accessible
psql $DATABASE_URL -c 'SELECT 1;'

# 4. Restart Trading Service
./scripts/restart_services.sh
```

---

## 5. On-Call Playbook

### 5.1 On-Call Engineer Responsibilities

**Primary Responsibilities**:
- Respond to PagerDuty alerts within 15 minutes
- Investigate and resolve critical issues within 1 hour
- Escalate to senior engineers if issue unresolved in 2 hours
- Document all incidents in `incidents.log`
- Conduct post-mortem within 24 hours of critical incidents

**Escalation Path**:
1. **L1 (On-Call Engineer)**: Initial response, basic troubleshooting
2. **L2 (Senior ML Engineer)**: Model-specific issues, complex debugging
3. **L3 (Engineering Lead)**: Architectural issues, rollback decisions
4. **L4 (Trading Desk Manager)**: Capital allocation, risk limits

---

### 5.2 Alert Severity Levels

**Critical (PagerDuty)**:
- Circuit breaker triggered
- Checkpoint rollback
- Trading halted unexpectedly
- Database connection lost
- All models erroring simultaneously

**Response**: Immediate (within 15 minutes)

**High (Slack + PagerDuty)**:
- High disagreement rate (> 60%)
- Latency spike (P99 > 75μs)
- Single model erroring (> 5% error rate)
- Daily loss approaching limit (> 80% of max)

**Response**: Within 30 minutes

**Medium (Slack)**:
- Model weight drift (> 15% change in 1 hour)
- Slippage above target (> 3 bps)
- Checkpoint swap failed (but fallback succeeded)

**Response**: Within 2 hours

**Low (Email)**:
- Daily summary report
- Weekly performance review
- Monthly A/B test results

**Response**: Next business day

---

### 5.3 Common Alert Scenarios

#### Scenario 1: Circuit Breaker Triggered (3 Consecutive Losses)

**Alert**:
```
🚨 CRITICAL: Circuit Breaker Triggered
Reason: 3 consecutive losses
Total Loss: $1,450
Trading: HALTED
Action: Investigate immediately
```

**Playbook**:
1. **Acknowledge alert** (PagerDuty): < 5 minutes
2. **Check recent trades**: See Section 4.4
3. **Identify root cause**: Low confidence? High disagreement? Volatility?
4. **Apply fix**: Hot-swap checkpoints, adjust confidence threshold, or wait for market stabilization
5. **Validate fix**: 5-minute canary period
6. **Resume trading** or **escalate to L2** if unresolved
7. **Document incident**: `incidents.log`
8. **Post-mortem**: Within 24 hours

**Estimated Time to Resolution**: 15-30 minutes

---

#### Scenario 2: High Disagreement Rate (> 70%)

**Alert**:
```
⚠️ WARNING: High Ensemble Disagreement
Disagreement Rate: 73.2%
Symbol: ES.FUT
Duration: 15 minutes
Action: Monitor closely
```

**Playbook**:
1. **Acknowledge alert** (Slack): < 10 minutes
2. **Check market volatility**: See Section 4.2
3. **If market regime shift** (volatility > 2x normal):
   - Monitor for 1 hour
   - Position sizes automatically reduced by circuit breaker
   - If disagreement persists > 1 hour, halt trading
4. **If specific model causing disagreement**:
   - Identify problematic model
   - Hot-swap to alternative checkpoint
   - Run canary validation
5. **Document incident**: `incidents.log`

**Estimated Time to Resolution**: 30-60 minutes

---

#### Scenario 3: Database Connection Lost

**Alert**:
```
🚨 CRITICAL: Database Connection Lost
Service: Trading Service
Duration: 30 seconds
Action: Investigate immediately
```

**Playbook**:
1. **Acknowledge alert** (PagerDuty): < 5 minutes
2. **Check PostgreSQL status**: See Section 4.5
3. **If PostgreSQL down**:
   - Restart PostgreSQL: `docker-compose restart postgres`
   - Wait for recovery (1-5 minutes)
   - Restart Trading Service
4. **If too many connections**:
   - Kill idle connections
   - Restart services to reset connection pools
5. **Verify trading operational**: `tli rollout status`
6. **Document incident**: `incidents.log`

**Estimated Time to Resolution**: 5-15 minutes

---

### 5.4 Emergency Contact List

**Engineering**:
- **On-Call Engineer**: PagerDuty rotation
- **Senior ML Engineer**: ml-team@foxhunt.trading
- **Engineering Lead**: eng-lead@foxhunt.trading

**Trading Desk**:
- **Trading Desk Manager**: trading-desk@foxhunt.trading
- **Head of Trading**: head-of-trading@foxhunt.trading

**Risk Management**:
- **Risk Manager**: risk@foxhunt.trading
- **Chief Risk Officer**: cro@foxhunt.trading

**Executive**:
- **CTO**: cto@foxhunt.trading
- **CEO**: ceo@foxhunt.trading

**Slack Channels**:
- **#trading-alerts**: Real-time alerts (monitored 24/7)
- **#ensemble-deployment**: Deployment updates
- **#incidents**: Post-mortem discussions

---

## 6. Common Issues & Solutions

### Issue 1: Model Fails to Load on Startup

**Error**: `Failed to load model checkpoint: /path/to/checkpoint.safetensors`

**Solution**: See Section 4.1 (Model Loading Failures)

---

### Issue 2: Inference Latency Exceeds Target

**Error**: `Ensemble aggregation latency P99: 127μs (target: < 50μs)`

**Solution**: See Section 4.3 (Latency Spikes)

---

### Issue 3: Checkpoint Swap Fails

**Error**: `Checkpoint swap failed: Validation timeout`

**Solution**:
```bash
# 1. Check if new checkpoint is valid
hexdump -C /path/to/new_checkpoint.safetensors | head -5

# 2. If invalid, abort swap (system keeps active buffer)
# 3. If valid but validation failed, check for:
#    - GPU memory exhaustion
#    - Network timeout (if checkpoint on MinIO)

# 4. Retry swap after resolving issue
tli ensemble swap --model <model_id> --checkpoint <path> --retry
```

---

### Issue 4: A/B Test Contamination

**Error**: `A/B test results invalid: Group assignment changed mid-test`

**Solution**:
```bash
# 1. Stop contaminated test
tli ab stop --test-id <uuid>

# 2. Verify deterministic hashing enabled
psql $DATABASE_URL -c "SELECT * FROM ab_test_config WHERE test_id='<uuid>';"

# Expected: hash_seed is consistent throughout test

# 3. Start new test with fresh group assignments
tli ab start --control <model> --treatment <model> --split 50/50 --duration 14d
```

---

### Issue 5: Daily Drawdown Approaching Limit

**Warning**: `Daily drawdown: 4.2% (limit: 5.0%)`

**Solution**:
```bash
# 1. Reduce position sizes immediately
psql $DATABASE_URL -c "UPDATE config SET value='0.50' WHERE key='position_size_multiplier';"

# 2. Increase confidence threshold
psql $DATABASE_URL -c "UPDATE config SET value='0.75' WHERE key='min_ensemble_confidence';"

# 3. Monitor closely for next 30 minutes
watch -n 60 'tli rollout status'

# 4. If drawdown exceeds 5%, circuit breaker will halt automatically

# 5. If approaching limit again, halt trading manually
tli rollout halt --reason "Approaching daily drawdown limit"
```

---

## 7. Emergency Contacts

### 7.1 24/7 Support

**PagerDuty**: Ensemble Trading System rotation
- **URL**: https://foxhunt.pagerduty.com/schedules/ensemble-trading
- **Escalation Policy**: L1 (15 min) → L2 (30 min) → L3 (1 hour)

**Slack**: #trading-alerts (monitored 24/7)
- **Bot**: @ensemble-bot (automated alerts)
- **On-Call**: @on-call (current on-call engineer)

### 7.2 Business Hours Support

**Email**: ml-team@foxhunt.trading (9 AM - 5 PM ET)
**Phone**: (555) 123-4567 (emergency hotline)

### 7.3 External Vendors

**Databento Support**: support@databento.com (market data issues)
**AWS Support**: Premium support (GPU/infrastructure issues)
**PostgreSQL DBA**: dba@foxhunt.trading (database emergencies)

---

## Appendix: Quick Reference Commands

```bash
# Health Checks
./scripts/quick_health_check.sh
tli ensemble status
tli rollout status

# Deployment
tli rollout start --phase <phase> --duration <days>
tli rollout advance --phase <phase> --capital-pct <pct>

# Rollback
tli rollout halt --reason "<reason>"
tli rollout rollback --reason "<reason>"
tli rollout resume

# Model Management
tli ensemble swap --model <model_id> --checkpoint <path>
tli ensemble validate --duration <minutes>

# A/B Testing
tli ab start --control <model> --treatment <model> --split 50/50 --duration <days>
tli ab status --test-id <uuid>
tli ab stop --test-id <uuid>

# Monitoring
curl http://localhost:9092/metrics | grep ensemble
psql $DATABASE_URL -c "SELECT * FROM ensemble_predictions ORDER BY timestamp DESC LIMIT 10;"

# Emergency
pkill -9 trading_service
./scripts/restart_services.sh
docker-compose restart postgres
```

---

**Document Status**: ✅ READY FOR PRODUCTION USE
**Last Updated**: 2025-10-14
**Next Review**: Monthly (or after major incidents)