foxhunt/docs/monitoring/ENSEMBLE_ALERT_RUNBOOKS.md

# Ensemble Alert Runbooks
# Operational Response Procedures

**Document Version**: 1.0
**Date**: 2025-10-14
**System**: Foxhunt HFT Trading System - 6-Model Ensemble
**Alert Count**: 22 alert rules across 7 categories

---

## Table of Contents

1. [Performance Degradation](#1-performance-degradation)
2. [Model Disagreement & Uncertainty](#2-model-disagreement--uncertainty)
3. [Latency & Performance](#3-latency--performance)
4. [Model Failures & Cascades](#4-model-failures--cascades)
5. [Model Weight Anomalies](#5-model-weight-anomalies)
6. [A/B Testing](#6-ab-testing)
7. [System Health](#7-system-health)

---

## 1. Performance Degradation

### 1.1 EnsembleSharpeRatioDropCritical

**Alert**: Sharpe ratio dropped >50% in 15 minutes
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 5 minutes

#### Symptoms
- Sharpe ratio <50% of baseline
- Trading strategy profitability severely degraded
- May indicate regime shift or model drift

#### Investigation Steps

```bash
# 1. Check current Sharpe ratio
curl -s http://localhost:9092/metrics | grep ensemble_model_pnl_contribution

# 2. Open Grafana dashboard
# Navigate to: http://localhost:3000/d/ensemble-ml-production

# 3. Check model disagreement rates
curl -s http://localhost:9092/metrics | grep ensemble_disagreement_rate

# 4. Review model weights (any single model dominating?)
curl -s http://localhost:9092/metrics | grep ensemble_model_weight

# 5. Check for market regime changes
# Review: VIX spike, news events, market hours (pre-market/after-hours)

# 6. Review recent P&L by model
psql $DATABASE_URL -c "
SELECT
    model_id,
    symbol,
    SUM(pnl_contribution_dollars) as total_pnl,
    COUNT(*) as prediction_count,
    AVG(confidence) as avg_confidence
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '1 hour'
GROUP BY model_id, symbol
ORDER BY total_pnl;
"
```

#### Response Actions

**IMMEDIATE (0-5 minutes)**:
1. Reduce position sizes by 50% across all symbols
2. Tighten stop-losses to -1% (from -2%)
3. Review all open positions - close any with confidence <0.6

**SHORT-TERM (5-15 minutes)**:
1. If any model contributing negative P&L >$500/hour → set weight to 0
2. Switch to defensive trading mode (reduced frequency, higher confidence threshold)
3. Review model predictions vs actual market moves (are predictions accurate?)

**ESCALATION**:
- If Sharpe ratio doesn't recover within 30 minutes → Page ML team
- If drawdown >10% → HALT automated trading immediately
- If issue persists >1 hour → Switch to manual trading or single best model

#### Resolution Criteria
- Sharpe ratio returns to >75% of baseline for 30+ minutes
- No single model dominating (max weight <60%)
- Disagreement rate <50%

---

### 1.2 EnsembleWinRateCollapse

**Alert**: Win rate collapsed below 40%
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 2 minutes

#### Symptoms
- Win rate <40% over 15-minute window
- Strategy actively losing money
- Immediate intervention required

#### Investigation Steps

```bash
# 1. Check current win rate by action
curl -s http://localhost:9092/metrics | grep ensemble_predictions_total

# 2. Calculate win rate
# Win rate = (winning trades) / (total trades)

# 3. Review P&L attribution by model
psql $DATABASE_URL -c "
SELECT
    model_id,
    COUNT(CASE WHEN pnl_contribution_dollars > 0 THEN 1 END) as winning_predictions,
    COUNT(*) as total_predictions,
    100.0 * COUNT(CASE WHEN pnl_contribution_dollars > 0 THEN 1 END) / COUNT(*) as win_rate_pct
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '15 minutes'
GROUP BY model_id
ORDER BY win_rate_pct;
"

# 4. Identify losing models
# Any model with win rate <40% should have weight reduced to 0
```

#### Response Actions

**IMMEDIATE (0-2 minutes)**:
1. **STOP ALL NEW POSITION ENTRIES** - close entry logic
2. Review all open positions - set stop-losses to current price +/- 0.5%
3. Identify models with negative P&L → set weight to 0 immediately

**SHORT-TERM (2-10 minutes)**:
1. Close all positions with unrealized loss >-2%
2. Switch to single best-performing model (highest win rate + positive P&L)
3. Reduce trading frequency by 75%

**ESCALATION**:
- If drawdown >10% → HALT all automated trading immediately
- Page ML team and risk manager
- Prepare incident report

#### Resolution Criteria
- Win rate returns to >50% for 30+ minutes
- P&L trend positive for 1+ hour
- All models contributing positive P&L

---

### 1.3 EnsembleNegativePnLTrend

**Alert**: Consistent negative P&L over 30 minutes
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 1 minute

#### Symptoms
- 30-minute P&L <-$1000
- Active capital loss
- Trading system hemorrhaging money

#### Investigation Steps

```bash
# 1. Check P&L by model
curl -s http://localhost:9092/metrics | grep ensemble_model_pnl_contribution_dollars

# 2. Review all open positions
psql $DATABASE_URL -c "
SELECT
    symbol,
    COUNT(*) as position_count,
    SUM(quantity) as net_position,
    SUM(unrealized_pnl) as total_unrealized_pnl
FROM positions
WHERE status = 'open'
GROUP BY symbol;
"

# 3. Identify losing models
psql $DATABASE_URL -c "
SELECT
    model_id,
    SUM(pnl_contribution_dollars) as total_pnl
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '30 minutes'
GROUP BY model_id
HAVING SUM(pnl_contribution_dollars) < -500
ORDER BY total_pnl;
"
```

#### Response Actions

**IMMEDIATE (0-1 minute)**:
1. **HALT ALL AUTOMATED TRADING** - stop order submission
2. Close all positions with unrealized loss
3. Set all model weights to 0 except best performer

**SHORT-TERM (1-5 minutes)**:
1. Review trading logs for errors
2. Check for broker connectivity issues
3. Verify market data feed is current (not stale)

**ESCALATION**:
- Page ML team, risk manager, and CTO immediately
- Prepare detailed incident report with:
  - Total P&L loss
  - Models contributing to loss
  - Market conditions during loss period
  - Root cause analysis

#### Resolution Criteria
- Automated trading halted
- All losing positions closed
- Root cause identified and documented
- ML team provides go/no-go decision for resuming trading

---

## 2. Model Disagreement & Uncertainty

### 2.1 EnsembleHighDisagreementCritical

**Alert**: Model disagreement >70% for 5 minutes
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 10 minutes

#### Symptoms
- Models fundamentally disagree on predictions
- Disagreement rate >70% sustained
- Prediction reliability compromised

#### Investigation Steps

```bash
# 1. Check disagreement rate by symbol
curl -s http://localhost:9092/metrics | grep ensemble_disagreement_rate

# 2. Review individual model predictions
psql $DATABASE_URL -c "
SELECT
    model_id,
    symbol,
    predicted_action,
    confidence,
    timestamp
FROM ensemble_model_predictions
WHERE timestamp > NOW() - INTERVAL '5 minutes'
    AND symbol = 'ES.FUT'  -- Replace with affected symbol
ORDER BY timestamp DESC
LIMIT 50;
"

# 3. Check for market regime shift
# Indicators: VIX spike, volatility increase, news events

# 4. Review model confidence scores
curl -s http://localhost:9092/metrics | grep ensemble_confidence_score
```

#### Response Actions

**IMMEDIATE (0-5 minutes)**:
1. Reduce position sizes by 50% for affected symbol
2. Increase confidence threshold from 0.7 to 0.85
3. Stop opening new positions for affected symbol

**SHORT-TERM (5-15 minutes)**:
1. If disagreement persists >15 minutes → pause trading for symbol
2. Review recent news/events for symbol
3. Check if disagreement is symbol-specific or system-wide

**LONG-TERM (15+ minutes)**:
1. If system-wide → may indicate market regime change
2. Consider retraining models on recent data
3. Review model weights - may need manual rebalancing

#### Resolution Criteria
- Disagreement rate <50% for 15+ minutes
- Model confidence >0.7
- Predictions aligned with market direction

---

### 2.2 EnsembleLowConfidenceHighDisagreement

**Alert**: Low confidence (<0.6) AND high disagreement (>0.7)
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 2 minutes

#### Symptoms
- Models uncertain AND disagreeing
- Worst-case scenario for prediction reliability
- Prediction quality at minimum

#### Investigation Steps

```bash
# 1. Check both metrics
curl -s http://localhost:9092/metrics | grep -E "ensemble_confidence_score|ensemble_disagreement_rate"

# 2. Review what models are predicting
psql $DATABASE_URL -c "
SELECT
    model_id,
    predicted_action,
    confidence,
    COUNT(*) as prediction_count
FROM ensemble_model_predictions
WHERE timestamp > NOW() - INTERVAL '2 minutes'
    AND symbol = 'ES.FUT'  -- Replace with affected symbol
GROUP BY model_id, predicted_action, confidence
ORDER BY model_id;
"
```

#### Response Actions

**IMMEDIATE (0-2 minutes)**:
1. **HALT ALL TRADING FOR AFFECTED SYMBOL** - immediate stop
2. Switch to observation mode (log predictions without execution)
3. Close all open positions for affected symbol with tight stop-losses

**SHORT-TERM (2-10 minutes)**:
1. Manually review market conditions
2. Check for:
   - Flash crash
   - News events
   - Exchange issues
   - Data feed problems

**ESCALATION**:
- Do NOT resume automated trading until:
  - Confidence >0.7 AND disagreement <0.5 for 15+ minutes
  - Manual review confirms market stability
  - ML team provides approval

#### Resolution Criteria
- Confidence >0.7
- Disagreement <0.5
- Manual approval from ML team

---

## 3. Latency & Performance

### 3.1 EnsembleAggregationLatencyP99High

**Alert**: Aggregation P99 latency >50μs for 1 minute
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 15 minutes

#### Symptoms
- P99 latency >50μs (SLA violation)
- HFT execution delays
- Potential alpha decay and slippage

#### Investigation Steps

```bash
# 1. Check current latency
curl -s http://localhost:9092/metrics | grep ensemble_aggregation_latency_microseconds

# 2. Check CPU utilization
top -n 1 | grep trading_service

# 3. Check concurrent prediction load
curl -s http://localhost:9092/metrics | grep ensemble_predictions_total | tail -5

# 4. Review latency breakdown by aggregation method
curl -s http://localhost:9092/metrics | grep ensemble_aggregation_latency_microseconds_bucket

# 5. Check for contention in model inference
ps aux | grep -E "trading_service|ml_inference"
```

#### Response Actions

**IMMEDIATE (0-5 minutes)**:
1. Check CPU/memory utilization on trading service host
2. Review concurrent prediction queue depth
3. Check for resource contention

**SHORT-TERM (5-15 minutes)**:
1. Scale to additional instances if CPU >80%
2. Review model inference queue depth
3. Check for I/O bottlenecks (disk, network)

**LONG-TERM (15+ minutes)**:
1. Profile ensemble aggregation code
2. Optimize hot paths
3. Consider GPU acceleration for large models
4. Review model sizes (TFT at 1.5-2.5GB may cause memory pressure)

#### Resolution Criteria
- P99 latency <50μs for 10+ minutes
- CPU utilization <70%
- No resource contention

---

## 4. Model Failures & Cascades

### 4.1 EnsembleModelFailureDetected

**Alert**: Single model failed to load or predict
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 10 minutes

#### Symptoms
- Model not producing predictions
- Checkpoint swap failed
- Ensemble operating with reduced capacity

#### Investigation Steps

```bash
# 1. Identify failed model
curl -s http://localhost:9092/metrics | grep checkpoint_swaps_total

# 2. Check trading service logs
journalctl -u trading_service -n 100 | grep -i "error\|failed\|panic"

# 3. Verify checkpoint file integrity in MinIO
mc ls minio/checkpoints/ | grep -E "DQN|PPO|MAMBA|TFT|TLOB"

# 4. Check disk space and memory
df -h
free -h

# 5. Attempt manual checkpoint reload
curl -X POST http://localhost:50052/admin/reload_checkpoint \
  -H "Content-Type: application/json" \
  -d '{"model_id": "DQN", "checkpoint_path": "/path/to/checkpoint.pt"}'
```

#### Response Actions

**IMMEDIATE (0-5 minutes)**:
1. Identify failed model from logs
2. Check error message for root cause:
   - File not found → verify MinIO path
   - Out of memory → check available RAM
   - Corrupted file → download fresh copy
   - CUDA error → check GPU status

**SHORT-TERM (5-15 minutes)**:
1. Attempt checkpoint reload
2. If reload fails → remove model from ensemble temporarily
3. Verify remaining models operational

**LONG-TERM (15+ minutes)**:
1. If issue persists → escalate to ML team
2. Review checkpoint training quality
3. Consider reverting to previous checkpoint

#### Resolution Criteria
- Model successfully loaded and predicting
- Checkpoint file integrity verified
- No errors in logs

---

### 4.2 EnsembleCascadeFailureDetected

**Alert**: Multiple models (2+) failed simultaneously
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 5 minutes

#### Symptoms
- 2+ models not producing predictions
- Ensemble integrity compromised
- Predictions unreliable

#### Investigation Steps

```bash
# 1. Count failed models
curl -s http://localhost:9092/metrics | grep ensemble_predictions_total | awk '{if ($2 == 0) print $0}'

# 2. Check trading service health
grpc_health_probe -addr=localhost:50052

# 3. Check infrastructure
nvidia-smi  # GPU health
free -h     # Memory availability
df -h       # Disk space
ip link     # Network connectivity

# 4. Check MinIO accessibility
mc ls minio/checkpoints/

# 5. Review trading service logs for common failure cause
journalctl -u trading_service -n 200 | grep -i "error\|failed\|panic" | sort | uniq -c
```

#### Response Actions

**IMMEDIATE (0-1 minute)**:
1. **HALT ALL AUTOMATED TRADING** - immediate stop
2. Switch to manual trading mode
3. Page on-call ML engineer immediately

**SHORT-TERM (1-5 minutes)**:
1. Review logs for common failure cause:
   - GPU failure → check nvidia-smi, restart service
   - Memory exhaustion → free memory, restart service
   - Network issues → check MinIO connectivity
   - Disk full → clean logs, free space
2. Attempt service restart if root cause identified

**ESCALATION**:
- Do NOT resume automated trading until:
  - >=4 models operational
  - Root cause identified and resolved
  - ML team provides go/no-go approval

#### Resolution Criteria
- >=4 models producing predictions
- All models passing health checks
- Root cause documented
- ML team approval for resuming trading

---

### 4.3 EnsembleCheckpointRollbackRateHigh

**Alert**: Checkpoint rollback rate >10% over 10 minutes
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 30 minutes

#### Symptoms
- New checkpoints failing canary validation
- Rollback rate >10%
- Model quality issues

#### Investigation Steps

```bash
# 1. Check rollback rate
curl -s http://localhost:9092/metrics | grep checkpoint_swaps_total

# 2. Review canary validation failure reasons
psql $DATABASE_URL -c "
SELECT
    model_id,
    checkpoint_version,
    validation_status,
    failure_reason,
    timestamp
FROM checkpoint_validations
WHERE validation_status = 'failed'
    AND timestamp > NOW() - INTERVAL '10 minutes'
ORDER BY timestamp DESC;
"

# 3. Review checkpoint training quality metrics
psql $DATABASE_URL -c "
SELECT
    model_id,
    epoch,
    train_loss,
    val_loss,
    sharpe_ratio,
    win_rate,
    timestamp
FROM model_training_metrics
WHERE timestamp > NOW() - INTERVAL '24 hours'
ORDER BY timestamp DESC
LIMIT 50;
"

# 4. Check training data quality
# Review for anomalies, missing data, distribution shifts
```

#### Response Actions

**IMMEDIATE (0-10 minutes)**:
1. Pause automated checkpoint updates
2. Review recent training quality metrics
3. Identify pattern in rollback failures

**SHORT-TERM (10-30 minutes)**:
1. Investigate training data quality issues
2. Review hyperparameter tuning results
3. Check for overfitting (train loss << val loss)

**LONG-TERM (30+ minutes)**:
1. If training data issue → fix data pipeline
2. If hyperparameter issue → retune models
3. If overfitting → add regularization, reduce model complexity
4. Escalate to ML training team

#### Resolution Criteria
- Rollback rate <5% for 1+ hour
- New checkpoints passing canary validation
- Training quality metrics within acceptable range

---

## 5. Model Weight Anomalies

### 5.1 EnsembleSingleModelDominance

**Alert**: Single model weight >70% for 10 minutes
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 20 minutes

#### Symptoms
- Single model has >70% weight
- Ensemble diversity lost
- Effectively operating as single model

#### Investigation Steps

```bash
# 1. Check model weights
curl -s http://localhost:9092/metrics | grep ensemble_model_weight

# 2. Review P&L attribution
psql $DATABASE_URL -c "
SELECT
    model_id,
    SUM(pnl_contribution_dollars) as total_pnl,
    COUNT(*) as prediction_count,
    AVG(confidence) as avg_confidence
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '1 hour'
GROUP BY model_id
ORDER BY total_pnl DESC;
"

# 3. Check if other models are producing predictions
curl -s http://localhost:9092/metrics | grep ensemble_predictions_total

# 4. Review why other models have low weights
# Check for silent failures, low confidence, poor performance
```

#### Response Actions

**IMMEDIATE (0-10 minutes)**:
1. Verify other models are producing valid predictions
2. Check P&L attribution - is dominant model actually best performer?
3. Review logs for silent model failures

**SHORT-TERM (10-20 minutes)**:
1. If other models underperforming → may be legitimate dominance
2. If other models failing silently → investigate failures
3. Consider manual weight rebalancing if needed

**LONG-TERM (20+ minutes)**:
1. Review ensemble weight update algorithm
2. Check for bias in weight calculation
3. Consider minimum weight thresholds (e.g., no model <10%)

#### Resolution Criteria
- No single model >60% weight
- All models contributing valid predictions
- Weight distribution reflects performance differences

---

### 5.2 EnsembleModelNegativePnLContribution

**Alert**: Model contributing negative P&L over 1 hour
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 10 minutes

#### Symptoms
- Model P&L <-$500 over 1 hour
- Model actively losing money
- Dragging down ensemble performance

#### Investigation Steps

```bash
# 1. Check P&L by model
curl -s http://localhost:9092/metrics | grep ensemble_model_pnl_contribution_dollars

# 2. Review model predictions
psql $DATABASE_URL -c "
SELECT
    predicted_action,
    actual_outcome,
    pnl_contribution_dollars,
    confidence,
    timestamp
FROM ensemble_model_predictions
WHERE model_id = 'DQN'  -- Replace with affected model
    AND timestamp > NOW() - INTERVAL '1 hour'
ORDER BY pnl_contribution_dollars
LIMIT 50;
"

# 3. Check for model drift
# Compare prediction distribution to training distribution

# 4. Review recent checkpoint updates
psql $DATABASE_URL -c "
SELECT
    checkpoint_version,
    loaded_at,
    validation_status
FROM checkpoint_validations
WHERE model_id = 'DQN'  -- Replace with affected model
ORDER BY loaded_at DESC
LIMIT 10;
"
```

#### Response Actions

**IMMEDIATE (0-5 minutes)**:
1. Reduce weight for affected model to 0 temporarily
2. Review model confidence scores (low confidence may indicate uncertainty)
3. Check if issue is symbol-specific or system-wide

**SHORT-TERM (5-15 minutes)**:
1. Review model training quality
2. Check for data distribution shift
3. Analyze prediction patterns for anomalies

**LONG-TERM (15+ minutes)**:
1. If model drift → retrain on recent data
2. If checkpoint issue → revert to previous checkpoint
3. If systematic issue → remove model from ensemble until fixed

#### Resolution Criteria
- Model P&L positive or neutral for 2+ hours
- Prediction patterns align with market direction
- No evidence of model drift

---

## 6. A/B Testing

### 6.1 EnsembleABTestImbalance

**Alert**: A/B test assignment >55/45 split for 10 minutes
**Severity**: WARNING
**PagerDuty**: NO
**MTTR**: 30 minutes

#### Symptoms
- Treatment/control imbalance
- Statistical validity compromised
- A/B test results may be biased

#### Investigation Steps

```bash
# 1. Check assignment distribution
curl -s http://localhost:9092/metrics | grep ab_test_assignments_total

# 2. Calculate exact split
psql $DATABASE_URL -c "
SELECT
    test_id,
    assignment_group,
    COUNT(*) as assignment_count,
    100.0 * COUNT(*) / SUM(COUNT(*)) OVER (PARTITION BY test_id) as percentage
FROM ab_test_assignments
WHERE test_id = 'test-001'  -- Replace with affected test
    AND timestamp > NOW() - INTERVAL '1 hour'
GROUP BY test_id, assignment_group;
"

# 3. Review user assignment hashing algorithm
# Check for bias in hash function or traffic sources
```

#### Response Actions

**IMMEDIATE (0-10 minutes)**:
1. Review user assignment logic
2. Check for bias in traffic sources
3. Verify random seed is properly set

**SHORT-TERM (10-30 minutes)**:
1. If bias in assignment → fix hashing algorithm
2. If bias in traffic → adjust sampling strategy
3. Consider restarting A/B test with corrected assignment

**LONG-TERM (30+ minutes)**:
1. Review A/B test framework code
2. Add monitoring for assignment balance
3. Implement automatic alerts for imbalance >52/48

#### Resolution Criteria
- Assignment split within 48-52% range
- No bias in hashing algorithm
- Traffic sources balanced

---

### 6.2 EnsembleABTestNegativeImpact

**Alert**: Treatment performing >15% worse than control for 30 minutes
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 5 minutes

#### Symptoms
- Treatment Sharpe ratio <-15% vs control
- Experimental model/config actively degrading performance
- Potential capital loss

#### Investigation Steps

```bash
# 1. Check metric differences
curl -s http://localhost:9092/metrics | grep ab_test_metric_difference

# 2. Review treatment vs control performance
psql $DATABASE_URL -c "
SELECT
    assignment_group,
    AVG(sharpe_ratio) as avg_sharpe,
    AVG(win_rate) as avg_win_rate,
    SUM(pnl) as total_pnl,
    COUNT(*) as trade_count
FROM ab_test_results
WHERE test_id = 'test-001'  -- Replace with affected test
    AND timestamp > NOW() - INTERVAL '30 minutes'
GROUP BY assignment_group;
"

# 3. Identify what changed in treatment
# Review experiment configuration, model checkpoints, hyperparameters
```

#### Response Actions

**IMMEDIATE (0-5 minutes)**:
1. **STOP A/B TEST IMMEDIATELY**
2. Revert all treatment group users to control
3. Review what changed in treatment configuration

**SHORT-TERM (5-15 minutes)**:
1. Document treatment configuration
2. Analyze why treatment underperformed
3. Review treatment predictions vs outcomes

**ESCALATION**:
- Do NOT roll out treatment variant
- Investigate root cause before retrying experiment
- Prepare incident report for ML team

#### Resolution Criteria
- All users reverted to control
- Root cause identified
- Experiment redesigned based on learnings

---

## 7. System Health

### 7.1 EnsembleServiceMemoryHigh

**Alert**: Trading service memory >85% for 2 minutes
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 10 minutes

#### Symptoms
- Memory usage >85%
- Risk of OOM kill
- Service instability

#### Investigation Steps

```bash
# 1. Check current memory usage
free -h
ps aux --sort=-%mem | head -10

# 2. Check trading service memory
ps aux | grep trading_service

# 3. Check loaded model sizes
curl -s http://localhost:50052/admin/model_info

# 4. Check for memory leaks
valgrind --leak-check=full cargo run -p trading_service --release

# 5. Review memory growth over time
# Check Grafana memory usage dashboard
```

#### Response Actions

**IMMEDIATE (0-5 minutes)**:
1. Check for memory leaks in ensemble coordinator
2. Review loaded model sizes:
   - DQN: ~75KB (OK)
   - PPO: ~42KB (OK)
   - MAMBA-2: 150-500MB (check if loaded)
   - TFT: 1.5-2.5GB (major consumer)
   - TLOB: ~50MB (OK)
3. Check for resource leaks (file handles, sockets)

**SHORT-TERM (5-15 minutes)**:
1. Consider restarting service during low-traffic window
2. Review memory allocation patterns
3. Check for unbounded caches or queues

**LONG-TERM (15+ minutes)**:
1. Profile memory usage with valgrind
2. Optimize model loading (lazy loading, memory-mapped files)
3. Scale to larger instance if needed (TFT requires significant RAM)

#### Resolution Criteria
- Memory usage <70%
- No memory leaks detected
- Service stable

---

### 7.2 EnsembleGPUInferenceFailure

**Alert**: GPU inference errors >1/sec for 5 minutes
**Severity**: CRITICAL
**PagerDuty**: YES
**MTTR**: 15 minutes

#### Symptoms
- GPU inference errors
- GPU-accelerated models (MAMBA-2, TFT) may fail
- Falling back to CPU (10-50x slower)

#### Investigation Steps

```bash
# 1. Check GPU health
nvidia-smi

# 2. Check GPU memory
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

# 3. Review CUDA errors
journalctl -u trading_service -n 200 | grep -i "cuda\|gpu"

# 4. Check GPU driver
cat /proc/driver/nvidia/version

# 5. Test GPU with simple operation
python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.device_count())"
```

#### Response Actions

**IMMEDIATE (0-5 minutes)**:
1. Check nvidia-smi output
2. Review CUDA error messages
3. Verify GPU not thermal throttling

**SHORT-TERM (5-15 minutes)**:
1. If GPU memory exhausted → restart service
2. If driver issue → reload nvidia driver: `sudo modprobe -r nvidia && sudo modprobe nvidia`
3. If thermal throttling → check cooling, reduce inference batch size

**LONG-TERM (15+ minutes)**:
1. If persistent GPU issues → disable GPU inference temporarily
2. Scale to larger GPU (RTX 3050 Ti → A100)
3. Optimize GPU memory usage (smaller batch sizes, model quantization)

#### Resolution Criteria
- GPU errors <0.1/minute
- GPU temperature <80°C
- All models running on GPU successfully

---

## Emergency Contacts

### On-Call Rotation

| Role | Primary | Backup |
|------|---------|--------|
| ML Engineer | @ml-oncall | @ml-team |
| Risk Manager | @risk-oncall | @risk-team |
| DevOps | @devops-oncall | @devops-team |
| CTO | @cto | - |

### Escalation Matrix

| Severity | Response Time | Escalate After |
|----------|---------------|----------------|
| CRITICAL | 5 minutes | 15 minutes |
| WARNING | 30 minutes | 2 hours |
| INFO | Next business day | - |

### Slack Channels

- **#foxhunt-ensemble-critical**: Critical alerts (PagerDuty)
- **#foxhunt-ensemble-warnings**: Warning alerts
- **#foxhunt-ensemble-info**: Info alerts (A/B tests)
- **#foxhunt-oncall**: On-call coordination

### PagerDuty Integrations

- **Ensemble Critical**: Integration key in alertmanager.yml
- **General Critical**: Integration key in alertmanager.yml

---

## Testing Alert Firing

### Simulate High Disagreement

```bash
# Generate test predictions with high disagreement
curl -X POST http://localhost:50052/admin/test_high_disagreement \
  -H "Content-Type: application/json" \
  -d '{"symbol": "ES.FUT", "duration_seconds": 300}'

# Expected: EnsembleHighDisagreementCritical fires within 5 minutes
```

### Simulate Model Failure

```bash
# Fail a specific model
curl -X POST http://localhost:50052/admin/fail_model \
  -H "Content-Type: application/json" \
  -d '{"model_id": "DQN"}'

# Expected: EnsembleModelFailureDetected fires immediately
```

### Simulate Cascade Failure

```bash
# Fail multiple models
curl -X POST http://localhost:50052/admin/fail_models \
  -H "Content-Type: application/json" \
  -d '{"model_ids": ["DQN", "PPO", "MAMBA-2"]}'

# Expected: EnsembleCascadeFailureDetected fires within 30 seconds
```

### Simulate Latency Spike

```bash
# Inject artificial latency
curl -X POST http://localhost:50052/admin/inject_latency \
  -H "Content-Type: application/json" \
  -d '{"latency_us": 75, "duration_seconds": 120}'

# Expected: EnsembleAggregationLatencyP99High fires within 1 minute
```

### Simulate Sharpe Ratio Drop

```bash
# Generate losing predictions
curl -X POST http://localhost:50052/admin/test_sharpe_drop \
  -H "Content-Type: application/json" \
  -d '{"symbol": "ES.FUT", "drop_percentage": 60, "duration_seconds": 900}'

# Expected: EnsembleSharpeRatioDropCritical fires within 15 minutes
```

---

## Verification Checklist

After deploying alert configuration:

- [ ] Prometheus reloaded with new alert rules
- [ ] AlertManager reloaded with new receivers
- [ ] PagerDuty integration keys configured
- [ ] Slack webhooks configured (3 channels)
- [ ] Test alerts fired successfully (5 test scenarios)
- [ ] Runbook URLs accessible
- [ ] Grafana dashboards linked to alerts
- [ ] On-call rotation configured in PagerDuty
- [ ] Escalation policies defined
- [ ] Team trained on runbook procedures

---

**Last Updated**: 2025-10-14
**Status**: Production Ready
**Alert Count**: 22 rules across 7 categories
**MTTR Target**: <15 minutes for critical alerts