foxhunt/docs/monitoring/ENSEMBLE_ALERT_RUNBOOKS.md

Ensemble Alert Runbooks

Operational Response Procedures

Document Version: 1.0 Date: 2025-10-14 System: Foxhunt HFT Trading System - 6-Model Ensemble Alert Count: 22 alert rules across 7 categories

---

Table of Contents

1. Performance Degradation
2. Model Disagreement & Uncertainty
3. Latency & Performance
4. Model Failures & Cascades
5. Model Weight Anomalies
6. A/B Testing
7. System Health

---

1. Performance Degradation

1.1 EnsembleSharpeRatioDropCritical

Alert: Sharpe ratio dropped >50% in 15 minutes Severity: CRITICAL PagerDuty: YES MTTR: 5 minutes

Symptoms

• Sharpe ratio <50% of baseline
• Trading strategy profitability severely degraded
• May indicate regime shift or model drift

Investigation Steps

# 1. Check current Sharpe ratio
curl -s http://localhost:9092/metrics | grep ensemble_model_pnl_contribution

# 2. Open Grafana dashboard
# Navigate to: http://localhost:3000/d/ensemble-ml-production

# 3. Check model disagreement rates
curl -s http://localhost:9092/metrics | grep ensemble_disagreement_rate

# 4. Review model weights (any single model dominating?)
curl -s http://localhost:9092/metrics | grep ensemble_model_weight

# 5. Check for market regime changes
# Review: VIX spike, news events, market hours (pre-market/after-hours)

# 6. Review recent P&L by model
psql $DATABASE_URL -c "
SELECT
    model_id,
    symbol,
    SUM(pnl_contribution_dollars) as total_pnl,
    COUNT(*) as prediction_count,
    AVG(confidence) as avg_confidence
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '1 hour'
GROUP BY model_id, symbol
ORDER BY total_pnl;
"

Response Actions

IMMEDIATE (0-5 minutes):

1. Reduce position sizes by 50% across all symbols
2. Tighten stop-losses to -1% (from -2%)
3. Review all open positions - close any with confidence <0.6

SHORT-TERM (5-15 minutes):

1. If any model contributing negative P&L >$500/hour → set weight to 0
2. Switch to defensive trading mode (reduced frequency, higher confidence threshold)
3. Review model predictions vs actual market moves (are predictions accurate?)

ESCALATION:

• If Sharpe ratio doesn't recover within 30 minutes → Page ML team
• If drawdown >10% → HALT automated trading immediately
• If issue persists >1 hour → Switch to manual trading or single best model

Resolution Criteria

• Sharpe ratio returns to >75% of baseline for 30+ minutes
• No single model dominating (max weight <60%)
• Disagreement rate <50%

---

1.2 EnsembleWinRateCollapse

Alert: Win rate collapsed below 40% Severity: CRITICAL PagerDuty: YES MTTR: 2 minutes

Symptoms

• Win rate <40% over 15-minute window
• Strategy actively losing money
• Immediate intervention required

Investigation Steps

# 1. Check current win rate by action
curl -s http://localhost:9092/metrics | grep ensemble_predictions_total

# 2. Calculate win rate
# Win rate = (winning trades) / (total trades)

# 3. Review P&L attribution by model
psql $DATABASE_URL -c "
SELECT
    model_id,
    COUNT(CASE WHEN pnl_contribution_dollars > 0 THEN 1 END) as winning_predictions,
    COUNT(*) as total_predictions,
    100.0 * COUNT(CASE WHEN pnl_contribution_dollars > 0 THEN 1 END) / COUNT(*) as win_rate_pct
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '15 minutes'
GROUP BY model_id
ORDER BY win_rate_pct;
"

# 4. Identify losing models
# Any model with win rate <40% should have weight reduced to 0

Response Actions

IMMEDIATE (0-2 minutes):

1. STOP ALL NEW POSITION ENTRIES - close entry logic
2. Review all open positions - set stop-losses to current price +/- 0.5%
3. Identify models with negative P&L → set weight to 0 immediately

SHORT-TERM (2-10 minutes):

1. Close all positions with unrealized loss >-2%
2. Switch to single best-performing model (highest win rate + positive P&L)
3. Reduce trading frequency by 75%

ESCALATION:

• If drawdown >10% → HALT all automated trading immediately
• Page ML team and risk manager
• Prepare incident report

Resolution Criteria

• Win rate returns to >50% for 30+ minutes
• P&L trend positive for 1+ hour
• All models contributing positive P&L

---

1.3 EnsembleNegativePnLTrend

Alert: Consistent negative P&L over 30 minutes Severity: CRITICAL PagerDuty: YES MTTR: 1 minute

Symptoms

• 30-minute P&L <-$1000
• Active capital loss
• Trading system hemorrhaging money

Investigation Steps

# 1. Check P&L by model
curl -s http://localhost:9092/metrics | grep ensemble_model_pnl_contribution_dollars

# 2. Review all open positions
psql $DATABASE_URL -c "
SELECT
    symbol,
    COUNT(*) as position_count,
    SUM(quantity) as net_position,
    SUM(unrealized_pnl) as total_unrealized_pnl
FROM positions
WHERE status = 'open'
GROUP BY symbol;
"

# 3. Identify losing models
psql $DATABASE_URL -c "
SELECT
    model_id,
    SUM(pnl_contribution_dollars) as total_pnl
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '30 minutes'
GROUP BY model_id
HAVING SUM(pnl_contribution_dollars) < -500
ORDER BY total_pnl;
"

Response Actions

IMMEDIATE (0-1 minute):

1. HALT ALL AUTOMATED TRADING - stop order submission
2. Close all positions with unrealized loss
3. Set all model weights to 0 except best performer

SHORT-TERM (1-5 minutes):

1. Review trading logs for errors
2. Check for broker connectivity issues
3. Verify market data feed is current (not stale)

ESCALATION:

• Page ML team, risk manager, and CTO immediately
• Prepare detailed incident report with:
  • Total P&L loss
  • Models contributing to loss
  • Market conditions during loss period
  • Root cause analysis

Resolution Criteria

• Automated trading halted
• All losing positions closed
• Root cause identified and documented
• ML team provides go/no-go decision for resuming trading

---

2. Model Disagreement & Uncertainty

2.1 EnsembleHighDisagreementCritical

Alert: Model disagreement >70% for 5 minutes Severity: CRITICAL PagerDuty: YES MTTR: 10 minutes

Symptoms

• Models fundamentally disagree on predictions
• Disagreement rate >70% sustained
• Prediction reliability compromised

Investigation Steps

# 1. Check disagreement rate by symbol
curl -s http://localhost:9092/metrics | grep ensemble_disagreement_rate

# 2. Review individual model predictions
psql $DATABASE_URL -c "
SELECT
    model_id,
    symbol,
    predicted_action,
    confidence,
    timestamp
FROM ensemble_model_predictions
WHERE timestamp > NOW() - INTERVAL '5 minutes'
    AND symbol = 'ES.FUT'  -- Replace with affected symbol
ORDER BY timestamp DESC
LIMIT 50;
"

# 3. Check for market regime shift
# Indicators: VIX spike, volatility increase, news events

# 4. Review model confidence scores
curl -s http://localhost:9092/metrics | grep ensemble_confidence_score

Response Actions

IMMEDIATE (0-5 minutes):

1. Reduce position sizes by 50% for affected symbol
2. Increase confidence threshold from 0.7 to 0.85
3. Stop opening new positions for affected symbol

SHORT-TERM (5-15 minutes):

1. If disagreement persists >15 minutes → pause trading for symbol
2. Review recent news/events for symbol
3. Check if disagreement is symbol-specific or system-wide

LONG-TERM (15+ minutes):

1. If system-wide → may indicate market regime change
2. Consider retraining models on recent data
3. Review model weights - may need manual rebalancing

Resolution Criteria

• Disagreement rate <50% for 15+ minutes
• Model confidence >0.7
• Predictions aligned with market direction

---

2.2 EnsembleLowConfidenceHighDisagreement

Alert: Low confidence (<0.6) AND high disagreement (>0.7) Severity: CRITICAL PagerDuty: YES MTTR: 2 minutes

Symptoms

• Models uncertain AND disagreeing
• Worst-case scenario for prediction reliability
• Prediction quality at minimum

Investigation Steps

# 1. Check both metrics
curl -s http://localhost:9092/metrics | grep -E "ensemble_confidence_score|ensemble_disagreement_rate"

# 2. Review what models are predicting
psql $DATABASE_URL -c "
SELECT
    model_id,
    predicted_action,
    confidence,
    COUNT(*) as prediction_count
FROM ensemble_model_predictions
WHERE timestamp > NOW() - INTERVAL '2 minutes'
    AND symbol = 'ES.FUT'  -- Replace with affected symbol
GROUP BY model_id, predicted_action, confidence
ORDER BY model_id;
"

Response Actions

IMMEDIATE (0-2 minutes):

1. HALT ALL TRADING FOR AFFECTED SYMBOL - immediate stop
2. Switch to observation mode (log predictions without execution)
3. Close all open positions for affected symbol with tight stop-losses

SHORT-TERM (2-10 minutes):

1. Manually review market conditions
2. Check for:
   • Flash crash
   • News events
   • Exchange issues
   • Data feed problems

ESCALATION:

• Do NOT resume automated trading until:
  • Confidence >0.7 AND disagreement <0.5 for 15+ minutes
  • Manual review confirms market stability
  • ML team provides approval

Resolution Criteria

• Confidence >0.7
• Disagreement <0.5
• Manual approval from ML team

---

3. Latency & Performance

3.1 EnsembleAggregationLatencyP99High

Alert: Aggregation P99 latency >50μs for 1 minute Severity: CRITICAL PagerDuty: YES MTTR: 15 minutes

Symptoms

• P99 latency >50μs (SLA violation)
• HFT execution delays
• Potential alpha decay and slippage

Investigation Steps

# 1. Check current latency
curl -s http://localhost:9092/metrics | grep ensemble_aggregation_latency_microseconds

# 2. Check CPU utilization
top -n 1 | grep trading_service

# 3. Check concurrent prediction load
curl -s http://localhost:9092/metrics | grep ensemble_predictions_total | tail -5

# 4. Review latency breakdown by aggregation method
curl -s http://localhost:9092/metrics | grep ensemble_aggregation_latency_microseconds_bucket

# 5. Check for contention in model inference
ps aux | grep -E "trading_service|ml_inference"

Response Actions

IMMEDIATE (0-5 minutes):

1. Check CPU/memory utilization on trading service host
2. Review concurrent prediction queue depth
3. Check for resource contention

SHORT-TERM (5-15 minutes):

1. Scale to additional instances if CPU >80%
2. Review model inference queue depth
3. Check for I/O bottlenecks (disk, network)

LONG-TERM (15+ minutes):

1. Profile ensemble aggregation code
2. Optimize hot paths
3. Consider GPU acceleration for large models
4. Review model sizes (TFT at 1.5-2.5GB may cause memory pressure)

Resolution Criteria

• P99 latency <50μs for 10+ minutes
• CPU utilization <70%
• No resource contention

---

4. Model Failures & Cascades

4.1 EnsembleModelFailureDetected

Alert: Single model failed to load or predict Severity: CRITICAL PagerDuty: YES MTTR: 10 minutes

Symptoms

• Model not producing predictions
• Checkpoint swap failed
• Ensemble operating with reduced capacity

Investigation Steps

# 1. Identify failed model
curl -s http://localhost:9092/metrics | grep checkpoint_swaps_total

# 2. Check trading service logs
journalctl -u trading_service -n 100 | grep -i "error\|failed\|panic"

# 3. Verify checkpoint file integrity in MinIO
mc ls minio/checkpoints/ | grep -E "DQN|PPO|MAMBA|TFT|TLOB"

# 4. Check disk space and memory
df -h
free -h

# 5. Attempt manual checkpoint reload
curl -X POST http://localhost:50052/admin/reload_checkpoint \
  -H "Content-Type: application/json" \
  -d '{"model_id": "DQN", "checkpoint_path": "/path/to/checkpoint.pt"}'

Response Actions

IMMEDIATE (0-5 minutes):

1. Identify failed model from logs
2. Check error message for root cause:
   • File not found → verify MinIO path
   • Out of memory → check available RAM
   • Corrupted file → download fresh copy
   • CUDA error → check GPU status

SHORT-TERM (5-15 minutes):

1. Attempt checkpoint reload
2. If reload fails → remove model from ensemble temporarily
3. Verify remaining models operational

LONG-TERM (15+ minutes):

1. If issue persists → escalate to ML team
2. Review checkpoint training quality
3. Consider reverting to previous checkpoint

Resolution Criteria

• Model successfully loaded and predicting
• Checkpoint file integrity verified
• No errors in logs

---

4.2 EnsembleCascadeFailureDetected

Alert: Multiple models (2+) failed simultaneously Severity: CRITICAL PagerDuty: YES MTTR: 5 minutes

Symptoms

• 2+ models not producing predictions
• Ensemble integrity compromised
• Predictions unreliable

Investigation Steps

# 1. Count failed models
curl -s http://localhost:9092/metrics | grep ensemble_predictions_total | awk '{if ($2 == 0) print $0}'

# 2. Check trading service health
grpc_health_probe -addr=localhost:50052

# 3. Check infrastructure
nvidia-smi  # GPU health
free -h     # Memory availability
df -h       # Disk space
ip link     # Network connectivity

# 4. Check MinIO accessibility
mc ls minio/checkpoints/

# 5. Review trading service logs for common failure cause
journalctl -u trading_service -n 200 | grep -i "error\|failed\|panic" | sort | uniq -c

Response Actions

IMMEDIATE (0-1 minute):

1. HALT ALL AUTOMATED TRADING - immediate stop
2. Switch to manual trading mode
3. Page on-call ML engineer immediately

SHORT-TERM (1-5 minutes):

1. Review logs for common failure cause:
   • GPU failure → check nvidia-smi, restart service
   • Memory exhaustion → free memory, restart service
   • Network issues → check MinIO connectivity
   • Disk full → clean logs, free space
2. Attempt service restart if root cause identified

ESCALATION:

• Do NOT resume automated trading until:

  • =4 models operational

  • Root cause identified and resolved
  • ML team provides go/no-go approval

Resolution Criteria

• =4 models producing predictions

• All models passing health checks
• Root cause documented
• ML team approval for resuming trading

---

4.3 EnsembleCheckpointRollbackRateHigh

Alert: Checkpoint rollback rate >10% over 10 minutes Severity: CRITICAL PagerDuty: YES MTTR: 30 minutes

Symptoms

• New checkpoints failing canary validation
• Rollback rate >10%
• Model quality issues

Investigation Steps

# 1. Check rollback rate
curl -s http://localhost:9092/metrics | grep checkpoint_swaps_total

# 2. Review canary validation failure reasons
psql $DATABASE_URL -c "
SELECT
    model_id,
    checkpoint_version,
    validation_status,
    failure_reason,
    timestamp
FROM checkpoint_validations
WHERE validation_status = 'failed'
    AND timestamp > NOW() - INTERVAL '10 minutes'
ORDER BY timestamp DESC;
"

# 3. Review checkpoint training quality metrics
psql $DATABASE_URL -c "
SELECT
    model_id,
    epoch,
    train_loss,
    val_loss,
    sharpe_ratio,
    win_rate,
    timestamp
FROM model_training_metrics
WHERE timestamp > NOW() - INTERVAL '24 hours'
ORDER BY timestamp DESC
LIMIT 50;
"

# 4. Check training data quality
# Review for anomalies, missing data, distribution shifts

Response Actions

IMMEDIATE (0-10 minutes):

1. Pause automated checkpoint updates
2. Review recent training quality metrics
3. Identify pattern in rollback failures

SHORT-TERM (10-30 minutes):

1. Investigate training data quality issues
2. Review hyperparameter tuning results
3. Check for overfitting (train loss << val loss)

LONG-TERM (30+ minutes):

1. If training data issue → fix data pipeline
2. If hyperparameter issue → retune models
3. If overfitting → add regularization, reduce model complexity
4. Escalate to ML training team

Resolution Criteria

• Rollback rate <5% for 1+ hour
• New checkpoints passing canary validation
• Training quality metrics within acceptable range

---

5. Model Weight Anomalies

5.1 EnsembleSingleModelDominance

Alert: Single model weight >70% for 10 minutes Severity: CRITICAL PagerDuty: YES MTTR: 20 minutes

Symptoms

• Single model has >70% weight
• Ensemble diversity lost
• Effectively operating as single model

Investigation Steps

# 1. Check model weights
curl -s http://localhost:9092/metrics | grep ensemble_model_weight

# 2. Review P&L attribution
psql $DATABASE_URL -c "
SELECT
    model_id,
    SUM(pnl_contribution_dollars) as total_pnl,
    COUNT(*) as prediction_count,
    AVG(confidence) as avg_confidence
FROM ensemble_predictions
WHERE timestamp > NOW() - INTERVAL '1 hour'
GROUP BY model_id
ORDER BY total_pnl DESC;
"

# 3. Check if other models are producing predictions
curl -s http://localhost:9092/metrics | grep ensemble_predictions_total

# 4. Review why other models have low weights
# Check for silent failures, low confidence, poor performance

Response Actions

IMMEDIATE (0-10 minutes):

1. Verify other models are producing valid predictions
2. Check P&L attribution - is dominant model actually best performer?
3. Review logs for silent model failures

SHORT-TERM (10-20 minutes):

1. If other models underperforming → may be legitimate dominance
2. If other models failing silently → investigate failures
3. Consider manual weight rebalancing if needed

LONG-TERM (20+ minutes):

1. Review ensemble weight update algorithm
2. Check for bias in weight calculation
3. Consider minimum weight thresholds (e.g., no model <10%)

Resolution Criteria

• No single model >60% weight
• All models contributing valid predictions
• Weight distribution reflects performance differences

---

5.2 EnsembleModelNegativePnLContribution

Alert: Model contributing negative P&L over 1 hour Severity: CRITICAL PagerDuty: YES MTTR: 10 minutes

Symptoms

• Model P&L <-$500 over 1 hour
• Model actively losing money
• Dragging down ensemble performance

Investigation Steps

# 1. Check P&L by model
curl -s http://localhost:9092/metrics | grep ensemble_model_pnl_contribution_dollars

# 2. Review model predictions
psql $DATABASE_URL -c "
SELECT
    predicted_action,
    actual_outcome,
    pnl_contribution_dollars,
    confidence,
    timestamp
FROM ensemble_model_predictions
WHERE model_id = 'DQN'  -- Replace with affected model
    AND timestamp > NOW() - INTERVAL '1 hour'
ORDER BY pnl_contribution_dollars
LIMIT 50;
"

# 3. Check for model drift
# Compare prediction distribution to training distribution

# 4. Review recent checkpoint updates
psql $DATABASE_URL -c "
SELECT
    checkpoint_version,
    loaded_at,
    validation_status
FROM checkpoint_validations
WHERE model_id = 'DQN'  -- Replace with affected model
ORDER BY loaded_at DESC
LIMIT 10;
"

Response Actions

IMMEDIATE (0-5 minutes):

1. Reduce weight for affected model to 0 temporarily
2. Review model confidence scores (low confidence may indicate uncertainty)
3. Check if issue is symbol-specific or system-wide

SHORT-TERM (5-15 minutes):

1. Review model training quality
2. Check for data distribution shift
3. Analyze prediction patterns for anomalies

LONG-TERM (15+ minutes):

1. If model drift → retrain on recent data
2. If checkpoint issue → revert to previous checkpoint
3. If systematic issue → remove model from ensemble until fixed

Resolution Criteria

• Model P&L positive or neutral for 2+ hours
• Prediction patterns align with market direction
• No evidence of model drift

---

6. A/B Testing

6.1 EnsembleABTestImbalance

Alert: A/B test assignment >55/45 split for 10 minutes Severity: WARNING PagerDuty: NO MTTR: 30 minutes

Symptoms

• Treatment/control imbalance
• Statistical validity compromised
• A/B test results may be biased

Investigation Steps

# 1. Check assignment distribution
curl -s http://localhost:9092/metrics | grep ab_test_assignments_total

# 2. Calculate exact split
psql $DATABASE_URL -c "
SELECT
    test_id,
    assignment_group,
    COUNT(*) as assignment_count,
    100.0 * COUNT(*) / SUM(COUNT(*)) OVER (PARTITION BY test_id) as percentage
FROM ab_test_assignments
WHERE test_id = 'test-001'  -- Replace with affected test
    AND timestamp > NOW() - INTERVAL '1 hour'
GROUP BY test_id, assignment_group;
"

# 3. Review user assignment hashing algorithm
# Check for bias in hash function or traffic sources

Response Actions

IMMEDIATE (0-10 minutes):

1. Review user assignment logic
2. Check for bias in traffic sources
3. Verify random seed is properly set

SHORT-TERM (10-30 minutes):

1. If bias in assignment → fix hashing algorithm
2. If bias in traffic → adjust sampling strategy
3. Consider restarting A/B test with corrected assignment

LONG-TERM (30+ minutes):

1. Review A/B test framework code
2. Add monitoring for assignment balance
3. Implement automatic alerts for imbalance >52/48

Resolution Criteria

• Assignment split within 48-52% range
• No bias in hashing algorithm
• Traffic sources balanced

---

6.2 EnsembleABTestNegativeImpact

Alert: Treatment performing >15% worse than control for 30 minutes Severity: CRITICAL PagerDuty: YES MTTR: 5 minutes

Symptoms

• Treatment Sharpe ratio <-15% vs control
• Experimental model/config actively degrading performance
• Potential capital loss

Investigation Steps

# 1. Check metric differences
curl -s http://localhost:9092/metrics | grep ab_test_metric_difference

# 2. Review treatment vs control performance
psql $DATABASE_URL -c "
SELECT
    assignment_group,
    AVG(sharpe_ratio) as avg_sharpe,
    AVG(win_rate) as avg_win_rate,
    SUM(pnl) as total_pnl,
    COUNT(*) as trade_count
FROM ab_test_results
WHERE test_id = 'test-001'  -- Replace with affected test
    AND timestamp > NOW() - INTERVAL '30 minutes'
GROUP BY assignment_group;
"

# 3. Identify what changed in treatment
# Review experiment configuration, model checkpoints, hyperparameters

Response Actions

IMMEDIATE (0-5 minutes):

1. STOP A/B TEST IMMEDIATELY
2. Revert all treatment group users to control
3. Review what changed in treatment configuration

SHORT-TERM (5-15 minutes):

1. Document treatment configuration
2. Analyze why treatment underperformed
3. Review treatment predictions vs outcomes

ESCALATION:

• Do NOT roll out treatment variant
• Investigate root cause before retrying experiment
• Prepare incident report for ML team

Resolution Criteria

• All users reverted to control
• Root cause identified
• Experiment redesigned based on learnings

---

7. System Health

7.1 EnsembleServiceMemoryHigh

Alert: Trading service memory >85% for 2 minutes Severity: CRITICAL PagerDuty: YES MTTR: 10 minutes

Symptoms

• Memory usage >85%
• Risk of OOM kill
• Service instability

Investigation Steps

# 1. Check current memory usage
free -h
ps aux --sort=-%mem | head -10

# 2. Check trading service memory
ps aux | grep trading_service

# 3. Check loaded model sizes
curl -s http://localhost:50052/admin/model_info

# 4. Check for memory leaks
valgrind --leak-check=full cargo run -p trading_service --release

# 5. Review memory growth over time
# Check Grafana memory usage dashboard

Response Actions

IMMEDIATE (0-5 minutes):

1. Check for memory leaks in ensemble coordinator
2. Review loaded model sizes:
   • DQN: ~75KB (OK)
   • PPO: ~42KB (OK)
   • MAMBA-2: 150-500MB (check if loaded)
   • TFT: 1.5-2.5GB (major consumer)
   • TLOB: ~50MB (OK)
3. Check for resource leaks (file handles, sockets)

SHORT-TERM (5-15 minutes):

1. Consider restarting service during low-traffic window
2. Review memory allocation patterns
3. Check for unbounded caches or queues

LONG-TERM (15+ minutes):

1. Profile memory usage with valgrind
2. Optimize model loading (lazy loading, memory-mapped files)
3. Scale to larger instance if needed (TFT requires significant RAM)

Resolution Criteria

• Memory usage <70%
• No memory leaks detected
• Service stable

---

7.2 EnsembleGPUInferenceFailure

Alert: GPU inference errors >1/sec for 5 minutes Severity: CRITICAL PagerDuty: YES MTTR: 15 minutes

Symptoms

• GPU inference errors
• GPU-accelerated models (MAMBA-2, TFT) may fail
• Falling back to CPU (10-50x slower)

Investigation Steps

# 1. Check GPU health
nvidia-smi

# 2. Check GPU memory
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

# 3. Review CUDA errors
journalctl -u trading_service -n 200 | grep -i "cuda\|gpu"

# 4. Check GPU driver
cat /proc/driver/nvidia/version

# 5. Test GPU with simple operation
python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.device_count())"

Response Actions

IMMEDIATE (0-5 minutes):

1. Check nvidia-smi output
2. Review CUDA error messages
3. Verify GPU not thermal throttling

SHORT-TERM (5-15 minutes):

1. If GPU memory exhausted → restart service
2. If driver issue → reload nvidia driver: sudo modprobe -r nvidia && sudo modprobe nvidia
3. If thermal throttling → check cooling, reduce inference batch size

LONG-TERM (15+ minutes):

1. If persistent GPU issues → disable GPU inference temporarily
2. Scale to larger GPU (RTX 3050 Ti → A100)
3. Optimize GPU memory usage (smaller batch sizes, model quantization)

Resolution Criteria

• GPU errors <0.1/minute
• GPU temperature <80°C
• All models running on GPU successfully

---

Emergency Contacts

On-Call Rotation

Role	Primary	Backup
ML Engineer	@ml-oncall	@ml-team
Risk Manager	@risk-oncall	@risk-team
DevOps	@devops-oncall	@devops-team
CTO	@cto	-

Escalation Matrix

Severity	Response Time	Escalate After
CRITICAL	5 minutes	15 minutes
WARNING	30 minutes	2 hours
INFO	Next business day	-

Slack Channels

• #foxhunt-ensemble-critical: Critical alerts (PagerDuty)
• #foxhunt-ensemble-warnings: Warning alerts
• #foxhunt-ensemble-info: Info alerts (A/B tests)
• #foxhunt-oncall: On-call coordination

PagerDuty Integrations

• Ensemble Critical: Integration key in alertmanager.yml
• General Critical: Integration key in alertmanager.yml

---

Testing Alert Firing

Simulate High Disagreement

# Generate test predictions with high disagreement
curl -X POST http://localhost:50052/admin/test_high_disagreement \
  -H "Content-Type: application/json" \
  -d '{"symbol": "ES.FUT", "duration_seconds": 300}'

# Expected: EnsembleHighDisagreementCritical fires within 5 minutes

Simulate Model Failure

# Fail a specific model
curl -X POST http://localhost:50052/admin/fail_model \
  -H "Content-Type: application/json" \
  -d '{"model_id": "DQN"}'

# Expected: EnsembleModelFailureDetected fires immediately

Simulate Cascade Failure

# Fail multiple models
curl -X POST http://localhost:50052/admin/fail_models \
  -H "Content-Type: application/json" \
  -d '{"model_ids": ["DQN", "PPO", "MAMBA-2"]}'

# Expected: EnsembleCascadeFailureDetected fires within 30 seconds

Simulate Latency Spike

# Inject artificial latency
curl -X POST http://localhost:50052/admin/inject_latency \
  -H "Content-Type: application/json" \
  -d '{"latency_us": 75, "duration_seconds": 120}'

# Expected: EnsembleAggregationLatencyP99High fires within 1 minute

Simulate Sharpe Ratio Drop

# Generate losing predictions
curl -X POST http://localhost:50052/admin/test_sharpe_drop \
  -H "Content-Type: application/json" \
  -d '{"symbol": "ES.FUT", "drop_percentage": 60, "duration_seconds": 900}'

# Expected: EnsembleSharpeRatioDropCritical fires within 15 minutes

---

Verification Checklist

After deploying alert configuration:

• Prometheus reloaded with new alert rules
• AlertManager reloaded with new receivers
• PagerDuty integration keys configured
• Slack webhooks configured (3 channels)
• Test alerts fired successfully (5 test scenarios)
• Runbook URLs accessible
• Grafana dashboards linked to alerts
• On-call rotation configured in PagerDuty
• Escalation policies defined
• Team trained on runbook procedures

---

Last Updated: 2025-10-14 Status: Production Ready Alert Count: 22 rules across 7 categories MTTR Target: <15 minutes for critical alerts