foxhunt/WAVE_D_OPERATIONAL_RUNBOOK.md

# Wave D Operational Runbook

**Version**: 1.0
**Date**: 2025-10-18
**Status**: 🟢 **PRODUCTION READY**
**Purpose**: Incident response guide for Wave D regime detection & adaptive strategies

---

## Table of Contents

1. [Common Issues & Resolutions](#common-issues--resolutions)
2. [Monitoring & Alerting](#monitoring--alerting)
3. [Performance Tuning](#performance-tuning)
4. [Rollback Procedures](#rollback-procedures)
5. [Quick Reference](#quick-reference)

---

## Common Issues & Resolutions

### Issue 1: CUSUM False Positives

**Symptom**:
- Alert: `CUSUMFalsePositiveSpike`
- Grafana: >100 structural breaks per hour
- Logs: Frequent "Structural break detected" messages
- Impact: Regime detection oversensitivity, frequent Crisis regime misclassification

**Root Cause**: CUSUM threshold too low for current market volatility.

**Diagnosis** (5 minutes):

```bash
# 1. Check current break frequency
psql -U foxhunt -d foxhunt -c "
  SELECT COUNT(*) AS break_count
  FROM regime_transitions
  WHERE symbol='ES.FUT'
    AND event_timestamp > NOW() - INTERVAL '1 hour'
    AND cusum_alert_triggered = TRUE;
"

# Expected: <10 breaks/hour
# Observed: >100 breaks/hour → CUSUM too sensitive

# 2. Check current CUSUM parameters
grep -A5 "CUSUM_THRESHOLD\|CUSUM_DRIFT" /opt/foxhunt/config/trading_agent.toml

# Current values:
#   cusum_threshold = 4.0
#   cusum_drift_allowance = 0.5

# 3. Measure current market volatility
tli trade ml adaptive-params --symbol ES.FUT | grep "ATR"

# If ATR >5.0 (high volatility), increase threshold
```

**Resolution Options**:

**Option 1: Increase CUSUM Threshold** (recommended, 10 minutes):
```bash
# Edit configuration
vim /opt/foxhunt/config/trading_agent.toml

# Change:
cusum_threshold = 5.0  # From 4.0 (25% less sensitive)

# Reload configuration
systemctl reload trading_agent_service

# Verify change applied
tail -f /var/log/foxhunt/trading_agent.log | grep "CUSUM threshold"
# Expected: "CUSUM threshold set to 5.0"
```

**Option 2: Increase Drift Allowance** (alternative, 10 minutes):
```bash
# Edit configuration
vim /opt/foxhunt/config/trading_agent.toml

# Change:
cusum_drift_allowance = 0.75  # From 0.5 (50% more tolerance)

# Reload and verify (same as Option 1)
```

**Option 3: Reduce CUSUM Weight in Ensemble** (last resort, 15 minutes):
```bash
# Edit source code
vim /home/jgrusewski/Work/foxhunt/ml/src/features/config.rs

# Change:
pub const REGIME_CLASSIFIER_WEIGHTS: RegimeWeights = RegimeWeights {
    cusum: 0.25,    // From 0.40 (reduce influence)
    trending: 0.35, // From 0.30 (increase influence)
    ranging: 0.25,  // From 0.20
    volatile: 0.15, // From 0.10
};

# Rebuild and deploy
cargo build -p trading_agent_service --release
systemctl restart trading_agent_service
```

**Verification** (30 minutes):
```bash
# Monitor break frequency for 30 minutes
watch -n 60 'psql -U foxhunt -d foxhunt -t -c "SELECT COUNT(*) FROM regime_transitions WHERE symbol='"'"'ES.FUT'"'"' AND event_timestamp > NOW() - INTERVAL '"'"'1 hour'"'"' AND cusum_alert_triggered = TRUE;"'

# Success criteria: <10 breaks per hour
# If still >50 breaks/hour: Apply Option 2 or 3
```

**Expected Behavior After Fix**:
- Break frequency: 5-10 per hour (down from 100+)
- False positive rate: <0.5% (test with `cargo test -p ml --test cusum_test -- test_false_positive_rate`)
- Regime stability: >0.7 (fewer Crisis regime misclassifications)

---

### Issue 2: ADX Initialization Failures

**Symptom**:
- Alert: `ADXInitializationFailure`
- Grafana: ADX stuck at 0.0 despite >10 minutes of data
- Logs: "ADX not initialized, need 28 bars" (repeated)
- Impact: Trend classification unavailable, falling back to CUSUM-only regime detection

**Root Cause**: Insufficient historical data to initialize ADX (requires 28 bars minimum).

**Diagnosis** (3 minutes):

```bash
# 1. Check current bar count
tli trade ml regime-status --symbol ES.FUT | grep "Bar count"

# Expected: Bar count ≥ 28
# Observed: Bar count < 28 → waiting for warmup

# 2. Verify data ingestion rate
psql -U foxhunt -d foxhunt -c "
  SELECT COUNT(*) AS bars_received
  FROM market_data
  WHERE symbol='ES.FUT'
    AND timestamp > NOW() - INTERVAL '10 minutes';
"

# Expected: >10 bars (1-minute intervals)
# Observed: 0 bars → data feed issue
```

**Resolution**:

**Scenario A: Bar count < 28 (waiting for warmup)** (no action required):
```bash
# Calculate remaining wait time
echo "Bars needed: $((28 - $(tli trade ml regime-status --symbol ES.FUT | grep "Bar count" | awk '{print $3}')))"
echo "Wait time: ~$((28 - bar_count)) minutes (assuming 1-minute bars)"

# ADX will auto-initialize when 28 bars accumulated
# Monitor progress:
watch -n 60 'tli trade ml regime-status --symbol ES.FUT | grep "Bar count\|ADX"'
```

**Scenario B: Bar count ≥ 28 but ADX still 0.0** (bug, 20 minutes):
```bash
# 1. Check ADX calculation logic
tail -100 /var/log/foxhunt/trading_agent.log | grep -i "adx"

# Look for errors like:
#   - "ADX calculation failed: division by zero"
#   - "TR array empty, cannot compute ATR"
#   - "DMI initialization failed"

# 2. Restart service to clear state
systemctl restart trading_agent_service

# 3. Wait 28 bars for re-initialization
sleep 1800  # 30 minutes for 28 bars + safety margin

# 4. Verify ADX initialized
tli trade ml regime-status --symbol ES.FUT | grep "ADX"

# Expected: ADX: 15-45 (non-zero)
# If still 0.0: File bug report + revert to Wave C features
```

**Scenario C: Data feed issue** (external dependency, 10 minutes):
```bash
# 1. Check Databento service status
curl -I https://hist.databento.com/v0/health

# Expected: HTTP 200
# Observed: HTTP 503 → Databento outage

# 2. Check local data acquisition service
systemctl status data_acquisition_service

# If inactive: systemctl start data_acquisition_service

# 3. Verify data ingestion resumed
watch -n 60 'psql -U foxhunt -d foxhunt -t -c "SELECT COUNT(*) FROM market_data WHERE symbol='"'"'ES.FUT'"'"' AND timestamp > NOW() - INTERVAL '"'"'5 minutes'"'"';"'

# Expected: Increasing count every minute
```

**Workaround** (if ADX initialization blocked for >1 hour):
```bash
# Temporarily disable ADX features (use CUSUM-only regime detection)
vim /opt/foxhunt/config/trading_agent.toml

# Add:
[features.wave_d]
disable_adx = true  # Fallback to CUSUM + Transition + Adaptive

# Reload
systemctl reload trading_agent_service

# Impact: Reduced regime detection accuracy (no trend strength signal)
```

**Expected Behavior After Fix**:
- ADX initializes within 28-30 bars (28-30 minutes for 1-minute bars)
- ADX value in range [0, 100], typically 15-45 for ES.FUT
- Trending regime detection accuracy improves (ADX >25 = trending)

---

### Issue 3: Transition Matrix Entropy Too High

**Symptom**:
- Alert: `TransitionMatrixEntropyHigh`
- Grafana: Shannon entropy >2.5 (random regime switches)
- Logs: "Regime stability low: 0.35" (expected >0.6)
- Impact: Unpredictable regime transitions, reduced adaptive strategy effectiveness

**Root Cause**: EMA alpha too high (over-reactive to recent transitions) or insufficient regime persistence.

**Diagnosis** (5 minutes):

```bash
# 1. Check current entropy
tli trade ml regime-status --symbol ES.FUT | grep "Entropy"

# Expected: 0.5-1.5 (low to moderate entropy)
# Observed: >2.5 (high entropy, near-random transitions)

# 2. Check transition matrix
psql -U foxhunt -d foxhunt -c "
  SELECT * FROM get_regime_transition_matrix('ES.FUT', 24);
"

# Look for:
#   - Uniform probabilities across all regimes (~12.5% each for 8 regimes)
#   - Low diagonal values (<0.5 = low regime persistence)

# 3. Check EMA alpha parameter
grep "TRANSITION_EMA_ALPHA" /opt/foxhunt/config/trading_agent.toml

# Current: 0.1 (fast adaptation)
# Target: 0.05 (slower adaptation)
```

**Resolution** (10 minutes):

```bash
# Decrease EMA alpha for slower adaptation
vim /opt/foxhunt/config/trading_agent.toml

# Change:
transition_ema_alpha = 0.05  # From 0.1 (50% slower)

# Reload
systemctl reload trading_agent_service

# Verify change applied
tail -f /var/log/foxhunt/trading_agent.log | grep "Transition EMA alpha"
# Expected: "Transition EMA alpha set to 0.05"
```

**Verification** (1 hour):
```bash
# Monitor entropy over 1 hour
watch -n 600 'tli trade ml regime-status --symbol ES.FUT | grep "Entropy"'

# Success criteria:
#   - Entropy drops from >2.5 to <1.5 within 1 hour
#   - Stability increases from <0.4 to >0.6
```

**Alternative Resolution** (if entropy remains high):
```bash
# Increase regime persistence threshold
vim /opt/foxhunt/config/trading_agent.toml

# Add:
[regime.persistence]
min_bars = 10  # From 5 (require 10 bars before allowing transition)

# Rebuild and deploy
cargo build -p trading_agent_service --release
systemctl restart trading_agent_service
```

**Expected Behavior After Fix**:
- Shannon entropy: 0.5-1.5 (structured regime transitions)
- Regime stability: >0.6 (high persistence)
- Transition frequency: 5-15 per day (down from >50 per hour)

---

### Issue 4: Adaptive Position Sizing Too Aggressive

**Symptom**:
- Alert: `PositionSizeMultiplierOutOfRange`
- Grafana: Position multiplier >2.0 (expected max 1.5x)
- Logs: "Position size: $150,000 (multiplier 2.0x)" (exceeds $100,000 baseline)
- Impact: Risk management compromised, potential over-leveraging

**Root Cause**: Trending regime multiplier misconfigured or regime detection too sensitive.

**Diagnosis** (3 minutes):

```bash
# 1. Check current position multiplier
tli trade ml adaptive-params --symbol ES.FUT | grep "Position Multiplier"

# Expected: 0.2-1.5x
# Observed: 2.0x → out of range

# 2. Check current regime
tli trade ml regime-status --symbol ES.FUT | grep "Regime:"

# If "Trending" → check multiplier config
# If "Crisis" or "Normal" → regime detection error

# 3. Check multiplier configuration
grep -A10 "POSITION_MULTIPLIERS" /opt/foxhunt/config/trading_agent.toml
```

**Resolution Option 1: Fix Multiplier Config** (5 minutes):
```bash
# Edit configuration
vim /opt/foxhunt/config/trading_agent.toml

# Ensure multipliers are within bounds:
[adaptive.position_multipliers]
normal = 1.0
trending = 1.3       # From 1.5 (reduce aggressiveness)
bull = 1.2           # From 1.4
bear = 0.8
sideways = 0.5
high_volatility = 0.5
crisis = 0.2

# Reload
systemctl reload trading_agent_service
```

**Resolution Option 2: Emergency Position Reduction** (immediate):
```bash
# If position multiplier already caused over-leveraging:

# 1. Reduce all open positions by 33%
tli trade positions --status OPEN | awk '{print $2}' | while read order_id; do
  tli trade modify --order-id $order_id --quantity-adjust -0.33
done

# 2. Set max position limit
tli trade config set --max-position-size 100000  # $100,000 hard cap

# 3. Restart trading agent with conservative settings
vim /opt/foxhunt/config/trading_agent.toml
# Set: adaptive_position_sizing = false (use static sizing)
systemctl restart trading_agent_service
```

**Verification** (15 minutes):
```bash
# Monitor position multiplier for 15 minutes
watch -n 60 'tli trade ml adaptive-params --symbol ES.FUT | grep "Position Multiplier"'

# Success criteria:
#   - Multiplier stays in [0.2, 1.5] range
#   - No alerts fired
#   - Risk budget utilization <80%
```

**Expected Behavior After Fix**:
- Position multiplier: 0.2-1.5x (within safe bounds)
- Maximum position size: $150,000 (1.5x × $100,000 baseline)
- Risk budget utilization: 40-70% (healthy range)

---

### Issue 5: Stop-Loss Multiplier Out of Range

**Symptom**:
- Alert: `StopLossMultiplierOutOfRange`
- Grafana: Stop-loss multiplier >5.0x ATR (expected max 4.0x)
- Logs: "Stop distance: $50.00 (ATR: $3.50, multiplier: 14.3x)" (calculation error)
- Impact: Risk management compromised, stops too wide (excessive loss potential)

**Root Cause**: ATR calculation error or multiplier config issue.

**Diagnosis** (3 minutes):

```bash
# 1. Check current stop-loss multiplier
tli trade ml adaptive-params --symbol ES.FUT | grep "Stop-Loss"

# Expected: 1.5-4.0x ATR
# Observed: 14.3x → out of range

# 2. Check ATR value
tli trade ml adaptive-params --symbol ES.FUT | grep "ATR"

# Expected: $2.50-$5.00 for ES.FUT
# Observed: $3.50 → reasonable
# Calculation: $50.00 / $3.50 = 14.3x → confirms multiplier issue

# 3. Check multiplier configuration
grep -A10 "STOPLOSS_MULTIPLIERS" /opt/foxhunt/config/trading_agent.toml
```

**Resolution** (5 minutes):

```bash
# Fix stop-loss multiplier configuration
vim /opt/foxhunt/config/trading_agent.toml

# Ensure multipliers are within bounds:
[adaptive.stoploss_multipliers]
normal = 2.0
trending = 2.5
bull = 2.0
bear = 2.5
sideways = 1.5
high_volatility = 3.0
crisis = 4.0         # Max value, widest stops

# Reload
systemctl reload trading_agent_service

# Verify change applied
tli trade ml adaptive-params --symbol ES.FUT | grep "Stop-Loss"

# Expected: 1.5-4.0x ATR
```

**Emergency Action** (if stops already too wide):
```bash
# Manually set stops to 2.0x ATR for all open positions
ATR=$(tli trade ml adaptive-params --symbol ES.FUT | grep "ATR" | awk '{print $3}')
STOP_DISTANCE=$(echo "$ATR * 2.0" | bc)

tli trade positions --status OPEN | awk '{print $2}' | while read order_id; do
  tli trade modify --order-id $order_id --stop-loss $STOP_DISTANCE
done
```

**Verification** (10 minutes):
```bash
# Monitor stop-loss multiplier
watch -n 60 'tli trade ml adaptive-params --symbol ES.FUT | grep "Stop-Loss"'

# Verify all open positions have correct stops
tli trade positions --status OPEN | grep "stop_loss"

# Success criteria:
#   - Multiplier in [1.5, 4.0] range
#   - Stop distance = ATR × multiplier (±5%)
```

**Expected Behavior After Fix**:
- Stop-loss multiplier: 1.5-4.0x ATR (regime-dependent)
- Stop distance: $5.00-$20.00 for ES.FUT (typical ATR $3.50)
- Max loss per trade: $20.00 (4.0x ATR in Crisis regime)

---

### Issue 6: Feature NaN/Inf Detected

**Symptom**:
- Alert: `FeatureDataQualityIssue`
- Grafana: `wave_d_feature_nan_count > 0` or `wave_d_feature_inf_count > 0`
- ML models: Training loss NaN
- Impact: ML models fail, trading halted

**Root Cause**: Division by zero or numerical instability in feature calculations.

**Diagnosis** (5 minutes):

```bash
# 1. Identify problematic feature
psql -U foxhunt -d foxhunt -c "
  SELECT feature_index, COUNT(*) AS error_count
  FROM ml_features
  WHERE symbol='ES.FUT'
    AND timestamp > NOW() - INTERVAL '1 hour'
    AND (feature_value IS NULL OR feature_value = 'NaN' OR feature_value = 'Infinity')
  GROUP BY feature_index
  ORDER BY error_count DESC
  LIMIT 5;
"

# Common problematic features:
#   - 223 (Regime Sharpe): Division by zero when std=0
#   - 224 (Risk Budget): Division by zero when max_position_size=0
#   - 218 (Shannon Entropy): log(0) when probability=0

# 2. Check logs for error details
grep -A5 "Invalid feature value" /var/log/foxhunt/ml_training.log | tail -20

# Look for stack traces pointing to specific calculations
```

**Resolution Patches**:

**Fix 1: Feature 223 (Regime Sharpe Ratio)** (10 minutes):
```rust
// File: /home/jgrusewski/Work/foxhunt/ml/src/features/regime_adaptive.rs
// Line: ~105

// BEFORE (causes NaN when std=0):
let sharpe = (mean / std) * (252.0_f64).sqrt();

// AFTER (safe):
let std = variance.sqrt();
let sharpe = if std > 1e-10 {
    (mean / std) * (252.0_f64).sqrt()
} else {
    0.0  // Return 0.0 instead of NaN when no volatility
};
```

**Fix 2: Feature 224 (Risk Budget Utilization)** (10 minutes):
```rust
// File: /home/jgrusewski/Work/foxhunt/ml/src/features/regime_adaptive.rs
// Line: ~125

// BEFORE (causes Inf when max_position_size=0):
let utilization = self.current_position_size / (position_mult * self.max_position_size);

// AFTER (safe):
let denominator = position_mult * self.max_position_size;
let utilization = if denominator > 1e-10 {
    (self.current_position_size / denominator).clamp(0.0, 1.0)
} else {
    0.0
};
```

**Fix 3: Feature 218 (Shannon Entropy)** (10 minutes):
```rust
// File: /home/jgrusewski/Work/foxhunt/ml/src/features/regime_transition.rs
// Line: ~85

// BEFORE (causes NaN when p=0):
let entropy: f64 = probabilities.iter()
    .map(|p| -p * p.log2())
    .sum();

// AFTER (safe):
let entropy: f64 = probabilities.iter()
    .filter(|&p| p > 1e-10)  // Skip zero probabilities
    .map(|p| -p * p.log2())
    .sum();
```

**Deployment** (15 minutes):
```bash
# 1. Apply fixes to source code
vim /home/jgrusewski/Work/foxhunt/ml/src/features/regime_adaptive.rs
vim /home/jgrusewski/Work/foxhunt/ml/src/features/regime_transition.rs

# 2. Run unit tests to verify fixes
cargo test -p ml --lib features::regime_adaptive -- test_all_features_finite
cargo test -p ml --lib features::regime_transition -- test_shannon_entropy_edge_cases

# Expected: All tests pass

# 3. Rebuild and deploy
cargo build --workspace --release
systemctl restart ml_training_service
systemctl restart trading_agent_service

# 4. Monitor for 10 minutes
watch -n 60 'psql -U foxhunt -d foxhunt -t -c "SELECT COUNT(*) FROM ml_features WHERE feature_index BETWEEN 201 AND 225 AND (feature_value IS NULL OR feature_value = '"'"'NaN'"'"' OR feature_value = '"'"'Infinity'"'"');"'

# Expected: 0 (no NaN/Inf values)
```

**Verification** (10 minutes):
```bash
# 1. Check Prometheus metrics
curl -s http://localhost:9090/api/v1/query?query=wave_d_feature_nan_count | jq '.data.result[0].value[1]'

# Expected: "0"

# 2. Validate all features finite
cargo test -p ml -- test_all_features_finite

# Expected: test result: ok
```

**Rollback** (if fixes fail or cause regressions):
```bash
# Disable Wave D features
vim /opt/foxhunt/config/trading_agent.toml
# Set: wave_d_enabled = false
systemctl reload trading_agent_service

# Revert to Wave C features (201 only)
tli trade ml predictions --symbol ES.FUT --limit 1 | jq '.features | length'
# Expected: 201
```

**Expected Behavior After Fix**:
- `wave_d_feature_nan_count == 0`
- `wave_d_feature_inf_count == 0`
- All 225 features within expected ranges
- ML model training loss converges (no NaN)

---

### Issue 7: Regime Flip-Flopping

**Symptom**:
- Alert: `RegimeFlipFloppingDetected`
- Grafana: >50 regime transitions per hour
- Trading: Excessive order placements/cancellations, increased slippage
- Impact: Strategy whipsawed, high transaction costs

**Root Cause**: Stability filter misconfiguration or high-frequency noise.

**Diagnosis** (5 minutes):

```bash
# 1. Check transition frequency
psql -U foxhunt -d foxhunt -c "
  SELECT COUNT(*) AS transitions_last_hour
  FROM regime_transitions
  WHERE symbol='ES.FUT'
    AND event_timestamp > NOW() - INTERVAL '1 hour';
"

# Expected: 5-15 transitions per hour
# Observed: >50 transitions → flip-flopping

# 2. Identify transition pattern
psql -U foxhunt -d foxhunt -c "
  SELECT from_regime, to_regime, COUNT(*) AS count
  FROM regime_transitions
  WHERE symbol='ES.FUT'
    AND event_timestamp > NOW() - INTERVAL '1 hour'
  GROUP BY from_regime, to_regime
  ORDER BY count DESC
  LIMIT 5;
"

# Look for:
#   - Normal ↔ Trending (back and forth)
#   - Bull ↔ Bear (oscillation)

# 3. Check CUSUM sensitivity
tli trade ml regime-status --symbol ES.FUT | grep "cusum"
```

**Resolution** (see Issue 1 for detailed CUSUM fixes):

**Quick Fix** (10 minutes):
```bash
# Increase stability window
vim /home/jgrusewski/Work/foxhunt/ml/src/features/config.rs

# Change:
pub const STABILITY_WINDOW: usize = 10; // From 5 (require 10 bars consensus)

# Rebuild and deploy
cargo build -p trading_agent_service --release
systemctl restart trading_agent_service
```

**Alternative Fix** (15 minutes):
```bash
# Reduce CUSUM weight in ensemble
vim /home/jgrusewski/Work/foxhunt/ml/src/features/config.rs

# Change:
pub const REGIME_CLASSIFIER_WEIGHTS: RegimeWeights = RegimeWeights {
    cusum: 0.25,    // From 0.40 (reduce CUSUM influence)
    trending: 0.35, // From 0.30 (increase trend following)
    ranging: 0.25,  // From 0.20
    volatile: 0.15, // From 0.10
};

# Rebuild and deploy (same as above)
```

**Verification** (1 hour):
```bash
# Monitor transition frequency for 1 hour
watch -n 60 'tli trade ml regime-transitions --symbol ES.FUT --limit 10'

# Success criteria:
#   - Transition frequency drops to <20 per hour
#   - Prometheus: `rate(regime_transitions_total[1h]) < 20`
#   - No alert fired for 1 hour
```

**Expected Behavior After Fix**:
- Transition frequency: 5-15 per hour (down from 50+)
- Regime persistence: >10 bars per regime (down from <5 bars)
- Reduced transaction costs due to fewer order cancellations

---

## Monitoring & Alerting

### Critical Alerts (PagerDuty)

#### Alert 1: FeatureDataQualityIssue
- **Condition**: `wave_d_feature_nan_count > 0 OR wave_d_feature_inf_count > 0`
- **For**: 1 minute
- **Severity**: Critical
- **Action**: See [Issue 6: Feature NaN/Inf Detected](#issue-6-feature-naninf-detected)

#### Alert 2: PositionSizeMultiplierOutOfRange
- **Condition**: `position_multiplier < 0.1 OR position_multiplier > 2.0`
- **For**: 1 minute
- **Severity**: Critical
- **Action**: See [Issue 4: Adaptive Position Sizing Too Aggressive](#issue-4-adaptive-position-sizing-too-aggressive)

#### Alert 3: StopLossMultiplierOutOfRange
- **Condition**: `stoploss_multiplier < 1.0 OR stoploss_multiplier > 5.0`
- **For**: 1 minute
- **Severity**: Critical
- **Action**: See [Issue 5: Stop-Loss Multiplier Out of Range](#issue-5-stop-loss-multiplier-out-of-range)

### Warning Alerts (Slack/Email)

#### Alert 4: RegimeFlipFloppingDetected
- **Condition**: `rate(regime_transitions_total[1h]) > 50`
- **For**: 5 minutes
- **Severity**: Warning
- **Action**: See [Issue 7: Regime Flip-Flopping](#issue-7-regime-flip-flopping)

#### Alert 5: CUSUMFalsePositiveSpike
- **Condition**: `rate(cusum_break_count[1h]) > 100`
- **For**: 5 minutes
- **Severity**: Warning
- **Action**: See [Issue 1: CUSUM False Positives](#issue-1-cusum-false-positives)

#### Alert 6: ADXInitializationFailure
- **Condition**: `adx{symbol!=""} == 0 AND up{job="trading_agent_service"} == 1`
- **For**: 10 minutes
- **Severity**: Warning
- **Action**: See [Issue 2: ADX Initialization Failures](#issue-2-adx-initialization-failures)

#### Alert 7: RiskBudgetOverutilization
- **Condition**: `risk_budget_utilization > 0.95`
- **For**: 5 minutes
- **Severity**: Warning
- **Action**: Reduce positions by 20% or widen stop-loss

#### Alert 8: FeatureExtractionLatencyHigh
- **Condition**: `histogram_quantile(0.99, wave_d_feature_extraction_duration_seconds) > 0.0001`
- **For**: 5 minutes
- **Severity**: Warning
- **Action**: Profile feature extraction, reduce symbol universe, or increase polling interval

### Prometheus Query Examples

```promql
# Current regime distribution
count by (regime) (current_regime)

# Transition frequency by symbol
rate(regime_transitions_total[1h]) * 3600

# Feature extraction P99 latency
histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m]))

# Risk budget utilization by regime
avg by (regime) (risk_budget_utilization)

# CUSUM false positive rate
rate(cusum_break_count[1h]) / on() group_left() (avg(bar_count) or vector(1))
```

---

## Performance Tuning

### Latency Optimization

**Target**: <65μs per bar (warm state), <500μs (cold start)

**Tuning Knobs**:

1. **VecDeque Pre-sizing** (5% latency reduction):
```rust
// Pre-allocate to maximum capacity
let mut breaks_window = VecDeque::with_capacity(100);
```

2. **Reduce Intermediate Allocations** (10% reduction):
```rust
// Use stack arrays for small buffers
let mut buffer: [f64; 10] = [0.0; 10];
// Instead of:
let mut buffer = vec![0.0; 10];
```

3. **Cache Regime State** (15% reduction):
```rust
// Cache last regime to avoid recomputation
if self.last_regime == regime && self.bar_count - self.last_update < 10 {
    return self.cached_features;
}
```

### Memory Optimization

**Target**: <500KB per symbol (100 symbols = <50MB total)

**Current Usage**:
- RegimeCUSUMFeatures: ~1KB (VecDeque with 100 capacity)
- RegimeADXFeatures: ~320 bytes
- RegimeTransitionFeatures: ~8KB (8×8 matrix)
- RegimeAdaptiveFeatures: ~200 bytes
- **Total**: ~10KB per symbol ✅ **50x under target**

**If Memory Exceeds Target**:
1. Reduce VecDeque capacity from 100 to 50
2. Use sparse matrix for transition probabilities
3. Limit number of tracked symbols to 50

### Throughput Scaling

**Target**: >18,000 bars/second (batch processing)

**Current**: ~18,000 bars/second ✅ **MEETS TARGET**

**If Throughput Degrades**:
1. Profile with `cargo flamegraph -p ml --test feature_extraction_bench`
2. Identify hot paths (likely ATR calculation or VecDeque operations)
3. Vectorize array operations where possible
4. Consider SIMD for bulk calculations

---

## Rollback Procedures

### Full Rollback to Wave C (201 Features)

**Duration**: 5 minutes
**Use Case**: Wave D causing production issues, unable to fix quickly

**Steps**:

1. **Disable Wave D Features**:
```bash
vim /opt/foxhunt/config/trading_agent.toml
# Set: wave_d_enabled = false
systemctl reload trading_agent_service
```

2. **Revert ML Models**:
```bash
cp /opt/foxhunt/models/wave_c/* /opt/foxhunt/models/current/
systemctl restart ml_training_service
systemctl restart trading_agent_service
```

3. **Revert Database Migration** (optional, only if schema causing issues):
```bash
cargo sqlx migrate revert --database-url postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
```

4. **Verify Rollback**:
```bash
tli trade ml predictions --symbol ES.FUT --limit 1 | jq '.features | length'
# Expected: 201 (Wave C only)
```

### Partial Rollback (Disable Specific Components)

**Disable CUSUM Only**:
```bash
vim /opt/foxhunt/config/trading_agent.toml
# Add: disable_cusum = true
systemctl reload trading_agent_service
```

**Disable ADX Only**:
```bash
# Add: disable_adx = true
systemctl reload trading_agent_service
```

**Disable Adaptive Strategies**:
```bash
# Set: adaptive_position_sizing = false
# Set: dynamic_stop_loss = false
systemctl reload trading_agent_service
```

---

## Quick Reference

### Configuration Files

- **Main Config**: `/opt/foxhunt/config/trading_agent.toml`
- **Feature Config**: `/home/jgrusewski/Work/foxhunt/ml/src/features/config.rs`
- **Regime Weights**: `/home/jgrusewski/Work/foxhunt/ml/src/features/config.rs` (line ~50)

### Log Files

- **Trading Agent**: `/var/log/foxhunt/trading_agent.log`
- **ML Training**: `/var/log/foxhunt/ml_training.log`
- **API Gateway**: `/var/log/foxhunt/api_gateway.log`
- **PostgreSQL**: `/var/log/postgresql/postgresql-*.log`

### Database Queries

```sql
-- Latest regime state
SELECT * FROM get_latest_regime('ES.FUT');

-- Transition matrix
SELECT * FROM get_regime_transition_matrix('ES.FUT', 24);

-- Adaptive performance by regime
SELECT * FROM get_regime_performance('ES.FUT', 24);

-- Recent transitions
SELECT * FROM regime_transitions
WHERE symbol='ES.FUT'
  AND event_timestamp > NOW() - INTERVAL '1 hour'
ORDER BY event_timestamp DESC
LIMIT 10;
```

### TLI Commands

```bash
# Regime status
tli trade ml regime-status --symbol ES.FUT

# Recent transitions
tli trade ml regime-transitions --symbol ES.FUT --limit 10

# Adaptive parameters
tli trade ml adaptive-params --symbol ES.FUT

# Regime performance
tli trade ml regime-performance --symbol ES.FUT

# Start predictions (paper trading)
tli trade ml start-predictions --interval 30 --symbols ES.FUT,NQ.FUT --paper-trading

# Stop predictions
tli trade ml stop-predictions
```

### Service Management

```bash
# Restart all services
systemctl restart api_gateway trading_service backtesting_service ml_training_service trading_agent_service

# Check service status
systemctl status trading_agent_service

# View logs in real-time
journalctl -u trading_agent_service -f

# Reload configuration (no downtime)
systemctl reload trading_agent_service
```

### Emergency Contacts

- **DevOps Lead**: @devops-lead (Slack), +1-XXX-XXX-XXXX
- **ML Engineer**: @ml-eng (Slack), +1-XXX-XXX-XXXX
- **On-Call Rotation**: PagerDuty schedule

---

**Document Version**: 1.0
**Last Updated**: 2025-10-18
**Status**: 🟢 **PRODUCTION READY**

**See Also**:
- [WAVE_D_PRODUCTION_CHECKLIST.md](WAVE_D_PRODUCTION_CHECKLIST.md) - Deployment checklist
- [WAVE_D_MONITORING_GUIDE.md](WAVE_D_MONITORING_GUIDE.md) - Grafana dashboards
- [WAVE_D_COMPLETION_SUMMARY.md](WAVE_D_COMPLETION_SUMMARY.md) - Executive summary