foxhunt/WAVE_D_OPERATIONAL_RUNBOOK.md

Wave D Operational Runbook

Version: 1.0 Date: 2025-10-18 Status: 🟢 PRODUCTION READY Purpose: Incident response guide for Wave D regime detection & adaptive strategies

---

Table of Contents

1. Common Issues & Resolutions
2. Monitoring & Alerting
3. Performance Tuning
4. Rollback Procedures
5. Quick Reference

---

Common Issues & Resolutions

Issue 1: CUSUM False Positives

Symptom:

• Alert: CUSUMFalsePositiveSpike
• Grafana: >100 structural breaks per hour
• Logs: Frequent "Structural break detected" messages
• Impact: Regime detection oversensitivity, frequent Crisis regime misclassification

Root Cause: CUSUM threshold too low for current market volatility.

Diagnosis (5 minutes):

# 1. Check current break frequency
psql -U foxhunt -d foxhunt -c "
  SELECT COUNT(*) AS break_count
  FROM regime_transitions
  WHERE symbol='ES.FUT'
    AND event_timestamp > NOW() - INTERVAL '1 hour'
    AND cusum_alert_triggered = TRUE;
"

# Expected: <10 breaks/hour
# Observed: >100 breaks/hour → CUSUM too sensitive

# 2. Check current CUSUM parameters
grep -A5 "CUSUM_THRESHOLD\|CUSUM_DRIFT" /opt/foxhunt/config/trading_agent.toml

# Current values:
#   cusum_threshold = 4.0
#   cusum_drift_allowance = 0.5

# 3. Measure current market volatility
tli trade ml adaptive-params --symbol ES.FUT | grep "ATR"

# If ATR >5.0 (high volatility), increase threshold

Resolution Options:

Option 1: Increase CUSUM Threshold (recommended, 10 minutes):

# Edit configuration
vim /opt/foxhunt/config/trading_agent.toml

# Change:
cusum_threshold = 5.0  # From 4.0 (25% less sensitive)

# Reload configuration
systemctl reload trading_agent_service

# Verify change applied
tail -f /var/log/foxhunt/trading_agent.log | grep "CUSUM threshold"
# Expected: "CUSUM threshold set to 5.0"

Option 2: Increase Drift Allowance (alternative, 10 minutes):

# Edit configuration
vim /opt/foxhunt/config/trading_agent.toml

# Change:
cusum_drift_allowance = 0.75  # From 0.5 (50% more tolerance)

# Reload and verify (same as Option 1)

Option 3: Reduce CUSUM Weight in Ensemble (last resort, 15 minutes):

# Edit source code
vim /home/jgrusewski/Work/foxhunt/ml/src/features/config.rs

# Change:
pub const REGIME_CLASSIFIER_WEIGHTS: RegimeWeights = RegimeWeights {
    cusum: 0.25,    // From 0.40 (reduce influence)
    trending: 0.35, // From 0.30 (increase influence)
    ranging: 0.25,  // From 0.20
    volatile: 0.15, // From 0.10
};

# Rebuild and deploy
cargo build -p trading_agent_service --release
systemctl restart trading_agent_service

Verification (30 minutes):

# Monitor break frequency for 30 minutes
watch -n 60 'psql -U foxhunt -d foxhunt -t -c "SELECT COUNT(*) FROM regime_transitions WHERE symbol='"'"'ES.FUT'"'"' AND event_timestamp > NOW() - INTERVAL '"'"'1 hour'"'"' AND cusum_alert_triggered = TRUE;"'

# Success criteria: <10 breaks per hour
# If still >50 breaks/hour: Apply Option 2 or 3

Expected Behavior After Fix:

• Break frequency: 5-10 per hour (down from 100+)
• False positive rate: <0.5% (test with cargo test -p ml --test cusum_test -- test_false_positive_rate)
• Regime stability: >0.7 (fewer Crisis regime misclassifications)

---

Issue 2: ADX Initialization Failures

Symptom:

• Alert: ADXInitializationFailure
• Grafana: ADX stuck at 0.0 despite >10 minutes of data
• Logs: "ADX not initialized, need 28 bars" (repeated)
• Impact: Trend classification unavailable, falling back to CUSUM-only regime detection

Root Cause: Insufficient historical data to initialize ADX (requires 28 bars minimum).

Diagnosis (3 minutes):

# 1. Check current bar count
tli trade ml regime-status --symbol ES.FUT | grep "Bar count"

# Expected: Bar count ≥ 28
# Observed: Bar count < 28 → waiting for warmup

# 2. Verify data ingestion rate
psql -U foxhunt -d foxhunt -c "
  SELECT COUNT(*) AS bars_received
  FROM market_data
  WHERE symbol='ES.FUT'
    AND timestamp > NOW() - INTERVAL '10 minutes';
"

# Expected: >10 bars (1-minute intervals)
# Observed: 0 bars → data feed issue

Resolution:

Scenario A: Bar count < 28 (waiting for warmup) (no action required):

# Calculate remaining wait time
echo "Bars needed: $((28 - $(tli trade ml regime-status --symbol ES.FUT | grep "Bar count" | awk '{print $3}')))"
echo "Wait time: ~$((28 - bar_count)) minutes (assuming 1-minute bars)"

# ADX will auto-initialize when 28 bars accumulated
# Monitor progress:
watch -n 60 'tli trade ml regime-status --symbol ES.FUT | grep "Bar count\|ADX"'

Scenario B: Bar count ≥ 28 but ADX still 0.0 (bug, 20 minutes):

# 1. Check ADX calculation logic
tail -100 /var/log/foxhunt/trading_agent.log | grep -i "adx"

# Look for errors like:
#   - "ADX calculation failed: division by zero"
#   - "TR array empty, cannot compute ATR"
#   - "DMI initialization failed"

# 2. Restart service to clear state
systemctl restart trading_agent_service

# 3. Wait 28 bars for re-initialization
sleep 1800  # 30 minutes for 28 bars + safety margin

# 4. Verify ADX initialized
tli trade ml regime-status --symbol ES.FUT | grep "ADX"

# Expected: ADX: 15-45 (non-zero)
# If still 0.0: File bug report + revert to Wave C features

Scenario C: Data feed issue (external dependency, 10 minutes):

# 1. Check Databento service status
curl -I https://hist.databento.com/v0/health

# Expected: HTTP 200
# Observed: HTTP 503 → Databento outage

# 2. Check local data acquisition service
systemctl status data_acquisition_service

# If inactive: systemctl start data_acquisition_service

# 3. Verify data ingestion resumed
watch -n 60 'psql -U foxhunt -d foxhunt -t -c "SELECT COUNT(*) FROM market_data WHERE symbol='"'"'ES.FUT'"'"' AND timestamp > NOW() - INTERVAL '"'"'5 minutes'"'"';"'

# Expected: Increasing count every minute

Workaround (if ADX initialization blocked for >1 hour):

# Temporarily disable ADX features (use CUSUM-only regime detection)
vim /opt/foxhunt/config/trading_agent.toml

# Add:
[features.wave_d]
disable_adx = true  # Fallback to CUSUM + Transition + Adaptive

# Reload
systemctl reload trading_agent_service

# Impact: Reduced regime detection accuracy (no trend strength signal)

Expected Behavior After Fix:

• ADX initializes within 28-30 bars (28-30 minutes for 1-minute bars)
• ADX value in range [0, 100], typically 15-45 for ES.FUT
• Trending regime detection accuracy improves (ADX >25 = trending)

---

Issue 3: Transition Matrix Entropy Too High

Symptom:

• Alert: TransitionMatrixEntropyHigh
• Grafana: Shannon entropy >2.5 (random regime switches)
• Logs: "Regime stability low: 0.35" (expected >0.6)
• Impact: Unpredictable regime transitions, reduced adaptive strategy effectiveness

Root Cause: EMA alpha too high (over-reactive to recent transitions) or insufficient regime persistence.

Diagnosis (5 minutes):

# 1. Check current entropy
tli trade ml regime-status --symbol ES.FUT | grep "Entropy"

# Expected: 0.5-1.5 (low to moderate entropy)
# Observed: >2.5 (high entropy, near-random transitions)

# 2. Check transition matrix
psql -U foxhunt -d foxhunt -c "
  SELECT * FROM get_regime_transition_matrix('ES.FUT', 24);
"

# Look for:
#   - Uniform probabilities across all regimes (~12.5% each for 8 regimes)
#   - Low diagonal values (<0.5 = low regime persistence)

# 3. Check EMA alpha parameter
grep "TRANSITION_EMA_ALPHA" /opt/foxhunt/config/trading_agent.toml

# Current: 0.1 (fast adaptation)
# Target: 0.05 (slower adaptation)

Resolution (10 minutes):

# Decrease EMA alpha for slower adaptation
vim /opt/foxhunt/config/trading_agent.toml

# Change:
transition_ema_alpha = 0.05  # From 0.1 (50% slower)

# Reload
systemctl reload trading_agent_service

# Verify change applied
tail -f /var/log/foxhunt/trading_agent.log | grep "Transition EMA alpha"
# Expected: "Transition EMA alpha set to 0.05"

Verification (1 hour):

# Monitor entropy over 1 hour
watch -n 600 'tli trade ml regime-status --symbol ES.FUT | grep "Entropy"'

# Success criteria:
#   - Entropy drops from >2.5 to <1.5 within 1 hour
#   - Stability increases from <0.4 to >0.6

Alternative Resolution (if entropy remains high):

# Increase regime persistence threshold
vim /opt/foxhunt/config/trading_agent.toml

# Add:
[regime.persistence]
min_bars = 10  # From 5 (require 10 bars before allowing transition)

# Rebuild and deploy
cargo build -p trading_agent_service --release
systemctl restart trading_agent_service

Expected Behavior After Fix:

• Shannon entropy: 0.5-1.5 (structured regime transitions)
• Regime stability: >0.6 (high persistence)
• Transition frequency: 5-15 per day (down from >50 per hour)

---

Issue 4: Adaptive Position Sizing Too Aggressive

Symptom:

• Alert: PositionSizeMultiplierOutOfRange
• Grafana: Position multiplier >2.0 (expected max 1.5x)
• Logs: "Position size: $150,000 (multiplier 2.0x)" (exceeds $100,000 baseline)
• Impact: Risk management compromised, potential over-leveraging

Root Cause: Trending regime multiplier misconfigured or regime detection too sensitive.

Diagnosis (3 minutes):

# 1. Check current position multiplier
tli trade ml adaptive-params --symbol ES.FUT | grep "Position Multiplier"

# Expected: 0.2-1.5x
# Observed: 2.0x → out of range

# 2. Check current regime
tli trade ml regime-status --symbol ES.FUT | grep "Regime:"

# If "Trending" → check multiplier config
# If "Crisis" or "Normal" → regime detection error

# 3. Check multiplier configuration
grep -A10 "POSITION_MULTIPLIERS" /opt/foxhunt/config/trading_agent.toml

Resolution Option 1: Fix Multiplier Config (5 minutes):

# Edit configuration
vim /opt/foxhunt/config/trading_agent.toml

# Ensure multipliers are within bounds:
[adaptive.position_multipliers]
normal = 1.0
trending = 1.3       # From 1.5 (reduce aggressiveness)
bull = 1.2           # From 1.4
bear = 0.8
sideways = 0.5
high_volatility = 0.5
crisis = 0.2

# Reload
systemctl reload trading_agent_service

Resolution Option 2: Emergency Position Reduction (immediate):

# If position multiplier already caused over-leveraging:

# 1. Reduce all open positions by 33%
tli trade positions --status OPEN | awk '{print $2}' | while read order_id; do
  tli trade modify --order-id $order_id --quantity-adjust -0.33
done

# 2. Set max position limit
tli trade config set --max-position-size 100000  # $100,000 hard cap

# 3. Restart trading agent with conservative settings
vim /opt/foxhunt/config/trading_agent.toml
# Set: adaptive_position_sizing = false (use static sizing)
systemctl restart trading_agent_service

Verification (15 minutes):

# Monitor position multiplier for 15 minutes
watch -n 60 'tli trade ml adaptive-params --symbol ES.FUT | grep "Position Multiplier"'

# Success criteria:
#   - Multiplier stays in [0.2, 1.5] range
#   - No alerts fired
#   - Risk budget utilization <80%

Expected Behavior After Fix:

• Position multiplier: 0.2-1.5x (within safe bounds)
• Maximum position size: $150,000 (1.5x × $100,000 baseline)
• Risk budget utilization: 40-70% (healthy range)

---

Issue 5: Stop-Loss Multiplier Out of Range

Symptom:

• Alert: StopLossMultiplierOutOfRange
• Grafana: Stop-loss multiplier >5.0x ATR (expected max 4.0x)
• Logs: "Stop distance: $50.00 (ATR: $3.50, multiplier: 14.3x)" (calculation error)
• Impact: Risk management compromised, stops too wide (excessive loss potential)

Root Cause: ATR calculation error or multiplier config issue.

Diagnosis (3 minutes):

# 1. Check current stop-loss multiplier
tli trade ml adaptive-params --symbol ES.FUT | grep "Stop-Loss"

# Expected: 1.5-4.0x ATR
# Observed: 14.3x → out of range

# 2. Check ATR value
tli trade ml adaptive-params --symbol ES.FUT | grep "ATR"

# Expected: $2.50-$5.00 for ES.FUT
# Observed: $3.50 → reasonable
# Calculation: $50.00 / $3.50 = 14.3x → confirms multiplier issue

# 3. Check multiplier configuration
grep -A10 "STOPLOSS_MULTIPLIERS" /opt/foxhunt/config/trading_agent.toml

Resolution (5 minutes):

# Fix stop-loss multiplier configuration
vim /opt/foxhunt/config/trading_agent.toml

# Ensure multipliers are within bounds:
[adaptive.stoploss_multipliers]
normal = 2.0
trending = 2.5
bull = 2.0
bear = 2.5
sideways = 1.5
high_volatility = 3.0
crisis = 4.0         # Max value, widest stops

# Reload
systemctl reload trading_agent_service

# Verify change applied
tli trade ml adaptive-params --symbol ES.FUT | grep "Stop-Loss"

# Expected: 1.5-4.0x ATR

Emergency Action (if stops already too wide):

# Manually set stops to 2.0x ATR for all open positions
ATR=$(tli trade ml adaptive-params --symbol ES.FUT | grep "ATR" | awk '{print $3}')
STOP_DISTANCE=$(echo "$ATR * 2.0" | bc)

tli trade positions --status OPEN | awk '{print $2}' | while read order_id; do
  tli trade modify --order-id $order_id --stop-loss $STOP_DISTANCE
done

Verification (10 minutes):

# Monitor stop-loss multiplier
watch -n 60 'tli trade ml adaptive-params --symbol ES.FUT | grep "Stop-Loss"'

# Verify all open positions have correct stops
tli trade positions --status OPEN | grep "stop_loss"

# Success criteria:
#   - Multiplier in [1.5, 4.0] range
#   - Stop distance = ATR × multiplier (±5%)

Expected Behavior After Fix:

• Stop-loss multiplier: 1.5-4.0x ATR (regime-dependent)
• Stop distance: $5.00-$20.00 for ES.FUT (typical ATR $3.50)
• Max loss per trade: $20.00 (4.0x ATR in Crisis regime)

---

Issue 6: Feature NaN/Inf Detected

Symptom:

• Alert: FeatureDataQualityIssue
• Grafana: wave_d_feature_nan_count > 0 or wave_d_feature_inf_count > 0
• ML models: Training loss NaN
• Impact: ML models fail, trading halted

Root Cause: Division by zero or numerical instability in feature calculations.

Diagnosis (5 minutes):

# 1. Identify problematic feature
psql -U foxhunt -d foxhunt -c "
  SELECT feature_index, COUNT(*) AS error_count
  FROM ml_features
  WHERE symbol='ES.FUT'
    AND timestamp > NOW() - INTERVAL '1 hour'
    AND (feature_value IS NULL OR feature_value = 'NaN' OR feature_value = 'Infinity')
  GROUP BY feature_index
  ORDER BY error_count DESC
  LIMIT 5;
"

# Common problematic features:
#   - 223 (Regime Sharpe): Division by zero when std=0
#   - 224 (Risk Budget): Division by zero when max_position_size=0
#   - 218 (Shannon Entropy): log(0) when probability=0

# 2. Check logs for error details
grep -A5 "Invalid feature value" /var/log/foxhunt/ml_training.log | tail -20

# Look for stack traces pointing to specific calculations

Resolution Patches:

Fix 1: Feature 223 (Regime Sharpe Ratio) (10 minutes):

// File: /home/jgrusewski/Work/foxhunt/ml/src/features/regime_adaptive.rs
// Line: ~105

// BEFORE (causes NaN when std=0):
let sharpe = (mean / std) * (252.0_f64).sqrt();

// AFTER (safe):
let std = variance.sqrt();
let sharpe = if std > 1e-10 {
    (mean / std) * (252.0_f64).sqrt()
} else {
    0.0  // Return 0.0 instead of NaN when no volatility
};

Fix 2: Feature 224 (Risk Budget Utilization) (10 minutes):

// File: /home/jgrusewski/Work/foxhunt/ml/src/features/regime_adaptive.rs
// Line: ~125

// BEFORE (causes Inf when max_position_size=0):
let utilization = self.current_position_size / (position_mult * self.max_position_size);

// AFTER (safe):
let denominator = position_mult * self.max_position_size;
let utilization = if denominator > 1e-10 {
    (self.current_position_size / denominator).clamp(0.0, 1.0)
} else {
    0.0
};

Fix 3: Feature 218 (Shannon Entropy) (10 minutes):

// File: /home/jgrusewski/Work/foxhunt/ml/src/features/regime_transition.rs
// Line: ~85

// BEFORE (causes NaN when p=0):
let entropy: f64 = probabilities.iter()
    .map(|p| -p * p.log2())
    .sum();

// AFTER (safe):
let entropy: f64 = probabilities.iter()
    .filter(|&p| p > 1e-10)  // Skip zero probabilities
    .map(|p| -p * p.log2())
    .sum();

Deployment (15 minutes):

# 1. Apply fixes to source code
vim /home/jgrusewski/Work/foxhunt/ml/src/features/regime_adaptive.rs
vim /home/jgrusewski/Work/foxhunt/ml/src/features/regime_transition.rs

# 2. Run unit tests to verify fixes
cargo test -p ml --lib features::regime_adaptive -- test_all_features_finite
cargo test -p ml --lib features::regime_transition -- test_shannon_entropy_edge_cases

# Expected: All tests pass

# 3. Rebuild and deploy
cargo build --workspace --release
systemctl restart ml_training_service
systemctl restart trading_agent_service

# 4. Monitor for 10 minutes
watch -n 60 'psql -U foxhunt -d foxhunt -t -c "SELECT COUNT(*) FROM ml_features WHERE feature_index BETWEEN 201 AND 225 AND (feature_value IS NULL OR feature_value = '"'"'NaN'"'"' OR feature_value = '"'"'Infinity'"'"');"'

# Expected: 0 (no NaN/Inf values)

Verification (10 minutes):

# 1. Check Prometheus metrics
curl -s http://localhost:9090/api/v1/query?query=wave_d_feature_nan_count | jq '.data.result[0].value[1]'

# Expected: "0"

# 2. Validate all features finite
cargo test -p ml -- test_all_features_finite

# Expected: test result: ok

Rollback (if fixes fail or cause regressions):

# Disable Wave D features
vim /opt/foxhunt/config/trading_agent.toml
# Set: wave_d_enabled = false
systemctl reload trading_agent_service

# Revert to Wave C features (201 only)
tli trade ml predictions --symbol ES.FUT --limit 1 | jq '.features | length'
# Expected: 201

Expected Behavior After Fix:

• wave_d_feature_nan_count == 0
• wave_d_feature_inf_count == 0
• All 225 features within expected ranges
• ML model training loss converges (no NaN)

---

Issue 7: Regime Flip-Flopping

Symptom:

• Alert: RegimeFlipFloppingDetected
• Grafana: >50 regime transitions per hour
• Trading: Excessive order placements/cancellations, increased slippage
• Impact: Strategy whipsawed, high transaction costs

Root Cause: Stability filter misconfiguration or high-frequency noise.

Diagnosis (5 minutes):

# 1. Check transition frequency
psql -U foxhunt -d foxhunt -c "
  SELECT COUNT(*) AS transitions_last_hour
  FROM regime_transitions
  WHERE symbol='ES.FUT'
    AND event_timestamp > NOW() - INTERVAL '1 hour';
"

# Expected: 5-15 transitions per hour
# Observed: >50 transitions → flip-flopping

# 2. Identify transition pattern
psql -U foxhunt -d foxhunt -c "
  SELECT from_regime, to_regime, COUNT(*) AS count
  FROM regime_transitions
  WHERE symbol='ES.FUT'
    AND event_timestamp > NOW() - INTERVAL '1 hour'
  GROUP BY from_regime, to_regime
  ORDER BY count DESC
  LIMIT 5;
"

# Look for:
#   - Normal ↔ Trending (back and forth)
#   - Bull ↔ Bear (oscillation)

# 3. Check CUSUM sensitivity
tli trade ml regime-status --symbol ES.FUT | grep "cusum"

Resolution (see Issue 1 for detailed CUSUM fixes):

Quick Fix (10 minutes):

# Increase stability window
vim /home/jgrusewski/Work/foxhunt/ml/src/features/config.rs

# Change:
pub const STABILITY_WINDOW: usize = 10; // From 5 (require 10 bars consensus)

# Rebuild and deploy
cargo build -p trading_agent_service --release
systemctl restart trading_agent_service

Alternative Fix (15 minutes):

# Reduce CUSUM weight in ensemble
vim /home/jgrusewski/Work/foxhunt/ml/src/features/config.rs

# Change:
pub const REGIME_CLASSIFIER_WEIGHTS: RegimeWeights = RegimeWeights {
    cusum: 0.25,    // From 0.40 (reduce CUSUM influence)
    trending: 0.35, // From 0.30 (increase trend following)
    ranging: 0.25,  // From 0.20
    volatile: 0.15, // From 0.10
};

# Rebuild and deploy (same as above)

Verification (1 hour):

# Monitor transition frequency for 1 hour
watch -n 60 'tli trade ml regime-transitions --symbol ES.FUT --limit 10'

# Success criteria:
#   - Transition frequency drops to <20 per hour
#   - Prometheus: `rate(regime_transitions_total[1h]) < 20`
#   - No alert fired for 1 hour

Expected Behavior After Fix:

• Transition frequency: 5-15 per hour (down from 50+)
• Regime persistence: >10 bars per regime (down from <5 bars)
• Reduced transaction costs due to fewer order cancellations

---

Monitoring & Alerting

Critical Alerts (PagerDuty)

Alert 1: FeatureDataQualityIssue

• Condition: wave_d_feature_nan_count > 0 OR wave_d_feature_inf_count > 0
• For: 1 minute
• Severity: Critical
• Action: See Issue 6: Feature NaN/Inf Detected

Alert 2: PositionSizeMultiplierOutOfRange

• Condition: position_multiplier < 0.1 OR position_multiplier > 2.0
• For: 1 minute
• Severity: Critical
• Action: See Issue 4: Adaptive Position Sizing Too Aggressive

Alert 3: StopLossMultiplierOutOfRange

• Condition: stoploss_multiplier < 1.0 OR stoploss_multiplier > 5.0
• For: 1 minute
• Severity: Critical
• Action: See Issue 5: Stop-Loss Multiplier Out of Range

Warning Alerts (Slack/Email)

Alert 4: RegimeFlipFloppingDetected

• Condition: rate(regime_transitions_total[1h]) > 50
• For: 5 minutes
• Severity: Warning
• Action: See Issue 7: Regime Flip-Flopping

Alert 5: CUSUMFalsePositiveSpike

• Condition: rate(cusum_break_count[1h]) > 100
• For: 5 minutes
• Severity: Warning
• Action: See Issue 1: CUSUM False Positives

Alert 6: ADXInitializationFailure

• Condition: adx{symbol!=""} == 0 AND up{job="trading_agent_service"} == 1
• For: 10 minutes
• Severity: Warning
• Action: See Issue 2: ADX Initialization Failures

Alert 7: RiskBudgetOverutilization

• Condition: risk_budget_utilization > 0.95
• For: 5 minutes
• Severity: Warning
• Action: Reduce positions by 20% or widen stop-loss

Alert 8: FeatureExtractionLatencyHigh

• Condition: histogram_quantile(0.99, wave_d_feature_extraction_duration_seconds) > 0.0001
• For: 5 minutes
• Severity: Warning
• Action: Profile feature extraction, reduce symbol universe, or increase polling interval

Prometheus Query Examples

# Current regime distribution
count by (regime) (current_regime)

# Transition frequency by symbol
rate(regime_transitions_total[1h]) * 3600

# Feature extraction P99 latency
histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m]))

# Risk budget utilization by regime
avg by (regime) (risk_budget_utilization)

# CUSUM false positive rate
rate(cusum_break_count[1h]) / on() group_left() (avg(bar_count) or vector(1))

---

Performance Tuning

Latency Optimization

Target: <65μs per bar (warm state), <500μs (cold start)

Tuning Knobs:

1. VecDeque Pre-sizing (5% latency reduction):

// Pre-allocate to maximum capacity
let mut breaks_window = VecDeque::with_capacity(100);

2. Reduce Intermediate Allocations (10% reduction):

// Use stack arrays for small buffers
let mut buffer: [f64; 10] = [0.0; 10];
// Instead of:
let mut buffer = vec![0.0; 10];

3. Cache Regime State (15% reduction):

// Cache last regime to avoid recomputation
if self.last_regime == regime && self.bar_count - self.last_update < 10 {
    return self.cached_features;
}

Memory Optimization

Target: <500KB per symbol (100 symbols = <50MB total)

Current Usage:

• RegimeCUSUMFeatures: ~1KB (VecDeque with 100 capacity)
• RegimeADXFeatures: ~320 bytes
• RegimeTransitionFeatures: ~8KB (8×8 matrix)
• RegimeAdaptiveFeatures: ~200 bytes
• Total: ~10KB per symbol ✅ 50x under target

If Memory Exceeds Target:

1. Reduce VecDeque capacity from 100 to 50
2. Use sparse matrix for transition probabilities
3. Limit number of tracked symbols to 50

Throughput Scaling

Target: >18,000 bars/second (batch processing)

Current: ~18,000 bars/second ✅ MEETS TARGET

If Throughput Degrades:

1. Profile with cargo flamegraph -p ml --test feature_extraction_bench
2. Identify hot paths (likely ATR calculation or VecDeque operations)
3. Vectorize array operations where possible
4. Consider SIMD for bulk calculations

---

Rollback Procedures

Full Rollback to Wave C (201 Features)

Duration: 5 minutes Use Case: Wave D causing production issues, unable to fix quickly

Steps:

1. Disable Wave D Features:

vim /opt/foxhunt/config/trading_agent.toml
# Set: wave_d_enabled = false
systemctl reload trading_agent_service

2. Revert ML Models:

cp /opt/foxhunt/models/wave_c/* /opt/foxhunt/models/current/
systemctl restart ml_training_service
systemctl restart trading_agent_service

3. Revert Database Migration (optional, only if schema causing issues):

cargo sqlx migrate revert --database-url postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

4. Verify Rollback:

tli trade ml predictions --symbol ES.FUT --limit 1 | jq '.features | length'
# Expected: 201 (Wave C only)

Partial Rollback (Disable Specific Components)

Disable CUSUM Only:

vim /opt/foxhunt/config/trading_agent.toml
# Add: disable_cusum = true
systemctl reload trading_agent_service

Disable ADX Only:

# Add: disable_adx = true
systemctl reload trading_agent_service

Disable Adaptive Strategies:

# Set: adaptive_position_sizing = false
# Set: dynamic_stop_loss = false
systemctl reload trading_agent_service

---

Quick Reference

Configuration Files

• Main Config: /opt/foxhunt/config/trading_agent.toml
• Feature Config: /home/jgrusewski/Work/foxhunt/ml/src/features/config.rs
• Regime Weights: /home/jgrusewski/Work/foxhunt/ml/src/features/config.rs (line ~50)

Log Files

• Trading Agent: /var/log/foxhunt/trading_agent.log
• ML Training: /var/log/foxhunt/ml_training.log
• API Gateway: /var/log/foxhunt/api_gateway.log
• PostgreSQL: /var/log/postgresql/postgresql-*.log

Database Queries

-- Latest regime state
SELECT * FROM get_latest_regime('ES.FUT');

-- Transition matrix
SELECT * FROM get_regime_transition_matrix('ES.FUT', 24);

-- Adaptive performance by regime
SELECT * FROM get_regime_performance('ES.FUT', 24);

-- Recent transitions
SELECT * FROM regime_transitions
WHERE symbol='ES.FUT'
  AND event_timestamp > NOW() - INTERVAL '1 hour'
ORDER BY event_timestamp DESC
LIMIT 10;

TLI Commands

# Regime status
tli trade ml regime-status --symbol ES.FUT

# Recent transitions
tli trade ml regime-transitions --symbol ES.FUT --limit 10

# Adaptive parameters
tli trade ml adaptive-params --symbol ES.FUT

# Regime performance
tli trade ml regime-performance --symbol ES.FUT

# Start predictions (paper trading)
tli trade ml start-predictions --interval 30 --symbols ES.FUT,NQ.FUT --paper-trading

# Stop predictions
tli trade ml stop-predictions

Service Management

# Restart all services
systemctl restart api_gateway trading_service backtesting_service ml_training_service trading_agent_service

# Check service status
systemctl status trading_agent_service

# View logs in real-time
journalctl -u trading_agent_service -f

# Reload configuration (no downtime)
systemctl reload trading_agent_service

Emergency Contacts

• DevOps Lead: @devops-lead (Slack), +1-XXX-XXX-XXXX
• ML Engineer: @ml-eng (Slack), +1-XXX-XXX-XXXX
• On-Call Rotation: PagerDuty schedule

---

Document Version: 1.0 Last Updated: 2025-10-18 Status: 🟢 PRODUCTION READY

See Also:

• WAVE_D_PRODUCTION_CHECKLIST.md - Deployment checklist
• WAVE_D_MONITORING_GUIDE.md - Grafana dashboards
• WAVE_D_COMPLETION_SUMMARY.md - Executive summary