foxhunt/WAVE_D_MONITORING_GUIDE.md

Wave D Monitoring Guide

Version: 1.0 Date: 2025-10-18 Status: 🟢 Production Ready

---

Table of Contents

1. Overview
2. Grafana Dashboards
3. Prometheus Metrics
4. Alert Thresholds
5. Logging Best Practices
6. Performance Monitoring
7. Data Quality Checks
8. Operational Playbooks

---

Overview

Wave D monitoring covers three critical areas:

1. Regime Detection: Track regime transitions, classifier performance, and stability
2. Adaptive Strategies: Monitor position sizing, stop-loss adjustments, and risk utilization
3. Feature Extraction: Validate feature quality, latency, and data integrity

Key Metrics Summary

Metric Category	Target	Alert Threshold	Dashboard
Regime Transitions	5-10/day	>50/hour	Regime Detection
CUSUM False Positives	<0.5%	>100/hour	Regime Detection
ADX Initialization	<28 bars	Stuck at 0 for >10min	Regime Detection
Feature Extraction Latency	P99 <50μs	P99 >100μs	Feature Performance
Feature NaN/Inf Count	0	>0	Feature Performance
Position Multiplier	[0.2, 1.5]	<0.1 or >2.0	Adaptive Strategies
Stop-Loss Multiplier	[1.5, 4.0]	<1.0 or >5.0	Adaptive Strategies
Risk Budget Utilization	<80%	>95%	Adaptive Strategies

---

Grafana Dashboards

Dashboard 1: Wave D - Regime Detection

Import Path: grafana/dashboards/wave_d_regime_detection.json Refresh Interval: 30 seconds Data Source: Prometheus

Panel 1.1: Current Regime (Gauge)

# Query
current_regime{symbol="ES.FUT"}

# Visualization: Gauge
# Thresholds:
#   - Normal: Green
#   - Trending: Blue
#   - Bull: Light Blue
#   - Bear: Orange
#   - Sideways: Yellow
#   - HighVolatility: Orange
#   - Crisis: Red

Purpose: Real-time regime classification for primary symbols.

Alert Conditions:

• Crisis regime for >4 hours → escalate to operations team
• Unknown regime → data quality issue

Panel 1.2: Regime Transitions Timeline (Time Series)

# Query
rate(regime_transitions_total{symbol="ES.FUT"}[5m]) * 3600

# Visualization: Time Series
# Unit: transitions per hour
# Alert: >50 transitions/hour (flip-flopping)

Purpose: Detect rapid regime switching (flip-flopping) that may indicate stability filter misconfiguration.

Interpretation:

• 5-10/day: Normal behavior
• 10-30/day: Volatile market conditions
• >50/day: Potential stability filter issue

Panel 1.3: CUSUM Statistics (Time Series)

# Query 1: S+ Normalized
cusum_s_plus{symbol="ES.FUT"} / cusum_threshold

# Query 2: S- Normalized
cusum_s_minus{symbol="ES.FUT"} / cusum_threshold

# Query 3: Break Count
increase(cusum_break_count{symbol="ES.FUT"}[1h])

# Visualization: Time Series (3 series)
# Alert: Break count >100/hour (false positive spike)

Purpose: Monitor structural break detection sensitivity.

Interpretation:

• S+/S- < 0.5: No significant drift
• S+/S- 0.5-1.0: Moderate drift (close to threshold)
• S+/S- > 1.0: Break detected
• Break count >100/hour: CUSUM threshold too sensitive

Panel 1.4: ADX Indicators (Time Series)

# Query 1: ADX
adx{symbol="ES.FUT"}

# Query 2: +DI
plus_di{symbol="ES.FUT"}

# Query 3: -DI
minus_di{symbol="ES.FUT"}

# Visualization: Time Series (3 series)
# Thresholds:
#   - ADX <20: Weak trend (gray zone)
#   - ADX 20-40: Moderate trend (yellow zone)
#   - ADX >40: Strong trend (green zone)

Purpose: Validate trend detection and directional movement.

Interpretation:

• +DI > -DI: Bullish pressure
• -DI > +DI: Bearish pressure
• ADX rising: Trend strengthening
• ADX falling: Trend weakening

Panel 1.5: Transition Probability Matrix (Heat Map)

# Query
transition_probability{from_regime=~".*", to_regime=~".*"}

# Visualization: Heat Map (N×N matrix)
# Color: Blue (low prob) → Red (high prob)

Purpose: Visualize regime transition patterns.

Interpretation:

• Diagonal (P(i→i)): High stability (dark red)
• Off-diagonal: Rare transitions (light blue)
• Crisis row: Typically transitions to Normal/Trending (recovery)

Panel 1.6: Regime Duration Distribution (Histogram)

# Query
histogram_quantile(0.5, regime_duration_bars{symbol="ES.FUT"})
histogram_quantile(0.75, regime_duration_bars{symbol="ES.FUT"})
histogram_quantile(0.95, regime_duration_bars{symbol="ES.FUT"})

# Visualization: Stat (3 values: P50, P75, P95)
# Unit: bars

Purpose: Understand typical regime lifetimes.

Interpretation:

• Trending: Longest duration (P50 ~30 bars)
• Crisis: Shortest duration (P50 ~5 bars)
• Normal: Moderate duration (P50 ~15 bars)

---

Dashboard 2: Wave D - Adaptive Strategies

Import Path: grafana/dashboards/wave_d_adaptive_strategies.json Refresh Interval: 30 seconds

Panel 2.1: Position Size Multiplier (Time Series)

# Query
position_multiplier{symbol="ES.FUT"}

# Visualization: Time Series
# Expected range: [0.2, 1.5]
# Alert: <0.1 or >2.0 (out of range)

Purpose: Track dynamic position sizing adjustments.

Interpretation:

• 1.5x: Trending regime (aggressive sizing)
• 1.0x: Normal regime (baseline)
• 0.5x: HighVolatility regime (risk reduction)
• 0.2x: Crisis regime (extreme risk reduction)

Panel 2.2: Stop-Loss Multiplier (Time Series)

# Query
stoploss_multiplier{symbol="ES.FUT"}

# Visualization: Time Series
# Expected range: [1.5, 4.0] × ATR
# Alert: <1.0 or >5.0 (out of range)

Purpose: Monitor dynamic stop-loss adjustments.

Interpretation:

• 4.0x ATR: Crisis regime (very wide stops)
• 2.5x ATR: Trending/Bear regime (wide stops)
• 2.0x ATR: Normal/Bull regime (standard stops)
• 1.5x ATR: Sideways regime (tight stops)

Panel 2.3: Risk Budget Utilization (Gauge)

# Query
risk_budget_utilization{symbol="ES.FUT"}

# Visualization: Gauge
# Thresholds:
#   - <50%: Green (safe)
#   - 50-80%: Yellow (moderate)
#   - 80-95%: Orange (high)
#   - >95%: Red (critical)

Purpose: Monitor risk exposure relative to regime-adjusted limits.

Interpretation:

• <50%: Underutilized capital
• 50-80%: Optimal range
• 80-95%: High utilization (monitor closely)
• >95%: Near limit (reduce position or widen stops)

Panel 2.4: Regime-Conditioned Sharpe Ratio (Table)

# Query
regime_sharpe{symbol="ES.FUT", regime=~".*"}

# Visualization: Table
# Group by: regime
# Sort by: regime_sharpe descending

Purpose: Compare strategy performance across regimes.

Expected Values:

• Trending: Sharpe 1.5-2.5 (best performance)
• Normal: Sharpe 1.0-1.5 (baseline)
• Sideways: Sharpe 0.5-1.0 (mean reversion)
• HighVolatility: Sharpe 0.0-0.5 (breakeven)
• Crisis: Sharpe -0.5-0.0 (capital preservation)

Panel 2.5: PnL by Regime (Bar Chart)

# Query
sum(regime_pnl{symbol="ES.FUT"}) by (regime)

# Visualization: Bar Chart
# X-axis: Regime
# Y-axis: Total PnL ($)

Purpose: Identify most profitable regimes.

Alert Conditions:

• Crisis PnL < -$10,000 → review risk limits
• Trending PnL < 0 → investigate trend-following strategy

Panel 2.6: Win Rate by Regime (Table)

# Query
win_rate{symbol="ES.FUT", regime=~".*"}

# Visualization: Table
# Format: Percentage (2 decimals)

Purpose: Validate strategy effectiveness per regime.

Expected Values:

• Overall: >55%
• Trending: 60-70% (trend-following advantage)
• Normal: 50-60% (baseline)
• Sideways: 45-55% (mean reversion challenges)
• Crisis: 30-50% (capital preservation mode)

---

Dashboard 3: Wave D - Feature Extraction Performance

Import Path: grafana/dashboards/wave_d_feature_performance.json Refresh Interval: 10 seconds

Panel 3.1: Feature Extraction Latency (Time Series)

# Query 1: P50
histogram_quantile(0.50, wave_d_feature_extraction_duration_seconds)

# Query 2: P90
histogram_quantile(0.90, wave_d_feature_extraction_duration_seconds)

# Query 3: P99
histogram_quantile(0.99, wave_d_feature_extraction_duration_seconds)

# Visualization: Time Series (3 series)
# Unit: microseconds (μs)
# Target: P99 <50μs
# Alert: P99 >100μs

Purpose: Monitor feature extraction performance.

Interpretation:

• P50 <10μs: Excellent performance
• P90 <30μs: Good performance
• P99 <50μs: Target met
• P99 >100μs: Performance degradation (investigate)

Panel 3.2: Feature Extraction Throughput (Stat)

# Query
rate(wave_d_feature_extraction_total[1m])

# Visualization: Stat
# Unit: bars per second
# Expected: >1000 bars/sec

Purpose: Validate feature extraction throughput under load.

Alert Conditions:

• <100 bars/sec → bottleneck in pipeline
• <10 bars/sec → critical performance issue

Panel 3.3: Feature NaN/Inf Count (Time Series)

# Query
wave_d_feature_nan_count + wave_d_feature_inf_count

# Visualization: Time Series
# Unit: count
# Alert: >0 (data quality issue)

Purpose: Detect numerical instability in feature calculations.

Expected Value: Always 0

Alert Conditions:

• 0 → immediate investigation required

• Identify feature index via logs: wave_d_feature_nan_count{feature_index="XXX"}

Panel 3.4: Feature Distribution Validation (Histogram)

# Query
wave_d_feature_value{feature_index=~"20[0-9]|21[0-9]|22[0-5]"}

# Visualization: Histogram (24 series, one per Wave D feature)
# Group by: feature_index

Purpose: Validate feature value ranges.

Expected Ranges:

• 201-202 (CUSUM S+/S-): [0.0, 1.5]
• 203 (Break Indicator): {0.0, 1.0}
• 204 (Direction): {-1.0, 0.0, 1.0}
• 205 (Time Since Break): [0.0, 100.0]
• 211-214 (ADX, DI, DX): [0, 100]
• 215 (Trend Classification): {0, 1, 2}
• 216, 220 (Stability, Change Prob): [0.0, 1.0]
• 218 (Shannon Entropy): [0, log₂(8)]
• 221 (Position Mult): [0.2, 1.5]
• 222 (Stop-Loss Mult): [1.5, 4.0]
• 224 (Risk Budget): [0.0, 1.0]

---

Prometheus Metrics

Regime Detection Metrics

# Current regime classification
current_regime{symbol="ES.FUT"} 2.0  # 0=Normal, 1=Trending, 2=Bull, etc.

# Regime transitions counter
regime_transitions_total{symbol="ES.FUT", from="Normal", to="Trending"} 15

# CUSUM statistics
cusum_s_plus{symbol="ES.FUT"} 0.45
cusum_s_minus{symbol="ES.FUT"} 0.12
cusum_break_count{symbol="ES.FUT"} 8
cusum_threshold{symbol="ES.FUT"} 4.0

# ADX indicators
adx{symbol="ES.FUT"} 32.5
plus_di{symbol="ES.FUT"} 28.3
minus_di{symbol="ES.FUT"} 15.7
trend_classification{symbol="ES.FUT"} 1.0  # 0=weak, 1=moderate, 2=strong

# Transition probabilities
transition_probability{symbol="ES.FUT", from="Normal", to="Normal"} 0.72
transition_probability{symbol="ES.FUT", from="Normal", to="Trending"} 0.18
stability_prob{symbol="ES.FUT"} 0.72
expected_duration{symbol="ES.FUT"} 3.6
shannon_entropy{symbol="ES.FUT"} 0.54

# Regime duration histogram
regime_duration_bars{symbol="ES.FUT", regime="Trending", le="10"} 5
regime_duration_bars{symbol="ES.FUT", regime="Trending", le="20"} 12
regime_duration_bars{symbol="ES.FUT", regime="Trending", le="+Inf"} 25

Adaptive Strategy Metrics

# Position sizing
position_multiplier{symbol="ES.FUT"} 1.5
current_position_size{symbol="ES.FUT"} 75000.0
max_position_size{symbol="ES.FUT"} 100000.0
risk_budget_utilization{symbol="ES.FUT"} 0.50

# Stop-loss
stoploss_multiplier{symbol="ES.FUT"} 2.5
atr_value{symbol="ES.FUT"} 12.50
stop_distance{symbol="ES.FUT"} 31.25

# Performance tracking
regime_sharpe{symbol="ES.FUT", regime="Trending"} 1.82
regime_pnl{symbol="ES.FUT", regime="Trending"} 3250.00
trade_count{symbol="ES.FUT", regime="Trending"} 8
win_rate{symbol="ES.FUT", regime="Trending"} 0.625

Feature Extraction Metrics

# Latency histogram
wave_d_feature_extraction_duration_seconds{le="0.00001"} 5432  # <10μs
wave_d_feature_extraction_duration_seconds{le="0.00005"} 9876  # <50μs
wave_d_feature_extraction_duration_seconds{le="0.0001"} 9950   # <100μs
wave_d_feature_extraction_duration_seconds{le="+Inf"} 10000

# Throughput counter
wave_d_feature_extraction_total 1234567

# Data quality
wave_d_feature_nan_count{feature_index="201"} 0
wave_d_feature_inf_count{feature_index="201"} 0
wave_d_feature_value{symbol="ES.FUT", feature_index="201"} 0.45

---

Alert Thresholds

Critical Alerts (Pager Duty)

1. Feature Data Quality Issue

alert: FeatureDataQualityIssue
expr: wave_d_feature_nan_count > 0 OR wave_d_feature_inf_count > 0
for: 1m
severity: critical
description: "Wave D features contain NaN or Inf values"
impact: "ML models will fail, trading halted"
action: |
  1. Check logs: `grep "Invalid feature value" /var/log/foxhunt/ml_training.log`
  2. Identify feature index: `wave_d_feature_nan_count{feature_index="XXX"}`
  3. Review feature calculation code for division by zero or sqrt(negative)
  4. Rollback to Wave C features if unable to fix quickly

2. Position Size Multiplier Out of Range

alert: PositionSizeMultiplierOutOfRange
expr: position_multiplier < 0.1 OR position_multiplier > 2.0
for: 1m
severity: critical
description: "Position multiplier outside expected range [0.2, 1.5]"
impact: "Risk management compromised, potential over-leveraging"
action: |
  1. Check regime classification: `tli trade ml regime-status --symbol ES.FUT`
  2. Verify position multiplier config: `grep POSITION_MULTIPLIERS ml/src/features/regime_adaptive.rs`
  3. Emergency: reduce all positions by 50%
  4. Investigate regime detection accuracy

3. Stop-Loss Multiplier Out of Range

alert: StopLossMultiplierOutOfRange
expr: stoploss_multiplier < 1.0 OR stoploss_multiplier > 5.0
for: 1m
severity: critical
description: "Stop-loss multiplier outside expected range [1.5, 4.0]"
impact: "Risk management compromised, stops too tight or too wide"
action: |
  1. Check ATR calculation: `tli trade ml adaptive-params --symbol ES.FUT`
  2. Verify stop-loss multiplier config: `grep STOPLOSS_MULTIPLIERS ml/src/features/regime_adaptive.rs`
  3. Emergency: manually set stops to 2.0x ATR
  4. Investigate regime transition logic

Warning Alerts (Slack/Email)

4. Regime Flip-Flopping Detected

alert: RegimeFlipFloppingDetected
expr: rate(regime_transitions_total[1h]) > 50
for: 5m
severity: warning
description: ">50 regime transitions per hour (flip-flopping)"
impact: "Excessive order placements/cancellations, increased slippage"
action: |
  1. Review Grafana dashboard: "Wave D - Regime Detection"
  2. Increase stability window: `pub const STABILITY_WINDOW: usize = 10;`
  3. Reduce CUSUM weight: `cusum: 0.25` (from 0.40)
  4. Increase CUSUM threshold: `cusum_threshold: 5.0` (from 4.0)
  5. Deploy config update and monitor for 1 hour

5. CUSUM False Positive Spike

alert: CUSUMFalsePositiveSpike
expr: rate(cusum_break_count[1h]) > 100
for: 5m
severity: warning
description: ">100 structural breaks per hour (false positive spike)"
impact: "Regime detection oversensitivity, frequent Crisis regime misclassification"
action: |
  1. Check current CUSUM threshold: `cusum_threshold{symbol="ES.FUT"}`
  2. Increase threshold: `cusum_threshold: 5.0` or `6.0`
  3. Increase drift allowance: `cusum_drift_allowance: 0.75` (from 0.5)
  4. Run test: `cargo test -p ml --test cusum_test -- test_false_positive_rate`
  5. Expected: false positive rate <0.5%

6. ADX Initialization Failure

alert: ADXInitializationFailure
expr: adx{symbol!=""} == 0 AND up{job="trading_agent_service"} == 1
for: 10m
severity: warning
description: "ADX stuck at 0.0 despite 10+ minutes of data"
impact: "Trend classification unavailable, falling back to other classifiers"
action: |
  1. Check bar count: `tli trade ml regime-status --symbol ES.FUT`
  2. Verify data ingestion: `psql -c "SELECT COUNT(*) FROM market_data WHERE symbol='ES.FUT' AND timestamp > NOW() - INTERVAL '10 minutes';"`
  3. If bar count <28: wait for initialization
  4. If bar count >28: investigate ADX calculation bug
  5. Review logs: `grep "ADX not initialized" /var/log/foxhunt/trading_agent.log`

7. Risk Budget Overutilization

alert: RiskBudgetOverutilization
expr: risk_budget_utilization > 0.95
for: 5m
severity: warning
description: "Risk budget >95% utilized"
impact: "Near position limits, reduced flexibility for new signals"
action: |
  1. Review current positions: `tli trade positions --status OPEN`
  2. Check regime: `tli trade ml regime-status --symbol ES.FUT`
  3. Options:
     a. Reduce position size by 20%
     b. Widen stop-loss to lower risk per share
     c. Close low-conviction trades
  4. Monitor for regime transition (may auto-adjust)

8. Feature Extraction Latency High

alert: FeatureExtractionLatencyHigh
expr: histogram_quantile(0.99, wave_d_feature_extraction_duration_seconds) > 0.0001
for: 5m
severity: warning
description: "Wave D feature extraction P99 latency >100μs (target: <50μs)"
impact: "Increased order submission latency, potential missed opportunities"
action: |
  1. Profile feature extraction: `cargo flamegraph -p ml --test feature_extraction_bench`
  2. Check CPU usage: `top -p $(pgrep trading_agent)`
  3. Investigate:
     - Excessive allocations in feature calculations
     - ATR calculation inefficiency
     - Regime transition matrix updates
  4. Optimize hot paths (cache intermediate results)
  5. Consider pre-computing static features

---

Logging Best Practices

Log Levels

• ERROR: System failures, data corruption, unrecoverable errors
• WARN: Degraded performance, missing data, recoverable errors
• INFO: Normal operations, regime transitions, adaptive adjustments
• DEBUG: Detailed diagnostics, feature values, intermediate calculations
• TRACE: Fine-grained execution flow (disabled in production)

Structured Logging Format

Use structured logging with key-value pairs for easy parsing and filtering.

Example (Rust with tracing crate):

use tracing::{info, warn, error};

// Regime transition (INFO)
info!(
    symbol = %symbol,
    from_regime = %old_regime,
    to_regime = %new_regime,
    confidence = %confidence,
    duration_bars = %duration,
    cusum_s_plus = %cusum_s_plus,
    cusum_s_minus = %cusum_s_minus,
    adx = %adx,
    "Regime transition detected"
);

// Adaptive strategy adjustment (INFO)
info!(
    symbol = %symbol,
    regime = %regime,
    old_position_mult = %old_mult,
    new_position_mult = %new_mult,
    old_stop_mult = %old_stop,
    new_stop_mult = %new_stop,
    risk_budget_util = %risk_budget,
    "Adaptive strategy parameters updated"
);

// Feature extraction error (ERROR)
error!(
    symbol = %symbol,
    feature_index = %idx,
    feature_name = %name,
    value = %value,
    error = %err,
    "Invalid feature value detected (NaN/Inf)"
);

// CUSUM false positive warning (WARN)
warn!(
    symbol = %symbol,
    cusum_threshold = %threshold,
    break_count_1h = %count,
    false_positive_rate = %rate,
    "CUSUM false positive rate exceeds 5% threshold"
);

// ADX initialization debug (DEBUG)
debug!(
    symbol = %symbol,
    bar_count = %count,
    bars_needed = %(28 - count),
    "ADX not yet initialized"
);

Log Aggregation (ELK Stack)

Elasticsearch Query Examples:

// Find all regime transitions to Crisis in last 24 hours
{
  "query": {
    "bool": {
      "must": [
        {"match": {"message": "Regime transition detected"}},
        {"match": {"to_regime": "Crisis"}},
        {"range": {"@timestamp": {"gte": "now-24h"}}}
      ]
    }
  }
}

// Find all feature NaN/Inf errors
{
  "query": {
    "bool": {
      "must": [
        {"match": {"level": "ERROR"}},
        {"match": {"message": "Invalid feature value detected"}},
        {"exists": {"field": "feature_index"}}
      ]
    }
  },
  "aggs": {
    "by_feature": {
      "terms": {"field": "feature_index"}
    }
  }
}

// Find all high-latency feature extractions (>100μs)
{
  "query": {
    "bool": {
      "must": [
        {"match": {"message": "Feature extraction completed"}},
        {"range": {"duration_us": {"gte": 100}}}
      ]
    }
  },
  "aggs": {
    "avg_latency": {"avg": {"field": "duration_us"}}
  }
}

---

Performance Monitoring

Latency Percentiles

Target SLOs:

• P50 (Median): <10μs per feature
• P90: <30μs per feature
• P99: <50μs per feature
• P99.9: <100μs per feature

Measurement:

use std::time::Instant;

let start = Instant::now();
let features = regime_adaptive.update(regime, return_value, position, &bars);
let duration = start.elapsed();

// Log latency
debug!(
    symbol = %symbol,
    duration_us = %duration.as_micros(),
    "Feature extraction completed"
);

// Emit metric
metrics::histogram!("wave_d_feature_extraction_duration_seconds", duration.as_secs_f64());

Grafana Query:

histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m]))

Throughput Monitoring

Target: >1000 bars/sec per symbol

Measurement:

// Increment counter on each feature extraction
metrics::counter!("wave_d_feature_extraction_total", 1, "symbol" => symbol.clone());

Grafana Query:

rate(wave_d_feature_extraction_total[1m])

Memory Usage

Target: <500KB per symbol (all Wave D state)

Measurement:

use std::mem::size_of_val;

let regime_cusum_size = size_of_val(&regime_cusum_features);
let adx_size = size_of_val(&adx_extractor);
let transition_size = size_of_val(&transition_features);
let adaptive_size = size_of_val(&adaptive_features);

let total_size = regime_cusum_size + adx_size + transition_size + adaptive_size;

info!(
    symbol = %symbol,
    total_size_kb = %(total_size / 1024),
    "Wave D memory usage"
);

Expected Sizes:

• RegimeCUSUMFeatures: ~1KB (VecDeque with 100 StructuralBreak capacity)
• AdxFeatureExtractor: ~320 bytes
• TransitionProbabilityFeatures: ~8KB (N×N transition matrix, N=8)
• RegimeAdaptiveFeatures: ~200 bytes (returns window + scalars)
• Total: ~10KB per symbol

---

Data Quality Checks

Feature Value Validation

Automated Tests (run every 5 minutes in production):

# Test script: scripts/validate_wave_d_features.sh
#!/bin/bash

# Extract latest 100 features from database
psql -U foxhunt -d foxhunt -c "
  SELECT feature_index, feature_value
  FROM ml_features
  WHERE symbol='ES.FUT'
    AND timestamp > NOW() - INTERVAL '5 minutes'
    AND feature_index BETWEEN 201 AND 225
  ORDER BY timestamp DESC
  LIMIT 2500;  -- 100 bars × 25 features
" -t -A -F "," > /tmp/wave_d_features.csv

# Validate feature ranges with Python
python3 <<EOF
import pandas as pd

df = pd.read_csv('/tmp/wave_d_features.csv', names=['index', 'value'])

# Check for NaN/Inf
nan_count = df['value'].isna().sum()
inf_count = (df['value'] == float('inf')).sum() + (df['value'] == float('-inf')).sum()

if nan_count > 0 or inf_count > 0:
    print(f"ERROR: {nan_count} NaN, {inf_count} Inf values detected")
    exit(1)

# Validate ranges
errors = []

# CUSUM (201-202): [0.0, 1.5]
cusum = df[df['index'].isin([201, 202])]
if (cusum['value'] < 0.0).any() or (cusum['value'] > 1.5).any():
    errors.append("CUSUM S+/S- out of range [0.0, 1.5]")

# ADX (211-214): [0, 100]
adx = df[df['index'].isin([211, 212, 213, 214])]
if (adx['value'] < 0.0).any() or (adx['value'] > 100.0).any():
    errors.append("ADX indicators out of range [0, 100]")

# Transition probabilities (216, 220): [0.0, 1.0]
probs = df[df['index'].isin([216, 220])]
if (probs['value'] < 0.0).any() or (probs['value'] > 1.0).any():
    errors.append("Transition probabilities out of range [0.0, 1.0]")

# Position multiplier (221): [0.2, 1.5]
pos_mult = df[df['index'] == 221]
if (pos_mult['value'] < 0.2).any() or (pos_mult['value'] > 1.5).any():
    errors.append("Position multiplier out of range [0.2, 1.5]")

# Risk budget (224): [0.0, 1.0]
risk = df[df['index'] == 224]
if (risk['value'] < 0.0).any() or (risk['value'] > 1.0).any():
    errors.append("Risk budget out of range [0.0, 1.0]")

if errors:
    print("ERRORS:")
    for error in errors:
        print(f"  - {error}")
    exit(1)

print("All Wave D features valid")
EOF

# Check exit code
if [ $? -eq 0 ]; then
  echo "$(date): Wave D feature validation PASSED" >> /var/log/foxhunt/feature_validation.log
else
  echo "$(date): Wave D feature validation FAILED" >> /var/log/foxhunt/feature_validation.log
  # Send alert
  curl -X POST https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
    -d '{"text": "🚨 Wave D feature validation FAILED"}'
fi

Cron Job:

*/5 * * * * /opt/foxhunt/scripts/validate_wave_d_features.sh

---

Operational Playbooks

Playbook 1: Regime Flip-Flopping

Symptoms:

• Alert: RegimeFlipFloppingDetected
• Grafana: >50 transitions per hour
• Trading: Excessive order placements/cancellations

Root Cause: Stability filter misconfiguration or high-frequency noise.

Diagnosis:

# 1. Check transition frequency
tli trade ml regime-transitions --symbol ES.FUT --limit 100

# 2. Identify transition pattern
psql -U foxhunt -d foxhunt -c "
  SELECT from_regime, to_regime, COUNT(*)
  FROM regime_transitions
  WHERE symbol='ES.FUT' AND timestamp > NOW() - INTERVAL '1 hour'
  GROUP BY from_regime, to_regime
  ORDER BY COUNT(*) DESC;
"

# 3. Check CUSUM sensitivity
tli trade ml regime-status --symbol ES.FUT | grep "cusum"

Resolution:

// Option 1: Increase stability window
pub const STABILITY_WINDOW: usize = 10; // From 5 to 10

// Option 2: Reduce CUSUM weight
pub const REGIME_CLASSIFIER_WEIGHTS: RegimeWeights = RegimeWeights {
    cusum: 0.25,    // From 0.40 to 0.25
    trending: 0.35, // From 0.30 to 0.35
    ranging: 0.25,  // From 0.20 to 0.25
    volatile: 0.15, // From 0.10 to 0.15
};

// Option 3: Increase CUSUM threshold
pub const CUSUM_THRESHOLD: f64 = 5.0; // From 4.0 to 5.0

Deployment:

# 1. Update config
vim ml/src/features/config.rs

# 2. Rebuild
cargo build -p trading_agent_service --release

# 3. Stop trading
tli trade ml stop

# 4. Deploy
systemctl restart trading_agent_service

# 5. Monitor for 1 hour
watch -n 60 'tli trade ml regime-transitions --symbol ES.FUT --limit 10'

Verification:

• Transition frequency <20 per hour
• Prometheus: rate(regime_transitions_total[1h]) < 20

---

Playbook 2: CUSUM False Positive Spike

Symptoms:

• Alert: CUSUMFalsePositiveSpike
• Grafana: >100 breaks per hour
• Logs: Frequent "Structural break detected" messages

Root Cause: CUSUM threshold too low for current market volatility.

Diagnosis:

# 1. Check break frequency
psql -U foxhunt -d foxhunt -c "
  SELECT COUNT(*)
  FROM regime_transitions
  WHERE symbol='ES.FUT'
    AND timestamp > NOW() - INTERVAL '1 hour'
    AND (from_regime != to_regime OR cusum_break_count > 0);
"

# 2. Check current CUSUM parameters
grep "cusum_threshold\|cusum_drift" ml/src/features/config.rs

# 3. Measure current market volatility
tli trade ml adaptive-params --symbol ES.FUT | grep "ATR"

Resolution:

// Increase CUSUM threshold (less sensitive)
pub const CUSUM_THRESHOLD: f64 = 5.0; // From 4.0

// OR increase drift allowance (more tolerance)
pub const CUSUM_DRIFT_ALLOWANCE: f64 = 0.75; // From 0.5

Deployment:

# Same steps as Playbook 1

Verification:

• Break count <10 per hour
• False positive rate <0.5%
• Run test: cargo test -p ml --test cusum_test -- test_false_positive_rate

---

Playbook 3: Feature NaN/Inf Detected

Symptoms:

• Alert: FeatureDataQualityIssue
• Grafana: wave_d_feature_nan_count > 0 or wave_d_feature_inf_count > 0
• ML models: Training loss NaN

Root Cause: Division by zero or numerical instability.

Diagnosis:

# 1. Identify problematic feature
psql -U foxhunt -d foxhunt -c "
  SELECT feature_index, COUNT(*)
  FROM ml_features
  WHERE symbol='ES.FUT'
    AND timestamp > NOW() - INTERVAL '1 hour'
    AND (feature_value IS NULL OR feature_value = 'NaN' OR feature_value = 'Infinity')
  GROUP BY feature_index
  ORDER BY COUNT(*) DESC;
"

# 2. Check logs for error details
grep "Invalid feature value" /var/log/foxhunt/ml_training.log | tail -20

# 3. Review feature calculation code
# Feature 223 (Regime Sharpe) → ml/src/features/regime_adaptive.rs:100-110
# Feature 224 (Risk Budget) → ml/src/features/regime_adaptive.rs:120-130

Resolution:

Common Fix 1: Sharpe Ratio (Feature 223)

// Add zero volatility check
let std = variance.sqrt();
if std > 1e-10 {
    (mean / std) * (252.0_f64).sqrt()
} else {
    0.0 // Return 0.0 instead of NaN
}

Common Fix 2: Risk Budget (Feature 224)

// Add zero position check
if self.max_position_size > 1e-10 {
    (self.current_position_size / (position_mult * self.max_position_size))
        .clamp(0.0, 1.0)
} else {
    0.0
}

Common Fix 3: Shannon Entropy (Feature 218)

// Filter zero probabilities before log
.filter(|&p| p > 1e-10) // Add this line
.map(|p| -p * p.log2())

Deployment:

# 1. Apply fix to affected feature
vim ml/src/features/regime_adaptive.rs

# 2. Run unit tests
cargo test -p ml --lib features::regime_adaptive -- test_all_features_finite

# 3. Rebuild and deploy
cargo build --workspace --release
systemctl restart ml_training_service
systemctl restart trading_agent_service

# 4. Monitor for 10 minutes
watch -n 60 'psql -U foxhunt -d foxhunt -t -c "SELECT COUNT(*) FROM ml_features WHERE feature_index BETWEEN 201 AND 225 AND (feature_value IS NULL OR feature_value = '"'"'NaN'"'"' OR feature_value = '"'"'Infinity'"'"');"'

Verification:

• wave_d_feature_nan_count == 0
• wave_d_feature_inf_count == 0
• Test: cargo test -p ml -- test_all_features_finite

---

Document Version: 1.0 Last Updated: 2025-10-18 Status: 🟢 Production Ready