foxhunt/WAVE_D_MONITORING_GUIDE.md

# Wave D Monitoring Guide

**Version**: 1.0
**Date**: 2025-10-18
**Status**: 🟢 **Production Ready**

---

## Table of Contents

1. [Overview](#overview)
2. [Grafana Dashboards](#grafana-dashboards)
3. [Prometheus Metrics](#prometheus-metrics)
4. [Alert Thresholds](#alert-thresholds)
5. [Logging Best Practices](#logging-best-practices)
6. [Performance Monitoring](#performance-monitoring)
7. [Data Quality Checks](#data-quality-checks)
8. [Operational Playbooks](#operational-playbooks)

---

## Overview

Wave D monitoring covers three critical areas:
1. **Regime Detection**: Track regime transitions, classifier performance, and stability
2. **Adaptive Strategies**: Monitor position sizing, stop-loss adjustments, and risk utilization
3. **Feature Extraction**: Validate feature quality, latency, and data integrity

### Key Metrics Summary

| Metric Category | Target | Alert Threshold | Dashboard |
|----------------|--------|-----------------|-----------|
| **Regime Transitions** | 5-10/day | >50/hour | Regime Detection |
| **CUSUM False Positives** | <0.5% | >100/hour | Regime Detection |
| **ADX Initialization** | <28 bars | Stuck at 0 for >10min | Regime Detection |
| **Feature Extraction Latency** | P99 <50μs | P99 >100μs | Feature Performance |
| **Feature NaN/Inf Count** | 0 | >0 | Feature Performance |
| **Position Multiplier** | [0.2, 1.5] | <0.1 or >2.0 | Adaptive Strategies |
| **Stop-Loss Multiplier** | [1.5, 4.0] | <1.0 or >5.0 | Adaptive Strategies |
| **Risk Budget Utilization** | <80% | >95% | Adaptive Strategies |

---

## Grafana Dashboards

### Dashboard 1: Wave D - Regime Detection

**Import Path**: `grafana/dashboards/wave_d_regime_detection.json`
**Refresh Interval**: 30 seconds
**Data Source**: Prometheus

#### Panel 1.1: Current Regime (Gauge)

```promql
# Query
current_regime{symbol="ES.FUT"}

# Visualization: Gauge
# Thresholds:
#   - Normal: Green
#   - Trending: Blue
#   - Bull: Light Blue
#   - Bear: Orange
#   - Sideways: Yellow
#   - HighVolatility: Orange
#   - Crisis: Red
```

**Purpose**: Real-time regime classification for primary symbols.

**Alert Conditions**:
- Crisis regime for >4 hours → escalate to operations team
- Unknown regime → data quality issue

#### Panel 1.2: Regime Transitions Timeline (Time Series)

```promql
# Query
rate(regime_transitions_total{symbol="ES.FUT"}[5m]) * 3600

# Visualization: Time Series
# Unit: transitions per hour
# Alert: >50 transitions/hour (flip-flopping)
```

**Purpose**: Detect rapid regime switching (flip-flopping) that may indicate stability filter misconfiguration.

**Interpretation**:
- **5-10/day**: Normal behavior
- **10-30/day**: Volatile market conditions
- **>50/day**: Potential stability filter issue

#### Panel 1.3: CUSUM Statistics (Time Series)

```promql
# Query 1: S+ Normalized
cusum_s_plus{symbol="ES.FUT"} / cusum_threshold

# Query 2: S- Normalized
cusum_s_minus{symbol="ES.FUT"} / cusum_threshold

# Query 3: Break Count
increase(cusum_break_count{symbol="ES.FUT"}[1h])

# Visualization: Time Series (3 series)
# Alert: Break count >100/hour (false positive spike)
```

**Purpose**: Monitor structural break detection sensitivity.

**Interpretation**:
- **S+/S- < 0.5**: No significant drift
- **S+/S- 0.5-1.0**: Moderate drift (close to threshold)
- **S+/S- > 1.0**: Break detected
- **Break count >100/hour**: CUSUM threshold too sensitive

#### Panel 1.4: ADX Indicators (Time Series)

```promql
# Query 1: ADX
adx{symbol="ES.FUT"}

# Query 2: +DI
plus_di{symbol="ES.FUT"}

# Query 3: -DI
minus_di{symbol="ES.FUT"}

# Visualization: Time Series (3 series)
# Thresholds:
#   - ADX <20: Weak trend (gray zone)
#   - ADX 20-40: Moderate trend (yellow zone)
#   - ADX >40: Strong trend (green zone)
```

**Purpose**: Validate trend detection and directional movement.

**Interpretation**:
- **+DI > -DI**: Bullish pressure
- **-DI > +DI**: Bearish pressure
- **ADX rising**: Trend strengthening
- **ADX falling**: Trend weakening

#### Panel 1.5: Transition Probability Matrix (Heat Map)

```promql
# Query
transition_probability{from_regime=~".*", to_regime=~".*"}

# Visualization: Heat Map (N×N matrix)
# Color: Blue (low prob) → Red (high prob)
```

**Purpose**: Visualize regime transition patterns.

**Interpretation**:
- **Diagonal (P(i→i))**: High stability (dark red)
- **Off-diagonal**: Rare transitions (light blue)
- **Crisis row**: Typically transitions to Normal/Trending (recovery)

#### Panel 1.6: Regime Duration Distribution (Histogram)

```promql
# Query
histogram_quantile(0.5, regime_duration_bars{symbol="ES.FUT"})
histogram_quantile(0.75, regime_duration_bars{symbol="ES.FUT"})
histogram_quantile(0.95, regime_duration_bars{symbol="ES.FUT"})

# Visualization: Stat (3 values: P50, P75, P95)
# Unit: bars
```

**Purpose**: Understand typical regime lifetimes.

**Interpretation**:
- **Trending**: Longest duration (P50 ~30 bars)
- **Crisis**: Shortest duration (P50 ~5 bars)
- **Normal**: Moderate duration (P50 ~15 bars)

---

### Dashboard 2: Wave D - Adaptive Strategies

**Import Path**: `grafana/dashboards/wave_d_adaptive_strategies.json`
**Refresh Interval**: 30 seconds

#### Panel 2.1: Position Size Multiplier (Time Series)

```promql
# Query
position_multiplier{symbol="ES.FUT"}

# Visualization: Time Series
# Expected range: [0.2, 1.5]
# Alert: <0.1 or >2.0 (out of range)
```

**Purpose**: Track dynamic position sizing adjustments.

**Interpretation**:
- **1.5x**: Trending regime (aggressive sizing)
- **1.0x**: Normal regime (baseline)
- **0.5x**: HighVolatility regime (risk reduction)
- **0.2x**: Crisis regime (extreme risk reduction)

#### Panel 2.2: Stop-Loss Multiplier (Time Series)

```promql
# Query
stoploss_multiplier{symbol="ES.FUT"}

# Visualization: Time Series
# Expected range: [1.5, 4.0] × ATR
# Alert: <1.0 or >5.0 (out of range)
```

**Purpose**: Monitor dynamic stop-loss adjustments.

**Interpretation**:
- **4.0x ATR**: Crisis regime (very wide stops)
- **2.5x ATR**: Trending/Bear regime (wide stops)
- **2.0x ATR**: Normal/Bull regime (standard stops)
- **1.5x ATR**: Sideways regime (tight stops)

#### Panel 2.3: Risk Budget Utilization (Gauge)

```promql
# Query
risk_budget_utilization{symbol="ES.FUT"}

# Visualization: Gauge
# Thresholds:
#   - <50%: Green (safe)
#   - 50-80%: Yellow (moderate)
#   - 80-95%: Orange (high)
#   - >95%: Red (critical)
```

**Purpose**: Monitor risk exposure relative to regime-adjusted limits.

**Interpretation**:
- **<50%**: Underutilized capital
- **50-80%**: Optimal range
- **80-95%**: High utilization (monitor closely)
- **>95%**: Near limit (reduce position or widen stops)

#### Panel 2.4: Regime-Conditioned Sharpe Ratio (Table)

```promql
# Query
regime_sharpe{symbol="ES.FUT", regime=~".*"}

# Visualization: Table
# Group by: regime
# Sort by: regime_sharpe descending
```

**Purpose**: Compare strategy performance across regimes.

**Expected Values**:
- **Trending**: Sharpe 1.5-2.5 (best performance)
- **Normal**: Sharpe 1.0-1.5 (baseline)
- **Sideways**: Sharpe 0.5-1.0 (mean reversion)
- **HighVolatility**: Sharpe 0.0-0.5 (breakeven)
- **Crisis**: Sharpe -0.5-0.0 (capital preservation)

#### Panel 2.5: PnL by Regime (Bar Chart)

```promql
# Query
sum(regime_pnl{symbol="ES.FUT"}) by (regime)

# Visualization: Bar Chart
# X-axis: Regime
# Y-axis: Total PnL ($)
```

**Purpose**: Identify most profitable regimes.

**Alert Conditions**:
- Crisis PnL < -$10,000 → review risk limits
- Trending PnL < 0 → investigate trend-following strategy

#### Panel 2.6: Win Rate by Regime (Table)

```promql
# Query
win_rate{symbol="ES.FUT", regime=~".*"}

# Visualization: Table
# Format: Percentage (2 decimals)
```

**Purpose**: Validate strategy effectiveness per regime.

**Expected Values**:
- **Overall**: >55%
- **Trending**: 60-70% (trend-following advantage)
- **Normal**: 50-60% (baseline)
- **Sideways**: 45-55% (mean reversion challenges)
- **Crisis**: 30-50% (capital preservation mode)

---

### Dashboard 3: Wave D - Feature Extraction Performance

**Import Path**: `grafana/dashboards/wave_d_feature_performance.json`
**Refresh Interval**: 10 seconds

#### Panel 3.1: Feature Extraction Latency (Time Series)

```promql
# Query 1: P50
histogram_quantile(0.50, wave_d_feature_extraction_duration_seconds)

# Query 2: P90
histogram_quantile(0.90, wave_d_feature_extraction_duration_seconds)

# Query 3: P99
histogram_quantile(0.99, wave_d_feature_extraction_duration_seconds)

# Visualization: Time Series (3 series)
# Unit: microseconds (μs)
# Target: P99 <50μs
# Alert: P99 >100μs
```

**Purpose**: Monitor feature extraction performance.

**Interpretation**:
- **P50 <10μs**: Excellent performance
- **P90 <30μs**: Good performance
- **P99 <50μs**: Target met
- **P99 >100μs**: Performance degradation (investigate)

#### Panel 3.2: Feature Extraction Throughput (Stat)

```promql
# Query
rate(wave_d_feature_extraction_total[1m])

# Visualization: Stat
# Unit: bars per second
# Expected: >1000 bars/sec
```

**Purpose**: Validate feature extraction throughput under load.

**Alert Conditions**:
- <100 bars/sec → bottleneck in pipeline
- <10 bars/sec → critical performance issue

#### Panel 3.3: Feature NaN/Inf Count (Time Series)

```promql
# Query
wave_d_feature_nan_count + wave_d_feature_inf_count

# Visualization: Time Series
# Unit: count
# Alert: >0 (data quality issue)
```

**Purpose**: Detect numerical instability in feature calculations.

**Expected Value**: Always 0

**Alert Conditions**:
- >0 → immediate investigation required
- Identify feature index via logs: `wave_d_feature_nan_count{feature_index="XXX"}`

#### Panel 3.4: Feature Distribution Validation (Histogram)

```promql
# Query
wave_d_feature_value{feature_index=~"20[0-9]|21[0-9]|22[0-5]"}

# Visualization: Histogram (24 series, one per Wave D feature)
# Group by: feature_index
```

**Purpose**: Validate feature value ranges.

**Expected Ranges**:
- **201-202** (CUSUM S+/S-): [0.0, 1.5]
- **203** (Break Indicator): {0.0, 1.0}
- **204** (Direction): {-1.0, 0.0, 1.0}
- **205** (Time Since Break): [0.0, 100.0]
- **211-214** (ADX, DI, DX): [0, 100]
- **215** (Trend Classification): {0, 1, 2}
- **216, 220** (Stability, Change Prob): [0.0, 1.0]
- **218** (Shannon Entropy): [0, log₂(8)]
- **221** (Position Mult): [0.2, 1.5]
- **222** (Stop-Loss Mult): [1.5, 4.0]
- **224** (Risk Budget): [0.0, 1.0]

---

## Prometheus Metrics

### Regime Detection Metrics

```yaml
# Current regime classification
current_regime{symbol="ES.FUT"} 2.0  # 0=Normal, 1=Trending, 2=Bull, etc.

# Regime transitions counter
regime_transitions_total{symbol="ES.FUT", from="Normal", to="Trending"} 15

# CUSUM statistics
cusum_s_plus{symbol="ES.FUT"} 0.45
cusum_s_minus{symbol="ES.FUT"} 0.12
cusum_break_count{symbol="ES.FUT"} 8
cusum_threshold{symbol="ES.FUT"} 4.0

# ADX indicators
adx{symbol="ES.FUT"} 32.5
plus_di{symbol="ES.FUT"} 28.3
minus_di{symbol="ES.FUT"} 15.7
trend_classification{symbol="ES.FUT"} 1.0  # 0=weak, 1=moderate, 2=strong

# Transition probabilities
transition_probability{symbol="ES.FUT", from="Normal", to="Normal"} 0.72
transition_probability{symbol="ES.FUT", from="Normal", to="Trending"} 0.18
stability_prob{symbol="ES.FUT"} 0.72
expected_duration{symbol="ES.FUT"} 3.6
shannon_entropy{symbol="ES.FUT"} 0.54

# Regime duration histogram
regime_duration_bars{symbol="ES.FUT", regime="Trending", le="10"} 5
regime_duration_bars{symbol="ES.FUT", regime="Trending", le="20"} 12
regime_duration_bars{symbol="ES.FUT", regime="Trending", le="+Inf"} 25
```

### Adaptive Strategy Metrics

```yaml
# Position sizing
position_multiplier{symbol="ES.FUT"} 1.5
current_position_size{symbol="ES.FUT"} 75000.0
max_position_size{symbol="ES.FUT"} 100000.0
risk_budget_utilization{symbol="ES.FUT"} 0.50

# Stop-loss
stoploss_multiplier{symbol="ES.FUT"} 2.5
atr_value{symbol="ES.FUT"} 12.50
stop_distance{symbol="ES.FUT"} 31.25

# Performance tracking
regime_sharpe{symbol="ES.FUT", regime="Trending"} 1.82
regime_pnl{symbol="ES.FUT", regime="Trending"} 3250.00
trade_count{symbol="ES.FUT", regime="Trending"} 8
win_rate{symbol="ES.FUT", regime="Trending"} 0.625
```

### Feature Extraction Metrics

```yaml
# Latency histogram
wave_d_feature_extraction_duration_seconds{le="0.00001"} 5432  # <10μs
wave_d_feature_extraction_duration_seconds{le="0.00005"} 9876  # <50μs
wave_d_feature_extraction_duration_seconds{le="0.0001"} 9950   # <100μs
wave_d_feature_extraction_duration_seconds{le="+Inf"} 10000

# Throughput counter
wave_d_feature_extraction_total 1234567

# Data quality
wave_d_feature_nan_count{feature_index="201"} 0
wave_d_feature_inf_count{feature_index="201"} 0
wave_d_feature_value{symbol="ES.FUT", feature_index="201"} 0.45
```

---

## Alert Thresholds

### Critical Alerts (Pager Duty)

#### 1. Feature Data Quality Issue

```yaml
alert: FeatureDataQualityIssue
expr: wave_d_feature_nan_count > 0 OR wave_d_feature_inf_count > 0
for: 1m
severity: critical
description: "Wave D features contain NaN or Inf values"
impact: "ML models will fail, trading halted"
action: |
  1. Check logs: `grep "Invalid feature value" /var/log/foxhunt/ml_training.log`
  2. Identify feature index: `wave_d_feature_nan_count{feature_index="XXX"}`
  3. Review feature calculation code for division by zero or sqrt(negative)
  4. Rollback to Wave C features if unable to fix quickly
```

#### 2. Position Size Multiplier Out of Range

```yaml
alert: PositionSizeMultiplierOutOfRange
expr: position_multiplier < 0.1 OR position_multiplier > 2.0
for: 1m
severity: critical
description: "Position multiplier outside expected range [0.2, 1.5]"
impact: "Risk management compromised, potential over-leveraging"
action: |
  1. Check regime classification: `tli trade ml regime-status --symbol ES.FUT`
  2. Verify position multiplier config: `grep POSITION_MULTIPLIERS ml/src/features/regime_adaptive.rs`
  3. Emergency: reduce all positions by 50%
  4. Investigate regime detection accuracy
```

#### 3. Stop-Loss Multiplier Out of Range

```yaml
alert: StopLossMultiplierOutOfRange
expr: stoploss_multiplier < 1.0 OR stoploss_multiplier > 5.0
for: 1m
severity: critical
description: "Stop-loss multiplier outside expected range [1.5, 4.0]"
impact: "Risk management compromised, stops too tight or too wide"
action: |
  1. Check ATR calculation: `tli trade ml adaptive-params --symbol ES.FUT`
  2. Verify stop-loss multiplier config: `grep STOPLOSS_MULTIPLIERS ml/src/features/regime_adaptive.rs`
  3. Emergency: manually set stops to 2.0x ATR
  4. Investigate regime transition logic
```

### Warning Alerts (Slack/Email)

#### 4. Regime Flip-Flopping Detected

```yaml
alert: RegimeFlipFloppingDetected
expr: rate(regime_transitions_total[1h]) > 50
for: 5m
severity: warning
description: ">50 regime transitions per hour (flip-flopping)"
impact: "Excessive order placements/cancellations, increased slippage"
action: |
  1. Review Grafana dashboard: "Wave D - Regime Detection"
  2. Increase stability window: `pub const STABILITY_WINDOW: usize = 10;`
  3. Reduce CUSUM weight: `cusum: 0.25` (from 0.40)
  4. Increase CUSUM threshold: `cusum_threshold: 5.0` (from 4.0)
  5. Deploy config update and monitor for 1 hour
```

#### 5. CUSUM False Positive Spike

```yaml
alert: CUSUMFalsePositiveSpike
expr: rate(cusum_break_count[1h]) > 100
for: 5m
severity: warning
description: ">100 structural breaks per hour (false positive spike)"
impact: "Regime detection oversensitivity, frequent Crisis regime misclassification"
action: |
  1. Check current CUSUM threshold: `cusum_threshold{symbol="ES.FUT"}`
  2. Increase threshold: `cusum_threshold: 5.0` or `6.0`
  3. Increase drift allowance: `cusum_drift_allowance: 0.75` (from 0.5)
  4. Run test: `cargo test -p ml --test cusum_test -- test_false_positive_rate`
  5. Expected: false positive rate <0.5%
```

#### 6. ADX Initialization Failure

```yaml
alert: ADXInitializationFailure
expr: adx{symbol!=""} == 0 AND up{job="trading_agent_service"} == 1
for: 10m
severity: warning
description: "ADX stuck at 0.0 despite 10+ minutes of data"
impact: "Trend classification unavailable, falling back to other classifiers"
action: |
  1. Check bar count: `tli trade ml regime-status --symbol ES.FUT`
  2. Verify data ingestion: `psql -c "SELECT COUNT(*) FROM market_data WHERE symbol='ES.FUT' AND timestamp > NOW() - INTERVAL '10 minutes';"`
  3. If bar count <28: wait for initialization
  4. If bar count >28: investigate ADX calculation bug
  5. Review logs: `grep "ADX not initialized" /var/log/foxhunt/trading_agent.log`
```

#### 7. Risk Budget Overutilization

```yaml
alert: RiskBudgetOverutilization
expr: risk_budget_utilization > 0.95
for: 5m
severity: warning
description: "Risk budget >95% utilized"
impact: "Near position limits, reduced flexibility for new signals"
action: |
  1. Review current positions: `tli trade positions --status OPEN`
  2. Check regime: `tli trade ml regime-status --symbol ES.FUT`
  3. Options:
     a. Reduce position size by 20%
     b. Widen stop-loss to lower risk per share
     c. Close low-conviction trades
  4. Monitor for regime transition (may auto-adjust)
```

#### 8. Feature Extraction Latency High

```yaml
alert: FeatureExtractionLatencyHigh
expr: histogram_quantile(0.99, wave_d_feature_extraction_duration_seconds) > 0.0001
for: 5m
severity: warning
description: "Wave D feature extraction P99 latency >100μs (target: <50μs)"
impact: "Increased order submission latency, potential missed opportunities"
action: |
  1. Profile feature extraction: `cargo flamegraph -p ml --test feature_extraction_bench`
  2. Check CPU usage: `top -p $(pgrep trading_agent)`
  3. Investigate:
     - Excessive allocations in feature calculations
     - ATR calculation inefficiency
     - Regime transition matrix updates
  4. Optimize hot paths (cache intermediate results)
  5. Consider pre-computing static features
```

---

## Logging Best Practices

### Log Levels

- **ERROR**: System failures, data corruption, unrecoverable errors
- **WARN**: Degraded performance, missing data, recoverable errors
- **INFO**: Normal operations, regime transitions, adaptive adjustments
- **DEBUG**: Detailed diagnostics, feature values, intermediate calculations
- **TRACE**: Fine-grained execution flow (disabled in production)

### Structured Logging Format

Use structured logging with key-value pairs for easy parsing and filtering.

**Example (Rust with `tracing` crate)**:
```rust
use tracing::{info, warn, error};

// Regime transition (INFO)
info!(
    symbol = %symbol,
    from_regime = %old_regime,
    to_regime = %new_regime,
    confidence = %confidence,
    duration_bars = %duration,
    cusum_s_plus = %cusum_s_plus,
    cusum_s_minus = %cusum_s_minus,
    adx = %adx,
    "Regime transition detected"
);

// Adaptive strategy adjustment (INFO)
info!(
    symbol = %symbol,
    regime = %regime,
    old_position_mult = %old_mult,
    new_position_mult = %new_mult,
    old_stop_mult = %old_stop,
    new_stop_mult = %new_stop,
    risk_budget_util = %risk_budget,
    "Adaptive strategy parameters updated"
);

// Feature extraction error (ERROR)
error!(
    symbol = %symbol,
    feature_index = %idx,
    feature_name = %name,
    value = %value,
    error = %err,
    "Invalid feature value detected (NaN/Inf)"
);

// CUSUM false positive warning (WARN)
warn!(
    symbol = %symbol,
    cusum_threshold = %threshold,
    break_count_1h = %count,
    false_positive_rate = %rate,
    "CUSUM false positive rate exceeds 5% threshold"
);

// ADX initialization debug (DEBUG)
debug!(
    symbol = %symbol,
    bar_count = %count,
    bars_needed = %(28 - count),
    "ADX not yet initialized"
);
```

### Log Aggregation (ELK Stack)

**Elasticsearch Query Examples**:

```json
// Find all regime transitions to Crisis in last 24 hours
{
  "query": {
    "bool": {
      "must": [
        {"match": {"message": "Regime transition detected"}},
        {"match": {"to_regime": "Crisis"}},
        {"range": {"@timestamp": {"gte": "now-24h"}}}
      ]
    }
  }
}

// Find all feature NaN/Inf errors
{
  "query": {
    "bool": {
      "must": [
        {"match": {"level": "ERROR"}},
        {"match": {"message": "Invalid feature value detected"}},
        {"exists": {"field": "feature_index"}}
      ]
    }
  },
  "aggs": {
    "by_feature": {
      "terms": {"field": "feature_index"}
    }
  }
}

// Find all high-latency feature extractions (>100μs)
{
  "query": {
    "bool": {
      "must": [
        {"match": {"message": "Feature extraction completed"}},
        {"range": {"duration_us": {"gte": 100}}}
      ]
    }
  },
  "aggs": {
    "avg_latency": {"avg": {"field": "duration_us"}}
  }
}
```

---

## Performance Monitoring

### Latency Percentiles

**Target SLOs**:
- **P50 (Median)**: <10μs per feature
- **P90**: <30μs per feature
- **P99**: <50μs per feature
- **P99.9**: <100μs per feature

**Measurement**:
```rust
use std::time::Instant;

let start = Instant::now();
let features = regime_adaptive.update(regime, return_value, position, &bars);
let duration = start.elapsed();

// Log latency
debug!(
    symbol = %symbol,
    duration_us = %duration.as_micros(),
    "Feature extraction completed"
);

// Emit metric
metrics::histogram!("wave_d_feature_extraction_duration_seconds", duration.as_secs_f64());
```

**Grafana Query**:
```promql
histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m]))
```

### Throughput Monitoring

**Target**: >1000 bars/sec per symbol

**Measurement**:
```rust
// Increment counter on each feature extraction
metrics::counter!("wave_d_feature_extraction_total", 1, "symbol" => symbol.clone());
```

**Grafana Query**:
```promql
rate(wave_d_feature_extraction_total[1m])
```

### Memory Usage

**Target**: <500KB per symbol (all Wave D state)

**Measurement**:
```rust
use std::mem::size_of_val;

let regime_cusum_size = size_of_val(&regime_cusum_features);
let adx_size = size_of_val(&adx_extractor);
let transition_size = size_of_val(&transition_features);
let adaptive_size = size_of_val(&adaptive_features);

let total_size = regime_cusum_size + adx_size + transition_size + adaptive_size;

info!(
    symbol = %symbol,
    total_size_kb = %(total_size / 1024),
    "Wave D memory usage"
);
```

**Expected Sizes**:
- **RegimeCUSUMFeatures**: ~1KB (VecDeque with 100 StructuralBreak capacity)
- **AdxFeatureExtractor**: ~320 bytes
- **TransitionProbabilityFeatures**: ~8KB (N×N transition matrix, N=8)
- **RegimeAdaptiveFeatures**: ~200 bytes (returns window + scalars)
- **Total**: ~10KB per symbol

---

## Data Quality Checks

### Feature Value Validation

**Automated Tests** (run every 5 minutes in production):

```bash
# Test script: scripts/validate_wave_d_features.sh
#!/bin/bash

# Extract latest 100 features from database
psql -U foxhunt -d foxhunt -c "
  SELECT feature_index, feature_value
  FROM ml_features
  WHERE symbol='ES.FUT'
    AND timestamp > NOW() - INTERVAL '5 minutes'
    AND feature_index BETWEEN 201 AND 225
  ORDER BY timestamp DESC
  LIMIT 2500;  -- 100 bars × 25 features
" -t -A -F "," > /tmp/wave_d_features.csv

# Validate feature ranges with Python
python3 <<EOF
import pandas as pd

df = pd.read_csv('/tmp/wave_d_features.csv', names=['index', 'value'])

# Check for NaN/Inf
nan_count = df['value'].isna().sum()
inf_count = (df['value'] == float('inf')).sum() + (df['value'] == float('-inf')).sum()

if nan_count > 0 or inf_count > 0:
    print(f"ERROR: {nan_count} NaN, {inf_count} Inf values detected")
    exit(1)

# Validate ranges
errors = []

# CUSUM (201-202): [0.0, 1.5]
cusum = df[df['index'].isin([201, 202])]
if (cusum['value'] < 0.0).any() or (cusum['value'] > 1.5).any():
    errors.append("CUSUM S+/S- out of range [0.0, 1.5]")

# ADX (211-214): [0, 100]
adx = df[df['index'].isin([211, 212, 213, 214])]
if (adx['value'] < 0.0).any() or (adx['value'] > 100.0).any():
    errors.append("ADX indicators out of range [0, 100]")

# Transition probabilities (216, 220): [0.0, 1.0]
probs = df[df['index'].isin([216, 220])]
if (probs['value'] < 0.0).any() or (probs['value'] > 1.0).any():
    errors.append("Transition probabilities out of range [0.0, 1.0]")

# Position multiplier (221): [0.2, 1.5]
pos_mult = df[df['index'] == 221]
if (pos_mult['value'] < 0.2).any() or (pos_mult['value'] > 1.5).any():
    errors.append("Position multiplier out of range [0.2, 1.5]")

# Risk budget (224): [0.0, 1.0]
risk = df[df['index'] == 224]
if (risk['value'] < 0.0).any() or (risk['value'] > 1.0).any():
    errors.append("Risk budget out of range [0.0, 1.0]")

if errors:
    print("ERRORS:")
    for error in errors:
        print(f"  - {error}")
    exit(1)

print("All Wave D features valid")
EOF

# Check exit code
if [ $? -eq 0 ]; then
  echo "$(date): Wave D feature validation PASSED" >> /var/log/foxhunt/feature_validation.log
else
  echo "$(date): Wave D feature validation FAILED" >> /var/log/foxhunt/feature_validation.log
  # Send alert
  curl -X POST https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
    -d '{"text": "🚨 Wave D feature validation FAILED"}'
fi
```

**Cron Job**:
```cron
*/5 * * * * /opt/foxhunt/scripts/validate_wave_d_features.sh
```

---

## Operational Playbooks

### Playbook 1: Regime Flip-Flopping

**Symptoms**:
- Alert: `RegimeFlipFloppingDetected`
- Grafana: >50 transitions per hour
- Trading: Excessive order placements/cancellations

**Root Cause**: Stability filter misconfiguration or high-frequency noise.

**Diagnosis**:
```bash
# 1. Check transition frequency
tli trade ml regime-transitions --symbol ES.FUT --limit 100

# 2. Identify transition pattern
psql -U foxhunt -d foxhunt -c "
  SELECT from_regime, to_regime, COUNT(*)
  FROM regime_transitions
  WHERE symbol='ES.FUT' AND timestamp > NOW() - INTERVAL '1 hour'
  GROUP BY from_regime, to_regime
  ORDER BY COUNT(*) DESC;
"

# 3. Check CUSUM sensitivity
tli trade ml regime-status --symbol ES.FUT | grep "cusum"
```

**Resolution**:
```rust
// Option 1: Increase stability window
pub const STABILITY_WINDOW: usize = 10; // From 5 to 10

// Option 2: Reduce CUSUM weight
pub const REGIME_CLASSIFIER_WEIGHTS: RegimeWeights = RegimeWeights {
    cusum: 0.25,    // From 0.40 to 0.25
    trending: 0.35, // From 0.30 to 0.35
    ranging: 0.25,  // From 0.20 to 0.25
    volatile: 0.15, // From 0.10 to 0.15
};

// Option 3: Increase CUSUM threshold
pub const CUSUM_THRESHOLD: f64 = 5.0; // From 4.0 to 5.0
```

**Deployment**:
```bash
# 1. Update config
vim ml/src/features/config.rs

# 2. Rebuild
cargo build -p trading_agent_service --release

# 3. Stop trading
tli trade ml stop

# 4. Deploy
systemctl restart trading_agent_service

# 5. Monitor for 1 hour
watch -n 60 'tli trade ml regime-transitions --symbol ES.FUT --limit 10'
```

**Verification**:
- Transition frequency <20 per hour
- Prometheus: `rate(regime_transitions_total[1h]) < 20`

---

### Playbook 2: CUSUM False Positive Spike

**Symptoms**:
- Alert: `CUSUMFalsePositiveSpike`
- Grafana: >100 breaks per hour
- Logs: Frequent "Structural break detected" messages

**Root Cause**: CUSUM threshold too low for current market volatility.

**Diagnosis**:
```bash
# 1. Check break frequency
psql -U foxhunt -d foxhunt -c "
  SELECT COUNT(*)
  FROM regime_transitions
  WHERE symbol='ES.FUT'
    AND timestamp > NOW() - INTERVAL '1 hour'
    AND (from_regime != to_regime OR cusum_break_count > 0);
"

# 2. Check current CUSUM parameters
grep "cusum_threshold\|cusum_drift" ml/src/features/config.rs

# 3. Measure current market volatility
tli trade ml adaptive-params --symbol ES.FUT | grep "ATR"
```

**Resolution**:
```rust
// Increase CUSUM threshold (less sensitive)
pub const CUSUM_THRESHOLD: f64 = 5.0; // From 4.0

// OR increase drift allowance (more tolerance)
pub const CUSUM_DRIFT_ALLOWANCE: f64 = 0.75; // From 0.5
```

**Deployment**:
```bash
# Same steps as Playbook 1
```

**Verification**:
- Break count <10 per hour
- False positive rate <0.5%
- Run test: `cargo test -p ml --test cusum_test -- test_false_positive_rate`

---

### Playbook 3: Feature NaN/Inf Detected

**Symptoms**:
- Alert: `FeatureDataQualityIssue`
- Grafana: `wave_d_feature_nan_count > 0` or `wave_d_feature_inf_count > 0`
- ML models: Training loss NaN

**Root Cause**: Division by zero or numerical instability.

**Diagnosis**:
```bash
# 1. Identify problematic feature
psql -U foxhunt -d foxhunt -c "
  SELECT feature_index, COUNT(*)
  FROM ml_features
  WHERE symbol='ES.FUT'
    AND timestamp > NOW() - INTERVAL '1 hour'
    AND (feature_value IS NULL OR feature_value = 'NaN' OR feature_value = 'Infinity')
  GROUP BY feature_index
  ORDER BY COUNT(*) DESC;
"

# 2. Check logs for error details
grep "Invalid feature value" /var/log/foxhunt/ml_training.log | tail -20

# 3. Review feature calculation code
# Feature 223 (Regime Sharpe) → ml/src/features/regime_adaptive.rs:100-110
# Feature 224 (Risk Budget) → ml/src/features/regime_adaptive.rs:120-130
```

**Resolution**:

**Common Fix 1: Sharpe Ratio (Feature 223)**
```rust
// Add zero volatility check
let std = variance.sqrt();
if std > 1e-10 {
    (mean / std) * (252.0_f64).sqrt()
} else {
    0.0 // Return 0.0 instead of NaN
}
```

**Common Fix 2: Risk Budget (Feature 224)**
```rust
// Add zero position check
if self.max_position_size > 1e-10 {
    (self.current_position_size / (position_mult * self.max_position_size))
        .clamp(0.0, 1.0)
} else {
    0.0
}
```

**Common Fix 3: Shannon Entropy (Feature 218)**
```rust
// Filter zero probabilities before log
.filter(|&p| p > 1e-10) // Add this line
.map(|p| -p * p.log2())
```

**Deployment**:
```bash
# 1. Apply fix to affected feature
vim ml/src/features/regime_adaptive.rs

# 2. Run unit tests
cargo test -p ml --lib features::regime_adaptive -- test_all_features_finite

# 3. Rebuild and deploy
cargo build --workspace --release
systemctl restart ml_training_service
systemctl restart trading_agent_service

# 4. Monitor for 10 minutes
watch -n 60 'psql -U foxhunt -d foxhunt -t -c "SELECT COUNT(*) FROM ml_features WHERE feature_index BETWEEN 201 AND 225 AND (feature_value IS NULL OR feature_value = '"'"'NaN'"'"' OR feature_value = '"'"'Infinity'"'"');"'
```

**Verification**:
- `wave_d_feature_nan_count == 0`
- `wave_d_feature_inf_count == 0`
- Test: `cargo test -p ml -- test_all_features_finite`

---

**Document Version**: 1.0
**Last Updated**: 2025-10-18
**Status**: 🟢 **Production Ready**