foxhunt/GRAFANA_WAVE_D_SETUP.md

# Grafana Wave D Dashboard Setup Guide

**Author**: Agent M2 - Grafana Dashboard Deployment Specialist
**Date**: 2025-10-19
**System**: Foxhunt HFT Trading System
**Version**: Wave D (225 features)

---

## Executive Summary

This guide provides step-by-step instructions for deploying the **Wave D Regime Detection & Adaptive Strategies** Grafana dashboard. The dashboard includes 8 panels covering regime transitions, feature extraction performance, regime distribution, adaptive strategy metrics, and 4 critical rollback alert panels.

**Dashboard File**: `/home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json`

**Key Capabilities**:
- Real-time regime transition monitoring with CUSUM alert visualization
- Feature extraction latency tracking (P50/P99/Average)
- Regime distribution pie chart (7 regime types)
- Adaptive strategy metrics (position sizing, stop-loss, Sharpe ratio, risk budget)
- 4 rollback alert panels (flip-flopping, false positives, data corruption, system health)

---

## Prerequisites

### 1. Infrastructure Requirements

**Docker Services (must be running)**:
```bash
# Check Docker services
docker-compose ps

# Expected services:
# - postgres (TimescaleDB)
# - prometheus
# - grafana
# - redis
# - vault
```

**Database Migration**:
```bash
# Ensure migration 045 is applied
cd /home/jgrusewski/Work/foxhunt
cargo sqlx migrate run

# Verify Wave D tables exist
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "\dt regime_*"

# Expected tables:
# - regime_states
# - regime_transitions
# - adaptive_strategy_metrics
```

### 2. Data Source Configuration

**PostgreSQL Data Source**:
- **Name**: `postgres`
- **Type**: PostgreSQL
- **Host**: `localhost:5432`
- **Database**: `foxhunt`
- **User**: `foxhunt`
- **Password**: `foxhunt_dev_password`
- **SSL Mode**: `disable` (development) / `require` (production)
- **Version**: TimescaleDB 2.x

**Prometheus Data Source**:
- **Name**: `prometheus`
- **Type**: Prometheus
- **URL**: `http://localhost:9090`
- **Access**: Server (default)
- **Scrape Interval**: 15s

---

## Installation

### Step 1: Configure Data Sources

#### Option A: Manual Configuration (Grafana UI)

1. **Login to Grafana**:
   ```bash
   # Open browser
   http://localhost:3000

   # Credentials
   Username: admin
   Password: foxhunt123
   ```

2. **Add PostgreSQL Data Source**:
   - Navigate to **Configuration** → **Data Sources** → **Add data source**
   - Select **PostgreSQL**
   - Configure:
     - Name: `postgres`
     - Host: `localhost:5432`
     - Database: `foxhunt`
     - User: `foxhunt`
     - Password: `foxhunt_dev_password`
     - SSL Mode: `disable`
     - Version: `12.0+`
     - TimescaleDB: **Enabled**
   - Click **Save & Test** (should see "Database Connection OK")

3. **Add Prometheus Data Source**:
   - Navigate to **Configuration** → **Data Sources** → **Add data source**
   - Select **Prometheus**
   - Configure:
     - Name: `prometheus`
     - URL: `http://localhost:9090`
     - Access: `Server (default)`
     - Scrape interval: `15s`
   - Click **Save & Test** (should see "Data source is working")

#### Option B: Automated Configuration (Recommended)

```bash
# Create Grafana provisioning directory
mkdir -p /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/datasources

# Create datasource configuration
cat > /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/datasources/wave_d.yml <<'EOF'
apiVersion: 1

datasources:
  - name: postgres
    type: postgres
    access: proxy
    url: localhost:5432
    database: foxhunt
    user: foxhunt
    secureJsonData:
      password: foxhunt_dev_password
    jsonData:
      sslmode: disable
      postgresVersion: 1200
      timescaledb: true
    isDefault: false
    editable: true

  - name: prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    editable: true
    jsonData:
      timeInterval: 15s
EOF

# Restart Grafana to apply configuration
docker-compose restart grafana
```

### Step 2: Import Wave D Dashboard

#### Option A: Manual Import (Grafana UI)

1. **Navigate to Dashboards**:
   - Click **+ (Create)** → **Import**

2. **Upload JSON**:
   - Click **Upload JSON file**
   - Select: `/home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json`
   - Or copy-paste the entire JSON content

3. **Configure Import**:
   - Dashboard name: **Wave D - Regime Detection & Adaptive Strategies** (auto-populated)
   - Folder: Select **Foxhunt** or create new folder
   - UID: `wave_d_regime_detection` (auto-populated)
   - PostgreSQL data source: Select `postgres`
   - Prometheus data source: Select `prometheus`

4. **Import**:
   - Click **Import**
   - Dashboard should load immediately with 8 panels

#### Option B: Automated Import (Recommended)

```bash
# Method 1: Grafana API (requires Grafana to be running)
GRAFANA_URL="http://localhost:3000"
GRAFANA_USER="admin"
GRAFANA_PASS="foxhunt123"
DASHBOARD_FILE="/home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json"

# Import dashboard via API
curl -X POST \
  -H "Content-Type: application/json" \
  -u "${GRAFANA_USER}:${GRAFANA_PASS}" \
  -d @"${DASHBOARD_FILE}" \
  "${GRAFANA_URL}/api/dashboards/db"

# Expected response: {"id":1,"slug":"wave-d-regime-detection","status":"success","uid":"wave_d_regime_detection","url":"/d/wave_d_regime_detection/wave-d-regime-detection","version":1}
```

```bash
# Method 2: Provisioning (persistent across Grafana restarts)
mkdir -p /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/dashboards

# Create provisioning config
cat > /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/dashboards/wave_d.yml <<'EOF'
apiVersion: 1

providers:
  - name: 'Wave D Dashboards'
    orgId: 1
    folder: 'Foxhunt'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /home/jgrusewski/Work/foxhunt/config/grafana/dashboards
      foldersFromFilesStructure: false
EOF

# Restart Grafana to apply provisioning
docker-compose restart grafana

# Dashboard will auto-load on startup
```

### Step 3: Verify Dashboard Functionality

```bash
# 1. Check data sources are connected
curl -u admin:foxhunt123 http://localhost:3000/api/datasources | jq '.[] | {name, type, url}'

# Expected output:
# {"name":"postgres","type":"postgres","url":"localhost:5432"}
# {"name":"prometheus","type":"prometheus","url":"http://localhost:9090"}

# 2. Test PostgreSQL queries
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt <<EOF
-- Test Panel 1: Regime Transitions Timeline
SELECT
  event_timestamp AS time,
  symbol,
  from_regime || ' → ' || to_regime AS metric
FROM regime_transitions
WHERE event_timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY event_timestamp ASC
LIMIT 5;

-- Test Panel 3: Regime Distribution
SELECT
  regime AS metric,
  COUNT(*) AS value
FROM regime_states
WHERE event_timestamp >= NOW() - INTERVAL '24 hours'
GROUP BY regime
ORDER BY value DESC;
EOF

# 3. Test Prometheus metrics
curl -s http://localhost:9090/api/v1/query?query=wave_d_feature_extraction_duration_seconds_bucket | jq '.data.result | length'

# Expected: >0 (metrics are being collected)

# 4. Open dashboard in browser
xdg-open "http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection" 2>/dev/null || \
open "http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection" 2>/dev/null || \
echo "Open manually: http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection"
```

---

## Dashboard Panels

### Panel 1: Regime Transitions Timeline (Timeseries)

**Purpose**: Visualize regime changes over time with CUSUM alert triggers.

**Data Source**: PostgreSQL (`postgres`)

**SQL Query**:
```sql
SELECT
  event_timestamp AS time,
  symbol,
  from_regime || ' → ' || to_regime AS metric,
  1 AS value,
  CASE
    WHEN cusum_alert_triggered THEN 'CUSUM Alert'
    ELSE 'Normal'
  END AS alert_type
FROM regime_transitions
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY event_timestamp ASC
```

**Visualization**:
- Type: Timeseries (points)
- X-axis: Time (24 hours)
- Y-axis: Regime transitions (discrete events)
- Legend: Transition labels (e.g., "Normal → Trending")
- Alert markers: Red points for CUSUM-triggered transitions (size: 12px)
- Normal markers: Colored points for regular transitions (size: 8px)

**Interpretation**:
- **5-10 transitions/day**: Normal market behavior
- **>30 transitions/hour**: WARNING - Potential flip-flopping
- **>50 transitions/hour**: CRITICAL - Trigger Level 1 rollback (ROLLBACK_PROCEDURES.md)
- **Red points**: CUSUM structural break detected (high confidence transition)

**Example Output**:
```
Time                 Transition           Alert Type
2025-10-19 10:15:00  Normal → Trending    Normal
2025-10-19 11:30:00  Trending → Volatile  CUSUM Alert  (RED)
2025-10-19 13:45:00  Volatile → Ranging   Normal
```

---

### Panel 2: Feature Extraction Latency (P50/P99) (Timeseries)

**Purpose**: Monitor Wave D feature extraction performance. Target: <1ms (1000μs).

**Data Source**: Prometheus (`prometheus`)

**PromQL Queries**:
```promql
# P50 Latency (median)
histogram_quantile(0.50, rate(wave_d_feature_extraction_duration_seconds_bucket[5m])) * 1000

# P99 Latency (99th percentile)
histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m])) * 1000

# Average Latency
avg(rate(wave_d_feature_extraction_duration_seconds_sum[5m]) / rate(wave_d_feature_extraction_duration_seconds_count[5m])) * 1000
```

**Visualization**:
- Type: Timeseries (smooth lines)
- X-axis: Time (24 hours)
- Y-axis: Latency (milliseconds)
- Legend: P50 (blue), P99 (orange, bold), Average (green)
- Thresholds:
  - Green: 0-1ms (target met)
  - Yellow: 1-2ms (warning)
  - Red: >2ms (critical, >2x target)

**Interpretation**:
- **P50 <0.5ms**: Excellent performance (50% of extractions)
- **P99 <1ms**: Target met (99% of extractions)
- **P99 1-2ms**: WARNING - Performance degradation
- **P99 >2ms**: CRITICAL - Trigger Level 1 rollback if persistent >15 min

**Example Prometheus Metrics**:
```promql
# Sample metrics (generated by ML service)
wave_d_feature_extraction_duration_seconds_bucket{le="0.001"} 450
wave_d_feature_extraction_duration_seconds_bucket{le="0.002"} 490
wave_d_feature_extraction_duration_seconds_bucket{le="+Inf"} 500
wave_d_feature_extraction_duration_seconds_sum 0.125
wave_d_feature_extraction_duration_seconds_count 500

# Calculated P99 = 0.25ms (excellent)
```

---

### Panel 3: Regime Distribution (24h) (Pie Chart)

**Purpose**: Visualize the percentage distribution of detected regimes over the last 24 hours.

**Data Source**: PostgreSQL (`postgres`)

**SQL Query**:
```sql
SELECT
  regime AS metric,
  COUNT(*) AS value
FROM regime_states
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
GROUP BY regime
ORDER BY value DESC
```

**Visualization**:
- Type: Pie chart
- Legend: Right side, table format with value and percentage
- Labels: Percentage on slices
- Color mapping (7 regime types):
  - **Normal**: Light green (default market conditions)
  - **Trending**: Green (directional movement)
  - **Ranging**: Blue (sideways/choppy)
  - **Volatile**: Orange (high volatility)
  - **Crisis**: Red (extreme conditions)
  - **Illiquid**: Yellow (low liquidity)
  - **Momentum**: Purple (strong directional)

**Interpretation**:
- **Normal 40-60%**: Healthy market balance
- **Trending 20-30%**: Good directional opportunities
- **Ranging 15-25%**: Consolidation phases
- **Volatile <10%**: Acceptable risk levels
- **Crisis <5%**: Rare events (expected)
- **Distribution changes >50% in 1 hour**: Potential market regime shift

**Example Output**:
```
Regime      Count    Percentage
Normal      450      45%
Trending    250      25%
Ranging     200      20%
Volatile    80       8%
Momentum    15       1.5%
Crisis      3        0.3%
Illiquid    2        0.2%
```

---

### Panel 4: Adaptive Strategy Metrics (Real-time) (Timeseries)

**Purpose**: Track position sizing and stop-loss adjustments by regime.

**Data Source**: PostgreSQL (`postgres`)

**SQL Queries** (4 metrics, dual Y-axis):

**Query A: Position Multiplier** (Left Y-axis: 0-2):
```sql
SELECT
  event_timestamp AS time,
  symbol || ' - ' || regime AS metric,
  position_multiplier AS value
FROM adaptive_strategy_metrics
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY event_timestamp ASC
```

**Query B: Stop-Loss Multiplier** (Left Y-axis: 1-5):
```sql
SELECT
  event_timestamp AS time,
  symbol || ' - ' || regime AS metric,
  stop_loss_multiplier AS value
FROM adaptive_strategy_metrics
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY event_timestamp ASC
```

**Query C: Regime Sharpe Ratio** (Right Y-axis: 0+):
```sql
SELECT
  event_timestamp AS time,
  symbol || ' - ' || regime AS metric,
  regime_sharpe AS value
FROM adaptive_strategy_metrics
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
  AND regime_sharpe IS NOT NULL
ORDER BY event_timestamp ASC
```

**Query D: Risk Budget Utilization** (Right Y-axis: 0-100%):
```sql
SELECT
  event_timestamp AS time,
  symbol || ' - ' || regime AS metric,
  risk_budget_utilization * 100 AS value
FROM adaptive_strategy_metrics
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
  AND risk_budget_utilization IS NOT NULL
ORDER BY event_timestamp ASC
```

**Visualization**:
- Type: Timeseries (smooth lines, dual Y-axis)
- X-axis: Time (24 hours)
- Left Y-axis: Position multiplier (0-2), Stop-loss multiplier (1-5)
- Right Y-axis: Sharpe ratio (0+), Risk budget (0-100%)
- Legend: Table format with mean, max, last value
- Colors:
  - Position Multiplier: Blue
  - Stop-Loss Multiplier: Orange
  - Regime Sharpe: Green
  - Risk Budget: Purple

**Interpretation**:

**Position Multiplier** (0.2x-1.5x range):
- **0.2x**: Crisis regime (minimal exposure)
- **0.5x**: Volatile regime (reduced size)
- **1.0x**: Normal regime (baseline)
- **1.5x**: Trending regime (max size)

**Stop-Loss Multiplier** (1.5x-4.0x ATR range):
- **1.5x ATR**: Trending regime (tight stops)
- **2.0x ATR**: Normal regime (baseline)
- **3.0x ATR**: Ranging regime (wider stops, avoid whipsaws)
- **4.0x ATR**: Volatile regime (max stops)

**Regime Sharpe Ratio** (>1.5 target):
- **<1.0**: Poor risk-adjusted returns (review strategy)
- **1.0-1.5**: Acceptable performance
- **>1.5**: Target met (expected +25-50% improvement vs. Wave C)
- **>2.0**: Excellent performance

**Risk Budget Utilization** (<80% target):
- **<50%**: Conservative (safe margin)
- **50-80%**: Target range (balanced risk)
- **80-100%**: WARNING - High risk exposure
- **>100%**: CRITICAL - Risk limit breach (should not occur)

**Example Output**:
```
Time                 Symbol - Regime    Pos Mult  Stop Mult  Sharpe  Risk %
2025-10-19 10:00:00  ES.FUT - Trending  1.5x      1.5x ATR   1.8     65%
2025-10-19 11:00:00  ES.FUT - Volatile  0.5x      4.0x ATR   1.2     45%
2025-10-19 12:00:00  ES.FUT - Ranging   1.0x      3.0x ATR   1.4     55%
```

---

### Panel 5: Rollback Alert - Flip-Flopping Detection (Stat)

**Purpose**: Monitor for excessive regime transitions (>50/hour triggers Level 1 rollback).

**Data Source**: PostgreSQL (`postgres`)

**SQL Query**:
```sql
SELECT
  COUNT(*) AS value
FROM regime_transitions
WHERE
  event_timestamp >= NOW() - INTERVAL '1 hour'
```

**Visualization**:
- Type: Stat (big number with background color)
- Thresholds:
  - Green: 0-29 transitions/hour (normal)
  - Yellow: 30-49 transitions/hour (warning)
  - Red: ≥50 transitions/hour (CRITICAL)
- Text: "Transitions/Hour" with large value

**Interpretation**:
- **0-10**: Normal market behavior
- **10-30**: Active regime changes (acceptable)
- **30-50**: WARNING - Potential flip-flopping
- **≥50**: CRITICAL - LEVEL 1 ROLLBACK REQUIRED (ROLLBACK_PROCEDURES.md)

**Alert Action**:
```bash
# If ≥50 transitions/hour, execute Level 1 rollback
cd /home/jgrusewski/Work/foxhunt
./LEVEL_1_ROLLBACK_TEST.sh  # Zero downtime, <1 minute
```

---

### Panel 6: Rollback Alert - False Positives (Stat)

**Purpose**: Monitor regime detection accuracy (>80% error rate triggers Level 1 rollback).

**Data Source**: Prometheus (`prometheus`)

**PromQL Query**:
```promql
(sum(regime_detection_errors_total) / sum(regime_detections_total)) * 100
```

**Visualization**:
- Type: Stat (big number with background color)
- Thresholds:
  - Green: 0-49% error rate (acceptable)
  - Yellow: 50-79% error rate (warning)
  - Red: ≥80% error rate (CRITICAL)
- Unit: Percentage (%)
- Text: "Error Rate (%)" with large value

**Interpretation**:
- **0-20%**: Excellent accuracy (>80% correct)
- **20-50%**: Acceptable accuracy (50-80% correct)
- **50-80%**: WARNING - High false positive rate
- **≥80%**: CRITICAL - LEVEL 1 ROLLBACK REQUIRED

**Alert Action**:
```bash
# If ≥80% error rate, execute Level 1 rollback
cd /home/jgrusewski/Work/foxhunt
./LEVEL_1_ROLLBACK_TEST.sh  # Zero downtime, <1 minute
```

**Note**: This metric requires Prometheus instrumentation in ML service:
```rust
// ml_training_service/src/metrics.rs
lazy_static! {
    pub static ref REGIME_DETECTIONS_TOTAL: IntCounter = register_int_counter!(
        "regime_detections_total", "Total regime detections"
    ).unwrap();

    pub static ref REGIME_DETECTION_ERRORS_TOTAL: IntCounter = register_int_counter!(
        "regime_detection_errors_total", "Total regime detection errors"
    ).unwrap();
}

// Increment on detection
REGIME_DETECTIONS_TOTAL.inc();

// Increment on error (NaN, Inf, out-of-range)
if regime.is_nan() || regime.is_infinite() {
    REGIME_DETECTION_ERRORS_TOTAL.inc();
}
```

---

### Panel 7: Rollback Alert - Data Corruption (Stat)

**Purpose**: Detect NaN/Inf values in Wave D features (triggers immediate Level 3 rollback).

**Data Source**: Prometheus (`prometheus`)

**PromQL Query**:
```promql
wave_d_features_nan_count + wave_d_features_inf_count
```

**Visualization**:
- Type: Stat (big number with background color)
- Thresholds:
  - Green: 0 (no corruption)
  - Red: ≥1 (ANY corruption is CRITICAL)
- Text: "NaN/Inf Count" with large value

**Interpretation**:
- **0**: No data corruption (normal)
- **≥1**: CRITICAL - IMMEDIATE LEVEL 3 ROLLBACK REQUIRED

**Alert Action**:
```bash
# If ANY NaN/Inf detected, execute Level 3 rollback IMMEDIATELY
cd /home/jgrusewski/Work/foxhunt
./LEVEL_3_ROLLBACK_TEST.sh  # Full rollback to Wave C, ~15 minutes
```

**Note**: This metric requires Prometheus instrumentation in ML service:
```rust
// ml_training_service/src/metrics.rs
lazy_static! {
    pub static ref WAVE_D_FEATURES_NAN_COUNT: IntCounter = register_int_counter!(
        "wave_d_features_nan_count", "Count of NaN values in Wave D features"
    ).unwrap();

    pub static ref WAVE_D_FEATURES_INF_COUNT: IntCounter = register_int_counter!(
        "wave_d_features_inf_count", "Count of Inf values in Wave D features"
    ).unwrap();
}

// Check features after extraction
for feature in &wave_d_features {
    if feature.is_nan() {
        WAVE_D_FEATURES_NAN_COUNT.inc();
        error!("NaN detected in Wave D feature extraction");
    }
    if feature.is_infinite() {
        WAVE_D_FEATURES_INF_COUNT.inc();
        error!("Inf detected in Wave D feature extraction");
    }
}
```

---

### Panel 8: System Health (Stat)

**Purpose**: Monitor service uptime (down >5 minutes triggers Level 3 rollback).

**Data Source**: Prometheus (`prometheus`)

**PromQL Queries** (3 services):
```promql
# ML Training Service
up{job="ml_training_service"}

# Trading Service
up{job="trading_service"}

# API Gateway
up{job="api_gateway"}
```

**Visualization**:
- Type: Stat (horizontal layout with 3 values)
- Mappings:
  - 0 → "DOWN" (red background)
  - 1 → "UP" (green background)
- Text size: Medium (24px)
- Display: Service name + status

**Interpretation**:
- **All services UP (1)**: Normal operation
- **Any service DOWN (0) for <5 minutes**: Transient issue (monitor)
- **Any service DOWN (0) for ≥5 minutes**: CRITICAL - LEVEL 3 ROLLBACK

**Alert Action**:
```bash
# If any service down ≥5 minutes, execute Level 3 rollback
cd /home/jgrusewski/Work/foxhunt
./LEVEL_3_ROLLBACK_TEST.sh  # Full rollback to Wave C, ~15 minutes
```

**Example Output**:
```
ML Training: UP (green)
Trading: UP (green)
API Gateway: DOWN (red)  # CRITICAL if >5 min
```

---

## Prometheus Metrics Configuration

### Required Metrics

The Wave D dashboard requires the following Prometheus metrics to be exposed by the ML Training Service:

**File**: `services/ml_training_service/src/metrics.rs`

```rust
use lazy_static::lazy_static;
use prometheus::{IntCounter, Histogram, register_int_counter, register_histogram};

lazy_static! {
    // Panel 2: Feature Extraction Latency
    pub static ref WAVE_D_FEATURE_EXTRACTION_DURATION: Histogram = register_histogram!(
        "wave_d_feature_extraction_duration_seconds",
        "Wave D feature extraction duration in seconds",
        vec![0.0001, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05]
    ).unwrap();

    // Panel 5: Flip-Flopping Detection (tracked in PostgreSQL)
    // Panel 6: False Positives
    pub static ref REGIME_DETECTIONS_TOTAL: IntCounter = register_int_counter!(
        "regime_detections_total",
        "Total regime detections performed"
    ).unwrap();

    pub static ref REGIME_DETECTION_ERRORS_TOTAL: IntCounter = register_int_counter!(
        "regime_detection_errors_total",
        "Total regime detection errors (NaN, Inf, out-of-range)"
    ).unwrap();

    // Panel 7: Data Corruption
    pub static ref WAVE_D_FEATURES_NAN_COUNT: IntCounter = register_int_counter!(
        "wave_d_features_nan_count",
        "Count of NaN values detected in Wave D features"
    ).unwrap();

    pub static ref WAVE_D_FEATURES_INF_COUNT: IntCounter = register_int_counter!(
        "wave_d_features_inf_count",
        "Count of Inf values detected in Wave D features"
    ).unwrap();

    // Panel 8: System Health (auto-collected by Prometheus)
    // Metric: up{job="ml_training_service"}
    // Metric: up{job="trading_service"}
    // Metric: up{job="api_gateway"}
}

// Usage in feature extraction code
pub fn extract_wave_d_features() -> Result<Vec<f64>, CommonError> {
    let _timer = WAVE_D_FEATURE_EXTRACTION_DURATION.start_timer();
    REGIME_DETECTIONS_TOTAL.inc();

    let features = /* extraction logic */;

    // Validate features
    for feature in &features {
        if feature.is_nan() {
            WAVE_D_FEATURES_NAN_COUNT.inc();
            REGIME_DETECTION_ERRORS_TOTAL.inc();
            return Err(CommonError::validation("NaN detected in Wave D features"));
        }
        if feature.is_infinite() {
            WAVE_D_FEATURES_INF_COUNT.inc();
            REGIME_DETECTION_ERRORS_TOTAL.inc();
            return Err(CommonError::validation("Inf detected in Wave D features"));
        }
    }

    Ok(features)
}
```

### Prometheus Scrape Configuration

**File**: `/etc/prometheus/prometheus.yml` (or Docker volume mount)

```yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # ML Training Service
  - job_name: 'ml_training_service'
    static_configs:
      - targets: ['localhost:9094']
    metrics_path: '/metrics'

  # Trading Service
  - job_name: 'trading_service'
    static_configs:
      - targets: ['localhost:9092']
    metrics_path: '/metrics'

  # API Gateway
  - job_name: 'api_gateway'
    static_configs:
      - targets: ['localhost:9091']
    metrics_path: '/metrics'

  # Backtesting Service
  - job_name: 'backtesting_service'
    static_configs:
      - targets: ['localhost:9093']
    metrics_path: '/metrics'
```

**Verify Metrics Collection**:
```bash
# Check Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job, health, lastScrape}'

# Expected output:
# {"job":"ml_training_service","health":"up","lastScrape":"2025-10-19T10:30:15Z"}
# {"job":"trading_service","health":"up","lastScrape":"2025-10-19T10:30:15Z"}
# {"job":"api_gateway","health":"up","lastScrape":"2025-10-19T10:30:15Z"}

# Test Wave D metrics
curl -s http://localhost:9094/metrics | grep wave_d_feature_extraction_duration_seconds

# Expected output (histogram buckets):
# wave_d_feature_extraction_duration_seconds_bucket{le="0.001"} 450
# wave_d_feature_extraction_duration_seconds_bucket{le="0.002"} 490
# wave_d_feature_extraction_duration_seconds_sum 0.125
# wave_d_feature_extraction_duration_seconds_count 500
```

---

## Alert Rules Configuration

### Prometheus Alert Rules

**File**: `/etc/prometheus/alerts/wave_d_rollback.yml`

```yaml
groups:
  - name: wave_d_rollback_triggers
    interval: 30s
    rules:
      # CRITICAL: Flip-flopping (>50 transitions/hour)
      - alert: WaveDFlipFlopping
        expr: rate(regime_transitions_total[1h]) > 50
        for: 5m
        labels:
          severity: critical
          rollback_level: level_1
        annotations:
          summary: "Wave D flip-flopping detected ({{ $value }} transitions/hour)"
          description: "Regime detection is changing states >50 times/hour. Recommend Level 1 rollback."
          runbook: "ROLLBACK_PROCEDURES.md#level-1-feature-only-rollback-zero-downtime"

      # CRITICAL: False positives (>80% error rate)
      - alert: WaveDFalsePositives
        expr: (sum(regime_detection_errors_total) / sum(regime_detections_total)) > 0.80
        for: 10m
        labels:
          severity: critical
          rollback_level: level_1
        annotations:
          summary: "Wave D false positive rate >80%"
          description: "Regime detection accuracy below threshold. Recommend Level 1 rollback."

      # WARNING: Performance degradation (>2x latency)
      - alert: WaveDLatencyDegradation
        expr: histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m])) > 0.002
        for: 15m
        labels:
          severity: warning
          rollback_level: level_1
        annotations:
          summary: "Wave D feature extraction latency >2ms (>2x target)"
          description: "Consider Level 1 rollback if latency persists."

      # CRITICAL: NaN/Inf in features
      - alert: WaveDDataCorruption
        expr: wave_d_features_nan_count > 0 OR wave_d_features_inf_count > 0
        for: 1m
        labels:
          severity: critical
          rollback_level: level_3
        annotations:
          summary: "Wave D data corruption detected (NaN/Inf values)"
          description: "IMMEDIATE LEVEL 3 ROLLBACK REQUIRED. Data integrity compromised."
          runbook: "ROLLBACK_PROCEDURES.md#level-3-full-rollback-to-wave-c"

      # CRITICAL: System unavailable
      - alert: FoxhuntSystemDown
        expr: up{job="foxhunt_services"} == 0
        for: 5m
        labels:
          severity: critical
          rollback_level: level_3
        annotations:
          summary: "Foxhunt system unavailable for >5 minutes"
          description: "Consider Level 3 rollback to Wave C baseline."
```

**Apply Alert Rules**:
```bash
# Reload Prometheus configuration
curl -X POST http://localhost:9090/-/reload

# Verify rules loaded
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | {name, rules: .rules | length}'

# Expected output:
# {"name":"wave_d_rollback_triggers","rules":5}
```

---

## Troubleshooting

### Issue 1: Dashboard Panels Show "No Data"

**Symptoms**:
- All panels show "No data" or empty graphs
- PostgreSQL queries return 0 rows
- Prometheus queries return empty results

**Diagnosis**:
```bash
# 1. Check database tables have data
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT COUNT(*) FROM regime_states;"
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT COUNT(*) FROM regime_transitions;"

# 2. Check Prometheus metrics
curl -s http://localhost:9094/metrics | grep wave_d_feature_extraction_duration_seconds_count

# 3. Check service is running and collecting metrics
docker-compose ps | grep ml_training_service
curl http://localhost:9094/health
```

**Solution**:
```bash
# If tables are empty, insert test data
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt <<EOF
-- Insert test regime state
INSERT INTO regime_states (symbol, event_timestamp, regime, confidence, cusum_s_plus, cusum_s_minus, adx, stability)
VALUES ('ES.FUT', NOW(), 'Trending', 0.85, 1.5, -0.3, 35.2, 0.92);

-- Insert test regime transition
INSERT INTO regime_transitions (symbol, event_timestamp, from_regime, to_regime, duration_bars, transition_probability, adx_at_transition, cusum_alert_triggered)
VALUES ('ES.FUT', NOW(), 'Normal', 'Trending', 45, 0.65, 32.1, FALSE);

-- Insert test adaptive strategy metrics
INSERT INTO adaptive_strategy_metrics (symbol, event_timestamp, regime, position_multiplier, stop_loss_multiplier, regime_sharpe, risk_budget_utilization, total_trades, winning_trades, total_pnl)
VALUES ('ES.FUT', NOW(), 'Trending', 1.5, 1.5, 1.8, 0.65, 10, 7, 1500);
EOF

# If Prometheus metrics missing, check service is exposing /metrics endpoint
curl http://localhost:9094/metrics | grep -E "wave_d|regime"

# Restart services if needed
docker-compose restart ml_training_service prometheus grafana
```

---

### Issue 2: PostgreSQL Data Source Connection Failed

**Symptoms**:
- Grafana shows "Database Connection Error"
- Panel queries fail with "Error reading from server"

**Diagnosis**:
```bash
# 1. Verify PostgreSQL is running
docker-compose ps | grep postgres

# 2. Test connection manually
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT version();"

# 3. Check Grafana data source health
curl -u admin:foxhunt123 http://localhost:3000/api/datasources/name/postgres | jq '.basicAuth, .url, .database'
```

**Solution**:
```bash
# 1. Update data source configuration in Grafana UI:
# - Host: localhost:5432 (not postgres:5432 if Grafana is NOT in Docker network)
# - Database: foxhunt
# - User: foxhunt
# - Password: foxhunt_dev_password
# - SSL Mode: disable

# 2. Or use correct Docker network hostname if Grafana is in same Docker network:
# - Host: postgres:5432

# 3. Restart Grafana
docker-compose restart grafana

# 4. Re-test data source in Grafana UI: Configuration → Data Sources → postgres → Save & Test
```

---

### Issue 3: Prometheus Metrics Not Showing

**Symptoms**:
- Panel 2 (Feature Extraction Latency) shows "No data"
- Panel 6, 7, 8 (Rollback alerts) show "No data"

**Diagnosis**:
```bash
# 1. Check Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="ml_training_service") | {health, lastError}'

# 2. Check Prometheus can scrape ML service
curl http://localhost:9094/metrics

# 3. Verify metric exists in Prometheus
curl -s 'http://localhost:9090/api/v1/query?query=wave_d_feature_extraction_duration_seconds_count' | jq '.data.result'
```

**Solution**:
```bash
# 1. Ensure ML service exposes /metrics endpoint
# Add to services/ml_training_service/src/main.rs:
#
# use prometheus::{Encoder, TextEncoder};
# use actix_web::{web, App, HttpResponse, HttpServer};
#
# async fn metrics_handler() -> HttpResponse {
#     let encoder = TextEncoder::new();
#     let metric_families = prometheus::gather();
#     let mut buffer = vec![];
#     encoder.encode(&metric_families, &mut buffer).unwrap();
#     HttpResponse::Ok().body(buffer)
# }
#
# HttpServer::new(|| {
#     App::new()
#         .route("/metrics", web::get().to(metrics_handler))
# })
# .bind("0.0.0.0:9094")?
# .run()
# .await?;

# 2. Update Prometheus scrape config (see "Prometheus Scrape Configuration" section)

# 3. Reload Prometheus
curl -X POST http://localhost:9090/-/reload

# 4. Wait 15-30 seconds for first scrape, then verify
curl -s 'http://localhost:9090/api/v1/query?query=up{job="ml_training_service"}' | jq '.data.result[0].value[1]'
# Expected: "1" (service is up)
```

---

### Issue 4: Dashboard Queries Timeout

**Symptoms**:
- Panels show "Timeout" error
- Queries take >30 seconds

**Diagnosis**:
```bash
# 1. Check query performance directly
time psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT
  event_timestamp AS time,
  symbol,
  from_regime || ' → ' || to_regime AS metric
FROM regime_transitions
WHERE event_timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY event_timestamp ASC
"

# 2. Check table sizes
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT
  schemaname,
  tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size,
  n_live_tup AS row_count
FROM pg_stat_user_tables
WHERE tablename LIKE 'regime_%'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
"

# 3. Check missing indexes
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT indexname, indexdef
FROM pg_indexes
WHERE tablename LIKE 'regime_%'
ORDER BY tablename, indexname;
"
```

**Solution**:
```bash
# 1. Ensure indexes from migration 045 are applied
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt <<EOF
-- Verify indexes exist (should already be created by migration 045)
-- If missing, create them:

-- regime_states indexes
CREATE INDEX IF NOT EXISTS idx_regime_states_symbol_timestamp ON regime_states(symbol, event_timestamp DESC);
CREATE INDEX IF NOT EXISTS idx_regime_states_regime ON regime_states(regime);
CREATE INDEX IF NOT EXISTS idx_regime_states_confidence ON regime_states(confidence DESC);

-- regime_transitions indexes
CREATE INDEX IF NOT EXISTS idx_regime_transitions_symbol_timestamp ON regime_transitions(symbol, event_timestamp DESC);
CREATE INDEX IF NOT EXISTS idx_regime_transitions_from_to ON regime_transitions(from_regime, to_regime);
CREATE INDEX IF NOT EXISTS idx_regime_transitions_symbol_from_to ON regime_transitions(symbol, from_regime, to_regime);

-- adaptive_strategy_metrics indexes
CREATE INDEX IF NOT EXISTS idx_adaptive_metrics_symbol_timestamp ON adaptive_strategy_metrics(symbol, event_timestamp DESC);
CREATE INDEX IF NOT EXISTS idx_adaptive_metrics_regime ON adaptive_strategy_metrics(regime);
CREATE INDEX IF NOT EXISTS idx_adaptive_metrics_sharpe ON adaptive_strategy_metrics(regime_sharpe DESC) WHERE regime_sharpe IS NOT NULL;

-- Analyze tables for query planner
ANALYZE regime_states;
ANALYZE regime_transitions;
ANALYZE adaptive_strategy_metrics;
EOF

# 2. Increase Grafana query timeout (default: 30s)
# Edit docker-compose.yml:
# grafana:
#   environment:
#     - GF_DATAPROXY_TIMEOUT=60

# 3. Restart Grafana
docker-compose restart grafana

# 4. Consider partitioning tables if >10M rows (see TimescaleDB hypertable conversion)
```

---

### Issue 5: Incorrect Time Range

**Symptoms**:
- Dashboard shows data from wrong time period
- "No data" but database has recent rows

**Diagnosis**:
```bash
# 1. Check database timestamps
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT
  'regime_states' AS table_name,
  MIN(event_timestamp) AS oldest,
  MAX(event_timestamp) AS newest,
  COUNT(*) AS total_rows
FROM regime_states
UNION ALL
SELECT
  'regime_transitions' AS table_name,
  MIN(event_timestamp) AS oldest,
  MAX(event_timestamp) AS newest,
  COUNT(*) AS total_rows
FROM regime_transitions;
"

# 2. Check Grafana time range picker
# Dashboard top-right: Should show "Last 24 hours" or "now-24h to now"

# 3. Check server time vs. dashboard time
date -u  # Server time (UTC)
# Compare with Grafana dashboard time picker
```

**Solution**:
```bash
# 1. Ensure database timestamps are in UTC (PostgreSQL TIMESTAMPTZ)
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SHOW timezone;"
# Expected: UTC

# 2. Update Grafana dashboard timezone
# Dashboard Settings → Time options → Timezone: UTC

# 3. Verify data exists in last 24 hours
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT COUNT(*) FROM regime_states WHERE event_timestamp >= NOW() - INTERVAL '24 hours';
"
# If 0, insert test data (see Issue 1 solution)
```

---

## Production Deployment Checklist

Before deploying to production, ensure:

### Database
- [ ] Migration 045 applied successfully (`cargo sqlx migrate run`)
- [ ] All 3 tables exist: `regime_states`, `regime_transitions`, `adaptive_strategy_metrics`
- [ ] All 3 functions exist: `get_latest_regime`, `get_regime_transition_matrix`, `get_regime_performance`
- [ ] Indexes verified with `\di regime_*` in psql
- [ ] Permissions granted to `foxhunt` user
- [ ] Backup scheduled (hourly for Wave D tables)

### Prometheus
- [ ] ML Training Service metrics endpoint exposed at `http://localhost:9094/metrics`
- [ ] Scrape config updated with all 4 services (API Gateway, Trading, Backtesting, ML Training)
- [ ] Alert rules loaded from `/etc/prometheus/alerts/wave_d_rollback.yml`
- [ ] Scrape interval: 15s
- [ ] Retention: 30 days minimum
- [ ] Storage: 10GB minimum for 30-day retention

### Grafana
- [ ] PostgreSQL data source configured with `postgres` UID
- [ ] Prometheus data source configured with `prometheus` UID
- [ ] Wave D dashboard imported successfully
- [ ] All 8 panels showing data (test with dummy data if needed)
- [ ] Alert rules linked to dashboard (see Panel 5-8)
- [ ] Dashboard starred/favorited for quick access
- [ ] Refresh interval: 10s
- [ ] Auto-refresh enabled
- [ ] Provisioning configured for persistent deployment

### Monitoring
- [ ] Prometheus alerts configured for 5 rollback triggers
- [ ] Alert notifications configured (Slack, PagerDuty, email)
- [ ] On-call rotation established for critical alerts
- [ ] Rollback procedures tested (LEVEL_1_ROLLBACK_TEST.sh, LEVEL_3_ROLLBACK_TEST.sh)
- [ ] Dashboard URL bookmarked for ops team
- [ ] Runbooks created for common issues (see "Troubleshooting" section)

### Performance
- [ ] Database indexes optimized (EXPLAIN ANALYZE on slow queries)
- [ ] Grafana query timeout increased to 60s (if needed)
- [ ] Prometheus storage optimized (SSD for fast queries)
- [ ] TimescaleDB hypertables configured (if >10M rows)
- [ ] Query performance baseline documented (<1s P99 for all panels)

### Security
- [ ] Grafana admin password changed from default (`admin/foxhunt123` → production password)
- [ ] PostgreSQL password changed from default (`foxhunt_dev_password` → production password)
- [ ] Grafana HTTPS enabled (production only)
- [ ] Prometheus metrics endpoint authentication enabled (production only)
- [ ] Database connections over SSL (production only)
- [ ] Audit logging enabled for Grafana configuration changes

---

## Next Steps

1. **Deploy Dashboard** (10 minutes):
   ```bash
   # Follow "Installation" section (automated method recommended)
   cd /home/jgrusewski/Work/foxhunt
   # ... (see Installation section)
   ```

2. **Configure Prometheus Metrics** (30 minutes):
   ```bash
   # Add metrics instrumentation to ML Training Service
   # See "Prometheus Metrics Configuration" section
   ```

3. **Test Dashboard with Live Data** (1 hour):
   ```bash
   # Run backtest to generate regime transitions
   cargo run --release -p backtesting_service --example wave_d_backtest

   # Verify data in dashboard
   xdg-open http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection
   ```

4. **Configure Alert Notifications** (30 minutes):
   ```bash
   # Add Prometheus Alertmanager config
   # See "Alert Rules Configuration" section
   ```

5. **Production Deployment** (as per ROLLBACK_PROCEDURES.md):
   ```bash
   # Complete "Production Deployment Checklist" above
   # Deploy with monitoring enabled
   # Monitor dashboard for 24 hours before live trading
   ```

---

## References

- **Dashboard File**: `/home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json`
- **Database Migration**: `/home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.sql`
- **Rollback Procedures**: `/home/jgrusewski/Work/foxhunt/ROLLBACK_PROCEDURES.md`
- **Grafana Documentation**: https://grafana.com/docs/grafana/latest/
- **Prometheus Documentation**: https://prometheus.io/docs/
- **TimescaleDB Documentation**: https://docs.timescale.com/

---

**END OF GUIDE**