foxhunt/GRAFANA_WAVE_D_SETUP.md

Grafana Wave D Dashboard Setup Guide

Author: Agent M2 - Grafana Dashboard Deployment Specialist Date: 2025-10-19 System: Foxhunt HFT Trading System Version: Wave D (225 features)

---

Executive Summary

This guide provides step-by-step instructions for deploying the Wave D Regime Detection & Adaptive Strategies Grafana dashboard. The dashboard includes 8 panels covering regime transitions, feature extraction performance, regime distribution, adaptive strategy metrics, and 4 critical rollback alert panels.

Dashboard File: /home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json

Key Capabilities:

• Real-time regime transition monitoring with CUSUM alert visualization
• Feature extraction latency tracking (P50/P99/Average)
• Regime distribution pie chart (7 regime types)
• Adaptive strategy metrics (position sizing, stop-loss, Sharpe ratio, risk budget)
• 4 rollback alert panels (flip-flopping, false positives, data corruption, system health)

---

Prerequisites

1. Infrastructure Requirements

Docker Services (must be running):

# Check Docker services
docker-compose ps

# Expected services:
# - postgres (TimescaleDB)
# - prometheus
# - grafana
# - redis
# - vault

Database Migration:

# Ensure migration 045 is applied
cd /home/jgrusewski/Work/foxhunt
cargo sqlx migrate run

# Verify Wave D tables exist
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "\dt regime_*"

# Expected tables:
# - regime_states
# - regime_transitions
# - adaptive_strategy_metrics

2. Data Source Configuration

PostgreSQL Data Source:

• Name: postgres
• Type: PostgreSQL
• Host: localhost:5432
• Database: foxhunt
• User: foxhunt
• Password: foxhunt_dev_password
• SSL Mode: disable (development) / require (production)
• Version: TimescaleDB 2.x

Prometheus Data Source:

• Name: prometheus
• Type: Prometheus
• URL: http://localhost:9090
• Access: Server (default)
• Scrape Interval: 15s

---

Installation

Step 1: Configure Data Sources

Option A: Manual Configuration (Grafana UI)

1. Login to Grafana:

   # Open browser
   http://localhost:3000
   
   # Credentials
   Username: admin
   Password: foxhunt123
   

2. Add PostgreSQL Data Source:

   • Navigate to Configuration → Data Sources → Add data source
   • Select PostgreSQL
   • Configure:
     • Name: postgres
     • Host: localhost:5432
     • Database: foxhunt
     • User: foxhunt
     • Password: foxhunt_dev_password
     • SSL Mode: disable
     • Version: 12.0+
     • TimescaleDB: Enabled
   • Click Save & Test (should see "Database Connection OK")

3. Add Prometheus Data Source:

   • Navigate to Configuration → Data Sources → Add data source
   • Select Prometheus
   • Configure:
     • Name: prometheus
     • URL: http://localhost:9090
     • Access: Server (default)
     • Scrape interval: 15s
   • Click Save & Test (should see "Data source is working")

Option B: Automated Configuration (Recommended)

# Create Grafana provisioning directory
mkdir -p /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/datasources

# Create datasource configuration
cat > /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/datasources/wave_d.yml <<'EOF'
apiVersion: 1

datasources:
  - name: postgres
    type: postgres
    access: proxy
    url: localhost:5432
    database: foxhunt
    user: foxhunt
    secureJsonData:
      password: foxhunt_dev_password
    jsonData:
      sslmode: disable
      postgresVersion: 1200
      timescaledb: true
    isDefault: false
    editable: true

  - name: prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    editable: true
    jsonData:
      timeInterval: 15s
EOF

# Restart Grafana to apply configuration
docker-compose restart grafana

Step 2: Import Wave D Dashboard

Option A: Manual Import (Grafana UI)

1. Navigate to Dashboards:

   • Click + (Create) → Import

2. Upload JSON:

   • Click Upload JSON file
   • Select: /home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json
   • Or copy-paste the entire JSON content

3. Configure Import:

   • Dashboard name: Wave D - Regime Detection & Adaptive Strategies (auto-populated)
   • Folder: Select Foxhunt or create new folder
   • UID: wave_d_regime_detection (auto-populated)
   • PostgreSQL data source: Select postgres
   • Prometheus data source: Select prometheus

4. Import:

   • Click Import
   • Dashboard should load immediately with 8 panels

Option B: Automated Import (Recommended)

# Method 1: Grafana API (requires Grafana to be running)
GRAFANA_URL="http://localhost:3000"
GRAFANA_USER="admin"
GRAFANA_PASS="foxhunt123"
DASHBOARD_FILE="/home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json"

# Import dashboard via API
curl -X POST \
  -H "Content-Type: application/json" \
  -u "${GRAFANA_USER}:${GRAFANA_PASS}" \
  -d @"${DASHBOARD_FILE}" \
  "${GRAFANA_URL}/api/dashboards/db"

# Expected response: {"id":1,"slug":"wave-d-regime-detection","status":"success","uid":"wave_d_regime_detection","url":"/d/wave_d_regime_detection/wave-d-regime-detection","version":1}

# Method 2: Provisioning (persistent across Grafana restarts)
mkdir -p /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/dashboards

# Create provisioning config
cat > /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/dashboards/wave_d.yml <<'EOF'
apiVersion: 1

providers:
  - name: 'Wave D Dashboards'
    orgId: 1
    folder: 'Foxhunt'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /home/jgrusewski/Work/foxhunt/config/grafana/dashboards
      foldersFromFilesStructure: false
EOF

# Restart Grafana to apply provisioning
docker-compose restart grafana

# Dashboard will auto-load on startup

Step 3: Verify Dashboard Functionality

# 1. Check data sources are connected
curl -u admin:foxhunt123 http://localhost:3000/api/datasources | jq '.[] | {name, type, url}'

# Expected output:
# {"name":"postgres","type":"postgres","url":"localhost:5432"}
# {"name":"prometheus","type":"prometheus","url":"http://localhost:9090"}

# 2. Test PostgreSQL queries
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt <<EOF
-- Test Panel 1: Regime Transitions Timeline
SELECT
  event_timestamp AS time,
  symbol,
  from_regime || ' → ' || to_regime AS metric
FROM regime_transitions
WHERE event_timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY event_timestamp ASC
LIMIT 5;

-- Test Panel 3: Regime Distribution
SELECT
  regime AS metric,
  COUNT(*) AS value
FROM regime_states
WHERE event_timestamp >= NOW() - INTERVAL '24 hours'
GROUP BY regime
ORDER BY value DESC;
EOF

# 3. Test Prometheus metrics
curl -s http://localhost:9090/api/v1/query?query=wave_d_feature_extraction_duration_seconds_bucket | jq '.data.result | length'

# Expected: >0 (metrics are being collected)

# 4. Open dashboard in browser
xdg-open "http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection" 2>/dev/null || \
open "http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection" 2>/dev/null || \
echo "Open manually: http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection"

---

Dashboard Panels

Panel 1: Regime Transitions Timeline (Timeseries)

Purpose: Visualize regime changes over time with CUSUM alert triggers.

Data Source: PostgreSQL (postgres)

SQL Query:

SELECT
  event_timestamp AS time,
  symbol,
  from_regime || ' → ' || to_regime AS metric,
  1 AS value,
  CASE
    WHEN cusum_alert_triggered THEN 'CUSUM Alert'
    ELSE 'Normal'
  END AS alert_type
FROM regime_transitions
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY event_timestamp ASC

Visualization:

• Type: Timeseries (points)
• X-axis: Time (24 hours)
• Y-axis: Regime transitions (discrete events)
• Legend: Transition labels (e.g., "Normal → Trending")
• Alert markers: Red points for CUSUM-triggered transitions (size: 12px)
• Normal markers: Colored points for regular transitions (size: 8px)

Interpretation:

• 5-10 transitions/day: Normal market behavior
• >30 transitions/hour: WARNING - Potential flip-flopping
• >50 transitions/hour: CRITICAL - Trigger Level 1 rollback (ROLLBACK_PROCEDURES.md)
• Red points: CUSUM structural break detected (high confidence transition)

Example Output:

Time                 Transition           Alert Type
2025-10-19 10:15:00  Normal → Trending    Normal
2025-10-19 11:30:00  Trending → Volatile  CUSUM Alert  (RED)
2025-10-19 13:45:00  Volatile → Ranging   Normal

---

Panel 2: Feature Extraction Latency (P50/P99) (Timeseries)

Purpose: Monitor Wave D feature extraction performance. Target: <1ms (1000μs).

Data Source: Prometheus (prometheus)

PromQL Queries:

# P50 Latency (median)
histogram_quantile(0.50, rate(wave_d_feature_extraction_duration_seconds_bucket[5m])) * 1000

# P99 Latency (99th percentile)
histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m])) * 1000

# Average Latency
avg(rate(wave_d_feature_extraction_duration_seconds_sum[5m]) / rate(wave_d_feature_extraction_duration_seconds_count[5m])) * 1000

Visualization:

• Type: Timeseries (smooth lines)
• X-axis: Time (24 hours)
• Y-axis: Latency (milliseconds)
• Legend: P50 (blue), P99 (orange, bold), Average (green)
• Thresholds:
  • Green: 0-1ms (target met)
  • Yellow: 1-2ms (warning)
  • Red: >2ms (critical, >2x target)

Interpretation:

• P50 <0.5ms: Excellent performance (50% of extractions)
• P99 <1ms: Target met (99% of extractions)
• P99 1-2ms: WARNING - Performance degradation
• P99 >2ms: CRITICAL - Trigger Level 1 rollback if persistent >15 min

Example Prometheus Metrics:

# Sample metrics (generated by ML service)
wave_d_feature_extraction_duration_seconds_bucket{le="0.001"} 450
wave_d_feature_extraction_duration_seconds_bucket{le="0.002"} 490
wave_d_feature_extraction_duration_seconds_bucket{le="+Inf"} 500
wave_d_feature_extraction_duration_seconds_sum 0.125
wave_d_feature_extraction_duration_seconds_count 500

# Calculated P99 = 0.25ms (excellent)

---

Panel 3: Regime Distribution (24h) (Pie Chart)

Purpose: Visualize the percentage distribution of detected regimes over the last 24 hours.

Data Source: PostgreSQL (postgres)

SQL Query:

SELECT
  regime AS metric,
  COUNT(*) AS value
FROM regime_states
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
GROUP BY regime
ORDER BY value DESC

Visualization:

• Type: Pie chart
• Legend: Right side, table format with value and percentage
• Labels: Percentage on slices
• Color mapping (7 regime types):
  • Normal: Light green (default market conditions)
  • Trending: Green (directional movement)
  • Ranging: Blue (sideways/choppy)
  • Volatile: Orange (high volatility)
  • Crisis: Red (extreme conditions)
  • Illiquid: Yellow (low liquidity)
  • Momentum: Purple (strong directional)

Interpretation:

• Normal 40-60%: Healthy market balance
• Trending 20-30%: Good directional opportunities
• Ranging 15-25%: Consolidation phases
• Volatile <10%: Acceptable risk levels
• Crisis <5%: Rare events (expected)
• Distribution changes >50% in 1 hour: Potential market regime shift

Example Output:

Regime      Count    Percentage
Normal      450      45%
Trending    250      25%
Ranging     200      20%
Volatile    80       8%
Momentum    15       1.5%
Crisis      3        0.3%
Illiquid    2        0.2%

---

Panel 4: Adaptive Strategy Metrics (Real-time) (Timeseries)

Purpose: Track position sizing and stop-loss adjustments by regime.

Data Source: PostgreSQL (postgres)

SQL Queries (4 metrics, dual Y-axis):

Query A: Position Multiplier (Left Y-axis: 0-2):

SELECT
  event_timestamp AS time,
  symbol || ' - ' || regime AS metric,
  position_multiplier AS value
FROM adaptive_strategy_metrics
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY event_timestamp ASC

Query B: Stop-Loss Multiplier (Left Y-axis: 1-5):

SELECT
  event_timestamp AS time,
  symbol || ' - ' || regime AS metric,
  stop_loss_multiplier AS value
FROM adaptive_strategy_metrics
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY event_timestamp ASC

Query C: Regime Sharpe Ratio (Right Y-axis: 0+):

SELECT
  event_timestamp AS time,
  symbol || ' - ' || regime AS metric,
  regime_sharpe AS value
FROM adaptive_strategy_metrics
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
  AND regime_sharpe IS NOT NULL
ORDER BY event_timestamp ASC

Query D: Risk Budget Utilization (Right Y-axis: 0-100%):

SELECT
  event_timestamp AS time,
  symbol || ' - ' || regime AS metric,
  risk_budget_utilization * 100 AS value
FROM adaptive_strategy_metrics
WHERE
  event_timestamp >= NOW() - INTERVAL '24 hours'
  AND risk_budget_utilization IS NOT NULL
ORDER BY event_timestamp ASC

Visualization:

• Type: Timeseries (smooth lines, dual Y-axis)
• X-axis: Time (24 hours)
• Left Y-axis: Position multiplier (0-2), Stop-loss multiplier (1-5)
• Right Y-axis: Sharpe ratio (0+), Risk budget (0-100%)
• Legend: Table format with mean, max, last value
• Colors:
  • Position Multiplier: Blue
  • Stop-Loss Multiplier: Orange
  • Regime Sharpe: Green
  • Risk Budget: Purple

Interpretation:

Position Multiplier (0.2x-1.5x range):

• 0.2x: Crisis regime (minimal exposure)
• 0.5x: Volatile regime (reduced size)
• 1.0x: Normal regime (baseline)
• 1.5x: Trending regime (max size)

Stop-Loss Multiplier (1.5x-4.0x ATR range):

• 1.5x ATR: Trending regime (tight stops)
• 2.0x ATR: Normal regime (baseline)
• 3.0x ATR: Ranging regime (wider stops, avoid whipsaws)
• 4.0x ATR: Volatile regime (max stops)

Regime Sharpe Ratio (>1.5 target):

• <1.0: Poor risk-adjusted returns (review strategy)
• 1.0-1.5: Acceptable performance
• >1.5: Target met (expected +25-50% improvement vs. Wave C)
• >2.0: Excellent performance

Risk Budget Utilization (<80% target):

• <50%: Conservative (safe margin)
• 50-80%: Target range (balanced risk)
• 80-100%: WARNING - High risk exposure
• >100%: CRITICAL - Risk limit breach (should not occur)

Example Output:

Time                 Symbol - Regime    Pos Mult  Stop Mult  Sharpe  Risk %
2025-10-19 10:00:00  ES.FUT - Trending  1.5x      1.5x ATR   1.8     65%
2025-10-19 11:00:00  ES.FUT - Volatile  0.5x      4.0x ATR   1.2     45%
2025-10-19 12:00:00  ES.FUT - Ranging   1.0x      3.0x ATR   1.4     55%

---

Panel 5: Rollback Alert - Flip-Flopping Detection (Stat)

Purpose: Monitor for excessive regime transitions (>50/hour triggers Level 1 rollback).

Data Source: PostgreSQL (postgres)

SQL Query:

SELECT
  COUNT(*) AS value
FROM regime_transitions
WHERE
  event_timestamp >= NOW() - INTERVAL '1 hour'

Visualization:

• Type: Stat (big number with background color)
• Thresholds:
  • Green: 0-29 transitions/hour (normal)
  • Yellow: 30-49 transitions/hour (warning)
  • Red: ≥50 transitions/hour (CRITICAL)
• Text: "Transitions/Hour" with large value

Interpretation:

• 0-10: Normal market behavior
• 10-30: Active regime changes (acceptable)
• 30-50: WARNING - Potential flip-flopping
• ≥50: CRITICAL - LEVEL 1 ROLLBACK REQUIRED (ROLLBACK_PROCEDURES.md)

Alert Action:

# If ≥50 transitions/hour, execute Level 1 rollback
cd /home/jgrusewski/Work/foxhunt
./LEVEL_1_ROLLBACK_TEST.sh  # Zero downtime, <1 minute

---

Panel 6: Rollback Alert - False Positives (Stat)

Purpose: Monitor regime detection accuracy (>80% error rate triggers Level 1 rollback).

Data Source: Prometheus (prometheus)

PromQL Query:

(sum(regime_detection_errors_total) / sum(regime_detections_total)) * 100

Visualization:

• Type: Stat (big number with background color)
• Thresholds:
  • Green: 0-49% error rate (acceptable)
  • Yellow: 50-79% error rate (warning)
  • Red: ≥80% error rate (CRITICAL)
• Unit: Percentage (%)
• Text: "Error Rate (%)" with large value

Interpretation:

• 0-20%: Excellent accuracy (>80% correct)
• 20-50%: Acceptable accuracy (50-80% correct)
• 50-80%: WARNING - High false positive rate
• ≥80%: CRITICAL - LEVEL 1 ROLLBACK REQUIRED

Alert Action:

# If ≥80% error rate, execute Level 1 rollback
cd /home/jgrusewski/Work/foxhunt
./LEVEL_1_ROLLBACK_TEST.sh  # Zero downtime, <1 minute

Note: This metric requires Prometheus instrumentation in ML service:

// ml_training_service/src/metrics.rs
lazy_static! {
    pub static ref REGIME_DETECTIONS_TOTAL: IntCounter = register_int_counter!(
        "regime_detections_total", "Total regime detections"
    ).unwrap();

    pub static ref REGIME_DETECTION_ERRORS_TOTAL: IntCounter = register_int_counter!(
        "regime_detection_errors_total", "Total regime detection errors"
    ).unwrap();
}

// Increment on detection
REGIME_DETECTIONS_TOTAL.inc();

// Increment on error (NaN, Inf, out-of-range)
if regime.is_nan() || regime.is_infinite() {
    REGIME_DETECTION_ERRORS_TOTAL.inc();
}

---

Panel 7: Rollback Alert - Data Corruption (Stat)

Purpose: Detect NaN/Inf values in Wave D features (triggers immediate Level 3 rollback).

Data Source: Prometheus (prometheus)

PromQL Query:

wave_d_features_nan_count + wave_d_features_inf_count

Visualization:

• Type: Stat (big number with background color)
• Thresholds:
  • Green: 0 (no corruption)
  • Red: ≥1 (ANY corruption is CRITICAL)
• Text: "NaN/Inf Count" with large value

Interpretation:

• 0: No data corruption (normal)
• ≥1: CRITICAL - IMMEDIATE LEVEL 3 ROLLBACK REQUIRED

Alert Action:

# If ANY NaN/Inf detected, execute Level 3 rollback IMMEDIATELY
cd /home/jgrusewski/Work/foxhunt
./LEVEL_3_ROLLBACK_TEST.sh  # Full rollback to Wave C, ~15 minutes

Note: This metric requires Prometheus instrumentation in ML service:

// ml_training_service/src/metrics.rs
lazy_static! {
    pub static ref WAVE_D_FEATURES_NAN_COUNT: IntCounter = register_int_counter!(
        "wave_d_features_nan_count", "Count of NaN values in Wave D features"
    ).unwrap();

    pub static ref WAVE_D_FEATURES_INF_COUNT: IntCounter = register_int_counter!(
        "wave_d_features_inf_count", "Count of Inf values in Wave D features"
    ).unwrap();
}

// Check features after extraction
for feature in &wave_d_features {
    if feature.is_nan() {
        WAVE_D_FEATURES_NAN_COUNT.inc();
        error!("NaN detected in Wave D feature extraction");
    }
    if feature.is_infinite() {
        WAVE_D_FEATURES_INF_COUNT.inc();
        error!("Inf detected in Wave D feature extraction");
    }
}

---

Panel 8: System Health (Stat)

Purpose: Monitor service uptime (down >5 minutes triggers Level 3 rollback).

Data Source: Prometheus (prometheus)

PromQL Queries (3 services):

# ML Training Service
up{job="ml_training_service"}

# Trading Service
up{job="trading_service"}

# API Gateway
up{job="api_gateway"}

Visualization:

• Type: Stat (horizontal layout with 3 values)
• Mappings:
  • 0 → "DOWN" (red background)
  • 1 → "UP" (green background)
• Text size: Medium (24px)
• Display: Service name + status

Interpretation:

• All services UP (1): Normal operation
• Any service DOWN (0) for <5 minutes: Transient issue (monitor)
• Any service DOWN (0) for ≥5 minutes: CRITICAL - LEVEL 3 ROLLBACK

Alert Action:

# If any service down ≥5 minutes, execute Level 3 rollback
cd /home/jgrusewski/Work/foxhunt
./LEVEL_3_ROLLBACK_TEST.sh  # Full rollback to Wave C, ~15 minutes

Example Output:

ML Training: UP (green)
Trading: UP (green)
API Gateway: DOWN (red)  # CRITICAL if >5 min

---

Prometheus Metrics Configuration

Required Metrics

The Wave D dashboard requires the following Prometheus metrics to be exposed by the ML Training Service:

File: services/ml_training_service/src/metrics.rs

use lazy_static::lazy_static;
use prometheus::{IntCounter, Histogram, register_int_counter, register_histogram};

lazy_static! {
    // Panel 2: Feature Extraction Latency
    pub static ref WAVE_D_FEATURE_EXTRACTION_DURATION: Histogram = register_histogram!(
        "wave_d_feature_extraction_duration_seconds",
        "Wave D feature extraction duration in seconds",
        vec![0.0001, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05]
    ).unwrap();

    // Panel 5: Flip-Flopping Detection (tracked in PostgreSQL)
    // Panel 6: False Positives
    pub static ref REGIME_DETECTIONS_TOTAL: IntCounter = register_int_counter!(
        "regime_detections_total",
        "Total regime detections performed"
    ).unwrap();

    pub static ref REGIME_DETECTION_ERRORS_TOTAL: IntCounter = register_int_counter!(
        "regime_detection_errors_total",
        "Total regime detection errors (NaN, Inf, out-of-range)"
    ).unwrap();

    // Panel 7: Data Corruption
    pub static ref WAVE_D_FEATURES_NAN_COUNT: IntCounter = register_int_counter!(
        "wave_d_features_nan_count",
        "Count of NaN values detected in Wave D features"
    ).unwrap();

    pub static ref WAVE_D_FEATURES_INF_COUNT: IntCounter = register_int_counter!(
        "wave_d_features_inf_count",
        "Count of Inf values detected in Wave D features"
    ).unwrap();

    // Panel 8: System Health (auto-collected by Prometheus)
    // Metric: up{job="ml_training_service"}
    // Metric: up{job="trading_service"}
    // Metric: up{job="api_gateway"}
}

// Usage in feature extraction code
pub fn extract_wave_d_features() -> Result<Vec<f64>, CommonError> {
    let _timer = WAVE_D_FEATURE_EXTRACTION_DURATION.start_timer();
    REGIME_DETECTIONS_TOTAL.inc();

    let features = /* extraction logic */;

    // Validate features
    for feature in &features {
        if feature.is_nan() {
            WAVE_D_FEATURES_NAN_COUNT.inc();
            REGIME_DETECTION_ERRORS_TOTAL.inc();
            return Err(CommonError::validation("NaN detected in Wave D features"));
        }
        if feature.is_infinite() {
            WAVE_D_FEATURES_INF_COUNT.inc();
            REGIME_DETECTION_ERRORS_TOTAL.inc();
            return Err(CommonError::validation("Inf detected in Wave D features"));
        }
    }

    Ok(features)
}

Prometheus Scrape Configuration

File: /etc/prometheus/prometheus.yml (or Docker volume mount)

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # ML Training Service
  - job_name: 'ml_training_service'
    static_configs:
      - targets: ['localhost:9094']
    metrics_path: '/metrics'

  # Trading Service
  - job_name: 'trading_service'
    static_configs:
      - targets: ['localhost:9092']
    metrics_path: '/metrics'

  # API Gateway
  - job_name: 'api_gateway'
    static_configs:
      - targets: ['localhost:9091']
    metrics_path: '/metrics'

  # Backtesting Service
  - job_name: 'backtesting_service'
    static_configs:
      - targets: ['localhost:9093']
    metrics_path: '/metrics'

Verify Metrics Collection:

# Check Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job, health, lastScrape}'

# Expected output:
# {"job":"ml_training_service","health":"up","lastScrape":"2025-10-19T10:30:15Z"}
# {"job":"trading_service","health":"up","lastScrape":"2025-10-19T10:30:15Z"}
# {"job":"api_gateway","health":"up","lastScrape":"2025-10-19T10:30:15Z"}

# Test Wave D metrics
curl -s http://localhost:9094/metrics | grep wave_d_feature_extraction_duration_seconds

# Expected output (histogram buckets):
# wave_d_feature_extraction_duration_seconds_bucket{le="0.001"} 450
# wave_d_feature_extraction_duration_seconds_bucket{le="0.002"} 490
# wave_d_feature_extraction_duration_seconds_sum 0.125
# wave_d_feature_extraction_duration_seconds_count 500

---

Alert Rules Configuration

Prometheus Alert Rules

File: /etc/prometheus/alerts/wave_d_rollback.yml

groups:
  - name: wave_d_rollback_triggers
    interval: 30s
    rules:
      # CRITICAL: Flip-flopping (>50 transitions/hour)
      - alert: WaveDFlipFlopping
        expr: rate(regime_transitions_total[1h]) > 50
        for: 5m
        labels:
          severity: critical
          rollback_level: level_1
        annotations:
          summary: "Wave D flip-flopping detected ({{ $value }} transitions/hour)"
          description: "Regime detection is changing states >50 times/hour. Recommend Level 1 rollback."
          runbook: "ROLLBACK_PROCEDURES.md#level-1-feature-only-rollback-zero-downtime"

      # CRITICAL: False positives (>80% error rate)
      - alert: WaveDFalsePositives
        expr: (sum(regime_detection_errors_total) / sum(regime_detections_total)) > 0.80
        for: 10m
        labels:
          severity: critical
          rollback_level: level_1
        annotations:
          summary: "Wave D false positive rate >80%"
          description: "Regime detection accuracy below threshold. Recommend Level 1 rollback."

      # WARNING: Performance degradation (>2x latency)
      - alert: WaveDLatencyDegradation
        expr: histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m])) > 0.002
        for: 15m
        labels:
          severity: warning
          rollback_level: level_1
        annotations:
          summary: "Wave D feature extraction latency >2ms (>2x target)"
          description: "Consider Level 1 rollback if latency persists."

      # CRITICAL: NaN/Inf in features
      - alert: WaveDDataCorruption
        expr: wave_d_features_nan_count > 0 OR wave_d_features_inf_count > 0
        for: 1m
        labels:
          severity: critical
          rollback_level: level_3
        annotations:
          summary: "Wave D data corruption detected (NaN/Inf values)"
          description: "IMMEDIATE LEVEL 3 ROLLBACK REQUIRED. Data integrity compromised."
          runbook: "ROLLBACK_PROCEDURES.md#level-3-full-rollback-to-wave-c"

      # CRITICAL: System unavailable
      - alert: FoxhuntSystemDown
        expr: up{job="foxhunt_services"} == 0
        for: 5m
        labels:
          severity: critical
          rollback_level: level_3
        annotations:
          summary: "Foxhunt system unavailable for >5 minutes"
          description: "Consider Level 3 rollback to Wave C baseline."

Apply Alert Rules:

# Reload Prometheus configuration
curl -X POST http://localhost:9090/-/reload

# Verify rules loaded
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | {name, rules: .rules | length}'

# Expected output:
# {"name":"wave_d_rollback_triggers","rules":5}

---

Troubleshooting

Issue 1: Dashboard Panels Show "No Data"

Symptoms:

• All panels show "No data" or empty graphs
• PostgreSQL queries return 0 rows
• Prometheus queries return empty results

Diagnosis:

# 1. Check database tables have data
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT COUNT(*) FROM regime_states;"
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT COUNT(*) FROM regime_transitions;"

# 2. Check Prometheus metrics
curl -s http://localhost:9094/metrics | grep wave_d_feature_extraction_duration_seconds_count

# 3. Check service is running and collecting metrics
docker-compose ps | grep ml_training_service
curl http://localhost:9094/health

Solution:

# If tables are empty, insert test data
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt <<EOF
-- Insert test regime state
INSERT INTO regime_states (symbol, event_timestamp, regime, confidence, cusum_s_plus, cusum_s_minus, adx, stability)
VALUES ('ES.FUT', NOW(), 'Trending', 0.85, 1.5, -0.3, 35.2, 0.92);

-- Insert test regime transition
INSERT INTO regime_transitions (symbol, event_timestamp, from_regime, to_regime, duration_bars, transition_probability, adx_at_transition, cusum_alert_triggered)
VALUES ('ES.FUT', NOW(), 'Normal', 'Trending', 45, 0.65, 32.1, FALSE);

-- Insert test adaptive strategy metrics
INSERT INTO adaptive_strategy_metrics (symbol, event_timestamp, regime, position_multiplier, stop_loss_multiplier, regime_sharpe, risk_budget_utilization, total_trades, winning_trades, total_pnl)
VALUES ('ES.FUT', NOW(), 'Trending', 1.5, 1.5, 1.8, 0.65, 10, 7, 1500);
EOF

# If Prometheus metrics missing, check service is exposing /metrics endpoint
curl http://localhost:9094/metrics | grep -E "wave_d|regime"

# Restart services if needed
docker-compose restart ml_training_service prometheus grafana

---

Issue 2: PostgreSQL Data Source Connection Failed

Symptoms:

• Grafana shows "Database Connection Error"
• Panel queries fail with "Error reading from server"

Diagnosis:

# 1. Verify PostgreSQL is running
docker-compose ps | grep postgres

# 2. Test connection manually
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT version();"

# 3. Check Grafana data source health
curl -u admin:foxhunt123 http://localhost:3000/api/datasources/name/postgres | jq '.basicAuth, .url, .database'

Solution:

# 1. Update data source configuration in Grafana UI:
# - Host: localhost:5432 (not postgres:5432 if Grafana is NOT in Docker network)
# - Database: foxhunt
# - User: foxhunt
# - Password: foxhunt_dev_password
# - SSL Mode: disable

# 2. Or use correct Docker network hostname if Grafana is in same Docker network:
# - Host: postgres:5432

# 3. Restart Grafana
docker-compose restart grafana

# 4. Re-test data source in Grafana UI: Configuration → Data Sources → postgres → Save & Test

---

Issue 3: Prometheus Metrics Not Showing

Symptoms:

• Panel 2 (Feature Extraction Latency) shows "No data"
• Panel 6, 7, 8 (Rollback alerts) show "No data"

Diagnosis:

# 1. Check Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="ml_training_service") | {health, lastError}'

# 2. Check Prometheus can scrape ML service
curl http://localhost:9094/metrics

# 3. Verify metric exists in Prometheus
curl -s 'http://localhost:9090/api/v1/query?query=wave_d_feature_extraction_duration_seconds_count' | jq '.data.result'

Solution:

# 1. Ensure ML service exposes /metrics endpoint
# Add to services/ml_training_service/src/main.rs:
#
# use prometheus::{Encoder, TextEncoder};
# use actix_web::{web, App, HttpResponse, HttpServer};
#
# async fn metrics_handler() -> HttpResponse {
#     let encoder = TextEncoder::new();
#     let metric_families = prometheus::gather();
#     let mut buffer = vec![];
#     encoder.encode(&metric_families, &mut buffer).unwrap();
#     HttpResponse::Ok().body(buffer)
# }
#
# HttpServer::new(|| {
#     App::new()
#         .route("/metrics", web::get().to(metrics_handler))
# })
# .bind("0.0.0.0:9094")?
# .run()
# .await?;

# 2. Update Prometheus scrape config (see "Prometheus Scrape Configuration" section)

# 3. Reload Prometheus
curl -X POST http://localhost:9090/-/reload

# 4. Wait 15-30 seconds for first scrape, then verify
curl -s 'http://localhost:9090/api/v1/query?query=up{job="ml_training_service"}' | jq '.data.result[0].value[1]'
# Expected: "1" (service is up)

---

Issue 4: Dashboard Queries Timeout

Symptoms:

• Panels show "Timeout" error
• Queries take >30 seconds

Diagnosis:

# 1. Check query performance directly
time psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT
  event_timestamp AS time,
  symbol,
  from_regime || ' → ' || to_regime AS metric
FROM regime_transitions
WHERE event_timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY event_timestamp ASC
"

# 2. Check table sizes
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT
  schemaname,
  tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size,
  n_live_tup AS row_count
FROM pg_stat_user_tables
WHERE tablename LIKE 'regime_%'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
"

# 3. Check missing indexes
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT indexname, indexdef
FROM pg_indexes
WHERE tablename LIKE 'regime_%'
ORDER BY tablename, indexname;
"

Solution:

# 1. Ensure indexes from migration 045 are applied
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt <<EOF
-- Verify indexes exist (should already be created by migration 045)
-- If missing, create them:

-- regime_states indexes
CREATE INDEX IF NOT EXISTS idx_regime_states_symbol_timestamp ON regime_states(symbol, event_timestamp DESC);
CREATE INDEX IF NOT EXISTS idx_regime_states_regime ON regime_states(regime);
CREATE INDEX IF NOT EXISTS idx_regime_states_confidence ON regime_states(confidence DESC);

-- regime_transitions indexes
CREATE INDEX IF NOT EXISTS idx_regime_transitions_symbol_timestamp ON regime_transitions(symbol, event_timestamp DESC);
CREATE INDEX IF NOT EXISTS idx_regime_transitions_from_to ON regime_transitions(from_regime, to_regime);
CREATE INDEX IF NOT EXISTS idx_regime_transitions_symbol_from_to ON regime_transitions(symbol, from_regime, to_regime);

-- adaptive_strategy_metrics indexes
CREATE INDEX IF NOT EXISTS idx_adaptive_metrics_symbol_timestamp ON adaptive_strategy_metrics(symbol, event_timestamp DESC);
CREATE INDEX IF NOT EXISTS idx_adaptive_metrics_regime ON adaptive_strategy_metrics(regime);
CREATE INDEX IF NOT EXISTS idx_adaptive_metrics_sharpe ON adaptive_strategy_metrics(regime_sharpe DESC) WHERE regime_sharpe IS NOT NULL;

-- Analyze tables for query planner
ANALYZE regime_states;
ANALYZE regime_transitions;
ANALYZE adaptive_strategy_metrics;
EOF

# 2. Increase Grafana query timeout (default: 30s)
# Edit docker-compose.yml:
# grafana:
#   environment:
#     - GF_DATAPROXY_TIMEOUT=60

# 3. Restart Grafana
docker-compose restart grafana

# 4. Consider partitioning tables if >10M rows (see TimescaleDB hypertable conversion)

---

Issue 5: Incorrect Time Range

Symptoms:

• Dashboard shows data from wrong time period
• "No data" but database has recent rows

Diagnosis:

# 1. Check database timestamps
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT
  'regime_states' AS table_name,
  MIN(event_timestamp) AS oldest,
  MAX(event_timestamp) AS newest,
  COUNT(*) AS total_rows
FROM regime_states
UNION ALL
SELECT
  'regime_transitions' AS table_name,
  MIN(event_timestamp) AS oldest,
  MAX(event_timestamp) AS newest,
  COUNT(*) AS total_rows
FROM regime_transitions;
"

# 2. Check Grafana time range picker
# Dashboard top-right: Should show "Last 24 hours" or "now-24h to now"

# 3. Check server time vs. dashboard time
date -u  # Server time (UTC)
# Compare with Grafana dashboard time picker

Solution:

# 1. Ensure database timestamps are in UTC (PostgreSQL TIMESTAMPTZ)
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SHOW timezone;"
# Expected: UTC

# 2. Update Grafana dashboard timezone
# Dashboard Settings → Time options → Timezone: UTC

# 3. Verify data exists in last 24 hours
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "
SELECT COUNT(*) FROM regime_states WHERE event_timestamp >= NOW() - INTERVAL '24 hours';
"
# If 0, insert test data (see Issue 1 solution)

---

Production Deployment Checklist

Before deploying to production, ensure:

Database

• Migration 045 applied successfully (cargo sqlx migrate run)
• All 3 tables exist: regime_states, regime_transitions, adaptive_strategy_metrics
• All 3 functions exist: get_latest_regime, get_regime_transition_matrix, get_regime_performance
• Indexes verified with \di regime_* in psql
• Permissions granted to foxhunt user
• Backup scheduled (hourly for Wave D tables)

Prometheus

• ML Training Service metrics endpoint exposed at http://localhost:9094/metrics
• Scrape config updated with all 4 services (API Gateway, Trading, Backtesting, ML Training)
• Alert rules loaded from /etc/prometheus/alerts/wave_d_rollback.yml
• Scrape interval: 15s
• Retention: 30 days minimum
• Storage: 10GB minimum for 30-day retention

Grafana

• PostgreSQL data source configured with postgres UID
• Prometheus data source configured with prometheus UID
• Wave D dashboard imported successfully
• All 8 panels showing data (test with dummy data if needed)
• Alert rules linked to dashboard (see Panel 5-8)
• Dashboard starred/favorited for quick access
• Refresh interval: 10s
• Auto-refresh enabled
• Provisioning configured for persistent deployment

Monitoring

• Prometheus alerts configured for 5 rollback triggers
• Alert notifications configured (Slack, PagerDuty, email)
• On-call rotation established for critical alerts
• Rollback procedures tested (LEVEL_1_ROLLBACK_TEST.sh, LEVEL_3_ROLLBACK_TEST.sh)
• Dashboard URL bookmarked for ops team
• Runbooks created for common issues (see "Troubleshooting" section)

Performance

• Database indexes optimized (EXPLAIN ANALYZE on slow queries)
• Grafana query timeout increased to 60s (if needed)
• Prometheus storage optimized (SSD for fast queries)
• TimescaleDB hypertables configured (if >10M rows)
• Query performance baseline documented (<1s P99 for all panels)

Security

• Grafana admin password changed from default (admin/foxhunt123 → production password)
• PostgreSQL password changed from default (foxhunt_dev_password → production password)
• Grafana HTTPS enabled (production only)
• Prometheus metrics endpoint authentication enabled (production only)
• Database connections over SSL (production only)
• Audit logging enabled for Grafana configuration changes

---

Next Steps

1. Deploy Dashboard (10 minutes):

   # Follow "Installation" section (automated method recommended)
   cd /home/jgrusewski/Work/foxhunt
   # ... (see Installation section)
   

2. Configure Prometheus Metrics (30 minutes):

   # Add metrics instrumentation to ML Training Service
   # See "Prometheus Metrics Configuration" section
   

3. Test Dashboard with Live Data (1 hour):

   # Run backtest to generate regime transitions
   cargo run --release -p backtesting_service --example wave_d_backtest
   
   # Verify data in dashboard
   xdg-open http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection
   

4. Configure Alert Notifications (30 minutes):

   # Add Prometheus Alertmanager config
   # See "Alert Rules Configuration" section
   

5. Production Deployment (as per ROLLBACK_PROCEDURES.md):

   # Complete "Production Deployment Checklist" above
   # Deploy with monitoring enabled
   # Monitor dashboard for 24 hours before live trading
   

---

References

• Dashboard File: /home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json
• Database Migration: /home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.sql
• Rollback Procedures: /home/jgrusewski/Work/foxhunt/ROLLBACK_PROCEDURES.md
• Grafana Documentation: https://grafana.com/docs/grafana/latest/
• Prometheus Documentation: https://prometheus.io/docs/
• TimescaleDB Documentation: https://docs.timescale.com/

---

END OF GUIDE