# Grafana Wave D Dashboard Setup Guide **Author**: Agent M2 - Grafana Dashboard Deployment Specialist **Date**: 2025-10-19 **System**: Foxhunt HFT Trading System **Version**: Wave D (225 features) --- ## Executive Summary This guide provides step-by-step instructions for deploying the **Wave D Regime Detection & Adaptive Strategies** Grafana dashboard. The dashboard includes 8 panels covering regime transitions, feature extraction performance, regime distribution, adaptive strategy metrics, and 4 critical rollback alert panels. **Dashboard File**: `/home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json` **Key Capabilities**: - Real-time regime transition monitoring with CUSUM alert visualization - Feature extraction latency tracking (P50/P99/Average) - Regime distribution pie chart (7 regime types) - Adaptive strategy metrics (position sizing, stop-loss, Sharpe ratio, risk budget) - 4 rollback alert panels (flip-flopping, false positives, data corruption, system health) --- ## Prerequisites ### 1. Infrastructure Requirements **Docker Services (must be running)**: ```bash # Check Docker services docker-compose ps # Expected services: # - postgres (TimescaleDB) # - prometheus # - grafana # - redis # - vault ``` **Database Migration**: ```bash # Ensure migration 045 is applied cd /home/jgrusewski/Work/foxhunt cargo sqlx migrate run # Verify Wave D tables exist psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "\dt regime_*" # Expected tables: # - regime_states # - regime_transitions # - adaptive_strategy_metrics ``` ### 2. Data Source Configuration **PostgreSQL Data Source**: - **Name**: `postgres` - **Type**: PostgreSQL - **Host**: `localhost:5432` - **Database**: `foxhunt` - **User**: `foxhunt` - **Password**: `foxhunt_dev_password` - **SSL Mode**: `disable` (development) / `require` (production) - **Version**: TimescaleDB 2.x **Prometheus Data Source**: - **Name**: `prometheus` - **Type**: Prometheus - **URL**: `http://localhost:9090` - **Access**: Server (default) - **Scrape Interval**: 15s --- ## Installation ### Step 1: Configure Data Sources #### Option A: Manual Configuration (Grafana UI) 1. **Login to Grafana**: ```bash # Open browser http://localhost:3000 # Credentials Username: admin Password: foxhunt123 ``` 2. **Add PostgreSQL Data Source**: - Navigate to **Configuration** → **Data Sources** → **Add data source** - Select **PostgreSQL** - Configure: - Name: `postgres` - Host: `localhost:5432` - Database: `foxhunt` - User: `foxhunt` - Password: `foxhunt_dev_password` - SSL Mode: `disable` - Version: `12.0+` - TimescaleDB: **Enabled** - Click **Save & Test** (should see "Database Connection OK") 3. **Add Prometheus Data Source**: - Navigate to **Configuration** → **Data Sources** → **Add data source** - Select **Prometheus** - Configure: - Name: `prometheus` - URL: `http://localhost:9090` - Access: `Server (default)` - Scrape interval: `15s` - Click **Save & Test** (should see "Data source is working") #### Option B: Automated Configuration (Recommended) ```bash # Create Grafana provisioning directory mkdir -p /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/datasources # Create datasource configuration cat > /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/datasources/wave_d.yml <<'EOF' apiVersion: 1 datasources: - name: postgres type: postgres access: proxy url: localhost:5432 database: foxhunt user: foxhunt secureJsonData: password: foxhunt_dev_password jsonData: sslmode: disable postgresVersion: 1200 timescaledb: true isDefault: false editable: true - name: prometheus type: prometheus access: proxy url: http://localhost:9090 isDefault: true editable: true jsonData: timeInterval: 15s EOF # Restart Grafana to apply configuration docker-compose restart grafana ``` ### Step 2: Import Wave D Dashboard #### Option A: Manual Import (Grafana UI) 1. **Navigate to Dashboards**: - Click **+ (Create)** → **Import** 2. **Upload JSON**: - Click **Upload JSON file** - Select: `/home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json` - Or copy-paste the entire JSON content 3. **Configure Import**: - Dashboard name: **Wave D - Regime Detection & Adaptive Strategies** (auto-populated) - Folder: Select **Foxhunt** or create new folder - UID: `wave_d_regime_detection` (auto-populated) - PostgreSQL data source: Select `postgres` - Prometheus data source: Select `prometheus` 4. **Import**: - Click **Import** - Dashboard should load immediately with 8 panels #### Option B: Automated Import (Recommended) ```bash # Method 1: Grafana API (requires Grafana to be running) GRAFANA_URL="http://localhost:3000" GRAFANA_USER="admin" GRAFANA_PASS="foxhunt123" DASHBOARD_FILE="/home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json" # Import dashboard via API curl -X POST \ -H "Content-Type: application/json" \ -u "${GRAFANA_USER}:${GRAFANA_PASS}" \ -d @"${DASHBOARD_FILE}" \ "${GRAFANA_URL}/api/dashboards/db" # Expected response: {"id":1,"slug":"wave-d-regime-detection","status":"success","uid":"wave_d_regime_detection","url":"/d/wave_d_regime_detection/wave-d-regime-detection","version":1} ``` ```bash # Method 2: Provisioning (persistent across Grafana restarts) mkdir -p /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/dashboards # Create provisioning config cat > /home/jgrusewski/Work/foxhunt/config/grafana/provisioning/dashboards/wave_d.yml <<'EOF' apiVersion: 1 providers: - name: 'Wave D Dashboards' orgId: 1 folder: 'Foxhunt' type: file disableDeletion: false updateIntervalSeconds: 10 allowUiUpdates: true options: path: /home/jgrusewski/Work/foxhunt/config/grafana/dashboards foldersFromFilesStructure: false EOF # Restart Grafana to apply provisioning docker-compose restart grafana # Dashboard will auto-load on startup ``` ### Step 3: Verify Dashboard Functionality ```bash # 1. Check data sources are connected curl -u admin:foxhunt123 http://localhost:3000/api/datasources | jq '.[] | {name, type, url}' # Expected output: # {"name":"postgres","type":"postgres","url":"localhost:5432"} # {"name":"prometheus","type":"prometheus","url":"http://localhost:9090"} # 2. Test PostgreSQL queries psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt <= NOW() - INTERVAL '24 hours' ORDER BY event_timestamp ASC LIMIT 5; -- Test Panel 3: Regime Distribution SELECT regime AS metric, COUNT(*) AS value FROM regime_states WHERE event_timestamp >= NOW() - INTERVAL '24 hours' GROUP BY regime ORDER BY value DESC; EOF # 3. Test Prometheus metrics curl -s http://localhost:9090/api/v1/query?query=wave_d_feature_extraction_duration_seconds_bucket | jq '.data.result | length' # Expected: >0 (metrics are being collected) # 4. Open dashboard in browser xdg-open "http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection" 2>/dev/null || \ open "http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection" 2>/dev/null || \ echo "Open manually: http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection" ``` --- ## Dashboard Panels ### Panel 1: Regime Transitions Timeline (Timeseries) **Purpose**: Visualize regime changes over time with CUSUM alert triggers. **Data Source**: PostgreSQL (`postgres`) **SQL Query**: ```sql SELECT event_timestamp AS time, symbol, from_regime || ' → ' || to_regime AS metric, 1 AS value, CASE WHEN cusum_alert_triggered THEN 'CUSUM Alert' ELSE 'Normal' END AS alert_type FROM regime_transitions WHERE event_timestamp >= NOW() - INTERVAL '24 hours' ORDER BY event_timestamp ASC ``` **Visualization**: - Type: Timeseries (points) - X-axis: Time (24 hours) - Y-axis: Regime transitions (discrete events) - Legend: Transition labels (e.g., "Normal → Trending") - Alert markers: Red points for CUSUM-triggered transitions (size: 12px) - Normal markers: Colored points for regular transitions (size: 8px) **Interpretation**: - **5-10 transitions/day**: Normal market behavior - **>30 transitions/hour**: WARNING - Potential flip-flopping - **>50 transitions/hour**: CRITICAL - Trigger Level 1 rollback (ROLLBACK_PROCEDURES.md) - **Red points**: CUSUM structural break detected (high confidence transition) **Example Output**: ``` Time Transition Alert Type 2025-10-19 10:15:00 Normal → Trending Normal 2025-10-19 11:30:00 Trending → Volatile CUSUM Alert (RED) 2025-10-19 13:45:00 Volatile → Ranging Normal ``` --- ### Panel 2: Feature Extraction Latency (P50/P99) (Timeseries) **Purpose**: Monitor Wave D feature extraction performance. Target: <1ms (1000μs). **Data Source**: Prometheus (`prometheus`) **PromQL Queries**: ```promql # P50 Latency (median) histogram_quantile(0.50, rate(wave_d_feature_extraction_duration_seconds_bucket[5m])) * 1000 # P99 Latency (99th percentile) histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m])) * 1000 # Average Latency avg(rate(wave_d_feature_extraction_duration_seconds_sum[5m]) / rate(wave_d_feature_extraction_duration_seconds_count[5m])) * 1000 ``` **Visualization**: - Type: Timeseries (smooth lines) - X-axis: Time (24 hours) - Y-axis: Latency (milliseconds) - Legend: P50 (blue), P99 (orange, bold), Average (green) - Thresholds: - Green: 0-1ms (target met) - Yellow: 1-2ms (warning) - Red: >2ms (critical, >2x target) **Interpretation**: - **P50 <0.5ms**: Excellent performance (50% of extractions) - **P99 <1ms**: Target met (99% of extractions) - **P99 1-2ms**: WARNING - Performance degradation - **P99 >2ms**: CRITICAL - Trigger Level 1 rollback if persistent >15 min **Example Prometheus Metrics**: ```promql # Sample metrics (generated by ML service) wave_d_feature_extraction_duration_seconds_bucket{le="0.001"} 450 wave_d_feature_extraction_duration_seconds_bucket{le="0.002"} 490 wave_d_feature_extraction_duration_seconds_bucket{le="+Inf"} 500 wave_d_feature_extraction_duration_seconds_sum 0.125 wave_d_feature_extraction_duration_seconds_count 500 # Calculated P99 = 0.25ms (excellent) ``` --- ### Panel 3: Regime Distribution (24h) (Pie Chart) **Purpose**: Visualize the percentage distribution of detected regimes over the last 24 hours. **Data Source**: PostgreSQL (`postgres`) **SQL Query**: ```sql SELECT regime AS metric, COUNT(*) AS value FROM regime_states WHERE event_timestamp >= NOW() - INTERVAL '24 hours' GROUP BY regime ORDER BY value DESC ``` **Visualization**: - Type: Pie chart - Legend: Right side, table format with value and percentage - Labels: Percentage on slices - Color mapping (7 regime types): - **Normal**: Light green (default market conditions) - **Trending**: Green (directional movement) - **Ranging**: Blue (sideways/choppy) - **Volatile**: Orange (high volatility) - **Crisis**: Red (extreme conditions) - **Illiquid**: Yellow (low liquidity) - **Momentum**: Purple (strong directional) **Interpretation**: - **Normal 40-60%**: Healthy market balance - **Trending 20-30%**: Good directional opportunities - **Ranging 15-25%**: Consolidation phases - **Volatile <10%**: Acceptable risk levels - **Crisis <5%**: Rare events (expected) - **Distribution changes >50% in 1 hour**: Potential market regime shift **Example Output**: ``` Regime Count Percentage Normal 450 45% Trending 250 25% Ranging 200 20% Volatile 80 8% Momentum 15 1.5% Crisis 3 0.3% Illiquid 2 0.2% ``` --- ### Panel 4: Adaptive Strategy Metrics (Real-time) (Timeseries) **Purpose**: Track position sizing and stop-loss adjustments by regime. **Data Source**: PostgreSQL (`postgres`) **SQL Queries** (4 metrics, dual Y-axis): **Query A: Position Multiplier** (Left Y-axis: 0-2): ```sql SELECT event_timestamp AS time, symbol || ' - ' || regime AS metric, position_multiplier AS value FROM adaptive_strategy_metrics WHERE event_timestamp >= NOW() - INTERVAL '24 hours' ORDER BY event_timestamp ASC ``` **Query B: Stop-Loss Multiplier** (Left Y-axis: 1-5): ```sql SELECT event_timestamp AS time, symbol || ' - ' || regime AS metric, stop_loss_multiplier AS value FROM adaptive_strategy_metrics WHERE event_timestamp >= NOW() - INTERVAL '24 hours' ORDER BY event_timestamp ASC ``` **Query C: Regime Sharpe Ratio** (Right Y-axis: 0+): ```sql SELECT event_timestamp AS time, symbol || ' - ' || regime AS metric, regime_sharpe AS value FROM adaptive_strategy_metrics WHERE event_timestamp >= NOW() - INTERVAL '24 hours' AND regime_sharpe IS NOT NULL ORDER BY event_timestamp ASC ``` **Query D: Risk Budget Utilization** (Right Y-axis: 0-100%): ```sql SELECT event_timestamp AS time, symbol || ' - ' || regime AS metric, risk_budget_utilization * 100 AS value FROM adaptive_strategy_metrics WHERE event_timestamp >= NOW() - INTERVAL '24 hours' AND risk_budget_utilization IS NOT NULL ORDER BY event_timestamp ASC ``` **Visualization**: - Type: Timeseries (smooth lines, dual Y-axis) - X-axis: Time (24 hours) - Left Y-axis: Position multiplier (0-2), Stop-loss multiplier (1-5) - Right Y-axis: Sharpe ratio (0+), Risk budget (0-100%) - Legend: Table format with mean, max, last value - Colors: - Position Multiplier: Blue - Stop-Loss Multiplier: Orange - Regime Sharpe: Green - Risk Budget: Purple **Interpretation**: **Position Multiplier** (0.2x-1.5x range): - **0.2x**: Crisis regime (minimal exposure) - **0.5x**: Volatile regime (reduced size) - **1.0x**: Normal regime (baseline) - **1.5x**: Trending regime (max size) **Stop-Loss Multiplier** (1.5x-4.0x ATR range): - **1.5x ATR**: Trending regime (tight stops) - **2.0x ATR**: Normal regime (baseline) - **3.0x ATR**: Ranging regime (wider stops, avoid whipsaws) - **4.0x ATR**: Volatile regime (max stops) **Regime Sharpe Ratio** (>1.5 target): - **<1.0**: Poor risk-adjusted returns (review strategy) - **1.0-1.5**: Acceptable performance - **>1.5**: Target met (expected +25-50% improvement vs. Wave C) - **>2.0**: Excellent performance **Risk Budget Utilization** (<80% target): - **<50%**: Conservative (safe margin) - **50-80%**: Target range (balanced risk) - **80-100%**: WARNING - High risk exposure - **>100%**: CRITICAL - Risk limit breach (should not occur) **Example Output**: ``` Time Symbol - Regime Pos Mult Stop Mult Sharpe Risk % 2025-10-19 10:00:00 ES.FUT - Trending 1.5x 1.5x ATR 1.8 65% 2025-10-19 11:00:00 ES.FUT - Volatile 0.5x 4.0x ATR 1.2 45% 2025-10-19 12:00:00 ES.FUT - Ranging 1.0x 3.0x ATR 1.4 55% ``` --- ### Panel 5: Rollback Alert - Flip-Flopping Detection (Stat) **Purpose**: Monitor for excessive regime transitions (>50/hour triggers Level 1 rollback). **Data Source**: PostgreSQL (`postgres`) **SQL Query**: ```sql SELECT COUNT(*) AS value FROM regime_transitions WHERE event_timestamp >= NOW() - INTERVAL '1 hour' ``` **Visualization**: - Type: Stat (big number with background color) - Thresholds: - Green: 0-29 transitions/hour (normal) - Yellow: 30-49 transitions/hour (warning) - Red: ≥50 transitions/hour (CRITICAL) - Text: "Transitions/Hour" with large value **Interpretation**: - **0-10**: Normal market behavior - **10-30**: Active regime changes (acceptable) - **30-50**: WARNING - Potential flip-flopping - **≥50**: CRITICAL - LEVEL 1 ROLLBACK REQUIRED (ROLLBACK_PROCEDURES.md) **Alert Action**: ```bash # If ≥50 transitions/hour, execute Level 1 rollback cd /home/jgrusewski/Work/foxhunt ./LEVEL_1_ROLLBACK_TEST.sh # Zero downtime, <1 minute ``` --- ### Panel 6: Rollback Alert - False Positives (Stat) **Purpose**: Monitor regime detection accuracy (>80% error rate triggers Level 1 rollback). **Data Source**: Prometheus (`prometheus`) **PromQL Query**: ```promql (sum(regime_detection_errors_total) / sum(regime_detections_total)) * 100 ``` **Visualization**: - Type: Stat (big number with background color) - Thresholds: - Green: 0-49% error rate (acceptable) - Yellow: 50-79% error rate (warning) - Red: ≥80% error rate (CRITICAL) - Unit: Percentage (%) - Text: "Error Rate (%)" with large value **Interpretation**: - **0-20%**: Excellent accuracy (>80% correct) - **20-50%**: Acceptable accuracy (50-80% correct) - **50-80%**: WARNING - High false positive rate - **≥80%**: CRITICAL - LEVEL 1 ROLLBACK REQUIRED **Alert Action**: ```bash # If ≥80% error rate, execute Level 1 rollback cd /home/jgrusewski/Work/foxhunt ./LEVEL_1_ROLLBACK_TEST.sh # Zero downtime, <1 minute ``` **Note**: This metric requires Prometheus instrumentation in ML service: ```rust // ml_training_service/src/metrics.rs lazy_static! { pub static ref REGIME_DETECTIONS_TOTAL: IntCounter = register_int_counter!( "regime_detections_total", "Total regime detections" ).unwrap(); pub static ref REGIME_DETECTION_ERRORS_TOTAL: IntCounter = register_int_counter!( "regime_detection_errors_total", "Total regime detection errors" ).unwrap(); } // Increment on detection REGIME_DETECTIONS_TOTAL.inc(); // Increment on error (NaN, Inf, out-of-range) if regime.is_nan() || regime.is_infinite() { REGIME_DETECTION_ERRORS_TOTAL.inc(); } ``` --- ### Panel 7: Rollback Alert - Data Corruption (Stat) **Purpose**: Detect NaN/Inf values in Wave D features (triggers immediate Level 3 rollback). **Data Source**: Prometheus (`prometheus`) **PromQL Query**: ```promql wave_d_features_nan_count + wave_d_features_inf_count ``` **Visualization**: - Type: Stat (big number with background color) - Thresholds: - Green: 0 (no corruption) - Red: ≥1 (ANY corruption is CRITICAL) - Text: "NaN/Inf Count" with large value **Interpretation**: - **0**: No data corruption (normal) - **≥1**: CRITICAL - IMMEDIATE LEVEL 3 ROLLBACK REQUIRED **Alert Action**: ```bash # If ANY NaN/Inf detected, execute Level 3 rollback IMMEDIATELY cd /home/jgrusewski/Work/foxhunt ./LEVEL_3_ROLLBACK_TEST.sh # Full rollback to Wave C, ~15 minutes ``` **Note**: This metric requires Prometheus instrumentation in ML service: ```rust // ml_training_service/src/metrics.rs lazy_static! { pub static ref WAVE_D_FEATURES_NAN_COUNT: IntCounter = register_int_counter!( "wave_d_features_nan_count", "Count of NaN values in Wave D features" ).unwrap(); pub static ref WAVE_D_FEATURES_INF_COUNT: IntCounter = register_int_counter!( "wave_d_features_inf_count", "Count of Inf values in Wave D features" ).unwrap(); } // Check features after extraction for feature in &wave_d_features { if feature.is_nan() { WAVE_D_FEATURES_NAN_COUNT.inc(); error!("NaN detected in Wave D feature extraction"); } if feature.is_infinite() { WAVE_D_FEATURES_INF_COUNT.inc(); error!("Inf detected in Wave D feature extraction"); } } ``` --- ### Panel 8: System Health (Stat) **Purpose**: Monitor service uptime (down >5 minutes triggers Level 3 rollback). **Data Source**: Prometheus (`prometheus`) **PromQL Queries** (3 services): ```promql # ML Training Service up{job="ml_training_service"} # Trading Service up{job="trading_service"} # API Gateway up{job="api_gateway"} ``` **Visualization**: - Type: Stat (horizontal layout with 3 values) - Mappings: - 0 → "DOWN" (red background) - 1 → "UP" (green background) - Text size: Medium (24px) - Display: Service name + status **Interpretation**: - **All services UP (1)**: Normal operation - **Any service DOWN (0) for <5 minutes**: Transient issue (monitor) - **Any service DOWN (0) for ≥5 minutes**: CRITICAL - LEVEL 3 ROLLBACK **Alert Action**: ```bash # If any service down ≥5 minutes, execute Level 3 rollback cd /home/jgrusewski/Work/foxhunt ./LEVEL_3_ROLLBACK_TEST.sh # Full rollback to Wave C, ~15 minutes ``` **Example Output**: ``` ML Training: UP (green) Trading: UP (green) API Gateway: DOWN (red) # CRITICAL if >5 min ``` --- ## Prometheus Metrics Configuration ### Required Metrics The Wave D dashboard requires the following Prometheus metrics to be exposed by the ML Training Service: **File**: `services/ml_training_service/src/metrics.rs` ```rust use lazy_static::lazy_static; use prometheus::{IntCounter, Histogram, register_int_counter, register_histogram}; lazy_static! { // Panel 2: Feature Extraction Latency pub static ref WAVE_D_FEATURE_EXTRACTION_DURATION: Histogram = register_histogram!( "wave_d_feature_extraction_duration_seconds", "Wave D feature extraction duration in seconds", vec![0.0001, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05] ).unwrap(); // Panel 5: Flip-Flopping Detection (tracked in PostgreSQL) // Panel 6: False Positives pub static ref REGIME_DETECTIONS_TOTAL: IntCounter = register_int_counter!( "regime_detections_total", "Total regime detections performed" ).unwrap(); pub static ref REGIME_DETECTION_ERRORS_TOTAL: IntCounter = register_int_counter!( "regime_detection_errors_total", "Total regime detection errors (NaN, Inf, out-of-range)" ).unwrap(); // Panel 7: Data Corruption pub static ref WAVE_D_FEATURES_NAN_COUNT: IntCounter = register_int_counter!( "wave_d_features_nan_count", "Count of NaN values detected in Wave D features" ).unwrap(); pub static ref WAVE_D_FEATURES_INF_COUNT: IntCounter = register_int_counter!( "wave_d_features_inf_count", "Count of Inf values detected in Wave D features" ).unwrap(); // Panel 8: System Health (auto-collected by Prometheus) // Metric: up{job="ml_training_service"} // Metric: up{job="trading_service"} // Metric: up{job="api_gateway"} } // Usage in feature extraction code pub fn extract_wave_d_features() -> Result, CommonError> { let _timer = WAVE_D_FEATURE_EXTRACTION_DURATION.start_timer(); REGIME_DETECTIONS_TOTAL.inc(); let features = /* extraction logic */; // Validate features for feature in &features { if feature.is_nan() { WAVE_D_FEATURES_NAN_COUNT.inc(); REGIME_DETECTION_ERRORS_TOTAL.inc(); return Err(CommonError::validation("NaN detected in Wave D features")); } if feature.is_infinite() { WAVE_D_FEATURES_INF_COUNT.inc(); REGIME_DETECTION_ERRORS_TOTAL.inc(); return Err(CommonError::validation("Inf detected in Wave D features")); } } Ok(features) } ``` ### Prometheus Scrape Configuration **File**: `/etc/prometheus/prometheus.yml` (or Docker volume mount) ```yaml global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: # ML Training Service - job_name: 'ml_training_service' static_configs: - targets: ['localhost:9094'] metrics_path: '/metrics' # Trading Service - job_name: 'trading_service' static_configs: - targets: ['localhost:9092'] metrics_path: '/metrics' # API Gateway - job_name: 'api_gateway' static_configs: - targets: ['localhost:9091'] metrics_path: '/metrics' # Backtesting Service - job_name: 'backtesting_service' static_configs: - targets: ['localhost:9093'] metrics_path: '/metrics' ``` **Verify Metrics Collection**: ```bash # Check Prometheus targets curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job, health, lastScrape}' # Expected output: # {"job":"ml_training_service","health":"up","lastScrape":"2025-10-19T10:30:15Z"} # {"job":"trading_service","health":"up","lastScrape":"2025-10-19T10:30:15Z"} # {"job":"api_gateway","health":"up","lastScrape":"2025-10-19T10:30:15Z"} # Test Wave D metrics curl -s http://localhost:9094/metrics | grep wave_d_feature_extraction_duration_seconds # Expected output (histogram buckets): # wave_d_feature_extraction_duration_seconds_bucket{le="0.001"} 450 # wave_d_feature_extraction_duration_seconds_bucket{le="0.002"} 490 # wave_d_feature_extraction_duration_seconds_sum 0.125 # wave_d_feature_extraction_duration_seconds_count 500 ``` --- ## Alert Rules Configuration ### Prometheus Alert Rules **File**: `/etc/prometheus/alerts/wave_d_rollback.yml` ```yaml groups: - name: wave_d_rollback_triggers interval: 30s rules: # CRITICAL: Flip-flopping (>50 transitions/hour) - alert: WaveDFlipFlopping expr: rate(regime_transitions_total[1h]) > 50 for: 5m labels: severity: critical rollback_level: level_1 annotations: summary: "Wave D flip-flopping detected ({{ $value }} transitions/hour)" description: "Regime detection is changing states >50 times/hour. Recommend Level 1 rollback." runbook: "ROLLBACK_PROCEDURES.md#level-1-feature-only-rollback-zero-downtime" # CRITICAL: False positives (>80% error rate) - alert: WaveDFalsePositives expr: (sum(regime_detection_errors_total) / sum(regime_detections_total)) > 0.80 for: 10m labels: severity: critical rollback_level: level_1 annotations: summary: "Wave D false positive rate >80%" description: "Regime detection accuracy below threshold. Recommend Level 1 rollback." # WARNING: Performance degradation (>2x latency) - alert: WaveDLatencyDegradation expr: histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m])) > 0.002 for: 15m labels: severity: warning rollback_level: level_1 annotations: summary: "Wave D feature extraction latency >2ms (>2x target)" description: "Consider Level 1 rollback if latency persists." # CRITICAL: NaN/Inf in features - alert: WaveDDataCorruption expr: wave_d_features_nan_count > 0 OR wave_d_features_inf_count > 0 for: 1m labels: severity: critical rollback_level: level_3 annotations: summary: "Wave D data corruption detected (NaN/Inf values)" description: "IMMEDIATE LEVEL 3 ROLLBACK REQUIRED. Data integrity compromised." runbook: "ROLLBACK_PROCEDURES.md#level-3-full-rollback-to-wave-c" # CRITICAL: System unavailable - alert: FoxhuntSystemDown expr: up{job="foxhunt_services"} == 0 for: 5m labels: severity: critical rollback_level: level_3 annotations: summary: "Foxhunt system unavailable for >5 minutes" description: "Consider Level 3 rollback to Wave C baseline." ``` **Apply Alert Rules**: ```bash # Reload Prometheus configuration curl -X POST http://localhost:9090/-/reload # Verify rules loaded curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | {name, rules: .rules | length}' # Expected output: # {"name":"wave_d_rollback_triggers","rules":5} ``` --- ## Troubleshooting ### Issue 1: Dashboard Panels Show "No Data" **Symptoms**: - All panels show "No data" or empty graphs - PostgreSQL queries return 0 rows - Prometheus queries return empty results **Diagnosis**: ```bash # 1. Check database tables have data psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT COUNT(*) FROM regime_states;" psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT COUNT(*) FROM regime_transitions;" # 2. Check Prometheus metrics curl -s http://localhost:9094/metrics | grep wave_d_feature_extraction_duration_seconds_count # 3. Check service is running and collecting metrics docker-compose ps | grep ml_training_service curl http://localhost:9094/health ``` **Solution**: ```bash # If tables are empty, insert test data psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt < HttpResponse { # let encoder = TextEncoder::new(); # let metric_families = prometheus::gather(); # let mut buffer = vec![]; # encoder.encode(&metric_families, &mut buffer).unwrap(); # HttpResponse::Ok().body(buffer) # } # # HttpServer::new(|| { # App::new() # .route("/metrics", web::get().to(metrics_handler)) # }) # .bind("0.0.0.0:9094")? # .run() # .await?; # 2. Update Prometheus scrape config (see "Prometheus Scrape Configuration" section) # 3. Reload Prometheus curl -X POST http://localhost:9090/-/reload # 4. Wait 15-30 seconds for first scrape, then verify curl -s 'http://localhost:9090/api/v1/query?query=up{job="ml_training_service"}' | jq '.data.result[0].value[1]' # Expected: "1" (service is up) ``` --- ### Issue 4: Dashboard Queries Timeout **Symptoms**: - Panels show "Timeout" error - Queries take >30 seconds **Diagnosis**: ```bash # 1. Check query performance directly time psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c " SELECT event_timestamp AS time, symbol, from_regime || ' → ' || to_regime AS metric FROM regime_transitions WHERE event_timestamp >= NOW() - INTERVAL '24 hours' ORDER BY event_timestamp ASC " # 2. Check table sizes psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c " SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size, n_live_tup AS row_count FROM pg_stat_user_tables WHERE tablename LIKE 'regime_%' ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC; " # 3. Check missing indexes psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c " SELECT indexname, indexdef FROM pg_indexes WHERE tablename LIKE 'regime_%' ORDER BY tablename, indexname; " ``` **Solution**: ```bash # 1. Ensure indexes from migration 045 are applied psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt <10M rows (see TimescaleDB hypertable conversion) ``` --- ### Issue 5: Incorrect Time Range **Symptoms**: - Dashboard shows data from wrong time period - "No data" but database has recent rows **Diagnosis**: ```bash # 1. Check database timestamps psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c " SELECT 'regime_states' AS table_name, MIN(event_timestamp) AS oldest, MAX(event_timestamp) AS newest, COUNT(*) AS total_rows FROM regime_states UNION ALL SELECT 'regime_transitions' AS table_name, MIN(event_timestamp) AS oldest, MAX(event_timestamp) AS newest, COUNT(*) AS total_rows FROM regime_transitions; " # 2. Check Grafana time range picker # Dashboard top-right: Should show "Last 24 hours" or "now-24h to now" # 3. Check server time vs. dashboard time date -u # Server time (UTC) # Compare with Grafana dashboard time picker ``` **Solution**: ```bash # 1. Ensure database timestamps are in UTC (PostgreSQL TIMESTAMPTZ) psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SHOW timezone;" # Expected: UTC # 2. Update Grafana dashboard timezone # Dashboard Settings → Time options → Timezone: UTC # 3. Verify data exists in last 24 hours psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c " SELECT COUNT(*) FROM regime_states WHERE event_timestamp >= NOW() - INTERVAL '24 hours'; " # If 0, insert test data (see Issue 1 solution) ``` --- ## Production Deployment Checklist Before deploying to production, ensure: ### Database - [ ] Migration 045 applied successfully (`cargo sqlx migrate run`) - [ ] All 3 tables exist: `regime_states`, `regime_transitions`, `adaptive_strategy_metrics` - [ ] All 3 functions exist: `get_latest_regime`, `get_regime_transition_matrix`, `get_regime_performance` - [ ] Indexes verified with `\di regime_*` in psql - [ ] Permissions granted to `foxhunt` user - [ ] Backup scheduled (hourly for Wave D tables) ### Prometheus - [ ] ML Training Service metrics endpoint exposed at `http://localhost:9094/metrics` - [ ] Scrape config updated with all 4 services (API Gateway, Trading, Backtesting, ML Training) - [ ] Alert rules loaded from `/etc/prometheus/alerts/wave_d_rollback.yml` - [ ] Scrape interval: 15s - [ ] Retention: 30 days minimum - [ ] Storage: 10GB minimum for 30-day retention ### Grafana - [ ] PostgreSQL data source configured with `postgres` UID - [ ] Prometheus data source configured with `prometheus` UID - [ ] Wave D dashboard imported successfully - [ ] All 8 panels showing data (test with dummy data if needed) - [ ] Alert rules linked to dashboard (see Panel 5-8) - [ ] Dashboard starred/favorited for quick access - [ ] Refresh interval: 10s - [ ] Auto-refresh enabled - [ ] Provisioning configured for persistent deployment ### Monitoring - [ ] Prometheus alerts configured for 5 rollback triggers - [ ] Alert notifications configured (Slack, PagerDuty, email) - [ ] On-call rotation established for critical alerts - [ ] Rollback procedures tested (LEVEL_1_ROLLBACK_TEST.sh, LEVEL_3_ROLLBACK_TEST.sh) - [ ] Dashboard URL bookmarked for ops team - [ ] Runbooks created for common issues (see "Troubleshooting" section) ### Performance - [ ] Database indexes optimized (EXPLAIN ANALYZE on slow queries) - [ ] Grafana query timeout increased to 60s (if needed) - [ ] Prometheus storage optimized (SSD for fast queries) - [ ] TimescaleDB hypertables configured (if >10M rows) - [ ] Query performance baseline documented (<1s P99 for all panels) ### Security - [ ] Grafana admin password changed from default (`admin/foxhunt123` → production password) - [ ] PostgreSQL password changed from default (`foxhunt_dev_password` → production password) - [ ] Grafana HTTPS enabled (production only) - [ ] Prometheus metrics endpoint authentication enabled (production only) - [ ] Database connections over SSL (production only) - [ ] Audit logging enabled for Grafana configuration changes --- ## Next Steps 1. **Deploy Dashboard** (10 minutes): ```bash # Follow "Installation" section (automated method recommended) cd /home/jgrusewski/Work/foxhunt # ... (see Installation section) ``` 2. **Configure Prometheus Metrics** (30 minutes): ```bash # Add metrics instrumentation to ML Training Service # See "Prometheus Metrics Configuration" section ``` 3. **Test Dashboard with Live Data** (1 hour): ```bash # Run backtest to generate regime transitions cargo run --release -p backtesting_service --example wave_d_backtest # Verify data in dashboard xdg-open http://localhost:3000/d/wave_d_regime_detection/wave-d-regime-detection ``` 4. **Configure Alert Notifications** (30 minutes): ```bash # Add Prometheus Alertmanager config # See "Alert Rules Configuration" section ``` 5. **Production Deployment** (as per ROLLBACK_PROCEDURES.md): ```bash # Complete "Production Deployment Checklist" above # Deploy with monitoring enabled # Monitor dashboard for 24 hours before live trading ``` --- ## References - **Dashboard File**: `/home/jgrusewski/Work/foxhunt/config/grafana/dashboards/wave_d_regime_detection.json` - **Database Migration**: `/home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.sql` - **Rollback Procedures**: `/home/jgrusewski/Work/foxhunt/ROLLBACK_PROCEDURES.md` - **Grafana Documentation**: https://grafana.com/docs/grafana/latest/ - **Prometheus Documentation**: https://prometheus.io/docs/ - **TimescaleDB Documentation**: https://docs.timescale.com/ --- **END OF GUIDE**