foxhunt/WAVE_D_ALERTS_DEPLOYMENT_GUIDE.md

Wave D Prometheus Alerts - Deployment Guide

Agent: M1 - Prometheus Alert Deployment Date: 2025-10-19 System: Foxhunt HFT Trading System Version: Wave D (225 features)

---

Executive Summary

This guide provides step-by-step instructions for deploying 9 Wave D Prometheus alert rules to monitor regime detection features and trigger automated rollback procedures when critical thresholds are exceeded.

Alert Categories:

• 5 Critical Alerts: Immediate rollback triggers (flip-flopping, false positives, data corruption, system down, latency)
• 4 Warning Alerts: Monitoring and early detection (memory leaks, regime coverage, transition rate, moderate errors)

Deployment Time: <5 minutes Validation Method: promtool syntax check + Prometheus UI verification Impact: Zero downtime (alerts load dynamically via Prometheus hot-reload)

---

Alert Rules Summary

Alert Name	Severity	Rollback Level	Trigger Condition	For Duration
WaveDFlipFlopping	Critical	Level 1	>50 transitions/hour	5 minutes
WaveDFalsePositives	Critical	Level 1	>80% error rate	10 minutes
WaveDDataCorruption	Critical	Level 3	NaN/Inf in features	1 minute
FoxhuntSystemDown	Critical	Level 3	System unavailable	5 minutes
WaveDLatencyDegradation	Warning	Level 1	>2ms P99 latency	15 minutes
WaveDMemoryLeak	Warning	Level 1	>20% RSS growth/hour	1 hour
WaveDRegimeCoverageHigh	Warning	None	>95% single regime	30 minutes
WaveDRegimeTransitionRateLow	Warning	None	<5 transitions/day	2 hours
WaveDDetectionErrorsModerate	Warning	None	20-80% error rate	30 minutes

---

Deployment Procedure

Prerequisites

1. Docker Compose Running:

   docker-compose ps | grep prometheus
   # Expected: foxhunt-prometheus running (healthy)
   

2. Prometheus Configuration Verified:

   ls -la config/prometheus/rules/wave_d_alerts.yml
   # Expected: File exists (created by Agent M1)
   

3. Prometheus Volume Mount Verified:

   docker inspect foxhunt-prometheus | jq '.[0].Mounts[] | select(.Destination == "/etc/prometheus/rules")'
   # Expected: Source = ./config/prometheus/rules (read-only)
   

Step 1: Validate Alert Syntax (30 seconds)

Using Docker Prometheus:

cd /home/jgrusewski/Work/foxhunt

# Validate alert rules syntax
docker exec foxhunt-prometheus promtool check rules /etc/prometheus/rules/wave_d_alerts.yml

# Expected output:
# Checking /etc/prometheus/rules/wave_d_alerts.yml
#   SUCCESS: 9 rules found

If promtool reports errors:

• Fix syntax errors in config/prometheus/rules/wave_d_alerts.yml
• Re-run validation
• Do NOT proceed until syntax is valid

Step 2: Hot-Reload Prometheus Configuration (15 seconds)

Option A: Prometheus Hot-Reload (Zero Downtime - PREFERRED):

# Send SIGHUP to Prometheus (reload config without restart)
docker exec foxhunt-prometheus kill -HUP 1

# Verify reload successful (check logs)
docker logs foxhunt-prometheus --tail 20

# Expected log message:
# "Completed loading of configuration file"

Option B: Full Prometheus Restart (5-10 seconds downtime):

# Only use if hot-reload fails
docker-compose restart prometheus

# Wait for health check
sleep 10
curl http://localhost:9090/-/healthy
# Expected: Prometheus is Healthy.

Step 3: Verify Alert Rules Loaded (30 seconds)

Method 1: Prometheus Web UI:

# Open browser to Prometheus alerts page
xdg-open http://localhost:9090/alerts

# OR use curl to list alerts
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules[].name'

# Expected output (9 alert names):
# WaveDFlipFlopping
# WaveDFalsePositives
# WaveDDataCorruption
# FoxhuntSystemDown
# WaveDLatencyDegradation
# WaveDMemoryLeak
# WaveDRegimeCoverageHigh
# WaveDRegimeTransitionRateLow
# WaveDDetectionErrorsModerate

Method 2: Prometheus API Query:

# Count loaded Wave D alerts
curl -s http://localhost:9090/api/v1/rules | \
  jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules | length'

# Expected: 9

Method 3: Check Alert Status:

# List all Wave D alerts with current state
curl -s http://localhost:9090/api/v1/alerts | \
  jq '.data.alerts[] | select(.labels.component | startswith("wave_d")) | {name: .labels.alertname, state: .state}'

# Expected (all alerts in "inactive" state before Wave D deployment):
# {"name":"WaveDFlipFlopping","state":"inactive"}
# {"name":"WaveDFalsePositives","state":"inactive"}
# ... (7 more)

Step 4: Validate Alert Annotations (15 seconds)

Check that all critical alerts have runbook URLs:

curl -s http://localhost:9090/api/v1/rules | \
  jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules[] | select(.labels.severity == "critical") | {name: .name, runbook: .annotations.runbook}'

# Expected: All 4 critical alerts have runbook URLs

Verify rollback_level labels:

curl -s http://localhost:9090/api/v1/rules | \
  jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules[] | {name: .name, rollback_level: .labels.rollback_level}'

# Expected:
# WaveDFlipFlopping → level_1
# WaveDFalsePositives → level_1
# WaveDDataCorruption → level_3
# FoxhuntSystemDown → level_3
# WaveDLatencyDegradation → level_1
# WaveDMemoryLeak → level_1
# (remaining alerts → none)

---

Testing Alert Rules

Synthetic Test Data (Pre-Deployment)

Before Wave D production deployment, test alerts using synthetic metrics:

Step 1: Expose Test Metrics Endpoint:

# Create test metrics endpoint (development only)
cat > /tmp/test_wave_d_metrics.prom <<'EOF'
# HELP regime_transitions_total Total regime transitions
# TYPE regime_transitions_total counter
regime_transitions_total 100

# HELP regime_detections_total Total regime detections
# TYPE regime_detections_total counter
regime_detections_total 1000

# HELP regime_detection_errors_total Regime detection errors
# TYPE regime_detection_errors_total counter
regime_detection_errors_total 850

# HELP wave_d_features_nan_count NaN values in Wave D features
# TYPE wave_d_features_nan_count gauge
wave_d_features_nan_count 0

# HELP wave_d_features_inf_count Inf values in Wave D features
# TYPE wave_d_features_inf_count gauge
wave_d_features_inf_count 0

# HELP wave_d_feature_extraction_duration_seconds Feature extraction latency
# TYPE wave_d_feature_extraction_duration_seconds histogram
wave_d_feature_extraction_duration_seconds_bucket{le="0.001"} 100
wave_d_feature_extraction_duration_seconds_bucket{le="0.002"} 200
wave_d_feature_extraction_duration_seconds_bucket{le="0.005"} 250
wave_d_feature_extraction_duration_seconds_bucket{le="+Inf"} 300
wave_d_feature_extraction_duration_seconds_sum 0.6
wave_d_feature_extraction_duration_seconds_count 300
EOF

# Serve test metrics (1-hour HTTP server)
cd /tmp
python3 -m http.server 8888 &
# Metrics available at: http://localhost:8888/test_wave_d_metrics.prom

Step 2: Configure Prometheus Scrape Job:

# Add to config/prometheus/prometheus.yml (temporary, for testing only)
scrape_configs:
  - job_name: 'wave_d_test_metrics'
    static_configs:
      - targets: ['host.docker.internal:8888']
    metrics_path: '/test_wave_d_metrics.prom'
    scrape_interval: 15s

Step 3: Reload Prometheus and Verify:

docker exec foxhunt-prometheus kill -HUP 1

# Wait 1 minute for metrics to populate
sleep 60

# Check if test alert fires (false positive rate >80%)
curl -s http://localhost:9090/api/v1/alerts | \
  jq '.data.alerts[] | select(.labels.alertname == "WaveDFalsePositives")'

# Expected: Alert in "pending" or "firing" state

Step 4: Cleanup Test Metrics:

# Remove test scrape job from prometheus.yml
# Kill test HTTP server
kill %1

# Reload Prometheus
docker exec foxhunt-prometheus kill -HUP 1

Production Alert Testing (Post-Deployment)

After Wave D production deployment, verify real metrics:

# Query Wave D metrics from production services
curl http://localhost:9091/metrics | grep wave_d
curl http://localhost:9092/metrics | grep regime

# Check alert evaluation results
curl -s http://localhost:9090/api/v1/rules | \
  jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules[] | {name: .name, state: .state, health: .health}'

# Expected: All alerts "inactive" if Wave D is healthy

---

Grafana Dashboard Integration

Add Wave D Alerts Panel to Grafana

Step 1: Access Grafana:

xdg-open http://localhost:3000
# Login: admin / foxhunt123

Step 2: Create Wave D Rollback Monitoring Dashboard:

Panel 1: Active Wave D Alerts:

ALERTS{component=~"wave_d.*", alertstate="firing"}

Panel 2: Regime Transitions per Hour:

rate(regime_transitions_total[1h]) * 3600

Panel 3: Regime Detection Error Rate:

sum(regime_detection_errors_total) / sum(regime_detections_total)

Panel 4: Wave D Feature Extraction Latency (P99):

histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m]))

Panel 5: Memory Growth Rate:

rate(process_resident_memory_bytes{job=~".*service"}[1h]) / process_resident_memory_bytes{job=~".*service"}

Panel 6: Data Quality Violations:

sum(wave_d_features_nan_count) + sum(wave_d_features_inf_count)

Grafana Alert Notification Channels

Configure Slack Notifications (Production):

# In Grafana UI:
# Alerting → Notification channels → New channel
# Type: Slack
# Webhook URL: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
# Channel: #production-alerts
# Test notification

Configure PagerDuty (Production):

# In Grafana UI:
# Alerting → Notification channels → New channel
# Type: PagerDuty
# Integration Key: <your-pagerduty-integration-key>
# Auto resolve alerts: true
# Test notification

---

Alertmanager Configuration (Optional)

For production deployments, configure Prometheus Alertmanager for advanced routing and grouping:

Step 1: Create Alertmanager Config

File: config/prometheus/alertmanager.yml

global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'severity', 'component']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Critical alerts → PagerDuty + Slack
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: true
    - match:
        severity: critical
      receiver: 'slack-critical'

    # Warning alerts → Slack only
    - match:
        severity: warning
      receiver: 'slack-warnings'

receivers:
  - name: 'default-receiver'
    slack_configs:
      - channel: '#production-alerts'
        title: 'Foxhunt Alert: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}'

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: '<your-pagerduty-integration-key>'
        description: '{{ .GroupLabels.alertname }}: {{ .Annotations.summary }}'
        severity: '{{ .Labels.severity }}'
        details:
          rollback_level: '{{ .Labels.rollback_level }}'
          runbook: '{{ .Annotations.runbook }}'

  - name: 'slack-critical'
    slack_configs:
      - channel: '#production-alerts'
        title: '🚨 CRITICAL: {{ .GroupLabels.alertname }}'
        text: |
          **Summary**: {{ .Annotations.summary }}
          **Rollback Level**: {{ .Labels.rollback_level }}
          **Runbook**: {{ .Annotations.runbook }}
        color: 'danger'

  - name: 'slack-warnings'
    slack_configs:
      - channel: '#wave-d-monitoring'
        title: '⚠️ WARNING: {{ .GroupLabels.alertname }}'
        text: '{{ .Annotations.summary }}'
        color: 'warning'

inhibit_rules:
  # Suppress warnings if critical alert is firing
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['component']

Step 2: Deploy Alertmanager

Update docker-compose.yml:

services:
  alertmanager:
    image: prom/alertmanager:latest
    container_name: foxhunt-alertmanager
    ports:
      - "9093:9093"
    volumes:
      - alertmanager_data:/alertmanager
      - ./config/prometheus/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
    networks:
      - foxhunt-network

volumes:
  alertmanager_data:

Update Prometheus Config:

# config/prometheus/prometheus.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

Restart Services:

docker-compose up -d alertmanager
docker-compose restart prometheus

---

Metrics Instrumentation Checklist

CRITICAL: Alerts will NOT fire if metrics are missing. Ensure all Wave D services expose these metrics:

Required Metrics

Metric Name	Type	Service	Description
regime_transitions_total	Counter	API Gateway, Trading Service	Total regime transitions
regime_detections_total	Counter	API Gateway, Trading Service	Total regime detections
regime_detection_errors_total	Counter	API Gateway, Trading Service	Regime detection errors
regime_states_count	Gauge	API Gateway, Trading Service	Current regime state counts
wave_d_features_nan_count	Gauge	ML Training Service	NaN values in features
wave_d_features_inf_count	Gauge	ML Training Service	Inf values in features
wave_d_feature_extraction_duration_seconds	Histogram	ML Training Service, Trading Service	Feature extraction latency
process_resident_memory_bytes	Gauge	All Services	RSS memory usage

Verification Commands

# Check if metrics are exposed
curl http://localhost:9091/metrics | grep -E "regime_|wave_d_"  # API Gateway
curl http://localhost:9092/metrics | grep -E "regime_|wave_d_"  # Trading Service
curl http://localhost:9094/metrics | grep -E "regime_|wave_d_"  # ML Training Service

# If metrics are missing, check Prometheus scrape targets
curl -s http://localhost:9090/api/v1/targets | \
  jq '.data.activeTargets[] | select(.labels.job | contains("service")) | {job: .labels.job, health: .health, lastError: .lastError}'

Add Missing Metrics (Example - Rust)

If Wave D metrics are missing, instrument services:

File: services/trading_service/src/metrics.rs

use prometheus::{Counter, Gauge, Histogram, register_counter, register_gauge, register_histogram};
use lazy_static::lazy_static;

lazy_static! {
    pub static ref REGIME_TRANSITIONS_TOTAL: Counter = register_counter!(
        "regime_transitions_total",
        "Total number of regime transitions detected"
    ).unwrap();

    pub static ref REGIME_DETECTIONS_TOTAL: Counter = register_counter!(
        "regime_detections_total",
        "Total number of regime detections performed"
    ).unwrap();

    pub static ref REGIME_DETECTION_ERRORS_TOTAL: Counter = register_counter!(
        "regime_detection_errors_total",
        "Total number of regime detection errors"
    ).unwrap();

    pub static ref WAVE_D_FEATURES_NAN_COUNT: Gauge = register_gauge!(
        "wave_d_features_nan_count",
        "Number of NaN values in Wave D features"
    ).unwrap();

    pub static ref WAVE_D_FEATURES_INF_COUNT: Gauge = register_gauge!(
        "wave_d_features_inf_count",
        "Number of Inf values in Wave D features"
    ).unwrap();

    pub static ref WAVE_D_FEATURE_EXTRACTION_DURATION: Histogram = register_histogram!(
        "wave_d_feature_extraction_duration_seconds",
        "Wave D feature extraction latency in seconds",
        vec![0.0001, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.05]
    ).unwrap();
}

Usage in Wave D Code:

// On regime transition
REGIME_TRANSITIONS_TOTAL.inc();

// On regime detection
REGIME_DETECTIONS_TOTAL.inc();
if detection_error {
    REGIME_DETECTION_ERRORS_TOTAL.inc();
}

// On feature extraction
let timer = WAVE_D_FEATURE_EXTRACTION_DURATION.start_timer();
let features = extract_wave_d_features()?;
timer.observe_duration();

// Check for NaN/Inf
let nan_count = features.iter().filter(|f| f.is_nan()).count();
let inf_count = features.iter().filter(|f| f.is_infinite()).count();
WAVE_D_FEATURES_NAN_COUNT.set(nan_count as f64);
WAVE_D_FEATURES_INF_COUNT.set(inf_count as f64);

---

Rollback Trigger Automation (Future Enhancement)

IMPORTANT: Current deployment requires MANUAL rollback execution. Future enhancement can automate rollback triggers.

Automated Rollback Script (Webhook Handler)

File: scripts/automated_rollback.sh

#!/bin/bash
# Automated rollback webhook handler (triggered by Alertmanager)
# WARNING: Use with extreme caution in production

ALERT_NAME="$1"
ROLLBACK_LEVEL="$2"

case "$ROLLBACK_LEVEL" in
  level_1)
    echo "Executing Level 1 rollback (feature-only, zero downtime)"
    /home/jgrusewski/Work/foxhunt/scripts/LEVEL_1_ROLLBACK.sh
    ;;
  level_2)
    echo "Executing Level 2 rollback (database rollback, ~5 min)"
    /home/jgrusewski/Work/foxhunt/scripts/LEVEL_2_ROLLBACK.sh
    ;;
  level_3)
    echo "CRITICAL: Level 3 rollback requested (full rollback, ~15 min)"
    echo "Sending emergency notification before rollback..."
    # Require human approval for Level 3
    exit 1
    ;;
  *)
    echo "Unknown rollback level: $ROLLBACK_LEVEL"
    exit 1
    ;;
esac

Alertmanager Webhook Configuration:

receivers:
  - name: 'automated-rollback'
    webhook_configs:
      - url: 'http://foxhunt-automation-server:5000/rollback'
        send_resolved: false
        http_config:
          bearer_token: '<secret-token>'

NOTE: Automated rollback is NOT RECOMMENDED for initial production deployment. Use manual rollback procedures until Wave D stability is proven.

---

Post-Deployment Validation

After deploying Wave D alerts, perform these validation steps:

Day 1: Alert Monitoring

• Open Grafana Wave D Rollback Monitoring dashboard
• Verify all 9 alerts are visible (inactive state)
• Check Prometheus /alerts page (no alerts firing)
• Review Slack #production-alerts channel (no false alarms)

Day 7: Alert Effectiveness Review

• Review alert history (how many alerts fired?)
• Analyze false positive rate (alerts that didn't require rollback)
• Adjust alert thresholds if needed (flip-flopping >50/hour too sensitive?)
• Document any alert tuning in WAVE_D_ALERTS_TUNING_LOG.md

Day 30: Alert Maturity Assessment

• Collect alert statistics (total fired, total resolved, avg duration)
• Evaluate rollback trigger accuracy (did alerts correctly predict issues?)
• Propose alert improvements (new metrics, adjusted thresholds, additional alerts)
• Update runbooks based on real incident response experience

---

Troubleshooting

Issue 1: Alerts Not Loading

Symptom: Prometheus /alerts page shows 0 Wave D alerts.

Diagnosis:

# Check if alert file exists in container
docker exec foxhunt-prometheus ls -la /etc/prometheus/rules/wave_d_alerts.yml

# Check Prometheus logs for errors
docker logs foxhunt-prometheus --tail 50 | grep -i error

Solution:

# Verify volume mount
docker inspect foxhunt-prometheus | jq '.[0].Mounts[] | select(.Destination == "/etc/prometheus/rules")'

# If mount is correct, reload Prometheus
docker exec foxhunt-prometheus kill -HUP 1

Issue 2: Alerts Stuck in "Pending" State

Symptom: Alerts show "pending" but never transition to "firing".

Diagnosis:

# Check alert evaluation interval
curl -s http://localhost:9090/api/v1/status/config | jq '.data.yaml' | grep evaluation_interval

# Check if metrics exist
curl -s http://localhost:9090/api/v1/query?query=regime_transitions_total

Solution:

# If metrics don't exist, alerts will never fire
# Ensure Wave D services are exposing metrics (see Metrics Instrumentation Checklist)

# If evaluation_interval is too high, reduce it
# config/prometheus/prometheus.yml: evaluation_interval: 15s

Issue 3: False Alarm Rate Too High

Symptom: Alerts firing too frequently, causing alert fatigue.

Diagnosis:

# Review alert history
curl -s http://localhost:9090/api/v1/query?query=ALERTS | \
  jq '.data.result[] | select(.metric.component | startswith("wave_d")) | {name: .metric.alertname, value: .value[1]}'

Solution:

# Adjust alert thresholds in wave_d_alerts.yml
# Example: Increase flip-flopping threshold from 50 to 100 transitions/hour
sed -i 's/rate(regime_transitions_total\[1h\]) > 50/rate(regime_transitions_total[1h]) > 100/' \
  config/prometheus/rules/wave_d_alerts.yml

# Reload Prometheus
docker exec foxhunt-prometheus kill -HUP 1

---

Success Criteria

Deployment is successful when:

• Syntax Validation: promtool reports "SUCCESS: 9 rules found"
• Prometheus Load: All 9 alerts visible in Prometheus /alerts UI
• Grafana Integration: Wave D dashboard shows alert panels with data
• Alert Evaluation: Alerts evaluate correctly (inactive when healthy, firing when threshold exceeded)
• Runbook Links: All critical alerts have accessible runbook URLs
• Notification Channels: Test alerts successfully sent to Slack/PagerDuty
• Metrics Availability: All required Wave D metrics exposed by services
• Zero False Alarms: No alerts firing during first 24 hours (assuming Wave D healthy)

---

Rollback (Alert Deployment Rollback)

If alert deployment causes issues (e.g., Prometheus crashes, alert spam):

Step 1: Disable Wave D Alerts:

# Rename alert file to disable
docker exec foxhunt-prometheus mv /etc/prometheus/rules/wave_d_alerts.yml /etc/prometheus/rules/wave_d_alerts.yml.disabled

# Reload Prometheus
docker exec foxhunt-prometheus kill -HUP 1

Step 2: Verify Alerts Removed:

curl -s http://localhost:9090/api/v1/rules | \
  jq '.data.groups[] | select(.name == "wave_d_rollback_triggers")'

# Expected: null (no results)

Step 3: Fix Issues and Re-deploy:

# Fix alert syntax or threshold issues
vim config/prometheus/rules/wave_d_alerts.yml

# Re-enable alerts
docker exec foxhunt-prometheus mv /etc/prometheus/rules/wave_d_alerts.yml.disabled /etc/prometheus/rules/wave_d_alerts.yml

# Reload Prometheus
docker exec foxhunt-prometheus kill -HUP 1

---

Next Steps

After successful Wave D alert deployment:

1. Production Deployment: Deploy Wave D features to production (see WAVE_D_DEPLOYMENT_GUIDE.md)
2. Monitoring Setup: Configure Grafana dashboards for real-time regime monitoring
3. Alertmanager Integration: Set up PagerDuty/Opsgenie for 24/7 on-call rotation
4. Runbook Testing: Validate all rollback procedures work as documented
5. Metrics Validation: Ensure all Wave D services expose required Prometheus metrics
6. Alert Tuning: Adjust thresholds based on real production data (first 7 days)
7. Automated Rollback: (Optional) Implement automated rollback webhook handler

---

References

• Alert Rules File: /home/jgrusewski/Work/foxhunt/config/prometheus/rules/wave_d_alerts.yml
• Rollback Procedures: /home/jgrusewski/Work/foxhunt/ROLLBACK_PROCEDURES.md
• Docker Compose: /home/jgrusewski/Work/foxhunt/docker-compose.yml
• Prometheus Config: /home/jgrusewski/Work/foxhunt/config/prometheus/prometheus.yml
• Wave D Documentation: /home/jgrusewski/Work/foxhunt/WAVE_D_DEPLOYMENT_GUIDE.md

---

END OF DEPLOYMENT GUIDE