foxhunt/WAVE_D_ALERTS_DEPLOYMENT_GUIDE.md

# Wave D Prometheus Alerts - Deployment Guide
**Agent**: M1 - Prometheus Alert Deployment
**Date**: 2025-10-19
**System**: Foxhunt HFT Trading System
**Version**: Wave D (225 features)

---

## Executive Summary

This guide provides step-by-step instructions for deploying **9 Wave D Prometheus alert rules** to monitor regime detection features and trigger automated rollback procedures when critical thresholds are exceeded.

**Alert Categories**:
- **5 Critical Alerts**: Immediate rollback triggers (flip-flopping, false positives, data corruption, system down, latency)
- **4 Warning Alerts**: Monitoring and early detection (memory leaks, regime coverage, transition rate, moderate errors)

**Deployment Time**: <5 minutes
**Validation Method**: promtool syntax check + Prometheus UI verification
**Impact**: Zero downtime (alerts load dynamically via Prometheus hot-reload)

---

## Alert Rules Summary

| Alert Name | Severity | Rollback Level | Trigger Condition | For Duration |
|------------|----------|----------------|-------------------|--------------|
| **WaveDFlipFlopping** | Critical | Level 1 | >50 transitions/hour | 5 minutes |
| **WaveDFalsePositives** | Critical | Level 1 | >80% error rate | 10 minutes |
| **WaveDDataCorruption** | Critical | Level 3 | NaN/Inf in features | 1 minute |
| **FoxhuntSystemDown** | Critical | Level 3 | System unavailable | 5 minutes |
| **WaveDLatencyDegradation** | Warning | Level 1 | >2ms P99 latency | 15 minutes |
| **WaveDMemoryLeak** | Warning | Level 1 | >20% RSS growth/hour | 1 hour |
| **WaveDRegimeCoverageHigh** | Warning | None | >95% single regime | 30 minutes |
| **WaveDRegimeTransitionRateLow** | Warning | None | <5 transitions/day | 2 hours |
| **WaveDDetectionErrorsModerate** | Warning | None | 20-80% error rate | 30 minutes |

---

## Deployment Procedure

### Prerequisites

1. **Docker Compose Running**:
   ```bash
   docker-compose ps | grep prometheus
   # Expected: foxhunt-prometheus running (healthy)
   ```

2. **Prometheus Configuration Verified**:
   ```bash
   ls -la config/prometheus/rules/wave_d_alerts.yml
   # Expected: File exists (created by Agent M1)
   ```

3. **Prometheus Volume Mount Verified**:
   ```bash
   docker inspect foxhunt-prometheus | jq '.[0].Mounts[] | select(.Destination == "/etc/prometheus/rules")'
   # Expected: Source = ./config/prometheus/rules (read-only)
   ```

### Step 1: Validate Alert Syntax (30 seconds)

**Using Docker Prometheus**:
```bash
cd /home/jgrusewski/Work/foxhunt

# Validate alert rules syntax
docker exec foxhunt-prometheus promtool check rules /etc/prometheus/rules/wave_d_alerts.yml

# Expected output:
# Checking /etc/prometheus/rules/wave_d_alerts.yml
#   SUCCESS: 9 rules found
```

**If promtool reports errors**:
- Fix syntax errors in `config/prometheus/rules/wave_d_alerts.yml`
- Re-run validation
- Do NOT proceed until syntax is valid

### Step 2: Hot-Reload Prometheus Configuration (15 seconds)

**Option A: Prometheus Hot-Reload (Zero Downtime - PREFERRED)**:
```bash
# Send SIGHUP to Prometheus (reload config without restart)
docker exec foxhunt-prometheus kill -HUP 1

# Verify reload successful (check logs)
docker logs foxhunt-prometheus --tail 20

# Expected log message:
# "Completed loading of configuration file"
```

**Option B: Full Prometheus Restart (5-10 seconds downtime)**:
```bash
# Only use if hot-reload fails
docker-compose restart prometheus

# Wait for health check
sleep 10
curl http://localhost:9090/-/healthy
# Expected: Prometheus is Healthy.
```

### Step 3: Verify Alert Rules Loaded (30 seconds)

**Method 1: Prometheus Web UI**:
```bash
# Open browser to Prometheus alerts page
xdg-open http://localhost:9090/alerts

# OR use curl to list alerts
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules[].name'

# Expected output (9 alert names):
# WaveDFlipFlopping
# WaveDFalsePositives
# WaveDDataCorruption
# FoxhuntSystemDown
# WaveDLatencyDegradation
# WaveDMemoryLeak
# WaveDRegimeCoverageHigh
# WaveDRegimeTransitionRateLow
# WaveDDetectionErrorsModerate
```

**Method 2: Prometheus API Query**:
```bash
# Count loaded Wave D alerts
curl -s http://localhost:9090/api/v1/rules | \
  jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules | length'

# Expected: 9
```

**Method 3: Check Alert Status**:
```bash
# List all Wave D alerts with current state
curl -s http://localhost:9090/api/v1/alerts | \
  jq '.data.alerts[] | select(.labels.component | startswith("wave_d")) | {name: .labels.alertname, state: .state}'

# Expected (all alerts in "inactive" state before Wave D deployment):
# {"name":"WaveDFlipFlopping","state":"inactive"}
# {"name":"WaveDFalsePositives","state":"inactive"}
# ... (7 more)
```

### Step 4: Validate Alert Annotations (15 seconds)

**Check that all critical alerts have runbook URLs**:
```bash
curl -s http://localhost:9090/api/v1/rules | \
  jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules[] | select(.labels.severity == "critical") | {name: .name, runbook: .annotations.runbook}'

# Expected: All 4 critical alerts have runbook URLs
```

**Verify rollback_level labels**:
```bash
curl -s http://localhost:9090/api/v1/rules | \
  jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules[] | {name: .name, rollback_level: .labels.rollback_level}'

# Expected:
# WaveDFlipFlopping → level_1
# WaveDFalsePositives → level_1
# WaveDDataCorruption → level_3
# FoxhuntSystemDown → level_3
# WaveDLatencyDegradation → level_1
# WaveDMemoryLeak → level_1
# (remaining alerts → none)
```

---

## Testing Alert Rules

### Synthetic Test Data (Pre-Deployment)

Before Wave D production deployment, test alerts using synthetic metrics:

**Step 1: Expose Test Metrics Endpoint**:
```bash
# Create test metrics endpoint (development only)
cat > /tmp/test_wave_d_metrics.prom <<'EOF'
# HELP regime_transitions_total Total regime transitions
# TYPE regime_transitions_total counter
regime_transitions_total 100

# HELP regime_detections_total Total regime detections
# TYPE regime_detections_total counter
regime_detections_total 1000

# HELP regime_detection_errors_total Regime detection errors
# TYPE regime_detection_errors_total counter
regime_detection_errors_total 850

# HELP wave_d_features_nan_count NaN values in Wave D features
# TYPE wave_d_features_nan_count gauge
wave_d_features_nan_count 0

# HELP wave_d_features_inf_count Inf values in Wave D features
# TYPE wave_d_features_inf_count gauge
wave_d_features_inf_count 0

# HELP wave_d_feature_extraction_duration_seconds Feature extraction latency
# TYPE wave_d_feature_extraction_duration_seconds histogram
wave_d_feature_extraction_duration_seconds_bucket{le="0.001"} 100
wave_d_feature_extraction_duration_seconds_bucket{le="0.002"} 200
wave_d_feature_extraction_duration_seconds_bucket{le="0.005"} 250
wave_d_feature_extraction_duration_seconds_bucket{le="+Inf"} 300
wave_d_feature_extraction_duration_seconds_sum 0.6
wave_d_feature_extraction_duration_seconds_count 300
EOF

# Serve test metrics (1-hour HTTP server)
cd /tmp
python3 -m http.server 8888 &
# Metrics available at: http://localhost:8888/test_wave_d_metrics.prom
```

**Step 2: Configure Prometheus Scrape Job**:
```yaml
# Add to config/prometheus/prometheus.yml (temporary, for testing only)
scrape_configs:
  - job_name: 'wave_d_test_metrics'
    static_configs:
      - targets: ['host.docker.internal:8888']
    metrics_path: '/test_wave_d_metrics.prom'
    scrape_interval: 15s
```

**Step 3: Reload Prometheus and Verify**:
```bash
docker exec foxhunt-prometheus kill -HUP 1

# Wait 1 minute for metrics to populate
sleep 60

# Check if test alert fires (false positive rate >80%)
curl -s http://localhost:9090/api/v1/alerts | \
  jq '.data.alerts[] | select(.labels.alertname == "WaveDFalsePositives")'

# Expected: Alert in "pending" or "firing" state
```

**Step 4: Cleanup Test Metrics**:
```bash
# Remove test scrape job from prometheus.yml
# Kill test HTTP server
kill %1

# Reload Prometheus
docker exec foxhunt-prometheus kill -HUP 1
```

### Production Alert Testing (Post-Deployment)

**After Wave D production deployment, verify real metrics**:

```bash
# Query Wave D metrics from production services
curl http://localhost:9091/metrics | grep wave_d
curl http://localhost:9092/metrics | grep regime

# Check alert evaluation results
curl -s http://localhost:9090/api/v1/rules | \
  jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules[] | {name: .name, state: .state, health: .health}'

# Expected: All alerts "inactive" if Wave D is healthy
```

---

## Grafana Dashboard Integration

### Add Wave D Alerts Panel to Grafana

**Step 1: Access Grafana**:
```bash
xdg-open http://localhost:3000
# Login: admin / foxhunt123
```

**Step 2: Create Wave D Rollback Monitoring Dashboard**:

**Panel 1: Active Wave D Alerts**:
```promql
ALERTS{component=~"wave_d.*", alertstate="firing"}
```

**Panel 2: Regime Transitions per Hour**:
```promql
rate(regime_transitions_total[1h]) * 3600
```

**Panel 3: Regime Detection Error Rate**:
```promql
sum(regime_detection_errors_total) / sum(regime_detections_total)
```

**Panel 4: Wave D Feature Extraction Latency (P99)**:
```promql
histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m]))
```

**Panel 5: Memory Growth Rate**:
```promql
rate(process_resident_memory_bytes{job=~".*service"}[1h]) / process_resident_memory_bytes{job=~".*service"}
```

**Panel 6: Data Quality Violations**:
```promql
sum(wave_d_features_nan_count) + sum(wave_d_features_inf_count)
```

### Grafana Alert Notification Channels

**Configure Slack Notifications** (Production):
```bash
# In Grafana UI:
# Alerting → Notification channels → New channel
# Type: Slack
# Webhook URL: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
# Channel: #production-alerts
# Test notification
```

**Configure PagerDuty** (Production):
```bash
# In Grafana UI:
# Alerting → Notification channels → New channel
# Type: PagerDuty
# Integration Key: <your-pagerduty-integration-key>
# Auto resolve alerts: true
# Test notification
```

---

## Alertmanager Configuration (Optional)

For production deployments, configure Prometheus Alertmanager for advanced routing and grouping:

### Step 1: Create Alertmanager Config

**File**: `config/prometheus/alertmanager.yml`

```yaml
global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'severity', 'component']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Critical alerts → PagerDuty + Slack
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: true
    - match:
        severity: critical
      receiver: 'slack-critical'

    # Warning alerts → Slack only
    - match:
        severity: warning
      receiver: 'slack-warnings'

receivers:
  - name: 'default-receiver'
    slack_configs:
      - channel: '#production-alerts'
        title: 'Foxhunt Alert: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}'

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: '<your-pagerduty-integration-key>'
        description: '{{ .GroupLabels.alertname }}: {{ .Annotations.summary }}'
        severity: '{{ .Labels.severity }}'
        details:
          rollback_level: '{{ .Labels.rollback_level }}'
          runbook: '{{ .Annotations.runbook }}'

  - name: 'slack-critical'
    slack_configs:
      - channel: '#production-alerts'
        title: '🚨 CRITICAL: {{ .GroupLabels.alertname }}'
        text: |
          **Summary**: {{ .Annotations.summary }}
          **Rollback Level**: {{ .Labels.rollback_level }}
          **Runbook**: {{ .Annotations.runbook }}
        color: 'danger'

  - name: 'slack-warnings'
    slack_configs:
      - channel: '#wave-d-monitoring'
        title: '⚠️ WARNING: {{ .GroupLabels.alertname }}'
        text: '{{ .Annotations.summary }}'
        color: 'warning'

inhibit_rules:
  # Suppress warnings if critical alert is firing
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['component']
```

### Step 2: Deploy Alertmanager

**Update docker-compose.yml**:
```yaml
services:
  alertmanager:
    image: prom/alertmanager:latest
    container_name: foxhunt-alertmanager
    ports:
      - "9093:9093"
    volumes:
      - alertmanager_data:/alertmanager
      - ./config/prometheus/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
    networks:
      - foxhunt-network

volumes:
  alertmanager_data:
```

**Update Prometheus Config**:
```yaml
# config/prometheus/prometheus.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']
```

**Restart Services**:
```bash
docker-compose up -d alertmanager
docker-compose restart prometheus
```

---

## Metrics Instrumentation Checklist

**CRITICAL**: Alerts will NOT fire if metrics are missing. Ensure all Wave D services expose these metrics:

### Required Metrics

| Metric Name | Type | Service | Description |
|-------------|------|---------|-------------|
| `regime_transitions_total` | Counter | API Gateway, Trading Service | Total regime transitions |
| `regime_detections_total` | Counter | API Gateway, Trading Service | Total regime detections |
| `regime_detection_errors_total` | Counter | API Gateway, Trading Service | Regime detection errors |
| `regime_states_count` | Gauge | API Gateway, Trading Service | Current regime state counts |
| `wave_d_features_nan_count` | Gauge | ML Training Service | NaN values in features |
| `wave_d_features_inf_count` | Gauge | ML Training Service | Inf values in features |
| `wave_d_feature_extraction_duration_seconds` | Histogram | ML Training Service, Trading Service | Feature extraction latency |
| `process_resident_memory_bytes` | Gauge | All Services | RSS memory usage |

### Verification Commands

```bash
# Check if metrics are exposed
curl http://localhost:9091/metrics | grep -E "regime_|wave_d_"  # API Gateway
curl http://localhost:9092/metrics | grep -E "regime_|wave_d_"  # Trading Service
curl http://localhost:9094/metrics | grep -E "regime_|wave_d_"  # ML Training Service

# If metrics are missing, check Prometheus scrape targets
curl -s http://localhost:9090/api/v1/targets | \
  jq '.data.activeTargets[] | select(.labels.job | contains("service")) | {job: .labels.job, health: .health, lastError: .lastError}'
```

### Add Missing Metrics (Example - Rust)

If Wave D metrics are missing, instrument services:

**File**: `services/trading_service/src/metrics.rs`

```rust
use prometheus::{Counter, Gauge, Histogram, register_counter, register_gauge, register_histogram};
use lazy_static::lazy_static;

lazy_static! {
    pub static ref REGIME_TRANSITIONS_TOTAL: Counter = register_counter!(
        "regime_transitions_total",
        "Total number of regime transitions detected"
    ).unwrap();

    pub static ref REGIME_DETECTIONS_TOTAL: Counter = register_counter!(
        "regime_detections_total",
        "Total number of regime detections performed"
    ).unwrap();

    pub static ref REGIME_DETECTION_ERRORS_TOTAL: Counter = register_counter!(
        "regime_detection_errors_total",
        "Total number of regime detection errors"
    ).unwrap();

    pub static ref WAVE_D_FEATURES_NAN_COUNT: Gauge = register_gauge!(
        "wave_d_features_nan_count",
        "Number of NaN values in Wave D features"
    ).unwrap();

    pub static ref WAVE_D_FEATURES_INF_COUNT: Gauge = register_gauge!(
        "wave_d_features_inf_count",
        "Number of Inf values in Wave D features"
    ).unwrap();

    pub static ref WAVE_D_FEATURE_EXTRACTION_DURATION: Histogram = register_histogram!(
        "wave_d_feature_extraction_duration_seconds",
        "Wave D feature extraction latency in seconds",
        vec![0.0001, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.05]
    ).unwrap();
}
```

**Usage in Wave D Code**:
```rust
// On regime transition
REGIME_TRANSITIONS_TOTAL.inc();

// On regime detection
REGIME_DETECTIONS_TOTAL.inc();
if detection_error {
    REGIME_DETECTION_ERRORS_TOTAL.inc();
}

// On feature extraction
let timer = WAVE_D_FEATURE_EXTRACTION_DURATION.start_timer();
let features = extract_wave_d_features()?;
timer.observe_duration();

// Check for NaN/Inf
let nan_count = features.iter().filter(|f| f.is_nan()).count();
let inf_count = features.iter().filter(|f| f.is_infinite()).count();
WAVE_D_FEATURES_NAN_COUNT.set(nan_count as f64);
WAVE_D_FEATURES_INF_COUNT.set(inf_count as f64);
```

---

## Rollback Trigger Automation (Future Enhancement)

**IMPORTANT**: Current deployment requires **MANUAL** rollback execution. Future enhancement can automate rollback triggers.

### Automated Rollback Script (Webhook Handler)

**File**: `scripts/automated_rollback.sh`

```bash
#!/bin/bash
# Automated rollback webhook handler (triggered by Alertmanager)
# WARNING: Use with extreme caution in production

ALERT_NAME="$1"
ROLLBACK_LEVEL="$2"

case "$ROLLBACK_LEVEL" in
  level_1)
    echo "Executing Level 1 rollback (feature-only, zero downtime)"
    /home/jgrusewski/Work/foxhunt/scripts/LEVEL_1_ROLLBACK.sh
    ;;
  level_2)
    echo "Executing Level 2 rollback (database rollback, ~5 min)"
    /home/jgrusewski/Work/foxhunt/scripts/LEVEL_2_ROLLBACK.sh
    ;;
  level_3)
    echo "CRITICAL: Level 3 rollback requested (full rollback, ~15 min)"
    echo "Sending emergency notification before rollback..."
    # Require human approval for Level 3
    exit 1
    ;;
  *)
    echo "Unknown rollback level: $ROLLBACK_LEVEL"
    exit 1
    ;;
esac
```

**Alertmanager Webhook Configuration**:
```yaml
receivers:
  - name: 'automated-rollback'
    webhook_configs:
      - url: 'http://foxhunt-automation-server:5000/rollback'
        send_resolved: false
        http_config:
          bearer_token: '<secret-token>'
```

**NOTE**: Automated rollback is **NOT RECOMMENDED** for initial production deployment. Use manual rollback procedures until Wave D stability is proven.

---

## Post-Deployment Validation

After deploying Wave D alerts, perform these validation steps:

### Day 1: Alert Monitoring

- [ ] Open Grafana Wave D Rollback Monitoring dashboard
- [ ] Verify all 9 alerts are visible (inactive state)
- [ ] Check Prometheus /alerts page (no alerts firing)
- [ ] Review Slack #production-alerts channel (no false alarms)

### Day 7: Alert Effectiveness Review

- [ ] Review alert history (how many alerts fired?)
- [ ] Analyze false positive rate (alerts that didn't require rollback)
- [ ] Adjust alert thresholds if needed (flip-flopping >50/hour too sensitive?)
- [ ] Document any alert tuning in WAVE_D_ALERTS_TUNING_LOG.md

### Day 30: Alert Maturity Assessment

- [ ] Collect alert statistics (total fired, total resolved, avg duration)
- [ ] Evaluate rollback trigger accuracy (did alerts correctly predict issues?)
- [ ] Propose alert improvements (new metrics, adjusted thresholds, additional alerts)
- [ ] Update runbooks based on real incident response experience

---

## Troubleshooting

### Issue 1: Alerts Not Loading

**Symptom**: Prometheus /alerts page shows 0 Wave D alerts.

**Diagnosis**:
```bash
# Check if alert file exists in container
docker exec foxhunt-prometheus ls -la /etc/prometheus/rules/wave_d_alerts.yml

# Check Prometheus logs for errors
docker logs foxhunt-prometheus --tail 50 | grep -i error
```

**Solution**:
```bash
# Verify volume mount
docker inspect foxhunt-prometheus | jq '.[0].Mounts[] | select(.Destination == "/etc/prometheus/rules")'

# If mount is correct, reload Prometheus
docker exec foxhunt-prometheus kill -HUP 1
```

### Issue 2: Alerts Stuck in "Pending" State

**Symptom**: Alerts show "pending" but never transition to "firing".

**Diagnosis**:
```bash
# Check alert evaluation interval
curl -s http://localhost:9090/api/v1/status/config | jq '.data.yaml' | grep evaluation_interval

# Check if metrics exist
curl -s http://localhost:9090/api/v1/query?query=regime_transitions_total
```

**Solution**:
```bash
# If metrics don't exist, alerts will never fire
# Ensure Wave D services are exposing metrics (see Metrics Instrumentation Checklist)

# If evaluation_interval is too high, reduce it
# config/prometheus/prometheus.yml: evaluation_interval: 15s
```

### Issue 3: False Alarm Rate Too High

**Symptom**: Alerts firing too frequently, causing alert fatigue.

**Diagnosis**:
```bash
# Review alert history
curl -s http://localhost:9090/api/v1/query?query=ALERTS | \
  jq '.data.result[] | select(.metric.component | startswith("wave_d")) | {name: .metric.alertname, value: .value[1]}'
```

**Solution**:
```bash
# Adjust alert thresholds in wave_d_alerts.yml
# Example: Increase flip-flopping threshold from 50 to 100 transitions/hour
sed -i 's/rate(regime_transitions_total\[1h\]) > 50/rate(regime_transitions_total[1h]) > 100/' \
  config/prometheus/rules/wave_d_alerts.yml

# Reload Prometheus
docker exec foxhunt-prometheus kill -HUP 1
```

---

## Success Criteria

Deployment is successful when:

- [ ] **Syntax Validation**: promtool reports "SUCCESS: 9 rules found"
- [ ] **Prometheus Load**: All 9 alerts visible in Prometheus /alerts UI
- [ ] **Grafana Integration**: Wave D dashboard shows alert panels with data
- [ ] **Alert Evaluation**: Alerts evaluate correctly (inactive when healthy, firing when threshold exceeded)
- [ ] **Runbook Links**: All critical alerts have accessible runbook URLs
- [ ] **Notification Channels**: Test alerts successfully sent to Slack/PagerDuty
- [ ] **Metrics Availability**: All required Wave D metrics exposed by services
- [ ] **Zero False Alarms**: No alerts firing during first 24 hours (assuming Wave D healthy)

---

## Rollback (Alert Deployment Rollback)

If alert deployment causes issues (e.g., Prometheus crashes, alert spam):

**Step 1: Disable Wave D Alerts**:
```bash
# Rename alert file to disable
docker exec foxhunt-prometheus mv /etc/prometheus/rules/wave_d_alerts.yml /etc/prometheus/rules/wave_d_alerts.yml.disabled

# Reload Prometheus
docker exec foxhunt-prometheus kill -HUP 1
```

**Step 2: Verify Alerts Removed**:
```bash
curl -s http://localhost:9090/api/v1/rules | \
  jq '.data.groups[] | select(.name == "wave_d_rollback_triggers")'

# Expected: null (no results)
```

**Step 3: Fix Issues and Re-deploy**:
```bash
# Fix alert syntax or threshold issues
vim config/prometheus/rules/wave_d_alerts.yml

# Re-enable alerts
docker exec foxhunt-prometheus mv /etc/prometheus/rules/wave_d_alerts.yml.disabled /etc/prometheus/rules/wave_d_alerts.yml

# Reload Prometheus
docker exec foxhunt-prometheus kill -HUP 1
```

---

## Next Steps

After successful Wave D alert deployment:

1. **Production Deployment**: Deploy Wave D features to production (see WAVE_D_DEPLOYMENT_GUIDE.md)
2. **Monitoring Setup**: Configure Grafana dashboards for real-time regime monitoring
3. **Alertmanager Integration**: Set up PagerDuty/Opsgenie for 24/7 on-call rotation
4. **Runbook Testing**: Validate all rollback procedures work as documented
5. **Metrics Validation**: Ensure all Wave D services expose required Prometheus metrics
6. **Alert Tuning**: Adjust thresholds based on real production data (first 7 days)
7. **Automated Rollback**: (Optional) Implement automated rollback webhook handler

---

## References

- **Alert Rules File**: `/home/jgrusewski/Work/foxhunt/config/prometheus/rules/wave_d_alerts.yml`
- **Rollback Procedures**: `/home/jgrusewski/Work/foxhunt/ROLLBACK_PROCEDURES.md`
- **Docker Compose**: `/home/jgrusewski/Work/foxhunt/docker-compose.yml`
- **Prometheus Config**: `/home/jgrusewski/Work/foxhunt/config/prometheus/prometheus.yml`
- **Wave D Documentation**: `/home/jgrusewski/Work/foxhunt/WAVE_D_DEPLOYMENT_GUIDE.md`

---

**END OF DEPLOYMENT GUIDE**