foxhunt/WAVE_D_ALERTS_QUICK_REFERENCE.md

# Wave D Alerts - Quick Reference Card
**Last Updated**: 2025-10-19 by Agent M1
**For**: Operations Team, On-Call Engineers
**System**: Foxhunt HFT Trading System

---

## 🚨 CRITICAL ALERTS (Immediate Action Required)

### WaveDFlipFlopping
- **Trigger**: >50 regime transitions/hour for 5 minutes
- **Action**: Execute Level 1 rollback (<1 minute, zero downtime)
- **Command**: `sed -i 's/enable_wave_d_regime: true/enable_wave_d_regime: false/' ml/src/features/config.rs && cargo build --release`
- **Runbook**: `ROLLBACK_PROCEDURES.md#level-1-feature-only-rollback-zero-downtime`

### WaveDFalsePositives
- **Trigger**: >80% regime detection error rate for 10 minutes
- **Action**: Execute Level 1 rollback (<1 minute, zero downtime)
- **Command**: Same as WaveDFlipFlopping
- **Runbook**: `ROLLBACK_PROCEDURES.md#level-1-feature-only-rollback-zero-downtime`

### WaveDDataCorruption
- **Trigger**: NaN/Inf values in Wave D features for 1 minute
- **Action**: IMMEDIATE Level 3 rollback (~15 minutes, full outage)
- **Command**: `git checkout <wave-c-commit> && cargo build --release`
- **Runbook**: `ROLLBACK_PROCEDURES.md#level-3-full-rollback-to-wave-c`
- **⚠️ WARNING**: This will PERMANENTLY DELETE all Wave D data!

### FoxhuntSystemDown
- **Trigger**: System unavailable for >5 minutes
- **Action**: Investigate → Consider Level 3 rollback if Wave D suspected
- **Command**: Check `docker-compose logs`, then execute Level 3 rollback if needed
- **Runbook**: `ROLLBACK_PROCEDURES.md#level-3-full-rollback-to-wave-c`

### WaveDMemoryLeak
- **Trigger**: >20% RSS memory growth/hour for 1 hour
- **Action**: Monitor → Execute Level 1 rollback if memory continues growing
- **Command**: Same as WaveDFlipFlopping
- **Runbook**: `ROLLBACK_PROCEDURES.md#level-1-feature-only-rollback-zero-downtime`

---

## ⚠️ WARNING ALERTS (Monitor & Investigate)

### WaveDLatencyDegradation
- **Trigger**: >2ms P99 feature extraction latency for 15 minutes
- **Action**: Monitor → Execute Level 1 rollback if persists >15 minutes
- **Impact**: Slower trading decisions, potential missed opportunities

### WaveDRegimeCoverageHigh
- **Trigger**: >95% single regime for 30 minutes
- **Action**: Investigate → Regime detection may not be discriminating properly
- **Impact**: Reduced adaptive strategy benefits

### WaveDRegimeTransitionRateLow
- **Trigger**: <5 regime transitions/day for 2 hours
- **Action**: Investigate → Check CUSUM thresholds, market volatility
- **Impact**: Adaptive strategies may not activate

### WaveDDetectionErrorsModerate
- **Trigger**: 20-80% error rate for 30 minutes
- **Action**: Monitor → If error rate approaches 80%, prepare for Level 1 rollback
- **Impact**: Degraded regime classification accuracy

---

## 📊 QUICK DIAGNOSTIC COMMANDS

### Check Alert Status
```bash
# View all Wave D alerts
curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.component | startswith("wave_d")) | {name: .labels.alertname, state: .state}'

# Open Prometheus UI
xdg-open http://localhost:9090/alerts
```

### Check Wave D Metrics
```bash
# Regime transitions (flip-flopping)
curl -s "http://localhost:9090/api/v1/query?query=rate(regime_transitions_total[1h])*3600" | jq '.data.result[0].value[1]'

# Error rate (false positives)
curl -s "http://localhost:9090/api/v1/query?query=sum(regime_detection_errors_total)/sum(regime_detections_total)" | jq '.data.result[0].value[1]'

# Data quality (NaN/Inf)
curl -s "http://localhost:9090/api/v1/query?query=wave_d_features_nan_count+wave_d_features_inf_count" | jq '.data.result[0].value[1]'

# Latency (P99)
curl -s "http://localhost:9090/api/v1/query?query=histogram_quantile(0.99,rate(wave_d_feature_extraction_duration_seconds_bucket[5m]))" | jq '.data.result[0].value[1]'
```

### Check Service Health
```bash
# All services
curl http://localhost:8080/health  # API Gateway
curl http://localhost:8081/health  # Trading Service
curl http://localhost:8082/health  # Backtesting Service
curl http://localhost:8095/health  # ML Training Service

# Prometheus health
curl http://localhost:9090/-/healthy
```

### Check System Resources
```bash
# Memory usage
docker stats --no-stream --format "table {{.Container}}\t{{.MemUsage}}\t{{.MemPerc}}"

# CPU usage
docker stats --no-stream --format "table {{.Container}}\t{{.CPUPerc}}"

# Disk space
df -h /
```

---

## 🔄 ROLLBACK QUICK REFERENCE

### Level 1: Feature-Only Rollback (Zero Downtime, <1 min)

**When**: WaveDFlipFlopping, WaveDFalsePositives, WaveDLatencyDegradation, WaveDMemoryLeak

**Steps**:
1. Disable Wave D features:
   ```bash
   sed -i 's/enable_wave_d_regime: true/enable_wave_d_regime: false/' ml/src/features/config.rs
   ```

2. Rebuild:
   ```bash
   cargo build --workspace --release
   ```

3. Rolling restart (zero downtime):
   ```bash
   kill -TERM $(pgrep -f trading_service) && sleep 5 && cargo run --release -p trading_service &
   kill -TERM $(pgrep -f ml_training_service) && sleep 5 && cargo run --release -p ml_training_service &
   kill -TERM $(pgrep -f backtesting_service) && sleep 5 && cargo run --release -p backtesting_service &
   kill -TERM $(pgrep -f api_gateway) && sleep 5 && cargo run --release -p api_gateway &
   ```

4. Validate:
   ```bash
   cargo run --release -p ml --example check_feature_count  # Should show 201
   ```

**Data Loss**: NONE (Wave D data preserved)

### Level 2: Database Rollback (~5 min, Planned Downtime)

**When**: Database migration failures, data integrity issues (non-critical)

**Steps**:
1. Backup:
   ```bash
   PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt > /tmp/backup_$(date +%s).sql
   ```

2. Stop services:
   ```bash
   kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")
   ```

3. Rollback migration:
   ```bash
   sqlx migrate revert
   ```

4. Disable Wave D (same as Level 1 Step 1)

5. Rebuild and restart (same as Level 1 Steps 2-3)

**Data Loss**: Wave D data DELETED (regime_states, regime_transitions, adaptive_strategy_metrics)

### Level 3: Full Rollback (~15 min, Full Outage)

**When**: WaveDDataCorruption, FoxhuntSystemDown (if Wave D suspected)

**Steps**:
1. Tag and backup:
   ```bash
   git tag "wave-d-emergency-rollback-$(date +%Y%m%d-%H%M%S)"
   PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt > /tmp/full_backup_$(date +%s).sql
   ```

2. Stop services:
   ```bash
   kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")
   ```

3. Rollback database (same as Level 2 Step 3)

4. Checkout Wave C:
   ```bash
   WAVE_C_COMMIT=$(git log --all --oneline --before="2025-10-17" | head -1 | awk '{print $1}')
   git stash push -m "Emergency rollback"
   git checkout "$WAVE_C_COMMIT"
   ```

5. Clean rebuild:
   ```bash
   cargo clean && cargo build --workspace --release
   ```

6. Manual restart (see Level 1 Step 3)

**Data Loss**: Complete Wave D removal (code + data)

---

## 📞 ESCALATION

### Severity Levels

| Severity | Response Time | Escalation Path |
|----------|--------------|-----------------|
| **WARNING** | 30 minutes | Primary On-Call only |
| **CRITICAL** | 15 minutes | Primary + Secondary On-Call |
| **CATASTROPHIC** | 5 minutes | Entire team + CTO |

### Contact Information

**Primary On-Call**: See ROLLBACK_PROCEDURES.md - Emergency Contacts section

**Slack Channels**:
- `#production-alerts` - Automated alerts
- `#incident-response` - Active incident coordination

**PagerDuty**: https://foxhunt.pagerduty.com (if configured)

---

## 🔍 TROUBLESHOOTING

### Alert Not Firing (Expected to Fire)

1. Check if metrics exist:
   ```bash
   curl http://localhost:9091/metrics | grep regime_transitions_total
   ```

2. Check if alert is loaded:
   ```bash
   curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules | length'
   # Expected: 9
   ```

3. Check alert evaluation:
   ```bash
   curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules[] | select(.name == "WaveDFlipFlopping")'
   ```

### False Alarm (Alert Firing When Shouldn't)

1. Check current metric value:
   ```bash
   curl -s "http://localhost:9090/api/v1/query?query=rate(regime_transitions_total[1h])*3600"
   ```

2. Adjust threshold if needed:
   ```bash
   vim config/prometheus/rules/wave_d_alerts.yml
   docker exec foxhunt-prometheus kill -HUP 1  # Reload
   ```

3. Document tuning decision in `WAVE_D_ALERTS_TUNING_LOG.md`

### Prometheus Unhealthy

1. Check logs:
   ```bash
   docker logs foxhunt-prometheus --tail 50
   ```

2. Check configuration:
   ```bash
   docker exec foxhunt-prometheus promtool check config /etc/prometheus/prometheus.yml
   ```

3. Restart if needed:
   ```bash
   docker-compose restart prometheus
   ```

---

## 📝 POST-INCIDENT CHECKLIST

After executing a rollback:

- [ ] Verify system stability (all services healthy for 1 hour)
- [ ] Notify stakeholders (Slack #production-alerts, email trading-team@foxhunt.ai)
- [ ] Document incident (`incidents/YYYY-MM-DD-wave-d-rollback.md`)
- [ ] Schedule root cause analysis (within 24 hours)
- [ ] Update runbooks based on lessons learned
- [ ] Plan re-deployment (after fix validated in staging)

---

## 📚 REFERENCES

- **Full Documentation**: `WAVE_D_ALERTS_DEPLOYMENT_GUIDE.md`
- **Rollback Procedures**: `ROLLBACK_PROCEDURES.md`
- **Completion Report**: `AGENT_M1_COMPLETION_REPORT.md`
- **Test Script**: `scripts/test_wave_d_alerts.sh`

---

**Keep this card accessible during on-call shifts!**

**Last Updated**: 2025-10-19