foxhunt/WAVE_D_ALERTS_QUICK_REFERENCE.md

Wave D Alerts - Quick Reference Card

Last Updated: 2025-10-19 by Agent M1 For: Operations Team, On-Call Engineers System: Foxhunt HFT Trading System

---

🚨 CRITICAL ALERTS (Immediate Action Required)

WaveDFlipFlopping

• Trigger: >50 regime transitions/hour for 5 minutes
• Action: Execute Level 1 rollback (<1 minute, zero downtime)
• Command: sed -i 's/enable_wave_d_regime: true/enable_wave_d_regime: false/' ml/src/features/config.rs && cargo build --release
• Runbook: ROLLBACK_PROCEDURES.md#level-1-feature-only-rollback-zero-downtime

WaveDFalsePositives

• Trigger: >80% regime detection error rate for 10 minutes
• Action: Execute Level 1 rollback (<1 minute, zero downtime)
• Command: Same as WaveDFlipFlopping
• Runbook: ROLLBACK_PROCEDURES.md#level-1-feature-only-rollback-zero-downtime

WaveDDataCorruption

• Trigger: NaN/Inf values in Wave D features for 1 minute
• Action: IMMEDIATE Level 3 rollback (~15 minutes, full outage)
• Command: git checkout <wave-c-commit> && cargo build --release
• Runbook: ROLLBACK_PROCEDURES.md#level-3-full-rollback-to-wave-c
• ⚠️ WARNING: This will PERMANENTLY DELETE all Wave D data!

FoxhuntSystemDown

• Trigger: System unavailable for >5 minutes
• Action: Investigate → Consider Level 3 rollback if Wave D suspected
• Command: Check docker-compose logs, then execute Level 3 rollback if needed
• Runbook: ROLLBACK_PROCEDURES.md#level-3-full-rollback-to-wave-c

WaveDMemoryLeak

• Trigger: >20% RSS memory growth/hour for 1 hour
• Action: Monitor → Execute Level 1 rollback if memory continues growing
• Command: Same as WaveDFlipFlopping
• Runbook: ROLLBACK_PROCEDURES.md#level-1-feature-only-rollback-zero-downtime

---

⚠️ WARNING ALERTS (Monitor & Investigate)

WaveDLatencyDegradation

• Trigger: >2ms P99 feature extraction latency for 15 minutes
• Action: Monitor → Execute Level 1 rollback if persists >15 minutes
• Impact: Slower trading decisions, potential missed opportunities

WaveDRegimeCoverageHigh

• Trigger: >95% single regime for 30 minutes
• Action: Investigate → Regime detection may not be discriminating properly
• Impact: Reduced adaptive strategy benefits

WaveDRegimeTransitionRateLow

• Trigger: <5 regime transitions/day for 2 hours
• Action: Investigate → Check CUSUM thresholds, market volatility
• Impact: Adaptive strategies may not activate

WaveDDetectionErrorsModerate

• Trigger: 20-80% error rate for 30 minutes
• Action: Monitor → If error rate approaches 80%, prepare for Level 1 rollback
• Impact: Degraded regime classification accuracy

---

📊 QUICK DIAGNOSTIC COMMANDS

Check Alert Status

# View all Wave D alerts
curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.component | startswith("wave_d")) | {name: .labels.alertname, state: .state}'

# Open Prometheus UI
xdg-open http://localhost:9090/alerts

Check Wave D Metrics

# Regime transitions (flip-flopping)
curl -s "http://localhost:9090/api/v1/query?query=rate(regime_transitions_total[1h])*3600" | jq '.data.result[0].value[1]'

# Error rate (false positives)
curl -s "http://localhost:9090/api/v1/query?query=sum(regime_detection_errors_total)/sum(regime_detections_total)" | jq '.data.result[0].value[1]'

# Data quality (NaN/Inf)
curl -s "http://localhost:9090/api/v1/query?query=wave_d_features_nan_count+wave_d_features_inf_count" | jq '.data.result[0].value[1]'

# Latency (P99)
curl -s "http://localhost:9090/api/v1/query?query=histogram_quantile(0.99,rate(wave_d_feature_extraction_duration_seconds_bucket[5m]))" | jq '.data.result[0].value[1]'

Check Service Health

# All services
curl http://localhost:8080/health  # API Gateway
curl http://localhost:8081/health  # Trading Service
curl http://localhost:8082/health  # Backtesting Service
curl http://localhost:8095/health  # ML Training Service

# Prometheus health
curl http://localhost:9090/-/healthy

Check System Resources

# Memory usage
docker stats --no-stream --format "table {{.Container}}\t{{.MemUsage}}\t{{.MemPerc}}"

# CPU usage
docker stats --no-stream --format "table {{.Container}}\t{{.CPUPerc}}"

# Disk space
df -h /

---

🔄 ROLLBACK QUICK REFERENCE

Level 1: Feature-Only Rollback (Zero Downtime, <1 min)

When: WaveDFlipFlopping, WaveDFalsePositives, WaveDLatencyDegradation, WaveDMemoryLeak

Steps:

1. Disable Wave D features:

   sed -i 's/enable_wave_d_regime: true/enable_wave_d_regime: false/' ml/src/features/config.rs
   

2. Rebuild:

   cargo build --workspace --release
   

3. Rolling restart (zero downtime):

   kill -TERM $(pgrep -f trading_service) && sleep 5 && cargo run --release -p trading_service &
   kill -TERM $(pgrep -f ml_training_service) && sleep 5 && cargo run --release -p ml_training_service &
   kill -TERM $(pgrep -f backtesting_service) && sleep 5 && cargo run --release -p backtesting_service &
   kill -TERM $(pgrep -f api_gateway) && sleep 5 && cargo run --release -p api_gateway &
   

4. Validate:

   cargo run --release -p ml --example check_feature_count  # Should show 201
   

Data Loss: NONE (Wave D data preserved)

Level 2: Database Rollback (~5 min, Planned Downtime)

When: Database migration failures, data integrity issues (non-critical)

Steps:

1. Backup:

   PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt > /tmp/backup_$(date +%s).sql
   

2. Stop services:

   kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")
   

3. Rollback migration:

   sqlx migrate revert
   

4. Disable Wave D (same as Level 1 Step 1)

5. Rebuild and restart (same as Level 1 Steps 2-3)

Data Loss: Wave D data DELETED (regime_states, regime_transitions, adaptive_strategy_metrics)

Level 3: Full Rollback (~15 min, Full Outage)

When: WaveDDataCorruption, FoxhuntSystemDown (if Wave D suspected)

Steps:

1. Tag and backup:

   git tag "wave-d-emergency-rollback-$(date +%Y%m%d-%H%M%S)"
   PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt > /tmp/full_backup_$(date +%s).sql
   

2. Stop services:

   kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")
   

3. Rollback database (same as Level 2 Step 3)

4. Checkout Wave C:

   WAVE_C_COMMIT=$(git log --all --oneline --before="2025-10-17" | head -1 | awk '{print $1}')
   git stash push -m "Emergency rollback"
   git checkout "$WAVE_C_COMMIT"
   

5. Clean rebuild:

   cargo clean && cargo build --workspace --release
   

6. Manual restart (see Level 1 Step 3)

Data Loss: Complete Wave D removal (code + data)

---

📞 ESCALATION

Severity Levels

Severity	Response Time	Escalation Path
WARNING	30 minutes	Primary On-Call only
CRITICAL	15 minutes	Primary + Secondary On-Call
CATASTROPHIC	5 minutes	Entire team + CTO

Contact Information

Primary On-Call: See ROLLBACK_PROCEDURES.md - Emergency Contacts section

Slack Channels:

• #production-alerts - Automated alerts
• #incident-response - Active incident coordination

PagerDuty: https://foxhunt.pagerduty.com (if configured)

---

🔍 TROUBLESHOOTING

Alert Not Firing (Expected to Fire)

1. Check if metrics exist:

   curl http://localhost:9091/metrics | grep regime_transitions_total
   

2. Check if alert is loaded:

   curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules | length'
   # Expected: 9
   

3. Check alert evaluation:

   curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name == "wave_d_rollback_triggers") | .rules[] | select(.name == "WaveDFlipFlopping")'
   

False Alarm (Alert Firing When Shouldn't)

1. Check current metric value:

   curl -s "http://localhost:9090/api/v1/query?query=rate(regime_transitions_total[1h])*3600"
   

2. Adjust threshold if needed:

   vim config/prometheus/rules/wave_d_alerts.yml
   docker exec foxhunt-prometheus kill -HUP 1  # Reload
   

3. Document tuning decision in WAVE_D_ALERTS_TUNING_LOG.md

Prometheus Unhealthy

1. Check logs:

   docker logs foxhunt-prometheus --tail 50
   

2. Check configuration:

   docker exec foxhunt-prometheus promtool check config /etc/prometheus/prometheus.yml
   

3. Restart if needed:

   docker-compose restart prometheus
   

---

📝 POST-INCIDENT CHECKLIST

After executing a rollback:

• Verify system stability (all services healthy for 1 hour)
• Notify stakeholders (Slack #production-alerts, email trading-team@foxhunt.ai)
• Document incident (incidents/YYYY-MM-DD-wave-d-rollback.md)
• Schedule root cause analysis (within 24 hours)
• Update runbooks based on lessons learned
• Plan re-deployment (after fix validated in staging)

---

📚 REFERENCES

• Full Documentation: WAVE_D_ALERTS_DEPLOYMENT_GUIDE.md
• Rollback Procedures: ROLLBACK_PROCEDURES.md
• Completion Report: AGENT_M1_COMPLETION_REPORT.md
• Test Script: scripts/test_wave_d_alerts.sh

---

Keep this card accessible during on-call shifts!

Last Updated: 2025-10-19