foxhunt/docs/archive/infrastructure/ALERTING_ARCHITECTURE.md

Foxhunt Alerting Architecture

Agent H5 | Production Monitoring System

---

🏗️ Alert Flow Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     METRICS COLLECTION                           │
├─────────────────────────────────────────────────────────────────┤
│  API Gateway (9091)  │  Trading Service (9092)  │  PostgreSQL   │
│  Backtesting (9093)  │  ML Training (9094)      │  Prometheus   │
└──────────────────────┬──────────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────────┐
│                    PROMETHEUS (9090)                             │
│                   Alert Evaluation                               │
├─────────────────────────────────────────────────────────────────┤
│  • 8 Alert Groups (32 total alerts)                             │
│  • Evaluation every 15-60s                                      │
│  • Time-series queries with thresholds                          │
│  • State: pending → firing → resolved                           │
└──────────────────────┬──────────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────────┐
│                  ALERTMANAGER (9093)                             │
│              Routing & Deduplication                             │
├─────────────────────────────────────────────────────────────────┤
│  • Group by: alertname, severity, component                     │
│  • Inhibition rules (suppress redundant alerts)                 │
│  • Route by severity and component                              │
└──────────┬──────────┬──────────┬──────────┬────────────────────┘
           │          │          │          │
           ▼          ▼          ▼          ▼
    ┌─────────┐  ┌────────┐  ┌──────┐  ┌────────┐
    │  Slack  │  │ Email  │  │Webhook│  │PagerDuty│
    │ 8 channels│ │Critical│  │Custom │  │ Future │
    └─────────┘  └────────┘  └──────┘  └────────┘

---

🎯 Alert Categories

1. Critical Alerts (Immediate Response)

Response Time: <5 minutes Channels: Slack + Email + Webhook

Latency
├── CriticalP99LatencyAPIGateway (>100ms for 1m)
├── CriticalP99LatencyTradingService (>100ms for 1m)
└── CriticalOrderProcessingLatency (>100ms for 30s)

Availability
├── CriticalServiceDown (unreachable for 30s)
└── DegradedSystemHealth (<75% services up)

Memory
├── CriticalMemoryGrowth (>10%/hour for 5m)
├── CriticalMemoryUsageAbsolute (>8GB)
└── CriticalSystemMemoryPressure (>90% system)

Trading/Risk
├── CriticalPositionLimitBreach (immediate)
├── HighDrawdown (>5%, immediate)
├── CriticalMarketDataStale (>5s old)
└── RiskCheckFailures (>5 in 5m)

Database
├── CriticalPostgreSQLDown (30s)
└── PostgreSQLConnectionPoolExhaustion (>90%)

2. Warning Alerts (Review within hours)

Response Time: <4 hours Channels: Slack only

Errors
├── HighErrorRateAPIGateway (>1% for 3m)
├── HighErrorRateTradingService (>1% for 3m)
└── HighOrderRejectionRate (>1% for 3m)

Resources
├── HighCPUUsage (>80% for 5m)
├── DiskSpaceLow (<15% for 5m)
└── DiskSpaceCritical (<10% for 2m)

ML Health
├── HighMLPredictionLatency (>50ms P99)
└── MLPredictionErrors (>1% for 3m)

Database
└── SlowDatabaseQueries (>100ms avg)

---

🔔 Notification Channels

Slack Channels (8 specialized)

#foxhunt-critical-latency    → P99 latency violations
#foxhunt-critical-outages    → Service down alerts
#foxhunt-critical-memory     → Memory leak detection
#foxhunt-critical-risk       → Risk management alerts
#foxhunt-critical-trading    → Trading system alerts
#foxhunt-critical-database   → Database failures
#foxhunt-warnings-errors     → Error rate warnings
#foxhunt-warnings-resources  → CPU/disk warnings
#foxhunt-warnings-ml         → ML model warnings

Email Recipients

oncall@foxhunt.local         → All critical service down
risk-team@foxhunt.local      → Critical risk alerts
monitoring@foxhunt.local     → Warning aggregates

Webhooks

http://localhost:5001/webhook            → Default
http://localhost:5001/critical-latency   → Latency alerts
http://localhost:5001/critical-service-down → Outages
http://localhost:5001/critical-memory    → Memory alerts
http://localhost:5001/critical-risk      → Risk alerts
http://localhost:5001/critical-trading   → Trading alerts
http://localhost:5001/critical-database  → Database alerts

---

🛡️ Alert Inhibition Rules

Suppression Logic

If Service Down
  ↓
  Suppress: All alerts from that service
  Why: Root cause is service unavailability

If System Health Degraded
  ↓
  Suppress: Individual service alerts
  Why: System-wide issue, not component-specific

If Critical Memory Alert
  ↓
  Suppress: Warning memory alerts
  Why: Critical takes precedence

If Database Down
  ↓
  Suppress: Slow queries, connection pool alerts
  Why: Root cause is database unavailability

If Market Data Stale
  ↓
  Suppress: Risk check failures (may be related)
  Why: Stale data causes risk failures

If Position Limit Breached
  ↓
  Suppress: Order rejection alerts
  Why: Orders rejected due to position limits

---

📊 Alert Timing Matrix

Alert Group	Evaluation	Group Wait	Group Interval	Repeat
Latency Critical	15s	0s	1m	15m
Service Down	15s	0s	30s	5m
Memory Critical	30s	0s	2m	10m
Trading/Risk	15s	0s	30s-1m	5-10m
Error Rates	15s	30s	5m	2h
Resources	30s	1m	5m	4h
ML Health	30s	1m	10m	4h
Aggregate	1m	10s	5m	4h

---

🔍 Alert States

┌──────────┐
│  Normal  │  No threshold breach
└─────┬────┘
      │ Threshold breached
      ▼
┌──────────┐
│ Pending  │  Waiting for "for" duration
└─────┬────┘
      │ Duration met
      ▼
┌──────────┐
│  Firing  │  Alert active, notifications sent
└─────┬────┘
      │ Issue resolved
      ▼
┌──────────┐
│ Resolved │  Recovery notification sent
└──────────┘

---

🎛️ Configuration Files

1. Alert Rules

File: config/prometheus/rules/production-alerts.yml

groups:
  - name: production-latency-critical
    interval: 15s
    rules:
      - alert: CriticalP99LatencyAPIGateway
        expr: histogram_quantile(0.99, rate(...)) > 0.1
        for: 1m
        labels:
          severity: critical
          component: latency

2. AlertManager Routing

File: config/prometheus/alertmanager-production.yml

route:
  receiver: 'default-webhook'
  group_by: ['alertname', 'severity', 'component']
  routes:
    - match:
        severity: critical
        component: latency
      receiver: 'critical-latency'
      group_wait: 0s

3. Prometheus Config

File: config/prometheus/prometheus.yml

rule_files:
  - "rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

---

🚀 Quick Commands

Check Alert Status

# View all firing alerts
curl -s http://localhost:9090/api/v1/alerts | \
  jq '.data.alerts[] | select(.state == "firing")'

# Count alerts by state
curl -s http://localhost:9090/api/v1/alerts | \
  jq '.data.alerts | group_by(.state) | map({state: .[0].state, count: length})'

# View specific alert
curl -s http://localhost:9090/api/v1/alerts | \
  jq '.data.alerts[] | select(.labels.alertname == "CriticalServiceDown")'

Reload Configuration

# Reload Prometheus
curl -X POST http://localhost:9090/-/reload

# Reload AlertManager
curl -X POST http://localhost:9093/-/reload

Test Alerts

# Run test suite
./scripts/test_alerting.sh

# Validate configuration
./scripts/validate_h5_alerting.sh

---

📈 Metrics Dashboard

Key Prometheus Queries

P99 Latency by Service

histogram_quantile(0.99,
  rate(grpc_server_handling_seconds_bucket[1m]))

Error Rate by Service

sum(rate(grpc_server_handled_total{grpc_code!="OK"}[5m])) by (job)
  /
sum(rate(grpc_server_handled_total[5m])) by (job)

Memory Growth Rate

((process_resident_memory_bytes
  - (process_resident_memory_bytes offset 1h))
  / (process_resident_memory_bytes offset 1h)) * 100

Service Availability

up{job=~"api_gateway|trading_service|backtesting_service|ml_training_service"}

---

🎯 Alert Priorities

P0 (Critical - Immediate)

• Service Down
• P99 Latency >100ms
• Memory Growth >10%/hour
• Position Limit Breach
• Drawdown >5%
• Market Data Stale

P1 (Warning - Hours)

• Error Rate >1%
• CPU >80%
• Disk <15%
• Slow Queries
• ML Prediction Errors

P2 (Info - Days)

• System health degradation
• Alert storms
• Configuration changes

---

📞 Escalation Path

Alert Fires
    ↓
Slack Notification (#critical-*)
    ↓
If no ACK in 5 minutes
    ↓
Email to oncall@foxhunt.local
    ↓
If no ACK in 10 minutes
    ↓
PagerDuty escalation (future)
    ↓
If no ACK in 15 minutes
    ↓
SMS to on-call engineer (future)

---

Last Updated: 2025-10-18 Agent: H5 Status: Production Ready Validation: 96.2% (26/27 checks passed)