foxhunt/docs/ALERT_REFERENCE.md

# Prometheus Alert Reference

**Last Updated**: 2025-10-03
**Wave**: 75 Agent 8
**Total Alerts**: 13
**Alert Groups**: 4

## Quick Reference

| Alert | Severity | Threshold | For | Receiver |
|-------|----------|-----------|-----|----------|
| AuthLatencySLAViolation | CRITICAL | p99 > 10μs | 1m | critical-alerts |
| HighAuthFailureRate | WARNING | >10% failures | 2m | warning-alerts |
| RedisConnectionFailure | CRITICAL | >0 errors/s | 1m | critical-alerts |
| RevocationCacheSizeExplosion | WARNING | >100k tokens | 5m | warning-alerts |
| LowCacheHitRate | WARNING | <90% hit rate | 5m | warning-alerts |
| NotifyListenerDisconnected | CRITICAL | disconnected | 1m | critical-alerts |
| HighConfigReloadLatency | WARNING | p95 > 100ms | 5m | warning-alerts |
| ConfigValidationFailures | WARNING | >0 failures/s | 2m | warning-alerts |
| CircuitBreakerOpen | CRITICAL | state > 1.5 | 1m | critical-alerts |
| BackendServiceUnhealthy | CRITICAL | status == 0 | 2m | critical-alerts |
| HighBackendLatency | WARNING | p99 > 100ms | 3m | warning-alerts |
| ConnectionPoolExhaustion | WARNING | >90% utilized | 5m | warning-alerts |
| ExcessiveRateLimiting | WARNING | >10 rejects/s | 5m | warning-alerts |

## Alert Groups

### 1. api_gateway_auth (5 alerts)

#### AuthLatencySLAViolation
- **Severity**: CRITICAL
- **Trigger**: `histogram_quantile(0.99, rate(api_gateway_auth_total_duration_microseconds_bucket[1m])) > 10`
- **For**: 1 minute
- **Description**: p99 auth latency exceeded 10μs SLA
- **Impact**: Authentication performance degradation
- **Action**:
  1. Check API Gateway auth service health
  2. Review recent code deployments
  3. Analyze auth request patterns
  4. Check database connection pool

#### HighAuthFailureRate
- **Severity**: WARNING
- **Trigger**: `100 * rate(api_gateway_auth_requests_failure[5m]) / rate(api_gateway_auth_requests_total[5m]) > 10`
- **For**: 2 minutes
- **Description**: Auth failure rate exceeded 10%
- **Impact**: Users unable to authenticate
- **Action**:
  1. Check auth logs for error patterns
  2. Verify JWT secret configuration
  3. Check token expiration settings
  4. Review RBAC policy changes

#### RedisConnectionFailure
- **Severity**: CRITICAL
- **Trigger**: `rate(api_gateway_auth_errors_redis_failure[1m]) > 0`
- **For**: 1 minute
- **Description**: JWT revocation Redis connection failed
- **Impact**: Revoked tokens may be accepted
- **Action**:
  1. Check Redis service status
  2. Verify network connectivity
  3. Review Redis authentication
  4. Check connection pool exhaustion

#### RevocationCacheSizeExplosion
- **Severity**: WARNING
- **Trigger**: `api_gateway_revoked_tokens_cached > 100000`
- **For**: 5 minutes
- **Description**: JWT revocation cache size excessive
- **Impact**: Memory pressure, potential OOM
- **Action**:
  1. Review token revocation rate
  2. Check cache TTL settings
  3. Implement cache eviction policy
  4. Consider Redis-only storage

#### LowCacheHitRate
- **Severity**: WARNING
- **Trigger**: `100 * rate(api_gateway_rbac_cache_hits[5m]) / (rate(api_gateway_rbac_cache_hits[5m]) + rate(api_gateway_rbac_cache_misses[5m])) < 90`
- **For**: 5 minutes
- **Description**: RBAC cache hit rate below 90%
- **Impact**: Increased database load, slower auth
- **Action**:
  1. Review cache size limits
  2. Check for cache invalidation storms
  3. Analyze RBAC query patterns
  4. Consider increasing cache TTL

### 2. api_gateway_config (3 alerts)

#### NotifyListenerDisconnected
- **Severity**: CRITICAL
- **Trigger**: `api_gateway_notify_listener_connected == 0`
- **For**: 1 minute
- **Description**: PostgreSQL NOTIFY listener disconnected
- **Impact**: Hot-reload capability lost
- **Action**:
  1. Check PostgreSQL connection
  2. Review database logs
  3. Verify NOTIFY/LISTEN permissions
  4. Restart config service if needed

#### HighConfigReloadLatency
- **Severity**: WARNING
- **Trigger**: `histogram_quantile(0.95, rate(api_gateway_config_reload_duration_milliseconds_bucket[1m])) > 100`
- **For**: 5 minutes
- **Description**: p95 config reload latency > 100ms
- **Impact**: Slow config propagation
- **Action**:
  1. Check database query performance
  2. Review config table indexes
  3. Analyze config payload size
  4. Check network latency to database

#### ConfigValidationFailures
- **Severity**: WARNING
- **Trigger**: `rate(api_gateway_config_validation_failure[5m]) > 0`
- **For**: 2 minutes
- **Description**: Invalid config updates detected
- **Impact**: Config changes rejected
- **Action**:
  1. Review recent config changes
  2. Check validation error logs
  3. Verify config schema compliance
  4. Validate JSON structure

### 3. api_gateway_proxy (4 alerts)

#### CircuitBreakerOpen
- **Severity**: CRITICAL
- **Trigger**: `api_gateway_circuit_breaker_state > 1.5`
- **For**: 1 minute
- **Description**: Backend service circuit breaker open
- **Impact**: Requests to backend service failing
- **Action**:
  1. Check backend service health
  2. Review error rate to backend
  3. Verify network connectivity
  4. Check backend response times

#### BackendServiceUnhealthy
- **Severity**: CRITICAL
- **Trigger**: `api_gateway_health_status == 0`
- **For**: 2 minutes
- **Description**: Health checks failing
- **Impact**: Backend service unavailable
- **Action**:
  1. Check backend service status
  2. Review health check endpoint
  3. Verify backend logs
  4. Restart backend service if needed

#### HighBackendLatency
- **Severity**: WARNING
- **Trigger**: `histogram_quantile(0.99, rate(api_gateway_backend_request_duration_milliseconds_bucket[1m])) > 100`
- **For**: 3 minutes
- **Description**: p99 latency to backend > 100ms
- **Impact**: Degraded user experience
- **Action**:
  1. Check backend service load
  2. Review database query performance
  3. Analyze slow endpoints
  4. Consider scaling backend

#### ConnectionPoolExhaustion
- **Severity**: WARNING
- **Trigger**: `100 * api_gateway_connection_pool_active / api_gateway_connection_pool_max > 90`
- **For**: 5 minutes
- **Description**: Connection pool nearly exhausted
- **Impact**: Connection timeouts, request queueing
- **Action**:
  1. Increase connection pool size
  2. Check for connection leaks
  3. Review connection timeout settings
  4. Monitor backend concurrency

### 4. api_gateway_rate_limiting (1 alert)

#### ExcessiveRateLimiting
- **Severity**: WARNING
- **Trigger**: `rate(api_gateway_auth_errors_rate_limited[1m]) > 10`
- **For**: 5 minutes
- **Description**: Rate limit rejections > 10/s
- **Impact**: Potential DDoS or misconfiguration
- **Action**:
  1. Identify source of excessive requests
  2. Review rate limit thresholds
  3. Check for legitimate traffic spikes
  4. Consider IP-based blocking

## Alert Routing

### Critical Alerts → critical-alerts receiver
- **Channels**: PagerDuty + Slack (#foxhunt-critical)
- **Group Wait**: 0s (immediate)
- **Repeat Interval**: 1 hour
- **Alerts**:
  - AuthLatencySLAViolation
  - RedisConnectionFailure
  - NotifyListenerDisconnected
  - CircuitBreakerOpen
  - BackendServiceUnhealthy

### Warning Alerts → warning-alerts receiver
- **Channels**: Slack (#foxhunt-warnings)
- **Group Wait**: 30s
- **Repeat Interval**: 4 hours
- **Alerts**:
  - HighAuthFailureRate
  - RevocationCacheSizeExplosion
  - LowCacheHitRate
  - HighConfigReloadLatency
  - ConfigValidationFailures
  - HighBackendLatency
  - ConnectionPoolExhaustion
  - ExcessiveRateLimiting

### Component-Specific Routing
- **auth** component → auth-alerts receiver (Slack #foxhunt-auth)
- **proxy** component → backend-alerts receiver (Slack #foxhunt-backend)
- **config** component → config-alerts receiver (Slack #foxhunt-config)

## Inhibition Rules

### 1. Circuit Breaker Inhibits Latency
```yaml
source_match:
  alertname: CircuitBreakerOpen
target_match:
  alertname: HighBackendLatency
equal: [service]
```
When circuit breaker opens, suppress latency alerts for same service.

### 2. Service Down Inhibits Circuit Breaker
```yaml
source_match:
  alertname: BackendServiceUnhealthy
target_match_re:
  alertname: HighBackendLatency|CircuitBreakerOpen
equal: [service]
```
When service is completely down, suppress derived alerts.

### 3. NOTIFY Disconnect Inhibits Config Alerts
```yaml
source_match:
  alertname: NotifyListenerDisconnected
target_match_re:
  alertname: HighConfigReloadLatency|ConfigValidationFailures
```
When NOTIFY listener fails, suppress downstream config alerts.

## Testing

### Test All Alerts
```bash
./test_alerts.sh
```

### Simulate Alert (requires amtool)
```bash
amtool alert add AuthLatencySLAViolation \
    --annotation=summary="P99 auth latency exceeded 10μs SLA" \
    --label=severity=critical \
    --label=component=auth \
    --alertmanager.url=http://localhost:9093
```

### View Active Alerts
```bash
curl -s http://localhost:9099/api/v1/alerts | jq '.data.alerts[]'
```

### Silence Alert
```bash
amtool silence add alertname=AuthLatencySLAViolation \
    --alertmanager.url=http://localhost:9093 \
    --comment="Maintenance window" \
    --duration=1h
```

## Metrics Reference

### Auth Metrics
- `api_gateway_auth_total_duration_microseconds_bucket` - Auth latency histogram
- `api_gateway_auth_requests_failure` - Failed auth requests counter
- `api_gateway_auth_requests_total` - Total auth requests counter
- `api_gateway_auth_errors_redis_failure` - Redis connection errors
- `api_gateway_revoked_tokens_cached` - Revoked tokens in cache
- `api_gateway_rbac_cache_hits` - RBAC cache hit counter
- `api_gateway_rbac_cache_misses` - RBAC cache miss counter
- `api_gateway_auth_errors_rate_limited` - Rate limited requests

### Config Metrics
- `api_gateway_notify_listener_connected` - NOTIFY listener status (0/1)
- `api_gateway_config_reload_duration_milliseconds_bucket` - Config reload latency
- `api_gateway_config_validation_failure` - Config validation failures

### Proxy Metrics
- `api_gateway_circuit_breaker_state` - Circuit breaker state (0=closed, 2=open)
- `api_gateway_health_status` - Backend health status (0=unhealthy, 1=healthy)
- `api_gateway_backend_request_duration_milliseconds_bucket` - Backend latency
- `api_gateway_connection_pool_active` - Active connections
- `api_gateway_connection_pool_max` - Max connection pool size

## Related Documentation

- **Alert Testing**: `/home/jgrusewski/Work/foxhunt/docs/WAVE75_AGENT8_ALERT_TESTING.md`
- **Alert Rules**: `/home/jgrusewski/Work/foxhunt/monitoring/prometheus/alerts/api_gateway_alerts.yml`
- **AlertManager Config**: `/home/jgrusewski/Work/foxhunt/monitoring/alertmanager/alertmanager.yml`
- **Prometheus Config**: `/home/jgrusewski/Work/foxhunt/deployment/monitoring/prometheus.yml`