# Alerting Rules Guide

**Last Updated**: 2025-10-22

---

## Alert Severity Levels

| Severity | Response Time | Notification |
|----------|---------------|--------------|
| Critical | 5 min | PagerDuty + Slack + Phone |
| Warning | 30 min | Slack |
| Info | Next day | Email |

---

## Critical Alerts

### System Down

```yaml
- alert: SystemDown
  expr: up{job=~"api-gateway|trading-service"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.job }} is down"
    description: "Service {{ $labels.job }} has been down for 1 minute"
```

### High Error Rate

```yaml
- alert: HighErrorRate
  expr: rate(grpc_requests_total{code!~"2.."}[5m]) / rate(grpc_requests_total[5m]) > 0.05
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "High error rate: {{ $value | humanizePercentage }}"
```

### Database Connection Pool Exhausted

```yaml
- alert: DatabasePoolExhausted
  expr: pg_stat_database_numbackends / pg_settings_max_connections > 0.9
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Database connection pool at {{ $value | humanizePercentage }}"
```

---

## Warning Alerts

### High Latency

```yaml
- alert: HighLatency
  expr: histogram_quantile(0.99, rate(grpc_request_duration_seconds_bucket[5m])) > 0.5
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "P99 latency is {{ $value }}s"
```

### High Memory Usage

```yaml
- alert: HighMemoryUsage
  expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.9
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Memory usage at {{ $value | humanizePercentage }}"
```

---

**End of Alerting Rules Guide**