Files
foxhunt/docs/monitoring/alerting-rules.md
jgrusewski 7458f1be01 feat(wave12): E2E validation complete - 225-feature pipeline ready
 Validation Results:
- PPO training: 24.2s (1 epoch, 950 samples, dim=225)
- Feature extraction: 105μs/bar (9.5x faster than target)
- Model checkpoint: 293KB (147KB actor + 146KB critic)
- GPU memory: 145MB used (96.4% headroom)
- Zero dimension mismatches

📊 Success Criteria (5/5):
 Feature dimension = 225 (Wave C 201 + Wave D 24)
 Model state_dim = 225
 Training completed without errors
 Checkpoint saved successfully
 No dimension mismatch errors

📁 Training Data Ready:
- ES.FUT: 2.9MB, 180 days
- NQ.FUT: 4.4MB, 180 days
- 6E.FUT: 2.8MB, 180 days
- ZN.FUT: 65KB, 90 days (clean)

🚀 Next: Full production model retraining (4 models, ~10min GPU time)

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-22 22:48:04 +02:00

1.7 KiB

Alerting Rules Guide

Last Updated: 2025-10-22


Alert Severity Levels

Severity Response Time Notification
Critical 5 min PagerDuty + Slack + Phone
Warning 30 min Slack
Info Next day Email

Critical Alerts

System Down

- alert: SystemDown
  expr: up{job=~"api-gateway|trading-service"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.job }} is down"
    description: "Service {{ $labels.job }} has been down for 1 minute"

High Error Rate

- alert: HighErrorRate
  expr: rate(grpc_requests_total{code!~"2.."}[5m]) / rate(grpc_requests_total[5m]) > 0.05
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "High error rate: {{ $value | humanizePercentage }}"

Database Connection Pool Exhausted

- alert: DatabasePoolExhausted
  expr: pg_stat_database_numbackends / pg_settings_max_connections > 0.9
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Database connection pool at {{ $value | humanizePercentage }}"

Warning Alerts

High Latency

- alert: HighLatency
  expr: histogram_quantile(0.99, rate(grpc_request_duration_seconds_bucket[5m])) > 0.5
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "P99 latency is {{ $value }}s"

High Memory Usage

- alert: HighMemoryUsage
  expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.9
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Memory usage at {{ $value | humanizePercentage }}"

End of Alerting Rules Guide