Files
foxhunt/docs/monitoring/grafana-setup.md
jgrusewski 7458f1be01 feat(wave12): E2E validation complete - 225-feature pipeline ready
 Validation Results:
- PPO training: 24.2s (1 epoch, 950 samples, dim=225)
- Feature extraction: 105μs/bar (9.5x faster than target)
- Model checkpoint: 293KB (147KB actor + 146KB critic)
- GPU memory: 145MB used (96.4% headroom)
- Zero dimension mismatches

📊 Success Criteria (5/5):
 Feature dimension = 225 (Wave C 201 + Wave D 24)
 Model state_dim = 225
 Training completed without errors
 Checkpoint saved successfully
 No dimension mismatch errors

📁 Training Data Ready:
- ES.FUT: 2.9MB, 180 days
- NQ.FUT: 4.4MB, 180 days
- 6E.FUT: 2.8MB, 180 days
- ZN.FUT: 65KB, 90 days (clean)

🚀 Next: Full production model retraining (4 models, ~10min GPU time)

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-22 22:48:04 +02:00

2.3 KiB

Grafana Setup Guide

Last Updated: 2025-10-22 Version: Grafana 10.1+


Installation

# Grafana is installed with kube-prometheus-stack
# Access Grafana
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

# Get admin password
kubectl get secret -n monitoring prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode

Dashboard Import

Foxhunt Trading Dashboard

{
  "dashboard": {
    "title": "Foxhunt Trading System",
    "panels": [
      {
        "title": "Order Submission Rate",
        "targets": [
          {
            "expr": "rate(orders_submitted_total[5m])"
          }
        ]
      },
      {
        "title": "API Gateway Latency P99",
        "targets": [
          {
            "expr": "histogram_quantile(0.99, rate(grpc_request_duration_seconds_bucket{service=\"api-gateway\"}[5m]))"
          }
        ]
      },
      {
        "title": "Active Positions",
        "targets": [
          {
            "expr": "positions_open_total"
          }
        ]
      },
      {
        "title": "Database Connection Pool Usage",
        "targets": [
          {
            "expr": "pg_stat_database_numbackends / pg_settings_max_connections * 100"
          }
        ]
      }
    ]
  }
}

Alert Configuration

Critical Alerts

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: foxhunt-alerts
  namespace: foxhunt
spec:
  groups:
  - name: foxhunt-critical
    interval: 30s
    rules:
    - alert: HighErrorRate
      expr: rate(grpc_requests_total{code!~"2.."}[5m]) > 0.1
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "High error rate detected"
        description: "Error rate is {{ $value | humanizePercentage }}"

    - alert: HighLatency
      expr: histogram_quantile(0.99, rate(grpc_request_duration_seconds_bucket[5m])) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High latency detected"
        description: "P99 latency is {{ $value }}s"

    - alert: DatabaseDown
      expr: up{job="postgresql"} == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "PostgreSQL is down"

End of Grafana Setup Guide