Files
foxhunt/docs/monitoring/prometheus-setup.md
jgrusewski 7458f1be01 feat(wave12): E2E validation complete - 225-feature pipeline ready
 Validation Results:
- PPO training: 24.2s (1 epoch, 950 samples, dim=225)
- Feature extraction: 105μs/bar (9.5x faster than target)
- Model checkpoint: 293KB (147KB actor + 146KB critic)
- GPU memory: 145MB used (96.4% headroom)
- Zero dimension mismatches

📊 Success Criteria (5/5):
 Feature dimension = 225 (Wave C 201 + Wave D 24)
 Model state_dim = 225
 Training completed without errors
 Checkpoint saved successfully
 No dimension mismatch errors

📁 Training Data Ready:
- ES.FUT: 2.9MB, 180 days
- NQ.FUT: 4.4MB, 180 days
- 6E.FUT: 2.8MB, 180 days
- ZN.FUT: 65KB, 90 days (clean)

🚀 Next: Full production model retraining (4 models, ~10min GPU time)

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-22 22:48:04 +02:00

2.0 KiB

Prometheus Setup Guide

Last Updated: 2025-10-22 Version: Prometheus 2.45+


Installation

# Install Prometheus Operator
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
    --namespace monitoring \
    --create-namespace \
    --set prometheus.prometheusSpec.retention=30d \
    --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi

Configuration

Service Monitors

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: foxhunt-services
  namespace: foxhunt
spec:
  selector:
    matchLabels:
      app: foxhunt
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

Scrape Configuration

scrape_configs:
  - job_name: 'foxhunt-api-gateway'
    kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
        - foxhunt
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
      action: keep
      regex: api-gateway
    - source_labels: [__meta_kubernetes_pod_name]
      target_label: pod

Key Metrics

Trading Service Metrics

# Request rate
rate(grpc_requests_total{service="trading-service"}[5m])

# Error rate
rate(grpc_requests_total{service="trading-service",code!~"2.."}[5m])

# Latency P99
histogram_quantile(0.99, rate(grpc_request_duration_seconds_bucket{service="trading-service"}[5m]))

# Order submission rate
rate(orders_submitted_total[5m])

# Position count
positions_open_total

Database Metrics

# Connection pool usage
pg_stat_database_numbackends / pg_settings_max_connections * 100

# Query duration P95
histogram_quantile(0.95, rate(pg_stat_statements_mean_exec_time_bucket[5m]))

# Cache hit ratio
pg_stat_database_blks_hit / (pg_stat_database_blks_hit + pg_stat_database_blks_read) * 100

End of Prometheus Setup Guide