# Prometheus Setup Guide

**Last Updated**: 2025-10-22
**Version**: Prometheus 2.45+

---

## Installation

```bash
# Install Prometheus Operator
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
    --namespace monitoring \
    --create-namespace \
    --set prometheus.prometheusSpec.retention=30d \
    --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi
```

---

## Configuration

### Service Monitors

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: foxhunt-services
  namespace: foxhunt
spec:
  selector:
    matchLabels:
      app: foxhunt
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics
```

### Scrape Configuration

```yaml
scrape_configs:
  - job_name: 'foxhunt-api-gateway'
    kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
        - foxhunt
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
      action: keep
      regex: api-gateway
    - source_labels: [__meta_kubernetes_pod_name]
      target_label: pod
```

---

## Key Metrics

### Trading Service Metrics

```promql
# Request rate
rate(grpc_requests_total{service="trading-service"}[5m])

# Error rate
rate(grpc_requests_total{service="trading-service",code!~"2.."}[5m])

# Latency P99
histogram_quantile(0.99, rate(grpc_request_duration_seconds_bucket{service="trading-service"}[5m]))

# Order submission rate
rate(orders_submitted_total[5m])

# Position count
positions_open_total
```

### Database Metrics

```promql
# Connection pool usage
pg_stat_database_numbackends / pg_settings_max_connections * 100

# Query duration P95
histogram_quantile(0.95, rate(pg_stat_statements_mean_exec_time_bucket[5m]))

# Cache hit ratio
pg_stat_database_blks_hit / (pg_stat_database_blks_hit + pg_stat_database_blks_read) * 100
```

---

**End of Prometheus Setup Guide**