foxhunt/services/api_gateway/METRICS_DEPLOYMENT.md

API Gateway Metrics - Deployment Guide

Quick Start

1. Run the Monitoring Stack

cd monitoring
docker-compose up -d

This starts:

• Prometheus on http://localhost:9099 (metrics collection)
• Grafana on http://localhost:3000 (visualization)
• AlertManager on http://localhost:9093 (alert routing)
• PostgreSQL Exporter on http://localhost:9187
• Redis Exporter on http://localhost:9121
• Node Exporter on http://localhost:9100

2. Access Grafana Dashboard

1. Open http://localhost:3000
2. Login: admin / foxhunt2025
3. Navigate to Dashboards → API Gateway - Authentication & Performance

3. Run Example to Generate Metrics

cd services/api_gateway
cargo run --example metrics_example

This will:

• Generate 100 successful auth events
• Record 4 auth failures
• Simulate 50 trading service requests
• Simulate 30 backtesting requests
• Update health status for all backends
• Expose metrics at http://localhost:9090/metrics

4. View Metrics in Prometheus

Open http://localhost:9099/graph and try these queries:

Authentication Success Rate:

100 * rate(api_gateway_auth_requests_success[1m]) / rate(api_gateway_auth_requests_total[1m])

Auth Latency p99:

histogram_quantile(0.99, rate(api_gateway_auth_total_duration_microseconds_bucket[1m]))

Backend Request Rate by Service:

rate(api_gateway_backend_requests_total[1m])

Cache Hit Rate:

100 * rate(api_gateway_jwt_cache_hits[1m]) / (rate(api_gateway_jwt_cache_hits[1m]) + rate(api_gateway_jwt_cache_misses[1m]))

Production Integration

API Gateway Service

use api_gateway::metrics::{GatewayMetrics, metrics_router};
use axum::Server;

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize metrics
    let metrics = GatewayMetrics::new()?;

    // Start Prometheus exporter on separate port
    let metrics_addr = "0.0.0.0:9090".parse()?;
    let metrics_router = metrics_router(metrics.registry());

    tokio::spawn(async move {
        Server::bind(&metrics_addr)
            .serve(metrics_router.into_make_service())
            .await
            .expect("Metrics server failed");
    });

    // Create auth interceptor with metrics
    let auth_interceptor = AuthInterceptor::new(
        jwt_service,
        revocation_service,
        authz_service,
        rate_limiter,
        audit_logger,
    );

    // Instrument auth interceptor to record metrics
    let instrumented_auth = InstrumentedAuthInterceptor::new(
        auth_interceptor,
        metrics.auth.clone(),
    );

    // Create gRPC server with metrics
    let server = Server::builder()
        .layer(instrumented_auth)
        .add_service(trading_proxy)
        .add_service(backtesting_proxy)
        .add_service(ml_training_proxy)
        .serve(addr)
        .await?;

    Ok(())
}

Docker Deployment

Add metrics port to docker-compose.yml:

services:
  api-gateway:
    image: foxhunt/api-gateway:latest
    ports:
      - "50051:50051"  # gRPC
      - "9090:9090"    # Prometheus metrics
    environment:
      - GATEWAY_BIND_ADDR=0.0.0.0:50051
      - METRICS_BIND_ADDR=0.0.0.0:9090
      - JWT_SECRET_FILE=/run/secrets/jwt_secret
      - REDIS_URL=redis://redis:6379
    networks:
      - foxhunt-monitoring

Update Prometheus to scrape API Gateway:

# monitoring/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'api_gateway'
    static_configs:
      - targets: ['api-gateway:9090']

Metrics Overview

Key Metrics to Monitor

Metric	Target	Alert Threshold	Description
api_gateway_auth_total_duration_microseconds (p99)	<10μs	>10μs	Total auth latency SLA
api_gateway_auth_requests_success / total	>99%	<95%	Auth success rate
api_gateway_circuit_breaker_state	0 (closed)	2 (open)	Backend health
api_gateway_health_status	1 (healthy)	0 (unhealthy)	Service availability
api_gateway_notify_listener_connected	1	0	Config hot-reload

Performance Targets

Authentication (HFT Requirements):

• JWT extraction: <0.5μs
• JWT validation: <1μs
• Revocation check: <500ns
• RBAC check: <100ns
• Rate limit check: <50ns
• Total auth: <10μs (p99)

Backend Proxy:

• Trading Service: <50ms (p99)
• Backtesting Service: <500ms (p99)
• ML Training Service: <5000ms (p99)

Configuration:

• Config reload: <50ms (p95)
• Config fetch: <25ms (p95)

Cache Performance:

• JWT cache hit rate: >99%
• RBAC cache hit rate: >95%
• Config cache hit rate: >90%

Alerting

Critical Alerts

1. AuthLatencySLAViolation: p99 auth latency >10μs for 1m
2. CircuitBreakerOpen: Backend circuit breaker opened
3. BackendServiceUnhealthy: Health checks failing for 2m
4. NotifyListenerDisconnected: Hot-reload capability lost
5. RedisConnectionFailure: JWT revocation unavailable

Warning Alerts

1. HighAuthFailureRate: Auth failure rate >10% for 2m
2. HighBackendLatency: Backend p99 >100ms for 3m
3. ConnectionPoolExhaustion: Pool utilization >90% for 5m
4. LowCacheHitRate: RBAC cache hit rate <90% for 5m
5. ExcessiveRateLimiting: Rate limit rejections >10/s for 5m

Grafana Dashboard Panels

Row 1: Authentication Overview

• Auth Requests: Total/Success/Failure rate
• Auth Success Rate: Percentage gauge with thresholds
• Auth SLA Compliance: <10μs target compliance

Row 2: Layer Latency Breakdown

• JWT Extraction p99: Sub-microsecond target
• JWT Validation p99: <1μs target
• Revocation Check p99: <500ns target
• RBAC Check p99: <100ns target
• Rate Limit Check p99: <50ns target
• Total Auth p99: <10μs SLA

Row 3: Error Analysis

• Errors by Type: Missing JWT, expired, revoked, etc.
• Cache Performance: JWT and RBAC cache hit rates

Row 4: Backend Services

• Request Latency: p99 by service
• Circuit Breaker States: Open/closed status
• Health Status: Table view of service health
• Connection Pool: Utilization percentage

Row 5: Configuration

• Reload Events: By type (auth, routing, rate_limit, backend)
• Hot-Reload Latency: p95 reload time
• NOTIFY Listener: Connection status indicator

Row 6: Rate Limiting

• Top 10 Rate-Limited Users: Highest rejection rates
• Active Entries: Total users being tracked

Troubleshooting

Metrics Not Appearing in Prometheus

1. Check API Gateway metrics endpoint:

   curl http://localhost:9090/metrics | grep api_gateway
   

2. Verify Prometheus target:

   curl http://localhost:9099/api/v1/targets | jq '.data.activeTargets[] | select(.job=="api_gateway")'
   

3. Check Prometheus logs:

   docker logs foxhunt-prometheus
   

Grafana Dashboard Not Loading

1. Verify Prometheus data source:

   • Grafana → Configuration → Data Sources
   • URL should be http://prometheus:9090

2. Import dashboard manually:

   • Grafana → Dashboards → Import
   • Upload monitoring/grafana/api_gateway_dashboard.json

High Metrics Cardinality

If Prometheus memory usage is high:

# Check metric cardinality
curl http://localhost:9099/api/v1/label/__name__/values | jq '. | length'

# Identify high-cardinality metrics
curl 'http://localhost:9099/api/v1/query?query=count({__name__=~".+"}) by (__name__)' | jq '.data.result | sort_by(.value[1] | tonumber) | reverse | .[0:10]'

Solutions:

• Reduce user_id label cardinality in per-user metrics
• Use recording rules for aggregated metrics
• Decrease retention period in Prometheus config

Alerts Not Firing

1. Verify alert rules syntax:

   docker exec foxhunt-prometheus promtool check rules /etc/prometheus/alerts/api_gateway_alerts.yml
   

2. Check active alerts:

   curl http://localhost:9099/api/v1/alerts | jq '.data.alerts'
   

3. Verify AlertManager:

   curl http://localhost:9093/api/v2/alerts | jq
   

Performance Impact

Metrics Overhead

• Counter increment: ~50ns
• Histogram observation: ~200ns
• Label lookup: ~10ns (cached)

Total overhead per authenticated request: <500ns (<0.005% of 10μs SLA)

Memory Usage

• Prometheus TSDB: ~50MB per million data points
• Grafana: ~200MB base + 10MB per dashboard
• API Gateway metrics: ~5MB per 100k requests

Network Bandwidth

• Metrics scrape: ~50KB per scrape
• Scrape interval: 5s
• Bandwidth: ~10KB/s per target

Production Checklist

• Prometheus deployed with persistent storage
• Grafana deployed with API Gateway dashboard
• AlertManager configured with notification channels
• PostgreSQL exporter connected to config database
• Redis exporter connected to revocation cache
• API Gateway metrics endpoint exposed on :9090
• Firewall rules allow Prometheus scraping
• HTTPS/TLS configured for Grafana
• Alert routing tested (Slack/PagerDuty)
• Metric retention policy configured (30 days default)
• Backup configured for Prometheus/Grafana data
• Monitoring stack monitored (meta-monitoring)

Next Steps

1. Configure Alerting:

   • Update Slack webhook in alertmanager.yml
   • Configure PagerDuty for critical alerts
   • Test alert routing

2. Customize Dashboard:

   • Add panels for business metrics
   • Configure alerts for your SLAs
   • Set up user-specific views

3. Integrate with CI/CD:

   • Add metrics checks to deployment pipeline
   • Canary deployments based on error rates
   • Automated rollback on SLA violations

4. Advanced Features:

   • Recording rules for complex queries
   • Federation for multi-cluster setup
   • Long-term storage (Thanos/Cortex)

Support

For issues or questions:

• Check logs: docker logs foxhunt-prometheus
• Review Grafana docs: https://grafana.com/docs/
• Prometheus docs: https://prometheus.io/docs/
• Internal docs: services/api_gateway/src/metrics/README.md