foxhunt/services/api_gateway/src/metrics

API Gateway Metrics System

Comprehensive Prometheus metrics for authentication, proxying, and configuration management.

Overview

The API Gateway exposes 60+ Prometheus metrics covering:

1. Authentication Performance (6-layer security)
2. Backend Proxy Metrics (Trading, Backtesting, ML Training services)
3. Configuration Hot-Reload (PostgreSQL NOTIFY/LISTEN)
4. Rate Limiting (Token bucket per-user)

Quick Start

1. Initialize Metrics

use api_gateway::metrics::GatewayMetrics;

#[tokio::main]
async fn main() -> Result<()> {
    // Create metrics registry
    let metrics = GatewayMetrics::new()?;

    // Start Prometheus exporter on :9090/metrics
    let metrics_router = api_gateway::metrics::metrics_router(metrics.registry());

    tokio::spawn(async move {
        axum::Server::bind(&"0.0.0.0:9090".parse().unwrap())
            .serve(metrics_router.into_make_service())
            .await
            .unwrap();
    });

    // Use metrics in auth interceptor
    let auth_interceptor = AuthInterceptor::new_with_metrics(
        jwt_service,
        revocation_service,
        authz_service,
        rate_limiter,
        audit_logger,
        metrics.auth.clone(),
    );

    Ok(())
}

2. Record Authentication Events

use std::time::Instant;

// Record authentication attempt
let start = Instant::now();

// Perform JWT extraction
let jwt_extraction_start = Instant::now();
let jwt = extract_jwt_from_header(request)?;
metrics.auth.jwt_extraction_duration_us.observe(
    jwt_extraction_start.elapsed().as_micros() as f64
);

// Validate JWT signature
let jwt_validation_start = Instant::now();
let claims = jwt_service.validate(&jwt)?;
metrics.auth.jwt_validation_duration_us.observe(
    jwt_validation_start.elapsed().as_micros() as f64
);

// Check revocation
let revocation_start = Instant::now();
let is_revoked = revocation_service.is_revoked(&claims.jti).await?;
metrics.auth.revocation_check_duration_us.observe(
    revocation_start.elapsed().as_micros() as f64
);

if is_revoked {
    metrics.auth.record_failure("revoked_jwt", Some(&claims.sub));
    return Err(AuthError::Revoked);
}

// Check RBAC permissions
let rbac_start = Instant::now();
let has_permission = authz_service.check_permission(&claims, method)?;
metrics.auth.rbac_check_duration_us.observe(
    rbac_start.elapsed().as_micros() as f64
);

// Check rate limit
let rate_limit_start = Instant::now();
let allowed = rate_limiter.check(&claims.sub)?;
metrics.auth.rate_limit_check_duration_us.observe(
    rate_limit_start.elapsed().as_micros() as f64
);

if !allowed {
    metrics.auth.record_rate_limit(&claims.sub);
    return Err(AuthError::RateLimited);
}

// Record success
let total_duration_us = start.elapsed().as_micros() as f64;
metrics.auth.record_success(total_duration_us);
metrics.auth.record_user_request(&claims.sub);

3. Record Backend Proxy Events

// Record backend request
let start = Instant::now();

match backend_client.execute_trade(request).await {
    Ok(response) => {
        let duration_ms = start.elapsed().as_millis() as f64;
        metrics.proxy.record_backend_success("trading", "ExecuteTrade", duration_ms);
    }
    Err(e) => {
        metrics.proxy.record_backend_failure("trading", "timeout");

        // Trip circuit breaker on repeated failures
        if should_trip_circuit_breaker() {
            metrics.proxy.record_circuit_breaker_trip("trading");
        }
    }
}

// Update health status
metrics.proxy.update_health_status("trading", true);

// Update connection pool stats
metrics.proxy.update_connection_pool("trading", active=10, idle=5, max=50);

4. Record Configuration Events

// Record NOTIFY event
metrics.config.notify_events_total.inc();

let start = Instant::now();
match process_config_update(payload).await {
    Ok(config) => {
        let duration_ms = start.elapsed().as_millis() as f64;
        metrics.config.record_notify_event(true);
        metrics.config.record_config_reload("auth", duration_ms);
    }
    Err(e) => {
        metrics.config.record_notify_event(false);
    }
}

// Update listener status
metrics.config.update_listener_status(true);

Metrics Reference

Authentication Metrics (api_gateway_auth_*)

Metric	Type	Description
auth_requests_total	Counter	Total authentication requests
auth_requests_success	Counter	Successful authentications
auth_requests_failure	Counter	Failed authentications
jwt_extraction_duration_microseconds	Histogram	JWT extraction latency (μs)
jwt_validation_duration_microseconds	Histogram	JWT validation latency (μs)
revocation_check_duration_microseconds	Histogram	Revocation check latency (μs)
rbac_check_duration_microseconds	Histogram	RBAC check latency (μs)
rate_limit_check_duration_microseconds	Histogram	Rate limit check latency (μs)
auth_total_duration_microseconds	Histogram	Total auth pipeline latency (μs)
auth_errors_missing_jwt	Counter	Missing Authorization header
auth_errors_invalid_jwt	Counter	Malformed JWT tokens
auth_errors_expired_jwt	Counter	Expired JWT tokens
auth_errors_revoked_jwt	Counter	Revoked JWT tokens
auth_errors_permission_denied	Counter	RBAC permission denied
auth_errors_rate_limited	Counter	Rate limit exceeded
auth_sla_met	Counter	Requests meeting <10μs SLA
auth_sla_exceeded	Counter	Requests exceeding <10μs SLA
jwt_cache_hits	Counter	JWT decoding key cache hits
rbac_cache_hits	Counter	RBAC permission cache hits

Proxy Metrics (api_gateway_backend_*)

Metric	Type	Description
backend_requests_total{service}	Counter	Requests to backend service
backend_requests_success{service}	Counter	Successful backend requests
backend_requests_failure{service,error_type}	Counter	Failed backend requests
backend_request_duration_milliseconds{service,method}	Histogram	Backend request latency (ms)
circuit_breaker_state{service}	Gauge	Circuit breaker state (0=closed, 2=open)
circuit_breaker_trips{service}	Counter	Circuit breaker trips
connection_pool_active{service}	Gauge	Active connections
connection_pool_idle{service}	Gauge	Idle connections
health_status{service}	Gauge	Health status (0=unhealthy, 1=healthy)
grpc_errors{service,status_code}	Counter	gRPC errors by status
routing_duration_microseconds	Histogram	Routing decision latency (μs)

Config Metrics (api_gateway_config_*)

Metric	Type	Description
notify_events_total	Counter	PostgreSQL NOTIFY events received
notify_events_success	Counter	Successfully processed events
config_cache_hits	Counter	Configuration cache hits
config_reload_duration_milliseconds	Histogram	Config reload latency (ms)
notify_listener_connected	Gauge	NOTIFY listener status
config_updates_auth	Counter	Auth config updates
config_updates_routing	Counter	Routing config updates
config_validation_failure	Counter	Config validation failures

Grafana Dashboard

The included Grafana dashboard (monitoring/grafana/api_gateway_dashboard.json) provides:

Panels

1. Authentication Overview

   • Request rate (total/success/failure)
   • Auth success rate (%)
   • SLA compliance (<10μs target)

2. Layer Latency Breakdown

   • JWT extraction p99
   • JWT validation p99
   • Revocation check p99
   • RBAC check p99
   • Rate limit check p99
   • Total auth p99

3. Error Analysis

   • Errors by type (missing JWT, expired, revoked, etc.)
   • Per-user failure tracking

4. Cache Performance

   • JWT cache hit rate
   • RBAC cache hit rate

5. Backend Services

   • Request latency by service (p99)
   • Circuit breaker states
   • Health status table
   • Connection pool utilization

6. Configuration

   • Config reload events by type
   • Hot-reload latency
   • NOTIFY listener status

7. Rate Limiting

   • Top 10 rate-limited users
   • Active rate limiter entries

Alerts

Critical alerts configured:

• Auth latency >10μs (SLA violation)
• High auth failure rate (>10%)
• Redis connection failure
• Circuit breaker open
• Backend service unhealthy
• NOTIFY listener disconnected

Prometheus Configuration

The included Prometheus config (monitoring/prometheus/prometheus.yml) scrapes:

• API Gateway: :9090/metrics
• Trading Service: :9091/metrics
• Backtesting Service: :9092/metrics
• ML Training Service: :9093/metrics
• PostgreSQL Exporter: :9187/metrics
• Redis Exporter: :9121/metrics

Alert Rules

See monitoring/prometheus/alerts/api_gateway_alerts.yml for:

• Auth SLA violations
• Backend health issues
• Configuration problems
• Rate limiting anomalies

Performance Targets

Metric	Target	Alert Threshold
Total auth latency (p99)	<10μs	>10μs
JWT validation (p99)	<1μs	>5μs
Revocation check (p99)	<500ns	>2μs
RBAC check (p99)	<100ns	>1μs
Backend latency (p99)	<50ms	>100ms
Config reload (p95)	<50ms	>100ms
JWT cache hit rate	>99%	<95%
RBAC cache hit rate	>95%	<90%

Docker Deployment

# docker-compose.yml
services:
  api-gateway:
    image: foxhunt/api-gateway:latest
    ports:
      - "50051:50051"  # gRPC
      - "9090:9090"    # Prometheus metrics
    environment:
      - GATEWAY_BIND_ADDR=0.0.0.0:50051
      - METRICS_BIND_ADDR=0.0.0.0:9090

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9099:9090"
    volumes:
      - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./monitoring/prometheus/alerts:/etc/prometheus/alerts

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - ./monitoring/grafana:/etc/grafana/provisioning/dashboards

Development

Running Tests

cargo test --package api_gateway --lib metrics

Benchmarking Metrics Overhead

cargo bench --bench metrics_overhead

Manual Testing with curl

# Query metrics endpoint
curl http://localhost:9090/metrics

# Filter auth metrics
curl http://localhost:9090/metrics | grep api_gateway_auth

# Get specific metric
curl -s http://localhost:9090/metrics | grep auth_requests_total

Architecture Decisions

Why Prometheus?

• Industry standard: Wide adoption in cloud-native systems
• Pull-based: No agent required, simple HTTP endpoint
• PromQL: Powerful query language for analysis
• Grafana integration: Excellent visualization support

Metric Design Principles

1. High cardinality avoidance: Labels limited to service/method/user_id
2. Histogram buckets: Optimized for HFT latencies (sub-microsecond to milliseconds)
3. Cache-friendly: Metrics recorded without allocation where possible
4. Zero overhead when disabled: Registry check at initialization only

Performance Considerations

• Metrics recording adds ~50ns overhead per counter increment
• Histogram observations add ~200ns overhead (quantile calculation)
• Total metrics overhead: <500ns per authenticated request

Troubleshooting

High cardinality issues

If Prometheus runs out of memory:

# Check metric cardinality
curl http://localhost:9090/api/v1/label/__name__/values | jq 'length'

# Identify high-cardinality metrics
curl http://localhost:9090/api/v1/query?query=count({__name__=~".+"}) by (__name__)

Solution: Reduce label diversity (e.g., aggregate per-user metrics)

Missing metrics

# Verify exporter is running
curl http://localhost:9090/metrics | head

# Check Prometheus targets
curl http://localhost:9099/api/v1/targets

Alert not firing

# Check alert rules syntax
promtool check rules monitoring/prometheus/alerts/api_gateway_alerts.yml

# Query alert status
curl http://localhost:9099/api/v1/alerts

Contributing

When adding new metrics:

1. Choose appropriate metric type (Counter/Gauge/Histogram)
2. Add to relevant module (auth_metrics.rs, proxy_metrics.rs, etc.)
3. Register in constructor (new() method)
4. Document in this README
5. Add Grafana panel to dashboard
6. Consider alert rules if critical

License

See workspace LICENSE file.