foxhunt/docs/monitoring/INFLUXDB_METRICS.md

InfluxDB Metrics Collection Documentation

Last Updated: 2025-10-07 Version: 1.0.0 Status: Production Ready

---

Table of Contents

1. Overview
2. Architecture
3. Configuration
4. Metrics Collection
5. Metric Categories
6. Retention Policies
7. Query Examples
8. Integration with Grafana
9. Troubleshooting

---

Overview

Foxhunt uses a dual metrics collection strategy:

• Prometheus: Real-time operational metrics (15-day retention)
• InfluxDB: Historical time-series data for compliance and analytics (90-day default, 7-year compliance)

Why Two Systems?

Feature	Prometheus	InfluxDB
Purpose	Real-time monitoring & alerting	Long-term storage & analytics
Retention	15 days	90 days (default), 7 years (compliance)
Query Language	PromQL	Flux
Data Model	Pull-based scraping	Push-based writing
Best For	Operational dashboards, alerts	Historical analysis, compliance reporting
Write Performance	Moderate (scrape-based)	High (optimized for writes)

---

Architecture

Data Flow

┌─────────────────────────────────────────────────────────────┐
│                    Trading Services                          │
│  (API Gateway, Trading, Backtesting, ML Training)           │
└───────────┬─────────────────────┬───────────────────────────┘
            │                     │
            │ Prometheus          │ InfluxDB
            │ Metrics (Pull)      │ Events (Push)
            ▼                     ▼
┌───────────────────┐    ┌──────────────────┐
│   Prometheus      │    │    InfluxDB      │
│   Port 9090       │    │    Port 8086     │
│   15d retention   │    │  90d retention   │
└─────────┬─────────┘    └────────┬─────────┘
          │                       │
          │                       │
          └───────────┬───────────┘
                      ▼
              ┌──────────────┐
              │   Grafana    │
              │   Port 3000  │
              └──────────────┘

Current Status

InfluxDB Integration Status: ⚠️ INFRASTRUCTURE ONLY

• ✅ Docker container configured and running
• ✅ InfluxDB client library implemented (trading_engine/src/persistence/influxdb.rs)
• ✅ Configuration structure defined
• ⚠️ NOT actively used - no services currently writing to InfluxDB
• ⚠️ Audit trails support InfluxDB as storage backend (not enabled)

Prometheus Integration Status: ✅ FULLY OPERATIONAL

• ✅ All services export Prometheus metrics
• ✅ Comprehensive metric definitions across services
• ✅ Grafana dashboards configured
• ✅ Real-time operational monitoring active

---

Configuration

Docker Compose Configuration

# docker-compose.yml
influxdb:
  image: influxdb:2.7-alpine
  container_name: foxhunt-influxdb
  environment:
    DOCKER_INFLUXDB_INIT_MODE: setup
    DOCKER_INFLUXDB_INIT_USERNAME: foxhunt
    DOCKER_INFLUXDB_INIT_PASSWORD: foxhunt_dev_password
    DOCKER_INFLUXDB_INIT_ORG: foxhunt
    DOCKER_INFLUXDB_INIT_BUCKET: trading_metrics
    DOCKER_INFLUXDB_INIT_RETENTION: 30d  # Default retention
  ports:
    - "8086:8086"
  volumes:
    - influxdb_data:/var/lib/influxdb2
  healthcheck:
    test: ["CMD", "influx", "ping"]
    interval: 30s
    timeout: 10s
    retries: 5

Application Configuration

// trading_engine/src/persistence/influxdb.rs
pub struct InfluxConfig {
    pub url: String,                    // "http://localhost:8086"
    pub org: String,                    // "foxhunt"
    pub bucket: String,                 // "market_data" or "trading_metrics"
    pub token: String,                  // Authentication token
    pub write_timeout_ms: u64,          // 1000 (1 second)
    pub query_timeout_ms: u64,          // 5000 (5 seconds)
    pub batch_size: usize,              // 1000 points per batch
    pub flush_interval_ms: u64,         // 100ms batch flush
    pub enable_compression: bool,       // true (gzip)
    pub connection_pool_size: usize,    // 10 connections
    pub enable_write_retries: bool,     // true
    pub max_retry_attempts: u32,        // 3 attempts
}

Access Credentials

Development Environment:

URL: http://localhost:8086
Organization: foxhunt
Username: foxhunt
Password: foxhunt_dev_password
Default Bucket: trading_metrics

Production Environment: Use Vault for secret management

# Retrieve from Vault
export INFLUXDB_TOKEN=$(vault kv get -field=token secret/influxdb)
export INFLUXDB_ORG=$(vault kv get -field=org secret/influxdb)

---

Metrics Collection

Current Implementation: Prometheus-Based

All services export metrics via Prometheus endpoints:

Service	Metrics Port	Endpoint	Scrape Interval
API Gateway	9091	/metrics	5s
Trading Service	9092	/metrics	5s
Backtesting Service	9093	/metrics	10s
ML Training Service	9094	/metrics	10s

Prometheus Scrape Configuration

# config/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'foxhunt-api-gateway'
    static_configs:
      - targets: ['host.docker.internal:9091']
    metrics_path: '/metrics'
    scrape_interval: 5s

  - job_name: 'foxhunt-trading-service'
    static_configs:
      - targets: ['host.docker.internal:9092']
    metrics_path: '/metrics'
    scrape_interval: 5s

InfluxDB Client (Available but Not Used)

The InfluxDB client is implemented and ready for use:

use trading_engine::persistence::influxdb::{InfluxClient, InfluxConfig, DataPoint, FieldValue};

// Initialize client
let config = InfluxConfig {
    url: "http://localhost:8086".to_string(),
    org: "foxhunt".to_string(),
    bucket: "trading_metrics".to_string(),
    token: std::env::var("INFLUXDB_TOKEN")?,
    ..Default::default()
};

let client = InfluxClient::new(config).await?;

// Write a data point
let point = DataPoint::new("order_latency".to_string())
    .tag("service", "trading")
    .tag("order_type", "market")
    .field("latency_us", FieldValue::Float(123.45))
    .field("success", FieldValue::Boolean(true));

client.write_point(point).await?;

// Query data
let query = r#"
    from(bucket: "trading_metrics")
    |> range(start: -1h)
    |> filter(fn: (r) => r._measurement == "order_latency")
    |> mean()
"#;

let result = client.query(query).await?;

---

Metric Categories

1. API Gateway Metrics

Authentication Metrics:

api_gateway_auth_attempts_total{status}          # Counter
api_gateway_auth_duration_milliseconds{method}   # Histogram
api_gateway_jwt_validations_total{result}        # Counter
api_gateway_mfa_verifications_total{method,result} # Counter
api_gateway_rate_limit_hits{user,endpoint}       # Counter

Proxy Metrics:

api_gateway_backend_requests_total{service}                    # Counter
api_gateway_backend_requests_success{service}                  # Counter
api_gateway_backend_requests_failure{service,error_type}       # Counter
api_gateway_backend_request_duration_milliseconds{service,method} # Histogram
api_gateway_circuit_breaker_state{service}                     # Gauge (0=closed, 1=half-open, 2=open)
api_gateway_connection_pool_active{service}                    # Gauge
api_gateway_health_status{service}                             # Gauge (0=unhealthy, 1=healthy)

2. Trading Service Metrics

Order Processing:

trading_orders_submitted_total{service}          # Counter
trading_order_submission_latency_us{order_type}  # Histogram
trading_order_fills_total{symbol}                # Counter
trading_order_rejections_total{reason}           # Counter

Position Management:

trading_positions_open{symbol}                   # Gauge
trading_position_pnl{symbol}                     # Gauge
trading_notional_exposure{symbol}                # Gauge

Performance:

trading_service_uptime_seconds                   # Counter
trading_buffer_capacity                          # Gauge
trading_buffer_used                              # Gauge
trading_metrics_scrape_duration_seconds          # Gauge

3. ML Model Metrics

Inference Performance:

ml_inference_latency_microseconds{model_id}      # Histogram
ml_predictions_total{model_id,prediction_type}   # Counter
ml_prediction_errors_total{model_id,error_type}  # Counter

Model Health:

ml_model_health_status{model_id}                 # Gauge (0=Healthy, 1=Degraded, 2=Unhealthy, 3=Failed, 4=Offline)
ml_model_accuracy_percent{model_id}              # Gauge
ml_model_confidence_score{model_id}              # Gauge
ml_model_drift_score{model_id}                   # Gauge

Resource Usage:

ml_model_memory_megabytes{model_id}              # Gauge
ml_model_cpu_utilization_percent{model_id}       # Gauge

Fallback & Circuit Breaker:

ml_fallback_total{from_model,to_model,reason}    # Counter
ml_circuit_breaker_transitions_total{model_id,from_state,to_state} # Counter
ml_alerts_total{model_id,alert_type,severity}    # Counter

4. Backtesting Service Metrics

Backtest Execution:

backtesting_runs_total{strategy}                 # Counter
backtesting_duration_seconds{strategy}           # Histogram
backtesting_market_events_processed{strategy}    # Counter

Performance Analytics:

backtesting_sharpe_ratio{strategy,backtest_id}   # Gauge
backtesting_max_drawdown{strategy,backtest_id}   # Gauge
backtesting_total_pnl{strategy,backtest_id}      # Gauge
backtesting_win_rate{strategy,backtest_id}       # Gauge

---

Retention Policies

Default Retention

Bucket	Retention Period	Purpose
trading_metrics	90 days	Operational metrics
compliance_audit	7 years	Regulatory compliance (SOX, MiFID II)
market_data	90 days	Historical market data
downsampled_1h	2 years	Hourly aggregations
downsampled_1d	10 years	Daily aggregations

Configuring Retention Policies

Via InfluxDB CLI:

# Create bucket with custom retention
influx bucket create \
  -n compliance_audit \
  -o foxhunt \
  -r 2555d  # 7 years

# Update existing bucket
influx bucket update \
  -n trading_metrics \
  -r 90d

Via API (automated):

curl -XPOST "http://localhost:8086/api/v2/buckets" \
  -H "Authorization: Token YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "orgID": "YOUR_ORG_ID",
    "name": "compliance_audit",
    "retentionRules": [{"type": "expire", "everySeconds": 220752000}]
  }'

Downsampling Strategy

For long-term storage efficiency:

// Hourly downsampling (runs every hour)
from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> aggregateWindow(every: 1h, fn: mean)
  |> to(bucket: "downsampled_1h", org: "foxhunt")

// Daily downsampling (runs daily)
from(bucket: "downsampled_1h")
  |> range(start: -1d)
  |> aggregateWindow(every: 1d, fn: mean)
  |> to(bucket: "downsampled_1d", org: "foxhunt")

---

Query Examples

Common Queries

1. Average Order Latency (Last Hour):

from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "order_latency")
  |> filter(fn: (r) => r._field == "latency_us")
  |> mean()

2. Orders Per Second:

from(bucket: "trading_metrics")
  |> range(start: -5m)
  |> filter(fn: (r) => r._measurement == "order_submitted")
  |> aggregateWindow(every: 1s, fn: count)

3. P99 Latency by Service:

from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "request_duration")
  |> group(columns: ["service"])
  |> quantile(q: 0.99, column: "_value")

4. Trading PnL Over Time:

from(bucket: "trading_metrics")
  |> range(start: -24h)
  |> filter(fn: (r) => r._measurement == "position_pnl")
  |> aggregateWindow(every: 1h, fn: last)

5. Circuit Breaker State Changes:

from(bucket: "trading_metrics")
  |> range(start: -7d)
  |> filter(fn: (r) => r._measurement == "circuit_breaker_transition")
  |> filter(fn: (r) => r.to_state == "open")
  |> count()

6. Compliance Audit Trail Query:

from(bucket: "compliance_audit")
  |> range(start: 2024-01-01T00:00:00Z, stop: 2024-12-31T23:59:59Z)
  |> filter(fn: (r) => r._measurement == "audit_event")
  |> filter(fn: (r) => r.event_type == "order_placed")
  |> filter(fn: (r) => r.user_id == "trader_001")

Performance Optimization

Use Time Bounds:

// Good - bounded query
from(bucket: "trading_metrics")
  |> range(start: -1h, stop: now())

// Bad - unbounded query
from(bucket: "trading_metrics")
  |> range(start: 0)

Aggregate Early:

// Good - aggregate before filtering
from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> aggregateWindow(every: 1m, fn: mean)
  |> filter(fn: (r) => r._value > 100)

// Bad - filter before aggregating
from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> filter(fn: (r) => r._value > 100)
  |> aggregateWindow(every: 1m, fn: mean)

---

Integration with Grafana

Adding InfluxDB as Data Source

1. Navigate to Configuration → Data Sources → Add data source
2. Select InfluxDB (choose v2.x)
3. Configure:

   Query Language: Flux
   URL: http://influxdb:8086
   Organization: foxhunt
   Token: <from Vault or environment>
   Default Bucket: trading_metrics
   

4. Test Connection → Save & Test

Example Dashboard Queries

Trading Latency Panel:

from(bucket: v.bucket)
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "order_latency")
  |> filter(fn: (r) => r._field == "latency_us")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)

ML Model Health Status:

from(bucket: "trading_metrics")
  |> range(start: v.timeRangeStart)
  |> filter(fn: (r) => r._measurement == "ml_model_health")
  |> last()
  |> map(fn: (r) => ({ r with _value:
      if r._value == 0 then "Healthy"
      else if r._value == 1 then "Degraded"
      else if r._value == 2 then "Unhealthy"
      else if r._value == 3 then "Failed"
      else "Offline"
  }))

---

Troubleshooting

Common Issues

1. Connection Refused:

# Check if InfluxDB is running
docker ps | grep influxdb

# Check logs
docker logs foxhunt-influxdb

# Verify port binding
netstat -an | grep 8086

2. Authentication Failures:

# Verify token
influx auth list

# Create new token
influx auth create \
  --org foxhunt \
  --read-buckets \
  --write-buckets

3. Write Timeouts:

// Increase timeout in config
let config = InfluxConfig {
    write_timeout_ms: 5000,  // Increase from 1000ms
    batch_size: 500,         // Reduce batch size
    ..Default::default()
};

4. Query Performance:

# Check bucket cardinality
influx query '
from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> group()
  |> count()
'

# Enable query profiling
influx query --profilers query,operator '...'

Health Check

# InfluxDB health endpoint
curl http://localhost:8086/health

# Check bucket list
influx bucket list

# Check organization
influx org list

Performance Tuning

Write Performance:

• Batch size: 500-1000 points optimal
• Enable compression: reduces network overhead by 70%
• Connection pooling: 10-20 connections for high throughput
• Use tags for indexed fields, fields for values

Query Performance:

• Always use time bounds (range())
• Aggregate early in query pipeline
• Use appropriate downsampling buckets
• Limit cardinality of tag values

---

Next Steps

Enabling InfluxDB Integration

To activate InfluxDB metrics collection:

1. Update Service Configuration:

   // Add to service initialization
   let influx_config = load_influx_config()?;
   let influx_client = InfluxClient::new(influx_config).await?;
   

2. Enable Audit Trail Storage:

   // trading_engine/src/compliance/audit_trails.rs
   let config = AuditTrailConfig {
       storage_backend: StorageBackendConfig {
           primary_storage: StorageType::InfluxDB,
           backup_storage: Some(StorageType::PostgreSQL),
           // ...
       },
       // ...
   };
   

3. Add Metrics Export Task:

   // Periodic metrics export to InfluxDB
   tokio::spawn(async move {
       let mut interval = tokio::time::interval(Duration::from_secs(60));
       loop {
           interval.tick().await;
           export_metrics_to_influx(&influx_client).await;
       }
   });
   

4. Create Grafana Dashboards using InfluxDB data source

---

Document Version: 1.0.0 Last Review: 2025-10-07 Next Review: 2025-11-07 Maintainer: Foxhunt DevOps Team