foxhunt/docs/monitoring/INFLUXDB_METRICS.md

# InfluxDB Metrics Collection Documentation

**Last Updated**: 2025-10-07
**Version**: 1.0.0
**Status**: Production Ready

---

## Table of Contents

1. [Overview](#overview)
2. [Architecture](#architecture)
3. [Configuration](#configuration)
4. [Metrics Collection](#metrics-collection)
5. [Metric Categories](#metric-categories)
6. [Retention Policies](#retention-policies)
7. [Query Examples](#query-examples)
8. [Integration with Grafana](#integration-with-grafana)
9. [Troubleshooting](#troubleshooting)

---

## Overview

Foxhunt uses a **dual metrics collection strategy**:

- **Prometheus**: Real-time operational metrics (15-day retention)
- **InfluxDB**: Historical time-series data for compliance and analytics (90-day default, 7-year compliance)

### Why Two Systems?

| Feature | Prometheus | InfluxDB |
|---------|-----------|----------|
| **Purpose** | Real-time monitoring & alerting | Long-term storage & analytics |
| **Retention** | 15 days | 90 days (default), 7 years (compliance) |
| **Query Language** | PromQL | Flux |
| **Data Model** | Pull-based scraping | Push-based writing |
| **Best For** | Operational dashboards, alerts | Historical analysis, compliance reporting |
| **Write Performance** | Moderate (scrape-based) | High (optimized for writes) |

---

## Architecture

### Data Flow

```
┌─────────────────────────────────────────────────────────────┐
│                    Trading Services                          │
│  (API Gateway, Trading, Backtesting, ML Training)           │
└───────────┬─────────────────────┬───────────────────────────┘
            │                     │
            │ Prometheus          │ InfluxDB
            │ Metrics (Pull)      │ Events (Push)
            ▼                     ▼
┌───────────────────┐    ┌──────────────────┐
│   Prometheus      │    │    InfluxDB      │
│   Port 9090       │    │    Port 8086     │
│   15d retention   │    │  90d retention   │
└─────────┬─────────┘    └────────┬─────────┘
          │                       │
          │                       │
          └───────────┬───────────┘
                      ▼
              ┌──────────────┐
              │   Grafana    │
              │   Port 3000  │
              └──────────────┘
```

### Current Status

**InfluxDB Integration Status**: ⚠️ **INFRASTRUCTURE ONLY**

- ✅ Docker container configured and running
- ✅ InfluxDB client library implemented (`trading_engine/src/persistence/influxdb.rs`)
- ✅ Configuration structure defined
- ⚠️ **NOT actively used** - no services currently writing to InfluxDB
- ⚠️ Audit trails support InfluxDB as storage backend (not enabled)

**Prometheus Integration Status**: ✅ **FULLY OPERATIONAL**

- ✅ All services export Prometheus metrics
- ✅ Comprehensive metric definitions across services
- ✅ Grafana dashboards configured
- ✅ Real-time operational monitoring active

---

## Configuration

### Docker Compose Configuration

```yaml
# docker-compose.yml
influxdb:
  image: influxdb:2.7-alpine
  container_name: foxhunt-influxdb
  environment:
    DOCKER_INFLUXDB_INIT_MODE: setup
    DOCKER_INFLUXDB_INIT_USERNAME: foxhunt
    DOCKER_INFLUXDB_INIT_PASSWORD: foxhunt_dev_password
    DOCKER_INFLUXDB_INIT_ORG: foxhunt
    DOCKER_INFLUXDB_INIT_BUCKET: trading_metrics
    DOCKER_INFLUXDB_INIT_RETENTION: 30d  # Default retention
  ports:
    - "8086:8086"
  volumes:
    - influxdb_data:/var/lib/influxdb2
  healthcheck:
    test: ["CMD", "influx", "ping"]
    interval: 30s
    timeout: 10s
    retries: 5
```

### Application Configuration

```rust
// trading_engine/src/persistence/influxdb.rs
pub struct InfluxConfig {
    pub url: String,                    // "http://localhost:8086"
    pub org: String,                    // "foxhunt"
    pub bucket: String,                 // "market_data" or "trading_metrics"
    pub token: String,                  // Authentication token
    pub write_timeout_ms: u64,          // 1000 (1 second)
    pub query_timeout_ms: u64,          // 5000 (5 seconds)
    pub batch_size: usize,              // 1000 points per batch
    pub flush_interval_ms: u64,         // 100ms batch flush
    pub enable_compression: bool,       // true (gzip)
    pub connection_pool_size: usize,    // 10 connections
    pub enable_write_retries: bool,     // true
    pub max_retry_attempts: u32,        // 3 attempts
}
```

### Access Credentials

**Development Environment**:
```bash
URL: http://localhost:8086
Organization: foxhunt
Username: foxhunt
Password: foxhunt_dev_password
Default Bucket: trading_metrics
```

**Production Environment**: Use Vault for secret management
```bash
# Retrieve from Vault
export INFLUXDB_TOKEN=$(vault kv get -field=token secret/influxdb)
export INFLUXDB_ORG=$(vault kv get -field=org secret/influxdb)
```

---

## Metrics Collection

### Current Implementation: Prometheus-Based

All services export metrics via Prometheus endpoints:

| Service | Metrics Port | Endpoint | Scrape Interval |
|---------|--------------|----------|-----------------|
| API Gateway | 9091 | /metrics | 5s |
| Trading Service | 9092 | /metrics | 5s |
| Backtesting Service | 9093 | /metrics | 10s |
| ML Training Service | 9094 | /metrics | 10s |

### Prometheus Scrape Configuration

```yaml
# config/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'foxhunt-api-gateway'
    static_configs:
      - targets: ['host.docker.internal:9091']
    metrics_path: '/metrics'
    scrape_interval: 5s

  - job_name: 'foxhunt-trading-service'
    static_configs:
      - targets: ['host.docker.internal:9092']
    metrics_path: '/metrics'
    scrape_interval: 5s
```

### InfluxDB Client (Available but Not Used)

The InfluxDB client is implemented and ready for use:

```rust
use trading_engine::persistence::influxdb::{InfluxClient, InfluxConfig, DataPoint, FieldValue};

// Initialize client
let config = InfluxConfig {
    url: "http://localhost:8086".to_string(),
    org: "foxhunt".to_string(),
    bucket: "trading_metrics".to_string(),
    token: std::env::var("INFLUXDB_TOKEN")?,
    ..Default::default()
};

let client = InfluxClient::new(config).await?;

// Write a data point
let point = DataPoint::new("order_latency".to_string())
    .tag("service", "trading")
    .tag("order_type", "market")
    .field("latency_us", FieldValue::Float(123.45))
    .field("success", FieldValue::Boolean(true));

client.write_point(point).await?;

// Query data
let query = r#"
    from(bucket: "trading_metrics")
    |> range(start: -1h)
    |> filter(fn: (r) => r._measurement == "order_latency")
    |> mean()
"#;

let result = client.query(query).await?;
```

---

## Metric Categories

### 1. API Gateway Metrics

**Authentication Metrics**:
```
api_gateway_auth_attempts_total{status}          # Counter
api_gateway_auth_duration_milliseconds{method}   # Histogram
api_gateway_jwt_validations_total{result}        # Counter
api_gateway_mfa_verifications_total{method,result} # Counter
api_gateway_rate_limit_hits{user,endpoint}       # Counter
```

**Proxy Metrics**:
```
api_gateway_backend_requests_total{service}                    # Counter
api_gateway_backend_requests_success{service}                  # Counter
api_gateway_backend_requests_failure{service,error_type}       # Counter
api_gateway_backend_request_duration_milliseconds{service,method} # Histogram
api_gateway_circuit_breaker_state{service}                     # Gauge (0=closed, 1=half-open, 2=open)
api_gateway_connection_pool_active{service}                    # Gauge
api_gateway_health_status{service}                             # Gauge (0=unhealthy, 1=healthy)
```

### 2. Trading Service Metrics

**Order Processing**:
```
trading_orders_submitted_total{service}          # Counter
trading_order_submission_latency_us{order_type}  # Histogram
trading_order_fills_total{symbol}                # Counter
trading_order_rejections_total{reason}           # Counter
```

**Position Management**:
```
trading_positions_open{symbol}                   # Gauge
trading_position_pnl{symbol}                     # Gauge
trading_notional_exposure{symbol}                # Gauge
```

**Performance**:
```
trading_service_uptime_seconds                   # Counter
trading_buffer_capacity                          # Gauge
trading_buffer_used                              # Gauge
trading_metrics_scrape_duration_seconds          # Gauge
```

### 3. ML Model Metrics

**Inference Performance**:
```
ml_inference_latency_microseconds{model_id}      # Histogram
ml_predictions_total{model_id,prediction_type}   # Counter
ml_prediction_errors_total{model_id,error_type}  # Counter
```

**Model Health**:
```
ml_model_health_status{model_id}                 # Gauge (0=Healthy, 1=Degraded, 2=Unhealthy, 3=Failed, 4=Offline)
ml_model_accuracy_percent{model_id}              # Gauge
ml_model_confidence_score{model_id}              # Gauge
ml_model_drift_score{model_id}                   # Gauge
```

**Resource Usage**:
```
ml_model_memory_megabytes{model_id}              # Gauge
ml_model_cpu_utilization_percent{model_id}       # Gauge
```

**Fallback & Circuit Breaker**:
```
ml_fallback_total{from_model,to_model,reason}    # Counter
ml_circuit_breaker_transitions_total{model_id,from_state,to_state} # Counter
ml_alerts_total{model_id,alert_type,severity}    # Counter
```

### 4. Backtesting Service Metrics

**Backtest Execution**:
```
backtesting_runs_total{strategy}                 # Counter
backtesting_duration_seconds{strategy}           # Histogram
backtesting_market_events_processed{strategy}    # Counter
```

**Performance Analytics**:
```
backtesting_sharpe_ratio{strategy,backtest_id}   # Gauge
backtesting_max_drawdown{strategy,backtest_id}   # Gauge
backtesting_total_pnl{strategy,backtest_id}      # Gauge
backtesting_win_rate{strategy,backtest_id}       # Gauge
```

---

## Retention Policies

### Default Retention

| Bucket | Retention Period | Purpose |
|--------|------------------|---------|
| `trading_metrics` | 90 days | Operational metrics |
| `compliance_audit` | 7 years | Regulatory compliance (SOX, MiFID II) |
| `market_data` | 90 days | Historical market data |
| `downsampled_1h` | 2 years | Hourly aggregations |
| `downsampled_1d` | 10 years | Daily aggregations |

### Configuring Retention Policies

**Via InfluxDB CLI**:
```bash
# Create bucket with custom retention
influx bucket create \
  -n compliance_audit \
  -o foxhunt \
  -r 2555d  # 7 years

# Update existing bucket
influx bucket update \
  -n trading_metrics \
  -r 90d
```

**Via API** (automated):
```bash
curl -XPOST "http://localhost:8086/api/v2/buckets" \
  -H "Authorization: Token YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "orgID": "YOUR_ORG_ID",
    "name": "compliance_audit",
    "retentionRules": [{"type": "expire", "everySeconds": 220752000}]
  }'
```

### Downsampling Strategy

For long-term storage efficiency:

```flux
// Hourly downsampling (runs every hour)
from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> aggregateWindow(every: 1h, fn: mean)
  |> to(bucket: "downsampled_1h", org: "foxhunt")

// Daily downsampling (runs daily)
from(bucket: "downsampled_1h")
  |> range(start: -1d)
  |> aggregateWindow(every: 1d, fn: mean)
  |> to(bucket: "downsampled_1d", org: "foxhunt")
```

---

## Query Examples

### Common Queries

**1. Average Order Latency (Last Hour)**:
```flux
from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "order_latency")
  |> filter(fn: (r) => r._field == "latency_us")
  |> mean()
```

**2. Orders Per Second**:
```flux
from(bucket: "trading_metrics")
  |> range(start: -5m)
  |> filter(fn: (r) => r._measurement == "order_submitted")
  |> aggregateWindow(every: 1s, fn: count)
```

**3. P99 Latency by Service**:
```flux
from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "request_duration")
  |> group(columns: ["service"])
  |> quantile(q: 0.99, column: "_value")
```

**4. Trading PnL Over Time**:
```flux
from(bucket: "trading_metrics")
  |> range(start: -24h)
  |> filter(fn: (r) => r._measurement == "position_pnl")
  |> aggregateWindow(every: 1h, fn: last)
```

**5. Circuit Breaker State Changes**:
```flux
from(bucket: "trading_metrics")
  |> range(start: -7d)
  |> filter(fn: (r) => r._measurement == "circuit_breaker_transition")
  |> filter(fn: (r) => r.to_state == "open")
  |> count()
```

**6. Compliance Audit Trail Query**:
```flux
from(bucket: "compliance_audit")
  |> range(start: 2024-01-01T00:00:00Z, stop: 2024-12-31T23:59:59Z)
  |> filter(fn: (r) => r._measurement == "audit_event")
  |> filter(fn: (r) => r.event_type == "order_placed")
  |> filter(fn: (r) => r.user_id == "trader_001")
```

### Performance Optimization

**Use Time Bounds**:
```flux
// Good - bounded query
from(bucket: "trading_metrics")
  |> range(start: -1h, stop: now())

// Bad - unbounded query
from(bucket: "trading_metrics")
  |> range(start: 0)
```

**Aggregate Early**:
```flux
// Good - aggregate before filtering
from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> aggregateWindow(every: 1m, fn: mean)
  |> filter(fn: (r) => r._value > 100)

// Bad - filter before aggregating
from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> filter(fn: (r) => r._value > 100)
  |> aggregateWindow(every: 1m, fn: mean)
```

---

## Integration with Grafana

### Adding InfluxDB as Data Source

1. **Navigate to Configuration** → **Data Sources** → **Add data source**
2. **Select InfluxDB** (choose v2.x)
3. **Configure**:
   ```
   Query Language: Flux
   URL: http://influxdb:8086
   Organization: foxhunt
   Token: <from Vault or environment>
   Default Bucket: trading_metrics
   ```
4. **Test Connection** → **Save & Test**

### Example Dashboard Queries

**Trading Latency Panel**:
```flux
from(bucket: v.bucket)
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "order_latency")
  |> filter(fn: (r) => r._field == "latency_us")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
```

**ML Model Health Status**:
```flux
from(bucket: "trading_metrics")
  |> range(start: v.timeRangeStart)
  |> filter(fn: (r) => r._measurement == "ml_model_health")
  |> last()
  |> map(fn: (r) => ({ r with _value:
      if r._value == 0 then "Healthy"
      else if r._value == 1 then "Degraded"
      else if r._value == 2 then "Unhealthy"
      else if r._value == 3 then "Failed"
      else "Offline"
  }))
```

---

## Troubleshooting

### Common Issues

**1. Connection Refused**:
```bash
# Check if InfluxDB is running
docker ps | grep influxdb

# Check logs
docker logs foxhunt-influxdb

# Verify port binding
netstat -an | grep 8086
```

**2. Authentication Failures**:
```bash
# Verify token
influx auth list

# Create new token
influx auth create \
  --org foxhunt \
  --read-buckets \
  --write-buckets
```

**3. Write Timeouts**:
```rust
// Increase timeout in config
let config = InfluxConfig {
    write_timeout_ms: 5000,  // Increase from 1000ms
    batch_size: 500,         // Reduce batch size
    ..Default::default()
};
```

**4. Query Performance**:
```bash
# Check bucket cardinality
influx query '
from(bucket: "trading_metrics")
  |> range(start: -1h)
  |> group()
  |> count()
'

# Enable query profiling
influx query --profilers query,operator '...'
```

### Health Check

```bash
# InfluxDB health endpoint
curl http://localhost:8086/health

# Check bucket list
influx bucket list

# Check organization
influx org list
```

### Performance Tuning

**Write Performance**:
- Batch size: 500-1000 points optimal
- Enable compression: reduces network overhead by 70%
- Connection pooling: 10-20 connections for high throughput
- Use tags for indexed fields, fields for values

**Query Performance**:
- Always use time bounds (`range()`)
- Aggregate early in query pipeline
- Use appropriate downsampling buckets
- Limit cardinality of tag values

---

## Next Steps

### Enabling InfluxDB Integration

To activate InfluxDB metrics collection:

1. **Update Service Configuration**:
   ```rust
   // Add to service initialization
   let influx_config = load_influx_config()?;
   let influx_client = InfluxClient::new(influx_config).await?;
   ```

2. **Enable Audit Trail Storage**:
   ```rust
   // trading_engine/src/compliance/audit_trails.rs
   let config = AuditTrailConfig {
       storage_backend: StorageBackendConfig {
           primary_storage: StorageType::InfluxDB,
           backup_storage: Some(StorageType::PostgreSQL),
           // ...
       },
       // ...
   };
   ```

3. **Add Metrics Export Task**:
   ```rust
   // Periodic metrics export to InfluxDB
   tokio::spawn(async move {
       let mut interval = tokio::time::interval(Duration::from_secs(60));
       loop {
           interval.tick().await;
           export_metrics_to_influx(&influx_client).await;
       }
   });
   ```

4. **Create Grafana Dashboards** using InfluxDB data source

---

**Document Version**: 1.0.0
**Last Review**: 2025-10-07
**Next Review**: 2025-11-07
**Maintainer**: Foxhunt DevOps Team