foxhunt/MONITORING_GUIDE.md

Foxhunt HFT Trading System - Comprehensive Monitoring Guide

🚀 Overview

This guide provides comprehensive instructions for setting up, configuring, and operating the monitoring infrastructure for the Foxhunt HFT Trading System. The monitoring stack is designed for ultra-low latency trading operations with enterprise-grade observability, alerting, and compliance reporting.

📊 Monitoring Architecture

Monitoring & Observability Architecture:
┌─────────────────────────────────────────────────────────────────────────┐
│                      Monitoring Data Flow                                │
├─────────────────────────────────────────────────────────────────────────┤
│  Data Sources (Ultra-High Frequency)                                   │
│  ├── Trading Service        → 1s scrape (order metrics)               │
│  ├── Risk Management        → 2s scrape (risk metrics)                │  
│  ├── TLI Interface          → 2s scrape (user metrics)                │
│  ├── ML Inference           → 10s scrape (model metrics)              │
│  └── System Resources       → 10s scrape (hardware metrics)           │
├─────────────────────────────────────────────────────────────────────────┤
│  Collection & Storage Layer                                            │
│  ├── Prometheus             → Metrics collection & storage             │
│  ├── Loki                   → Log aggregation                         │
│  ├── Tempo                  → Distributed tracing                     │
│  └── InfluxDB               → High-frequency time series              │
├─────────────────────────────────────────────────────────────────────────┤
│  Processing & Analytics                                                 │
│  ├── AlertManager           → Real-time alerting                      │
│  ├── Grafana                → Visualization & dashboards              │
│  ├── Custom Analytics       → HFT-specific calculations               │
│  └── Compliance Reporting   → Regulatory compliance                   │
├─────────────────────────────────────────────────────────────────────────┤
│  Notification & Response                                                │
│  ├── Slack Integration      → Team notifications                      │
│  ├── Email Alerts           → Executive notifications                 │
│  ├── PagerDuty               → On-call escalation                     │
│  ├── SMS/Voice               → Emergency notifications                │
│  └── Auto-Remediation       → Automated response actions             │
└─────────────────────────────────────────────────────────────────────────┘

🔧 Installation & Setup

Prerequisites

System Requirements:

# Monitoring Server Specifications
CPU: 16+ cores (Intel Xeon or AMD EPYC)
Memory: 64GB+ RAM (128GB recommended)
Storage: 1TB+ NVMe SSD for metrics storage
Network: 10Gbps+ connection to trading infrastructure
OS: Ubuntu 22.04 LTS or RHEL 8+

Required Software:

# Update system
sudo apt update && sudo apt upgrade -y

# Install Docker and Docker Compose
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# Install additional monitoring tools
sudo apt install -y \
    prometheus \
    prometheus-alertmanager \
    prometheus-node-exporter \
    grafana \
    net-tools \
    htop \
    iotop \
    nethogs

Quick Start Deployment

1. Deploy Monitoring Stack:

# Clone repository and navigate to monitoring
cd /path/to/foxhunt
cp docker-compose.monitoring.yml docker-compose.monitoring.production.yml

# Customize production monitoring configuration
nano docker-compose.monitoring.production.yml

# Deploy full monitoring stack
docker-compose -f docker-compose.monitoring.production.yml up -d

# Verify deployment
docker-compose -f docker-compose.monitoring.production.yml ps

2. Access Monitoring Services:

# Service endpoints
Grafana:      http://localhost:3000      (admin/admin)
Prometheus:   http://localhost:9090
AlertManager: http://localhost:9093
Loki:         http://localhost:3100
Tempo:        http://localhost:3200

📈 Prometheus Configuration

Production Configuration

Core Prometheus Config (/etc/prometheus/prometheus.yml):

global:
  scrape_interval: 5s        # High frequency for HFT
  evaluation_interval: 5s    # Fast alert evaluation
  scrape_timeout: 3s
  external_labels:
    cluster: 'foxhunt-production'
    environment: 'production'
    datacenter: 'primary'

# Alert rule files
rule_files:
  - "/etc/prometheus/rules/trading-critical.yml"
  - "/etc/prometheus/rules/trading-performance.yml"
  - "/etc/prometheus/rules/risk-management.yml"
  - "/etc/prometheus/rules/system-health.yml"
  - "/etc/prometheus/rules/compliance.yml"

# AlertManager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']
      timeout: 10s
      api_version: v2

# Scrape configurations optimized for HFT
scrape_configs:
  # ULTRA-HIGH PRIORITY - Trading Services (1s scrape)
  - job_name: 'foxhunt-trading'
    static_configs:
      - targets: ['trading-service:9001']
    scrape_interval: 1s
    scrape_timeout: 500ms
    metrics_path: /metrics
    honor_labels: true
    relabel_configs:
      - source_labels: [__address__]
        target_label: service_type
        replacement: trading
      - source_labels: [__address__]
        target_label: criticality
        replacement: ultra_high

  # HIGH PRIORITY - Risk Management (2s scrape)
  - job_name: 'foxhunt-risk'
    static_configs:
      - targets: ['risk-service:9002']
    scrape_interval: 2s
    scrape_timeout: 1s
    metrics_path: /metrics
    relabel_configs:
      - source_labels: [__address__]
        target_label: service_type
        replacement: risk
      - source_labels: [__address__]
        target_label: criticality
        replacement: high

  # TLI Interface (2s scrape)
  - job_name: 'foxhunt-tli'
    static_configs:
      - targets: ['tli-service:9003']
    scrape_interval: 2s
    scrape_timeout: 1s
    metrics_path: /metrics

  # ML Services (5s scrape)
  - job_name: 'foxhunt-ml'
    static_configs:
      - targets: ['ml-service:9004']
    scrape_interval: 5s
    scrape_timeout: 2s
    metrics_path: /metrics

  # Backtesting Service (10s scrape)
  - job_name: 'foxhunt-backtesting'
    static_configs:
      - targets: ['backtesting-service:9005']
    scrape_interval: 10s
    scrape_timeout: 5s
    metrics_path: /metrics

  # Infrastructure Services
  - job_name: 'postgres'
    static_configs:
      - targets: ['postgres-exporter:9187']
    scrape_interval: 15s

  - job_name: 'redis'
    static_configs:
      - targets: ['redis-exporter:9121']
    scrape_interval: 10s

  - job_name: 'influxdb'
    static_configs:
      - targets: ['influxdb:8086']
    scrape_interval: 30s
    metrics_path: /metrics

  # System monitoring
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
    scrape_interval: 10s

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']
    scrape_interval: 10s

# Storage configuration for HFT workloads
storage:
  tsdb:
    retention.time: 30d
    retention.size: 100GB
    wal-compression: true
    wal-segment-size: 256MB
    min-block-duration: 2h
    max-block-duration: 24h

# Query configuration
global:
  query_timeout: 2m
  query_max_concurrency: 20
  query_max_samples: 50000000

Critical Alert Rules

Trading Performance Alerts (/etc/prometheus/rules/trading-critical.yml):

groups:
  - name: trading.critical
    interval: 5s
    rules:
      # Ultra-low latency alerts
      - alert: OrderSubmissionLatencyHigh
        expr: histogram_quantile(0.99, rate(order_submission_duration_seconds_bucket[30s])) > 0.000050
        for: 10s
        labels:
          severity: critical
          component: trading
          team: trading
        annotations:
          summary: "Order submission latency exceeding 50μs"
          description: "P99 order submission latency is {{ $value }}s, exceeding 50μs threshold"
          impact: "High-frequency trading strategy performance degraded"
          action: "Check CPU affinity, network latency, and system resources"

      - alert: OrderFillRateLow
        expr: rate(orders_filled_total[1m]) / rate(orders_submitted_total[1m]) < 0.95
        for: 30s
        labels:
          severity: critical
          component: trading
          team: trading
        annotations:
          summary: "Order fill rate below 95%"
          description: "Order fill rate is {{ $value | humanizePercentage }}"
          impact: "Trading strategy execution quality degraded"

      - alert: TradingServiceDown
        expr: up{job="foxhunt-trading"} == 0
        for: 5s
        labels:
          severity: critical
          component: trading
          team: trading
        annotations:
          summary: "Trading service is down"
          description: "Trading service has been down for more than 5 seconds"
          impact: "All trading operations halted"
          action: "Immediate investigation required"

  - name: risk.critical
    interval: 5s
    rules:
      - alert: RiskLimitsBreached
        expr: current_position_risk > risk_limit_threshold
        for: 0s
        labels:
          severity: critical
          component: risk
          team: risk
        annotations:
          summary: "Risk limits breached"
          description: "Current position risk {{ $value }} exceeds limit"
          impact: "Potential significant financial loss"
          action: "Activate risk controls and position reduction"

      - alert: VaRExceeded
        expr: daily_var_utilization > 0.95
        for: 10s
        labels:
          severity: critical
          component: risk
          team: risk
        annotations:
          summary: "VaR utilization exceeding 95%"
          description: "Daily VaR utilization is {{ $value | humanizePercentage }}"
          impact: "Approaching daily risk limits"

      - alert: DrawdownExcessive
        expr: current_drawdown_pct > max_allowed_drawdown_pct
        for: 30s
        labels:
          severity: critical
          component: risk
          team: risk
        annotations:
          summary: "Drawdown exceeds maximum allowed"
          description: "Current drawdown {{ $value }}% exceeds {{ $labels.max_allowed_drawdown_pct }}%"
          impact: "Strategy performance significantly degraded"

  - name: system.critical
    interval: 10s
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 90
        for: 2m
        labels:
          severity: critical
          component: system
          team: operations
        annotations:
          summary: "High CPU usage detected"
          description: "CPU usage is {{ $value }}% on {{ $labels.instance }}"
          impact: "System performance degradation, potential latency increase"

      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes > 0.90
        for: 2m
        labels:
          severity: critical
          component: system
          team: operations
        annotations:
          summary: "High memory usage detected"
          description: "Memory usage is {{ $value | humanizePercentage }} on {{ $labels.instance }}"

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"}) < 0.10
        for: 5m
        labels:
          severity: warning
          component: system
          team: operations
        annotations:
          summary: "Low disk space"
          description: "Disk space usage is {{ $value | humanizePercentage }} on {{ $labels.instance }}"

  - name: performance.critical
    interval: 1s
    rules:
      - alert: NetworkLatencyHigh
        expr: histogram_quantile(0.99, rate(network_request_duration_seconds_bucket[30s])) > 0.001
        for: 15s
        labels:
          severity: critical
          component: network
          team: operations
        annotations:
          summary: "Network latency exceeding 1ms"
          description: "P99 network latency is {{ $value }}s"
          impact: "Trading latency significantly impacted"

      - alert: DatabaseQuerySlow
        expr: histogram_quantile(0.95, rate(database_query_duration_seconds_bucket[1m])) > 0.010
        for: 30s
        labels:
          severity: warning
          component: database
          team: operations
        annotations:
          summary: "Database queries slow"
          description: "P95 database query time is {{ $value }}s"

📊 Grafana Dashboard Configuration

Production Dashboards Setup

1. Deploy Pre-built Dashboards:

# Copy dashboard configurations
cp -r config/grafana/dashboards/* /var/lib/grafana/dashboards/

# Import dashboards via API
for dashboard in config/grafana/dashboards/*.json; do
    curl -X POST http://admin:admin@localhost:3000/api/dashboards/db \
        -H 'Content-Type: application/json' \
        -d @"$dashboard"
done

2. Core Dashboard Overview:

a) HFT Trading Performance Dashboard:

• Order Flow Metrics: Submission rate, fill rate, cancellation rate
• Latency Monitoring: P50, P95, P99 order latencies
• Market Data: Feed latency, throughput, gaps
• Position Tracking: Real-time positions, PnL, exposure
• Strategy Performance: Sharpe ratio, win rate, max drawdown

b) System Health Dashboard:

• CPU Metrics: Usage per core, CPU affinity effectiveness
• Memory Monitoring: Usage, allocation patterns, GC pressure
• Network Performance: Bandwidth, packet loss, latency
• Disk I/O: IOPS, latency, queue depth
• GPU Utilization: CUDA usage, memory allocation

c) Risk Management Dashboard:

• Real-time Risk Metrics: VaR, expected shortfall, exposure
• Position Limits: Current vs. maximum positions
• Drawdown Analysis: Current, maximum, recovery time
• Stress Testing: Scenario analysis results
• Compliance Status: Regulatory requirement adherence

d) Business Executive Dashboard:

• Daily P&L: Realized/unrealized gains/losses
• Trading Volume: Notional, share count, order count
• Performance Attribution: Strategy contribution analysis
• Cost Analysis: Trading costs, slippage, market impact
• Regulatory Compliance: Trade reporting status

Custom Dashboard JSON Configuration

Trading Performance Dashboard (trading-performance.json):

{
  "dashboard": {
    "id": null,
    "title": "Foxhunt HFT Trading Performance",
    "tags": ["foxhunt", "trading", "hft"],
    "timezone": "browser",
    "panels": [
      {
        "id": 1,
        "title": "Order Submission Latency (P99)",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.99, rate(order_submission_duration_seconds_bucket[30s]))",
            "legendFormat": "P99 Latency"
          }
        ],
        "yAxes": [
          {
            "label": "Latency (seconds)",
            "max": 0.0001,
            "min": 0
          }
        ],
        "alert": {
          "conditions": [
            {
              "evaluator": {
                "params": [0.00005],
                "type": "gt"
              },
              "operator": {
                "type": "and"
              },
              "query": {
                "params": ["A", "5m", "now"]
              },
              "reducer": {
                "params": [],
                "type": "last"
              },
              "type": "query"
            }
          ],
          "executionErrorState": "alerting",
          "for": "10s",
          "frequency": "1s",
          "handler": 1,
          "name": "High Order Latency",
          "noDataState": "no_data",
          "notifications": []
        },
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 0,
          "y": 0
        }
      },
      {
        "id": 2,
        "title": "Orders Per Second",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(orders_submitted_total[1m])",
            "legendFormat": "Submitted"
          },
          {
            "expr": "rate(orders_filled_total[1m])",
            "legendFormat": "Filled"
          },
          {
            "expr": "rate(orders_cancelled_total[1m])",
            "legendFormat": "Cancelled"
          }
        ],
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 12,
          "y": 0
        }
      }
    ],
    "time": {
      "from": "now-1h",
      "to": "now"
    },
    "refresh": "1s"
  }
}

🚨 AlertManager Configuration

Production Alert Configuration

AlertManager Config (/etc/alertmanager/alertmanager.yml):

global:
  smtp_smarthost: 'smtp.company.com:587'
  smtp_from: 'foxhunt-alerts@company.com'
  smtp_require_tls: true
  slack_api_url: 'YOUR_SLACK_WEBHOOK_URL'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

# Alert routing strategy
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 5s
  group_interval: 10s
  repeat_interval: 2m
  receiver: 'default'
  
  routes:
    # CRITICAL TRADING ALERTS - Immediate escalation
    - match:
        severity: critical
        component: trading
      receiver: 'trading-critical'
      group_wait: 0s
      group_interval: 30s
      repeat_interval: 1m
      continue: true

    # CRITICAL RISK ALERTS - Immediate escalation
    - match:
        severity: critical
        component: risk
      receiver: 'risk-critical'
      group_wait: 0s
      group_interval: 30s
      repeat_interval: 1m
      continue: true

    # SYSTEM CRITICAL - Operations team
    - match:
        severity: critical
        component: system
      receiver: 'system-critical'
      group_wait: 10s
      group_interval: 1m
      repeat_interval: 5m

    # WARNING ALERTS - Standard routing
    - match:
        severity: warning
      receiver: 'warning-alerts'
      group_wait: 2m
      group_interval: 5m
      repeat_interval: 30m

# Alert receivers with escalation
receivers:
  # Default fallback
  - name: 'default'
    slack_configs:
      - channel: '#general-alerts'
        title: 'Foxhunt Alert'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

  # Critical trading alerts with multi-channel escalation
  - name: 'trading-critical'
    # Immediate Slack notification
    slack_configs:
      - channel: '#trading-critical'
        title: '🚨 CRITICAL TRADING ALERT'
        text: |
          Alert: {{ range .Alerts }}{{ .Annotations.summary }}{{ end }}
          Impact: {{ range .Alerts }}{{ .Annotations.impact }}{{ end }}
          Action: {{ range .Alerts }}{{ .Annotations.action }}{{ end }}
        send_resolved: true
        color: 'danger'
    
    # Email to trading team
    email_configs:
      - to: 'trading-team@company.com'
        subject: '🚨 CRITICAL: Foxhunt Trading Alert'
        body: |
          CRITICAL TRADING ALERT
          
          Alert: {{ range .Alerts }}{{ .Annotations.summary }}{{ end }}
          Description: {{ range .Alerts }}{{ .Annotations.description }}{{ end }}
          Impact: {{ range .Alerts }}{{ .Annotations.impact }}{{ end }}
          Required Action: {{ range .Alerts }}{{ .Annotations.action }}{{ end }}
          
          Time: {{ range .Alerts }}{{ .StartsAt }}{{ end }}
          
          Dashboard: http://grafana:3000/d/trading-performance
          
        headers:
          Priority: 'urgent'
          Importance: 'high'
    
    # PagerDuty for on-call escalation
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_SERVICE_KEY'
        description: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
        details:
          alert: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
          impact: '{{ range .Alerts }}{{ .Annotations.impact }}{{ end }}'
          action: '{{ range .Alerts }}{{ .Annotations.action }}{{ end }}'
        client: 'Foxhunt AlertManager'
        client_url: 'http://alertmanager:9093'

  # Critical risk alerts
  - name: 'risk-critical'
    slack_configs:
      - channel: '#risk-critical'
        title: '🚨 CRITICAL RISK ALERT'
        text: |
          Alert: {{ range .Alerts }}{{ .Annotations.summary }}{{ end }}
          Impact: {{ range .Alerts }}{{ .Annotations.impact }}{{ end }}
        color: 'danger'
    
    email_configs:
      - to: 'risk-team@company.com,cro@company.com'
        subject: '🚨 CRITICAL: Foxhunt Risk Alert'
        body: |
          CRITICAL RISK ALERT
          
          Alert: {{ range .Alerts }}{{ .Annotations.summary }}{{ end }}
          Description: {{ range .Alerts }}{{ .Annotations.description }}{{ end }}
          Impact: {{ range .Alerts }}{{ .Annotations.impact }}{{ end }}
          
          Immediate risk management action required.
          
          Dashboard: http://grafana:3000/d/risk-management

  # System critical alerts
  - name: 'system-critical'
    slack_configs:
      - channel: '#ops-critical'
        title: '⚠️ CRITICAL SYSTEM ALERT'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
        color: 'warning'
    
    email_configs:
      - to: 'ops-team@company.com'
        subject: '⚠️ CRITICAL: Foxhunt System Alert'

  # Warning alerts
  - name: 'warning-alerts'
    slack_configs:
      - channel: '#monitoring'
        title: 'Foxhunt Warning'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
        color: 'warning'

# Inhibition rules to prevent alert storms
inhibit_rules:
  # Inhibit all other alerts if trading service is completely down
  - source_match:
      alertname: TradingServiceDown
    target_match_re:
      component: trading
    equal: ['instance']

  # Inhibit individual service alerts if the whole node is down
  - source_match:
      alertname: NodeDown
    target_match_re:
      alertname: (ServiceDown|HighLatency|.*Error)
    equal: ['instance']

  # Inhibit memory alerts if disk is full (likely log/data overflow)
  - source_match:
      alertname: DiskSpaceLow
    target_match:
      alertname: HighMemoryUsage
    equal: ['instance']

📋 Daily Monitoring Operations

Morning Checklist (Pre-Market)

Daily Monitoring Startup Script (morning-monitoring-check.sh):

#!/bin/bash
# Daily Morning Monitoring Health Check

echo "=== Foxhunt Monitoring Health Check - $(date) ==="

# 1. Verify all monitoring services are running
echo "1. Checking monitoring services..."
services=("prometheus" "grafana" "alertmanager" "loki" "tempo")
for service in "${services[@]}"; do
    if docker ps | grep -q "foxhunt-$service"; then
        echo "   ✅ $service: Running"
    else
        echo "   ❌ $service: DOWN - CRITICAL"
        exit 1
    fi
done

# 2. Check Prometheus targets
echo "2. Checking Prometheus targets..."
curl -s http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | select(.health != "up") | .labels.job + ": " + .health' > /tmp/down_targets.txt
if [ -s /tmp/down_targets.txt ]; then
    echo "   ❌ Down targets detected:"
    cat /tmp/down_targets.txt
    exit 1
else
    echo "   ✅ All targets healthy"
fi

# 3. Verify critical metrics are being collected
echo "3. Verifying critical metrics..."
critical_metrics=(
    "order_submission_duration_seconds"
    "up{job=\"foxhunt-trading\"}"
    "daily_pnl_usd"
    "current_position_risk"
)

for metric in "${critical_metrics[@]}"; do
    result=$(curl -s "http://localhost:9090/api/v1/query?query=$metric" | jq -r '.data.result | length')
    if [ "$result" -gt 0 ]; then
        echo "   ✅ $metric: Data available"
    else
        echo "   ❌ $metric: NO DATA - CRITICAL"
        exit 1
    fi
done

# 4. Check AlertManager status
echo "4. Checking AlertManager..."
alerts=$(curl -s http://localhost:9093/api/v1/alerts | jq -r '.data[] | select(.status.state == "firing") | .labels.alertname')
if [ -n "$alerts" ]; then
    echo "   ⚠️  Active alerts:"
    echo "$alerts" | while read alert; do
        echo "      - $alert"
    done
else
    echo "   ✅ No active alerts"
fi

# 5. Verify Grafana dashboards
echo "5. Checking Grafana dashboards..."
dashboard_count=$(curl -s http://admin:admin@localhost:3000/api/search | jq '. | length')
if [ "$dashboard_count" -ge 6 ]; then
    echo "   ✅ Grafana: $dashboard_count dashboards loaded"
else
    echo "   ❌ Grafana: Missing dashboards ($dashboard_count found)"
fi

# 6. Check data retention and storage
echo "6. Checking storage and retention..."
prometheus_storage=$(df -h /var/lib/prometheus | awk 'NR==2 {print $5}' | sed 's/%//')
if [ "$prometheus_storage" -lt 80 ]; then
    echo "   ✅ Prometheus storage: ${prometheus_storage}% used"
else
    echo "   ⚠️  Prometheus storage: ${prometheus_storage}% used - Consider cleanup"
fi

echo "=== Morning Health Check Complete ==="
echo "Dashboard: http://localhost:3000/d/foxhunt-overview"
echo "Prometheus: http://localhost:9090"
echo "AlertManager: http://localhost:9093"

Real-Time Monitoring Operations

1. Critical Metrics Dashboard URLs:

# Quick access URLs for operations team
GRAFANA_BASE="http://localhost:3000"

# Primary monitoring dashboards
echo "Real-time Trading Performance: ${GRAFANA_BASE}/d/trading-performance"
echo "System Health Overview: ${GRAFANA_BASE}/d/system-health" 
echo "Risk Management: ${GRAFANA_BASE}/d/risk-management"
echo "HFT Latency Monitor: ${GRAFANA_BASE}/d/hft-latency-monitor"
echo "Business Executive View: ${GRAFANA_BASE}/d/business-executive"
echo "Compliance Audit: ${GRAFANA_BASE}/d/compliance-audit"

2. Key Metrics to Monitor Throughout Day:

# Ultra-critical metrics (1-second monitoring)
- order_submission_latency_p99 < 50μs
- up{job="foxhunt-trading"} == 1
- current_position_risk < risk_limit_threshold

# High-priority metrics (5-second monitoring)  
- fill_rate_percentage > 95%
- daily_pnl_usd (tracking)
- system_cpu_usage < 80%
- system_memory_usage < 85%

# Standard metrics (30-second monitoring)
- network_latency_p95 < 1ms
- database_query_duration_p95 < 10ms
- gpu_utilization_percentage
- disk_io_latency_p99

3. Alert Response Procedures:

Critical Trading Alert Response:

#!/bin/bash
# critical-trading-alert-response.sh

echo "CRITICAL TRADING ALERT RECEIVED - $(date)"
echo "Performing immediate diagnostics..."

# 1. Check service status
curl -f http://localhost:50051/health || echo "❌ Trading service health check failed"

# 2. Check current latency
current_latency=$(curl -s http://localhost:9090/api/v1/query?query=histogram_quantile\(0.99,rate\(order_submission_duration_seconds_bucket\[30s\]\)\) | jq -r '.data.result[0].value[1]')
echo "Current P99 latency: ${current_latency}s"

# 3. Check system resources
echo "System resources:"
top -bn1 | head -20
free -h
iostat -x 1 1

# 4. Check network connectivity to exchanges
echo "Exchange connectivity:"
ping -c 3 ib-gateway.internal
ping -c 3 fix.icmarkets.com

# 5. Check for obvious issues
echo "Recent errors:"
docker logs foxhunt-trading-service --tail=50 | grep -i error

echo "DIAGNOSTICS COMPLETE - Manual investigation required"

🛠️ Troubleshooting Common Issues

High Latency Issues

1. Diagnose Latency Spikes:

# Check CPU frequency scaling
cat /proc/cpuinfo | grep MHz
sudo cpupower frequency-info

# Verify CPU affinity is working
for pid in $(pgrep -f foxhunt-trading); do
    taskset -p $pid
done

# Check for network issues
ss -tulpn | grep :50051
netstat -i
sar -n DEV 1 5

# Check memory allocation
cat /proc/meminfo | grep -E "(MemAvailable|Hugepages)"
numactl --show

2. Fix Common Latency Issues:

# Reset CPU governor to performance
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Disable CPU idle states
sudo cpupower idle-set -D 0

# Restart services with proper affinity
docker-compose restart foxhunt-trading-service

# Clear system caches if memory pressure detected
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

Monitoring Service Issues

1. Prometheus Issues:

# Check Prometheus storage
df -h /var/lib/prometheus
du -sh /var/lib/prometheus/*

# Check configuration syntax
docker exec foxhunt-prometheus promtool check config /etc/prometheus/prometheus.yml

# Check rule files
docker exec foxhunt-prometheus promtool check rules /etc/prometheus/rules/*.yml

# Restart Prometheus if needed
docker-compose restart foxhunt-prometheus

2. Grafana Issues:

# Check Grafana logs
docker logs foxhunt-grafana --tail=100

# Test database connectivity
docker exec foxhunt-grafana grafana-cli admin reset-admin-password admin

# Reload dashboards
for dashboard in config/grafana/dashboards/*.json; do
    curl -X POST http://admin:admin@localhost:3000/api/dashboards/db \
        -H 'Content-Type: application/json' \
        -d @"$dashboard"
done

3. AlertManager Issues:

# Check AlertManager configuration
docker exec foxhunt-alertmanager amtool config show

# Test alert routing
docker exec foxhunt-alertmanager amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml

# Silence alerts temporarily
curl -X POST http://localhost:9093/api/v1/silences \
    -H 'Content-Type: application/json' \
    -d '{
        "matchers": [{"name": "alertname", "value": "TestAlert"}],
        "startsAt": "2023-01-01T00:00:00Z",
        "endsAt": "2023-01-01T01:00:00Z",
        "comment": "Temporary silence for maintenance"
    }'

📊 Performance Optimization

Monitoring Stack Optimization

1. Prometheus Optimization:

# /etc/prometheus/prometheus.yml optimizations
global:
  scrape_interval: 5s          # Balance between data resolution and overhead
  evaluation_interval: 5s      # Fast alert evaluation
  scrape_timeout: 3s           # Prevent hanging scrapes

# Storage optimizations
storage:
  tsdb:
    retention.time: 30d        # Adjust based on storage capacity
    retention.size: 100GB
    wal-compression: true      # Reduce storage usage
    wal-segment-size: 256MB    # Larger segments for better performance
    min-block-duration: 2h     # Larger blocks for better query performance
    max-block-duration: 24h

2. Query Optimization:

# Enable query logging
docker exec foxhunt-prometheus \
    kill -HUP $(pgrep prometheus)

# Monitor slow queries
tail -f /var/lib/prometheus/query.log | grep -E "slow|timeout"

# Optimize expensive queries using recording rules
cat > /etc/prometheus/rules/recording-rules.yml << EOF
groups:
  - name: performance.rules
    interval: 10s
    rules:
      - record: trading:latency_p99_5m
        expr: histogram_quantile(0.99, rate(order_submission_duration_seconds_bucket[5m]))
      
      - record: trading:order_rate_1m  
        expr: rate(orders_submitted_total[1m])
        
      - record: system:cpu_usage_5m
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
EOF

3. Grafana Performance Optimization:

# Grafana configuration optimizations
cat > /etc/grafana/grafana.ini << EOF
[database]
# Use PostgreSQL for better performance at scale
type = postgres
host = postgres:5432
name = grafana
user = grafana
password = ${GRAFANA_DB_PASSWORD}

[server]
# Performance settings
enable_gzip = true
router_logging = false

[analytics]
reporting_enabled = false
check_for_updates = false

[metrics]
enabled = true
interval_seconds = 10

[caching]
enabled = true
EOF

# Restart Grafana with optimizations
docker-compose restart foxhunt-grafana

📈 Advanced Analytics

Custom Metric Calculations

HFT-Specific Metrics:

# Sharpe Ratio calculation (rolling 24h)
sharpe_ratio_24h = (avg_over_time(daily_returns_pct[24h]) - risk_free_rate) / stddev_over_time(daily_returns_pct[24h])

# Maximum Adverse Excursion (MAE)
max_adverse_excursion = max_over_time((entry_price - min_price_during_trade) / entry_price[1h])

# Market Impact calculation
market_impact_bps = (execution_price - arrival_price) / arrival_price * 10000

# Slippage analysis
slippage_bps = (fill_price - limit_price) / limit_price * 10000

# Fill ratio by time of day
fill_ratio_by_hour = rate(orders_filled_total[1h]) / rate(orders_submitted_total[1h]) by (hour)

Compliance Reporting

Automated Compliance Metrics:

# Best execution monitoring
best_execution_compliance = (
    orders_routed_to_best_venue_total / orders_submitted_total
) by (symbol, venue)

# Transaction reporting completeness
transaction_reporting_coverage = (
    reported_transactions_total / executed_transactions_total
)

# MiFID II compliance score
mifid_ii_compliance_score = (
    best_execution_compliance * 0.4 +
    transaction_reporting_coverage * 0.3 +
    trade_surveillance_coverage * 0.3
)

# SOX compliance monitoring
sox_audit_trail_completeness = (
    audit_events_logged_total / business_events_total
)

🔐 Security Monitoring

Security-Specific Alerts

Security Alert Rules:

groups:
  - name: security.critical
    rules:
      - alert: UnauthorizedAccess
        expr: rate(http_requests_total{status=~"401|403"}[5m]) > 10
        labels:
          severity: critical
          component: security
        annotations:
          summary: "High rate of unauthorized access attempts"

      - alert: AnomalousLoginPattern
        expr: |
          (
            rate(login_attempts_total[1h]) 
            > 
            avg_over_time(login_attempts_total[24h:1h]) + 3 * stddev_over_time(login_attempts_total[24h:1h])
          )
        labels:
          severity: warning
          component: security
        annotations:
          summary: "Anomalous login pattern detected"

      - alert: PrivilegeEscalation
        expr: rate(privilege_escalation_events_total[5m]) > 0
        labels:
          severity: critical
          component: security
        annotations:
          summary: "Privilege escalation attempt detected"

---

Documentation Status: Production-ready comprehensive monitoring guide Last Updated: 2025-09-24 Version: Production v1.0.0 Covers: Prometheus, Grafana, AlertManager, Security, Operations