foxhunt/TROUBLESHOOTING.md

Foxhunt HFT Trading System - Comprehensive Troubleshooting Guide

🚀 Overview

This comprehensive troubleshooting guide provides solutions for common issues, debugging procedures, and emergency response protocols for the Foxhunt HFT Trading System. The guide is organized by system component and severity level to enable rapid issue resolution in production environments.

🚨 Emergency Response Procedures

CRITICAL: Trading System Down

Immediate Actions (0-2 minutes):

#!/bin/bash
# emergency-response.sh - Execute immediately for trading outages

echo "🚨 EMERGENCY: Trading system outage detected at $(date)"

# 1. IMMEDIATE SAFETY - Activate kill switch
curl -X POST http://localhost:50052/api/v1/emergency/kill-switch -d '{"reason":"system_outage","operator":"emergency_response"}'

# 2. Check system status
echo "Checking system status..."
docker-compose ps | grep -E "(trading|risk|ml)"

# 3. Check critical services health
services=("trading-service" "risk-service" "tli" "postgres" "redis")
for service in "${services[@]}"; do
    if curl -f -m 5 "http://localhost:50051/health" 2>/dev/null; then
        echo "✅ $service: Healthy"
    else
        echo "❌ $service: DOWN - CRITICAL"
    fi
done

# 4. Check system resources
echo "System resources:"
echo "CPU: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | awk -F% '{print $1}')"
echo "Memory: $(free | grep Mem | awk '{printf("%.2f%%\n", $3/$2 * 100.0)}')"
echo "Disk: $(df -h / | awk 'NR==2 {print $5}')"

# 5. Emergency restart if needed
read -p "Attempt emergency restart? (y/N): " -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]; then
    echo "Performing emergency restart..."
    docker-compose restart trading-service risk-service
    sleep 30
    
    # Re-check after restart
    if curl -f "http://localhost:50051/health"; then
        echo "✅ System restored"
        # Deactivate kill switch
        curl -X DELETE http://localhost:50052/api/v1/emergency/kill-switch
    else
        echo "❌ System still down - Escalate to Level 2 support"
    fi
fi

echo "Emergency response complete - Manual investigation required"

CRITICAL: Risk Limits Breached

Risk Emergency Protocol:

#!/bin/bash
# risk-emergency.sh - Risk limit breach response

echo "🚨 RISK EMERGENCY: Limits breached at $(date)"

# 1. Get current risk metrics
current_var=$(curl -s http://localhost:50052/api/v1/risk/var/current | jq -r '.current_var')
position_risk=$(curl -s http://localhost:50052/api/v1/risk/positions/aggregate | jq -r '.total_risk')
drawdown=$(curl -s http://localhost:50051/api/v1/trading/performance/drawdown | jq -r '.current_drawdown_pct')

echo "Current VaR: $current_var"
echo "Position Risk: $position_risk"
echo "Drawdown: $drawdown%"

# 2. Risk assessment
if (( $(echo "$drawdown > 10" | bc -l) )); then
    echo "SEVERE: Drawdown exceeds 10% - Immediate action required"
    
    # Emergency position reduction
    curl -X POST http://localhost:50051/api/v1/trading/emergency/reduce-positions \
        -d '{"reduction_percentage": 50, "reason": "risk_breach"}'
    
    # Notify risk management
    curl -X POST http://localhost:9093/api/v1/alerts \
        -H 'Content-Type: application/json' \
        -d '{
            "alerts": [{
                "labels": {
                    "alertname": "EmergencyRiskBreach",
                    "severity": "critical"
                },
                "annotations": {
                    "summary": "Emergency risk breach - immediate action taken"
                }
            }]
        }'
fi

# 3. Generate emergency risk report
./scripts/generate-emergency-risk-report.sh

echo "Risk emergency protocol completed"

🔧 System Component Troubleshooting

Trading Service Issues

High Order Latency (>50μs)

Diagnosis:

# Check current latency metrics
curl -s http://localhost:9090/api/v1/query?query=histogram_quantile\(0.99,rate\(order_submission_duration_seconds_bucket\[30s\]\)\) | jq -r '.data.result[0].value[1]'

# Check CPU affinity
for pid in $(pgrep -f trading-service); do
    echo "PID $pid CPU affinity:"
    taskset -p $pid
done

# Check CPU frequency scaling
cat /proc/cpuinfo | grep MHz | head -4
sudo cpupower frequency-info

# Check for CPU throttling
dmesg | grep -i "cpu.*throttled" | tail -5

# Network latency to exchanges
ping -c 5 ib-gateway.internal
ping -c 5 fix.icmarkets.com

# Check system interrupts
cat /proc/interrupts | grep -E "(eth0|timer)"

Solutions:

# Fix 1: Reset CPU governor
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Fix 2: Disable CPU idle states  
sudo cpupower idle-set -D 0

# Fix 3: Set CPU affinity for trading cores
docker exec foxhunt-trading-service taskset -cp 2-5 1

# Fix 4: Increase network buffer sizes
echo 'net.core.rmem_max = 268435456' | sudo tee -a /etc/sysctl.conf
echo 'net.core.wmem_max = 268435456' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Fix 5: Restart trading service with optimizations
docker-compose restart foxhunt-trading-service

# Verify fix
sleep 30
current_latency=$(curl -s http://localhost:9090/api/v1/query?query=histogram_quantile\(0.99,rate\(order_submission_duration_seconds_bucket\[30s\]\)\) | jq -r '.data.result[0].value[1]')
echo "Current latency after fixes: ${current_latency}s"

Order Fill Rate Low (<95%)

Diagnosis:

# Check fill rate metrics
fill_rate=$(curl -s http://localhost:9090/api/v1/query?query=rate\(orders_filled_total\[1m\]\)/rate\(orders_submitted_total\[1m\]\) | jq -r '.data.result[0].value[1]')
echo "Current fill rate: $(echo "$fill_rate * 100" | bc)%"

# Check order routing
curl -s http://localhost:50051/api/v1/trading/routing/stats | jq '.'

# Check broker connectivity
curl -f http://localhost:50051/api/v1/brokers/ib/health
curl -f http://localhost:50051/api/v1/brokers/icmarkets/health

# Check market conditions
curl -s http://localhost:50051/api/v1/market-data/status | jq '.feeds[] | {symbol, latency, status}'

Solutions:

# Fix 1: Update order routing algorithm
curl -X POST http://localhost:50051/api/v1/trading/routing/optimize \
    -d '{"algorithm": "intelligent", "consider_fill_rate": true}'

# Fix 2: Adjust order pricing strategy  
curl -X POST http://localhost:50051/api/v1/trading/strategy/pricing \
    -d '{"mode": "aggressive", "spread_tolerance": 0.02}'

# Fix 3: Check and restart broker connections
docker exec foxhunt-trading-service ./scripts/restart-broker-connections.sh

# Fix 4: Enable additional liquidity venues
curl -X POST http://localhost:50051/api/v1/trading/venues/enable \
    -d '{"venues": ["EDGX", "BZX", "ARCA"]}'

Trading Service Won't Start

Diagnosis:

# Check Docker container status
docker ps -a | grep trading-service

# Check logs for startup errors
docker logs foxhunt-trading-service --tail=100

# Check configuration validity
docker exec foxhunt-trading-service ./trading_service --check-config

# Check database connectivity
docker exec foxhunt-trading-service pg_isready -h postgres -p 5432

# Check port conflicts
netstat -tulpn | grep :50051
lsof -i :50051

Solutions:

# Fix 1: Database connection issue
export DATABASE_URL="postgresql://foxhunt_user:${POSTGRES_PASSWORD}@postgres:5432/foxhunt_production"
docker-compose restart postgres
sleep 20
docker-compose restart foxhunt-trading-service

# Fix 2: Port conflict resolution
docker stop $(docker ps -q --filter "publish=50051")
docker-compose up -d foxhunt-trading-service

# Fix 3: Configuration file corruption
docker exec foxhunt-trading-service cp /etc/foxhunt/config/trading.toml.backup /etc/foxhunt/config/trading.toml
docker-compose restart foxhunt-trading-service

# Fix 4: Memory/resource constraints
docker update --memory=8g --cpus="4" foxhunt-trading-service
docker-compose restart foxhunt-trading-service

# Fix 5: Complete service rebuild if needed
docker-compose down foxhunt-trading-service
docker-compose build foxhunt-trading-service
docker-compose up -d foxhunt-trading-service

Database Issues

PostgreSQL Connection Pool Exhausted

Diagnosis:

# Check active connections
docker exec foxhunt-postgres-primary psql -U postgres -c "
    SELECT count(*) as active_connections, state 
    FROM pg_stat_activity 
    GROUP BY state;"

# Check connection pool configuration
docker exec foxhunt-trading-service cat /etc/foxhunt/config/database.toml | grep -A 5 "pool"

# Check long-running queries
docker exec foxhunt-postgres-primary psql -U postgres -c "
    SELECT pid, now() - pg_stat_activity.query_start AS duration, query 
    FROM pg_stat_activity 
    WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes';"

Solutions:

# Fix 1: Kill long-running queries
docker exec foxhunt-postgres-primary psql -U postgres -c "
    SELECT pg_terminate_backend(pid) 
    FROM pg_stat_activity 
    WHERE (now() - pg_stat_activity.query_start) > interval '10 minutes';"

# Fix 2: Increase connection pool size
docker exec foxhunt-trading-service sed -i 's/pool_size = 20/pool_size = 50/' /etc/foxhunt/config/database.toml
docker-compose restart foxhunt-trading-service

# Fix 3: Optimize PostgreSQL configuration
docker exec foxhunt-postgres-primary psql -U postgres -c "
    ALTER SYSTEM SET max_connections = 200;
    ALTER SYSTEM SET shared_buffers = '4GB';
    ALTER SYSTEM SET effective_cache_size = '12GB';
    SELECT pg_reload_conf();"

# Fix 4: Connection leak detection
docker logs foxhunt-trading-service | grep -i "connection" | tail -20

Redis Cluster Node Down

Diagnosis:

# Check cluster status
docker exec foxhunt-redis-node-1 redis-cli --cluster info

# Check individual node status
for i in {1..6}; do
    echo "Node $i:"
    docker exec foxhunt-redis-node-$i redis-cli ping 2>/dev/null || echo "DOWN"
done

# Check cluster configuration
docker exec foxhunt-redis-node-1 redis-cli --cluster nodes

Solutions:

# Fix 1: Restart failed node
failed_node=$(docker exec foxhunt-redis-node-1 redis-cli --cluster nodes | grep "fail" | cut -d' ' -f2)
if [ -n "$failed_node" ]; then
    docker-compose restart foxhunt-redis-node-${failed_node: -1}
    sleep 10
    docker exec foxhunt-redis-node-1 redis-cli --cluster fix redis-node-1:7001
fi

# Fix 2: Remove and re-add failed node
# docker exec foxhunt-redis-node-1 redis-cli --cluster del-node redis-node-1:7001 ${failed_node}
# docker exec foxhunt-redis-node-1 redis-cli --cluster add-node redis-node-X:700X redis-node-1:7001

# Fix 3: Complete cluster reset (LAST RESORT)
# ./scripts/reset-redis-cluster.sh

InfluxDB Write Timeouts

Diagnosis:

# Check InfluxDB status
curl -f http://localhost:8086/health

# Check write performance
docker logs foxhunt-influxdb --tail=100 | grep -i "timeout\|error"

# Check disk I/O
iostat -x 1 5 | grep -E "(Device|influxdb|nvme)"

# Check memory usage
docker stats foxhunt-influxdb --no-stream

Solutions:

# Fix 1: Increase write timeout
curl -X POST http://localhost:8086/api/v2/config \
    -H 'Authorization: Token $INFLUX_TOKEN' \
    -d '{"storage-write-timeout": "30s"}'

# Fix 2: Optimize batch size
docker exec foxhunt-trading-service sed -i 's/batch_size = 1000/batch_size = 5000/' /etc/foxhunt/config/influxdb.toml
docker-compose restart foxhunt-trading-service

# Fix 3: Add more memory to InfluxDB
docker update --memory=8g foxhunt-influxdb
docker-compose restart foxhunt-influxdb

# Fix 4: Enable compression
curl -X POST http://localhost:8086/api/v2/config \
    -H 'Authorization: Token $INFLUX_TOKEN' \
    -d '{"storage-series-file-max-concurrent-compactions": 4}'

Performance Issues

High CPU Usage (>90%)

Diagnosis:

# Identify top CPU consumers
top -bn2 -d1 | grep -E "foxhunt|trading" | head -10

# Check CPU per core usage
mpstat -P ALL 1 3

# Check for CPU-intensive processes
ps aux --sort=-%cpu | head -15

# Check for CPU throttling
dmesg | grep -i "cpu.*throttled" | tail -10

# Check context switches
vmstat 1 5

Solutions:

# Fix 1: Scale CPU-intensive services
docker update --cpus="6" foxhunt-ml-service
docker update --cpus="4" foxhunt-trading-service

# Fix 2: Optimize CPU affinity
./scripts/set-cpu-affinity.sh

# Fix 3: Reduce monitoring frequency temporarily
docker exec foxhunt-prometheus sed -i 's/scrape_interval: 1s/scrape_interval: 5s/' /etc/prometheus/prometheus.yml
docker-compose restart foxhunt-prometheus

# Fix 4: Check for runaway processes
pkill -f "stress\|cpu-burn\|yes"

# Fix 5: Enable CPU frequency scaling optimization
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Memory Leaks

Diagnosis:

# Check memory usage trends
docker stats --no-stream | grep foxhunt

# Check for memory leaks in specific services
docker exec foxhunt-trading-service cat /proc/self/status | grep -E "(VmSize|VmRSS|VmHWM)"

# System memory analysis
free -h
cat /proc/meminfo | grep -E "(MemAvailable|MemFree|Cached|Buffers)"

# Check for OOM killer activity
dmesg | grep -i "killed process"
journalctl -u docker | grep -i "oom"

Solutions:

# Fix 1: Restart services showing memory growth
docker-compose restart foxhunt-ml-service  # Usually the main culprit
sleep 30
docker stats --no-stream | grep foxhunt

# Fix 2: Increase memory limits temporarily
docker update --memory=16g foxhunt-ml-service
docker update --memory=8g foxhunt-trading-service

# Fix 3: Force garbage collection (if applicable)
curl -X POST http://localhost:50053/api/v1/ml/gc

# Fix 4: Clear system caches
sync
echo 3 | sudo tee /proc/sys/vm/drop_caches

# Fix 5: Enable memory profiling
docker exec foxhunt-trading-service kill -USR1 $(pgrep trading_service)

Disk I/O Bottleneck

Diagnosis:

# Check disk I/O statistics
iostat -x 1 5

# Check disk space usage
df -h
du -sh /var/lib/docker/volumes/* | sort -hr | head -10

# Check I/O wait
top -bn1 | grep "wa"
vmstat 1 5

# Check which processes are causing I/O
iotop -ao1 -d1 | head -20

Solutions:

# Fix 1: Move high-I/O operations to faster storage
docker volume create --driver local --opt type=tmpfs --opt device=tmpfs foxhunt-temp
docker run -v foxhunt-temp:/tmp foxhunt/trading-service

# Fix 2: Optimize database I/O
docker exec foxhunt-postgres-primary psql -U postgres -c "
    ALTER SYSTEM SET wal_buffers = '16MB';
    ALTER SYSTEM SET checkpoint_completion_target = 0.7;
    SELECT pg_reload_conf();"

# Fix 3: Clean up old log files
docker exec foxhunt-trading-service find /var/log -name "*.log" -mtime +7 -delete
docker system prune -f

# Fix 4: Adjust I/O scheduler
echo mq-deadline | sudo tee /sys/block/nvme0n1/queue/scheduler

# Fix 5: Increase I/O priority for critical services
docker exec foxhunt-trading-service ionice -c 1 -n 4 -p $(pgrep trading_service)

Network Issues

High Network Latency

Diagnosis:

# Check network latency to exchanges
ping -c 10 ib-gateway.internal | tail -1
ping -c 10 fix.icmarkets.com | tail -1

# Check network interface statistics
cat /proc/net/dev | grep -E "(eth0|enp)"
ethtool eth0 | grep -E "(Speed|Link)"

# Check for packet loss
ping -c 100 -i 0.2 ib-gateway.internal | grep -E "(packet loss|rtt)"

# Check network buffer utilization
ss -tuln | grep :50051
netstat -s | grep -E "(retrans|drop|error)"

Solutions:

# Fix 1: Optimize network buffers
echo 'net.core.rmem_max = 536870912' | sudo tee -a /etc/sysctl.conf
echo 'net.core.wmem_max = 536870912' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 131072 536870912' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Fix 2: Disable TCP features for lower latency
echo 'net.ipv4.tcp_timestamps = 0' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_sack = 0' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Fix 3: Set network interface to performance mode
ethtool -C eth0 adaptive-rx off adaptive-tx off
ethtool -G eth0 rx 4096 tx 4096

# Fix 4: Use dedicated network namespace
ip netns add trading
ip netns exec trading ip link set lo up
ip link set eth0 netns trading

# Fix 5: Restart networking services
sudo systemctl restart networking
docker-compose restart foxhunt-trading-service

ML/GPU Issues

CUDA Out of Memory

Diagnosis:

# Check GPU memory usage
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
docker exec foxhunt-ml-service nvidia-smi

# Check CUDA version compatibility
docker exec foxhunt-ml-service nvcc --version
docker exec foxhunt-ml-service python -c "import torch; print(torch.cuda.is_available())"

# Check ML service logs
docker logs foxhunt-ml-service --tail=100 | grep -i "cuda\|memory\|gpu"

Solutions:

# Fix 1: Clear GPU memory cache
docker exec foxhunt-ml-service python -c "
import torch
torch.cuda.empty_cache()
print('GPU cache cleared')
"

# Fix 2: Reduce batch size
curl -X POST http://localhost:50053/api/v1/ml/config \
    -d '{"batch_size": 64, "gradient_accumulation_steps": 2}'

# Fix 3: Enable gradient checkpointing
curl -X POST http://localhost:50053/api/v1/ml/config \
    -d '{"gradient_checkpointing": true, "mixed_precision": true}'

# Fix 4: Restart ML service
docker-compose restart foxhunt-ml-service
sleep 60
nvidia-smi  # Verify GPU memory is freed

# Fix 5: Use model parallelism
curl -X POST http://localhost:50053/api/v1/ml/config \
    -d '{"model_parallel": true, "tensor_parallel_size": 2}'

Model Inference Timeouts

Diagnosis:

# Check inference latency
curl -s http://localhost:9090/api/v1/query?query=histogram_quantile\(0.95,rate\(ml_inference_duration_seconds_bucket\[1m\]\)\)

# Check model loading status
curl -s http://localhost:50053/api/v1/ml/models/status | jq '.'

# Check GPU utilization
nvidia-smi dmon -s u -c 10

# Check model cache
docker exec foxhunt-ml-service ls -la /var/lib/foxhunt/ml/models/

Solutions:

# Fix 1: Warm up models
curl -X POST http://localhost:50053/api/v1/ml/models/warmup

# Fix 2: Enable model caching
curl -X POST http://localhost:50053/api/v1/ml/config \
    -d '{"enable_model_cache": true, "cache_size_gb": 4}'

# Fix 3: Use TensorRT optimization
curl -X POST http://localhost:50053/api/v1/ml/optimize \
    -d '{"engine": "tensorrt", "precision": "fp16"}'

# Fix 4: Increase inference timeout
docker exec foxhunt-ml-service sed -i 's/timeout = 5000/timeout = 15000/' /etc/foxhunt/config/ml.toml
docker-compose restart foxhunt-ml-service

# Fix 5: Scale ML service
docker-compose scale foxhunt-ml-service=2

🔍 Diagnostic Tools and Scripts

System Health Check Script

comprehensive-health-check.sh:

#!/bin/bash
# Comprehensive system health check

echo "=== Foxhunt System Health Check - $(date) ==="

# Function to check service health
check_service() {
    local service=$1
    local url=$2
    if curl -f -m 5 "$url" >/dev/null 2>&1; then
        echo "✅ $service: Healthy"
        return 0
    else
        echo "❌ $service: Unhealthy"
        return 1
    fi
}

# Function to check metrics
check_metric() {
    local name=$1
    local query=$2
    local threshold=$3
    local comparison=$4
    
    local value=$(curl -s "http://localhost:9090/api/v1/query?query=$query" | jq -r '.data.result[0].value[1] // "0"')
    
    if [ "$comparison" = "lt" ] && (( $(echo "$value < $threshold" | bc -l) )); then
        echo "✅ $name: $value (< $threshold)"
        return 0
    elif [ "$comparison" = "gt" ] && (( $(echo "$value > $threshold" | bc -l) )); then
        echo "✅ $name: $value (> $threshold)"
        return 0
    elif [ "$comparison" = "eq" ] && (( $(echo "$value == $threshold" | bc -l) )); then
        echo "✅ $name: $value (= $threshold)"
        return 0
    else
        echo "❌ $name: $value (fails $comparison $threshold)"
        return 1
    fi
}

# 1. Service Health Checks
echo "1. Service Health:"
check_service "Trading Service" "http://localhost:50051/health"
check_service "Risk Service" "http://localhost:50052/health" 
check_service "ML Service" "http://localhost:50053/health"
check_service "TLI" "http://localhost:3000/health"
check_service "Prometheus" "http://localhost:9090/-/ready"
check_service "Grafana" "http://localhost:3000/api/health"

# 2. Performance Metrics
echo -e "\n2. Performance Metrics:"
check_metric "Order Latency P99" "histogram_quantile(0.99,rate(order_submission_duration_seconds_bucket[30s]))" "0.00005" "lt"
check_metric "Fill Rate" "rate(orders_filled_total[1m])/rate(orders_submitted_total[1m])" "0.95" "gt"
check_metric "Trading Service Up" "up{job=\"foxhunt-trading\"}" "1" "eq"

# 3. System Resources
echo -e "\n3. System Resources:"
cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | awk -F% '{print $1}')
check_metric "CPU Usage" "echo $cpu_usage" "90" "lt"

mem_usage=$(free | grep Mem | awk '{printf("%.1f"), $3/$2 * 100.0}')
check_metric "Memory Usage" "echo $mem_usage" "85" "lt"

# 4. Database Health
echo -e "\n4. Database Health:"
if docker exec foxhunt-postgres-primary pg_isready -U postgres >/dev/null 2>&1; then
    echo "✅ PostgreSQL: Ready"
else
    echo "❌ PostgreSQL: Not ready"
fi

if docker exec foxhunt-redis-node-1 redis-cli ping >/dev/null 2>&1; then
    echo "✅ Redis: Ready"
else
    echo "❌ Redis: Not ready"
fi

if curl -f http://localhost:8086/health >/dev/null 2>&1; then
    echo "✅ InfluxDB: Ready"  
else
    echo "❌ InfluxDB: Not ready"
fi

# 5. Risk Management
echo -e "\n5. Risk Management:"
check_metric "Current Drawdown" "current_drawdown_pct" "10" "lt"
check_metric "VaR Utilization" "daily_var_utilization" "0.95" "lt"
check_metric "Position Risk" "current_position_risk/risk_limit_threshold" "1.0" "lt"

# 6. Security Status
echo -e "\n6. Security Status:"
if docker exec foxhunt-vault-1 vault status >/dev/null 2>&1; then
    echo "✅ Vault: Sealed status OK"
else
    echo "❌ Vault: Issue detected"
fi

# SSL certificate validity
cert_days=$(echo | openssl s_client -connect localhost:3000 2>/dev/null | openssl x509 -noout -dates | grep notAfter | cut -d= -f2 | xargs -I {} date -d {} +%s)
current_days=$(date +%s)
days_until_expiry=$(( (cert_days - current_days) / 86400 ))

if [ $days_until_expiry -gt 30 ]; then
    echo "✅ SSL Certificate: ${days_until_expiry} days remaining"
else
    echo "⚠️  SSL Certificate: Only ${days_until_expiry} days remaining"
fi

echo -e "\n=== Health Check Complete ==="

Performance Analysis Script

performance-analysis.sh:

#!/bin/bash
# Detailed performance analysis

echo "=== Performance Analysis - $(date) ==="

# 1. Latency Analysis
echo "1. Latency Analysis:"
echo "   Order Submission (last 5 minutes):"
curl -s "http://localhost:9090/api/v1/query?query=histogram_quantile(0.50,rate(order_submission_duration_seconds_bucket[5m]))" | jq -r '.data.result[0].value[1] // "N/A"' | xargs -I {} echo "   P50: {} seconds"
curl -s "http://localhost:9090/api/v1/query?query=histogram_quantile(0.95,rate(order_submission_duration_seconds_bucket[5m]))" | jq -r '.data.result[0].value[1] // "N/A"' | xargs -I {} echo "   P95: {} seconds"
curl -s "http://localhost:9090/api/v1/query?query=histogram_quantile(0.99,rate(order_submission_duration_seconds_bucket[5m]))" | jq -r '.data.result[0].value[1] // "N/A"' | xargs -I {} echo "   P99: {} seconds"

# 2. Throughput Analysis
echo -e "\n2. Throughput Analysis:"
orders_per_sec=$(curl -s "http://localhost:9090/api/v1/query?query=rate(orders_submitted_total[1m])" | jq -r '.data.result[0].value[1] // "0"')
fills_per_sec=$(curl -s "http://localhost:9090/api/v1/query?query=rate(orders_filled_total[1m])" | jq -r '.data.result[0].value[1] // "0"')
echo "   Orders/sec: $orders_per_sec"
echo "   Fills/sec: $fills_per_sec"
echo "   Fill Rate: $(echo "scale=2; $fills_per_sec * 100 / $orders_per_sec" | bc)%"

# 3. Resource Utilization
echo -e "\n3. Resource Utilization:"
echo "   CPU Usage by Service:"
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}" | grep foxhunt

echo -e "\n   Memory Usage by Service:"
docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}" | grep foxhunt

echo -e "\n   GPU Utilization:"
if command -v nvidia-smi &> /dev/null; then
    nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv,noheader,nounits
else
    echo "   No GPU detected"
fi

# 4. Network Performance
echo -e "\n4. Network Performance:"
echo "   Exchange Connectivity:"
ping -c 3 ib-gateway.internal 2>/dev/null | grep "min/avg/max" || echo "   IB Gateway: Unreachable"
ping -c 3 fix.icmarkets.com 2>/dev/null | grep "min/avg/max" || echo "   ICMarkets: Unreachable"

# 5. Database Performance
echo -e "\n5. Database Performance:"
if docker exec foxhunt-postgres-primary psql -U postgres -c "SELECT count(*) as active_connections FROM pg_stat_activity WHERE state = 'active';" 2>/dev/null; then
    echo "   PostgreSQL: Connected"
else
    echo "   PostgreSQL: Connection failed"
fi

redis_ops=$(docker exec foxhunt-redis-node-1 redis-cli --latency-history -i 1 2>/dev/null | head -1 || echo "Redis: Connection failed")
echo "   Redis: $redis_ops"

echo -e "\n=== Performance Analysis Complete ==="

Log Analysis Script

log-analysis.sh:

#!/bin/bash
# Analyze system logs for issues

echo "=== Log Analysis - $(date) ==="

# 1. Error Analysis
echo "1. Recent Errors (last 1 hour):"
services=("foxhunt-trading-service" "foxhunt-risk-service" "foxhunt-ml-service" "foxhunt-tli")
for service in "${services[@]}"; do
    error_count=$(docker logs "$service" --since=1h 2>/dev/null | grep -c -i "error\|exception\|failed\|panic")
    if [ "$error_count" -gt 0 ]; then
        echo "   $service: $error_count errors"
        docker logs "$service" --since=1h | grep -i "error\|exception\|failed\|panic" | tail -3 | sed 's/^/     /'
    else
        echo "   $service: No errors"
    fi
done

# 2. Performance Warnings
echo -e "\n2. Performance Warnings (last 30 minutes):"
for service in "${services[@]}"; do
    perf_warnings=$(docker logs "$service" --since=30m 2>/dev/null | grep -c -i "slow\|timeout\|latency\|performance")
    if [ "$perf_warnings" -gt 0 ]; then
        echo "   $service: $perf_warnings performance warnings"
        docker logs "$service" --since=30m | grep -i "slow\|timeout\|latency\|performance" | tail -2 | sed 's/^/     /'
    fi
done

# 3. System Events
echo -e "\n3. System Events:"
echo "   Memory pressure events:"
dmesg | grep -i "oom\|memory" | tail -3 | sed 's/^/     /'

echo "   CPU throttling events:"
dmesg | grep -i "cpu.*throttled" | tail -3 | sed 's/^/     /'

echo "   Disk I/O errors:"
dmesg | grep -i "i/o error\|disk.*error" | tail -3 | sed 's/^/     /'

# 4. Security Events
echo -e "\n4. Security Events:"
echo "   Authentication failures:"
docker logs foxhunt-tli --since=1h 2>/dev/null | grep -c "authentication.*failed\|unauthorized\|access.*denied" | xargs -I {} echo "   TLI: {} auth failures"

echo "   Suspicious network activity:"
ss -tuln | grep -c ":50051.*ESTABLISHED" | xargs -I {} echo "   Active trading connections: {}"

echo -e "\n=== Log Analysis Complete ==="

📞 Escalation Procedures

Level 1 Support (Operations Team)

Scope: Basic system monitoring, service restarts, configuration changes Response Time: 5 minutes Contact: ops-team@company.com

Escalation Triggers:

• Service health checks fail
• Basic performance metrics outside normal ranges
• Standard monitoring alerts

Level 2 Support (Engineering Team)

Scope: Code-level debugging, database optimization, performance tuning Response Time: 15 minutes
Contact: engineering@company.com

Escalation Triggers:

• Level 1 unable to resolve within 30 minutes
• Performance degradation >20%
• Data corruption or integrity issues

Level 3 Support (Architecture Team)

Scope: System architecture changes, major performance optimization, security incidents Response Time: 1 hour Contact: architecture@company.com

Escalation Triggers:

• System-wide performance issues
• Security breaches or suspected attacks
• Major infrastructure failures

Emergency Escalation

Scope: Trading system down, major financial impact, regulatory issues Response Time: Immediate Contact: emergency@company.com, +1-XXX-XXX-XXXX

Escalation Triggers:

• Trading system completely unavailable >5 minutes
• Risk limits breached with potential major losses
• Regulatory compliance violations
• Security incidents with data exposure

---

Documentation Status: Production-ready comprehensive troubleshooting guide Last Updated: 2025-09-24
Version: Production v1.0.0 Covers: Emergency response, diagnostics, performance optimization, escalation