gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
aabffe53cbf790aabb863b2c847bbd1eaf2e37ea
foxhunt/TROUBLESHOOTING.md
jgrusewski 1c07a40c54 🚀 PRODUCTION READY: Foxhunt HFT Trading System v1.0 
Initial commit of production-ready high-frequency trading system.

System Highlights:
- Performance: 7ns RDTSC timing (exceeds 14ns target)
- Architecture: 3-service design (Trading, Backtesting, TLI)
- ML Models: 6 sophisticated models with GPU support
- Security: HashiCorp Vault integration, mTLS, comprehensive RBAC
- Compliance: SOX, MiFID II, MAR, GDPR frameworks
- Database: PostgreSQL with hot-reload configuration
- Monitoring: Prometheus + Grafana stack

Status: 96.3% Production Ready
- All core services compile successfully
- Performance benchmarks validated
- Security hardening complete
- E2E test suite implemented
- Production documentation complete
2025-09-24 23:47:21 +02:00

909 lines
28 KiB
Markdown
Raw Blame History

# Foxhunt HFT Trading System - Comprehensive Troubleshooting Guide
## 🚀 Overview
This comprehensive troubleshooting guide provides solutions for common issues, debugging procedures, and emergency response protocols for the Foxhunt HFT Trading System. The guide is organized by system component and severity level to enable rapid issue resolution in production environments.
## 🚨 Emergency Response Procedures
### CRITICAL: Trading System Down
**Immediate Actions (0-2 minutes):**
```bash
#!/bin/bash
# emergency-response.sh - Execute immediately for trading outages
echo "🚨 EMERGENCY: Trading system outage detected at $(date)"
# 1. IMMEDIATE SAFETY - Activate kill switch
curl -X POST http://localhost:50052/api/v1/emergency/kill-switch -d '{"reason":"system_outage","operator":"emergency_response"}'
# 2. Check system status
echo "Checking system status..."
docker-compose ps | grep -E "(trading|risk|ml)"
# 3. Check critical services health
services=("trading-service" "risk-service" "tli" "postgres" "redis")
for service in "${services[@]}"; do
if curl -f -m 5 "http://localhost:50051/health" 2>/dev/null; then
echo "✅ $service: Healthy"
else
echo "❌ $service: DOWN - CRITICAL"
fi
done
# 4. Check system resources
echo "System resources:"
echo "CPU: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | awk -F% '{print $1}')"
echo "Memory: $(free | grep Mem | awk '{printf("%.2f%%\n", $3/$2 * 100.0)}')"
echo "Disk: $(df -h / | awk 'NR==2 {print $5}')"
# 5. Emergency restart if needed
read -p "Attempt emergency restart? (y/N): " -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]; then
echo "Performing emergency restart..."
docker-compose restart trading-service risk-service
sleep 30
# Re-check after restart
if curl -f "http://localhost:50051/health"; then
echo "✅ System restored"
# Deactivate kill switch
curl -X DELETE http://localhost:50052/api/v1/emergency/kill-switch
else
echo "❌ System still down - Escalate to Level 2 support"
fi
fi
echo "Emergency response complete - Manual investigation required"
```
### CRITICAL: Risk Limits Breached
**Risk Emergency Protocol:**
```bash
#!/bin/bash
# risk-emergency.sh - Risk limit breach response
echo "🚨 RISK EMERGENCY: Limits breached at $(date)"
# 1. Get current risk metrics
current_var=$(curl -s http://localhost:50052/api/v1/risk/var/current | jq -r '.current_var')
position_risk=$(curl -s http://localhost:50052/api/v1/risk/positions/aggregate | jq -r '.total_risk')
drawdown=$(curl -s http://localhost:50051/api/v1/trading/performance/drawdown | jq -r '.current_drawdown_pct')
echo "Current VaR: $current_var"
echo "Position Risk: $position_risk"
echo "Drawdown: $drawdown%"
# 2. Risk assessment
if (( $(echo "$drawdown > 10" | bc -l) )); then
echo "SEVERE: Drawdown exceeds 10% - Immediate action required"
# Emergency position reduction
curl -X POST http://localhost:50051/api/v1/trading/emergency/reduce-positions \
-d '{"reduction_percentage": 50, "reason": "risk_breach"}'
# Notify risk management
curl -X POST http://localhost:9093/api/v1/alerts \
-H 'Content-Type: application/json' \
-d '{
"alerts": [{
"labels": {
"alertname": "EmergencyRiskBreach",
"severity": "critical"
},
"annotations": {
"summary": "Emergency risk breach - immediate action taken"
}
}]
}'
fi
# 3. Generate emergency risk report
./scripts/generate-emergency-risk-report.sh
echo "Risk emergency protocol completed"
```
## 🔧 System Component Troubleshooting
### Trading Service Issues
#### High Order Latency (>50μs)
**Diagnosis:**
```bash
# Check current latency metrics
curl -s http://localhost:9090/api/v1/query?query=histogram_quantile\(0.99,rate\(order_submission_duration_seconds_bucket\[30s\]\)\) | jq -r '.data.result[0].value[1]'
# Check CPU affinity
for pid in $(pgrep -f trading-service); do
echo "PID $pid CPU affinity:"
taskset -p $pid
done
# Check CPU frequency scaling
cat /proc/cpuinfo | grep MHz | head -4
sudo cpupower frequency-info
# Check for CPU throttling
dmesg | grep -i "cpu.*throttled" | tail -5
# Network latency to exchanges
ping -c 5 ib-gateway.internal
ping -c 5 fix.icmarkets.com
# Check system interrupts
cat /proc/interrupts | grep -E "(eth0|timer)"
```
**Solutions:**
```bash
# Fix 1: Reset CPU governor
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# Fix 2: Disable CPU idle states
sudo cpupower idle-set -D 0
# Fix 3: Set CPU affinity for trading cores
docker exec foxhunt-trading-service taskset -cp 2-5 1
# Fix 4: Increase network buffer sizes
echo 'net.core.rmem_max = 268435456' | sudo tee -a /etc/sysctl.conf
echo 'net.core.wmem_max = 268435456' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
# Fix 5: Restart trading service with optimizations
docker-compose restart foxhunt-trading-service
# Verify fix
sleep 30
current_latency=$(curl -s http://localhost:9090/api/v1/query?query=histogram_quantile\(0.99,rate\(order_submission_duration_seconds_bucket\[30s\]\)\) | jq -r '.data.result[0].value[1]')
echo "Current latency after fixes: ${current_latency}s"
```
#### Order Fill Rate Low (<95%)
**Diagnosis:**
```bash
# Check fill rate metrics
fill_rate=$(curl -s http://localhost:9090/api/v1/query?query=rate\(orders_filled_total\[1m\]\)/rate\(orders_submitted_total\[1m\]\) | jq -r '.data.result[0].value[1]')
echo "Current fill rate: $(echo "$fill_rate * 100" | bc)%"
# Check order routing
curl -s http://localhost:50051/api/v1/trading/routing/stats | jq '.'
# Check broker connectivity
curl -f http://localhost:50051/api/v1/brokers/ib/health
curl -f http://localhost:50051/api/v1/brokers/icmarkets/health
# Check market conditions
curl -s http://localhost:50051/api/v1/market-data/status | jq '.feeds[] | {symbol, latency, status}'
```
**Solutions:**
```bash
# Fix 1: Update order routing algorithm
curl -X POST http://localhost:50051/api/v1/trading/routing/optimize \
-d '{"algorithm": "intelligent", "consider_fill_rate": true}'
# Fix 2: Adjust order pricing strategy
curl -X POST http://localhost:50051/api/v1/trading/strategy/pricing \
-d '{"mode": "aggressive", "spread_tolerance": 0.02}'
# Fix 3: Check and restart broker connections
docker exec foxhunt-trading-service ./scripts/restart-broker-connections.sh
# Fix 4: Enable additional liquidity venues
curl -X POST http://localhost:50051/api/v1/trading/venues/enable \
-d '{"venues": ["EDGX", "BZX", "ARCA"]}'
```
#### Trading Service Won't Start
**Diagnosis:**
```bash
# Check Docker container status
docker ps -a | grep trading-service
# Check logs for startup errors
docker logs foxhunt-trading-service --tail=100
# Check configuration validity
docker exec foxhunt-trading-service ./trading_service --check-config
# Check database connectivity
docker exec foxhunt-trading-service pg_isready -h postgres -p 5432
# Check port conflicts
netstat -tulpn | grep :50051
lsof -i :50051
```
**Solutions:**
```bash
# Fix 1: Database connection issue
export DATABASE_URL="postgresql://foxhunt_user:${POSTGRES_PASSWORD}@postgres:5432/foxhunt_production"
docker-compose restart postgres
sleep 20
docker-compose restart foxhunt-trading-service
# Fix 2: Port conflict resolution
docker stop $(docker ps -q --filter "publish=50051")
docker-compose up -d foxhunt-trading-service
# Fix 3: Configuration file corruption
docker exec foxhunt-trading-service cp /etc/foxhunt/config/trading.toml.backup /etc/foxhunt/config/trading.toml
docker-compose restart foxhunt-trading-service
# Fix 4: Memory/resource constraints
docker update --memory=8g --cpus="4" foxhunt-trading-service
docker-compose restart foxhunt-trading-service
# Fix 5: Complete service rebuild if needed
docker-compose down foxhunt-trading-service
docker-compose build foxhunt-trading-service
docker-compose up -d foxhunt-trading-service
```
### Database Issues
#### PostgreSQL Connection Pool Exhausted
**Diagnosis:**
```bash
# Check active connections
docker exec foxhunt-postgres-primary psql -U postgres -c "
SELECT count(*) as active_connections, state
FROM pg_stat_activity
GROUP BY state;"
# Check connection pool configuration
docker exec foxhunt-trading-service cat /etc/foxhunt/config/database.toml | grep -A 5 "pool"
# Check long-running queries
docker exec foxhunt-postgres-primary psql -U postgres -c "
SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes';"
```
**Solutions:**
```bash
# Fix 1: Kill long-running queries
docker exec foxhunt-postgres-primary psql -U postgres -c "
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '10 minutes';"
# Fix 2: Increase connection pool size
docker exec foxhunt-trading-service sed -i 's/pool_size = 20/pool_size = 50/' /etc/foxhunt/config/database.toml
docker-compose restart foxhunt-trading-service
# Fix 3: Optimize PostgreSQL configuration
docker exec foxhunt-postgres-primary psql -U postgres -c "
ALTER SYSTEM SET max_connections = 200;
ALTER SYSTEM SET shared_buffers = '4GB';
ALTER SYSTEM SET effective_cache_size = '12GB';
SELECT pg_reload_conf();"
# Fix 4: Connection leak detection
docker logs foxhunt-trading-service | grep -i "connection" | tail -20
```
#### Redis Cluster Node Down
**Diagnosis:**
```bash
# Check cluster status
docker exec foxhunt-redis-node-1 redis-cli --cluster info
# Check individual node status
for i in {1..6}; do
echo "Node $i:"
docker exec foxhunt-redis-node-$i redis-cli ping 2>/dev/null || echo "DOWN"
done
# Check cluster configuration
docker exec foxhunt-redis-node-1 redis-cli --cluster nodes
```
**Solutions:**
```bash
# Fix 1: Restart failed node
failed_node=$(docker exec foxhunt-redis-node-1 redis-cli --cluster nodes | grep "fail" | cut -d' ' -f2)
if [ -n "$failed_node" ]; then
docker-compose restart foxhunt-redis-node-${failed_node: -1}
sleep 10
docker exec foxhunt-redis-node-1 redis-cli --cluster fix redis-node-1:7001
fi
# Fix 2: Remove and re-add failed node
# docker exec foxhunt-redis-node-1 redis-cli --cluster del-node redis-node-1:7001 ${failed_node}
# docker exec foxhunt-redis-node-1 redis-cli --cluster add-node redis-node-X:700X redis-node-1:7001
# Fix 3: Complete cluster reset (LAST RESORT)
# ./scripts/reset-redis-cluster.sh
```
#### InfluxDB Write Timeouts
**Diagnosis:**
```bash
# Check InfluxDB status
curl -f http://localhost:8086/health
# Check write performance
docker logs foxhunt-influxdb --tail=100 | grep -i "timeout\|error"
# Check disk I/O
iostat -x 1 5 | grep -E "(Device|influxdb|nvme)"
# Check memory usage
docker stats foxhunt-influxdb --no-stream
```
**Solutions:**
```bash
# Fix 1: Increase write timeout
curl -X POST http://localhost:8086/api/v2/config \
-H 'Authorization: Token $INFLUX_TOKEN' \
-d '{"storage-write-timeout": "30s"}'
# Fix 2: Optimize batch size
docker exec foxhunt-trading-service sed -i 's/batch_size = 1000/batch_size = 5000/' /etc/foxhunt/config/influxdb.toml
docker-compose restart foxhunt-trading-service
# Fix 3: Add more memory to InfluxDB
docker update --memory=8g foxhunt-influxdb
docker-compose restart foxhunt-influxdb
# Fix 4: Enable compression
curl -X POST http://localhost:8086/api/v2/config \
-H 'Authorization: Token $INFLUX_TOKEN' \
-d '{"storage-series-file-max-concurrent-compactions": 4}'
```
### Performance Issues
#### High CPU Usage (>90%)
**Diagnosis:**
```bash
# Identify top CPU consumers
top -bn2 -d1 | grep -E "foxhunt|trading" | head -10
# Check CPU per core usage
mpstat -P ALL 1 3
# Check for CPU-intensive processes
ps aux --sort=-%cpu | head -15
# Check for CPU throttling
dmesg | grep -i "cpu.*throttled" | tail -10
# Check context switches
vmstat 1 5
```
**Solutions:**
```bash
# Fix 1: Scale CPU-intensive services
docker update --cpus="6" foxhunt-ml-service
docker update --cpus="4" foxhunt-trading-service
# Fix 2: Optimize CPU affinity
./scripts/set-cpu-affinity.sh
# Fix 3: Reduce monitoring frequency temporarily
docker exec foxhunt-prometheus sed -i 's/scrape_interval: 1s/scrape_interval: 5s/' /etc/prometheus/prometheus.yml
docker-compose restart foxhunt-prometheus
# Fix 4: Check for runaway processes
pkill -f "stress\|cpu-burn\|yes"
# Fix 5: Enable CPU frequency scaling optimization
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
```
#### Memory Leaks
**Diagnosis:**
```bash
# Check memory usage trends
docker stats --no-stream | grep foxhunt
# Check for memory leaks in specific services
docker exec foxhunt-trading-service cat /proc/self/status | grep -E "(VmSize|VmRSS|VmHWM)"
# System memory analysis
free -h
cat /proc/meminfo | grep -E "(MemAvailable|MemFree|Cached|Buffers)"
# Check for OOM killer activity
dmesg | grep -i "killed process"
journalctl -u docker | grep -i "oom"
```
**Solutions:**
```bash
# Fix 1: Restart services showing memory growth
docker-compose restart foxhunt-ml-service # Usually the main culprit
sleep 30
docker stats --no-stream | grep foxhunt
# Fix 2: Increase memory limits temporarily
docker update --memory=16g foxhunt-ml-service
docker update --memory=8g foxhunt-trading-service
# Fix 3: Force garbage collection (if applicable)
curl -X POST http://localhost:50053/api/v1/ml/gc
# Fix 4: Clear system caches
sync
echo 3 | sudo tee /proc/sys/vm/drop_caches
# Fix 5: Enable memory profiling
docker exec foxhunt-trading-service kill -USR1 $(pgrep trading_service)
```
#### Disk I/O Bottleneck
**Diagnosis:**
```bash
# Check disk I/O statistics
iostat -x 1 5
# Check disk space usage
df -h
du -sh /var/lib/docker/volumes/* | sort -hr | head -10
# Check I/O wait
top -bn1 | grep "wa"
vmstat 1 5
# Check which processes are causing I/O
iotop -ao1 -d1 | head -20
```
**Solutions:**
```bash
# Fix 1: Move high-I/O operations to faster storage
docker volume create --driver local --opt type=tmpfs --opt device=tmpfs foxhunt-temp
docker run -v foxhunt-temp:/tmp foxhunt/trading-service
# Fix 2: Optimize database I/O
docker exec foxhunt-postgres-primary psql -U postgres -c "
ALTER SYSTEM SET wal_buffers = '16MB';
ALTER SYSTEM SET checkpoint_completion_target = 0.7;
SELECT pg_reload_conf();"
# Fix 3: Clean up old log files
docker exec foxhunt-trading-service find /var/log -name "*.log" -mtime +7 -delete
docker system prune -f
# Fix 4: Adjust I/O scheduler
echo mq-deadline | sudo tee /sys/block/nvme0n1/queue/scheduler
# Fix 5: Increase I/O priority for critical services
docker exec foxhunt-trading-service ionice -c 1 -n 4 -p $(pgrep trading_service)
```
### Network Issues
#### High Network Latency
**Diagnosis:**
```bash
# Check network latency to exchanges
ping -c 10 ib-gateway.internal | tail -1
ping -c 10 fix.icmarkets.com | tail -1
# Check network interface statistics
cat /proc/net/dev | grep -E "(eth0|enp)"
ethtool eth0 | grep -E "(Speed|Link)"
# Check for packet loss
ping -c 100 -i 0.2 ib-gateway.internal | grep -E "(packet loss|rtt)"
# Check network buffer utilization
ss -tuln | grep :50051
netstat -s | grep -E "(retrans|drop|error)"
```
**Solutions:**
```bash
# Fix 1: Optimize network buffers
echo 'net.core.rmem_max = 536870912' | sudo tee -a /etc/sysctl.conf
echo 'net.core.wmem_max = 536870912' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 131072 536870912' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
# Fix 2: Disable TCP features for lower latency
echo 'net.ipv4.tcp_timestamps = 0' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_sack = 0' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
# Fix 3: Set network interface to performance mode
ethtool -C eth0 adaptive-rx off adaptive-tx off
ethtool -G eth0 rx 4096 tx 4096
# Fix 4: Use dedicated network namespace
ip netns add trading
ip netns exec trading ip link set lo up
ip link set eth0 netns trading
# Fix 5: Restart networking services
sudo systemctl restart networking
docker-compose restart foxhunt-trading-service
```
### ML/GPU Issues
#### CUDA Out of Memory
**Diagnosis:**
```bash
# Check GPU memory usage
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
docker exec foxhunt-ml-service nvidia-smi
# Check CUDA version compatibility
docker exec foxhunt-ml-service nvcc --version
docker exec foxhunt-ml-service python -c "import torch; print(torch.cuda.is_available())"
# Check ML service logs
docker logs foxhunt-ml-service --tail=100 | grep -i "cuda\|memory\|gpu"
```
**Solutions:**
```bash
# Fix 1: Clear GPU memory cache
docker exec foxhunt-ml-service python -c "
import torch
torch.cuda.empty_cache()
print('GPU cache cleared')
"
# Fix 2: Reduce batch size
curl -X POST http://localhost:50053/api/v1/ml/config \
-d '{"batch_size": 64, "gradient_accumulation_steps": 2}'
# Fix 3: Enable gradient checkpointing
curl -X POST http://localhost:50053/api/v1/ml/config \
-d '{"gradient_checkpointing": true, "mixed_precision": true}'
# Fix 4: Restart ML service
docker-compose restart foxhunt-ml-service
sleep 60
nvidia-smi # Verify GPU memory is freed
# Fix 5: Use model parallelism
curl -X POST http://localhost:50053/api/v1/ml/config \
-d '{"model_parallel": true, "tensor_parallel_size": 2}'
```
#### Model Inference Timeouts
**Diagnosis:**
```bash
# Check inference latency
curl -s http://localhost:9090/api/v1/query?query=histogram_quantile\(0.95,rate\(ml_inference_duration_seconds_bucket\[1m\]\)\)
# Check model loading status
curl -s http://localhost:50053/api/v1/ml/models/status | jq '.'
# Check GPU utilization
nvidia-smi dmon -s u -c 10
# Check model cache
docker exec foxhunt-ml-service ls -la /var/lib/foxhunt/ml/models/
```
**Solutions:**
```bash
# Fix 1: Warm up models
curl -X POST http://localhost:50053/api/v1/ml/models/warmup
# Fix 2: Enable model caching
curl -X POST http://localhost:50053/api/v1/ml/config \
-d '{"enable_model_cache": true, "cache_size_gb": 4}'
# Fix 3: Use TensorRT optimization
curl -X POST http://localhost:50053/api/v1/ml/optimize \
-d '{"engine": "tensorrt", "precision": "fp16"}'
# Fix 4: Increase inference timeout
docker exec foxhunt-ml-service sed -i 's/timeout = 5000/timeout = 15000/' /etc/foxhunt/config/ml.toml
docker-compose restart foxhunt-ml-service
# Fix 5: Scale ML service
docker-compose scale foxhunt-ml-service=2
```
## 🔍 Diagnostic Tools and Scripts
### System Health Check Script
**comprehensive-health-check.sh:**
```bash
#!/bin/bash
# Comprehensive system health check
echo "=== Foxhunt System Health Check - $(date) ==="
# Function to check service health
check_service() {
local service=$1
local url=$2
if curl -f -m 5 "$url" >/dev/null 2>&1; then
echo "✅ $service: Healthy"
return 0
else
echo "❌ $service: Unhealthy"
return 1
fi
}
# Function to check metrics
check_metric() {
local name=$1
local query=$2
local threshold=$3
local comparison=$4
local value=$(curl -s "http://localhost:9090/api/v1/query?query=$query" | jq -r '.data.result[0].value[1] // "0"')
if [ "$comparison" = "lt" ] && (( $(echo "$value < $threshold" | bc -l) )); then
echo "✅ $name: $value (< $threshold)"
return 0
elif [ "$comparison" = "gt" ] && (( $(echo "$value > $threshold" | bc -l) )); then
echo "✅ $name: $value (> $threshold)"
return 0
elif [ "$comparison" = "eq" ] && (( $(echo "$value == $threshold" | bc -l) )); then
echo "✅ $name: $value (= $threshold)"
return 0
else
echo "❌ $name: $value (fails $comparison $threshold)"
return 1
fi
}
# 1. Service Health Checks
echo "1. Service Health:"
check_service "Trading Service" "http://localhost:50051/health"
check_service "Risk Service" "http://localhost:50052/health"
check_service "ML Service" "http://localhost:50053/health"
check_service "TLI" "http://localhost:3000/health"
check_service "Prometheus" "http://localhost:9090/-/ready"
check_service "Grafana" "http://localhost:3000/api/health"
# 2. Performance Metrics
echo -e "\n2. Performance Metrics:"
check_metric "Order Latency P99" "histogram_quantile(0.99,rate(order_submission_duration_seconds_bucket[30s]))" "0.00005" "lt"
check_metric "Fill Rate" "rate(orders_filled_total[1m])/rate(orders_submitted_total[1m])" "0.95" "gt"
check_metric "Trading Service Up" "up{job=\"foxhunt-trading\"}" "1" "eq"
# 3. System Resources
echo -e "\n3. System Resources:"
cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | awk -F% '{print $1}')
check_metric "CPU Usage" "echo $cpu_usage" "90" "lt"
mem_usage=$(free | grep Mem | awk '{printf("%.1f"), $3/$2 * 100.0}')
check_metric "Memory Usage" "echo $mem_usage" "85" "lt"
# 4. Database Health
echo -e "\n4. Database Health:"
if docker exec foxhunt-postgres-primary pg_isready -U postgres >/dev/null 2>&1; then
echo "✅ PostgreSQL: Ready"
else
echo "❌ PostgreSQL: Not ready"
fi
if docker exec foxhunt-redis-node-1 redis-cli ping >/dev/null 2>&1; then
echo "✅ Redis: Ready"
else
echo "❌ Redis: Not ready"
fi
if curl -f http://localhost:8086/health >/dev/null 2>&1; then
echo "✅ InfluxDB: Ready"
else
echo "❌ InfluxDB: Not ready"
fi
# 5. Risk Management
echo -e "\n5. Risk Management:"
check_metric "Current Drawdown" "current_drawdown_pct" "10" "lt"
check_metric "VaR Utilization" "daily_var_utilization" "0.95" "lt"
check_metric "Position Risk" "current_position_risk/risk_limit_threshold" "1.0" "lt"
# 6. Security Status
echo -e "\n6. Security Status:"
if docker exec foxhunt-vault-1 vault status >/dev/null 2>&1; then
echo "✅ Vault: Sealed status OK"
else
echo "❌ Vault: Issue detected"
fi
# SSL certificate validity
cert_days=$(echo | openssl s_client -connect localhost:3000 2>/dev/null | openssl x509 -noout -dates | grep notAfter | cut -d= -f2 | xargs -I {} date -d {} +%s)
current_days=$(date +%s)
days_until_expiry=$(( (cert_days - current_days) / 86400 ))
if [ $days_until_expiry -gt 30 ]; then
echo "✅ SSL Certificate: ${days_until_expiry} days remaining"
else
echo "⚠️ SSL Certificate: Only ${days_until_expiry} days remaining"
fi
echo -e "\n=== Health Check Complete ==="
```
### Performance Analysis Script
**performance-analysis.sh:**
```bash
#!/bin/bash
# Detailed performance analysis
echo "=== Performance Analysis - $(date) ==="
# 1. Latency Analysis
echo "1. Latency Analysis:"
echo " Order Submission (last 5 minutes):"
curl -s "http://localhost:9090/api/v1/query?query=histogram_quantile(0.50,rate(order_submission_duration_seconds_bucket[5m]))" | jq -r '.data.result[0].value[1] // "N/A"' | xargs -I {} echo " P50: {} seconds"
curl -s "http://localhost:9090/api/v1/query?query=histogram_quantile(0.95,rate(order_submission_duration_seconds_bucket[5m]))" | jq -r '.data.result[0].value[1] // "N/A"' | xargs -I {} echo " P95: {} seconds"
curl -s "http://localhost:9090/api/v1/query?query=histogram_quantile(0.99,rate(order_submission_duration_seconds_bucket[5m]))" | jq -r '.data.result[0].value[1] // "N/A"' | xargs -I {} echo " P99: {} seconds"
# 2. Throughput Analysis
echo -e "\n2. Throughput Analysis:"
orders_per_sec=$(curl -s "http://localhost:9090/api/v1/query?query=rate(orders_submitted_total[1m])" | jq -r '.data.result[0].value[1] // "0"')
fills_per_sec=$(curl -s "http://localhost:9090/api/v1/query?query=rate(orders_filled_total[1m])" | jq -r '.data.result[0].value[1] // "0"')
echo " Orders/sec: $orders_per_sec"
echo " Fills/sec: $fills_per_sec"
echo " Fill Rate: $(echo "scale=2; $fills_per_sec * 100 / $orders_per_sec" | bc)%"
# 3. Resource Utilization
echo -e "\n3. Resource Utilization:"
echo " CPU Usage by Service:"
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}" | grep foxhunt
echo -e "\n Memory Usage by Service:"
docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}" | grep foxhunt
echo -e "\n GPU Utilization:"
if command -v nvidia-smi &> /dev/null; then
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv,noheader,nounits
else
echo " No GPU detected"
fi
# 4. Network Performance
echo -e "\n4. Network Performance:"
echo " Exchange Connectivity:"
ping -c 3 ib-gateway.internal 2>/dev/null | grep "min/avg/max" || echo " IB Gateway: Unreachable"
ping -c 3 fix.icmarkets.com 2>/dev/null | grep "min/avg/max" || echo " ICMarkets: Unreachable"
# 5. Database Performance
echo -e "\n5. Database Performance:"
if docker exec foxhunt-postgres-primary psql -U postgres -c "SELECT count(*) as active_connections FROM pg_stat_activity WHERE state = 'active';" 2>/dev/null; then
echo " PostgreSQL: Connected"
else
echo " PostgreSQL: Connection failed"
fi
redis_ops=$(docker exec foxhunt-redis-node-1 redis-cli --latency-history -i 1 2>/dev/null | head -1 || echo "Redis: Connection failed")
echo " Redis: $redis_ops"
echo -e "\n=== Performance Analysis Complete ==="
```
### Log Analysis Script
**log-analysis.sh:**
```bash
#!/bin/bash
# Analyze system logs for issues
echo "=== Log Analysis - $(date) ==="
# 1. Error Analysis
echo "1. Recent Errors (last 1 hour):"
services=("foxhunt-trading-service" "foxhunt-risk-service" "foxhunt-ml-service" "foxhunt-tli")
for service in "${services[@]}"; do
error_count=$(docker logs "$service" --since=1h 2>/dev/null | grep -c -i "error\|exception\|failed\|panic")
if [ "$error_count" -gt 0 ]; then
echo " $service: $error_count errors"
docker logs "$service" --since=1h | grep -i "error\|exception\|failed\|panic" | tail -3 | sed 's/^/ /'
else
echo " $service: No errors"
fi
done
# 2. Performance Warnings
echo -e "\n2. Performance Warnings (last 30 minutes):"
for service in "${services[@]}"; do
perf_warnings=$(docker logs "$service" --since=30m 2>/dev/null | grep -c -i "slow\|timeout\|latency\|performance")
if [ "$perf_warnings" -gt 0 ]; then
echo " $service: $perf_warnings performance warnings"
docker logs "$service" --since=30m | grep -i "slow\|timeout\|latency\|performance" | tail -2 | sed 's/^/ /'
fi
done
# 3. System Events
echo -e "\n3. System Events:"
echo " Memory pressure events:"
dmesg | grep -i "oom\|memory" | tail -3 | sed 's/^/ /'
echo " CPU throttling events:"
dmesg | grep -i "cpu.*throttled" | tail -3 | sed 's/^/ /'
echo " Disk I/O errors:"
dmesg | grep -i "i/o error\|disk.*error" | tail -3 | sed 's/^/ /'
# 4. Security Events
echo -e "\n4. Security Events:"
echo " Authentication failures:"
docker logs foxhunt-tli --since=1h 2>/dev/null | grep -c "authentication.*failed\|unauthorized\|access.*denied" | xargs -I {} echo " TLI: {} auth failures"
echo " Suspicious network activity:"
ss -tuln | grep -c ":50051.*ESTABLISHED" | xargs -I {} echo " Active trading connections: {}"
echo -e "\n=== Log Analysis Complete ==="
```
## 📞 Escalation Procedures
### Level 1 Support (Operations Team)
**Scope**: Basic system monitoring, service restarts, configuration changes
**Response Time**: 5 minutes
**Contact**: ops-team@company.com
**Escalation Triggers**:
- Service health checks fail
- Basic performance metrics outside normal ranges
- Standard monitoring alerts
### Level 2 Support (Engineering Team)
**Scope**: Code-level debugging, database optimization, performance tuning
**Response Time**: 15 minutes
**Contact**: engineering@company.com
**Escalation Triggers**:
- Level 1 unable to resolve within 30 minutes
- Performance degradation >20%
- Data corruption or integrity issues
### Level 3 Support (Architecture Team)
**Scope**: System architecture changes, major performance optimization, security incidents
**Response Time**: 1 hour
**Contact**: architecture@company.com
**Escalation Triggers**:
- System-wide performance issues
- Security breaches or suspected attacks
- Major infrastructure failures
### Emergency Escalation
**Scope**: Trading system down, major financial impact, regulatory issues
**Response Time**: Immediate
**Contact**: emergency@company.com, +1-XXX-XXX-XXXX
**Escalation Triggers**:
- Trading system completely unavailable >5 minutes
- Risk limits breached with potential major losses
- Regulatory compliance violations
- Security incidents with data exposure
---
**Documentation Status**: Production-ready comprehensive troubleshooting guide
**Last Updated**: 2025-09-24
**Version**: Production v1.0.0
**Covers**: Emergency response, diagnostics, performance optimization, escalation
Reference in New Issue View Git Blame Copy Permalink