foxhunt/docs/TROUBLESHOOTING_GUIDE.md

Foxhunt HFT Trading System - Troubleshooting Guide

Version: 1.0 Last Updated: 2025-10-03 Wave: 67 - Production Operations Audience: Operators, SREs, On-Call Engineers

---

Quick Diagnosis Decision Tree

SERVICE NOT RESPONDING?
│
├─ YES → Is process running? (pgrep -a service_name)
│  │
│  ├─ NO → Start service
│  │      └─ Still fails? → Check logs → See [Service Startup Failures](#service-startup-failures)
│  │
│  └─ YES → Is health endpoint responding? (grpcurl health check)
│     │
│     ├─ NO → Check logs for errors → See [Service Health Failures](#service-health-failures)
│     │
│     └─ YES → Check client connectivity → See [Network Issues](#network-issues)
│
└─ NO → PERFORMANCE DEGRADATION?
   │
   ├─ YES → High latency? (>10ms P99)
   │  │
   │  ├─ YES → Check database query times → See [Database Performance](#database-performance)
   │  │       Check memory pressure → See [Memory Issues](#memory-issues)
   │  │       Check CPU usage → See [CPU Issues](#cpu-issues)
   │  │
   │  └─ NO → High error rate? (>1%)
   │         └─ YES → Check service logs → See [Error Analysis](#error-analysis)
   │
   └─ NO → AUTHENTICATION FAILING?
          └─ YES → Is auth enabled? → See [Authentication Issues](#authentication-issues)

---

Table of Contents

1. Service Startup Failures
2. Service Health Failures
3. Network Issues
4. Database Performance
5. Memory Issues
6. CPU Issues
7. Authentication Issues
8. Configuration Issues
9. Integration Test Failures
10. Emergency Escalation

---

Service Startup Failures

Symptom: Service Won't Start

Decision Tree:

Service won't start?
│
├─ Check logs: tail -100 /var/log/foxhunt/SERVICE.log
│  │
│  ├─ "Address already in use" → Port conflict
│  │  └─ Solution: lsof -i :PORT → kill PID → retry
│  │
│  ├─ "Database connection failed" → Database issue
│  │  └─ Solution: psql $DATABASE_URL -c "SELECT 1;" → Fix DB → retry
│  │
│  ├─ "Configuration file not found" → Config missing
│  │  └─ Solution: Check .env.production exists → See [Configuration Issues](#configuration-issues)
│  │
│  └─ "Permission denied" → File permissions
│     └─ Solution: chown foxhunt_service:foxhunt_service /path/to/service → retry

Common Startup Errors

Error: "Address already in use"

Symptom:

ERROR: Failed to bind to address 0.0.0.0:50051
Error: Address already in use (os error 98)

Diagnosis:

# Find what's using the port
lsof -i :50051

# Check if old process is still running
pgrep -a trading_service

Solution:

# Kill old process
OLD_PID=$(lsof -t -i :50051)
kill -TERM $OLD_PID

# Wait for clean shutdown
sleep 5

# Force kill if still running
kill -9 $OLD_PID

# Restart service
./scripts/start-all-services.sh

Error: "Database connection refused"

Symptom:

ERROR: Failed to connect to database
Error: Connection refused (os error 111)

Diagnosis:

# Check PostgreSQL status
sudo systemctl status postgresql

# Check database connectivity
psql $DATABASE_URL -c "SELECT version();"

# Check database logs
sudo tail -50 /var/log/postgresql/postgresql-14-main.log

Solution:

# Restart PostgreSQL if down
sudo systemctl restart postgresql

# Wait for PostgreSQL to be ready
sleep 10

# Verify connection
psql $DATABASE_URL -c "SELECT 1;"

# Retry service startup
./scripts/start-all-services.sh

Error: "Configuration file not found"

Symptom:

ERROR: Configuration file not found: /etc/foxhunt/trading_service.toml

Diagnosis:

# Check if config file exists
ls -la /etc/foxhunt/trading_service.toml

# Check environment file
ls -la .env.production

# Check file permissions
ls -la /etc/foxhunt/*.toml

Solution:

# Copy from template
sudo cp /etc/foxhunt/trading_service.toml.example /etc/foxhunt/trading_service.toml

# Set correct permissions
sudo chown foxhunt_service:foxhunt_service /etc/foxhunt/trading_service.toml

# Verify Wave 66 environment template exists
ls -la .env.production.example

# Copy and configure
cp .env.production.example .env.production
vim .env.production  # Fill in production values

---

Service Health Failures

Symptom: Health Check Fails

Decision Tree:

Health check fails?
│
├─ Check service logs: tail -100 /var/log/foxhunt/SERVICE.log
│  │
│  ├─ Recent PANIC/FATAL? → Service crashed
│  │  └─ Solution: Investigate crash → See [Service Crash Investigation](#service-crash-investigation)
│  │
│  ├─ "Connection pool exhausted" → Database overload
│  │  └─ Solution: Check DB connections → See [Database Performance](#database-performance)
│  │
│  ├─ "Out of memory" → Memory pressure
│  │  └─ Solution: Check memory usage → See [Memory Issues](#memory-issues)
│  │
│  └─ No recent errors → Slow response
│     └─ Solution: Check CPU/latency → See [Performance Degradation](#performance-degradation)

Service Crash Investigation

Symptom:

Health check returns: "Service Unavailable"
Process not running (pgrep returns nothing)

Diagnosis:

# Check for core dumps
ls -lt /var/crash/ | head -5

# Check service logs for panic
tail -200 /var/log/foxhunt/trading_service.log | grep -A 20 "PANIC\|FATAL\|panic"

# Check system logs
sudo dmesg | tail -50 | grep -i "kill\|oom\|segfault"

# Check if OOM killed the service
sudo grep -i "killed process" /var/log/syslog | tail -10

Common Crash Causes:

1. Out of Memory (OOM):

# Evidence:
sudo grep "Out of memory" /var/log/syslog

# Solution:
# 1. Reduce memory pressure (see [Memory Issues](#memory-issues))
# 2. Increase system memory
# 3. Configure OOM score to protect critical services
echo -1000 > /proc/$(pgrep trading_service)/oom_score_adj

2. Panic/Unwrap on None:

# Evidence in logs:
# "thread 'main' panicked at 'called `Option::unwrap()` on a `None` value'"

# Solution:
# This is a code bug - file incident report
# Emergency: Rollback to previous version
./scripts/emergency-rollback.sh

3. Database Connection Failure:

# Evidence:
# "Failed to acquire database connection from pool"

# Solution:
# Check database health
sudo systemctl status postgresql
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"

# If too many connections, kill idle ones:
psql $DATABASE_URL -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND state_change < now() - interval '1 hour';"

---

Network Issues

Symptom: Cannot Connect to Service

Decision Tree:

Cannot connect to service?
│
├─ Can ping server? (ping server_ip)
│  │
│  ├─ NO → Network/routing issue
│  │  └─ Solution: Check network connectivity → Escalate to network team
│  │
│  └─ YES → Can telnet to port? (telnet server_ip 50051)
│     │
│     ├─ NO → Firewall blocking
│     │  └─ Solution: Check iptables → Add firewall rule
│     │
│     └─ YES → gRPC handshake failing
│        └─ Solution: Check TLS certificates → See [TLS Issues](#tls-issues)

Firewall Issues

Diagnosis:

# Check if ports are listening
netstat -tlnp | grep -E '(50051|50052|50053)'

# Check iptables rules
sudo iptables -L -n -v | grep -E '(50051|50052|50053)'

# Test connectivity from client
telnet trading-service-host 50051

Solution:

# Allow gRPC ports through firewall
sudo iptables -A INPUT -p tcp --dport 50051 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 50052 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 50053 -j ACCEPT

# Save rules
sudo iptables-save > /etc/iptables/rules.v4

# Verify
sudo iptables -L -n -v | grep -E '(50051|50052|50053)'

TLS Issues

Symptom:

ERROR: SSL/TLS handshake failed
Error: certificate verify failed

Diagnosis:

# Check certificate validity
openssl x509 -in /etc/foxhunt/certs/server.crt -noout -dates

# Check certificate chain
openssl verify -CAfile /etc/foxhunt/certs/ca.crt /etc/foxhunt/certs/server.crt

# Test TLS connection
openssl s_client -connect localhost:50051 -CAfile /etc/foxhunt/certs/ca.crt

Solution:

# Regenerate certificates if expired
./scripts/generate-certificates.sh

# Update service configuration with new certificates
vim .env.production
# TLS_CERT_PATH=/etc/foxhunt/certs/server.crt
# TLS_KEY_PATH=/etc/foxhunt/certs/server.key

# Restart services
./scripts/start-all-services.sh

---

Database Performance

Symptom: Slow Queries / High Latency

Decision Tree:

Slow database queries?
│
├─ Check active queries: SELECT * FROM pg_stat_activity WHERE state != 'idle';
│  │
│  ├─ Long-running queries? (duration > 1s)
│  │  └─ Solution: Identify slow queries → Optimize/kill
│  │
│  ├─ Many idle connections? (state = 'idle')
│  │  └─ Solution: Terminate idle connections → Reduce connection pool
│  │
│  └─ Connection pool exhausted?
│     └─ Solution: Increase pool size OR reduce query concurrency

Slow Query Diagnosis

Diagnosis:

# Find slow queries
psql $DATABASE_URL -c "
SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY duration DESC
LIMIT 10;
"

# Check table bloat
psql $DATABASE_URL -c "
SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 10;
"

# Check index usage
psql $DATABASE_URL -c "
SELECT schemaname, tablename, indexname, idx_scan, idx_tup_read, idx_tup_fetch
FROM pg_stat_user_indexes
WHERE idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC
LIMIT 10;
"

Solution:

# Kill long-running queries
psql $DATABASE_URL -c "SELECT pg_terminate_backend(PID);" # Replace PID

# Run VACUUM ANALYZE to update statistics
psql $DATABASE_URL -c "VACUUM ANALYZE;"

# Rebuild indexes if fragmented
psql $DATABASE_URL -c "REINDEX DATABASE foxhunt_production;"

# Add missing indexes (example)
psql $DATABASE_URL -c "CREATE INDEX CONCURRENTLY idx_orders_created_at ON orders(created_at);"

Connection Pool Exhaustion

Symptom:

ERROR: connection pool timeout
ERROR: remaining connection slots are reserved for non-replication superuser connections

Diagnosis:

# Check current connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"

# Check max connections
psql $DATABASE_URL -c "SHOW max_connections;"

# Check connections by state
psql $DATABASE_URL -c "
SELECT state, count(*)
FROM pg_stat_activity
GROUP BY state
ORDER BY count DESC;
"

Solution:

# Kill idle connections
psql $DATABASE_URL -c "
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
AND state_change < now() - interval '5 minutes';
"

# Increase max_connections (PostgreSQL config)
sudo vim /etc/postgresql/14/main/postgresql.conf
# max_connections = 200  # Increase from default 100

# Restart PostgreSQL
sudo systemctl restart postgresql

# Reduce service connection pool size (Wave 66 config)
vim .env.production
# DATABASE_POOL_SIZE=20  # Reduce from 50

---

Memory Issues

Symptom: High Memory Usage

Decision Tree:

High memory usage? (>90%)
│
├─ Check top processes: top -o %MEM
│  │
│  ├─ Service consuming most memory?
│  │  └─ Solution: Check for memory leak → Restart service → Monitor
│  │
│  ├─ PostgreSQL consuming most memory?
│  │  └─ Solution: Adjust shared_buffers → Tune memory settings
│  │
│  └─ Redis consuming most memory?
│     └─ Solution: Check cache size → Adjust TTLs → See Wave 66 thresholds

Memory Leak Investigation

Diagnosis:

# Check memory usage by process
ps aux | grep -E '(trading_service|backtesting_service|ml_training_service)' | \
    awk '{print $11, "Memory:", $4"%", "RSS:", $6/1024 "MB"}'

# Check memory growth over time
while true; do
    date >> /tmp/memory_usage.log
    ps aux | grep trading_service | awk '{print $6}' >> /tmp/memory_usage.log
    sleep 60
done

Solution:

# Emergency: Restart leaking service
systemctl restart trading_service

# Long-term: Investigate with Valgrind
valgrind --leak-check=full --log-file=/tmp/valgrind.log ./target/release/trading_service

# Check Wave 66 cache TTLs (may be too long)
grep -r "CACHE_TTL" /home/jgrusewski/Work/foxhunt/common/src/thresholds.rs

# Reduce cache sizes
vim .env.production
# POSITION_CACHE_SIZE=1000  # Reduce from default
# VAR_CACHE_SIZE=500

PostgreSQL Memory Tuning

Diagnosis:

# Check PostgreSQL memory settings
psql $DATABASE_URL -c "SHOW shared_buffers;"
psql $DATABASE_URL -c "SHOW work_mem;"
psql $DATABASE_URL -c "SHOW maintenance_work_mem;"

# Check current memory usage
free -h

Solution:

# Tune PostgreSQL memory (for 128GB system)
sudo vim /etc/postgresql/14/main/postgresql.conf

# Recommended settings:
# shared_buffers = 32GB          # 25% of total RAM
# effective_cache_size = 96GB    # 75% of total RAM
# work_mem = 64MB                # Depends on max_connections
# maintenance_work_mem = 2GB
# wal_buffers = 16MB

# Apply changes
sudo systemctl restart postgresql

---

CPU Issues

Symptom: High CPU Usage

Decision Tree:

High CPU usage? (>85%)
│
├─ Check top processes: top
│  │
│  ├─ Service using high CPU?
│  │  └─ Solution: Check for infinite loop → Profile code → Fix
│  │
│  ├─ PostgreSQL using high CPU?
│  │  └─ Solution: Check slow queries → Optimize → Add indexes
│  │
│  └─ System processes using CPU? (kernel, interrupts)
│     └─ Solution: Check for hardware issues → Escalate

CPU Profiling

Diagnosis:

# Check CPU usage by service
top -b -n 1 | grep -E '(trading_service|backtesting_service|ml_training_service)'

# Check CPU affinity (Wave 66 CPU_AFFINITY_CORES setting)
taskset -cp $(pgrep trading_service)

# Profile with perf
sudo perf record -p $(pgrep trading_service) -g -- sleep 10
sudo perf report

Solution:

# Set CPU affinity per Wave 66 configuration
taskset -cp 2,3,4,5 $(pgrep trading_service)

# Check if SIMD is enabled (Wave 66 ENABLE_SIMD)
grep "ENABLE_SIMD" .env.production

# Verify SIMD optimizations are working
./target/release/trading_service --version | grep -i simd

---

Authentication Issues

Symptom: Authentication Failures

Wave 63 Authentication Status: Designed but NOT ENABLED

Decision Tree:

Authentication failing?
│
├─ Is authentication enabled? (Check main.rs for .layer(auth_layer))
│  │
│  ├─ NO → Authentication is disabled (Wave 63 design, Wave 64+ implementation)
│  │  └─ Solution: Enable by uncommenting .layer(auth_layer) in main.rs
│  │
│  └─ YES → Check authentication logs
│     │
│     ├─ "Invalid JWT token" → Token issue
│     │  └─ Solution: Check token expiration → Regenerate token
│     │
│     ├─ "Certificate verification failed" → mTLS issue
│     │  └─ Solution: Check client certificates → See [TLS Issues](#tls-issues)
│     │
│     └─ "Rate limit exceeded" → Rate limiting triggered
│        └─ Solution: Check rate limiter config → Adjust limits

Enabling Authentication (Wave 63)

Current State:

• ✅ Authentication architecture designed (Wave 63 Agent 2)
• ✅ Implementation complete (auth_interceptor.rs)
• ✅ Tonic upgrade complete (Wave 64 Agent 1 - enables HTTP-layer middleware)
• ⚠️ NOT ENABLED - Requires uncommenting .layer(auth_layer)

To Enable:

// File: /home/jgrusewski/Work/foxhunt/services/trading_service/src/main.rs
// Around line 315:

let server = Server::builder()
    .tls_config(tls_config.to_server_tls_config())?
    .layer(auth_layer)  // <- UNCOMMENT THIS LINE
    .add_service(trading_service_server)
    .add_service(risk_service_server)
    .add_service(ml_service_server)
    .add_service(monitoring_service_server)
    .serve_with_shutdown(addr, shutdown_signal());

Rebuild and Deploy:

# Rebuild with authentication enabled
cargo build --release --bin trading_service

# Deploy
./scripts/start-all-services.sh

# Test authentication
grpcurl -H "Authorization: Bearer YOUR_JWT_TOKEN" \
    -d '{}' \
    localhost:50051 trading.TradingService/GetOrders

JWT Token Issues

Diagnosis:

# Check token expiration
echo "YOUR_JWT_TOKEN" | cut -d'.' -f2 | base64 -d | jq '.exp'

# Compare with current time
date +%s

Solution:

# Generate new JWT token (example)
./scripts/generate-jwt-token.sh --user admin --expires 86400

# Update client configuration with new token
vim /path/to/client/config.toml
# auth_token = "new_jwt_token_here"

---

Configuration Issues

Wave 66 Configuration System

Configuration Tiers:

1. Compile-Time ✅: /home/jgrusewski/Work/foxhunt/common/src/thresholds.rs
2. Runtime 📋: .env.production (requires restart)
3. Database 📋: Hot-reload (Wave 68 - not yet implemented)

Common Issues:

Missing Environment Variables

Symptom:

ERROR: Environment variable DATABASE_URL not found

Solution:

# Check Wave 66 environment template
cat .env.production.example | grep DATABASE_URL

# Copy and configure
cp .env.production.example .env.production
vim .env.production  # Fill in production values

# Verify
source .env.production
echo $DATABASE_URL

Configuration Value Out of Range

Symptom:

ERROR: Invalid value for MAX_LATENCY_US: must be between 10 and 1000

Solution:

# Check Wave 66 thresholds
grep "MAX_LATENCY" /home/jgrusewski/Work/foxhunt/common/src/thresholds.rs

# Update .env.production with valid value
vim .env.production
# MAX_LATENCY_US=50  # Within valid range

# Restart service
./scripts/start-all-services.sh

---

Integration Test Failures

Symptom: Integration Tests Won't Compile

Wave 66 Agent 12 Status: 418 unit tests passing, integration tests BLOCKED

Known Issues:

• tests/fixtures/mod.rs: Missing type imports (TliError, EventSeverity)
• tests/failure_scenario_tests.rs: 14 compilation errors
• services/ml_training_service/src/data_loader.rs: Unsafe PgPool initialization

Temporary Workaround:

# Run only unit tests (skip integration)
cargo test --workspace --lib

# Run specific crate tests
cargo test -p adaptive-strategy  # 69 tests
cargo test -p common             # 68 tests
cargo test -p trading_engine     # 281 tests

Long-Term Fix (Future Wave):

# Fix type imports in fixtures
# File: tests/fixtures/mod.rs
use common::errors::TliError;
use common::types::EventSeverity;

# Fix PgPool initialization
# File: services/ml_training_service/src/data_loader.rs
// Remove unsafe std::mem::zeroed()
// Add proper PgPool initialization

---

Emergency Escalation

Escalation Matrix

Issue Severity	Response Time	Escalation Path
P0 - Critical	<15 min	On-call engineer → Trading Ops Lead → CTO
P1 - High	<1 hour	On-call engineer → Trading Ops Lead
P2 - Medium	<4 hours	On-call engineer → Queue for business hours
P3 - Low	<24 hours	Queue for business hours

P0 - Critical Incidents

Criteria:

• Trading service completely down
• Data corruption detected
• Security breach
• Financial loss occurring

Immediate Actions:

# 1. STOP TRADING
./scripts/emergency-stop.sh "P0 INCIDENT: [reason]"

# 2. Collect diagnostic data
./scripts/collect-emergency-diagnostics.sh

# 3. Notify stakeholders
# Send alert via PagerDuty, Slack, email

# 4. Create incident report
echo "P0 INCIDENT $(date)" >> /var/log/foxhunt/incidents.log
echo "Details: [describe issue]" >> /var/log/foxhunt/incidents.log

Diagnostic Data Collection

#!/bin/bash
# /home/jgrusewski/Work/foxhunt/scripts/collect-emergency-diagnostics.sh

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
DIAG_DIR="/tmp/foxhunt_diagnostics_$TIMESTAMP"

mkdir -p "$DIAG_DIR"

echo "Collecting diagnostic data to $DIAG_DIR..."

# Service logs
cp /var/log/foxhunt/*.log "$DIAG_DIR/"

# System logs
sudo cp /var/log/syslog "$DIAG_DIR/"
sudo dmesg > "$DIAG_DIR/dmesg.log"

# Process information
ps aux > "$DIAG_DIR/processes.txt"
pgrep -a foxhunt > "$DIAG_DIR/foxhunt_processes.txt"

# System resources
free -h > "$DIAG_DIR/memory.txt"
df -h > "$DIAG_DIR/disk.txt"
top -b -n 1 > "$DIAG_DIR/top.txt"

# Network
netstat -tlnp > "$DIAG_DIR/network.txt"
sudo iptables -L -n -v > "$DIAG_DIR/firewall.txt"

# Database
psql $DATABASE_URL -c "SELECT * FROM pg_stat_activity;" > "$DIAG_DIR/db_activity.txt"

# Configuration (sanitized)
cp .env.production "$DIAG_DIR/env.txt"
sed -i 's/PASSWORD=.*/PASSWORD=***REDACTED***/g' "$DIAG_DIR/env.txt"

# Compress
tar -czf "$DIAG_DIR.tar.gz" "$DIAG_DIR/"

echo "Diagnostic data collected: $DIAG_DIR.tar.gz"

---

Appendix: Wave 66 Configuration Reference

Centralized Constants

Location: /home/jgrusewski/Work/foxhunt/common/src/thresholds.rs

Key Constants:

// Risk Management
pub const BREACH_SOFT_PCT: Decimal = Decimal::from_f64_retain(90.0);
pub const BREACH_HARD_PCT: Decimal = Decimal::from_f64_retain(100.0);
pub const BREACH_CRITICAL_PCT: Decimal = Decimal::from_f64_retain(120.0);

// Cache TTLs
pub const POSITION_CACHE_TTL: Duration = Duration::from_secs(300);      // 5 min
pub const COMPLIANCE_CACHE_TTL: Duration = Duration::from_secs(86400);  // 24 hours
pub const VAR_CACHE_TTL: Duration = Duration::from_secs(3600);          // 1 hour

// Database
pub const QUERY_TIMEOUT: Duration = Duration::from_secs(30);
pub const CONNECTION_POOL_SIZE: u32 = 50;

// Performance
pub const MAX_LATENCY_US: u64 = 50;
pub const ENABLE_SIMD: bool = true;

Documentation: See /home/jgrusewski/Work/foxhunt/docs/CONFIGURATION_QUICK_REFERENCE.md

---

Document Version: 1.0 Wave: 67 Agent 10 - Troubleshooting Guide Maintained By: Foxhunt Operations Team Last Review: 2025-10-03

For Emergencies: Execute /home/jgrusewski/Work/foxhunt/scripts/emergency-stop.sh and escalate to On-Call Engineer.