foxhunt/docs/TROUBLESHOOTING_GUIDE.md

# Foxhunt HFT Trading System - Troubleshooting Guide

**Version**: 1.0
**Last Updated**: 2025-10-03
**Wave**: 67 - Production Operations
**Audience**: Operators, SREs, On-Call Engineers

---

## Quick Diagnosis Decision Tree

```
SERVICE NOT RESPONDING?
│
├─ YES → Is process running? (pgrep -a service_name)
│  │
│  ├─ NO → Start service
│  │      └─ Still fails? → Check logs → See [Service Startup Failures](#service-startup-failures)
│  │
│  └─ YES → Is health endpoint responding? (grpcurl health check)
│     │
│     ├─ NO → Check logs for errors → See [Service Health Failures](#service-health-failures)
│     │
│     └─ YES → Check client connectivity → See [Network Issues](#network-issues)
│
└─ NO → PERFORMANCE DEGRADATION?
   │
   ├─ YES → High latency? (>10ms P99)
   │  │
   │  ├─ YES → Check database query times → See [Database Performance](#database-performance)
   │  │       Check memory pressure → See [Memory Issues](#memory-issues)
   │  │       Check CPU usage → See [CPU Issues](#cpu-issues)
   │  │
   │  └─ NO → High error rate? (>1%)
   │         └─ YES → Check service logs → See [Error Analysis](#error-analysis)
   │
   └─ NO → AUTHENTICATION FAILING?
          └─ YES → Is auth enabled? → See [Authentication Issues](#authentication-issues)
```

---

## Table of Contents

1. [Service Startup Failures](#service-startup-failures)
2. [Service Health Failures](#service-health-failures)
3. [Network Issues](#network-issues)
4. [Database Performance](#database-performance)
5. [Memory Issues](#memory-issues)
6. [CPU Issues](#cpu-issues)
7. [Authentication Issues](#authentication-issues)
8. [Configuration Issues](#configuration-issues)
9. [Integration Test Failures](#integration-test-failures)
10. [Emergency Escalation](#emergency-escalation)

---

## Service Startup Failures

### Symptom: Service Won't Start

**Decision Tree**:
```
Service won't start?
│
├─ Check logs: tail -100 /var/log/foxhunt/SERVICE.log
│  │
│  ├─ "Address already in use" → Port conflict
│  │  └─ Solution: lsof -i :PORT → kill PID → retry
│  │
│  ├─ "Database connection failed" → Database issue
│  │  └─ Solution: psql $DATABASE_URL -c "SELECT 1;" → Fix DB → retry
│  │
│  ├─ "Configuration file not found" → Config missing
│  │  └─ Solution: Check .env.production exists → See [Configuration Issues](#configuration-issues)
│  │
│  └─ "Permission denied" → File permissions
│     └─ Solution: chown foxhunt_service:foxhunt_service /path/to/service → retry
```

### Common Startup Errors

#### Error: "Address already in use"

**Symptom**:
```
ERROR: Failed to bind to address 0.0.0.0:50051
Error: Address already in use (os error 98)
```

**Diagnosis**:
```bash
# Find what's using the port
lsof -i :50051

# Check if old process is still running
pgrep -a trading_service
```

**Solution**:
```bash
# Kill old process
OLD_PID=$(lsof -t -i :50051)
kill -TERM $OLD_PID

# Wait for clean shutdown
sleep 5

# Force kill if still running
kill -9 $OLD_PID

# Restart service
./scripts/start-all-services.sh
```

#### Error: "Database connection refused"

**Symptom**:
```
ERROR: Failed to connect to database
Error: Connection refused (os error 111)
```

**Diagnosis**:
```bash
# Check PostgreSQL status
sudo systemctl status postgresql

# Check database connectivity
psql $DATABASE_URL -c "SELECT version();"

# Check database logs
sudo tail -50 /var/log/postgresql/postgresql-14-main.log
```

**Solution**:
```bash
# Restart PostgreSQL if down
sudo systemctl restart postgresql

# Wait for PostgreSQL to be ready
sleep 10

# Verify connection
psql $DATABASE_URL -c "SELECT 1;"

# Retry service startup
./scripts/start-all-services.sh
```

#### Error: "Configuration file not found"

**Symptom**:
```
ERROR: Configuration file not found: /etc/foxhunt/trading_service.toml
```

**Diagnosis**:
```bash
# Check if config file exists
ls -la /etc/foxhunt/trading_service.toml

# Check environment file
ls -la .env.production

# Check file permissions
ls -la /etc/foxhunt/*.toml
```

**Solution**:
```bash
# Copy from template
sudo cp /etc/foxhunt/trading_service.toml.example /etc/foxhunt/trading_service.toml

# Set correct permissions
sudo chown foxhunt_service:foxhunt_service /etc/foxhunt/trading_service.toml

# Verify Wave 66 environment template exists
ls -la .env.production.example

# Copy and configure
cp .env.production.example .env.production
vim .env.production  # Fill in production values
```

---

## Service Health Failures

### Symptom: Health Check Fails

**Decision Tree**:
```
Health check fails?
│
├─ Check service logs: tail -100 /var/log/foxhunt/SERVICE.log
│  │
│  ├─ Recent PANIC/FATAL? → Service crashed
│  │  └─ Solution: Investigate crash → See [Service Crash Investigation](#service-crash-investigation)
│  │
│  ├─ "Connection pool exhausted" → Database overload
│  │  └─ Solution: Check DB connections → See [Database Performance](#database-performance)
│  │
│  ├─ "Out of memory" → Memory pressure
│  │  └─ Solution: Check memory usage → See [Memory Issues](#memory-issues)
│  │
│  └─ No recent errors → Slow response
│     └─ Solution: Check CPU/latency → See [Performance Degradation](#performance-degradation)
```

### Service Crash Investigation

**Symptom**:
```
Health check returns: "Service Unavailable"
Process not running (pgrep returns nothing)
```

**Diagnosis**:
```bash
# Check for core dumps
ls -lt /var/crash/ | head -5

# Check service logs for panic
tail -200 /var/log/foxhunt/trading_service.log | grep -A 20 "PANIC\|FATAL\|panic"

# Check system logs
sudo dmesg | tail -50 | grep -i "kill\|oom\|segfault"

# Check if OOM killed the service
sudo grep -i "killed process" /var/log/syslog | tail -10
```

**Common Crash Causes**:

1. **Out of Memory (OOM)**:
```bash
# Evidence:
sudo grep "Out of memory" /var/log/syslog

# Solution:
# 1. Reduce memory pressure (see [Memory Issues](#memory-issues))
# 2. Increase system memory
# 3. Configure OOM score to protect critical services
echo -1000 > /proc/$(pgrep trading_service)/oom_score_adj
```

2. **Panic/Unwrap on None**:
```bash
# Evidence in logs:
# "thread 'main' panicked at 'called `Option::unwrap()` on a `None` value'"

# Solution:
# This is a code bug - file incident report
# Emergency: Rollback to previous version
./scripts/emergency-rollback.sh
```

3. **Database Connection Failure**:
```bash
# Evidence:
# "Failed to acquire database connection from pool"

# Solution:
# Check database health
sudo systemctl status postgresql
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"

# If too many connections, kill idle ones:
psql $DATABASE_URL -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND state_change < now() - interval '1 hour';"
```

---

## Network Issues

### Symptom: Cannot Connect to Service

**Decision Tree**:
```
Cannot connect to service?
│
├─ Can ping server? (ping server_ip)
│  │
│  ├─ NO → Network/routing issue
│  │  └─ Solution: Check network connectivity → Escalate to network team
│  │
│  └─ YES → Can telnet to port? (telnet server_ip 50051)
│     │
│     ├─ NO → Firewall blocking
│     │  └─ Solution: Check iptables → Add firewall rule
│     │
│     └─ YES → gRPC handshake failing
│        └─ Solution: Check TLS certificates → See [TLS Issues](#tls-issues)
```

### Firewall Issues

**Diagnosis**:
```bash
# Check if ports are listening
netstat -tlnp | grep -E '(50051|50052|50053)'

# Check iptables rules
sudo iptables -L -n -v | grep -E '(50051|50052|50053)'

# Test connectivity from client
telnet trading-service-host 50051
```

**Solution**:
```bash
# Allow gRPC ports through firewall
sudo iptables -A INPUT -p tcp --dport 50051 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 50052 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 50053 -j ACCEPT

# Save rules
sudo iptables-save > /etc/iptables/rules.v4

# Verify
sudo iptables -L -n -v | grep -E '(50051|50052|50053)'
```

### TLS Issues

**Symptom**:
```
ERROR: SSL/TLS handshake failed
Error: certificate verify failed
```

**Diagnosis**:
```bash
# Check certificate validity
openssl x509 -in /etc/foxhunt/certs/server.crt -noout -dates

# Check certificate chain
openssl verify -CAfile /etc/foxhunt/certs/ca.crt /etc/foxhunt/certs/server.crt

# Test TLS connection
openssl s_client -connect localhost:50051 -CAfile /etc/foxhunt/certs/ca.crt
```

**Solution**:
```bash
# Regenerate certificates if expired
./scripts/generate-certificates.sh

# Update service configuration with new certificates
vim .env.production
# TLS_CERT_PATH=/etc/foxhunt/certs/server.crt
# TLS_KEY_PATH=/etc/foxhunt/certs/server.key

# Restart services
./scripts/start-all-services.sh
```

---

## Database Performance

### Symptom: Slow Queries / High Latency

**Decision Tree**:
```
Slow database queries?
│
├─ Check active queries: SELECT * FROM pg_stat_activity WHERE state != 'idle';
│  │
│  ├─ Long-running queries? (duration > 1s)
│  │  └─ Solution: Identify slow queries → Optimize/kill
│  │
│  ├─ Many idle connections? (state = 'idle')
│  │  └─ Solution: Terminate idle connections → Reduce connection pool
│  │
│  └─ Connection pool exhausted?
│     └─ Solution: Increase pool size OR reduce query concurrency
```

### Slow Query Diagnosis

**Diagnosis**:
```bash
# Find slow queries
psql $DATABASE_URL -c "
SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY duration DESC
LIMIT 10;
"

# Check table bloat
psql $DATABASE_URL -c "
SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 10;
"

# Check index usage
psql $DATABASE_URL -c "
SELECT schemaname, tablename, indexname, idx_scan, idx_tup_read, idx_tup_fetch
FROM pg_stat_user_indexes
WHERE idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC
LIMIT 10;
"
```

**Solution**:
```bash
# Kill long-running queries
psql $DATABASE_URL -c "SELECT pg_terminate_backend(PID);" # Replace PID

# Run VACUUM ANALYZE to update statistics
psql $DATABASE_URL -c "VACUUM ANALYZE;"

# Rebuild indexes if fragmented
psql $DATABASE_URL -c "REINDEX DATABASE foxhunt_production;"

# Add missing indexes (example)
psql $DATABASE_URL -c "CREATE INDEX CONCURRENTLY idx_orders_created_at ON orders(created_at);"
```

### Connection Pool Exhaustion

**Symptom**:
```
ERROR: connection pool timeout
ERROR: remaining connection slots are reserved for non-replication superuser connections
```

**Diagnosis**:
```bash
# Check current connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"

# Check max connections
psql $DATABASE_URL -c "SHOW max_connections;"

# Check connections by state
psql $DATABASE_URL -c "
SELECT state, count(*)
FROM pg_stat_activity
GROUP BY state
ORDER BY count DESC;
"
```

**Solution**:
```bash
# Kill idle connections
psql $DATABASE_URL -c "
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
AND state_change < now() - interval '5 minutes';
"

# Increase max_connections (PostgreSQL config)
sudo vim /etc/postgresql/14/main/postgresql.conf
# max_connections = 200  # Increase from default 100

# Restart PostgreSQL
sudo systemctl restart postgresql

# Reduce service connection pool size (Wave 66 config)
vim .env.production
# DATABASE_POOL_SIZE=20  # Reduce from 50
```

---

## Memory Issues

### Symptom: High Memory Usage

**Decision Tree**:
```
High memory usage? (>90%)
│
├─ Check top processes: top -o %MEM
│  │
│  ├─ Service consuming most memory?
│  │  └─ Solution: Check for memory leak → Restart service → Monitor
│  │
│  ├─ PostgreSQL consuming most memory?
│  │  └─ Solution: Adjust shared_buffers → Tune memory settings
│  │
│  └─ Redis consuming most memory?
│     └─ Solution: Check cache size → Adjust TTLs → See Wave 66 thresholds
```

### Memory Leak Investigation

**Diagnosis**:
```bash
# Check memory usage by process
ps aux | grep -E '(trading_service|backtesting_service|ml_training_service)' | \
    awk '{print $11, "Memory:", $4"%", "RSS:", $6/1024 "MB"}'

# Check memory growth over time
while true; do
    date >> /tmp/memory_usage.log
    ps aux | grep trading_service | awk '{print $6}' >> /tmp/memory_usage.log
    sleep 60
done
```

**Solution**:
```bash
# Emergency: Restart leaking service
systemctl restart trading_service

# Long-term: Investigate with Valgrind
valgrind --leak-check=full --log-file=/tmp/valgrind.log ./target/release/trading_service

# Check Wave 66 cache TTLs (may be too long)
grep -r "CACHE_TTL" /home/jgrusewski/Work/foxhunt/common/src/thresholds.rs

# Reduce cache sizes
vim .env.production
# POSITION_CACHE_SIZE=1000  # Reduce from default
# VAR_CACHE_SIZE=500
```

### PostgreSQL Memory Tuning

**Diagnosis**:
```bash
# Check PostgreSQL memory settings
psql $DATABASE_URL -c "SHOW shared_buffers;"
psql $DATABASE_URL -c "SHOW work_mem;"
psql $DATABASE_URL -c "SHOW maintenance_work_mem;"

# Check current memory usage
free -h
```

**Solution**:
```bash
# Tune PostgreSQL memory (for 128GB system)
sudo vim /etc/postgresql/14/main/postgresql.conf

# Recommended settings:
# shared_buffers = 32GB          # 25% of total RAM
# effective_cache_size = 96GB    # 75% of total RAM
# work_mem = 64MB                # Depends on max_connections
# maintenance_work_mem = 2GB
# wal_buffers = 16MB

# Apply changes
sudo systemctl restart postgresql
```

---

## CPU Issues

### Symptom: High CPU Usage

**Decision Tree**:
```
High CPU usage? (>85%)
│
├─ Check top processes: top
│  │
│  ├─ Service using high CPU?
│  │  └─ Solution: Check for infinite loop → Profile code → Fix
│  │
│  ├─ PostgreSQL using high CPU?
│  │  └─ Solution: Check slow queries → Optimize → Add indexes
│  │
│  └─ System processes using CPU? (kernel, interrupts)
│     └─ Solution: Check for hardware issues → Escalate
```

### CPU Profiling

**Diagnosis**:
```bash
# Check CPU usage by service
top -b -n 1 | grep -E '(trading_service|backtesting_service|ml_training_service)'

# Check CPU affinity (Wave 66 CPU_AFFINITY_CORES setting)
taskset -cp $(pgrep trading_service)

# Profile with perf
sudo perf record -p $(pgrep trading_service) -g -- sleep 10
sudo perf report
```

**Solution**:
```bash
# Set CPU affinity per Wave 66 configuration
taskset -cp 2,3,4,5 $(pgrep trading_service)

# Check if SIMD is enabled (Wave 66 ENABLE_SIMD)
grep "ENABLE_SIMD" .env.production

# Verify SIMD optimizations are working
./target/release/trading_service --version | grep -i simd
```

---

## Authentication Issues

### Symptom: Authentication Failures

**Wave 63 Authentication Status**: Designed but **NOT ENABLED**

**Decision Tree**:
```
Authentication failing?
│
├─ Is authentication enabled? (Check main.rs for .layer(auth_layer))
│  │
│  ├─ NO → Authentication is disabled (Wave 63 design, Wave 64+ implementation)
│  │  └─ Solution: Enable by uncommenting .layer(auth_layer) in main.rs
│  │
│  └─ YES → Check authentication logs
│     │
│     ├─ "Invalid JWT token" → Token issue
│     │  └─ Solution: Check token expiration → Regenerate token
│     │
│     ├─ "Certificate verification failed" → mTLS issue
│     │  └─ Solution: Check client certificates → See [TLS Issues](#tls-issues)
│     │
│     └─ "Rate limit exceeded" → Rate limiting triggered
│        └─ Solution: Check rate limiter config → Adjust limits
```

### Enabling Authentication (Wave 63)

**Current State**:
- ✅ Authentication architecture designed (Wave 63 Agent 2)
- ✅ Implementation complete (`auth_interceptor.rs`)
- ✅ Tonic upgrade complete (Wave 64 Agent 1 - enables HTTP-layer middleware)
- ⚠️ **NOT ENABLED** - Requires uncommenting `.layer(auth_layer)`

**To Enable**:
```rust
// File: /home/jgrusewski/Work/foxhunt/services/trading_service/src/main.rs
// Around line 315:

let server = Server::builder()
    .tls_config(tls_config.to_server_tls_config())?
    .layer(auth_layer)  // <- UNCOMMENT THIS LINE
    .add_service(trading_service_server)
    .add_service(risk_service_server)
    .add_service(ml_service_server)
    .add_service(monitoring_service_server)
    .serve_with_shutdown(addr, shutdown_signal());
```

**Rebuild and Deploy**:
```bash
# Rebuild with authentication enabled
cargo build --release --bin trading_service

# Deploy
./scripts/start-all-services.sh

# Test authentication
grpcurl -H "Authorization: Bearer YOUR_JWT_TOKEN" \
    -d '{}' \
    localhost:50051 trading.TradingService/GetOrders
```

### JWT Token Issues

**Diagnosis**:
```bash
# Check token expiration
echo "YOUR_JWT_TOKEN" | cut -d'.' -f2 | base64 -d | jq '.exp'

# Compare with current time
date +%s
```

**Solution**:
```bash
# Generate new JWT token (example)
./scripts/generate-jwt-token.sh --user admin --expires 86400

# Update client configuration with new token
vim /path/to/client/config.toml
# auth_token = "new_jwt_token_here"
```

---

## Configuration Issues

### Wave 66 Configuration System

**Configuration Tiers**:
1. **Compile-Time** ✅: `/home/jgrusewski/Work/foxhunt/common/src/thresholds.rs`
2. **Runtime** 📋: `.env.production` (requires restart)
3. **Database** 📋: Hot-reload (Wave 68 - not yet implemented)

**Common Issues**:

#### Missing Environment Variables

**Symptom**:
```
ERROR: Environment variable DATABASE_URL not found
```

**Solution**:
```bash
# Check Wave 66 environment template
cat .env.production.example | grep DATABASE_URL

# Copy and configure
cp .env.production.example .env.production
vim .env.production  # Fill in production values

# Verify
source .env.production
echo $DATABASE_URL
```

#### Configuration Value Out of Range

**Symptom**:
```
ERROR: Invalid value for MAX_LATENCY_US: must be between 10 and 1000
```

**Solution**:
```bash
# Check Wave 66 thresholds
grep "MAX_LATENCY" /home/jgrusewski/Work/foxhunt/common/src/thresholds.rs

# Update .env.production with valid value
vim .env.production
# MAX_LATENCY_US=50  # Within valid range

# Restart service
./scripts/start-all-services.sh
```

---

## Integration Test Failures

### Symptom: Integration Tests Won't Compile

**Wave 66 Agent 12 Status**: 418 unit tests passing, integration tests **BLOCKED**

**Known Issues**:
- `tests/fixtures/mod.rs`: Missing type imports (TliError, EventSeverity)
- `tests/failure_scenario_tests.rs`: 14 compilation errors
- `services/ml_training_service/src/data_loader.rs`: Unsafe PgPool initialization

**Temporary Workaround**:
```bash
# Run only unit tests (skip integration)
cargo test --workspace --lib

# Run specific crate tests
cargo test -p adaptive-strategy  # 69 tests
cargo test -p common             # 68 tests
cargo test -p trading_engine     # 281 tests
```

**Long-Term Fix** (Future Wave):
```bash
# Fix type imports in fixtures
# File: tests/fixtures/mod.rs
use common::errors::TliError;
use common::types::EventSeverity;

# Fix PgPool initialization
# File: services/ml_training_service/src/data_loader.rs
// Remove unsafe std::mem::zeroed()
// Add proper PgPool initialization
```

---

## Emergency Escalation

### Escalation Matrix

| Issue Severity | Response Time | Escalation Path |
|----------------|---------------|-----------------|
| **P0 - Critical** | <15 min | On-call engineer → Trading Ops Lead → CTO |
| **P1 - High** | <1 hour | On-call engineer → Trading Ops Lead |
| **P2 - Medium** | <4 hours | On-call engineer → Queue for business hours |
| **P3 - Low** | <24 hours | Queue for business hours |

### P0 - Critical Incidents

**Criteria**:
- Trading service completely down
- Data corruption detected
- Security breach
- Financial loss occurring

**Immediate Actions**:
```bash
# 1. STOP TRADING
./scripts/emergency-stop.sh "P0 INCIDENT: [reason]"

# 2. Collect diagnostic data
./scripts/collect-emergency-diagnostics.sh

# 3. Notify stakeholders
# Send alert via PagerDuty, Slack, email

# 4. Create incident report
echo "P0 INCIDENT $(date)" >> /var/log/foxhunt/incidents.log
echo "Details: [describe issue]" >> /var/log/foxhunt/incidents.log
```

### Diagnostic Data Collection

```bash
#!/bin/bash
# /home/jgrusewski/Work/foxhunt/scripts/collect-emergency-diagnostics.sh

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
DIAG_DIR="/tmp/foxhunt_diagnostics_$TIMESTAMP"

mkdir -p "$DIAG_DIR"

echo "Collecting diagnostic data to $DIAG_DIR..."

# Service logs
cp /var/log/foxhunt/*.log "$DIAG_DIR/"

# System logs
sudo cp /var/log/syslog "$DIAG_DIR/"
sudo dmesg > "$DIAG_DIR/dmesg.log"

# Process information
ps aux > "$DIAG_DIR/processes.txt"
pgrep -a foxhunt > "$DIAG_DIR/foxhunt_processes.txt"

# System resources
free -h > "$DIAG_DIR/memory.txt"
df -h > "$DIAG_DIR/disk.txt"
top -b -n 1 > "$DIAG_DIR/top.txt"

# Network
netstat -tlnp > "$DIAG_DIR/network.txt"
sudo iptables -L -n -v > "$DIAG_DIR/firewall.txt"

# Database
psql $DATABASE_URL -c "SELECT * FROM pg_stat_activity;" > "$DIAG_DIR/db_activity.txt"

# Configuration (sanitized)
cp .env.production "$DIAG_DIR/env.txt"
sed -i 's/PASSWORD=.*/PASSWORD=***REDACTED***/g' "$DIAG_DIR/env.txt"

# Compress
tar -czf "$DIAG_DIR.tar.gz" "$DIAG_DIR/"

echo "Diagnostic data collected: $DIAG_DIR.tar.gz"
```

---

## Appendix: Wave 66 Configuration Reference

### Centralized Constants

**Location**: `/home/jgrusewski/Work/foxhunt/common/src/thresholds.rs`

**Key Constants**:
```rust
// Risk Management
pub const BREACH_SOFT_PCT: Decimal = Decimal::from_f64_retain(90.0);
pub const BREACH_HARD_PCT: Decimal = Decimal::from_f64_retain(100.0);
pub const BREACH_CRITICAL_PCT: Decimal = Decimal::from_f64_retain(120.0);

// Cache TTLs
pub const POSITION_CACHE_TTL: Duration = Duration::from_secs(300);      // 5 min
pub const COMPLIANCE_CACHE_TTL: Duration = Duration::from_secs(86400);  // 24 hours
pub const VAR_CACHE_TTL: Duration = Duration::from_secs(3600);          // 1 hour

// Database
pub const QUERY_TIMEOUT: Duration = Duration::from_secs(30);
pub const CONNECTION_POOL_SIZE: u32 = 50;

// Performance
pub const MAX_LATENCY_US: u64 = 50;
pub const ENABLE_SIMD: bool = true;
```

**Documentation**: See `/home/jgrusewski/Work/foxhunt/docs/CONFIGURATION_QUICK_REFERENCE.md`

---

**Document Version**: 1.0
**Wave**: 67 Agent 10 - Troubleshooting Guide
**Maintained By**: Foxhunt Operations Team
**Last Review**: 2025-10-03

**For Emergencies**: Execute `/home/jgrusewski/Work/foxhunt/scripts/emergency-stop.sh` and escalate to On-Call Engineer.