foxhunt/tests/e2e_helpers/INFRASTRUCTURE_VALIDATION.md

Infrastructure Health Check Validation

Overview

The infrastructure_health_check.sh script validates all Docker Compose infrastructure services required for the Foxhunt HFT trading system. This script is designed to be run by other agents and developers before deploying or testing services.

Usage

# Run from project root
./tests/e2e_helpers/infrastructure_health_check.sh

# Check exit code
echo $?
# 0 = all services healthy
# 1 = critical services down (BLOCKER)
# 2 = optional services down (WARNING)

Services Validated

Critical Infrastructure (Exit 1 if down)

1. PostgreSQL (TimescaleDB) - Port 5432

   • Container: foxhunt-postgres
   • Checks: Container health + connection test
   • Required for: All services (authentication, orders, positions, state)

2. Redis - Port 6379

   • Container: foxhunt-redis
   • Checks: Container health + PING command
   • Required for: Caching, session management, rate limiting

3. Vault - Port 8200

   • Container: foxhunt-vault
   • Checks: Container health + initialized + unsealed status
   • Required for: Secrets management, API keys, credentials

Monitoring Infrastructure (Exit 2 if down)

4. Prometheus - Port 9090

   • Container: foxhunt-prometheus
   • Checks: Container health + HTTP endpoint
   • Used for: Metrics collection, alerting

5. Grafana - Port 3000

   • Container: foxhunt-grafana
   • Checks: Container health + API health endpoint
   • Used for: Dashboards, visualization
   • Credentials: admin / foxhunt123

6. InfluxDB - Port 8086 (OPTIONAL)

   • Container: foxhunt-influxdb
   • Checks: Container health + HTTP endpoint
   • Used for: Time-series metrics (not currently deployed)
   • Status: Defined in docker-compose.yml but not started

Application Services (Exit 2 if down)

7. API Gateway - Port 50051

   • Container: foxhunt-api-gateway
   • Checks: Container health
   • Purpose: Authentication, routing, rate limiting

8. Trading Service - Port 50052

   • Container: foxhunt-trading-service
   • Checks: Container health
   • Purpose: Core trading logic, order execution

9. Backtesting Service - Port 50053

   • Container: foxhunt-backtesting-service
   • Checks: Container health
   • Purpose: Strategy testing, performance analytics

10. ML Training Service - Port 50054

    • Container: foxhunt-ml-training-service
    • Checks: Container health
    • Purpose: Model training, feature engineering

Storage Services (Exit 2 if down)

11. MinIO - Ports 9000 (API), 9001 (Console)
    • Container: foxhunt-minio
    • Checks: Container health + HTTP endpoint
    • Purpose: S3-compatible storage for E2E testing
    • Credentials: foxhunt_test / foxhunt_test_password

Validation Results

Current Status (2025-10-12)

✓ All 17 checks passed
✓ Zero critical failures
✓ Zero optional failures
✓ System ready for deployment

Service Health Matrix

Service	Status	Port	Container Health	Connectivity
PostgreSQL	✅ HEALTHY	5432	✅ healthy	✅ OK
Redis	✅ HEALTHY	6379	✅ healthy	✅ PONG
Vault	✅ HEALTHY	8200	✅ healthy	✅ unsealed
Prometheus	✅ HEALTHY	9090	✅ healthy	✅ OK
Grafana	✅ HEALTHY	3000	✅ healthy	✅ API OK
InfluxDB	⚠️ NOT DEPLOYED	8086	N/A	N/A
API Gateway	✅ HEALTHY	50051	✅ healthy	N/A
Trading Service	✅ HEALTHY	50052	✅ healthy	N/A
Backtesting Service	✅ HEALTHY	50053	✅ healthy	N/A
ML Training Service	✅ HEALTHY	50054	✅ healthy	N/A
MinIO	✅ HEALTHY	9000/9001	✅ healthy	✅ OK

Troubleshooting

Critical Service Down

If a critical service (PostgreSQL, Redis, Vault) is down:

# Check logs
docker-compose logs <service_name>

# Restart specific service
docker-compose restart <service_name>

# Restart all infrastructure
docker-compose restart postgres redis vault

# Full rebuild if needed
docker-compose down
docker-compose up -d

PostgreSQL Connection Failed

# Verify container is healthy
docker-compose ps postgres

# Check PostgreSQL logs
docker-compose logs postgres

# Test connection manually
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c 'SELECT 1'

# Verify migrations
cargo sqlx migrate run

Redis Connection Failed

# Verify container is healthy
docker-compose ps redis

# Check Redis logs
docker-compose logs redis

# Test connection manually
docker exec foxhunt-redis redis-cli ping

# Check memory usage
docker exec foxhunt-redis redis-cli INFO memory

Vault Sealed/Not Initialized

# Check Vault status
curl http://localhost:8200/v1/sys/health | jq .

# Vault is running in dev mode and should auto-unseal
# If sealed, restart Vault
docker-compose restart vault

# Verify dev token
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=foxhunt-dev-root
docker exec foxhunt-vault vault status

InfluxDB Not Running

InfluxDB is defined in docker-compose.yml but not currently started. To enable:

# Start InfluxDB
docker-compose up -d influxdb

# Verify health
docker-compose ps influxdb
curl http://localhost:8086/health

# Access web UI
open http://localhost:8086
# Credentials: foxhunt / foxhunt_dev_password

Application Service Down

# Check service logs
docker-compose logs api_gateway
docker-compose logs trading_service
docker-compose logs backtesting_service
docker-compose logs ml_training_service

# Restart service
docker-compose restart <service_name>

# Rebuild service if needed
docker-compose up -d --build <service_name>

Integration with Other Scripts

This health check script is designed to be used by other testing scripts:

#!/bin/bash
# Example: E2E test runner

# Validate infrastructure first
./tests/e2e_helpers/infrastructure_health_check.sh
HEALTH_EXIT=$?

if [ $HEALTH_EXIT -eq 1 ]; then
    echo "CRITICAL: Infrastructure not ready"
    exit 1
elif [ $HEALTH_EXIT -eq 2 ]; then
    echo "WARNING: Some optional services down, continuing..."
fi

# Proceed with tests
cargo test --test e2e_trading_integration

Manual Verification Commands

PostgreSQL

psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
\dt  # List tables
\q   # Quit

Redis

docker exec -it foxhunt-redis redis-cli
PING  # Should return PONG
INFO
QUIT

Vault

curl http://localhost:8200/v1/sys/health | jq .
# Should show: initialized=true, sealed=false

Prometheus

# Check targets
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

# Web UI
open http://localhost:9090

Grafana

# API health
curl http://localhost:3000/api/health | jq .

# Web UI
open http://localhost:3000
# Login: admin / foxhunt123

MinIO

# API health
curl http://localhost:9000/minio/health/live

# Console UI
open http://localhost:9001
# Login: foxhunt_test / foxhunt_test_password

Performance Benchmarks

From Wave 137 validation (Agent 314):

• Health check duration: ~2-3 seconds
• Container health checks: 11 containers validated
• Connectivity tests: 7 services tested
• Total checks: 17 validation points
• Success rate: 100% (17/17 passed)
• Zero false positives: All checks accurate

Exit Codes Reference

Exit Code	Status	Meaning	Action Required
0	SUCCESS	All services healthy	Proceed with deployment/testing
1	CRITICAL	Critical services down	Fix PostgreSQL/Redis/Vault before proceeding
2	WARNING	Optional services down	Can proceed but monitoring limited

Maintenance Notes

Adding New Services

To add a new service to validation:

1. Determine if service is critical or optional
2. Add check in appropriate section (2, 3, 4, or 5)
3. Use check_docker_container for container health
4. Add connectivity test if applicable
5. Update this documentation

Modifying Critical Services

If a service changes from optional to critical:

1. Change last parameter of check_docker_container from "false" to "true"
2. Update documentation in "Services Validated" section
3. Update "Service Health Matrix" table

Removing Services

If a service is deprecated:

1. Remove check from script
2. Update documentation
3. Update exit code expectations in dependent scripts

Related Documentation

• /home/jgrusewski/Work/foxhunt/CLAUDE.md - System architecture and credentials
• /home/jgrusewski/Work/foxhunt/docker-compose.yml - Service definitions
• /home/jgrusewski/Work/foxhunt/WAVE_137_VALIDATION_SUMMARY.md - E2E validation results

Last Updated

• Date: 2025-10-12
• Agent: Agent 314
• Wave: 137
• Status: ALL SERVICES VALIDATED ✅