foxhunt/docs/archive/historical/HEALTH_CHECK_README.md

Foxhunt HFT Health Check Tools

Wave 75 Agent 6 deliverables for comprehensive service health validation.

Quick Start

# Run quick health check (13 checks, <10 seconds)
./quick_health_check.sh

# Run comprehensive health check (35+ checks, detailed logging)
./health_check.sh

# View current system status
cat health_status_summary.txt

# Generate development TLS certificates (if services are failing)
./generate_dev_certs.sh

Tools Overview

1. Quick Health Check (quick_health_check.sh)

Purpose: Fast validation for CI/CD and monitoring

Features:

• 13 essential health checks
• Completes in <10 seconds
• Clean exit codes (0=healthy, 1=degraded, 2=unhealthy)
• Color-coded output
• No external dependencies (uses docker exec)

Checks:

• 4 gRPC services (ports 50050-50053)
• 5 infrastructure services (PostgreSQL, Redis, Vault, Prometheus, Grafana)
• Docker container health
• Service process status

Usage:

./quick_health_check.sh
echo $?  # Check exit code

Output Example:

=== Foxhunt HFT Quick Health Check ===

[1/4] Checking gRPC Services...
  ✓ Trading Service (port 50051)
  ✗ Backtesting Service (port 50052) - NOT RESPONDING

[2/4] Checking Infrastructure Services...
  ✓ PostgreSQL (port 5433)
  ✓ Redis (port 6380)
  ✓ Vault (port 8200) - UNSEALED
  ✓ Prometheus (port 9099)
  ✓ Grafana (port 3000)

=== Summary ===
Total Checks: 13
Passed: 10
Warnings: 0
Failed: 3
Overall Status: DEGRADED

2. Comprehensive Health Check (health_check.sh)

Purpose: Detailed system validation with logging

Features:

• 35+ comprehensive checks
• Detailed logging to logs/health_check_YYYYMMDD_HHMMSS.log
• Prerequisite verification
• Infrastructure service validation
• gRPC service health checks
• Docker container monitoring
• Process resource tracking
• Inter-service communication tests
• Hot-reload validation
• Service log error scanning

Usage:

./health_check.sh

# View latest log
ls -lt logs/health_check_*.log | head -1
tail -100 logs/health_check_20251003_151333.log

Checks Include:

1. Prerequisites: grpcurl, psql, curl, jq, docker
2. Docker Containers: Health status, unhealthy detection
3. Infrastructure Services:
   • PostgreSQL (connection, table count)
   • Redis (ping, memory usage)
   • Vault (health, sealed status)
   • InfluxDB (optional)
   • Prometheus (health endpoint)
   • Grafana (API health, database status)
4. gRPC Services: Port listening, service list, health endpoint
5. Service Processes: Running status, CPU/memory usage
6. System Resources: CPU, memory, disk usage
7. Inter-Service Communication: API Gateway routing
8. Hot-Reload: PostgreSQL NOTIFY/LISTEN, config_settings table
9. Service Logs: Error scanning

3. TLS Certificate Generator (generate_dev_certs.sh)

Purpose: Generate development TLS certificates for gRPC services

Features:

• Generates CA certificate
• Generates server certificates with SAN (localhost + service names)
• Generates client certificates (for mutual TLS)
• Proper file permissions
• Can run as sudo (for /etc/foxhunt) or locally (./certs_dev)

Usage:

# Generate in /etc/foxhunt/certs (requires sudo)
sudo ./generate_dev_certs.sh

# Generate in local directory (no sudo)
./generate_dev_certs.sh
# Then manually copy to /etc/foxhunt/certs

Generated Files:

• ca.crt - Certificate Authority certificate
• ca.key - Certificate Authority private key
• server.crt - Server certificate (valid for localhost + service names)
• server.key - Server private key
• client.crt - Client certificate (for mutual TLS)
• client.key - Client private key

Subject Alternative Names (SAN):

• localhost
• trading.foxhunt.local
• backtesting.foxhunt.local
• ml-training.foxhunt.local
• api-gateway.foxhunt.local
• 127.0.0.1
• 0.0.0.0

4. Health Status Summary (health_status_summary.txt)

Purpose: Visual summary of current system health

View:

cat health_status_summary.txt

Exit Codes

All health check scripts use standard exit codes:

• 0 - HEALTHY: All checks passed
• 1 - DEGRADED: Some checks failed (<5 failures)
• 2 - UNHEALTHY: Many checks failed (≥5 failures)

Integration with CI/CD

GitHub Actions

name: Health Check
on: [push, pull_request]

jobs:
  health_check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Start Infrastructure
        run: docker-compose up -d postgres redis vault

      - name: Run Quick Health Check
        run: ./quick_health_check.sh

      - name: Upload Health Logs
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: health-logs
          path: logs/health_check_*.log

Kubernetes Probes

livenessProbe:
  exec:
    command:
    - grpcurl
    - -plaintext
    - localhost:50051
    - grpc.health.v1.Health/Check
  initialDelaySeconds: 10
  periodSeconds: 30

readinessProbe:
  exec:
    command:
    - grpcurl
    - -plaintext
    - localhost:50051
    - grpc.health.v1.Health/Check
  initialDelaySeconds: 5
  periodSeconds: 10

Cron Monitoring

# Add to crontab for continuous monitoring
*/5 * * * * /path/to/foxhunt/quick_health_check.sh || \
    echo "Health check failed!" | mail -s "Foxhunt Health Alert" ops@company.com

Common Issues and Fixes

Issue: All gRPC services failing with TLS errors

Symptom:

Failed to read certificate file: /etc/foxhunt/certs/server.crt
No such file or directory (os error 2)

Fix:

# Generate development certificates
sudo ./generate_dev_certs.sh

# Restart services
sudo systemctl restart foxhunt-*

# Verify
./quick_health_check.sh

Issue: PostgreSQL connection refused

Symptom:

[FAIL] PostgreSQL is NOT healthy

Fix:

# Check Docker container
docker ps | grep postgres

# Check credentials (from docker-compose.yml)
PGPASSWORD=test_password psql -h localhost -p 5433 -U foxhunt_test -d foxhunt_test

# Verify port mapping
docker port api_gateway_test_postgres

Issue: Redis not responding

Symptom:

[FAIL] Redis is NOT healthy

Fix:

# Check Docker container
docker ps | grep redis

# Test via Docker
docker exec api_gateway_test_redis redis-cli PING

# Check port mapping
docker port api_gateway_test_redis

Issue: Vault sealed

Symptom:

[WARN] Vault is healthy but SEALED

Fix:

# Unseal Vault (requires unseal keys)
vault operator unseal <unseal_key_1>
vault operator unseal <unseal_key_2>
vault operator unseal <unseal_key_3>

# Check status
vault status

File Locations

/home/jgrusewski/Work/foxhunt/
├── health_check.sh                    # Comprehensive health check
├── quick_health_check.sh              # Fast health check
├── generate_dev_certs.sh              # TLS certificate generator
├── health_status_summary.txt          # Visual status summary
├── HEALTH_CHECK_README.md             # This file
└── logs/
    ├── health_check_*.log             # Detailed health check logs
    ├── trading_service.log            # Service logs
    ├── backtesting_service.log
    └── ml_training_service.log

Documentation

Comprehensive health validation report:

• docs/WAVE75_AGENT6_HEALTH_VALIDATION.md - Full analysis, root causes, remediation

Service Ports Reference

gRPC Application Services

• 50050 - API Gateway
• 50051 - Trading Service
• 50052 - Backtesting Service
• 50053 - ML Training Service

Infrastructure Services

• 5433 - PostgreSQL (mapped from 5432)
• 6380 - Redis (mapped from 6379)
• 8200 - Vault
• 8086 - InfluxDB (optional)
• 9099 - Prometheus (mapped from 9090)
• 3000 - Grafana

Monitoring Exporters

• 9187 - PostgreSQL Exporter
• 9121 - Redis Exporter
• 9100 - Node Exporter
• 9093 - AlertManager

Best Practices

1. Run quick health check after every deployment

   ./quick_health_check.sh || exit 1
   

2. Monitor health check logs for patterns

   grep -i "FAIL" logs/health_check_*.log | sort | uniq -c
   

3. Set up automated alerts for health failures

   # Cron job example
   */5 * * * * /path/to/quick_health_check.sh || notify-send "Health check failed"
   

4. Include health checks in pre-deployment validation

   # In deployment script
   ./health_check.sh
   if [ $? -ne 0 ]; then
       echo "Pre-deployment health check failed!"
       exit 1
   fi
   

5. Use comprehensive checks for root cause analysis

   # When issues occur
   ./health_check.sh > /tmp/detailed_health.log 2>&1
   cat /tmp/detailed_health.log | less
   

Support

For issues or questions:

1. Check docs/WAVE75_AGENT6_HEALTH_VALIDATION.md for detailed analysis
2. Review service logs in logs/ directory
3. Run ./health_check.sh for comprehensive diagnostics

Version History

• 2025-10-03 - Initial release (Wave 75 Agent 6)
  • Comprehensive health check script (35+ checks)
  • Quick health check script (13 checks)
  • TLS certificate generator
  • Complete documentation and remediation guide