foxhunt/HEALTH_CHECK_README.md

# Foxhunt HFT Health Check Tools

Wave 75 Agent 6 deliverables for comprehensive service health validation.

## Quick Start

```bash
# Run quick health check (13 checks, <10 seconds)
./quick_health_check.sh

# Run comprehensive health check (35+ checks, detailed logging)
./health_check.sh

# View current system status
cat health_status_summary.txt

# Generate development TLS certificates (if services are failing)
./generate_dev_certs.sh
```

## Tools Overview

### 1. Quick Health Check (`quick_health_check.sh`)

**Purpose**: Fast validation for CI/CD and monitoring

**Features**:
- 13 essential health checks
- Completes in <10 seconds
- Clean exit codes (0=healthy, 1=degraded, 2=unhealthy)
- Color-coded output
- No external dependencies (uses docker exec)

**Checks**:
- 4 gRPC services (ports 50050-50053)
- 5 infrastructure services (PostgreSQL, Redis, Vault, Prometheus, Grafana)
- Docker container health
- Service process status

**Usage**:
```bash
./quick_health_check.sh
echo $?  # Check exit code
```

**Output Example**:
```
=== Foxhunt HFT Quick Health Check ===

[1/4] Checking gRPC Services...
  ✓ Trading Service (port 50051)
  ✗ Backtesting Service (port 50052) - NOT RESPONDING

[2/4] Checking Infrastructure Services...
  ✓ PostgreSQL (port 5433)
  ✓ Redis (port 6380)
  ✓ Vault (port 8200) - UNSEALED
  ✓ Prometheus (port 9099)
  ✓ Grafana (port 3000)

=== Summary ===
Total Checks: 13
Passed: 10
Warnings: 0
Failed: 3
Overall Status: DEGRADED
```

### 2. Comprehensive Health Check (`health_check.sh`)

**Purpose**: Detailed system validation with logging

**Features**:
- 35+ comprehensive checks
- Detailed logging to `logs/health_check_YYYYMMDD_HHMMSS.log`
- Prerequisite verification
- Infrastructure service validation
- gRPC service health checks
- Docker container monitoring
- Process resource tracking
- Inter-service communication tests
- Hot-reload validation
- Service log error scanning

**Usage**:
```bash
./health_check.sh

# View latest log
ls -lt logs/health_check_*.log | head -1
tail -100 logs/health_check_20251003_151333.log
```

**Checks Include**:
1. **Prerequisites**: grpcurl, psql, curl, jq, docker
2. **Docker Containers**: Health status, unhealthy detection
3. **Infrastructure Services**:
   - PostgreSQL (connection, table count)
   - Redis (ping, memory usage)
   - Vault (health, sealed status)
   - InfluxDB (optional)
   - Prometheus (health endpoint)
   - Grafana (API health, database status)
4. **gRPC Services**: Port listening, service list, health endpoint
5. **Service Processes**: Running status, CPU/memory usage
6. **System Resources**: CPU, memory, disk usage
7. **Inter-Service Communication**: API Gateway routing
8. **Hot-Reload**: PostgreSQL NOTIFY/LISTEN, config_settings table
9. **Service Logs**: Error scanning

### 3. TLS Certificate Generator (`generate_dev_certs.sh`)

**Purpose**: Generate development TLS certificates for gRPC services

**Features**:
- Generates CA certificate
- Generates server certificates with SAN (localhost + service names)
- Generates client certificates (for mutual TLS)
- Proper file permissions
- Can run as sudo (for /etc/foxhunt) or locally (./certs_dev)

**Usage**:
```bash
# Generate in /etc/foxhunt/certs (requires sudo)
sudo ./generate_dev_certs.sh

# Generate in local directory (no sudo)
./generate_dev_certs.sh
# Then manually copy to /etc/foxhunt/certs
```

**Generated Files**:
- `ca.crt` - Certificate Authority certificate
- `ca.key` - Certificate Authority private key
- `server.crt` - Server certificate (valid for localhost + service names)
- `server.key` - Server private key
- `client.crt` - Client certificate (for mutual TLS)
- `client.key` - Client private key

**Subject Alternative Names (SAN)**:
- `localhost`
- `trading.foxhunt.local`
- `backtesting.foxhunt.local`
- `ml-training.foxhunt.local`
- `api-gateway.foxhunt.local`
- `127.0.0.1`
- `0.0.0.0`

### 4. Health Status Summary (`health_status_summary.txt`)

**Purpose**: Visual summary of current system health

**View**:
```bash
cat health_status_summary.txt
```

## Exit Codes

All health check scripts use standard exit codes:

- `0` - **HEALTHY**: All checks passed
- `1` - **DEGRADED**: Some checks failed (<5 failures)
- `2` - **UNHEALTHY**: Many checks failed (≥5 failures)

## Integration with CI/CD

### GitHub Actions

```yaml
name: Health Check
on: [push, pull_request]

jobs:
  health_check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Start Infrastructure
        run: docker-compose up -d postgres redis vault

      - name: Run Quick Health Check
        run: ./quick_health_check.sh

      - name: Upload Health Logs
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: health-logs
          path: logs/health_check_*.log
```

### Kubernetes Probes

```yaml
livenessProbe:
  exec:
    command:
    - grpcurl
    - -plaintext
    - localhost:50051
    - grpc.health.v1.Health/Check
  initialDelaySeconds: 10
  periodSeconds: 30

readinessProbe:
  exec:
    command:
    - grpcurl
    - -plaintext
    - localhost:50051
    - grpc.health.v1.Health/Check
  initialDelaySeconds: 5
  periodSeconds: 10
```

### Cron Monitoring

```bash
# Add to crontab for continuous monitoring
*/5 * * * * /path/to/foxhunt/quick_health_check.sh || \
    echo "Health check failed!" | mail -s "Foxhunt Health Alert" ops@company.com
```

## Common Issues and Fixes

### Issue: All gRPC services failing with TLS errors

**Symptom**:
```
Failed to read certificate file: /etc/foxhunt/certs/server.crt
No such file or directory (os error 2)
```

**Fix**:
```bash
# Generate development certificates
sudo ./generate_dev_certs.sh

# Restart services
sudo systemctl restart foxhunt-*

# Verify
./quick_health_check.sh
```

### Issue: PostgreSQL connection refused

**Symptom**:
```
[FAIL] PostgreSQL is NOT healthy
```

**Fix**:
```bash
# Check Docker container
docker ps | grep postgres

# Check credentials (from docker-compose.yml)
PGPASSWORD=test_password psql -h localhost -p 5433 -U foxhunt_test -d foxhunt_test

# Verify port mapping
docker port api_gateway_test_postgres
```

### Issue: Redis not responding

**Symptom**:
```
[FAIL] Redis is NOT healthy
```

**Fix**:
```bash
# Check Docker container
docker ps | grep redis

# Test via Docker
docker exec api_gateway_test_redis redis-cli PING

# Check port mapping
docker port api_gateway_test_redis
```

### Issue: Vault sealed

**Symptom**:
```
[WARN] Vault is healthy but SEALED
```

**Fix**:
```bash
# Unseal Vault (requires unseal keys)
vault operator unseal <unseal_key_1>
vault operator unseal <unseal_key_2>
vault operator unseal <unseal_key_3>

# Check status
vault status
```

## File Locations

```
/home/jgrusewski/Work/foxhunt/
├── health_check.sh                    # Comprehensive health check
├── quick_health_check.sh              # Fast health check
├── generate_dev_certs.sh              # TLS certificate generator
├── health_status_summary.txt          # Visual status summary
├── HEALTH_CHECK_README.md             # This file
└── logs/
    ├── health_check_*.log             # Detailed health check logs
    ├── trading_service.log            # Service logs
    ├── backtesting_service.log
    └── ml_training_service.log
```

## Documentation

Comprehensive health validation report:
- **docs/WAVE75_AGENT6_HEALTH_VALIDATION.md** - Full analysis, root causes, remediation

## Service Ports Reference

### gRPC Application Services
- **50050** - API Gateway
- **50051** - Trading Service
- **50052** - Backtesting Service
- **50053** - ML Training Service

### Infrastructure Services
- **5433** - PostgreSQL (mapped from 5432)
- **6380** - Redis (mapped from 6379)
- **8200** - Vault
- **8086** - InfluxDB (optional)
- **9099** - Prometheus (mapped from 9090)
- **3000** - Grafana

### Monitoring Exporters
- **9187** - PostgreSQL Exporter
- **9121** - Redis Exporter
- **9100** - Node Exporter
- **9093** - AlertManager

## Best Practices

1. **Run quick health check after every deployment**
   ```bash
   ./quick_health_check.sh || exit 1
   ```

2. **Monitor health check logs for patterns**
   ```bash
   grep -i "FAIL" logs/health_check_*.log | sort | uniq -c
   ```

3. **Set up automated alerts for health failures**
   ```bash
   # Cron job example
   */5 * * * * /path/to/quick_health_check.sh || notify-send "Health check failed"
   ```

4. **Include health checks in pre-deployment validation**
   ```bash
   # In deployment script
   ./health_check.sh
   if [ $? -ne 0 ]; then
       echo "Pre-deployment health check failed!"
       exit 1
   fi
   ```

5. **Use comprehensive checks for root cause analysis**
   ```bash
   # When issues occur
   ./health_check.sh > /tmp/detailed_health.log 2>&1
   cat /tmp/detailed_health.log | less
   ```

## Support

For issues or questions:
1. Check `docs/WAVE75_AGENT6_HEALTH_VALIDATION.md` for detailed analysis
2. Review service logs in `logs/` directory
3. Run `./health_check.sh` for comprehensive diagnostics

## Version History

- **2025-10-03** - Initial release (Wave 75 Agent 6)
  - Comprehensive health check script (35+ checks)
  - Quick health check script (13 checks)
  - TLS certificate generator
  - Complete documentation and remediation guide