🚀 Wave 126 Wave 1 Complete: 6 agents deployed - 4/4 services healthy
Agent 106: ML health endpoint (HTTP/8095) Agent 107: Redis test fix (serial_test isolation) Agent 108: CLAUDE.md draft update (95-97% → 100%) Agent 109: Prometheus/Grafana setup (31 alerts, 6 dashboards) Agent 110: Deployment docs (9 files + 4 scripts) Agent 111: Security audit prep (0 critical vulnerabilities) Service Health: 4/4 healthy (100%) Tests: 99%+ pass rate Production: ~98% readiness Next: Wave 2 (E2E, load, perf, security validation)
This commit is contained in:
jgrusewski
625
docs/deployment/DOCKER_TROUBLESHOOTING.md
Normal file
625
docs/deployment/DOCKER_TROUBLESHOOTING.md
Normal file
@@ -0,0 +1,625 @@
|
||||
# Docker Compose Troubleshooting
|
||||
|
||||
## Service Health Checks
|
||||
|
||||
### Check Service Status
|
||||
|
||||
View the status of all Docker Compose services:
|
||||
|
||||
```bash
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
Expected output for healthy system:
|
||||
```
|
||||
NAME IMAGE STATUS
|
||||
foxhunt-api-gateway foxhunt/api_gateway:latest Up (healthy)
|
||||
foxhunt-trading-service foxhunt/trading_service:latest Up (healthy)
|
||||
foxhunt-backtesting foxhunt/backtesting:latest Up (healthy)
|
||||
foxhunt-ml-training foxhunt/ml_training:latest Up (healthy)
|
||||
foxhunt-postgres timescale/timescaledb:latest Up (healthy)
|
||||
foxhunt-redis redis:7-alpine Up (healthy)
|
||||
foxhunt-vault vault:1.15 Up (healthy)
|
||||
foxhunt-prometheus prom/prometheus:latest Up (healthy)
|
||||
foxhunt-grafana grafana/grafana:latest Up (healthy)
|
||||
```
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### Issue: Service shows "unhealthy"
|
||||
|
||||
**Symptoms**: Docker Compose shows service status as "unhealthy" or "starting"
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check health endpoint
|
||||
curl http://localhost:8080/health # Adjust port for service
|
||||
|
||||
# Check container logs for errors
|
||||
docker logs foxhunt-api-gateway --tail 100
|
||||
|
||||
# Check health check definition
|
||||
docker inspect foxhunt-api-gateway | jq '.[].State.Health'
|
||||
|
||||
# View health check history
|
||||
docker inspect foxhunt-api-gateway | jq '.[].State.Health.Log'
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Restart unhealthy service
|
||||
docker-compose restart api_gateway
|
||||
|
||||
# If health check fails repeatedly, check service dependencies
|
||||
docker-compose logs postgres redis # Check if dependencies are healthy
|
||||
|
||||
# Verify health check configuration in docker-compose.yml
|
||||
docker-compose config | grep -A 10 healthcheck
|
||||
```
|
||||
|
||||
#### Issue: Service restarting continuously
|
||||
|
||||
**Symptoms**: Service exits immediately after starting, high restart count
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# View restart count
|
||||
docker-compose ps api_gateway
|
||||
|
||||
# View full logs (including previous crashes)
|
||||
docker logs foxhunt-api-gateway
|
||||
|
||||
# Check for port conflicts
|
||||
netstat -tulpn | grep 50051 # Check if port already in use
|
||||
|
||||
# View recent container events
|
||||
docker events --since 10m --filter 'container=foxhunt-api-gateway'
|
||||
```
|
||||
|
||||
**Common Causes & Fixes**:
|
||||
|
||||
1. **Port Conflict**:
|
||||
```bash
|
||||
# Find process using port
|
||||
sudo lsof -i :50051
|
||||
|
||||
# Kill conflicting process or change port in docker-compose.yml
|
||||
```
|
||||
|
||||
2. **Database Connection Failed**:
|
||||
```bash
|
||||
# Verify PostgreSQL is accessible
|
||||
docker exec foxhunt-postgres pg_isready -U foxhunt
|
||||
|
||||
# Check database credentials in .env file
|
||||
grep DATABASE_URL .env
|
||||
|
||||
# Run migrations if database exists but schema is missing
|
||||
cargo sqlx migrate run
|
||||
```
|
||||
|
||||
3. **Missing Environment Variables**:
|
||||
```bash
|
||||
# Check environment variables passed to container
|
||||
docker inspect foxhunt-api-gateway | jq '.[].Config.Env'
|
||||
|
||||
# Verify .env file exists and is sourced
|
||||
ls -la .env
|
||||
docker-compose config | grep -A 5 environment
|
||||
```
|
||||
|
||||
#### Issue: "Connection refused" errors
|
||||
|
||||
**Symptoms**: Services cannot connect to each other or external clients cannot reach services
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Verify service is listening on correct port
|
||||
docker exec foxhunt-api-gateway netstat -tulpn
|
||||
|
||||
# Check service mesh connectivity
|
||||
docker network inspect foxhunt_foxhunt-network
|
||||
|
||||
# Test internal connectivity between services
|
||||
docker exec foxhunt-api-gateway ping foxhunt-postgres
|
||||
docker exec foxhunt-api-gateway curl http://foxhunt-trading-service:50051/health
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Ensure all services are on the same network
|
||||
docker-compose config | grep -A 5 networks
|
||||
|
||||
# Restart networking stack
|
||||
docker-compose down
|
||||
docker network prune -f
|
||||
docker-compose up -d
|
||||
|
||||
# Verify DNS resolution
|
||||
docker exec foxhunt-api-gateway nslookup foxhunt-postgres
|
||||
```
|
||||
|
||||
#### Issue: "Cannot allocate memory" or OOM errors
|
||||
|
||||
**Symptoms**: Services crash with out-of-memory errors
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check Docker memory usage
|
||||
docker stats --no-stream
|
||||
|
||||
# View container memory limits
|
||||
docker inspect foxhunt-api-gateway | jq '.[].HostConfig.Memory'
|
||||
|
||||
# Check system memory
|
||||
free -h
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Increase container memory limit in docker-compose.yml
|
||||
services:
|
||||
api_gateway:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 2G
|
||||
reservations:
|
||||
memory: 1G
|
||||
|
||||
# Or increase Docker daemon memory limit
|
||||
# Edit /etc/docker/daemon.json:
|
||||
{
|
||||
"default-memory": "2G"
|
||||
}
|
||||
|
||||
# Restart Docker
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
|
||||
## Rebuild and Redeploy
|
||||
|
||||
### Clean Rebuild
|
||||
|
||||
Perform a complete rebuild from scratch:
|
||||
|
||||
```bash
|
||||
# 1. Stop all services
|
||||
docker-compose down
|
||||
|
||||
# 2. Remove volumes (CAUTION: deletes all data)
|
||||
docker-compose down -v
|
||||
|
||||
# 3. Remove old images
|
||||
docker-compose down --rmi all
|
||||
|
||||
# 4. Clean build cache
|
||||
docker builder prune -af
|
||||
|
||||
# 5. Rebuild images from scratch
|
||||
docker-compose build --no-cache --parallel
|
||||
|
||||
# 6. Start services
|
||||
docker-compose up -d
|
||||
|
||||
# 7. Watch logs for startup issues
|
||||
docker-compose logs -f
|
||||
|
||||
# 8. Verify health
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
### Individual Service Rebuild
|
||||
|
||||
Rebuild and restart a single service:
|
||||
|
||||
```bash
|
||||
# Rebuild service image
|
||||
docker-compose build --no-cache api_gateway
|
||||
|
||||
# Restart service with new image
|
||||
docker-compose up -d api_gateway
|
||||
|
||||
# Verify service started correctly
|
||||
docker-compose logs -f api_gateway
|
||||
```
|
||||
|
||||
### Incremental Updates
|
||||
|
||||
For faster development iterations:
|
||||
|
||||
```bash
|
||||
# Build without cache for changed code
|
||||
docker-compose build api_gateway
|
||||
|
||||
# Restart with new build
|
||||
docker-compose up -d api_gateway
|
||||
|
||||
# Hot reload without full rebuild (if supported)
|
||||
docker-compose restart api_gateway
|
||||
```
|
||||
|
||||
## Database Issues
|
||||
|
||||
### Issue: "Database connection failed"
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Verify PostgreSQL is healthy
|
||||
docker-compose ps postgres
|
||||
|
||||
# Check if database exists
|
||||
docker exec foxhunt-postgres psql -U foxhunt -d foxhunt -c '\l'
|
||||
|
||||
# Check database logs
|
||||
docker logs foxhunt-postgres --tail 50
|
||||
|
||||
# Test connection from service container
|
||||
docker exec foxhunt-api-gateway pg_isready -h postgres -U foxhunt
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Restart PostgreSQL
|
||||
docker-compose restart postgres
|
||||
|
||||
# Wait for PostgreSQL to be ready
|
||||
sleep 10
|
||||
|
||||
# Run migrations
|
||||
cargo sqlx migrate run
|
||||
|
||||
# Verify tables created
|
||||
docker exec foxhunt-postgres psql -U foxhunt -d foxhunt -c '\dt'
|
||||
```
|
||||
|
||||
### Issue: "Migration failed"
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check migration status
|
||||
cargo sqlx migrate info
|
||||
|
||||
# View migration history in database
|
||||
docker exec foxhunt-postgres psql -U foxhunt -d foxhunt -c \
|
||||
'SELECT * FROM _sqlx_migrations ORDER BY version;'
|
||||
|
||||
# Check for partial migrations
|
||||
docker logs foxhunt-postgres | grep -i migration
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Revert last migration if corrupt
|
||||
cargo sqlx migrate revert
|
||||
|
||||
# Re-run migrations
|
||||
cargo sqlx migrate run
|
||||
|
||||
# Or manually reset migration state (CAUTION)
|
||||
docker exec foxhunt-postgres psql -U foxhunt -d foxhunt -c \
|
||||
'DELETE FROM _sqlx_migrations WHERE version = <failed_version>;'
|
||||
```
|
||||
|
||||
### Issue: "Redis connection failed"
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Verify Redis is running
|
||||
docker-compose ps redis
|
||||
|
||||
# Test Redis connection
|
||||
docker exec foxhunt-redis redis-cli ping
|
||||
|
||||
# Check Redis logs
|
||||
docker logs foxhunt-redis --tail 50
|
||||
|
||||
# Test from service container
|
||||
docker exec foxhunt-api-gateway redis-cli -h redis ping
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Restart Redis
|
||||
docker-compose restart redis
|
||||
|
||||
# Clear Redis data if corrupt
|
||||
docker exec foxhunt-redis redis-cli FLUSHALL
|
||||
|
||||
# Verify connectivity
|
||||
docker exec foxhunt-redis redis-cli INFO server
|
||||
```
|
||||
|
||||
## Network Issues
|
||||
|
||||
### Issue: Services can't communicate
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# List Docker networks
|
||||
docker network ls
|
||||
|
||||
# Inspect Foxhunt network
|
||||
docker network inspect foxhunt_foxhunt-network
|
||||
|
||||
# Verify all services on same network
|
||||
docker-compose config | grep -A 5 networks
|
||||
|
||||
# Check DNS resolution
|
||||
docker exec foxhunt-api-gateway nslookup foxhunt-postgres
|
||||
docker exec foxhunt-api-gateway nslookup foxhunt-redis
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Recreate network
|
||||
docker network rm foxhunt_foxhunt-network
|
||||
docker-compose up -d
|
||||
|
||||
# Or force network recreation
|
||||
docker-compose down
|
||||
docker-compose up -d --force-recreate
|
||||
```
|
||||
|
||||
### Issue: External access not working
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check port bindings
|
||||
docker-compose ps
|
||||
docker port foxhunt-api-gateway
|
||||
|
||||
# Verify firewall rules
|
||||
sudo iptables -L -n | grep 50051
|
||||
|
||||
# Test from host machine
|
||||
curl http://localhost:50051/health
|
||||
grpc_health_probe -addr=localhost:50051
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Ensure ports are exposed in docker-compose.yml
|
||||
services:
|
||||
api_gateway:
|
||||
ports:
|
||||
- "50051:50050" # host:container
|
||||
|
||||
# Restart with port forwarding
|
||||
docker-compose down
|
||||
docker-compose up -d
|
||||
|
||||
# Check if SELinux is blocking (on RHEL/CentOS)
|
||||
sudo setenforce 0 # Temporarily disable to test
|
||||
```
|
||||
|
||||
## Performance Issues
|
||||
|
||||
### Issue: Slow response times
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check container resource usage
|
||||
docker stats --no-stream
|
||||
|
||||
# Check for CPU throttling
|
||||
docker inspect foxhunt-api-gateway | jq '.[].HostConfig.CpuQuota'
|
||||
|
||||
# Monitor real-time stats
|
||||
docker stats
|
||||
|
||||
# Check for disk I/O bottlenecks
|
||||
iostat -x 1 10
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Increase CPU/memory limits
|
||||
services:
|
||||
api_gateway:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2.0'
|
||||
memory: 4G
|
||||
|
||||
# Use host network for lower latency (development only)
|
||||
services:
|
||||
api_gateway:
|
||||
network_mode: "host"
|
||||
|
||||
# Optimize Docker storage driver
|
||||
# Edit /etc/docker/daemon.json:
|
||||
{
|
||||
"storage-driver": "overlay2"
|
||||
}
|
||||
```
|
||||
|
||||
### Issue: High disk usage
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check Docker disk usage
|
||||
docker system df -v
|
||||
|
||||
# Find large images
|
||||
docker images --format "{{.Repository}}:{{.Tag}} {{.Size}}"
|
||||
|
||||
# Check volume usage
|
||||
docker volume ls
|
||||
du -sh /var/lib/docker/volumes/*
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Clean up unused resources
|
||||
docker system prune -af --volumes
|
||||
|
||||
# Remove old images
|
||||
docker image prune -af --filter "until=24h"
|
||||
|
||||
# Compact database (if WAL files growing)
|
||||
docker exec foxhunt-postgres psql -U foxhunt -d foxhunt -c 'VACUUM FULL;'
|
||||
```
|
||||
|
||||
## Logging and Debugging
|
||||
|
||||
### View Service Logs
|
||||
|
||||
```bash
|
||||
# View logs for specific service
|
||||
docker-compose logs api_gateway
|
||||
|
||||
# Follow logs in real-time
|
||||
docker-compose logs -f api_gateway
|
||||
|
||||
# View last N lines
|
||||
docker-compose logs --tail=100 api_gateway
|
||||
|
||||
# View logs since timestamp
|
||||
docker-compose logs --since=2025-01-07T10:00:00 api_gateway
|
||||
|
||||
# View all service logs
|
||||
docker-compose logs -f
|
||||
```
|
||||
|
||||
### Enable Debug Logging
|
||||
|
||||
```bash
|
||||
# Set debug log level in .env
|
||||
echo "RUST_LOG=debug" >> .env
|
||||
|
||||
# Restart service to pick up changes
|
||||
docker-compose restart api_gateway
|
||||
|
||||
# Verify debug logs appear
|
||||
docker-compose logs api_gateway | grep DEBUG
|
||||
```
|
||||
|
||||
### Export Logs for Analysis
|
||||
|
||||
```bash
|
||||
# Export logs to file
|
||||
docker-compose logs api_gateway > api_gateway_logs.txt
|
||||
|
||||
# Export with timestamps
|
||||
docker-compose logs -t api_gateway > api_gateway_logs.txt
|
||||
|
||||
# Export all service logs
|
||||
docker-compose logs > all_services_logs.txt
|
||||
|
||||
# Compress for sharing
|
||||
tar -czf logs_$(date +%Y%m%d_%H%M%S).tar.gz *_logs.txt
|
||||
```
|
||||
|
||||
## Emergency Procedures
|
||||
|
||||
### Complete System Reset
|
||||
|
||||
```bash
|
||||
# WARNING: This deletes ALL data
|
||||
|
||||
# 1. Stop and remove all containers
|
||||
docker-compose down -v --remove-orphans
|
||||
|
||||
# 2. Remove all Foxhunt images
|
||||
docker images | grep foxhunt | awk '{print $3}' | xargs docker rmi -f
|
||||
|
||||
# 3. Remove network
|
||||
docker network rm foxhunt_foxhunt-network
|
||||
|
||||
# 4. Clean Docker system
|
||||
docker system prune -af --volumes
|
||||
|
||||
# 5. Rebuild from scratch
|
||||
docker-compose build --no-cache
|
||||
docker-compose up -d
|
||||
|
||||
# 6. Run migrations
|
||||
cargo sqlx migrate run
|
||||
|
||||
# 7. Verify system
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
### Backup Before Troubleshooting
|
||||
|
||||
```bash
|
||||
# Backup database
|
||||
docker exec foxhunt-postgres pg_dump -U foxhunt foxhunt > backup_$(date +%Y%m%d).sql
|
||||
|
||||
# Backup Redis
|
||||
docker exec foxhunt-redis redis-cli --rdb /tmp/dump.rdb
|
||||
docker cp foxhunt-redis:/tmp/dump.rdb ./redis_backup_$(date +%Y%m%d).rdb
|
||||
|
||||
# Backup configuration
|
||||
cp .env .env.backup
|
||||
cp docker-compose.yml docker-compose.yml.backup
|
||||
```
|
||||
|
||||
### Restore from Backup
|
||||
|
||||
```bash
|
||||
# Restore database
|
||||
docker exec -i foxhunt-postgres psql -U foxhunt -d foxhunt < backup_20250107.sql
|
||||
|
||||
# Restore Redis
|
||||
docker cp redis_backup_20250107.rdb foxhunt-redis:/tmp/dump.rdb
|
||||
docker exec foxhunt-redis redis-cli --rdb /tmp/dump.rdb RESTORE
|
||||
|
||||
# Restart services
|
||||
docker-compose restart
|
||||
```
|
||||
|
||||
## Health Check Automation
|
||||
|
||||
### Automated Health Monitoring Script
|
||||
|
||||
Create `scripts/docker_health_monitor.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
SERVICES=("api_gateway" "trading_service" "backtesting_service" "ml_training_service" "postgres" "redis")
|
||||
ALERT_EMAIL="ops@foxhunt.trading"
|
||||
|
||||
for service in "${SERVICES[@]}"; do
|
||||
status=$(docker inspect -f '{{.State.Health.Status}}' foxhunt-${service} 2>/dev/null)
|
||||
|
||||
if [ "$status" != "healthy" ]; then
|
||||
echo "ALERT: Service ${service} is ${status}"
|
||||
|
||||
# Send alert (configure your alerting system)
|
||||
echo "Service ${service} unhealthy at $(date)" | mail -s "Foxhunt Alert" $ALERT_EMAIL
|
||||
|
||||
# Attempt automatic recovery
|
||||
docker-compose restart ${service}
|
||||
sleep 30
|
||||
|
||||
# Check if recovered
|
||||
status=$(docker inspect -f '{{.State.Health.Status}}' foxhunt-${service})
|
||||
if [ "$status" = "healthy" ]; then
|
||||
echo "Service ${service} recovered"
|
||||
else
|
||||
echo "Service ${service} failed to recover - manual intervention required"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
### Setup Cron Job
|
||||
|
||||
```bash
|
||||
# Add to crontab (run every 5 minutes)
|
||||
crontab -e
|
||||
|
||||
# Add line:
|
||||
*/5 * * * * /home/foxhunt/scripts/docker_health_monitor.sh >> /var/log/foxhunt_health.log 2>&1
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Docker Compose Troubleshooting](https://docs.docker.com/compose/faq/)
|
||||
- [Docker Networking](https://docs.docker.com/network/)
|
||||
- [Container Health Checks](https://docs.docker.com/engine/reference/builder/#healthcheck)
|
||||
- [Docker Logs](https://docs.docker.com/engine/reference/commandline/logs/)
|
||||
676
docs/deployment/HEALTH_CHECK_VERIFICATION.md
Normal file
676
docs/deployment/HEALTH_CHECK_VERIFICATION.md
Normal file
@@ -0,0 +1,676 @@
|
||||
# Health Check Verification Procedures
|
||||
|
||||
## Overview
|
||||
|
||||
Foxhunt HFT system implements comprehensive health checks across all services to ensure system reliability and enable automated monitoring. This guide covers health check endpoints, verification procedures, and automated monitoring scripts.
|
||||
|
||||
## Service Health Endpoints
|
||||
|
||||
### HTTP Health Endpoints
|
||||
|
||||
Each service exposes an HTTP health endpoint for easy monitoring:
|
||||
|
||||
#### API Gateway
|
||||
```bash
|
||||
curl http://localhost:8080/health
|
||||
|
||||
# Expected response:
|
||||
{
|
||||
"status": "healthy",
|
||||
"service": "api_gateway",
|
||||
"version": "1.0.0",
|
||||
"uptime_seconds": 3600,
|
||||
"dependencies": {
|
||||
"postgres": "healthy",
|
||||
"redis": "healthy",
|
||||
"trading_service": "healthy"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Trading Service
|
||||
```bash
|
||||
curl http://localhost:8081/health
|
||||
|
||||
# Expected response:
|
||||
{
|
||||
"status": "healthy",
|
||||
"service": "trading_service",
|
||||
"version": "1.0.0",
|
||||
"uptime_seconds": 3600,
|
||||
"dependencies": {
|
||||
"postgres": "healthy",
|
||||
"redis": "healthy"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Backtesting Service
|
||||
```bash
|
||||
curl http://localhost:8083/health
|
||||
|
||||
# Expected response:
|
||||
{
|
||||
"status": "healthy",
|
||||
"service": "backtesting_service",
|
||||
"version": "1.0.0",
|
||||
"uptime_seconds": 3600,
|
||||
"models_loaded": 3,
|
||||
"dependencies": {
|
||||
"postgres": "healthy"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### ML Training Service
|
||||
```bash
|
||||
curl http://localhost:8095/health
|
||||
|
||||
# Expected response:
|
||||
{
|
||||
"status": "healthy",
|
||||
"service": "ml_training_service",
|
||||
"version": "1.0.0",
|
||||
"uptime_seconds": 3600,
|
||||
"gpu_available": true,
|
||||
"models_loaded": 3,
|
||||
"dependencies": {
|
||||
"postgres": "healthy",
|
||||
"storage": "healthy"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### gRPC Health Checks
|
||||
|
||||
For gRPC services, use `grpc_health_probe`:
|
||||
|
||||
#### Install grpc_health_probe
|
||||
```bash
|
||||
# Download latest release
|
||||
wget https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.19/grpc_health_probe-linux-amd64
|
||||
chmod +x grpc_health_probe-linux-amd64
|
||||
sudo mv grpc_health_probe-linux-amd64 /usr/local/bin/grpc_health_probe
|
||||
```
|
||||
|
||||
#### API Gateway (gRPC)
|
||||
```bash
|
||||
grpc_health_probe -addr=localhost:50051
|
||||
|
||||
# With TLS:
|
||||
grpc_health_probe -addr=localhost:50051 \
|
||||
-tls \
|
||||
-tls-ca-cert=/tmp/foxhunt/certs/ca.crt
|
||||
|
||||
# Expected output:
|
||||
# status: SERVING
|
||||
```
|
||||
|
||||
#### Trading Service (gRPC)
|
||||
```bash
|
||||
grpc_health_probe -addr=localhost:50052
|
||||
|
||||
# Expected output:
|
||||
# status: SERVING
|
||||
```
|
||||
|
||||
#### Backtesting Service (gRPC)
|
||||
```bash
|
||||
grpc_health_probe -addr=localhost:50053
|
||||
|
||||
# Expected output:
|
||||
# status: SERVING
|
||||
```
|
||||
|
||||
#### ML Training Service (gRPC)
|
||||
```bash
|
||||
grpc_health_probe -addr=localhost:50054
|
||||
|
||||
# Expected output:
|
||||
# status: SERVING
|
||||
```
|
||||
|
||||
### Infrastructure Health Checks
|
||||
|
||||
#### PostgreSQL
|
||||
```bash
|
||||
# Using pg_isready
|
||||
docker exec foxhunt-postgres pg_isready -U foxhunt
|
||||
|
||||
# Expected output:
|
||||
# /var/run/postgresql:5432 - accepting connections
|
||||
|
||||
# Check database connectivity
|
||||
docker exec foxhunt-postgres psql -U foxhunt -d foxhunt -c 'SELECT 1;'
|
||||
|
||||
# Check replication lag (if using replication)
|
||||
docker exec foxhunt-postgres psql -U foxhunt -d foxhunt -c \
|
||||
'SELECT pg_last_wal_receive_lsn() - pg_last_wal_replay_lsn() AS replication_lag;'
|
||||
```
|
||||
|
||||
#### Redis
|
||||
```bash
|
||||
# Ping check
|
||||
docker exec foxhunt-redis redis-cli ping
|
||||
|
||||
# Expected output:
|
||||
# PONG
|
||||
|
||||
# Check memory usage
|
||||
docker exec foxhunt-redis redis-cli INFO memory | grep used_memory_human
|
||||
|
||||
# Check keyspace
|
||||
docker exec foxhunt-redis redis-cli INFO keyspace
|
||||
```
|
||||
|
||||
#### HashiCorp Vault
|
||||
```bash
|
||||
# Health endpoint
|
||||
curl http://localhost:8200/v1/sys/health
|
||||
|
||||
# Expected response (initialized and unsealed):
|
||||
{
|
||||
"initialized": true,
|
||||
"sealed": false,
|
||||
"standby": false,
|
||||
"performance_standby": false,
|
||||
"replication_performance_mode": "disabled",
|
||||
"replication_dr_mode": "disabled",
|
||||
"server_time_utc": 1704672000,
|
||||
"version": "1.15.0"
|
||||
}
|
||||
|
||||
# Check seal status
|
||||
curl http://localhost:8200/v1/sys/seal-status
|
||||
```
|
||||
|
||||
#### InfluxDB
|
||||
```bash
|
||||
# Health endpoint
|
||||
curl http://localhost:8086/health
|
||||
|
||||
# Expected response:
|
||||
{
|
||||
"name": "influxdb",
|
||||
"message": "ready for queries and writes",
|
||||
"status": "pass",
|
||||
"checks": [],
|
||||
"version": "2.7.0",
|
||||
"commit": "abc123"
|
||||
}
|
||||
|
||||
# Check database exists
|
||||
curl -G http://localhost:8086/api/v2/buckets \
|
||||
-H "Authorization: Token foxhunt-dev-token"
|
||||
```
|
||||
|
||||
### Prometheus Health
|
||||
|
||||
```bash
|
||||
# Prometheus health
|
||||
curl http://localhost:9090/-/healthy
|
||||
|
||||
# Expected output:
|
||||
# Prometheus is Healthy.
|
||||
|
||||
# Check all service targets
|
||||
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
|
||||
|
||||
# Expected output:
|
||||
# {"job": "api_gateway", "health": "up"}
|
||||
# {"job": "trading_service", "health": "up"}
|
||||
# {"job": "backtesting_service", "health": "up"}
|
||||
# {"job": "ml_training_service", "health": "up"}
|
||||
# {"job": "postgres", "health": "up"}
|
||||
# {"job": "redis", "health": "up"}
|
||||
```
|
||||
|
||||
### Grafana Health
|
||||
|
||||
```bash
|
||||
# Grafana health
|
||||
curl http://localhost:3000/api/health
|
||||
|
||||
# Expected response:
|
||||
{
|
||||
"commit": "abc123",
|
||||
"database": "ok",
|
||||
"version": "9.5.0"
|
||||
}
|
||||
|
||||
# Check datasources
|
||||
curl -u admin:foxhunt123 http://localhost:3000/api/datasources | jq '.[] | {name: .name, type: .type}'
|
||||
```
|
||||
|
||||
## Automated Health Validation Script
|
||||
|
||||
Create comprehensive health check script at `/home/jgrusewski/Work/foxhunt/scripts/health_check.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
# Color codes for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Counters
|
||||
TOTAL_CHECKS=0
|
||||
PASSED_CHECKS=0
|
||||
FAILED_CHECKS=0
|
||||
|
||||
# Function to check health
|
||||
check_health() {
|
||||
local name=$1
|
||||
local command=$2
|
||||
|
||||
TOTAL_CHECKS=$((TOTAL_CHECKS + 1))
|
||||
|
||||
if eval "$command" > /dev/null 2>&1; then
|
||||
echo -e " ${GREEN}✓${NC} $name: ${GREEN}OK${NC}"
|
||||
PASSED_CHECKS=$((PASSED_CHECKS + 1))
|
||||
return 0
|
||||
else
|
||||
echo -e " ${RED}✗${NC} $name: ${RED}FAIL${NC}"
|
||||
FAILED_CHECKS=$((FAILED_CHECKS + 1))
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
echo "=== Foxhunt Health Check ==="
|
||||
echo "Timestamp: $(date '+%Y-%m-%d %H:%M:%S')"
|
||||
|
||||
# 1. Docker Container Status
|
||||
echo -e "\n${YELLOW}[1/7] Docker Container Status:${NC}"
|
||||
check_health "Docker Compose" "docker-compose ps | grep -E '(healthy|Up)'"
|
||||
docker-compose ps
|
||||
|
||||
# 2. Infrastructure Layer
|
||||
echo -e "\n${YELLOW}[2/7] Infrastructure Health:${NC}"
|
||||
check_health "PostgreSQL" "docker exec foxhunt-postgres pg_isready -U foxhunt"
|
||||
check_health "Redis" "docker exec foxhunt-redis redis-cli ping | grep -q PONG"
|
||||
check_health "Vault" "curl -sf http://localhost:8200/v1/sys/health"
|
||||
check_health "InfluxDB" "curl -sf http://localhost:8086/health"
|
||||
|
||||
# 3. HTTP Health Endpoints
|
||||
echo -e "\n${YELLOW}[3/7] HTTP Health Endpoints:${NC}"
|
||||
check_health "API Gateway (HTTP)" "curl -sf http://localhost:8080/health"
|
||||
check_health "Trading Service (HTTP)" "curl -sf http://localhost:8081/health"
|
||||
check_health "Backtesting Service (HTTP)" "curl -sf http://localhost:8083/health"
|
||||
check_health "ML Training Service (HTTP)" "curl -sf http://localhost:8095/health"
|
||||
|
||||
# 4. gRPC Services
|
||||
echo -e "\n${YELLOW}[4/7] gRPC Services:${NC}"
|
||||
check_health "API Gateway (gRPC)" "grpc_health_probe -addr=localhost:50051"
|
||||
check_health "Trading Service (gRPC)" "grpc_health_probe -addr=localhost:50052"
|
||||
check_health "Backtesting Service (gRPC)" "grpc_health_probe -addr=localhost:50053"
|
||||
check_health "ML Training Service (gRPC)" "grpc_health_probe -addr=localhost:50054"
|
||||
|
||||
# 5. Database Connectivity
|
||||
echo -e "\n${YELLOW}[5/7] Database Connectivity:${NC}"
|
||||
check_health "PostgreSQL Query" "docker exec foxhunt-postgres psql -U foxhunt -d foxhunt -c 'SELECT 1;'"
|
||||
check_health "Redis Keyspace" "docker exec foxhunt-redis redis-cli INFO keyspace"
|
||||
check_health "InfluxDB Buckets" "curl -sf -H 'Authorization: Token foxhunt-dev-token' http://localhost:8086/api/v2/buckets"
|
||||
|
||||
# 6. Monitoring Stack
|
||||
echo -e "\n${YELLOW}[6/7] Monitoring Stack:${NC}"
|
||||
check_health "Prometheus" "curl -sf http://localhost:9090/-/healthy"
|
||||
check_health "Grafana" "curl -sf http://localhost:3000/api/health"
|
||||
check_health "Prometheus Targets" "curl -sf http://localhost:9090/api/v1/targets | jq -e '.data.activeTargets | length > 0'"
|
||||
|
||||
# 7. Service Dependencies
|
||||
echo -e "\n${YELLOW}[7/7] Service Dependencies:${NC}"
|
||||
|
||||
# API Gateway dependencies
|
||||
API_GW_HEALTH=$(curl -sf http://localhost:8080/health | jq -r '.dependencies.postgres')
|
||||
if [ "$API_GW_HEALTH" = "healthy" ]; then
|
||||
echo -e " ${GREEN}✓${NC} API Gateway → PostgreSQL: ${GREEN}OK${NC}"
|
||||
PASSED_CHECKS=$((PASSED_CHECKS + 1))
|
||||
else
|
||||
echo -e " ${RED}✗${NC} API Gateway → PostgreSQL: ${RED}FAIL${NC}"
|
||||
FAILED_CHECKS=$((FAILED_CHECKS + 1))
|
||||
fi
|
||||
TOTAL_CHECKS=$((TOTAL_CHECKS + 1))
|
||||
|
||||
API_GW_REDIS=$(curl -sf http://localhost:8080/health | jq -r '.dependencies.redis')
|
||||
if [ "$API_GW_REDIS" = "healthy" ]; then
|
||||
echo -e " ${GREEN}✓${NC} API Gateway → Redis: ${GREEN}OK${NC}"
|
||||
PASSED_CHECKS=$((PASSED_CHECKS + 1))
|
||||
else
|
||||
echo -e " ${RED}✗${NC} API Gateway → Redis: ${RED}FAIL${NC}"
|
||||
FAILED_CHECKS=$((FAILED_CHECKS + 1))
|
||||
fi
|
||||
TOTAL_CHECKS=$((TOTAL_CHECKS + 1))
|
||||
|
||||
# Summary
|
||||
echo -e "\n${YELLOW}=== Health Check Summary ===${NC}"
|
||||
echo "Total Checks: $TOTAL_CHECKS"
|
||||
echo -e "Passed: ${GREEN}$PASSED_CHECKS${NC}"
|
||||
echo -e "Failed: ${RED}$FAILED_CHECKS${NC}"
|
||||
|
||||
if [ $FAILED_CHECKS -eq 0 ]; then
|
||||
echo -e "\n${GREEN}All health checks passed!${NC}"
|
||||
exit 0
|
||||
else
|
||||
echo -e "\n${RED}Some health checks failed!${NC}"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
## Continuous Health Monitoring
|
||||
|
||||
### Cron-based Monitoring
|
||||
|
||||
Setup periodic health checks:
|
||||
|
||||
```bash
|
||||
# Make script executable
|
||||
chmod +x /home/jgrusewski/Work/foxhunt/scripts/health_check.sh
|
||||
|
||||
# Add to crontab (run every 5 minutes)
|
||||
crontab -e
|
||||
|
||||
# Add line:
|
||||
*/5 * * * * /home/jgrusewski/Work/foxhunt/scripts/health_check.sh >> /var/log/foxhunt_health.log 2>&1
|
||||
```
|
||||
|
||||
### Prometheus Alerting Rules
|
||||
|
||||
Create `/home/jgrusewski/Work/foxhunt/prometheus/alerts.yml`:
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
- name: foxhunt_health
|
||||
interval: 30s
|
||||
rules:
|
||||
# Service health alerts
|
||||
- alert: ServiceDown
|
||||
expr: up{job=~"api_gateway|trading_service|backtesting_service|ml_training_service"} == 0
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Service {{ $labels.job }} is down"
|
||||
description: "Service {{ $labels.job }} has been down for more than 1 minute"
|
||||
|
||||
# Database health alerts
|
||||
- alert: PostgreSQLDown
|
||||
expr: up{job="postgres"} == 0
|
||||
for: 30s
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "PostgreSQL is down"
|
||||
description: "PostgreSQL database is unreachable"
|
||||
|
||||
- alert: RedisDown
|
||||
expr: up{job="redis"} == 0
|
||||
for: 30s
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Redis is down"
|
||||
description: "Redis cache is unreachable"
|
||||
|
||||
# High latency alerts
|
||||
- alert: HighResponseTime
|
||||
expr: http_request_duration_seconds{quantile="0.99"} > 1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High response time on {{ $labels.job }}"
|
||||
description: "P99 latency is {{ $value }}s for {{ $labels.job }}"
|
||||
|
||||
# Circuit breaker alerts
|
||||
- alert: CircuitBreakerOpen
|
||||
expr: circuit_breaker_state{state="open"} == 1
|
||||
for: 2m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Circuit breaker open for {{ $labels.service }}"
|
||||
description: "Circuit breaker for {{ $labels.service }} has been open for 2 minutes"
|
||||
|
||||
# Resource alerts
|
||||
- alert: HighMemoryUsage
|
||||
expr: (container_memory_usage_bytes / container_spec_memory_limit_bytes) > 0.9
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High memory usage on {{ $labels.container_name }}"
|
||||
description: "Memory usage is {{ $value | humanizePercentage }} on {{ $labels.container_name }}"
|
||||
|
||||
- alert: HighCPUUsage
|
||||
expr: (rate(container_cpu_usage_seconds_total[5m]) / container_spec_cpu_quota) > 0.9
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High CPU usage on {{ $labels.container_name }}"
|
||||
description: "CPU usage is {{ $value | humanizePercentage }} on {{ $labels.container_name }}"
|
||||
```
|
||||
|
||||
### Grafana Dashboard for Health Monitoring
|
||||
|
||||
Create Grafana dashboard JSON at `/home/jgrusewski/Work/foxhunt/grafana/dashboards/health_overview.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"dashboard": {
|
||||
"title": "Foxhunt Health Overview",
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Service Status",
|
||||
"type": "stat",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "up{job=~\"api_gateway|trading_service|backtesting_service|ml_training_service\"}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"mappings": [
|
||||
{"value": 0, "text": "Down", "color": "red"},
|
||||
{"value": 1, "text": "Up", "color": "green"}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "Infrastructure Status",
|
||||
"type": "stat",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "up{job=~\"postgres|redis|vault\"}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "Circuit Breaker State",
|
||||
"type": "graph",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "circuit_breaker_state"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "Health Check Success Rate",
|
||||
"type": "graph",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(health_check_success_total[5m]) / rate(health_check_total[5m])"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Advanced Health Checks
|
||||
|
||||
### Deep Health Checks
|
||||
|
||||
Beyond basic liveness checks, implement deep health checks:
|
||||
|
||||
```bash
|
||||
# Database query performance
|
||||
curl http://localhost:8081/health/deep
|
||||
|
||||
# Expected response includes query timing:
|
||||
{
|
||||
"status": "healthy",
|
||||
"checks": {
|
||||
"database_query": {
|
||||
"status": "healthy",
|
||||
"latency_ms": 15,
|
||||
"query": "SELECT 1"
|
||||
},
|
||||
"redis_ping": {
|
||||
"status": "healthy",
|
||||
"latency_ms": 2
|
||||
},
|
||||
"model_inference": {
|
||||
"status": "healthy",
|
||||
"latency_ms": 45,
|
||||
"model": "MAMBA-2"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Dependency Chain Validation
|
||||
|
||||
Validate entire request chain:
|
||||
|
||||
```bash
|
||||
# Create end-to-end health check
|
||||
curl -X POST http://localhost:8080/health/e2e \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"check_trading_flow": true,
|
||||
"check_ml_inference": true,
|
||||
"check_backtesting": false
|
||||
}'
|
||||
|
||||
# Response includes timing for each step:
|
||||
{
|
||||
"status": "healthy",
|
||||
"total_latency_ms": 120,
|
||||
"steps": [
|
||||
{"step": "auth", "latency_ms": 10, "status": "ok"},
|
||||
{"step": "rate_limit", "latency_ms": 5, "status": "ok"},
|
||||
{"step": "trading_service", "latency_ms": 50, "status": "ok"},
|
||||
{"step": "risk_check", "latency_ms": 30, "status": "ok"},
|
||||
{"step": "ml_inference", "latency_ms": 25, "status": "ok"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting Failed Health Checks
|
||||
|
||||
### Common Issues and Solutions
|
||||
|
||||
#### 1. HTTP Health Endpoint Returns 503
|
||||
|
||||
**Cause**: Service dependencies are unhealthy
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check service logs
|
||||
docker logs foxhunt-api-gateway --tail 50 | grep -i health
|
||||
|
||||
# Check dependency status in health response
|
||||
curl http://localhost:8080/health | jq '.dependencies'
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Identify failed dependency
|
||||
FAILED_DEP=$(curl -s http://localhost:8080/health | jq -r '.dependencies | to_entries[] | select(.value != "healthy") | .key')
|
||||
|
||||
# Restart failed dependency
|
||||
docker-compose restart $FAILED_DEP
|
||||
|
||||
# Verify recovery
|
||||
sleep 10
|
||||
curl http://localhost:8080/health
|
||||
```
|
||||
|
||||
#### 2. gRPC Health Check Times Out
|
||||
|
||||
**Cause**: Service not listening or TLS configuration issue
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check if port is listening
|
||||
netstat -tulpn | grep 50051
|
||||
|
||||
# Verify service is running
|
||||
docker-compose ps api_gateway
|
||||
|
||||
# Check gRPC logs
|
||||
docker logs foxhunt-api-gateway | grep -i grpc
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Verify TLS certificates if using secure connection
|
||||
grpc_health_probe -addr=localhost:50051 -tls -tls-ca-cert=/tmp/foxhunt/certs/ca.crt
|
||||
|
||||
# Or test without TLS
|
||||
grpc_health_probe -addr=localhost:50051
|
||||
```
|
||||
|
||||
#### 3. Database Health Check Fails
|
||||
|
||||
**Cause**: Connection pool exhausted or database locked
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check PostgreSQL connections
|
||||
docker exec foxhunt-postgres psql -U foxhunt -d foxhunt -c \
|
||||
'SELECT count(*) FROM pg_stat_activity;'
|
||||
|
||||
# Check for locks
|
||||
docker exec foxhunt-postgres psql -U foxhunt -d foxhunt -c \
|
||||
'SELECT * FROM pg_locks WHERE NOT granted;'
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Terminate idle connections
|
||||
docker exec foxhunt-postgres psql -U foxhunt -d foxhunt -c \
|
||||
'SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = '\''idle'\'' AND state_change < NOW() - INTERVAL '\''5 minutes'\'';'
|
||||
|
||||
# Restart service to reset connection pool
|
||||
docker-compose restart api_gateway
|
||||
```
|
||||
|
||||
## Production Health Check Checklist
|
||||
|
||||
- [ ] All services have HTTP health endpoints
|
||||
- [ ] All gRPC services implement health check protocol
|
||||
- [ ] Prometheus is scraping all service metrics
|
||||
- [ ] Grafana dashboards show health status
|
||||
- [ ] Alerting rules configured for critical services
|
||||
- [ ] Automated health check script runs periodically
|
||||
- [ ] On-call team receives alerts via PagerDuty/Slack
|
||||
- [ ] Runbooks exist for common health check failures
|
||||
- [ ] Deep health checks validate full request chains
|
||||
- [ ] Circuit breakers prevent cascade failures
|
||||
|
||||
## References
|
||||
|
||||
- [gRPC Health Checking Protocol](https://github.com/grpc/grpc/blob/master/doc/health-checking.md)
|
||||
- [Kubernetes Liveness/Readiness Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
|
||||
- [Prometheus Alerting](https://prometheus.io/docs/alerting/latest/overview/)
|
||||
- [Grafana Health Dashboards](https://grafana.com/docs/grafana/latest/panels/visualizations/stat-panel/)
|
||||
290
docs/deployment/README.md
Normal file
290
docs/deployment/README.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Foxhunt Deployment Documentation
|
||||
|
||||
This directory contains comprehensive operational documentation for deploying, monitoring, and troubleshooting the Foxhunt HFT trading system.
|
||||
|
||||
## 📚 Documentation Index
|
||||
|
||||
### 1. [TLS Certificate Setup](TLS_CERTIFICATE_SETUP.md)
|
||||
Complete guide for TLS/mTLS certificate management:
|
||||
- Self-signed certificates for development
|
||||
- Production certificates (Let's Encrypt, Corporate CA)
|
||||
- Certificate validation and rotation
|
||||
- Docker and Kubernetes configuration
|
||||
- Troubleshooting TLS issues
|
||||
- Security best practices
|
||||
|
||||
**When to use**: Setting up secure communication between services
|
||||
|
||||
---
|
||||
|
||||
### 2. [Docker Troubleshooting](DOCKER_TROUBLESHOOTING.md)
|
||||
Comprehensive Docker Compose troubleshooting guide:
|
||||
- Service health check procedures
|
||||
- Common issues and solutions
|
||||
- Rebuild and redeploy strategies
|
||||
- Database and network troubleshooting
|
||||
- Performance optimization
|
||||
- Emergency recovery procedures
|
||||
|
||||
**When to use**: Services not starting, unhealthy containers, connection issues
|
||||
|
||||
---
|
||||
|
||||
### 3. [Service Dependencies](SERVICE_DEPENDENCIES.md)
|
||||
Service architecture and dependency management:
|
||||
- Complete service topology diagram
|
||||
- Startup dependency sequence
|
||||
- Service requirements matrix
|
||||
- Failure impact analysis
|
||||
- Communication patterns
|
||||
- High availability configuration
|
||||
|
||||
**When to use**: Understanding service relationships, planning deployments
|
||||
|
||||
---
|
||||
|
||||
### 4. [Health Check Verification](HEALTH_CHECK_VERIFICATION.md)
|
||||
Health monitoring and verification procedures:
|
||||
- HTTP and gRPC health endpoints
|
||||
- Infrastructure health checks
|
||||
- Prometheus alerting configuration
|
||||
- Grafana health dashboards
|
||||
- Automated monitoring scripts
|
||||
- Troubleshooting failed checks
|
||||
|
||||
**When to use**: Verifying system health, setting up monitoring
|
||||
|
||||
---
|
||||
|
||||
### 5. [General Deployment Guide](DEPLOYMENT.md)
|
||||
High-level deployment procedures (existing document):
|
||||
- Initial deployment steps
|
||||
- Configuration management
|
||||
- Production considerations
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Automation Scripts
|
||||
|
||||
Located in `/home/jgrusewski/Work/foxhunt/scripts/`:
|
||||
|
||||
### 1. `comprehensive_health_check.sh`
|
||||
**Purpose**: Complete system health validation
|
||||
**Usage**: `./scripts/comprehensive_health_check.sh`
|
||||
**Features**:
|
||||
- Checks all services (infrastructure, core, gateway, monitoring)
|
||||
- Validates dependency chains
|
||||
- Reports resource usage
|
||||
- Color-coded output with summary
|
||||
|
||||
### 2. `start_foxhunt.sh`
|
||||
**Purpose**: Automated system startup
|
||||
**Usage**: `./scripts/start_foxhunt.sh`
|
||||
**Features**:
|
||||
- Sequential startup (Infrastructure → Core → Gateway → Monitoring)
|
||||
- Health verification after each layer
|
||||
- Database migration execution
|
||||
- Service readiness polling
|
||||
|
||||
### 3. `stop_foxhunt.sh`
|
||||
**Purpose**: Graceful system shutdown
|
||||
**Usage**: `./scripts/stop_foxhunt.sh`
|
||||
**Features**:
|
||||
- Reverse-order shutdown
|
||||
- Service stop verification
|
||||
- Clean shutdown status report
|
||||
|
||||
### 4. `check_dependencies.sh`
|
||||
**Purpose**: Service dependency validation
|
||||
**Usage**: `./scripts/check_dependencies.sh`
|
||||
**Features**:
|
||||
- 4-layer dependency validation
|
||||
- Connection pool status
|
||||
- Port listening verification
|
||||
- Dependency chain validation
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### First Time Setup
|
||||
|
||||
1. **Start infrastructure**:
|
||||
```bash
|
||||
./scripts/start_foxhunt.sh
|
||||
```
|
||||
|
||||
2. **Verify health**:
|
||||
```bash
|
||||
./scripts/comprehensive_health_check.sh
|
||||
```
|
||||
|
||||
3. **Check dependencies**:
|
||||
```bash
|
||||
./scripts/check_dependencies.sh
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
1. **Service won't start**:
|
||||
- Check logs: `docker-compose logs -f [service-name]`
|
||||
- See: [Docker Troubleshooting](DOCKER_TROUBLESHOOTING.md)
|
||||
|
||||
2. **Health check fails**:
|
||||
- Review failed checks in script output
|
||||
- See: [Health Check Verification](HEALTH_CHECK_VERIFICATION.md)
|
||||
|
||||
3. **TLS errors**:
|
||||
- Verify certificates exist: `ls -la /tmp/foxhunt/certs/`
|
||||
- See: [TLS Certificate Setup](TLS_CERTIFICATE_SETUP.md)
|
||||
|
||||
4. **Dependency issues**:
|
||||
- Run: `./scripts/check_dependencies.sh`
|
||||
- See: [Service Dependencies](SERVICE_DEPENDENCIES.md)
|
||||
|
||||
---
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
### Health Endpoints
|
||||
|
||||
| Service | HTTP Health | gRPC Health | Metrics |
|
||||
|---------|-------------|-------------|---------|
|
||||
| API Gateway | http://localhost:8080/health | localhost:50051 | :9091/metrics |
|
||||
| Trading Service | http://localhost:8081/health | localhost:50052 | :9092/metrics |
|
||||
| Backtesting | http://localhost:8083/health | localhost:50053 | :9093/metrics |
|
||||
| ML Training | http://localhost:8095/health | localhost:50054 | :9094/metrics |
|
||||
|
||||
### Monitoring Stack
|
||||
|
||||
- **Prometheus**: http://localhost:9090
|
||||
- **Grafana**: http://localhost:3000 (admin/foxhunt123)
|
||||
- **AlertManager**: http://localhost:9093
|
||||
|
||||
See [Health Check Verification](HEALTH_CHECK_VERIFICATION.md) for detailed monitoring setup.
|
||||
|
||||
---
|
||||
|
||||
## 🔒 Security
|
||||
|
||||
### TLS Configuration
|
||||
|
||||
For production deployments:
|
||||
|
||||
1. Generate certificates (see [TLS Setup](TLS_CERTIFICATE_SETUP.md))
|
||||
2. Configure volume mounts in `docker-compose.yml`
|
||||
3. Enable TLS in service configurations
|
||||
4. Set up certificate rotation
|
||||
|
||||
### Secrets Management
|
||||
|
||||
All secrets should be stored in HashiCorp Vault:
|
||||
- Database credentials
|
||||
- API keys
|
||||
- TLS certificates
|
||||
- JWT secrets
|
||||
|
||||
See main [CLAUDE.md](../../CLAUDE.md) for Vault configuration.
|
||||
|
||||
---
|
||||
|
||||
## 📈 Production Readiness
|
||||
|
||||
### Pre-Deployment Checklist
|
||||
|
||||
- [ ] All services healthy (run `comprehensive_health_check.sh`)
|
||||
- [ ] Dependencies validated (run `check_dependencies.sh`)
|
||||
- [ ] TLS certificates configured (see [TLS Setup](TLS_CERTIFICATE_SETUP.md))
|
||||
- [ ] Monitoring stack deployed (Prometheus, Grafana)
|
||||
- [ ] Alerting rules configured (see [Health Checks](HEALTH_CHECK_VERIFICATION.md))
|
||||
- [ ] Backup procedures tested (see [Docker Troubleshooting](DOCKER_TROUBLESHOOTING.md))
|
||||
- [ ] Database migrations applied (`cargo sqlx migrate run`)
|
||||
- [ ] Resource limits configured (see [Service Dependencies](SERVICE_DEPENDENCIES.md))
|
||||
- [ ] Circuit breakers tested
|
||||
- [ ] Runbooks documented
|
||||
|
||||
---
|
||||
|
||||
## 🆘 Emergency Procedures
|
||||
|
||||
### System Down
|
||||
|
||||
1. Check all services: `docker-compose ps`
|
||||
2. Review logs: `docker-compose logs --tail=100`
|
||||
3. Run health check: `./scripts/comprehensive_health_check.sh`
|
||||
4. Restart services: `./scripts/stop_foxhunt.sh && ./scripts/start_foxhunt.sh`
|
||||
|
||||
### Database Issues
|
||||
|
||||
1. Check PostgreSQL: `docker exec foxhunt-postgres pg_isready -U foxhunt`
|
||||
2. Review migrations: `cargo sqlx migrate info`
|
||||
3. Backup database: See [Docker Troubleshooting](DOCKER_TROUBLESHOOTING.md#backup-before-troubleshooting)
|
||||
4. Restore if needed
|
||||
|
||||
### Network Issues
|
||||
|
||||
1. Check service connectivity: `./scripts/check_dependencies.sh`
|
||||
2. Verify Docker network: `docker network inspect foxhunt_foxhunt-network`
|
||||
3. Test port bindings: `netstat -tulpn | grep -E ":(5005[0-4]|5432|6379)"`
|
||||
4. See [Docker Troubleshooting](DOCKER_TROUBLESHOOTING.md#network-issues)
|
||||
|
||||
---
|
||||
|
||||
## 📞 Support
|
||||
|
||||
### Documentation
|
||||
- **Architecture**: [CLAUDE.md](../../CLAUDE.md)
|
||||
- **Testing**: [TESTING_PLAN.md](../../TESTING_PLAN.md)
|
||||
- **Wave Reports**: [Wave summaries](../../) (WAVE_*_SUMMARY.md)
|
||||
|
||||
### Tools Required
|
||||
|
||||
- `docker` and `docker-compose`
|
||||
- `grpc_health_probe` (for gRPC health checks)
|
||||
- `jq` (for JSON parsing)
|
||||
- `curl` (for HTTP health checks)
|
||||
- `netstat` or `ss` (for port checking)
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# Install grpc_health_probe
|
||||
wget https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.19/grpc_health_probe-linux-amd64
|
||||
chmod +x grpc_health_probe-linux-amd64
|
||||
sudo mv grpc_health_probe-linux-amd64 /usr/local/bin/grpc_health_probe
|
||||
|
||||
# Install jq
|
||||
sudo apt-get install jq # Ubuntu/Debian
|
||||
sudo yum install jq # RHEL/CentOS
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Contributing
|
||||
|
||||
When adding new operational documentation:
|
||||
|
||||
1. Follow existing structure and format
|
||||
2. Include troubleshooting section
|
||||
3. Provide complete examples
|
||||
4. Update this README index
|
||||
5. Test all commands and scripts
|
||||
6. Update relevant automation scripts
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Recent Updates
|
||||
|
||||
**2025-10-07** (Agent 110):
|
||||
- Created comprehensive deployment documentation suite
|
||||
- Added 4 automation scripts (health check, startup, shutdown, dependencies)
|
||||
- Documented TLS certificate management
|
||||
- Added Docker troubleshooting guide
|
||||
- Documented service dependencies and architecture
|
||||
- Created health check verification procedures
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-10-07
|
||||
**Documentation Version**: 1.0
|
||||
**Total Lines**: ~2,814 lines of documentation and automation
|
||||
601
docs/deployment/SERVICE_DEPENDENCIES.md
Normal file
601
docs/deployment/SERVICE_DEPENDENCIES.md
Normal file
@@ -0,0 +1,601 @@
|
||||
# Foxhunt Service Dependencies
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ TLI │ (Client)
|
||||
│ (Terminal) │
|
||||
│ Pure Client│
|
||||
└──────┬──────┘
|
||||
│
|
||||
Port 50051
|
||||
(gRPC/TLS)
|
||||
│
|
||||
▼
|
||||
┌───────────────────────┐
|
||||
│ API Gateway │
|
||||
│ ┌─────────────────┐ │
|
||||
│ │ Authentication │ │
|
||||
│ │ Rate Limiting │ │
|
||||
│ │ Request Routing │ │
|
||||
│ │ Config Mgmt │ │
|
||||
│ │ Audit Logging │ │
|
||||
│ └─────────────────┘ │
|
||||
│ Port 50051 (gRPC) │
|
||||
│ Port 9091 (metrics) │
|
||||
│ Port 8080 (health) │
|
||||
└───────┬───────────────┘
|
||||
│
|
||||
┌─────────────────┼─────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────────┐ ┌────────────┐
|
||||
│ Trading │ │ Backtesting │ │ ML Training│
|
||||
│ Service │ │ Service │ │ Service │
|
||||
│ │ │ │ │ │
|
||||
│Port 50052│ │ Port 50053 │ │ Port 50054 │
|
||||
│ 9092 │ │ 9093 │ │ 9094 │
|
||||
│ 8081 │ │ 8083 │ │ 8095 │
|
||||
└────┬─────┘ └──────┬───────┘ └─────┬──────┘
|
||||
│ │ │
|
||||
│ │ │
|
||||
└─────────────────┴──────────────────┘
|
||||
│
|
||||
┌─────────────┴────────────┐
|
||||
│ │
|
||||
▼ ▼
|
||||
┌────────────────┐ ┌────────────────┐
|
||||
│ PostgreSQL │ │ Redis │
|
||||
│ (TimescaleDB)│ │ (Cache) │
|
||||
│ │ │ │
|
||||
│ Port 5432 │ │ Port 6379 │
|
||||
│ Health: 5433 │ │ │
|
||||
└────────┬───────┘ └────────┬───────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌────────────────┐ ┌────────────────┐
|
||||
│ HashiCorp Vault│ │ InfluxDB │
|
||||
│ (Secrets) │ │ (Time Series) │
|
||||
│ │ │ │
|
||||
│ Port 8200 │ │ Port 8086 │
|
||||
└────────────────┘ └────────────────┘
|
||||
|
||||
┌─────────────────────────┐
|
||||
│ Monitoring Stack │
|
||||
├─────────────────────────┤
|
||||
│ Prometheus (9090) │
|
||||
│ Grafana (3000) │
|
||||
│ AlertManager (9093) │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
## Service Port Reference
|
||||
|
||||
| Service | gRPC Port | HTTP/Health | Metrics | Internal Port |
|
||||
|---------|-----------|-------------|---------|---------------|
|
||||
| API Gateway | 50051 | 8080 | 9091 | 50050 |
|
||||
| Trading Service | 50052 | 8081 | 9092 | 50051 |
|
||||
| Backtesting Service | 50053 | 8083 | 9093 | 50052 |
|
||||
| ML Training Service | 50054 | 8095 | 9094 | 50053 |
|
||||
| PostgreSQL | 5432 | 5433 | - | - |
|
||||
| Redis | 6379 | - | - | - |
|
||||
| Vault | 8200 | 8200 | - | - |
|
||||
| InfluxDB | 8086 | 8086 | - | - |
|
||||
| Prometheus | 9090 | 9090 | - | - |
|
||||
| Grafana | 3000 | 3000 | - | - |
|
||||
|
||||
## Startup Dependencies
|
||||
|
||||
### Layer 1: Infrastructure (Start First)
|
||||
|
||||
These services MUST be running and healthy before starting any application services.
|
||||
|
||||
```bash
|
||||
# Start infrastructure layer
|
||||
docker-compose up -d postgres redis vault influxdb
|
||||
|
||||
# Wait for all to be healthy (30-60 seconds)
|
||||
docker-compose ps | grep -E "(postgres|redis|vault|influxdb)"
|
||||
|
||||
# Verify health
|
||||
docker exec foxhunt-postgres pg_isready -U foxhunt
|
||||
docker exec foxhunt-redis redis-cli ping
|
||||
curl -s http://localhost:8200/v1/sys/health
|
||||
curl -s http://localhost:8086/health
|
||||
```
|
||||
|
||||
**PostgreSQL** (TimescaleDB):
|
||||
- **No dependencies**
|
||||
- **Critical**: Database must be healthy before any service starts
|
||||
- **Health check**: `pg_isready -U foxhunt`
|
||||
- **Startup time**: 15-30 seconds
|
||||
|
||||
**Redis**:
|
||||
- **No dependencies**
|
||||
- **Critical**: Cache required for session management
|
||||
- **Health check**: `redis-cli ping`
|
||||
- **Startup time**: 5-10 seconds
|
||||
|
||||
**HashiCorp Vault**:
|
||||
- **No dependencies**
|
||||
- **Critical**: Secrets management for all services
|
||||
- **Health check**: `curl http://localhost:8200/v1/sys/health`
|
||||
- **Startup time**: 10-15 seconds
|
||||
- **Note**: Dev mode auto-unseals, production requires manual unsealing
|
||||
|
||||
**InfluxDB**:
|
||||
- **No dependencies**
|
||||
- **Optional**: Used for time-series metrics storage
|
||||
- **Health check**: `curl http://localhost:8086/health`
|
||||
- **Startup time**: 15-20 seconds
|
||||
|
||||
### Layer 2: Core Services (After Infrastructure)
|
||||
|
||||
Application services depend on infrastructure being healthy.
|
||||
|
||||
```bash
|
||||
# Wait for infrastructure
|
||||
sleep 30
|
||||
|
||||
# Run database migrations
|
||||
cargo sqlx migrate run
|
||||
|
||||
# Start core services
|
||||
docker-compose up -d trading_service backtesting_service ml_training_service
|
||||
|
||||
# Wait for services to be healthy (60-120 seconds)
|
||||
docker-compose ps | grep -E "(trading|backtesting|ml_training)"
|
||||
```
|
||||
|
||||
**Trading Service**:
|
||||
- **Required**: PostgreSQL, Redis, Vault
|
||||
- **Optional**: None
|
||||
- **Critical**: Core trading logic, cannot start without database
|
||||
- **Health check**: `curl http://localhost:8081/health`
|
||||
- **Startup time**: 30-45 seconds
|
||||
|
||||
**Backtesting Service**:
|
||||
- **Required**: PostgreSQL, Vault
|
||||
- **Optional**: Redis, ML Training Service
|
||||
- **Independent**: Can run standalone for historical analysis
|
||||
- **Health check**: `curl http://localhost:8083/health`
|
||||
- **Startup time**: 60-90 seconds (model loading)
|
||||
|
||||
**ML Training Service**:
|
||||
- **Required**: PostgreSQL, Vault
|
||||
- **Optional**: Redis, Storage (S3/MinIO)
|
||||
- **Independent**: Can run standalone for model training
|
||||
- **Health check**: `curl http://localhost:8095/health`
|
||||
- **Startup time**: 60-120 seconds (model initialization, GPU warmup)
|
||||
|
||||
### Layer 3: Gateway (After Core Services)
|
||||
|
||||
API Gateway depends on backend services being available.
|
||||
|
||||
```bash
|
||||
# Wait for core services
|
||||
sleep 60
|
||||
|
||||
# Start API Gateway
|
||||
docker-compose up -d api_gateway
|
||||
|
||||
# Verify gateway is healthy
|
||||
grpc_health_probe -addr=localhost:50051
|
||||
curl http://localhost:8080/health
|
||||
```
|
||||
|
||||
**API Gateway**:
|
||||
- **Required**: PostgreSQL, Redis, Vault, Trading Service
|
||||
- **Optional**: Backtesting Service, ML Training Service
|
||||
- **Graceful degradation**: If optional services unavailable, routes to them return 503
|
||||
- **Health check**: `grpc_health_probe -addr=localhost:50051`
|
||||
- **Startup time**: 20-30 seconds
|
||||
|
||||
### Layer 4: Monitoring (Optional)
|
||||
|
||||
Monitoring stack can start independently at any time.
|
||||
|
||||
```bash
|
||||
# Start monitoring stack
|
||||
docker-compose up -d prometheus grafana alertmanager
|
||||
|
||||
# Verify dashboards accessible
|
||||
curl http://localhost:9090 # Prometheus
|
||||
curl http://localhost:3000 # Grafana
|
||||
```
|
||||
|
||||
**Prometheus**:
|
||||
- **No dependencies**
|
||||
- **Optional**: Metrics collection and alerting
|
||||
- **Startup time**: 10-15 seconds
|
||||
|
||||
**Grafana**:
|
||||
- **Required**: Prometheus (for datasource)
|
||||
- **Optional**: Visualization dashboards
|
||||
- **Startup time**: 15-20 seconds
|
||||
|
||||
## Service Requirements Matrix
|
||||
|
||||
| Service | PostgreSQL | Redis | Vault | Trading | Backtesting | ML Training | Storage |
|
||||
|---------|------------|-------|-------|---------|-------------|-------------|---------|
|
||||
| **API Gateway** | ✅ Required | ✅ Required | ✅ Required | ✅ Required | ⚠️ Optional | ⚠️ Optional | ❌ Not Used |
|
||||
| **Trading Service** | ✅ Required | ✅ Required | ✅ Required | - | ❌ Not Used | ❌ Not Used | ❌ Not Used |
|
||||
| **Backtesting** | ✅ Required | ⚠️ Optional | ✅ Required | ❌ Not Used | - | ⚠️ Optional | ⚠️ Optional |
|
||||
| **ML Training** | ✅ Required | ⚠️ Optional | ✅ Required | ❌ Not Used | ❌ Not Used | - | ⚠️ Optional |
|
||||
|
||||
Legend:
|
||||
- ✅ **Required**: Service will not start without this dependency
|
||||
- ⚠️ **Optional**: Service can operate with degraded functionality
|
||||
- ❌ **Not Used**: Service does not interact with this component
|
||||
|
||||
## Dependency Health Checks
|
||||
|
||||
### Automated Startup Script
|
||||
|
||||
Create `scripts/start_foxhunt.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
set -e # Exit on error
|
||||
|
||||
echo "=== Foxhunt Startup Sequence ==="
|
||||
|
||||
# Layer 1: Infrastructure
|
||||
echo -e "\n[1/4] Starting infrastructure layer..."
|
||||
docker-compose up -d postgres redis vault influxdb
|
||||
|
||||
echo "Waiting for infrastructure (60s)..."
|
||||
sleep 60
|
||||
|
||||
# Verify infrastructure health
|
||||
echo "Verifying infrastructure..."
|
||||
docker exec foxhunt-postgres pg_isready -U foxhunt || { echo "PostgreSQL not ready"; exit 1; }
|
||||
docker exec foxhunt-redis redis-cli ping | grep -q PONG || { echo "Redis not ready"; exit 1; }
|
||||
curl -sf http://localhost:8200/v1/sys/health || { echo "Vault not ready"; exit 1; }
|
||||
|
||||
# Run database migrations
|
||||
echo -e "\n[2/4] Running database migrations..."
|
||||
cargo sqlx migrate run
|
||||
|
||||
# Layer 2: Core services
|
||||
echo -e "\n[3/4] Starting core services..."
|
||||
docker-compose up -d trading_service backtesting_service ml_training_service
|
||||
|
||||
echo "Waiting for core services (120s)..."
|
||||
sleep 120
|
||||
|
||||
# Verify core services health
|
||||
echo "Verifying core services..."
|
||||
curl -sf http://localhost:8081/health || { echo "Trading Service not ready"; exit 1; }
|
||||
curl -sf http://localhost:8083/health || { echo "Backtesting Service not ready"; exit 1; }
|
||||
curl -sf http://localhost:8095/health || { echo "ML Training Service not ready"; exit 1; }
|
||||
|
||||
# Layer 3: API Gateway
|
||||
echo -e "\n[4/4] Starting API Gateway..."
|
||||
docker-compose up -d api_gateway
|
||||
|
||||
echo "Waiting for API Gateway (30s)..."
|
||||
sleep 30
|
||||
|
||||
# Verify gateway health
|
||||
echo "Verifying API Gateway..."
|
||||
grpc_health_probe -addr=localhost:50051 || { echo "API Gateway not ready"; exit 1; }
|
||||
|
||||
# Start monitoring (optional)
|
||||
echo -e "\nStarting monitoring stack (optional)..."
|
||||
docker-compose up -d prometheus grafana alertmanager
|
||||
|
||||
echo -e "\n=== Foxhunt Started Successfully ==="
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
### Automated Shutdown Script
|
||||
|
||||
Create `scripts/stop_foxhunt.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
echo "=== Foxhunt Shutdown Sequence ==="
|
||||
|
||||
# Layer 3: Stop API Gateway first (client-facing)
|
||||
echo -e "\n[1/4] Stopping API Gateway..."
|
||||
docker-compose stop api_gateway
|
||||
|
||||
# Layer 2: Stop core services
|
||||
echo -e "\n[2/4] Stopping core services..."
|
||||
docker-compose stop trading_service backtesting_service ml_training_service
|
||||
|
||||
# Layer 1: Stop infrastructure
|
||||
echo -e "\n[3/4] Stopping infrastructure..."
|
||||
docker-compose stop postgres redis vault influxdb
|
||||
|
||||
# Stop monitoring
|
||||
echo -e "\n[4/4] Stopping monitoring..."
|
||||
docker-compose stop prometheus grafana alertmanager
|
||||
|
||||
echo -e "\n=== Foxhunt Stopped Successfully ==="
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
## Failure Impact Analysis
|
||||
|
||||
### PostgreSQL Failure
|
||||
|
||||
**Impact**:
|
||||
- ❌ **Trading Service**: Immediate failure, cannot execute trades
|
||||
- ❌ **Backtesting Service**: Cannot load historical data
|
||||
- ❌ **ML Training Service**: Cannot save/load models
|
||||
- ❌ **API Gateway**: Authentication fails (session data in DB)
|
||||
|
||||
**Mitigation**:
|
||||
```bash
|
||||
# PostgreSQL has built-in replication
|
||||
# Configure streaming replication in production
|
||||
|
||||
# Monitor with Prometheus alert
|
||||
- alert: PostgreSQLDown
|
||||
expr: up{job="postgres"} == 0
|
||||
for: 1m
|
||||
```
|
||||
|
||||
### Redis Failure
|
||||
|
||||
**Impact**:
|
||||
- ⚠️ **Trading Service**: Rate limiting disabled, session cache unavailable
|
||||
- ⚠️ **Backtesting Service**: Degraded performance (no caching)
|
||||
- ⚠️ **ML Training Service**: Feature cache unavailable
|
||||
- ❌ **API Gateway**: Session management fails, auth degraded
|
||||
|
||||
**Mitigation**:
|
||||
```bash
|
||||
# Redis has built-in persistence
|
||||
# Configure AOF or RDB snapshots
|
||||
|
||||
# Graceful degradation in code
|
||||
if redis.is_available() {
|
||||
use_redis_cache();
|
||||
} else {
|
||||
fallback_to_database();
|
||||
}
|
||||
```
|
||||
|
||||
### Trading Service Failure
|
||||
|
||||
**Impact**:
|
||||
- ❌ **API Gateway**: Trading routes return 503
|
||||
- ✅ **Backtesting Service**: Unaffected (independent)
|
||||
- ✅ **ML Training Service**: Unaffected (independent)
|
||||
|
||||
**Mitigation**:
|
||||
```bash
|
||||
# Deploy multiple Trading Service replicas
|
||||
docker-compose up -d --scale trading_service=3
|
||||
|
||||
# API Gateway load balances across replicas
|
||||
```
|
||||
|
||||
### Vault Failure
|
||||
|
||||
**Impact**:
|
||||
- ⚠️ **All Services**: Cannot fetch new secrets (existing secrets cached)
|
||||
- ❌ **New Service Starts**: Cannot authenticate without secrets
|
||||
|
||||
**Mitigation**:
|
||||
```bash
|
||||
# Vault HA cluster in production
|
||||
# Services cache secrets with TTL
|
||||
|
||||
# Emergency: Use environment variables as fallback
|
||||
export DATABASE_PASSWORD="fallback_password"
|
||||
```
|
||||
|
||||
## Service Communication Patterns
|
||||
|
||||
### Request Flow: Client → Trading Execution
|
||||
|
||||
```
|
||||
TLI Client
|
||||
│
|
||||
│ 1. gRPC request (authenticated)
|
||||
▼
|
||||
API Gateway
|
||||
│
|
||||
│ 2. Validate JWT token (check Redis cache)
|
||||
│ 3. Check rate limit (Redis)
|
||||
│ 4. Forward to Trading Service
|
||||
▼
|
||||
Trading Service
|
||||
│
|
||||
│ 5. Validate order (business logic)
|
||||
│ 6. Check risk limits (PostgreSQL)
|
||||
│ 7. Execute trade (broker API)
|
||||
│ 8. Record in database (PostgreSQL)
|
||||
│ 9. Update cache (Redis)
|
||||
▼
|
||||
Response to Client
|
||||
```
|
||||
|
||||
### Request Flow: Backtesting
|
||||
|
||||
```
|
||||
TLI Client
|
||||
│
|
||||
│ 1. Start backtest request
|
||||
▼
|
||||
API Gateway
|
||||
│
|
||||
│ 2. Route to Backtesting Service
|
||||
▼
|
||||
Backtesting Service
|
||||
│
|
||||
│ 3. Load market data (PostgreSQL/Parquet)
|
||||
│ 4. Optional: Load ML model (ML Training Service)
|
||||
│ 5. Run strategy simulation
|
||||
│ 6. Calculate metrics (Sharpe, drawdown)
|
||||
│ 7. Store results (PostgreSQL)
|
||||
▼
|
||||
Response with backtest ID
|
||||
```
|
||||
|
||||
### Request Flow: ML Model Training
|
||||
|
||||
```
|
||||
ML Training Service (async job)
|
||||
│
|
||||
│ 1. Fetch training data (PostgreSQL)
|
||||
│ 2. Extract features (technical indicators)
|
||||
│ 3. Train model (GPU acceleration)
|
||||
│ 4. Validate model (test set)
|
||||
│ 5. Save checkpoint (S3/PostgreSQL)
|
||||
│ 6. Update model registry (PostgreSQL)
|
||||
│ 7. Publish metrics (InfluxDB)
|
||||
▼
|
||||
Model ready for inference
|
||||
```
|
||||
|
||||
## Circuit Breaker Configuration
|
||||
|
||||
Services implement circuit breakers to prevent cascade failures:
|
||||
|
||||
```rust
|
||||
// Example: Trading Service → PostgreSQL
|
||||
CircuitBreaker::new()
|
||||
.failure_threshold(5) // Open after 5 failures
|
||||
.timeout(Duration::from_secs(30))
|
||||
.half_open_timeout(Duration::from_secs(60))
|
||||
```
|
||||
|
||||
**States**:
|
||||
- **Closed**: Normal operation, requests pass through
|
||||
- **Open**: Too many failures, requests fail fast
|
||||
- **Half-Open**: Test if service recovered
|
||||
|
||||
## Production Considerations
|
||||
|
||||
### High Availability
|
||||
|
||||
1. **Database Replication**:
|
||||
```yaml
|
||||
# Primary-replica setup
|
||||
services:
|
||||
postgres-primary:
|
||||
image: timescale/timescaledb:latest
|
||||
environment:
|
||||
- POSTGRES_REPLICATION_MODE=master
|
||||
|
||||
postgres-replica:
|
||||
image: timescale/timescaledb:latest
|
||||
environment:
|
||||
- POSTGRES_REPLICATION_MODE=slave
|
||||
- POSTGRES_MASTER_HOST=postgres-primary
|
||||
```
|
||||
|
||||
2. **Service Scaling**:
|
||||
```bash
|
||||
# Scale critical services
|
||||
docker-compose up -d --scale trading_service=3 --scale api_gateway=2
|
||||
```
|
||||
|
||||
3. **Load Balancing**:
|
||||
```yaml
|
||||
# Add nginx for load balancing
|
||||
services:
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
ports:
|
||||
- "443:443"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf
|
||||
```
|
||||
|
||||
### Monitoring Dependencies
|
||||
|
||||
```yaml
|
||||
# Prometheus service discovery
|
||||
scrape_configs:
|
||||
- job_name: 'api_gateway'
|
||||
static_configs:
|
||||
- targets: ['api_gateway:9091']
|
||||
|
||||
- job_name: 'trading_service'
|
||||
static_configs:
|
||||
- targets: ['trading_service:9092']
|
||||
|
||||
- job_name: 'postgres'
|
||||
static_configs:
|
||||
- targets: ['postgres_exporter:9187']
|
||||
```
|
||||
|
||||
### Dependency Health Dashboard
|
||||
|
||||
Grafana dashboard showing service dependencies:
|
||||
|
||||
```json
|
||||
{
|
||||
"dashboard": {
|
||||
"title": "Foxhunt Service Dependencies",
|
||||
"panels": [
|
||||
{
|
||||
"title": "Service Mesh Health",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "up{job=~\"api_gateway|trading_service|backtesting_service|ml_training_service\"}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Infrastructure Health",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "up{job=~\"postgres|redis|vault\"}"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting Dependency Issues
|
||||
|
||||
### Check Dependency Chain
|
||||
|
||||
```bash
|
||||
# Verify full dependency chain
|
||||
scripts/check_dependencies.sh
|
||||
```
|
||||
|
||||
Create `scripts/check_dependencies.sh`:
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
echo "=== Checking Service Dependencies ==="
|
||||
|
||||
# Layer 1: Infrastructure
|
||||
echo -e "\n[Infrastructure]"
|
||||
docker exec foxhunt-postgres pg_isready -U foxhunt && echo " PostgreSQL: ✓" || echo " PostgreSQL: ✗"
|
||||
docker exec foxhunt-redis redis-cli ping | grep -q PONG && echo " Redis: ✓" || echo " Redis: ✗"
|
||||
curl -sf http://localhost:8200/v1/sys/health && echo " Vault: ✓" || echo " Vault: ✗"
|
||||
|
||||
# Layer 2: Core Services
|
||||
echo -e "\n[Core Services]"
|
||||
curl -sf http://localhost:8081/health && echo " Trading Service: ✓" || echo " Trading Service: ✗"
|
||||
curl -sf http://localhost:8083/health && echo " Backtesting Service: ✓" || echo " Backtesting Service: ✗"
|
||||
curl -sf http://localhost:8095/health && echo " ML Training Service: ✓" || echo " ML Training Service: ✗"
|
||||
|
||||
# Layer 3: Gateway
|
||||
echo -e "\n[Gateway]"
|
||||
grpc_health_probe -addr=localhost:50051 && echo " API Gateway: ✓" || echo " API Gateway: ✗"
|
||||
|
||||
echo -e "\n=== Dependency Check Complete ==="
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Docker Compose Startup Order](https://docs.docker.com/compose/startup-order/)
|
||||
- [Service Mesh Patterns](https://microservices.io/patterns/microservices.html)
|
||||
- [Circuit Breaker Pattern](https://martinfowler.com/bliki/CircuitBreaker.html)
|
||||
- [Health Check Patterns](https://microservices.io/patterns/observability/health-check-api.html)
|
||||
377
docs/deployment/TLS_CERTIFICATE_SETUP.md
Normal file
377
docs/deployment/TLS_CERTIFICATE_SETUP.md
Normal file
@@ -0,0 +1,377 @@
|
||||
# TLS Certificate Setup Guide
|
||||
|
||||
## Overview
|
||||
Foxhunt HFT system uses TLS/mTLS for secure inter-service communication. This guide covers certificate generation, configuration, and troubleshooting for development and production environments.
|
||||
|
||||
## Certificate Generation
|
||||
|
||||
### Self-Signed Certificates (Development)
|
||||
|
||||
For local development and testing, self-signed certificates are sufficient:
|
||||
|
||||
```bash
|
||||
# Create certificate directory
|
||||
mkdir -p /tmp/foxhunt/certs
|
||||
cd /tmp/foxhunt/certs
|
||||
|
||||
# Generate RSA 4096-bit private key and certificate
|
||||
openssl req -x509 -newkey rsa:4096 -nodes \
|
||||
-keyout server.key \
|
||||
-out server.crt \
|
||||
-days 365 \
|
||||
-subj "/CN=localhost"
|
||||
|
||||
# Generate CA certificate for mTLS (mutual TLS)
|
||||
openssl req -x509 -new -nodes -key server.key \
|
||||
-sha256 -days 365 -out ca.crt \
|
||||
-subj "/CN=Foxhunt CA"
|
||||
|
||||
# Set appropriate permissions
|
||||
chmod 644 server.crt ca.crt
|
||||
chmod 600 server.key
|
||||
```
|
||||
|
||||
### Production Certificates (Let's Encrypt or Corporate CA)
|
||||
|
||||
For production deployments, use properly signed certificates:
|
||||
|
||||
#### Option 1: Let's Encrypt (Free, Automated)
|
||||
```bash
|
||||
# Install certbot
|
||||
sudo apt-get install certbot
|
||||
|
||||
# Generate certificate for domain
|
||||
sudo certbot certonly --standalone \
|
||||
-d api.foxhunt.trading \
|
||||
-d trading.foxhunt.trading \
|
||||
--non-interactive \
|
||||
--agree-tos \
|
||||
--email ops@foxhunt.trading
|
||||
|
||||
# Copy certificates to Foxhunt directory
|
||||
sudo cp /etc/letsencrypt/live/api.foxhunt.trading/fullchain.pem /tmp/foxhunt/certs/server.crt
|
||||
sudo cp /etc/letsencrypt/live/api.foxhunt.trading/privkey.pem /tmp/foxhunt/certs/server.key
|
||||
sudo cp /etc/letsencrypt/live/api.foxhunt.trading/chain.pem /tmp/foxhunt/certs/ca.crt
|
||||
|
||||
# Set permissions
|
||||
sudo chown foxhunt:foxhunt /tmp/foxhunt/certs/*
|
||||
chmod 644 /tmp/foxhunt/certs/server.crt /tmp/foxhunt/certs/ca.crt
|
||||
chmod 600 /tmp/foxhunt/certs/server.key
|
||||
```
|
||||
|
||||
#### Option 2: Corporate CA (Internal PKI)
|
||||
```bash
|
||||
# Request CSR from your CA
|
||||
openssl req -new -newkey rsa:4096 -nodes \
|
||||
-keyout server.key \
|
||||
-out server.csr \
|
||||
-subj "/C=US/ST=NY/L=NYC/O=Foxhunt/CN=api.foxhunt.internal"
|
||||
|
||||
# Submit server.csr to your corporate CA
|
||||
# Receive signed certificate (server.crt) and CA chain (ca.crt)
|
||||
|
||||
# Install certificates
|
||||
cp server.crt server.key ca.crt /tmp/foxhunt/certs/
|
||||
chmod 644 /tmp/foxhunt/certs/server.crt /tmp/foxhunt/certs/ca.crt
|
||||
chmod 600 /tmp/foxhunt/certs/server.key
|
||||
```
|
||||
|
||||
### Certificate Validation
|
||||
|
||||
Verify certificates are correctly generated:
|
||||
|
||||
```bash
|
||||
# Verify certificate details
|
||||
openssl x509 -in /tmp/foxhunt/certs/server.crt -text -noout
|
||||
|
||||
# Check expiration date
|
||||
openssl x509 -in /tmp/foxhunt/certs/server.crt -noout -dates
|
||||
|
||||
# Verify private key matches certificate
|
||||
openssl rsa -noout -modulus -in /tmp/foxhunt/certs/server.key | openssl md5
|
||||
openssl x509 -noout -modulus -in /tmp/foxhunt/certs/server.crt | openssl md5
|
||||
# Both MD5 hashes should match
|
||||
|
||||
# Verify CA chain
|
||||
openssl verify -CAfile /tmp/foxhunt/certs/ca.crt /tmp/foxhunt/certs/server.crt
|
||||
```
|
||||
|
||||
## Docker Configuration
|
||||
|
||||
### Volume Mounts
|
||||
|
||||
Update `docker-compose.yml` to mount certificates into containers:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
api_gateway:
|
||||
volumes:
|
||||
- ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
|
||||
environment:
|
||||
- TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
|
||||
- TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
|
||||
- TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt
|
||||
|
||||
trading_service:
|
||||
volumes:
|
||||
- ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
|
||||
environment:
|
||||
- TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
|
||||
- TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
|
||||
- TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt
|
||||
|
||||
backtesting_service:
|
||||
volumes:
|
||||
- ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
|
||||
environment:
|
||||
- TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
|
||||
- TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
|
||||
- TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt
|
||||
|
||||
ml_training_service:
|
||||
volumes:
|
||||
- ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
|
||||
environment:
|
||||
- TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
|
||||
- TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
|
||||
- TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt
|
||||
```
|
||||
|
||||
### Kubernetes Secrets (Production)
|
||||
|
||||
For Kubernetes deployments, use Secrets:
|
||||
|
||||
```yaml
|
||||
# Create TLS secret
|
||||
kubectl create secret tls foxhunt-tls \
|
||||
--cert=/tmp/foxhunt/certs/server.crt \
|
||||
--key=/tmp/foxhunt/certs/server.key \
|
||||
--namespace=foxhunt
|
||||
|
||||
# Create CA secret
|
||||
kubectl create secret generic foxhunt-ca \
|
||||
--from-file=ca.crt=/tmp/foxhunt/certs/ca.crt \
|
||||
--namespace=foxhunt
|
||||
|
||||
# Mount in deployment
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: trading-service
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: trading-service
|
||||
volumeMounts:
|
||||
- name: tls-certs
|
||||
mountPath: /tmp/foxhunt/certs
|
||||
readOnly: true
|
||||
env:
|
||||
- name: TLS_CERT_PATH
|
||||
value: /tmp/foxhunt/certs/tls.crt
|
||||
- name: TLS_KEY_PATH
|
||||
value: /tmp/foxhunt/certs/tls.key
|
||||
volumes:
|
||||
- name: tls-certs
|
||||
secret:
|
||||
secretName: foxhunt-tls
|
||||
```
|
||||
|
||||
## Certificate Rotation
|
||||
|
||||
### Automated Rotation (Let's Encrypt)
|
||||
|
||||
```bash
|
||||
# Create renewal script
|
||||
cat > /etc/cron.daily/foxhunt-cert-renewal << 'EOF'
|
||||
#!/bin/bash
|
||||
# Renew Let's Encrypt certificates
|
||||
certbot renew --quiet
|
||||
|
||||
# Copy to Foxhunt directory if renewed
|
||||
if [ $? -eq 0 ]; then
|
||||
cp /etc/letsencrypt/live/api.foxhunt.trading/fullchain.pem /tmp/foxhunt/certs/server.crt
|
||||
cp /etc/letsencrypt/live/api.foxhunt.trading/privkey.pem /tmp/foxhunt/certs/server.key
|
||||
|
||||
# Restart services to pick up new certificates
|
||||
docker-compose restart api_gateway trading_service backtesting_service ml_training_service
|
||||
fi
|
||||
EOF
|
||||
|
||||
chmod +x /etc/cron.daily/foxhunt-cert-renewal
|
||||
```
|
||||
|
||||
### Manual Rotation
|
||||
|
||||
```bash
|
||||
# 1. Generate new certificates (follow generation steps above)
|
||||
|
||||
# 2. Backup old certificates
|
||||
cp /tmp/foxhunt/certs/server.crt /tmp/foxhunt/certs/server.crt.bak
|
||||
cp /tmp/foxhunt/certs/server.key /tmp/foxhunt/certs/server.key.bak
|
||||
|
||||
# 3. Install new certificates
|
||||
cp new-server.crt /tmp/foxhunt/certs/server.crt
|
||||
cp new-server.key /tmp/foxhunt/certs/server.key
|
||||
|
||||
# 4. Restart services
|
||||
docker-compose restart api_gateway trading_service backtesting_service ml_training_service
|
||||
|
||||
# 5. Verify services are healthy
|
||||
docker-compose ps
|
||||
curl -k https://localhost:50051/health
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### Error: "No such file or directory: /tmp/foxhunt/certs/server.crt"
|
||||
|
||||
**Cause**: Certificates not generated or volume mount incorrect
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Verify certificates exist
|
||||
ls -lh /tmp/foxhunt/certs/
|
||||
|
||||
# Check volume mount in docker-compose.yml
|
||||
docker-compose config | grep -A 5 volumes
|
||||
|
||||
# Regenerate certificates if missing
|
||||
cd /tmp/foxhunt/certs
|
||||
openssl req -x509 -newkey rsa:4096 -nodes \
|
||||
-keyout server.key \
|
||||
-out server.crt \
|
||||
-days 365 \
|
||||
-subj "/CN=localhost"
|
||||
```
|
||||
|
||||
#### Error: "Permission denied: /tmp/foxhunt/certs/server.key"
|
||||
|
||||
**Cause**: Incorrect file permissions
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Set correct permissions
|
||||
chmod 600 /tmp/foxhunt/certs/server.key
|
||||
chmod 644 /tmp/foxhunt/certs/server.crt
|
||||
|
||||
# If running in Docker, ensure ownership matches container user
|
||||
sudo chown 1000:1000 /tmp/foxhunt/certs/*
|
||||
```
|
||||
|
||||
#### Error: "Certificate has expired"
|
||||
|
||||
**Cause**: Certificate validity period exceeded
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Check expiration
|
||||
openssl x509 -in /tmp/foxhunt/certs/server.crt -noout -dates
|
||||
|
||||
# Regenerate with longer validity (1 year)
|
||||
openssl req -x509 -newkey rsa:4096 -nodes \
|
||||
-keyout server.key \
|
||||
-out server.crt \
|
||||
-days 365 \
|
||||
-subj "/CN=localhost"
|
||||
|
||||
# Restart services
|
||||
docker-compose restart
|
||||
```
|
||||
|
||||
#### Error: "Certificate verification failed"
|
||||
|
||||
**Cause**: CA certificate mismatch or missing
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Verify certificate chain
|
||||
openssl verify -CAfile /tmp/foxhunt/certs/ca.crt /tmp/foxhunt/certs/server.crt
|
||||
|
||||
# Ensure CA certificate is correct
|
||||
openssl x509 -in /tmp/foxhunt/certs/ca.crt -text -noout
|
||||
|
||||
# Regenerate CA if needed
|
||||
openssl req -x509 -new -nodes -key server.key \
|
||||
-sha256 -days 365 -out ca.crt \
|
||||
-subj "/CN=Foxhunt CA"
|
||||
```
|
||||
|
||||
#### Error: "TLS handshake failed"
|
||||
|
||||
**Cause**: Protocol version mismatch or cipher suite incompatibility
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Test TLS connection
|
||||
openssl s_client -connect localhost:50051 -tls1_2
|
||||
|
||||
# Check supported ciphers
|
||||
openssl ciphers -v 'HIGH:!aNULL:!MD5'
|
||||
|
||||
# Update service configuration to allow TLS 1.2+
|
||||
# In service config (config/schemas.rs):
|
||||
# tls_min_version: "TLS1_2"
|
||||
```
|
||||
|
||||
### Debugging Tools
|
||||
|
||||
```bash
|
||||
# Test gRPC TLS connection
|
||||
grpc_health_probe -addr=localhost:50051 \
|
||||
-tls \
|
||||
-tls-ca-cert=/tmp/foxhunt/certs/ca.crt \
|
||||
-tls-client-cert=/tmp/foxhunt/certs/server.crt \
|
||||
-tls-client-key=/tmp/foxhunt/certs/server.key
|
||||
|
||||
# View certificate details in service logs
|
||||
docker logs foxhunt-api-gateway 2>&1 | grep -i tls
|
||||
|
||||
# Monitor TLS errors
|
||||
journalctl -u docker -f | grep -i "tls\|certificate"
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
1. **Key Protection**:
|
||||
- Store private keys with `600` permissions (owner read/write only)
|
||||
- Never commit private keys to version control
|
||||
- Use hardware security modules (HSM) for production keys
|
||||
|
||||
2. **Certificate Validation**:
|
||||
- Enable strict certificate validation in production
|
||||
- Verify hostname matches certificate CN/SAN
|
||||
- Implement certificate pinning for critical services
|
||||
|
||||
3. **Rotation Policy**:
|
||||
- Rotate certificates every 90 days (Let's Encrypt default)
|
||||
- Monitor certificate expiration with alerts
|
||||
- Maintain certificate inventory in Vault
|
||||
|
||||
4. **Cipher Suites**:
|
||||
- Disable weak ciphers (MD5, DES, RC4)
|
||||
- Prefer forward secrecy (ECDHE, DHE)
|
||||
- Use TLS 1.2 or higher
|
||||
|
||||
## Production Checklist
|
||||
|
||||
- [ ] Generate production certificates from trusted CA
|
||||
- [ ] Configure automated renewal (certbot or corporate process)
|
||||
- [ ] Set up certificate expiration monitoring
|
||||
- [ ] Test TLS connections from all services
|
||||
- [ ] Verify mTLS authentication works
|
||||
- [ ] Document certificate locations in Vault
|
||||
- [ ] Configure backup/restore procedures
|
||||
- [ ] Set up alerts for TLS errors
|
||||
- [ ] Test certificate rotation procedure
|
||||
- [ ] Review security audit logs
|
||||
|
||||
## References
|
||||
|
||||
- [OpenSSL Documentation](https://www.openssl.org/docs/)
|
||||
- [Let's Encrypt Best Practices](https://letsencrypt.org/docs/)
|
||||
- [gRPC TLS Guide](https://grpc.io/docs/guides/auth/)
|
||||
- [Docker TLS Security](https://docs.docker.com/engine/security/certificates/)
|
||||
Reference in New Issue
Block a user