foxhunt/docs/deployment

Foxhunt Deployment Documentation

This directory contains comprehensive operational documentation for deploying, monitoring, and troubleshooting the Foxhunt HFT trading system.

📚 Documentation Index

1. TLS Certificate Setup

Complete guide for TLS/mTLS certificate management:

• Self-signed certificates for development
• Production certificates (Let's Encrypt, Corporate CA)
• Certificate validation and rotation
• Docker and Kubernetes configuration
• Troubleshooting TLS issues
• Security best practices

When to use: Setting up secure communication between services

---

2. Docker Troubleshooting

Comprehensive Docker Compose troubleshooting guide:

• Service health check procedures
• Common issues and solutions
• Rebuild and redeploy strategies
• Database and network troubleshooting
• Performance optimization
• Emergency recovery procedures

When to use: Services not starting, unhealthy containers, connection issues

---

3. Service Dependencies

Service architecture and dependency management:

• Complete service topology diagram
• Startup dependency sequence
• Service requirements matrix
• Failure impact analysis
• Communication patterns
• High availability configuration

When to use: Understanding service relationships, planning deployments

---

4. Health Check Verification

Health monitoring and verification procedures:

• HTTP and gRPC health endpoints
• Infrastructure health checks
• Prometheus alerting configuration
• Grafana health dashboards
• Automated monitoring scripts
• Troubleshooting failed checks

When to use: Verifying system health, setting up monitoring

---

5. General Deployment Guide

High-level deployment procedures (existing document):

• Initial deployment steps
• Configuration management
• Production considerations

---

🛠️ Automation Scripts

Located in /home/jgrusewski/Work/foxhunt/scripts/:

1. comprehensive_health_check.sh

Purpose: Complete system health validation Usage: ./scripts/comprehensive_health_check.sh Features:

• Checks all services (infrastructure, core, gateway, monitoring)
• Validates dependency chains
• Reports resource usage
• Color-coded output with summary

2. start_foxhunt.sh

Purpose: Automated system startup Usage: ./scripts/start_foxhunt.sh Features:

• Sequential startup (Infrastructure → Core → Gateway → Monitoring)
• Health verification after each layer
• Database migration execution
• Service readiness polling

3. stop_foxhunt.sh

Purpose: Graceful system shutdown Usage: ./scripts/stop_foxhunt.sh Features:

• Reverse-order shutdown
• Service stop verification
• Clean shutdown status report

4. check_dependencies.sh

Purpose: Service dependency validation Usage: ./scripts/check_dependencies.sh Features:

• 4-layer dependency validation
• Connection pool status
• Port listening verification
• Dependency chain validation

---

🚀 Quick Start

First Time Setup

1. Start infrastructure:

   ./scripts/start_foxhunt.sh
   

2. Verify health:

   ./scripts/comprehensive_health_check.sh
   

3. Check dependencies:

   ./scripts/check_dependencies.sh
   

Troubleshooting

1. Service won't start:

   • Check logs: docker-compose logs -f [service-name]
   • See: Docker Troubleshooting

2. Health check fails:

   • Review failed checks in script output
   • See: Health Check Verification

3. TLS errors:

   • Verify certificates exist: ls -la /tmp/foxhunt/certs/
   • See: TLS Certificate Setup

4. Dependency issues:

   • Run: ./scripts/check_dependencies.sh
   • See: Service Dependencies

---

📊 Monitoring

Health Endpoints

Service	HTTP Health	gRPC Health	Metrics
API Gateway	http://localhost:8080/health	localhost:50051	:9091/metrics
Trading Service	http://localhost:8081/health	localhost:50052	:9092/metrics
Backtesting	http://localhost:8083/health	localhost:50053	:9093/metrics
ML Training	http://localhost:8095/health	localhost:50054	:9094/metrics

Monitoring Stack

• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000 (admin/foxhunt123)
• AlertManager: http://localhost:9093

See Health Check Verification for detailed monitoring setup.

---

🔒 Security

TLS Configuration

For production deployments:

1. Generate certificates (see TLS Setup)
2. Configure volume mounts in docker-compose.yml
3. Enable TLS in service configurations
4. Set up certificate rotation

Secrets Management

All secrets should be stored in HashiCorp Vault:

• Database credentials
• API keys
• TLS certificates
• JWT secrets

See main CLAUDE.md for Vault configuration.

---

📈 Production Readiness

Pre-Deployment Checklist

• All services healthy (run comprehensive_health_check.sh)
• Dependencies validated (run check_dependencies.sh)
• TLS certificates configured (see TLS Setup)
• Monitoring stack deployed (Prometheus, Grafana)
• Alerting rules configured (see Health Checks)
• Backup procedures tested (see Docker Troubleshooting)
• Database migrations applied (cargo sqlx migrate run)
• Resource limits configured (see Service Dependencies)
• Circuit breakers tested
• Runbooks documented

---

🆘 Emergency Procedures

System Down

1. Check all services: docker-compose ps
2. Review logs: docker-compose logs --tail=100
3. Run health check: ./scripts/comprehensive_health_check.sh
4. Restart services: ./scripts/stop_foxhunt.sh && ./scripts/start_foxhunt.sh

Database Issues

1. Check PostgreSQL: docker exec foxhunt-postgres pg_isready -U foxhunt
2. Review migrations: cargo sqlx migrate info
3. Backup database: See Docker Troubleshooting
4. Restore if needed

Network Issues

1. Check service connectivity: ./scripts/check_dependencies.sh
2. Verify Docker network: docker network inspect foxhunt_foxhunt-network
3. Test port bindings: netstat -tulpn | grep -E ":(5005[0-4]|5432|6379)"
4. See Docker Troubleshooting

---

📞 Support

Documentation

• Architecture: CLAUDE.md
• Testing: TESTING_PLAN.md
• Wave Reports: Wave summaries (WAVE_*_SUMMARY.md)

Tools Required

• docker and docker-compose
• grpc_health_probe (for gRPC health checks)
• jq (for JSON parsing)
• curl (for HTTP health checks)
• netstat or ss (for port checking)

Installation

# Install grpc_health_probe
wget https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.19/grpc_health_probe-linux-amd64
chmod +x grpc_health_probe-linux-amd64
sudo mv grpc_health_probe-linux-amd64 /usr/local/bin/grpc_health_probe

# Install jq
sudo apt-get install jq  # Ubuntu/Debian
sudo yum install jq      # RHEL/CentOS

---

📝 Contributing

When adding new operational documentation:

1. Follow existing structure and format
2. Include troubleshooting section
3. Provide complete examples
4. Update this README index
5. Test all commands and scripts
6. Update relevant automation scripts

---

🔄 Recent Updates

2025-10-07 (Agent 110):

• Created comprehensive deployment documentation suite
• Added 4 automation scripts (health check, startup, shutdown, dependencies)
• Documented TLS certificate management
• Added Docker troubleshooting guide
• Documented service dependencies and architecture
• Created health check verification procedures

---

Last Updated: 2025-10-07 Documentation Version: 1.0 Total Lines: ~2,814 lines of documentation and automation