foxhunt/docs/deployment/README.md

# Foxhunt Deployment Documentation

This directory contains comprehensive operational documentation for deploying, monitoring, and troubleshooting the Foxhunt HFT trading system.

## 📚 Documentation Index

### 1. [TLS Certificate Setup](TLS_CERTIFICATE_SETUP.md)
Complete guide for TLS/mTLS certificate management:
- Self-signed certificates for development
- Production certificates (Let's Encrypt, Corporate CA)
- Certificate validation and rotation
- Docker and Kubernetes configuration
- Troubleshooting TLS issues
- Security best practices

**When to use**: Setting up secure communication between services

---

### 2. [Docker Troubleshooting](DOCKER_TROUBLESHOOTING.md)
Comprehensive Docker Compose troubleshooting guide:
- Service health check procedures
- Common issues and solutions
- Rebuild and redeploy strategies
- Database and network troubleshooting
- Performance optimization
- Emergency recovery procedures

**When to use**: Services not starting, unhealthy containers, connection issues

---

### 3. [Service Dependencies](SERVICE_DEPENDENCIES.md)
Service architecture and dependency management:
- Complete service topology diagram
- Startup dependency sequence
- Service requirements matrix
- Failure impact analysis
- Communication patterns
- High availability configuration

**When to use**: Understanding service relationships, planning deployments

---

### 4. [Health Check Verification](HEALTH_CHECK_VERIFICATION.md)
Health monitoring and verification procedures:
- HTTP and gRPC health endpoints
- Infrastructure health checks
- Prometheus alerting configuration
- Grafana health dashboards
- Automated monitoring scripts
- Troubleshooting failed checks

**When to use**: Verifying system health, setting up monitoring

---

### 5. [General Deployment Guide](DEPLOYMENT.md)
High-level deployment procedures (existing document):
- Initial deployment steps
- Configuration management
- Production considerations

---

## 🛠️ Automation Scripts

Located in `/home/jgrusewski/Work/foxhunt/scripts/`:

### 1. `comprehensive_health_check.sh`
**Purpose**: Complete system health validation
**Usage**: `./scripts/comprehensive_health_check.sh`
**Features**:
- Checks all services (infrastructure, core, gateway, monitoring)
- Validates dependency chains
- Reports resource usage
- Color-coded output with summary

### 2. `start_foxhunt.sh`
**Purpose**: Automated system startup
**Usage**: `./scripts/start_foxhunt.sh`
**Features**:
- Sequential startup (Infrastructure → Core → Gateway → Monitoring)
- Health verification after each layer
- Database migration execution
- Service readiness polling

### 3. `stop_foxhunt.sh`
**Purpose**: Graceful system shutdown
**Usage**: `./scripts/stop_foxhunt.sh`
**Features**:
- Reverse-order shutdown
- Service stop verification
- Clean shutdown status report

### 4. `check_dependencies.sh`
**Purpose**: Service dependency validation
**Usage**: `./scripts/check_dependencies.sh`
**Features**:
- 4-layer dependency validation
- Connection pool status
- Port listening verification
- Dependency chain validation

---

## 🚀 Quick Start

### First Time Setup

1. **Start infrastructure**:
   ```bash
   ./scripts/start_foxhunt.sh
   ```

2. **Verify health**:
   ```bash
   ./scripts/comprehensive_health_check.sh
   ```

3. **Check dependencies**:
   ```bash
   ./scripts/check_dependencies.sh
   ```

### Troubleshooting

1. **Service won't start**:
   - Check logs: `docker-compose logs -f [service-name]`
   - See: [Docker Troubleshooting](DOCKER_TROUBLESHOOTING.md)

2. **Health check fails**:
   - Review failed checks in script output
   - See: [Health Check Verification](HEALTH_CHECK_VERIFICATION.md)

3. **TLS errors**:
   - Verify certificates exist: `ls -la /tmp/foxhunt/certs/`
   - See: [TLS Certificate Setup](TLS_CERTIFICATE_SETUP.md)

4. **Dependency issues**:
   - Run: `./scripts/check_dependencies.sh`
   - See: [Service Dependencies](SERVICE_DEPENDENCIES.md)

---

## 📊 Monitoring

### Health Endpoints

| Service | HTTP Health | gRPC Health | Metrics |
|---------|-------------|-------------|---------|
| API Gateway | http://localhost:8080/health | localhost:50051 | :9091/metrics |
| Trading Service | http://localhost:8081/health | localhost:50052 | :9092/metrics |
| Backtesting | http://localhost:8083/health | localhost:50053 | :9093/metrics |
| ML Training | http://localhost:8095/health | localhost:50054 | :9094/metrics |

### Monitoring Stack

- **Prometheus**: http://localhost:9090
- **Grafana**: http://localhost:3000 (admin/foxhunt123)
- **AlertManager**: http://localhost:9093

See [Health Check Verification](HEALTH_CHECK_VERIFICATION.md) for detailed monitoring setup.

---

## 🔒 Security

### TLS Configuration

For production deployments:

1. Generate certificates (see [TLS Setup](TLS_CERTIFICATE_SETUP.md))
2. Configure volume mounts in `docker-compose.yml`
3. Enable TLS in service configurations
4. Set up certificate rotation

### Secrets Management

All secrets should be stored in HashiCorp Vault:
- Database credentials
- API keys
- TLS certificates
- JWT secrets

See main [CLAUDE.md](../../CLAUDE.md) for Vault configuration.

---

## 📈 Production Readiness

### Pre-Deployment Checklist

- [ ] All services healthy (run `comprehensive_health_check.sh`)
- [ ] Dependencies validated (run `check_dependencies.sh`)
- [ ] TLS certificates configured (see [TLS Setup](TLS_CERTIFICATE_SETUP.md))
- [ ] Monitoring stack deployed (Prometheus, Grafana)
- [ ] Alerting rules configured (see [Health Checks](HEALTH_CHECK_VERIFICATION.md))
- [ ] Backup procedures tested (see [Docker Troubleshooting](DOCKER_TROUBLESHOOTING.md))
- [ ] Database migrations applied (`cargo sqlx migrate run`)
- [ ] Resource limits configured (see [Service Dependencies](SERVICE_DEPENDENCIES.md))
- [ ] Circuit breakers tested
- [ ] Runbooks documented

---

## 🆘 Emergency Procedures

### System Down

1. Check all services: `docker-compose ps`
2. Review logs: `docker-compose logs --tail=100`
3. Run health check: `./scripts/comprehensive_health_check.sh`
4. Restart services: `./scripts/stop_foxhunt.sh && ./scripts/start_foxhunt.sh`

### Database Issues

1. Check PostgreSQL: `docker exec foxhunt-postgres pg_isready -U foxhunt`
2. Review migrations: `cargo sqlx migrate info`
3. Backup database: See [Docker Troubleshooting](DOCKER_TROUBLESHOOTING.md#backup-before-troubleshooting)
4. Restore if needed

### Network Issues

1. Check service connectivity: `./scripts/check_dependencies.sh`
2. Verify Docker network: `docker network inspect foxhunt_foxhunt-network`
3. Test port bindings: `netstat -tulpn | grep -E ":(5005[0-4]|5432|6379)"`
4. See [Docker Troubleshooting](DOCKER_TROUBLESHOOTING.md#network-issues)

---

## 📞 Support

### Documentation
- **Architecture**: [CLAUDE.md](../../CLAUDE.md)
- **Testing**: [TESTING_PLAN.md](../../TESTING_PLAN.md)
- **Wave Reports**: [Wave summaries](../../) (WAVE_*_SUMMARY.md)

### Tools Required

- `docker` and `docker-compose`
- `grpc_health_probe` (for gRPC health checks)
- `jq` (for JSON parsing)
- `curl` (for HTTP health checks)
- `netstat` or `ss` (for port checking)

### Installation

```bash
# Install grpc_health_probe
wget https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.19/grpc_health_probe-linux-amd64
chmod +x grpc_health_probe-linux-amd64
sudo mv grpc_health_probe-linux-amd64 /usr/local/bin/grpc_health_probe

# Install jq
sudo apt-get install jq  # Ubuntu/Debian
sudo yum install jq      # RHEL/CentOS
```

---

## 📝 Contributing

When adding new operational documentation:

1. Follow existing structure and format
2. Include troubleshooting section
3. Provide complete examples
4. Update this README index
5. Test all commands and scripts
6. Update relevant automation scripts

---

## 🔄 Recent Updates

**2025-10-07** (Agent 110):
- Created comprehensive deployment documentation suite
- Added 4 automation scripts (health check, startup, shutdown, dependencies)
- Documented TLS certificate management
- Added Docker troubleshooting guide
- Documented service dependencies and architecture
- Created health check verification procedures

---

**Last Updated**: 2025-10-07
**Documentation Version**: 1.0
**Total Lines**: ~2,814 lines of documentation and automation