foxhunt/docs/deployment/SERVICE_DEPENDENCIES.md

# Foxhunt Service Dependencies

## Architecture Overview

```
                       ┌─────────────┐
                       │     TLI     │ (Client)
                       │  (Terminal) │
                       │  Pure Client│
                       └──────┬──────┘
                              │
                         Port 50051
                         (gRPC/TLS)
                              │
                              ▼
                  ┌───────────────────────┐
                  │    API Gateway        │
                  │  ┌─────────────────┐  │
                  │  │ Authentication  │  │
                  │  │ Rate Limiting   │  │
                  │  │ Request Routing │  │
                  │  │ Config Mgmt     │  │
                  │  │ Audit Logging   │  │
                  │  └─────────────────┘  │
                  │  Port 50051 (gRPC)    │
                  │  Port 9091 (metrics)  │
                  │  Port 8080 (health)   │
                  └───────┬───────────────┘
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
        ▼                 ▼                 ▼
  ┌──────────┐    ┌──────────────┐   ┌────────────┐
  │ Trading  │    │ Backtesting  │   │ ML Training│
  │ Service  │    │   Service    │   │  Service   │
  │          │    │              │   │            │
  │Port 50052│    │  Port 50053  │   │ Port 50054 │
  │    9092  │    │      9093    │   │    9094    │
  │    8081  │    │      8083    │   │    8095    │
  └────┬─────┘    └──────┬───────┘   └─────┬──────┘
       │                 │                  │
       │                 │                  │
       └─────────────────┴──────────────────┘
                         │
           ┌─────────────┴────────────┐
           │                          │
           ▼                          ▼
   ┌────────────────┐        ┌────────────────┐
   │   PostgreSQL   │        │     Redis      │
   │   (TimescaleDB)│        │    (Cache)     │
   │                │        │                │
   │    Port 5432   │        │   Port 6379    │
   │ Health: 5433   │        │                │
   └────────┬───────┘        └────────┬───────┘
            │                         │
            ▼                         ▼
   ┌────────────────┐        ┌────────────────┐
   │ HashiCorp Vault│        │   InfluxDB     │
   │   (Secrets)    │        │  (Time Series) │
   │                │        │                │
   │    Port 8200   │        │   Port 8086    │
   └────────────────┘        └────────────────┘

           ┌─────────────────────────┐
           │   Monitoring Stack      │
           ├─────────────────────────┤
           │ Prometheus (9090)       │
           │ Grafana (3000)          │
           │ AlertManager (9093)     │
           └─────────────────────────┘
```

## Service Port Reference

| Service | gRPC Port | HTTP/Health | Metrics | Internal Port |
|---------|-----------|-------------|---------|---------------|
| API Gateway | 50051 | 8080 | 9091 | 50050 |
| Trading Service | 50052 | 8081 | 9092 | 50051 |
| Backtesting Service | 50053 | 8083 | 9093 | 50052 |
| ML Training Service | 50054 | 8095 | 9094 | 50053 |
| PostgreSQL | 5432 | 5433 | - | - |
| Redis | 6379 | - | - | - |
| Vault | 8200 | 8200 | - | - |
| InfluxDB | 8086 | 8086 | - | - |
| Prometheus | 9090 | 9090 | - | - |
| Grafana | 3000 | 3000 | - | - |

## Startup Dependencies

### Layer 1: Infrastructure (Start First)

These services MUST be running and healthy before starting any application services.

```bash
# Start infrastructure layer
docker-compose up -d postgres redis vault influxdb

# Wait for all to be healthy (30-60 seconds)
docker-compose ps | grep -E "(postgres|redis|vault|influxdb)"

# Verify health
docker exec foxhunt-postgres pg_isready -U foxhunt
docker exec foxhunt-redis redis-cli ping
curl -s http://localhost:8200/v1/sys/health
curl -s http://localhost:8086/health
```

**PostgreSQL** (TimescaleDB):
- **No dependencies**
- **Critical**: Database must be healthy before any service starts
- **Health check**: `pg_isready -U foxhunt`
- **Startup time**: 15-30 seconds

**Redis**:
- **No dependencies**
- **Critical**: Cache required for session management
- **Health check**: `redis-cli ping`
- **Startup time**: 5-10 seconds

**HashiCorp Vault**:
- **No dependencies**
- **Critical**: Secrets management for all services
- **Health check**: `curl http://localhost:8200/v1/sys/health`
- **Startup time**: 10-15 seconds
- **Note**: Dev mode auto-unseals, production requires manual unsealing

**InfluxDB**:
- **No dependencies**
- **Optional**: Used for time-series metrics storage
- **Health check**: `curl http://localhost:8086/health`
- **Startup time**: 15-20 seconds

### Layer 2: Core Services (After Infrastructure)

Application services depend on infrastructure being healthy.

```bash
# Wait for infrastructure
sleep 30

# Run database migrations
cargo sqlx migrate run

# Start core services
docker-compose up -d trading_service backtesting_service ml_training_service

# Wait for services to be healthy (60-120 seconds)
docker-compose ps | grep -E "(trading|backtesting|ml_training)"
```

**Trading Service**:
- **Required**: PostgreSQL, Redis, Vault
- **Optional**: None
- **Critical**: Core trading logic, cannot start without database
- **Health check**: `curl http://localhost:8081/health`
- **Startup time**: 30-45 seconds

**Backtesting Service**:
- **Required**: PostgreSQL, Vault
- **Optional**: Redis, ML Training Service
- **Independent**: Can run standalone for historical analysis
- **Health check**: `curl http://localhost:8083/health`
- **Startup time**: 60-90 seconds (model loading)

**ML Training Service**:
- **Required**: PostgreSQL, Vault
- **Optional**: Redis, Storage (S3/MinIO)
- **Independent**: Can run standalone for model training
- **Health check**: `curl http://localhost:8095/health`
- **Startup time**: 60-120 seconds (model initialization, GPU warmup)

### Layer 3: Gateway (After Core Services)

API Gateway depends on backend services being available.

```bash
# Wait for core services
sleep 60

# Start API Gateway
docker-compose up -d api_gateway

# Verify gateway is healthy
grpc_health_probe -addr=localhost:50051
curl http://localhost:8080/health
```

**API Gateway**:
- **Required**: PostgreSQL, Redis, Vault, Trading Service
- **Optional**: Backtesting Service, ML Training Service
- **Graceful degradation**: If optional services unavailable, routes to them return 503
- **Health check**: `grpc_health_probe -addr=localhost:50051`
- **Startup time**: 20-30 seconds

### Layer 4: Monitoring (Optional)

Monitoring stack can start independently at any time.

```bash
# Start monitoring stack
docker-compose up -d prometheus grafana alertmanager

# Verify dashboards accessible
curl http://localhost:9090  # Prometheus
curl http://localhost:3000  # Grafana
```

**Prometheus**:
- **No dependencies**
- **Optional**: Metrics collection and alerting
- **Startup time**: 10-15 seconds

**Grafana**:
- **Required**: Prometheus (for datasource)
- **Optional**: Visualization dashboards
- **Startup time**: 15-20 seconds

## Service Requirements Matrix

| Service | PostgreSQL | Redis | Vault | Trading | Backtesting | ML Training | Storage |
|---------|------------|-------|-------|---------|-------------|-------------|---------|
| **API Gateway** | ✅ Required | ✅ Required | ✅ Required | ✅ Required | ⚠️ Optional | ⚠️ Optional | ❌ Not Used |
| **Trading Service** | ✅ Required | ✅ Required | ✅ Required | - | ❌ Not Used | ❌ Not Used | ❌ Not Used |
| **Backtesting** | ✅ Required | ⚠️ Optional | ✅ Required | ❌ Not Used | - | ⚠️ Optional | ⚠️ Optional |
| **ML Training** | ✅ Required | ⚠️ Optional | ✅ Required | ❌ Not Used | ❌ Not Used | - | ⚠️ Optional |

Legend:
- ✅ **Required**: Service will not start without this dependency
- ⚠️ **Optional**: Service can operate with degraded functionality
- ❌ **Not Used**: Service does not interact with this component

## Dependency Health Checks

### Automated Startup Script

Create `scripts/start_foxhunt.sh`:

```bash
#!/bin/bash

set -e  # Exit on error

echo "=== Foxhunt Startup Sequence ==="

# Layer 1: Infrastructure
echo -e "\n[1/4] Starting infrastructure layer..."
docker-compose up -d postgres redis vault influxdb

echo "Waiting for infrastructure (60s)..."
sleep 60

# Verify infrastructure health
echo "Verifying infrastructure..."
docker exec foxhunt-postgres pg_isready -U foxhunt || { echo "PostgreSQL not ready"; exit 1; }
docker exec foxhunt-redis redis-cli ping | grep -q PONG || { echo "Redis not ready"; exit 1; }
curl -sf http://localhost:8200/v1/sys/health || { echo "Vault not ready"; exit 1; }

# Run database migrations
echo -e "\n[2/4] Running database migrations..."
cargo sqlx migrate run

# Layer 2: Core services
echo -e "\n[3/4] Starting core services..."
docker-compose up -d trading_service backtesting_service ml_training_service

echo "Waiting for core services (120s)..."
sleep 120

# Verify core services health
echo "Verifying core services..."
curl -sf http://localhost:8081/health || { echo "Trading Service not ready"; exit 1; }
curl -sf http://localhost:8083/health || { echo "Backtesting Service not ready"; exit 1; }
curl -sf http://localhost:8095/health || { echo "ML Training Service not ready"; exit 1; }

# Layer 3: API Gateway
echo -e "\n[4/4] Starting API Gateway..."
docker-compose up -d api_gateway

echo "Waiting for API Gateway (30s)..."
sleep 30

# Verify gateway health
echo "Verifying API Gateway..."
grpc_health_probe -addr=localhost:50051 || { echo "API Gateway not ready"; exit 1; }

# Start monitoring (optional)
echo -e "\nStarting monitoring stack (optional)..."
docker-compose up -d prometheus grafana alertmanager

echo -e "\n=== Foxhunt Started Successfully ==="
docker-compose ps
```

### Automated Shutdown Script

Create `scripts/stop_foxhunt.sh`:

```bash
#!/bin/bash

echo "=== Foxhunt Shutdown Sequence ==="

# Layer 3: Stop API Gateway first (client-facing)
echo -e "\n[1/4] Stopping API Gateway..."
docker-compose stop api_gateway

# Layer 2: Stop core services
echo -e "\n[2/4] Stopping core services..."
docker-compose stop trading_service backtesting_service ml_training_service

# Layer 1: Stop infrastructure
echo -e "\n[3/4] Stopping infrastructure..."
docker-compose stop postgres redis vault influxdb

# Stop monitoring
echo -e "\n[4/4] Stopping monitoring..."
docker-compose stop prometheus grafana alertmanager

echo -e "\n=== Foxhunt Stopped Successfully ==="
docker-compose ps
```

## Failure Impact Analysis

### PostgreSQL Failure

**Impact**:
- ❌ **Trading Service**: Immediate failure, cannot execute trades
- ❌ **Backtesting Service**: Cannot load historical data
- ❌ **ML Training Service**: Cannot save/load models
- ❌ **API Gateway**: Authentication fails (session data in DB)

**Mitigation**:
```bash
# PostgreSQL has built-in replication
# Configure streaming replication in production

# Monitor with Prometheus alert
- alert: PostgreSQLDown
  expr: up{job="postgres"} == 0
  for: 1m
```

### Redis Failure

**Impact**:
- ⚠️ **Trading Service**: Rate limiting disabled, session cache unavailable
- ⚠️ **Backtesting Service**: Degraded performance (no caching)
- ⚠️ **ML Training Service**: Feature cache unavailable
- ❌ **API Gateway**: Session management fails, auth degraded

**Mitigation**:
```bash
# Redis has built-in persistence
# Configure AOF or RDB snapshots

# Graceful degradation in code
if redis.is_available() {
    use_redis_cache();
} else {
    fallback_to_database();
}
```

### Trading Service Failure

**Impact**:
- ❌ **API Gateway**: Trading routes return 503
- ✅ **Backtesting Service**: Unaffected (independent)
- ✅ **ML Training Service**: Unaffected (independent)

**Mitigation**:
```bash
# Deploy multiple Trading Service replicas
docker-compose up -d --scale trading_service=3

# API Gateway load balances across replicas
```

### Vault Failure

**Impact**:
- ⚠️ **All Services**: Cannot fetch new secrets (existing secrets cached)
- ❌ **New Service Starts**: Cannot authenticate without secrets

**Mitigation**:
```bash
# Vault HA cluster in production
# Services cache secrets with TTL

# Emergency: Use environment variables as fallback
export DATABASE_PASSWORD="fallback_password"
```

## Service Communication Patterns

### Request Flow: Client → Trading Execution

```
TLI Client
   │
   │ 1. gRPC request (authenticated)
   ▼
API Gateway
   │
   │ 2. Validate JWT token (check Redis cache)
   │ 3. Check rate limit (Redis)
   │ 4. Forward to Trading Service
   ▼
Trading Service
   │
   │ 5. Validate order (business logic)
   │ 6. Check risk limits (PostgreSQL)
   │ 7. Execute trade (broker API)
   │ 8. Record in database (PostgreSQL)
   │ 9. Update cache (Redis)
   ▼
Response to Client
```

### Request Flow: Backtesting

```
TLI Client
   │
   │ 1. Start backtest request
   ▼
API Gateway
   │
   │ 2. Route to Backtesting Service
   ▼
Backtesting Service
   │
   │ 3. Load market data (PostgreSQL/Parquet)
   │ 4. Optional: Load ML model (ML Training Service)
   │ 5. Run strategy simulation
   │ 6. Calculate metrics (Sharpe, drawdown)
   │ 7. Store results (PostgreSQL)
   ▼
Response with backtest ID
```

### Request Flow: ML Model Training

```
ML Training Service (async job)
   │
   │ 1. Fetch training data (PostgreSQL)
   │ 2. Extract features (technical indicators)
   │ 3. Train model (GPU acceleration)
   │ 4. Validate model (test set)
   │ 5. Save checkpoint (S3/PostgreSQL)
   │ 6. Update model registry (PostgreSQL)
   │ 7. Publish metrics (InfluxDB)
   ▼
Model ready for inference
```

## Circuit Breaker Configuration

Services implement circuit breakers to prevent cascade failures:

```rust
// Example: Trading Service → PostgreSQL
CircuitBreaker::new()
    .failure_threshold(5)        // Open after 5 failures
    .timeout(Duration::from_secs(30))
    .half_open_timeout(Duration::from_secs(60))
```

**States**:
- **Closed**: Normal operation, requests pass through
- **Open**: Too many failures, requests fail fast
- **Half-Open**: Test if service recovered

## Production Considerations

### High Availability

1. **Database Replication**:
```yaml
# Primary-replica setup
services:
  postgres-primary:
    image: timescale/timescaledb:latest
    environment:
      - POSTGRES_REPLICATION_MODE=master

  postgres-replica:
    image: timescale/timescaledb:latest
    environment:
      - POSTGRES_REPLICATION_MODE=slave
      - POSTGRES_MASTER_HOST=postgres-primary
```

2. **Service Scaling**:
```bash
# Scale critical services
docker-compose up -d --scale trading_service=3 --scale api_gateway=2
```

3. **Load Balancing**:
```yaml
# Add nginx for load balancing
services:
  nginx:
    image: nginx:alpine
    ports:
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
```

### Monitoring Dependencies

```yaml
# Prometheus service discovery
scrape_configs:
  - job_name: 'api_gateway'
    static_configs:
      - targets: ['api_gateway:9091']

  - job_name: 'trading_service'
    static_configs:
      - targets: ['trading_service:9092']

  - job_name: 'postgres'
    static_configs:
      - targets: ['postgres_exporter:9187']
```

### Dependency Health Dashboard

Grafana dashboard showing service dependencies:

```json
{
  "dashboard": {
    "title": "Foxhunt Service Dependencies",
    "panels": [
      {
        "title": "Service Mesh Health",
        "targets": [
          {
            "expr": "up{job=~\"api_gateway|trading_service|backtesting_service|ml_training_service\"}"
          }
        ]
      },
      {
        "title": "Infrastructure Health",
        "targets": [
          {
            "expr": "up{job=~\"postgres|redis|vault\"}"
          }
        ]
      }
    ]
  }
}
```

## Troubleshooting Dependency Issues

### Check Dependency Chain

```bash
# Verify full dependency chain
scripts/check_dependencies.sh
```

Create `scripts/check_dependencies.sh`:
```bash
#!/bin/bash

echo "=== Checking Service Dependencies ==="

# Layer 1: Infrastructure
echo -e "\n[Infrastructure]"
docker exec foxhunt-postgres pg_isready -U foxhunt && echo "  PostgreSQL: ✓" || echo "  PostgreSQL: ✗"
docker exec foxhunt-redis redis-cli ping | grep -q PONG && echo "  Redis: ✓" || echo "  Redis: ✗"
curl -sf http://localhost:8200/v1/sys/health && echo "  Vault: ✓" || echo "  Vault: ✗"

# Layer 2: Core Services
echo -e "\n[Core Services]"
curl -sf http://localhost:8081/health && echo "  Trading Service: ✓" || echo "  Trading Service: ✗"
curl -sf http://localhost:8083/health && echo "  Backtesting Service: ✓" || echo "  Backtesting Service: ✗"
curl -sf http://localhost:8095/health && echo "  ML Training Service: ✓" || echo "  ML Training Service: ✗"

# Layer 3: Gateway
echo -e "\n[Gateway]"
grpc_health_probe -addr=localhost:50051 && echo "  API Gateway: ✓" || echo "  API Gateway: ✗"

echo -e "\n=== Dependency Check Complete ==="
```

## References

- [Docker Compose Startup Order](https://docs.docker.com/compose/startup-order/)
- [Service Mesh Patterns](https://microservices.io/patterns/microservices.html)
- [Circuit Breaker Pattern](https://martinfowler.com/bliki/CircuitBreaker.html)
- [Health Check Patterns](https://microservices.io/patterns/observability/health-check-api.html)