foxhunt/docs/deployment/SERVICE_DEPENDENCIES.md

Foxhunt Service Dependencies

Architecture Overview

                       ┌─────────────┐
                       │     TLI     │ (Client)
                       │  (Terminal) │
                       │  Pure Client│
                       └──────┬──────┘
                              │
                         Port 50051
                         (gRPC/TLS)
                              │
                              ▼
                  ┌───────────────────────┐
                  │    API Gateway        │
                  │  ┌─────────────────┐  │
                  │  │ Authentication  │  │
                  │  │ Rate Limiting   │  │
                  │  │ Request Routing │  │
                  │  │ Config Mgmt     │  │
                  │  │ Audit Logging   │  │
                  │  └─────────────────┘  │
                  │  Port 50051 (gRPC)    │
                  │  Port 9091 (metrics)  │
                  │  Port 8080 (health)   │
                  └───────┬───────────────┘
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
        ▼                 ▼                 ▼
  ┌──────────┐    ┌──────────────┐   ┌────────────┐
  │ Trading  │    │ Backtesting  │   │ ML Training│
  │ Service  │    │   Service    │   │  Service   │
  │          │    │              │   │            │
  │Port 50052│    │  Port 50053  │   │ Port 50054 │
  │    9092  │    │      9093    │   │    9094    │
  │    8081  │    │      8083    │   │    8095    │
  └────┬─────┘    └──────┬───────┘   └─────┬──────┘
       │                 │                  │
       │                 │                  │
       └─────────────────┴──────────────────┘
                         │
           ┌─────────────┴────────────┐
           │                          │
           ▼                          ▼
   ┌────────────────┐        ┌────────────────┐
   │   PostgreSQL   │        │     Redis      │
   │   (TimescaleDB)│        │    (Cache)     │
   │                │        │                │
   │    Port 5432   │        │   Port 6379    │
   │ Health: 5433   │        │                │
   └────────┬───────┘        └────────┬───────┘
            │                         │
            ▼                         ▼
   ┌────────────────┐        ┌────────────────┐
   │ HashiCorp Vault│        │   InfluxDB     │
   │   (Secrets)    │        │  (Time Series) │
   │                │        │                │
   │    Port 8200   │        │   Port 8086    │
   └────────────────┘        └────────────────┘

           ┌─────────────────────────┐
           │   Monitoring Stack      │
           ├─────────────────────────┤
           │ Prometheus (9090)       │
           │ Grafana (3000)          │
           │ AlertManager (9093)     │
           └─────────────────────────┘

Service Port Reference

Service	gRPC Port	HTTP/Health	Metrics	Internal Port
API Gateway	50051	8080	9091	50050
Trading Service	50052	8081	9092	50051
Backtesting Service	50053	8083	9093	50052
ML Training Service	50054	8095	9094	50053
PostgreSQL	5432	5433	-	-
Redis	6379	-	-	-
Vault	8200	8200	-	-
InfluxDB	8086	8086	-	-
Prometheus	9090	9090	-	-
Grafana	3000	3000	-	-

Startup Dependencies

Layer 1: Infrastructure (Start First)

These services MUST be running and healthy before starting any application services.

# Start infrastructure layer
docker-compose up -d postgres redis vault influxdb

# Wait for all to be healthy (30-60 seconds)
docker-compose ps | grep -E "(postgres|redis|vault|influxdb)"

# Verify health
docker exec foxhunt-postgres pg_isready -U foxhunt
docker exec foxhunt-redis redis-cli ping
curl -s http://localhost:8200/v1/sys/health
curl -s http://localhost:8086/health

PostgreSQL (TimescaleDB):

• No dependencies
• Critical: Database must be healthy before any service starts
• Health check: pg_isready -U foxhunt
• Startup time: 15-30 seconds

Redis:

• No dependencies
• Critical: Cache required for session management
• Health check: redis-cli ping
• Startup time: 5-10 seconds

HashiCorp Vault:

• No dependencies
• Critical: Secrets management for all services
• Health check: curl http://localhost:8200/v1/sys/health
• Startup time: 10-15 seconds
• Note: Dev mode auto-unseals, production requires manual unsealing

InfluxDB:

• No dependencies
• Optional: Used for time-series metrics storage
• Health check: curl http://localhost:8086/health
• Startup time: 15-20 seconds

Layer 2: Core Services (After Infrastructure)

Application services depend on infrastructure being healthy.

# Wait for infrastructure
sleep 30

# Run database migrations
cargo sqlx migrate run

# Start core services
docker-compose up -d trading_service backtesting_service ml_training_service

# Wait for services to be healthy (60-120 seconds)
docker-compose ps | grep -E "(trading|backtesting|ml_training)"

Trading Service:

• Required: PostgreSQL, Redis, Vault
• Optional: None
• Critical: Core trading logic, cannot start without database
• Health check: curl http://localhost:8081/health
• Startup time: 30-45 seconds

Backtesting Service:

• Required: PostgreSQL, Vault
• Optional: Redis, ML Training Service
• Independent: Can run standalone for historical analysis
• Health check: curl http://localhost:8083/health
• Startup time: 60-90 seconds (model loading)

ML Training Service:

• Required: PostgreSQL, Vault
• Optional: Redis, Storage (S3/MinIO)
• Independent: Can run standalone for model training
• Health check: curl http://localhost:8095/health
• Startup time: 60-120 seconds (model initialization, GPU warmup)

Layer 3: Gateway (After Core Services)

API Gateway depends on backend services being available.

# Wait for core services
sleep 60

# Start API Gateway
docker-compose up -d api_gateway

# Verify gateway is healthy
grpc_health_probe -addr=localhost:50051
curl http://localhost:8080/health

API Gateway:

• Required: PostgreSQL, Redis, Vault, Trading Service
• Optional: Backtesting Service, ML Training Service
• Graceful degradation: If optional services unavailable, routes to them return 503
• Health check: grpc_health_probe -addr=localhost:50051
• Startup time: 20-30 seconds

Layer 4: Monitoring (Optional)

Monitoring stack can start independently at any time.

# Start monitoring stack
docker-compose up -d prometheus grafana alertmanager

# Verify dashboards accessible
curl http://localhost:9090  # Prometheus
curl http://localhost:3000  # Grafana

Prometheus:

• No dependencies
• Optional: Metrics collection and alerting
• Startup time: 10-15 seconds

Grafana:

• Required: Prometheus (for datasource)
• Optional: Visualization dashboards
• Startup time: 15-20 seconds

Service Requirements Matrix

Service	PostgreSQL	Redis	Vault	Trading	Backtesting	ML Training	Storage
API Gateway	✅ Required	✅ Required	✅ Required	✅ Required	⚠️ Optional	⚠️ Optional	❌ Not Used
Trading Service	✅ Required	✅ Required	✅ Required	-	❌ Not Used	❌ Not Used	❌ Not Used
Backtesting	✅ Required	⚠️ Optional	✅ Required	❌ Not Used	-	⚠️ Optional	⚠️ Optional
ML Training	✅ Required	⚠️ Optional	✅ Required	❌ Not Used	❌ Not Used	-	⚠️ Optional

Legend:

• ✅ Required: Service will not start without this dependency
• ⚠️ Optional: Service can operate with degraded functionality
• ❌ Not Used: Service does not interact with this component

Dependency Health Checks

Automated Startup Script

Create scripts/start_foxhunt.sh:

#!/bin/bash

set -e  # Exit on error

echo "=== Foxhunt Startup Sequence ==="

# Layer 1: Infrastructure
echo -e "\n[1/4] Starting infrastructure layer..."
docker-compose up -d postgres redis vault influxdb

echo "Waiting for infrastructure (60s)..."
sleep 60

# Verify infrastructure health
echo "Verifying infrastructure..."
docker exec foxhunt-postgres pg_isready -U foxhunt || { echo "PostgreSQL not ready"; exit 1; }
docker exec foxhunt-redis redis-cli ping | grep -q PONG || { echo "Redis not ready"; exit 1; }
curl -sf http://localhost:8200/v1/sys/health || { echo "Vault not ready"; exit 1; }

# Run database migrations
echo -e "\n[2/4] Running database migrations..."
cargo sqlx migrate run

# Layer 2: Core services
echo -e "\n[3/4] Starting core services..."
docker-compose up -d trading_service backtesting_service ml_training_service

echo "Waiting for core services (120s)..."
sleep 120

# Verify core services health
echo "Verifying core services..."
curl -sf http://localhost:8081/health || { echo "Trading Service not ready"; exit 1; }
curl -sf http://localhost:8083/health || { echo "Backtesting Service not ready"; exit 1; }
curl -sf http://localhost:8095/health || { echo "ML Training Service not ready"; exit 1; }

# Layer 3: API Gateway
echo -e "\n[4/4] Starting API Gateway..."
docker-compose up -d api_gateway

echo "Waiting for API Gateway (30s)..."
sleep 30

# Verify gateway health
echo "Verifying API Gateway..."
grpc_health_probe -addr=localhost:50051 || { echo "API Gateway not ready"; exit 1; }

# Start monitoring (optional)
echo -e "\nStarting monitoring stack (optional)..."
docker-compose up -d prometheus grafana alertmanager

echo -e "\n=== Foxhunt Started Successfully ==="
docker-compose ps

Automated Shutdown Script

Create scripts/stop_foxhunt.sh:

#!/bin/bash

echo "=== Foxhunt Shutdown Sequence ==="

# Layer 3: Stop API Gateway first (client-facing)
echo -e "\n[1/4] Stopping API Gateway..."
docker-compose stop api_gateway

# Layer 2: Stop core services
echo -e "\n[2/4] Stopping core services..."
docker-compose stop trading_service backtesting_service ml_training_service

# Layer 1: Stop infrastructure
echo -e "\n[3/4] Stopping infrastructure..."
docker-compose stop postgres redis vault influxdb

# Stop monitoring
echo -e "\n[4/4] Stopping monitoring..."
docker-compose stop prometheus grafana alertmanager

echo -e "\n=== Foxhunt Stopped Successfully ==="
docker-compose ps

Failure Impact Analysis

PostgreSQL Failure

Impact:

• ❌ Trading Service: Immediate failure, cannot execute trades
• ❌ Backtesting Service: Cannot load historical data
• ❌ ML Training Service: Cannot save/load models
• ❌ API Gateway: Authentication fails (session data in DB)

Mitigation:

# PostgreSQL has built-in replication
# Configure streaming replication in production

# Monitor with Prometheus alert
- alert: PostgreSQLDown
  expr: up{job="postgres"} == 0
  for: 1m

Redis Failure

Impact:

• ⚠️ Trading Service: Rate limiting disabled, session cache unavailable
• ⚠️ Backtesting Service: Degraded performance (no caching)
• ⚠️ ML Training Service: Feature cache unavailable
• ❌ API Gateway: Session management fails, auth degraded

Mitigation:

# Redis has built-in persistence
# Configure AOF or RDB snapshots

# Graceful degradation in code
if redis.is_available() {
    use_redis_cache();
} else {
    fallback_to_database();
}

Trading Service Failure

Impact:

• ❌ API Gateway: Trading routes return 503
• ✅ Backtesting Service: Unaffected (independent)
• ✅ ML Training Service: Unaffected (independent)

Mitigation:

# Deploy multiple Trading Service replicas
docker-compose up -d --scale trading_service=3

# API Gateway load balances across replicas

Vault Failure

Impact:

• ⚠️ All Services: Cannot fetch new secrets (existing secrets cached)
• ❌ New Service Starts: Cannot authenticate without secrets

Mitigation:

# Vault HA cluster in production
# Services cache secrets with TTL

# Emergency: Use environment variables as fallback
export DATABASE_PASSWORD="fallback_password"

Service Communication Patterns

Request Flow: Client → Trading Execution

TLI Client
   │
   │ 1. gRPC request (authenticated)
   ▼
API Gateway
   │
   │ 2. Validate JWT token (check Redis cache)
   │ 3. Check rate limit (Redis)
   │ 4. Forward to Trading Service
   ▼
Trading Service
   │
   │ 5. Validate order (business logic)
   │ 6. Check risk limits (PostgreSQL)
   │ 7. Execute trade (broker API)
   │ 8. Record in database (PostgreSQL)
   │ 9. Update cache (Redis)
   ▼
Response to Client

Request Flow: Backtesting

TLI Client
   │
   │ 1. Start backtest request
   ▼
API Gateway
   │
   │ 2. Route to Backtesting Service
   ▼
Backtesting Service
   │
   │ 3. Load market data (PostgreSQL/Parquet)
   │ 4. Optional: Load ML model (ML Training Service)
   │ 5. Run strategy simulation
   │ 6. Calculate metrics (Sharpe, drawdown)
   │ 7. Store results (PostgreSQL)
   ▼
Response with backtest ID

Request Flow: ML Model Training

ML Training Service (async job)
   │
   │ 1. Fetch training data (PostgreSQL)
   │ 2. Extract features (technical indicators)
   │ 3. Train model (GPU acceleration)
   │ 4. Validate model (test set)
   │ 5. Save checkpoint (S3/PostgreSQL)
   │ 6. Update model registry (PostgreSQL)
   │ 7. Publish metrics (InfluxDB)
   ▼
Model ready for inference

Circuit Breaker Configuration

Services implement circuit breakers to prevent cascade failures:

// Example: Trading Service → PostgreSQL
CircuitBreaker::new()
    .failure_threshold(5)        // Open after 5 failures
    .timeout(Duration::from_secs(30))
    .half_open_timeout(Duration::from_secs(60))

States:

• Closed: Normal operation, requests pass through
• Open: Too many failures, requests fail fast
• Half-Open: Test if service recovered

Production Considerations

High Availability

1. Database Replication:

# Primary-replica setup
services:
  postgres-primary:
    image: timescale/timescaledb:latest
    environment:
      - POSTGRES_REPLICATION_MODE=master

  postgres-replica:
    image: timescale/timescaledb:latest
    environment:
      - POSTGRES_REPLICATION_MODE=slave
      - POSTGRES_MASTER_HOST=postgres-primary

2. Service Scaling:

# Scale critical services
docker-compose up -d --scale trading_service=3 --scale api_gateway=2

3. Load Balancing:

# Add nginx for load balancing
services:
  nginx:
    image: nginx:alpine
    ports:
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf

Monitoring Dependencies

# Prometheus service discovery
scrape_configs:
  - job_name: 'api_gateway'
    static_configs:
      - targets: ['api_gateway:9091']

  - job_name: 'trading_service'
    static_configs:
      - targets: ['trading_service:9092']

  - job_name: 'postgres'
    static_configs:
      - targets: ['postgres_exporter:9187']

Dependency Health Dashboard

Grafana dashboard showing service dependencies:

{
  "dashboard": {
    "title": "Foxhunt Service Dependencies",
    "panels": [
      {
        "title": "Service Mesh Health",
        "targets": [
          {
            "expr": "up{job=~\"api_gateway|trading_service|backtesting_service|ml_training_service\"}"
          }
        ]
      },
      {
        "title": "Infrastructure Health",
        "targets": [
          {
            "expr": "up{job=~\"postgres|redis|vault\"}"
          }
        ]
      }
    ]
  }
}

Troubleshooting Dependency Issues

Check Dependency Chain

# Verify full dependency chain
scripts/check_dependencies.sh

Create scripts/check_dependencies.sh:

#!/bin/bash

echo "=== Checking Service Dependencies ==="

# Layer 1: Infrastructure
echo -e "\n[Infrastructure]"
docker exec foxhunt-postgres pg_isready -U foxhunt && echo "  PostgreSQL: ✓" || echo "  PostgreSQL: ✗"
docker exec foxhunt-redis redis-cli ping | grep -q PONG && echo "  Redis: ✓" || echo "  Redis: ✗"
curl -sf http://localhost:8200/v1/sys/health && echo "  Vault: ✓" || echo "  Vault: ✗"

# Layer 2: Core Services
echo -e "\n[Core Services]"
curl -sf http://localhost:8081/health && echo "  Trading Service: ✓" || echo "  Trading Service: ✗"
curl -sf http://localhost:8083/health && echo "  Backtesting Service: ✓" || echo "  Backtesting Service: ✗"
curl -sf http://localhost:8095/health && echo "  ML Training Service: ✓" || echo "  ML Training Service: ✗"

# Layer 3: Gateway
echo -e "\n[Gateway]"
grpc_health_probe -addr=localhost:50051 && echo "  API Gateway: ✓" || echo "  API Gateway: ✗"

echo -e "\n=== Dependency Check Complete ==="

References

• Docker Compose Startup Order
• Service Mesh Patterns
• Circuit Breaker Pattern
• Health Check Patterns