foxhunt/SERVICE_ROLLBACK_MATRIX.md

Service Rollback Matrix

Purpose: Quick reference for service-specific rollback procedures Last Updated: 2025-10-18

---

📊 Service Dependency Map

┌─────────────────────────────────────────┐
│         Infrastructure Layer            │
│  PostgreSQL │ Redis │ Vault │ InfluxDB  │
└──────────────┬──────────────────────────┘
               │
        ┌──────┴──────┐
        ▼             ▼
┌──────────────┐  ┌──────────────┐
│Trading Service│  │Backtesting  │
│  (Port 50052)│  │Service       │
└──────┬────────┘  └──────┬───────┘
       │                  │
       └──────────┬───────┘
                  ▼
         ┌──────────────┐
         │ML Training   │
         │Service       │
         └──────┬────────┘
                │
         ┌──────┴───────┐
         ▼              ▼
┌──────────────┐  ┌──────────────┐
│Trading Agent │  │  API Gateway  │
│Service       │  │  (Port 50051) │
└──────────────┘  └───────────────┘

Rollback Order (bottom-up): API Gateway → Trading Agent → ML Training → Backtesting → Trading → Infrastructure

---

🔧 Service Rollback Details

1. Trading Service

Port: 50052 (gRPC), 9092 (Metrics) Dependencies: PostgreSQL, Redis, Vault Critical: ⚠️ YES - Handles all trade execution

Rollback Commands

# Stop service
docker-compose stop trading_service

# Tag current version
docker tag foxhunt_trading_service:latest foxhunt_trading_service:backup_$(date +%Y%m%d_%H%M%S)

# Rollback to previous version
docker tag foxhunt_trading_service:wave_c_stable foxhunt_trading_service:latest

# Restart with rollback version
docker-compose up -d trading_service

# Verify health
grpc_health_probe -addr=localhost:50052
curl http://localhost:9092/metrics

Time Estimates

• Stop: 0.5s
• Start: 1.2s
• Health Check: 2s
• Total Downtime: 3.7 seconds

Verification Steps

# Check service status
docker logs foxhunt-trading-service --tail 50 | grep -E "Starting|Listening|Ready"

# Test gRPC endpoint
grpcurl -plaintext localhost:50052 grpc.health.v1.Health/Check

# Verify database connectivity
psql $DATABASE_URL -c "SELECT COUNT(*) FROM trading_events WHERE created_at > NOW() - INTERVAL '1 hour';"

Rollback Success Criteria

• ✅ Service status: healthy
• ✅ gRPC endpoint responsive
• ✅ Database connectivity confirmed
• ✅ Redis connection active
• ✅ Metrics endpoint returning data

---

2. API Gateway

Port: 50051 (gRPC), 9091 (Metrics) Dependencies: All backend services Critical: ⚠️ YES - Single entry point for all clients

Rollback Commands

# Stop service (clients will fail during this time)
docker-compose stop api_gateway

# Tag and rollback
docker tag foxhunt_api_gateway:latest foxhunt_api_gateway:backup_$(date +%Y%m%d_%H%M%S)
docker tag foxhunt_api_gateway:wave_c_stable foxhunt_api_gateway:latest

# Restart
docker-compose up -d api_gateway

# Verify health
grpc_health_probe -addr=localhost:50051

Time Estimates

• Stop: 0.5s
• Start: 1.5s
• Health Check: 2s
• Total Downtime: 4.0 seconds

Verification Steps

# Test authentication
tli auth login --username test_user

# Test routing to backend services
tli trade ml submit --symbol ES.FUT --action BUY --quantity 1 --test-mode

# Check rate limiting
for i in {1..10}; do curl -s http://localhost:50051/health; done

Rollback Success Criteria

• ✅ Service status: healthy
• ✅ JWT authentication working
• ✅ Rate limiting operational
• ✅ Audit logging active
• ✅ All backend service routes functional

---

3. Backtesting Service

Port: 50053 (gRPC), 9093 (Metrics), 8082 (Health) Dependencies: PostgreSQL, Redis, Vault Critical: ❌ NO - Non-critical for live trading

Rollback Commands

# Stop service (no impact on live trading)
docker-compose stop backtesting_service

# Tag and rollback
docker tag foxhunt_backtesting_service:latest foxhunt_backtesting_service:backup_$(date +%Y%m%d_%H%M%S)
docker tag foxhunt_backtesting_service:wave_c_stable foxhunt_backtesting_service:latest

# Restart
docker-compose up -d backtesting_service

# Verify health
curl http://localhost:8082/health

Time Estimates

• Stop: 0.5s
• Start: 2.0s
• Health Check: 3s
• Total Downtime: 5.5 seconds (zero impact on trading)

Verification Steps

# Test backtest execution
tli backtest ml run --symbol ES.FUT --start 2024-01-01 --end 2024-01-02 --test-mode

# Verify DBN data loading
docker logs foxhunt-backtesting-service | grep -i "dbn"

# Check performance
curl http://localhost:9093/metrics | grep backtest_duration

Rollback Success Criteria

• ✅ Service status: healthy
• ✅ DBN data loading functional
• ✅ Backtest execution working
• ✅ Performance within baseline

---

4. ML Training Service

Port: 50054 (gRPC), 9094 (Metrics), 8095 (Health) Dependencies: PostgreSQL, Redis, Vault, MinIO, GPU Critical: ❌ NO - Non-critical for live trading

Rollback Commands

# Stop service (no impact on live trading)
docker-compose stop ml_training_service

# Tag and rollback
docker tag foxhunt_ml_training_service:latest foxhunt_ml_training_service:backup_$(date +%Y%m%d_%H%M%S)
docker tag foxhunt_ml_training_service:wave_c_stable foxhunt_ml_training_service:latest

# Restart (GPU initialization takes time)
docker-compose up -d ml_training_service

# Verify health
curl http://localhost:8095/health
nvidia-smi  # Verify GPU allocated

Time Estimates

• Stop: 0.5s
• Start: 8.0s (GPU initialization)
• Health Check: 5s
• Total Downtime: 13.5 seconds (zero impact on trading)

Verification Steps

# Verify GPU access
docker exec foxhunt-ml-training-service nvidia-smi

# Test model loading
docker logs foxhunt-ml-training-service | grep -E "CUDA|GPU|Model loaded"

# Check MinIO connectivity
curl http://localhost:9000/minio/health/live

Rollback Success Criteria

• ✅ Service status: healthy
• ✅ GPU accessible (CUDA available)
• ✅ MinIO connectivity confirmed
• ✅ Model loading functional
• ✅ Training pipeline operational

---

5. Trading Agent Service

Port: 50055 (gRPC), 9095 (Metrics), 8083 (Health) Dependencies: Trading Service, PostgreSQL, Redis, Vault Critical: ⚠️ YES - Orchestrates trading decisions

Rollback Commands

# Stop service (disables autonomous trading)
docker-compose stop trading_agent_service

# Tag and rollback
docker tag foxhunt_trading_agent_service:latest foxhunt_trading_agent_service:backup_$(date +%Y%m%d_%H%M%S)
docker tag foxhunt_trading_agent_service:wave_c_stable foxhunt_trading_agent_service:latest

# Restart
docker-compose up -d trading_agent_service

# Verify health
curl http://localhost:8083/health

Time Estimates

• Stop: 0.5s
• Start: 1.5s
• Health Check: 2s
• Total Downtime: 4.0 seconds

Verification Steps

# Test agent decision making
tli agent status

# Verify portfolio allocation
psql $DATABASE_URL -c "SELECT * FROM portfolio_allocations ORDER BY created_at DESC LIMIT 5;"

# Check trading universe
psql $DATABASE_URL -c "SELECT * FROM trading_universes ORDER BY created_at DESC LIMIT 1;"

Rollback Success Criteria

• ✅ Service status: healthy
• ✅ Agent decision loop operational
• ✅ Portfolio allocation functional
• ✅ Universe selection working
• ✅ Order submission to Trading Service successful

---

🚦 Rollback Coordination Matrix

Zero-Downtime Rollback Order (for non-critical services)

1. ML Training Service (13.5s) - Start first (longest)
2. Backtesting Service (5.5s)
3. Trading Agent Service (4.0s)
4. Trading Service (3.7s)
5. API Gateway (4.0s) - Last (shortest downtime)

Total Time: ~31 seconds (services rolled in parallel where possible)

Critical-Path Rollback Order (for trading disruption)

1. Stop Trading: Pause TLI trading (30s)
2. API Gateway: Rollback (4.0s) - Prevents new requests
3. Trading Service: Rollback (3.7s) - Core trading logic
4. Trading Agent Service: Rollback (4.0s) - Decision orchestration
5. Backtesting/ML Training: Rollback in parallel (13.5s) - Non-critical
6. Resume Trading: Re-enable TLI (30s)

Total Time: ~85 seconds (~1.5 minutes)

---

🔍 Service Health Check Commands

Quick Health Check (all services)

docker-compose ps | grep foxhunt | grep -v "Up.*healthy" && echo "❌ Unhealthy services found" || echo "✅ All services healthy"

Individual Service Health Checks

# Trading Service
grpc_health_probe -addr=localhost:50052 && curl -s http://localhost:9092/health

# API Gateway
grpc_health_probe -addr=localhost:50051 && curl -s http://localhost:9091/health

# Backtesting Service
curl -s http://localhost:8082/health

# ML Training Service
curl -s http://localhost:8095/health && docker exec foxhunt-ml-training-service nvidia-smi > /dev/null

# Trading Agent Service
curl -s http://localhost:8083/health

Comprehensive System Health Check

#!/bin/bash
echo "=== Foxhunt System Health Check ==="

services=("trading_service:50052" "api_gateway:50051" "backtesting_service:8082" "ml_training_service:8095" "trading_agent_service:8083")

for svc in "${services[@]}"; do
    name=$(echo $svc | cut -d: -f1)
    port=$(echo $svc | cut -d: -f2)

    if [[ $port -lt 10000 ]]; then
        # HTTP health check
        if curl -sf http://localhost:$port/health > /dev/null; then
            echo "✅ $name (HTTP $port) - Healthy"
        else
            echo "❌ $name (HTTP $port) - Unhealthy"
        fi
    else
        # gRPC health check
        if grpc_health_probe -addr=localhost:$port > /dev/null 2>&1; then
            echo "✅ $name (gRPC $port) - Healthy"
        else
            echo "❌ $name (gRPC $port) - Unhealthy"
        fi
    fi
done

echo ""
echo "=== Infrastructure Health ==="
psql $DATABASE_URL -c "SELECT 1;" > /dev/null && echo "✅ PostgreSQL - Connected" || echo "❌ PostgreSQL - Failed"
redis-cli ping > /dev/null && echo "✅ Redis - Connected" || echo "❌ Redis - Failed"
curl -s http://localhost:8200/v1/sys/health | jq -r .initialized > /dev/null && echo "✅ Vault - Initialized" || echo "❌ Vault - Failed"

---

📊 Rollback Decision Matrix

Severity	Symptom	Rollback Type	Estimated Time
P0 - Critical	Trading halted, data corruption	Full System	5-10 min
P1 - High	Service crashes, performance degradation >50%	Service-Specific	2-5 min
P2 - Medium	Migration failure, non-critical service down	Database or Service	1-3 min
P3 - Low	Minor bugs, logging issues	Hotfix (no rollback)	N/A

---

🛡️ Rollback Safety Checklist

Before initiating any service rollback:

• Alert stakeholders - Notify team in Slack/incident channel
• Stop trading (if critical service) - Pause TLI trading
• Verify backup exists - Confirm previous version tagged
• Check dependencies - Ensure no breaking changes between versions
• Monitor ready - Open Grafana dashboard for real-time metrics
• Logs captured - Save current logs before rollback
• Database snapshot - Create backup if rolling back DB migrations

After rollback:

• Health checks pass - All services reporting healthy
• Smoke tests pass - Basic functionality verified
• Metrics baseline - Performance within ±10% of previous
• No error logs - Check logs for 5 minutes post-rollback
• Trading enabled - Resume live trading if stopped
• Incident documented - Log reason and outcome

---

Last Updated: 2025-10-18 Next Review: After any major deployment or incident