foxhunt/docs/archive/feature_implementation/SERVICE_ROLLBACK_MATRIX.md

# Service Rollback Matrix

**Purpose**: Quick reference for service-specific rollback procedures
**Last Updated**: 2025-10-18

---

## 📊 Service Dependency Map

```
┌─────────────────────────────────────────┐
│         Infrastructure Layer            │
│  PostgreSQL │ Redis │ Vault │ InfluxDB  │
└──────────────┬──────────────────────────┘
               │
        ┌──────┴──────┐
        ▼             ▼
┌──────────────┐  ┌──────────────┐
│Trading Service│  │Backtesting  │
│  (Port 50052)│  │Service       │
└──────┬────────┘  └──────┬───────┘
       │                  │
       └──────────┬───────┘
                  ▼
         ┌──────────────┐
         │ML Training   │
         │Service       │
         └──────┬────────┘
                │
         ┌──────┴───────┐
         ▼              ▼
┌──────────────┐  ┌──────────────┐
│Trading Agent │  │  API Gateway  │
│Service       │  │  (Port 50051) │
└──────────────┘  └───────────────┘
```

**Rollback Order** (bottom-up): API Gateway → Trading Agent → ML Training → Backtesting → Trading → Infrastructure

---

## 🔧 Service Rollback Details

### 1. Trading Service
**Port**: 50052 (gRPC), 9092 (Metrics)
**Dependencies**: PostgreSQL, Redis, Vault
**Critical**: ⚠️ **YES** - Handles all trade execution

#### Rollback Commands
```bash
# Stop service
docker-compose stop trading_service

# Tag current version
docker tag foxhunt_trading_service:latest foxhunt_trading_service:backup_$(date +%Y%m%d_%H%M%S)

# Rollback to previous version
docker tag foxhunt_trading_service:wave_c_stable foxhunt_trading_service:latest

# Restart with rollback version
docker-compose up -d trading_service

# Verify health
grpc_health_probe -addr=localhost:50052
curl http://localhost:9092/metrics
```

#### Time Estimates
- Stop: 0.5s
- Start: 1.2s
- Health Check: 2s
- **Total Downtime**: **3.7 seconds**

#### Verification Steps
```bash
# Check service status
docker logs foxhunt-trading-service --tail 50 | grep -E "Starting|Listening|Ready"

# Test gRPC endpoint
grpcurl -plaintext localhost:50052 grpc.health.v1.Health/Check

# Verify database connectivity
psql $DATABASE_URL -c "SELECT COUNT(*) FROM trading_events WHERE created_at > NOW() - INTERVAL '1 hour';"
```

#### Rollback Success Criteria
- ✅ Service status: `healthy`
- ✅ gRPC endpoint responsive
- ✅ Database connectivity confirmed
- ✅ Redis connection active
- ✅ Metrics endpoint returning data

---

### 2. API Gateway
**Port**: 50051 (gRPC), 9091 (Metrics)
**Dependencies**: All backend services
**Critical**: ⚠️ **YES** - Single entry point for all clients

#### Rollback Commands
```bash
# Stop service (clients will fail during this time)
docker-compose stop api_gateway

# Tag and rollback
docker tag foxhunt_api_gateway:latest foxhunt_api_gateway:backup_$(date +%Y%m%d_%H%M%S)
docker tag foxhunt_api_gateway:wave_c_stable foxhunt_api_gateway:latest

# Restart
docker-compose up -d api_gateway

# Verify health
grpc_health_probe -addr=localhost:50051
```

#### Time Estimates
- Stop: 0.5s
- Start: 1.5s
- Health Check: 2s
- **Total Downtime**: **4.0 seconds**

#### Verification Steps
```bash
# Test authentication
tli auth login --username test_user

# Test routing to backend services
tli trade ml submit --symbol ES.FUT --action BUY --quantity 1 --test-mode

# Check rate limiting
for i in {1..10}; do curl -s http://localhost:50051/health; done
```

#### Rollback Success Criteria
- ✅ Service status: `healthy`
- ✅ JWT authentication working
- ✅ Rate limiting operational
- ✅ Audit logging active
- ✅ All backend service routes functional

---

### 3. Backtesting Service
**Port**: 50053 (gRPC), 9093 (Metrics), 8082 (Health)
**Dependencies**: PostgreSQL, Redis, Vault
**Critical**: ❌ **NO** - Non-critical for live trading

#### Rollback Commands
```bash
# Stop service (no impact on live trading)
docker-compose stop backtesting_service

# Tag and rollback
docker tag foxhunt_backtesting_service:latest foxhunt_backtesting_service:backup_$(date +%Y%m%d_%H%M%S)
docker tag foxhunt_backtesting_service:wave_c_stable foxhunt_backtesting_service:latest

# Restart
docker-compose up -d backtesting_service

# Verify health
curl http://localhost:8082/health
```

#### Time Estimates
- Stop: 0.5s
- Start: 2.0s
- Health Check: 3s
- **Total Downtime**: **5.5 seconds** (zero impact on trading)

#### Verification Steps
```bash
# Test backtest execution
tli backtest ml run --symbol ES.FUT --start 2024-01-01 --end 2024-01-02 --test-mode

# Verify DBN data loading
docker logs foxhunt-backtesting-service | grep -i "dbn"

# Check performance
curl http://localhost:9093/metrics | grep backtest_duration
```

#### Rollback Success Criteria
- ✅ Service status: `healthy`
- ✅ DBN data loading functional
- ✅ Backtest execution working
- ✅ Performance within baseline

---

### 4. ML Training Service
**Port**: 50054 (gRPC), 9094 (Metrics), 8095 (Health)
**Dependencies**: PostgreSQL, Redis, Vault, MinIO, GPU
**Critical**: ❌ **NO** - Non-critical for live trading

#### Rollback Commands
```bash
# Stop service (no impact on live trading)
docker-compose stop ml_training_service

# Tag and rollback
docker tag foxhunt_ml_training_service:latest foxhunt_ml_training_service:backup_$(date +%Y%m%d_%H%M%S)
docker tag foxhunt_ml_training_service:wave_c_stable foxhunt_ml_training_service:latest

# Restart (GPU initialization takes time)
docker-compose up -d ml_training_service

# Verify health
curl http://localhost:8095/health
nvidia-smi  # Verify GPU allocated
```

#### Time Estimates
- Stop: 0.5s
- Start: 8.0s (GPU initialization)
- Health Check: 5s
- **Total Downtime**: **13.5 seconds** (zero impact on trading)

#### Verification Steps
```bash
# Verify GPU access
docker exec foxhunt-ml-training-service nvidia-smi

# Test model loading
docker logs foxhunt-ml-training-service | grep -E "CUDA|GPU|Model loaded"

# Check MinIO connectivity
curl http://localhost:9000/minio/health/live
```

#### Rollback Success Criteria
- ✅ Service status: `healthy`
- ✅ GPU accessible (CUDA available)
- ✅ MinIO connectivity confirmed
- ✅ Model loading functional
- ✅ Training pipeline operational

---

### 5. Trading Agent Service
**Port**: 50055 (gRPC), 9095 (Metrics), 8083 (Health)
**Dependencies**: Trading Service, PostgreSQL, Redis, Vault
**Critical**: ⚠️ **YES** - Orchestrates trading decisions

#### Rollback Commands
```bash
# Stop service (disables autonomous trading)
docker-compose stop trading_agent_service

# Tag and rollback
docker tag foxhunt_trading_agent_service:latest foxhunt_trading_agent_service:backup_$(date +%Y%m%d_%H%M%S)
docker tag foxhunt_trading_agent_service:wave_c_stable foxhunt_trading_agent_service:latest

# Restart
docker-compose up -d trading_agent_service

# Verify health
curl http://localhost:8083/health
```

#### Time Estimates
- Stop: 0.5s
- Start: 1.5s
- Health Check: 2s
- **Total Downtime**: **4.0 seconds**

#### Verification Steps
```bash
# Test agent decision making
tli agent status

# Verify portfolio allocation
psql $DATABASE_URL -c "SELECT * FROM portfolio_allocations ORDER BY created_at DESC LIMIT 5;"

# Check trading universe
psql $DATABASE_URL -c "SELECT * FROM trading_universes ORDER BY created_at DESC LIMIT 1;"
```

#### Rollback Success Criteria
- ✅ Service status: `healthy`
- ✅ Agent decision loop operational
- ✅ Portfolio allocation functional
- ✅ Universe selection working
- ✅ Order submission to Trading Service successful

---

## 🚦 Rollback Coordination Matrix

### Zero-Downtime Rollback Order (for non-critical services)
1. ML Training Service (13.5s) - Start first (longest)
2. Backtesting Service (5.5s)
3. Trading Agent Service (4.0s)
4. Trading Service (3.7s)
5. API Gateway (4.0s) - Last (shortest downtime)

**Total Time**: ~31 seconds (services rolled in parallel where possible)

### Critical-Path Rollback Order (for trading disruption)
1. **Stop Trading**: Pause TLI trading (30s)
2. **API Gateway**: Rollback (4.0s) - Prevents new requests
3. **Trading Service**: Rollback (3.7s) - Core trading logic
4. **Trading Agent Service**: Rollback (4.0s) - Decision orchestration
5. **Backtesting/ML Training**: Rollback in parallel (13.5s) - Non-critical
6. **Resume Trading**: Re-enable TLI (30s)

**Total Time**: ~85 seconds (~1.5 minutes)

---

## 🔍 Service Health Check Commands

### Quick Health Check (all services)
```bash
docker-compose ps | grep foxhunt | grep -v "Up.*healthy" && echo "❌ Unhealthy services found" || echo "✅ All services healthy"
```

### Individual Service Health Checks
```bash
# Trading Service
grpc_health_probe -addr=localhost:50052 && curl -s http://localhost:9092/health

# API Gateway
grpc_health_probe -addr=localhost:50051 && curl -s http://localhost:9091/health

# Backtesting Service
curl -s http://localhost:8082/health

# ML Training Service
curl -s http://localhost:8095/health && docker exec foxhunt-ml-training-service nvidia-smi > /dev/null

# Trading Agent Service
curl -s http://localhost:8083/health
```

### Comprehensive System Health Check
```bash
#!/bin/bash
echo "=== Foxhunt System Health Check ==="

services=("trading_service:50052" "api_gateway:50051" "backtesting_service:8082" "ml_training_service:8095" "trading_agent_service:8083")

for svc in "${services[@]}"; do
    name=$(echo $svc | cut -d: -f1)
    port=$(echo $svc | cut -d: -f2)

    if [[ $port -lt 10000 ]]; then
        # HTTP health check
        if curl -sf http://localhost:$port/health > /dev/null; then
            echo "✅ $name (HTTP $port) - Healthy"
        else
            echo "❌ $name (HTTP $port) - Unhealthy"
        fi
    else
        # gRPC health check
        if grpc_health_probe -addr=localhost:$port > /dev/null 2>&1; then
            echo "✅ $name (gRPC $port) - Healthy"
        else
            echo "❌ $name (gRPC $port) - Unhealthy"
        fi
    fi
done

echo ""
echo "=== Infrastructure Health ==="
psql $DATABASE_URL -c "SELECT 1;" > /dev/null && echo "✅ PostgreSQL - Connected" || echo "❌ PostgreSQL - Failed"
redis-cli ping > /dev/null && echo "✅ Redis - Connected" || echo "❌ Redis - Failed"
curl -s http://localhost:8200/v1/sys/health | jq -r .initialized > /dev/null && echo "✅ Vault - Initialized" || echo "❌ Vault - Failed"
```

---

## 📊 Rollback Decision Matrix

| Severity | Symptom | Rollback Type | Estimated Time |
|---|---|---|---|
| **P0 - Critical** | Trading halted, data corruption | Full System | 5-10 min |
| **P1 - High** | Service crashes, performance degradation >50% | Service-Specific | 2-5 min |
| **P2 - Medium** | Migration failure, non-critical service down | Database or Service | 1-3 min |
| **P3 - Low** | Minor bugs, logging issues | Hotfix (no rollback) | N/A |

---

## 🛡️ Rollback Safety Checklist

Before initiating any service rollback:

- [ ] **Alert stakeholders** - Notify team in Slack/incident channel
- [ ] **Stop trading** (if critical service) - Pause TLI trading
- [ ] **Verify backup exists** - Confirm previous version tagged
- [ ] **Check dependencies** - Ensure no breaking changes between versions
- [ ] **Monitor ready** - Open Grafana dashboard for real-time metrics
- [ ] **Logs captured** - Save current logs before rollback
- [ ] **Database snapshot** - Create backup if rolling back DB migrations

After rollback:

- [ ] **Health checks pass** - All services reporting healthy
- [ ] **Smoke tests pass** - Basic functionality verified
- [ ] **Metrics baseline** - Performance within ±10% of previous
- [ ] **No error logs** - Check logs for 5 minutes post-rollback
- [ ] **Trading enabled** - Resume live trading if stopped
- [ ] **Incident documented** - Log reason and outcome

---

**Last Updated**: 2025-10-18
**Next Review**: After any major deployment or incident