# Staging Environment Guide - Wave D Deployment Testing

**Created**: 2025-10-19 by Agent E1
**Status**: READY FOR WAVE D ROLLBACK TESTING
**Purpose**: Isolated staging environment for Wave D validation and rollback procedures

---

## Table of Contents

1. [Overview](#overview)
2. [Infrastructure Setup](#infrastructure-setup)
3. [Deployment Procedures](#deployment-procedures)
4. [Testing Procedures](#testing-procedures)
5. [Rollback Procedures](#rollback-procedures)
6. [Monitoring](#monitoring)
7. [Troubleshooting](#troubleshooting)

---

## Overview

The staging environment is a complete, isolated replica of the production system designed for Wave D testing. It runs in parallel with the development environment using offset ports.

### Key Features

- **Isolated Infrastructure**: Separate PostgreSQL, Redis, Vault, MinIO instances
- **Port Isolation**: All ports offset from dev to allow parallel operation
- **Wave D Support**: Migration 045 pre-applied with regime detection tables
- **Real Data**: Uses test data from `test_data/` directory (ES.FUT, NQ.FUT)
- **All 5 Microservices**: API Gateway, Trading Service, Backtesting Service, ML Training Service, Trading Agent Service

### Environment Comparison

| Component | Development | Staging | Port Offset |
|-----------|------------|---------|-------------|
| PostgreSQL | 5432 | 5433 | +1 |
| Redis | 6379 | 6380 | +1 |
| Vault | 8200 | 8201 | +1 |
| MinIO API | 9000 | 9002 | +2 |
| MinIO Console | 9001 | 9003 | +2 |
| API Gateway | 50051 | 50061 | +10 |
| Trading Service | 50052 | 50062 | +10 |
| Backtesting Service | 50053 | 50063 | +10 |
| ML Training Service | 50054 | 50064 | +10 |
| Trading Agent Service | 50055 | 50065 | +10 |

---

## Infrastructure Setup

### Prerequisites

1. Docker and Docker Compose installed
2. NVIDIA Docker runtime (for ML Training Service GPU support)
3. At least 16GB RAM available
4. Test data files in `test_data/` directory

### Quick Start

```bash
# 1. Deploy staging infrastructure
cd /home/jgrusewski/Work/foxhunt
docker-compose -f docker-compose.staging.yml up -d

# 2. Wait for services to be healthy (30-60 seconds)
docker ps --filter "name=staging" --format "table {{.Names}}\t{{.Status}}"

# 3. Verify database migration (already applied during Agent E1)
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "\dt" | grep regime

# Expected output:
#  regime_states
#  regime_transitions
#  adaptive_strategy_metrics
```

### Infrastructure Components

#### PostgreSQL Staging (foxhunt-postgres-staging)
- **Image**: `timescale/timescaledb:latest-pg16`
- **Port**: 5433 (external) → 5432 (internal)
- **Database**: `foxhunt_staging`
- **User**: `foxhunt`
- **Password**: `foxhunt_staging_password`
- **Connection**: `postgresql://foxhunt:foxhunt_staging_password@localhost:5433/foxhunt_staging`

#### Redis Staging (foxhunt-redis-staging)
- **Image**: `redis:7-alpine`
- **Port**: 6380 (external) → 6379 (internal)
- **Max Memory**: 2GB with `allkeys-lru` eviction
- **Connection**: `redis://localhost:6380`

#### Vault Staging (foxhunt-vault-staging)
- **Image**: `hashicorp/vault:1.15`
- **Port**: 8201 (external) → 8200 (internal)
- **Mode**: Dev mode (DO NOT use in production)
- **Root Token**: `foxhunt-staging-root`
- **Connection**: `http://localhost:8201`

#### MinIO Staging (foxhunt-minio-staging)
- **Image**: `minio/minio:latest`
- **API Port**: 9002 (external) → 9000 (internal)
- **Console Port**: 9003 (external) → 9001 (internal)
- **Access Key**: `foxhunt`
- **Secret Key**: `foxhunt_staging_password`
- **Bucket**: `ml-models-staging`
- **Console UI**: `http://localhost:9003`

---

## Deployment Procedures

### Full Deployment

```bash
# Deploy all staging services
docker-compose -f docker-compose.staging.yml up -d

# Expected output:
# Creating network "foxhunt-staging-network"
# Creating foxhunt-postgres-staging ... done
# Creating foxhunt-redis-staging    ... done
# Creating foxhunt-vault-staging     ... done
# Creating foxhunt-minio-staging     ... done
# Creating foxhunt-trading-service-staging ... done
# Creating foxhunt-backtesting-service-staging ... done
# Creating foxhunt-ml-training-service-staging ... done
# Creating foxhunt-trading-agent-service-staging ... done
# Creating foxhunt-api-gateway-staging ... done
```

### Selective Deployment

```bash
# Deploy infrastructure only
docker-compose -f docker-compose.staging.yml up -d postgres_staging redis_staging vault_staging minio_staging

# Deploy specific service (e.g., backtesting)
docker-compose -f docker-compose.staging.yml up -d backtesting_service_staging

# Scale down (stop all services but keep data)
docker-compose -f docker-compose.staging.yml down

# Complete cleanup (removes all data volumes - USE WITH CAUTION)
docker-compose -f docker-compose.staging.yml down -v
```

### Verify Deployment

```bash
# Check all staging containers
docker ps --filter "name=staging" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# Check service health
docker-compose -f docker-compose.staging.yml ps

# View logs for specific service
docker logs -f foxhunt-api-gateway-staging

# View logs for all services
docker-compose -f docker-compose.staging.yml logs -f
```

---

## Testing Procedures

### 1. Database Migration Testing

```bash
# Verify Wave D tables exist
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "
SELECT tablename FROM pg_tables
WHERE schemaname = 'public'
AND (tablename LIKE '%regime%' OR tablename LIKE '%adaptive%')
ORDER BY tablename;
"

# Expected output:
#  adaptive_strategy_metrics
#  regime_states
#  regime_transitions

# Test regime functions
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "
SELECT routine_name FROM information_schema.routines
WHERE routine_schema = 'public'
AND routine_name LIKE '%regime%'
ORDER BY routine_name;
"

# Expected output:
#  get_latest_regime
#  get_regime_performance
#  get_regime_transition_matrix
```

### 2. Service Health Checks

```bash
# API Gateway health (port 50061)
grpc_health_probe -addr=localhost:50061 || echo "API Gateway not healthy"

# Trading Service health
docker exec foxhunt-trading-service-staging /usr/local/bin/grpc_health_probe -addr=localhost:50051

# Backtesting Service health
curl -f http://localhost:8093/health || echo "Backtesting Service not healthy"

# ML Training Service health
curl -f http://localhost:8097/health || echo "ML Training Service not healthy"

# Trading Agent Service health
curl -f http://localhost:8085/health || echo "Trading Agent Service not healthy"
```

### 3. Load Test Data

```bash
# Verify test data files are accessible
docker exec foxhunt-backtesting-service-staging ls -lh /workspace/test_data/real/databento/

# Expected files:
#  ES.FUT_ohlcv-1m_2024-01-02.dbn
#  NQ.FUT_ohlcv-1m_2024-01-02.dbn
#  (and other DBN files)
```

### 4. End-to-End Smoke Tests

**Create E2E Test Script** (`staging_e2e_tests.sh`):

```bash
#!/bin/bash
# Staging E2E Smoke Tests - Wave D Validation

set -e

STAGING_GATEWAY="localhost:50061"
STAGING_DB="postgresql://foxhunt:foxhunt_staging_password@localhost:5433/foxhunt_staging"

echo "==================================="
echo "Staging E2E Smoke Tests - Wave D"
echo "==================================="

# Test 1: Database connectivity
echo "[TEST 1] Database connectivity..."
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "SELECT 1;" > /dev/null
echo "✓ Database connection successful"

# Test 2: Redis connectivity
echo "[TEST 2] Redis connectivity..."
docker exec foxhunt-redis-staging redis-cli ping | grep -q PONG
echo "✓ Redis connection successful"

# Test 3: Vault connectivity
echo "[TEST 3] Vault connectivity..."
docker exec foxhunt-vault-staging vault status > /dev/null
echo "✓ Vault connection successful"

# Test 4: MinIO connectivity
echo "[TEST 4] MinIO connectivity..."
curl -s http://localhost:9002/minio/health/live | grep -q "200"
echo "✓ MinIO connection successful"

# Test 5: Wave D migration verification
echo "[TEST 5] Wave D migration verification..."
REGIME_TABLES=$(docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -t -c "
SELECT COUNT(*) FROM pg_tables
WHERE schemaname = 'public'
AND (tablename LIKE '%regime%' OR tablename LIKE '%adaptive%');
")
if [ "$REGIME_TABLES" -eq 3 ]; then
    echo "✓ All 3 Wave D tables present"
else
    echo "✗ Expected 3 Wave D tables, found $REGIME_TABLES"
    exit 1
fi

# Test 6: Service health checks
echo "[TEST 6] Service health checks..."
for SERVICE in trading backtesting ml_training trading_agent api_gateway; do
    CONTAINER="foxhunt-${SERVICE//_/-}-service-staging"
    if [ "$SERVICE" = "api_gateway" ]; then
        CONTAINER="foxhunt-api-gateway-staging"
    fi

    if docker ps --filter "name=$CONTAINER" --filter "health=healthy" | grep -q "$CONTAINER"; then
        echo "✓ $SERVICE service healthy"
    else
        echo "✗ $SERVICE service NOT healthy"
        exit 1
    fi
done

# Test 7: Test data accessibility
echo "[TEST 7] Test data accessibility..."
ES_FILE=$(docker exec foxhunt-backtesting-service-staging ls /workspace/test_data/real/databento/ | grep "ES.FUT" | head -1)
if [ -n "$ES_FILE" ]; then
    echo "✓ ES.FUT test data accessible"
else
    echo "✗ ES.FUT test data NOT found"
    exit 1
fi

echo "==================================="
echo "All staging E2E tests PASSED ✓"
echo "==================================="
```

**Run E2E Tests**:

```bash
chmod +x staging_e2e_tests.sh
./staging_e2e_tests.sh
```

---

## Rollback Procedures

### Level 1: Service Rollback (No Data Loss)

If Wave D services have issues, rollback to pre-Wave D state while preserving data.

```bash
# 1. Stop all staging services
docker-compose -f docker-compose.staging.yml stop

# 2. Revert to previous service images (if available)
# Note: This requires pre-Wave D images to be tagged
docker tag foxhunt-trading-service:pre-wave-d foxhunt-trading-service:latest
docker tag foxhunt-api-gateway:pre-wave-d foxhunt-api-gateway:latest
# ... repeat for all services

# 3. Restart services with rollback images
docker-compose -f docker-compose.staging.yml up -d

# 4. Verify rollback
docker-compose -f docker-compose.staging.yml ps
```

### Level 2: Database Rollback (Revert Migration)

If Wave D database changes cause issues, rollback migration 045.

```bash
# 1. Stop all services accessing the database
docker-compose -f docker-compose.staging.yml stop trading_service_staging backtesting_service_staging ml_training_service_staging trading_agent_service_staging api_gateway_staging

# 2. Apply rollback migration (down script)
docker cp /home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.down.sql foxhunt-postgres-staging:/tmp/
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -f /tmp/045_wave_d_regime_tracking.down.sql

# 3. Verify Wave D tables removed
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "\dt" | grep -v regime

# 4. Restart services
docker-compose -f docker-compose.staging.yml up -d
```

### Level 3: Full Environment Reset (Nuclear Option)

Complete staging environment reset to clean state.

```bash
# WARNING: This destroys ALL staging data

# 1. Stop and remove all containers
docker-compose -f docker-compose.staging.yml down

# 2. Remove all volumes (data loss)
docker volume rm foxhunt-postgres-staging-data
docker volume rm foxhunt-redis-staging-data
docker volume rm foxhunt-vault-staging-data
docker volume rm foxhunt-minio-staging-data

# 3. Remove network
docker network rm foxhunt-staging-network

# 4. Redeploy from scratch
docker-compose -f docker-compose.staging.yml up -d

# 5. Reapply migrations
docker cp /home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.sql foxhunt-postgres-staging:/tmp/
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -f /tmp/045_wave_d_regime_tracking.sql
```

---

## Monitoring

### Container Monitoring

```bash
# View all staging containers
docker ps --filter "name=staging"

# Resource usage
docker stats --filter "name=staging" --no-stream

# Logs (real-time)
docker-compose -f docker-compose.staging.yml logs -f

# Logs (specific service)
docker logs -f foxhunt-backtesting-service-staging

# Logs (tail last 100 lines)
docker logs --tail 100 foxhunt-api-gateway-staging
```

### Database Monitoring

```bash
# Active connections
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "
SELECT count(*) as active_connections FROM pg_stat_activity WHERE state = 'active';
"

# Database size
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "
SELECT pg_size_pretty(pg_database_size('foxhunt_staging')) as db_size;
"

# Table sizes
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "
SELECT tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables WHERE schemaname = 'public' ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
"
```

### Redis Monitoring

```bash
# Redis info
docker exec foxhunt-redis-staging redis-cli INFO | grep -E "used_memory|connected_clients|uptime"

# Redis keys count
docker exec foxhunt-redis-staging redis-cli DBSIZE

# Redis memory usage
docker exec foxhunt-redis-staging redis-cli INFO memory | grep used_memory_human
```

### Service-Specific Monitoring

```bash
# Trading Service metrics (Prometheus format)
curl http://localhost:9102/metrics | grep -E "^foxhunt"

# Backtesting Service metrics
curl http://localhost:9103/metrics | grep -E "^foxhunt"

# ML Training Service metrics
curl http://localhost:9104/metrics | grep -E "^foxhunt"

# Trading Agent Service metrics
curl http://localhost:9105/metrics | grep -E "^foxhunt"

# API Gateway metrics
curl http://localhost:9101/metrics | grep -E "^foxhunt"
```

---

## Troubleshooting

### Common Issues

#### Issue 1: Port Already in Use

**Symptom**: `ERROR: for foxhunt-postgres-staging  Cannot start service postgres_staging: driver failed...`

**Solution**:
```bash
# Check what's using the port
lsof -i :5433

# Kill conflicting process or stop dev environment
docker-compose down

# Restart staging
docker-compose -f docker-compose.staging.yml up -d
```

#### Issue 2: Unhealthy Service

**Symptom**: Service shows `health: starting` or `unhealthy` status

**Solution**:
```bash
# Check service logs
docker logs foxhunt-<service>-staging

# Restart specific service
docker-compose -f docker-compose.staging.yml restart <service>_staging

# If persistent, rebuild
docker-compose -f docker-compose.staging.yml up -d --build <service>_staging
```

#### Issue 3: Database Migration Failed

**Symptom**: Wave D tables not present

**Solution**:
```bash
# Reapply migration manually
docker cp /home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.sql foxhunt-postgres-staging:/tmp/
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -f /tmp/045_wave_d_regime_tracking.sql

# Verify
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "\dt" | grep regime
```

#### Issue 4: Test Data Not Found

**Symptom**: `USE_DBN_DATA=true` but backtests fail

**Solution**:
```bash
# Verify test data is mounted
docker exec foxhunt-backtesting-service-staging ls -lh /workspace/test_data/real/databento/

# If missing, check docker-compose volume mount
# Should have: - ./test_data:/workspace/test_data:ro

# Restart service after fixing
docker-compose -f docker-compose.staging.yml restart backtesting_service_staging
```

#### Issue 5: Out of Memory

**Symptom**: Services crashing with OOM errors

**Solution**:
```bash
# Check Docker memory limits
docker stats --no-stream

# Increase Docker Desktop memory allocation (Settings → Resources → Memory)
# Recommended: 16GB for full staging deployment

# Or run infrastructure + specific services only
docker-compose -f docker-compose.staging.yml up -d postgres_staging redis_staging vault_staging backtesting_service_staging
```

### Getting Help

```bash
# View all staging logs for last 5 minutes
docker-compose -f docker-compose.staging.yml logs --since 5m

# Export logs for debugging
docker-compose -f docker-compose.staging.yml logs > staging_logs_$(date +%Y%m%d_%H%M%S).txt

# Check system resources
docker system df
docker system info
```

---

## Configuration Files

### Primary Files

- **`docker-compose.staging.yml`**: Staging Docker Compose configuration
- **`.env.staging`**: Staging environment variables
- **`migrations/045_wave_d_regime_tracking.sql`**: Wave D migration (up)
- **`migrations/045_wave_d_regime_tracking.down.sql`**: Wave D migration (down/rollback)

### Environment Variables (.env.staging)

Key staging-specific variables:

```bash
ENVIRONMENT=staging
JWT_SECRET=staging_jwt_secret_for_wave_d_testing_change_for_real_deployment
DATABASE_URL=postgresql://foxhunt:foxhunt_staging_password@localhost:5433/foxhunt_staging
REDIS_URL=redis://localhost:6380
VAULT_ADDR=http://localhost:8201
USE_DBN_DATA=true
```

---

## Next Steps for Wave D Testing

### Pre-Deployment Checklist

- [ ] All staging infrastructure services healthy
- [ ] Migration 045 applied successfully
- [ ] All 3 Wave D tables present (regime_states, regime_transitions, adaptive_strategy_metrics)
- [ ] Test data files accessible (ES.FUT, NQ.FUT)
- [ ] All 5 microservices deployed and healthy
- [ ] E2E smoke tests passing

### Wave D Validation Tests

1. **Regime Detection Testing**:
   - Load historical data (ES.FUT, NQ.FUT)
   - Verify regime classification (Trending/Ranging/Volatile)
   - Check regime transition logging
   - Validate CUSUM break detection

2. **Adaptive Strategy Testing**:
   - Test position sizing multipliers (0.2x-1.5x)
   - Test dynamic stop-loss (1.5x-4.0x ATR)
   - Verify regime-conditioned Sharpe calculation
   - Check risk budget utilization

3. **Performance Testing**:
   - Measure regime detection latency (<50μs target)
   - Test under high throughput (10k bars/sec)
   - Validate feature extraction (225 features)
   - Check memory usage (no leaks)

4. **Rollback Testing**:
   - Test Level 1 rollback (service revert)
   - Test Level 2 rollback (migration down)
   - Test Level 3 rollback (full reset)
   - Verify data integrity after rollback

### Success Criteria

- ✅ All E2E tests passing
- ✅ No service health issues for 1 hour continuous operation
- ✅ Regime detection accuracy >85% on test data
- ✅ Rollback procedures validated successfully
- ✅ Performance targets met (<50μs regime detection)

---

## Agent E1 Completion Summary

**Status**: ✅ **COMPLETE**

**Deliverables**:
1. ✅ `docker-compose.staging.yml` - Complete staging environment configuration
2. ✅ `.env.staging` - Staging environment variables
3. ✅ Staging infrastructure deployed (PostgreSQL, Redis, Vault, MinIO)
4. ✅ Migration 045 applied successfully to staging database
5. ✅ All 3 Wave D tables created and verified
6. ✅ Test data loaded and accessible (ES.FUT, NQ.FUT)
7. ✅ `STAGING_ENVIRONMENT_GUIDE.md` - Comprehensive documentation

**Next Agent**: E2 - Service Deployment & E2E Testing

---

**Document Version**: 1.0
**Last Updated**: 2025-10-19
**Maintained By**: Agent E1 (Staging Environment Deployment)