foxhunt/STAGING_ENVIRONMENT_GUIDE.md

Staging Environment Guide - Wave D Deployment Testing

Created: 2025-10-19 by Agent E1 Status: READY FOR WAVE D ROLLBACK TESTING Purpose: Isolated staging environment for Wave D validation and rollback procedures

---

Table of Contents

1. Overview
2. Infrastructure Setup
3. Deployment Procedures
4. Testing Procedures
5. Rollback Procedures
6. Monitoring
7. Troubleshooting

---

Overview

The staging environment is a complete, isolated replica of the production system designed for Wave D testing. It runs in parallel with the development environment using offset ports.

Key Features

• Isolated Infrastructure: Separate PostgreSQL, Redis, Vault, MinIO instances
• Port Isolation: All ports offset from dev to allow parallel operation
• Wave D Support: Migration 045 pre-applied with regime detection tables
• Real Data: Uses test data from test_data/ directory (ES.FUT, NQ.FUT)
• All 5 Microservices: API Gateway, Trading Service, Backtesting Service, ML Training Service, Trading Agent Service

Environment Comparison

Component	Development	Staging	Port Offset
PostgreSQL	5432	5433	+1
Redis	6379	6380	+1
Vault	8200	8201	+1
MinIO API	9000	9002	+2
MinIO Console	9001	9003	+2
API Gateway	50051	50061	+10
Trading Service	50052	50062	+10
Backtesting Service	50053	50063	+10
ML Training Service	50054	50064	+10
Trading Agent Service	50055	50065	+10

---

Infrastructure Setup

Prerequisites

1. Docker and Docker Compose installed
2. NVIDIA Docker runtime (for ML Training Service GPU support)
3. At least 16GB RAM available
4. Test data files in test_data/ directory

Quick Start

# 1. Deploy staging infrastructure
cd /home/jgrusewski/Work/foxhunt
docker-compose -f docker-compose.staging.yml up -d

# 2. Wait for services to be healthy (30-60 seconds)
docker ps --filter "name=staging" --format "table {{.Names}}\t{{.Status}}"

# 3. Verify database migration (already applied during Agent E1)
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "\dt" | grep regime

# Expected output:
#  regime_states
#  regime_transitions
#  adaptive_strategy_metrics

Infrastructure Components

PostgreSQL Staging (foxhunt-postgres-staging)

• Image: timescale/timescaledb:latest-pg16
• Port: 5433 (external) → 5432 (internal)
• Database: foxhunt_staging
• User: foxhunt
• Password: foxhunt_staging_password
• Connection: postgresql://foxhunt:foxhunt_staging_password@localhost:5433/foxhunt_staging

Redis Staging (foxhunt-redis-staging)

• Image: redis:7-alpine
• Port: 6380 (external) → 6379 (internal)
• Max Memory: 2GB with allkeys-lru eviction
• Connection: redis://localhost:6380

Vault Staging (foxhunt-vault-staging)

• Image: hashicorp/vault:1.15
• Port: 8201 (external) → 8200 (internal)
• Mode: Dev mode (DO NOT use in production)
• Root Token: foxhunt-staging-root
• Connection: http://localhost:8201

MinIO Staging (foxhunt-minio-staging)

• Image: minio/minio:latest
• API Port: 9002 (external) → 9000 (internal)
• Console Port: 9003 (external) → 9001 (internal)
• Access Key: foxhunt
• Secret Key: foxhunt_staging_password
• Bucket: ml-models-staging
• Console UI: http://localhost:9003

---

Deployment Procedures

Full Deployment

# Deploy all staging services
docker-compose -f docker-compose.staging.yml up -d

# Expected output:
# Creating network "foxhunt-staging-network"
# Creating foxhunt-postgres-staging ... done
# Creating foxhunt-redis-staging    ... done
# Creating foxhunt-vault-staging     ... done
# Creating foxhunt-minio-staging     ... done
# Creating foxhunt-trading-service-staging ... done
# Creating foxhunt-backtesting-service-staging ... done
# Creating foxhunt-ml-training-service-staging ... done
# Creating foxhunt-trading-agent-service-staging ... done
# Creating foxhunt-api-gateway-staging ... done

Selective Deployment

# Deploy infrastructure only
docker-compose -f docker-compose.staging.yml up -d postgres_staging redis_staging vault_staging minio_staging

# Deploy specific service (e.g., backtesting)
docker-compose -f docker-compose.staging.yml up -d backtesting_service_staging

# Scale down (stop all services but keep data)
docker-compose -f docker-compose.staging.yml down

# Complete cleanup (removes all data volumes - USE WITH CAUTION)
docker-compose -f docker-compose.staging.yml down -v

Verify Deployment

# Check all staging containers
docker ps --filter "name=staging" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# Check service health
docker-compose -f docker-compose.staging.yml ps

# View logs for specific service
docker logs -f foxhunt-api-gateway-staging

# View logs for all services
docker-compose -f docker-compose.staging.yml logs -f

---

Testing Procedures

1. Database Migration Testing

# Verify Wave D tables exist
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "
SELECT tablename FROM pg_tables
WHERE schemaname = 'public'
AND (tablename LIKE '%regime%' OR tablename LIKE '%adaptive%')
ORDER BY tablename;
"

# Expected output:
#  adaptive_strategy_metrics
#  regime_states
#  regime_transitions

# Test regime functions
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "
SELECT routine_name FROM information_schema.routines
WHERE routine_schema = 'public'
AND routine_name LIKE '%regime%'
ORDER BY routine_name;
"

# Expected output:
#  get_latest_regime
#  get_regime_performance
#  get_regime_transition_matrix

2. Service Health Checks

# API Gateway health (port 50061)
grpc_health_probe -addr=localhost:50061 || echo "API Gateway not healthy"

# Trading Service health
docker exec foxhunt-trading-service-staging /usr/local/bin/grpc_health_probe -addr=localhost:50051

# Backtesting Service health
curl -f http://localhost:8093/health || echo "Backtesting Service not healthy"

# ML Training Service health
curl -f http://localhost:8097/health || echo "ML Training Service not healthy"

# Trading Agent Service health
curl -f http://localhost:8085/health || echo "Trading Agent Service not healthy"

3. Load Test Data

# Verify test data files are accessible
docker exec foxhunt-backtesting-service-staging ls -lh /workspace/test_data/real/databento/

# Expected files:
#  ES.FUT_ohlcv-1m_2024-01-02.dbn
#  NQ.FUT_ohlcv-1m_2024-01-02.dbn
#  (and other DBN files)

4. End-to-End Smoke Tests

Create E2E Test Script (staging_e2e_tests.sh):

#!/bin/bash
# Staging E2E Smoke Tests - Wave D Validation

set -e

STAGING_GATEWAY="localhost:50061"
STAGING_DB="postgresql://foxhunt:foxhunt_staging_password@localhost:5433/foxhunt_staging"

echo "==================================="
echo "Staging E2E Smoke Tests - Wave D"
echo "==================================="

# Test 1: Database connectivity
echo "[TEST 1] Database connectivity..."
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "SELECT 1;" > /dev/null
echo "✓ Database connection successful"

# Test 2: Redis connectivity
echo "[TEST 2] Redis connectivity..."
docker exec foxhunt-redis-staging redis-cli ping | grep -q PONG
echo "✓ Redis connection successful"

# Test 3: Vault connectivity
echo "[TEST 3] Vault connectivity..."
docker exec foxhunt-vault-staging vault status > /dev/null
echo "✓ Vault connection successful"

# Test 4: MinIO connectivity
echo "[TEST 4] MinIO connectivity..."
curl -s http://localhost:9002/minio/health/live | grep -q "200"
echo "✓ MinIO connection successful"

# Test 5: Wave D migration verification
echo "[TEST 5] Wave D migration verification..."
REGIME_TABLES=$(docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -t -c "
SELECT COUNT(*) FROM pg_tables
WHERE schemaname = 'public'
AND (tablename LIKE '%regime%' OR tablename LIKE '%adaptive%');
")
if [ "$REGIME_TABLES" -eq 3 ]; then
    echo "✓ All 3 Wave D tables present"
else
    echo "✗ Expected 3 Wave D tables, found $REGIME_TABLES"
    exit 1
fi

# Test 6: Service health checks
echo "[TEST 6] Service health checks..."
for SERVICE in trading backtesting ml_training trading_agent api_gateway; do
    CONTAINER="foxhunt-${SERVICE//_/-}-service-staging"
    if [ "$SERVICE" = "api_gateway" ]; then
        CONTAINER="foxhunt-api-gateway-staging"
    fi

    if docker ps --filter "name=$CONTAINER" --filter "health=healthy" | grep -q "$CONTAINER"; then
        echo "✓ $SERVICE service healthy"
    else
        echo "✗ $SERVICE service NOT healthy"
        exit 1
    fi
done

# Test 7: Test data accessibility
echo "[TEST 7] Test data accessibility..."
ES_FILE=$(docker exec foxhunt-backtesting-service-staging ls /workspace/test_data/real/databento/ | grep "ES.FUT" | head -1)
if [ -n "$ES_FILE" ]; then
    echo "✓ ES.FUT test data accessible"
else
    echo "✗ ES.FUT test data NOT found"
    exit 1
fi

echo "==================================="
echo "All staging E2E tests PASSED ✓"
echo "==================================="

Run E2E Tests:

chmod +x staging_e2e_tests.sh
./staging_e2e_tests.sh

---

Rollback Procedures

Level 1: Service Rollback (No Data Loss)

If Wave D services have issues, rollback to pre-Wave D state while preserving data.

# 1. Stop all staging services
docker-compose -f docker-compose.staging.yml stop

# 2. Revert to previous service images (if available)
# Note: This requires pre-Wave D images to be tagged
docker tag foxhunt-trading-service:pre-wave-d foxhunt-trading-service:latest
docker tag foxhunt-api-gateway:pre-wave-d foxhunt-api-gateway:latest
# ... repeat for all services

# 3. Restart services with rollback images
docker-compose -f docker-compose.staging.yml up -d

# 4. Verify rollback
docker-compose -f docker-compose.staging.yml ps

Level 2: Database Rollback (Revert Migration)

If Wave D database changes cause issues, rollback migration 045.

# 1. Stop all services accessing the database
docker-compose -f docker-compose.staging.yml stop trading_service_staging backtesting_service_staging ml_training_service_staging trading_agent_service_staging api_gateway_staging

# 2. Apply rollback migration (down script)
docker cp /home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.down.sql foxhunt-postgres-staging:/tmp/
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -f /tmp/045_wave_d_regime_tracking.down.sql

# 3. Verify Wave D tables removed
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "\dt" | grep -v regime

# 4. Restart services
docker-compose -f docker-compose.staging.yml up -d

Level 3: Full Environment Reset (Nuclear Option)

Complete staging environment reset to clean state.

# WARNING: This destroys ALL staging data

# 1. Stop and remove all containers
docker-compose -f docker-compose.staging.yml down

# 2. Remove all volumes (data loss)
docker volume rm foxhunt-postgres-staging-data
docker volume rm foxhunt-redis-staging-data
docker volume rm foxhunt-vault-staging-data
docker volume rm foxhunt-minio-staging-data

# 3. Remove network
docker network rm foxhunt-staging-network

# 4. Redeploy from scratch
docker-compose -f docker-compose.staging.yml up -d

# 5. Reapply migrations
docker cp /home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.sql foxhunt-postgres-staging:/tmp/
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -f /tmp/045_wave_d_regime_tracking.sql

---

Monitoring

Container Monitoring

# View all staging containers
docker ps --filter "name=staging"

# Resource usage
docker stats --filter "name=staging" --no-stream

# Logs (real-time)
docker-compose -f docker-compose.staging.yml logs -f

# Logs (specific service)
docker logs -f foxhunt-backtesting-service-staging

# Logs (tail last 100 lines)
docker logs --tail 100 foxhunt-api-gateway-staging

Database Monitoring

# Active connections
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "
SELECT count(*) as active_connections FROM pg_stat_activity WHERE state = 'active';
"

# Database size
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "
SELECT pg_size_pretty(pg_database_size('foxhunt_staging')) as db_size;
"

# Table sizes
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "
SELECT tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables WHERE schemaname = 'public' ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
"

Redis Monitoring

# Redis info
docker exec foxhunt-redis-staging redis-cli INFO | grep -E "used_memory|connected_clients|uptime"

# Redis keys count
docker exec foxhunt-redis-staging redis-cli DBSIZE

# Redis memory usage
docker exec foxhunt-redis-staging redis-cli INFO memory | grep used_memory_human

Service-Specific Monitoring

# Trading Service metrics (Prometheus format)
curl http://localhost:9102/metrics | grep -E "^foxhunt"

# Backtesting Service metrics
curl http://localhost:9103/metrics | grep -E "^foxhunt"

# ML Training Service metrics
curl http://localhost:9104/metrics | grep -E "^foxhunt"

# Trading Agent Service metrics
curl http://localhost:9105/metrics | grep -E "^foxhunt"

# API Gateway metrics
curl http://localhost:9101/metrics | grep -E "^foxhunt"

---

Troubleshooting

Common Issues

Issue 1: Port Already in Use

Symptom: ERROR: for foxhunt-postgres-staging Cannot start service postgres_staging: driver failed...

Solution:

# Check what's using the port
lsof -i :5433

# Kill conflicting process or stop dev environment
docker-compose down

# Restart staging
docker-compose -f docker-compose.staging.yml up -d

Issue 2: Unhealthy Service

Symptom: Service shows health: starting or unhealthy status

Solution:

# Check service logs
docker logs foxhunt-<service>-staging

# Restart specific service
docker-compose -f docker-compose.staging.yml restart <service>_staging

# If persistent, rebuild
docker-compose -f docker-compose.staging.yml up -d --build <service>_staging

Issue 3: Database Migration Failed

Symptom: Wave D tables not present

Solution:

# Reapply migration manually
docker cp /home/jgrusewski/Work/foxhunt/migrations/045_wave_d_regime_tracking.sql foxhunt-postgres-staging:/tmp/
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -f /tmp/045_wave_d_regime_tracking.sql

# Verify
docker exec foxhunt-postgres-staging psql -U foxhunt -d foxhunt_staging -c "\dt" | grep regime

Issue 4: Test Data Not Found

Symptom: USE_DBN_DATA=true but backtests fail

Solution:

# Verify test data is mounted
docker exec foxhunt-backtesting-service-staging ls -lh /workspace/test_data/real/databento/

# If missing, check docker-compose volume mount
# Should have: - ./test_data:/workspace/test_data:ro

# Restart service after fixing
docker-compose -f docker-compose.staging.yml restart backtesting_service_staging

Issue 5: Out of Memory

Symptom: Services crashing with OOM errors

Solution:

# Check Docker memory limits
docker stats --no-stream

# Increase Docker Desktop memory allocation (Settings → Resources → Memory)
# Recommended: 16GB for full staging deployment

# Or run infrastructure + specific services only
docker-compose -f docker-compose.staging.yml up -d postgres_staging redis_staging vault_staging backtesting_service_staging

Getting Help

# View all staging logs for last 5 minutes
docker-compose -f docker-compose.staging.yml logs --since 5m

# Export logs for debugging
docker-compose -f docker-compose.staging.yml logs > staging_logs_$(date +%Y%m%d_%H%M%S).txt

# Check system resources
docker system df
docker system info

---

Configuration Files

Primary Files

• docker-compose.staging.yml: Staging Docker Compose configuration
• .env.staging: Staging environment variables
• migrations/045_wave_d_regime_tracking.sql: Wave D migration (up)
• migrations/045_wave_d_regime_tracking.down.sql: Wave D migration (down/rollback)

Environment Variables (.env.staging)

Key staging-specific variables:

ENVIRONMENT=staging
JWT_SECRET=staging_jwt_secret_for_wave_d_testing_change_for_real_deployment
DATABASE_URL=postgresql://foxhunt:foxhunt_staging_password@localhost:5433/foxhunt_staging
REDIS_URL=redis://localhost:6380
VAULT_ADDR=http://localhost:8201
USE_DBN_DATA=true

---

Next Steps for Wave D Testing

Pre-Deployment Checklist

• All staging infrastructure services healthy
• Migration 045 applied successfully
• All 3 Wave D tables present (regime_states, regime_transitions, adaptive_strategy_metrics)
• Test data files accessible (ES.FUT, NQ.FUT)
• All 5 microservices deployed and healthy
• E2E smoke tests passing

Wave D Validation Tests

1. Regime Detection Testing:

   • Load historical data (ES.FUT, NQ.FUT)
   • Verify regime classification (Trending/Ranging/Volatile)
   • Check regime transition logging
   • Validate CUSUM break detection

2. Adaptive Strategy Testing:

   • Test position sizing multipliers (0.2x-1.5x)
   • Test dynamic stop-loss (1.5x-4.0x ATR)
   • Verify regime-conditioned Sharpe calculation
   • Check risk budget utilization

3. Performance Testing:

   • Measure regime detection latency (<50μs target)
   • Test under high throughput (10k bars/sec)
   • Validate feature extraction (225 features)
   • Check memory usage (no leaks)

4. Rollback Testing:

   • Test Level 1 rollback (service revert)
   • Test Level 2 rollback (migration down)
   • Test Level 3 rollback (full reset)
   • Verify data integrity after rollback

Success Criteria

• ✅ All E2E tests passing
• ✅ No service health issues for 1 hour continuous operation
• ✅ Regime detection accuracy >85% on test data
• ✅ Rollback procedures validated successfully
• ✅ Performance targets met (<50μs regime detection)

---

Agent E1 Completion Summary

Status: ✅ COMPLETE

Deliverables:

1. ✅ docker-compose.staging.yml - Complete staging environment configuration
2. ✅ .env.staging - Staging environment variables
3. ✅ Staging infrastructure deployed (PostgreSQL, Redis, Vault, MinIO)
4. ✅ Migration 045 applied successfully to staging database
5. ✅ All 3 Wave D tables created and verified
6. ✅ Test data loaded and accessible (ES.FUT, NQ.FUT)
7. ✅ STAGING_ENVIRONMENT_GUIDE.md - Comprehensive documentation

Next Agent: E2 - Service Deployment & E2E Testing

---

Document Version: 1.0 Last Updated: 2025-10-19 Maintained By: Agent E1 (Staging Environment Deployment)