foxhunt/WAVE_D_PRODUCTION_DEPLOYMENT_PLAN.md

Wave D Production Deployment Plan

Document Version: 1.0 Date: 2025-10-19 Status: Ready for Execution Total Timeline: 26-28 hours (2-4h deployment + 24h observation + 30m certification)

---

Executive Summary

This document provides a comprehensive, step-by-step deployment plan for Wave D (Regime Detection & Adaptive Strategies) to production. The system has achieved 100% production readiness with all blockers resolved, 99.4% test pass rate (2,062/2,074), and 922x average performance vs. targets.

Deployment Scope:

• 5 microservices (API Gateway, Trading, ML Training, Trading Agent, Backtesting)
• Database migration 045 (3 new tables: regime_states, regime_transitions, adaptive_strategy_metrics)
• 225-feature extraction pipeline (201 Wave C + 24 Wave D)
• 8 regime detection modules + 4 adaptive strategies
• Monitoring stack (Prometheus alerts, Grafana dashboards)

Risk Level: LOW (independent microservices, per-service rollback capability, comprehensive monitoring)

---

Deployment Architecture

┌─────────────────────────────────────────────────────────────┐
│                    DEPLOYMENT SEQUENCE                       │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  Phase 1: Pre-Deployment (30m)                              │
│  └─> Infrastructure validation                              │
│      Database backup                                         │
│      Service compilation                                     │
│      Monitoring setup                                        │
│                                                              │
│  Phase 2: Database Migration (15m)                          │
│  └─> Migration 045 (regime detection tables)                │
│      Validation & rollback prep                              │
│                                                              │
│  Phase 3: Service Deployment (60m)                          │
│  └─> API Gateway (15m)                                       │
│      Trading Service (10m)                                   │
│      ML Training Service (10m)                               │
│      Trading Agent Service (15m)                             │
│      Backtesting Service (10m)                               │
│                                                              │
│  Phase 4: Smoke Tests (20m)                                 │
│  └─> 5 critical tests (auth, regime, order, ML, monitoring) │
│                                                              │
│  Phase 5: Monitoring Configuration (15m)                    │
│  └─> 10 Prometheus alerts (3 critical + 5 warning + 2 perf) │
│      3 notification channels (Slack, Email, PagerDuty)       │
│                                                              │
│  Phase 6: Post-Deployment Validation (60m)                  │
│  └─> Service health baseline (T+5)                          │
│      Wave D feature validation (T+10)                        │
│      Trading functionality (T+20)                            │
│      Performance baseline (T+30)                             │
│      Monitoring validation (T+45)                            │
│      Initial assessment (T+60)                               │
│                                                              │
│  Phase 7: 24-Hour Observation                               │
│  └─> Intensive monitoring (T+0 to T+6h, every 30m)          │
│      Regular monitoring (T+6 to T+24h, every 2h)            │
│                                                              │
│  Phase 8: Production Certification (30m)                    │
│  └─> 25-item checklist (100% required)                      │
│      Stakeholder sign-off                                    │
│      GO/NO-GO decision                                       │
│                                                              │
└─────────────────────────────────────────────────────────────┘

---

Phase 1: Pre-Deployment Checklist (30 minutes)

A. Infrastructure Health Verification (10 minutes)

Docker Services Check:

# 1. Verify all services up
docker-compose ps
# Expected: All services "Up (healthy)"
#   - PostgreSQL (5432)
#   - Redis (6379)
#   - Vault (8200, unsealed)
#   - Grafana (3000)
#   - Prometheus (9090)
#   - InfluxDB (8086)

# 2. Database connectivity
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT version();"
# Expected: PostgreSQL 15.x with TimescaleDB

# 3. Database backup (CRITICAL - do not skip)
pg_dump -h localhost -U foxhunt -d foxhunt > /backup/foxhunt_pre_wave_d_$(date +%Y%m%d_%H%M%S).sql
# Verify backup size > 0 bytes

# 4. Vault secrets validation
vault kv get secret/foxhunt/database
vault kv get secret/foxhunt/jwt
vault kv get secret/foxhunt/redis
vault kv get secret/foxhunt/databento
# Expected: All secrets accessible

B. Service Readiness Check (10 minutes)

Build All Services:

# 1. Clean build (release mode)
cargo build --workspace --release
# Expected: 0 errors, 0 warnings

# 2. Verify binary artifacts
ls -lh target/release/{api_gateway,trading_service,backtesting_service,ml_training_service,trading_agent_service}
# Expected: All binaries present with recent timestamps

# 3. Validate configuration
cat config/production.toml
# Verify: Correct endpoints, ports, TLS settings

# 4. Test suite validation
cargo test --workspace --release
# Expected: 2,062/2,074 passing (99.4%)

C. Monitoring & Alerting Setup (10 minutes)

Grafana Dashboard Import:

# 1. Import Wave D dashboards
curl -X POST http://admin:foxhunt123@localhost:3000/api/dashboards/db \
  -H "Content-Type: application/json" \
  -d @grafana/wave_d_regime_detection.json

curl -X POST http://admin:foxhunt123@localhost:3000/api/dashboards/db \
  -H "Content-Type: application/json" \
  -d @grafana/wave_d_adaptive_strategies.json

# 2. Configure Prometheus alerts
cp prometheus/wave_d_alerts.yml /etc/prometheus/alerts/
curl -X POST http://localhost:9090/-/reload

# 3. Verify alerts loaded
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].name'
# Expected: ["wave_d_regime_detection", "wave_d_performance"]

D. Network & Port Validation (5 minutes)

Verify Ports Available:

# Check all service ports available
lsof -i :50051  # API Gateway (should be empty)
lsof -i :50052  # Trading Service (should be empty)
lsof -i :50053  # Backtesting Service (should be empty)
lsof -i :50054  # ML Training Service (should be empty)
lsof -i :50055  # Trading Agent Service (should be empty)

# Verify grpc_health_probe available
which grpc_health_probe

Pre-Deployment GO/NO-GO Decision

All criteria MUST pass before proceeding:

• All 6 Docker services healthy
• Database backup completed (size > 0)
• Vault secrets accessible
• All services compiled (release mode)
• Test suite >= 99% pass rate
• All service ports available
• Grafana dashboards imported
• Prometheus alerts loaded

If ANY item fails: STOP and remediate before continuing

---

Phase 2: Database Migration (15 minutes)

A. Pre-Migration Validation (5 minutes)

# 1. Verify current migration state
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT version FROM _sqlx_migrations ORDER BY version DESC LIMIT 1;"
# Expected: Last migration < 045

# 2. Verify backup exists
ls -lh /backup/foxhunt_pre_wave_d_*.sql
# Expected: File size > 100MB, recent timestamp

# 3. Test migration on staging (DRY RUN)
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt_staging \
  < migrations/045_regime_detection.sql
# Expected: 0 errors

# 4. Verify staging tables created
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt_staging \
  -c "\dt regime_*"
# Expected: 3 tables (regime_states, regime_transitions, adaptive_strategy_metrics)

B. Production Migration Execution (5 minutes)

# 1. Stop all services (if deploying during market hours)
# NOTE: Skip if deploying after market close
killall -TERM api_gateway trading_service ml_training_service trading_agent_service backtesting_service

# 2. Apply migration
cd /home/jgrusewski/Work/foxhunt
cargo sqlx migrate run
# Expected output: "Applied 045/migrate regime detection (0.234s)"

# 3. Verify migration applied
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT version FROM _sqlx_migrations WHERE version = 45;"
# Expected: 1 row returned

# 4. Verify table structure
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "\d regime_states"
# Expected: Columns include id, symbol, regime_type, confidence, timestamp

C. Post-Migration Validation (5 minutes)

# 1. Verify all 3 tables exist
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT table_name FROM information_schema.tables WHERE table_name LIKE 'regime%';"
# Expected: 3 rows

# 2. Verify indexes created
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT indexname FROM pg_indexes WHERE tablename LIKE 'regime%';"
# Expected: Multiple indexes

# 3. Test INSERT/SELECT permissions
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "INSERT INTO regime_states (symbol, regime_type, confidence, timestamp)
      VALUES ('TEST.FUT', 'trending', 0.95, NOW());"
# Expected: INSERT 0 1

psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT COUNT(*) FROM regime_states;"
# Expected: 1 (test row)

# 4. Cleanup test data
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "DELETE FROM regime_states WHERE symbol = 'TEST.FUT';"

Migration Success Criteria

• Migration version 45 recorded in _sqlx_migrations
• 3 tables created: regime_states, regime_transitions, adaptive_strategy_metrics
• All indexes created successfully
• INSERT/SELECT permissions validated
• Zero errors in migration output
• Staging migration tested successfully

Rollback Procedure (if migration fails)

# 1. Stop all services immediately
killall -TERM api_gateway trading_service ml_training_service trading_agent_service backtesting_service

# 2. Drop Wave D tables (safe because additive migration)
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt <<EOF
DROP TABLE IF EXISTS adaptive_strategy_metrics CASCADE;
DROP TABLE IF EXISTS regime_transitions CASCADE;
DROP TABLE IF EXISTS regime_states CASCADE;
DELETE FROM _sqlx_migrations WHERE version = 45;
EOF

# 3. Verify rollback
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "\dt regime_*"
# Expected: No rows (tables dropped)

# 4. Restart services with pre-Wave D binaries
git checkout e1834ac4  # Last commit before Wave D
cargo build --workspace --release
# Deploy pre-Wave D services (see Phase 3)

---

Phase 3: Service Deployment Sequence (60 minutes)

Deployment Order: API Gateway -> Trading Service -> ML Training -> Trading Agent -> Backtesting

Service 1: API Gateway (15 minutes)

A. Configuration Update (5 minutes):

# 1. Verify production configuration
cat config/production.toml | grep -A 5 "\[api_gateway\]"
# Expected: host = "0.0.0.0", port = 50051, tls_enabled = true

# 2. Update Vault secrets (if needed)
vault kv put secret/foxhunt/api_gateway \
  jwt_secret="$(openssl rand -base64 32)" \
  jwt_expiry_minutes=60 \
  rate_limit_requests_per_minute=1000

# 3. Set environment variables
export FOXHUNT_ENV=production
export FOXHUNT_LOG_LEVEL=info
export RUST_BACKTRACE=1

B. Service Deployment (5 minutes):

# 1. Start API Gateway
cd /home/jgrusewski/Work/foxhunt
nohup target/release/api_gateway > /var/log/foxhunt/api_gateway.log 2>&1 &
echo $! > /var/run/foxhunt/api_gateway.pid

# 2. Wait for startup (max 30 seconds)
for i in {1..30}; do
  grpc_health_probe -addr=localhost:50051 && break
  echo "Waiting for API Gateway... ($i/30)"
  sleep 1
done

# 3. Verify process running
ps aux | grep api_gateway | grep -v grep

C. Health Validation (5 minutes):

# 1. gRPC health check
grpc_health_probe -addr=localhost:50051
# Expected: status: SERVING

# 2. HTTP health endpoint
curl -f http://localhost:8080/health
# Expected: {"status":"healthy","service":"api_gateway"}

# 3. Prometheus metrics
curl -f http://localhost:9091/metrics | grep api_gateway_up
# Expected: api_gateway_up 1

# 4. Test authentication
curl -X POST http://localhost:50051/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"test123"}'
# Expected: {"token":"eyJ...","expires_in":3600}

# 5. Verify logs clean
tail -n 50 /var/log/foxhunt/api_gateway.log | grep -i error
# Expected: No error lines

Rollback: If health check fails, execute Service 1 Rollback Procedure

---

Service 2: Trading Service (10 minutes)

A. Configuration & Deployment (5 minutes):

# 1. Verify config
cat config/production.toml | grep -A 5 "\[trading_service\]"

# 2. Start Trading Service
nohup target/release/trading_service > /var/log/foxhunt/trading_service.log 2>&1 &
echo $! > /var/run/foxhunt/trading_service.pid

# 3. Wait for startup
for i in {1..30}; do
  grpc_health_probe -addr=localhost:50052 && break
  echo "Waiting for Trading Service... ($i/30)"
  sleep 1
done

B. Health Validation (5 minutes):

# 1. gRPC health check
grpc_health_probe -addr=localhost:50052
# Expected: status: SERVING

# 2. HTTP health endpoint
curl -f http://localhost:8081/health

# 3. Verify database connectivity
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT COUNT(*) FROM positions;"
# Expected: >= 0 rows

# 4. Test order submission (via API Gateway)
JWT_TOKEN=$(curl -s -X POST http://localhost:50051/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"test123"}' | jq -r '.token')

curl -X POST http://localhost:50051/api/v1/trading/order \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"symbol":"ES.FUT","side":"BUY","quantity":1,"order_type":"LIMIT","price":5000}'
# Expected: {"order_id":"...","status":"PENDING"}

# 5. Verify logs
tail -n 50 /var/log/foxhunt/trading_service.log | grep -i error
# Expected: No error lines

Rollback: If health check fails, execute Service 2 Rollback Procedure

---

Service 3: ML Training Service (10 minutes)

A. Configuration & Deployment (5 minutes):

# 1. Verify config
cat config/production.toml | grep -A 5 "\[ml_training_service\]"

# 2. Verify CUDA availability
nvidia-smi
# Expected: GPU detected, driver loaded

# 3. Start ML Training Service
nohup target/release/ml_training_service > /var/log/foxhunt/ml_training_service.log 2>&1 &
echo $! > /var/run/foxhunt/ml_training_service.pid

# 4. Wait for startup
for i in {1..30}; do
  grpc_health_probe -addr=localhost:50054 && break
  echo "Waiting for ML Training Service... ($i/30)"
  sleep 1
done

B. Health Validation (5 minutes):

# 1. gRPC health check
grpc_health_probe -addr=localhost:50054

# 2. HTTP health endpoint
curl -f http://localhost:8095/health

# 3. Test regime detection (via API Gateway)
curl -X GET http://localhost:50051/api/v1/ml/regime?symbol=ES.FUT \
  -H "Authorization: Bearer $JWT_TOKEN"
# Expected: {"regime":"trending","confidence":0.85,"timestamp":"..."}

# 4. Verify GPU memory usage
nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits
# Expected: < 500 MB (well within 4GB budget)

# 5. Verify logs
tail -n 50 /var/log/foxhunt/ml_training_service.log | grep -i error

Rollback: If health check fails, execute Service 3 Rollback Procedure

---

Service 4: Trading Agent Service (15 minutes)

A. Configuration & Deployment (7 minutes):

# 1. Verify config
cat config/production.toml | grep -A 5 "\[trading_agent_service\]"

# 2. Verify dependencies (Trading + ML Training must be healthy)
grpc_health_probe -addr=localhost:50052  # Trading Service
grpc_health_probe -addr=localhost:50054  # ML Training Service
# Both must return SERVING

# 3. Start Trading Agent Service
nohup target/release/trading_agent_service > /var/log/foxhunt/trading_agent_service.log 2>&1 &
echo $! > /var/run/foxhunt/trading_agent_service.pid

# 4. Wait for startup (longer due to model loading)
for i in {1..60}; do
  grpc_health_probe -addr=localhost:50055 && break
  echo "Waiting for Trading Agent Service... ($i/60)"
  sleep 1
done

B. Health Validation (8 minutes):

# 1. gRPC health check
grpc_health_probe -addr=localhost:50055

# 2. HTTP health endpoint
curl -f http://localhost:8082/health

# 3. Test universe selection
curl -X GET http://localhost:50051/api/v1/trading-agent/universe \
  -H "Authorization: Bearer $JWT_TOKEN"
# Expected: {"symbols":["ES.FUT","NQ.FUT","6E.FUT","ZN.FUT"]}

# 4. Test asset selection
curl -X POST http://localhost:50051/api/v1/trading-agent/assets \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"symbols":["ES.FUT","NQ.FUT"]}'
# Expected: {"selected":["ES.FUT"],"scores":[0.85,0.72]}

# 5. Test portfolio allocation (CRITICAL - regime-adaptive sizing)
curl -X POST http://localhost:50051/api/v1/trading-agent/allocate \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"symbols":["ES.FUT"],"total_capital":100000}'
# Expected: {"allocations":{"ES.FUT":25000},"regime":"trending"}

# 6. Verify regime-adaptive position sizing active
grep -i "kelly_criterion_regime_adaptive" /var/log/foxhunt/trading_agent_service.log
# Expected: Function called, no errors

# 7. Verify logs
tail -n 50 /var/log/foxhunt/trading_agent_service.log | grep -i error

Rollback: If health check fails, execute Service 4 Rollback Procedure

---

Service 5: Backtesting Service (10 minutes)

A. Configuration & Deployment (5 minutes):

# 1. Verify config
cat config/production.toml | grep -A 5 "\[backtesting_service\]"

# 2. Start Backtesting Service
nohup target/release/backtesting_service > /var/log/foxhunt/backtesting_service.log 2>&1 &
echo $! > /var/run/foxhunt/backtesting_service.pid

# 3. Wait for startup
for i in {1..30}; do
  grpc_health_probe -addr=localhost:50053 && break
  echo "Waiting for Backtesting Service... ($i/30)"
  sleep 1
done

B. Health Validation (5 minutes):

# 1. gRPC health check
grpc_health_probe -addr=localhost:50053

# 2. HTTP health endpoint
curl -f http://localhost:8082/health

# 3. Test backtest execution
curl -X POST http://localhost:50051/api/v1/backtest/run \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "symbol":"ES.FUT",
    "start_date":"2024-01-01",
    "end_date":"2024-01-02",
    "strategy":"wave_d"
  }'
# Expected: {"backtest_id":"...","status":"RUNNING"}

# 4. Verify DBN data loading
tail -n 100 /var/log/foxhunt/backtesting_service.log | grep "DBN data loaded"
# Expected: Load time < 10ms

# 5. Verify logs
tail -n 50 /var/log/foxhunt/backtesting_service.log | grep -i error

Rollback: If health check fails, execute Service 5 Rollback Procedure

---

Deployment Success Criteria

All criteria MUST pass before proceeding to Phase 4:

• All 5 services return SERVING on gRPC health checks
• All HTTP /health endpoints return 200 OK
• All Prometheus /metrics endpoints accessible
• No errors in service logs
• Inter-service communication verified (Trading Agent -> Trading Service)
• Database connectivity verified (all services can query regime tables)
• Regime detection operational (ML Training Service)
• Adaptive position sizing operational (Trading Agent Service)

---

Phase 4: Smoke Test Procedures (20 minutes)

5 Critical Tests - All MUST pass before declaring deployment successful

Test 1: Authentication & Authorization (5 minutes)

# 1. Test JWT login
JWT_TOKEN=$(curl -s -X POST http://localhost:50051/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"test123"}' | jq -r '.token')

echo "JWT Token: $JWT_TOKEN"
# Expected: Non-empty token starting with "eyJ"

# 2. Test token validation
curl -X GET http://localhost:50051/api/v1/auth/validate \
  -H "Authorization: Bearer $JWT_TOKEN"
# Expected: {"valid":true,"user":"admin","expires_in":3600}

# 3. Test unauthorized access (should fail)
curl -X GET http://localhost:50051/api/v1/trading/positions
# Expected: HTTP 401 Unauthorized

# 4. Test authorized access (should succeed)
curl -X GET http://localhost:50051/api/v1/trading/positions \
  -H "Authorization: Bearer $JWT_TOKEN"
# Expected: HTTP 200, {"positions":[...]}

# 5. Test rate limiting
for i in {1..100}; do
  curl -s http://localhost:50051/api/v1/auth/validate \
    -H "Authorization: Bearer $JWT_TOKEN" > /dev/null
done
# Expected: Some requests return HTTP 429 (rate limit exceeded)

Success Criteria:

• JWT token generated successfully
• Token validation passes
• Unauthorized requests blocked
• Authorized requests succeed
• Rate limiting active

---

Test 2: Regime Detection & Adaptive Strategy (5 minutes)

# 1. Get current regime for ES.FUT
REGIME=$(curl -s -X GET http://localhost:50051/api/v1/ml/regime?symbol=ES.FUT \
  -H "Authorization: Bearer $JWT_TOKEN" | jq -r '.regime')
echo "Current Regime: $REGIME"
# Expected: One of: trending, ranging, volatile

# 2. Verify regime confidence
CONFIDENCE=$(curl -s -X GET http://localhost:50051/api/v1/ml/regime?symbol=ES.FUT \
  -H "Authorization: Bearer $JWT_TOKEN" | jq -r '.confidence')
echo "Regime Confidence: $CONFIDENCE"
# Expected: 0.0 < confidence < 1.0

# 3. Test regime transitions
curl -s -X GET http://localhost:50051/api/v1/ml/regime/transitions?symbol=ES.FUT&limit=10 \
  -H "Authorization: Bearer $JWT_TOKEN" | jq '.'
# Expected: Array of recent regime transitions

# 4. Test adaptive position sizing
ALLOCATION=$(curl -s -X POST http://localhost:50051/api/v1/trading-agent/allocate \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"symbols":["ES.FUT"],"total_capital":100000}' | jq -r '.allocations["ES.FUT"]')
echo "Position Size: $ALLOCATION"
# Expected: 10000 < allocation < 50000 (regime-adjusted)

# 5. Verify adaptive metrics
curl -s -X GET http://localhost:50051/api/v1/ml/adaptive-metrics?symbol=ES.FUT \
  -H "Authorization: Bearer $JWT_TOKEN" | jq '.'
# Expected: {"position_multiplier":0.2-1.5,"stop_multiplier":1.5-4.0}

Success Criteria:

• Regime detection returns valid regime type
• Confidence score between 0-1
• Regime transitions queryable
• Position sizing adaptive to regime
• Adaptive metrics present

---

Test 3: Order Submission & Execution (3 minutes)

# 1. Submit market order
ORDER_ID=$(curl -s -X POST http://localhost:50051/api/v1/trading/order \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "symbol":"ES.FUT",
    "side":"BUY",
    "quantity":1,
    "order_type":"MARKET"
  }' | jq -r '.order_id')
echo "Order ID: $ORDER_ID"

# 2. Check order status
curl -s -X GET "http://localhost:50051/api/v1/trading/order/$ORDER_ID" \
  -H "Authorization: Bearer $JWT_TOKEN" | jq '.'
# Expected: {"status":"FILLED" or "PENDING"}

# 3. Verify position created
curl -s -X GET http://localhost:50051/api/v1/trading/positions \
  -H "Authorization: Bearer $JWT_TOKEN" | jq '.[] | select(.symbol=="ES.FUT")'
# Expected: Position with quantity=1

# 4. Test dynamic stop-loss applied
STOP_PRICE=$(curl -s -X GET http://localhost:50051/api/v1/trading/positions \
  -H "Authorization: Bearer $JWT_TOKEN" | jq -r '.[] | select(.symbol=="ES.FUT") | .stop_loss')
echo "Stop Loss: $STOP_PRICE"
# Expected: Non-null stop price (ATR-based, 1.5x-4.0x multiplier)

Success Criteria:

• Order submitted successfully
• Order status queryable
• Position created after fill
• Dynamic stop-loss applied

---

Test 4: ML Predictions & Feature Extraction (4 minutes)

# 1. Trigger ML prediction
PREDICTION=$(curl -s -X POST http://localhost:50051/api/v1/ml/predict \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"symbol":"ES.FUT","horizon":60}' | jq -r '.signal')
echo "ML Prediction: $PREDICTION"
# Expected: BUY, SELL, or HOLD

# 2. Verify 225 features extracted
FEATURE_COUNT=$(curl -s -X GET http://localhost:50051/api/v1/ml/features?symbol=ES.FUT \
  -H "Authorization: Bearer $JWT_TOKEN" | jq '.features | length')
echo "Feature Count: $FEATURE_COUNT"
# Expected: 225 (201 Wave C + 24 Wave D)

# 3. Verify Wave D features present (indices 201-224)
curl -s -X GET http://localhost:50051/api/v1/ml/features?symbol=ES.FUT&indices=201-224 \
  -H "Authorization: Bearer $JWT_TOKEN" | jq '.features | keys'
# Expected: Array with 24 feature names (cusum_*, adx_*, transition_prob_*, adaptive_*)

# 4. Test model ensemble prediction
curl -s -X POST http://localhost:50051/api/v1/ml/predict/ensemble \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"symbol":"ES.FUT","models":["mamba2","dqn","ppo","tft"]}' | jq '.'
# Expected: {"consensus":"BUY","confidence":0.75}

# 5. Verify GPU inference latency
INFERENCE_TIME=$(curl -s http://localhost:9094/metrics | grep ml_inference_duration_seconds | awk '{print $2}')
echo "Inference Latency: ${INFERENCE_TIME}s"
# Expected: < 0.005 (5ms)

Success Criteria:

• ML prediction returns valid signal
• 225 features extracted
• Wave D features (201-224) present
• Ensemble prediction functional
• Inference latency < 5ms

---

Test 5: Database Persistence & Monitoring (3 minutes)

# 1. Verify regime_states table has data
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT COUNT(*) FROM regime_states WHERE symbol = 'ES.FUT';"
# Expected: >= 1 rows

# 2. Verify adaptive_strategy_metrics table has data
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT COUNT(*) FROM adaptive_strategy_metrics WHERE symbol = 'ES.FUT';"
# Expected: >= 1 rows

# 3. Verify Prometheus scraping all services
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets | map(select(.health=="up")) | length'
# Expected: >= 5 (all services up)

# 4. Verify Grafana dashboards accessible
curl -s -u admin:foxhunt123 http://localhost:3000/api/dashboards/db/wave-d-regime-detection | jq '.dashboard.title'
# Expected: "Wave D - Regime Detection"

# 5. Verify Prometheus alerts loaded
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.name | contains("regime")) | .name'
# Expected: 8 alerts (3 critical, 5 warning)

Success Criteria:

• Regime data persisted to database
• All 3 Wave D tables populated
• Prometheus scraping all services
• Grafana dashboards accessible
• Prometheus alerts loaded

---

Smoke Test Summary

echo "=== SMOKE TEST RESULTS ==="
echo "Test 1: Authentication         [PASS/FAIL]"
echo "Test 2: Regime Detection       [PASS/FAIL]"
echo "Test 3: Order Execution        [PASS/FAIL]"
echo "Test 4: ML Predictions         [PASS/FAIL]"
echo "Test 5: Database & Monitoring  [PASS/FAIL]"
echo "==========================="
echo "Overall Status: [5/5 PASS = PRODUCTION READY]"

Failure Response:

• If ANY test fails: STOP, investigate (15 min limit), rollback if necessary
• If 2+ tests fail: IMMEDIATE ROLLBACK (do not proceed)
• If 1 test fails: Investigate, fix or rollback within 15 minutes

---

Phase 5: Monitoring Alerts Configuration (15 minutes)

Prometheus Alert Rules

File: /etc/prometheus/alerts/wave_d_alerts.yml

Alert Configuration:

• 3 Critical Alerts (page immediately, 5-minute response time)
• 5 Warning Alerts (investigate within 1 hour)
• 2 Performance Alerts (24-hour monitoring)

Critical Alerts:

1. RegimeFlipFlopping: >10 transitions/5min (indicates unstable regime detection)
2. RegimeFalsePositives: >30% false positive rate (degrades adaptive performance)
3. RegimeNaNInfValues: NaN/Inf in features (causes ML model failures)

Warning Alerts: 4. RegimeDetectionLatencyHigh: P99 >50μs (delays adaptive adjustments) 5. RegimeCoverageLow: <80% high-confidence regimes (underperformance risk) 6. AdaptivePositionSizerOutOfRange: Multiplier <0.2 or >1.5 (extreme conditions) 7. DynamicStopLossOutOfRange: Multiplier <1.5 or >4.0 (stops too tight/wide) 8. RegimeTransitionProbabilityAnomaly: >30% change in 1h (market regime shift)

Performance Alerts: 9. RegimeAdaptivePerformanceDegraded: Sharpe <1.5 for 2h (adaptive strategies not working) 10. WaveDVsWaveCPerformanceRegression: Wave D < Wave C baseline for 24h (regression)

Alert Deployment

# 1. Copy alert rules
sudo cp prometheus/wave_d_alerts.yml /etc/prometheus/alerts/

# 2. Validate syntax
promtool check rules /etc/prometheus/alerts/wave_d_alerts.yml
# Expected: SUCCESS - 10 rules loaded

# 3. Reload Prometheus
curl -X POST http://localhost:9090/-/reload

# 4. Verify alerts loaded
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].name'
# Expected: ["wave_d_regime_detection", "wave_d_performance"]

# 5. Check alert count
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].rules | length'
# Expected: [8, 2]

Notification Channels

# 1. Configure Slack
curl -X POST http://admin:foxhunt123@localhost:3000/api/alert-notifications \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Slack - Foxhunt Alerts",
    "type": "slack",
    "isDefault": true,
    "settings": {
      "url": "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK",
      "recipient": "#foxhunt-alerts"
    }
  }'

# 2. Configure Email
curl -X POST http://admin:foxhunt123@localhost:3000/api/alert-notifications \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Email - Operations Team",
    "type": "email",
    "settings": {
      "addresses": "ops@foxhunt.ai"
    }
  }'

# 3. Configure PagerDuty (critical only)
curl -X POST http://admin:foxhunt123@localhost:3000/api/alert-notifications \
  -H "Content-Type: application/json" \
  -d '{
    "name": "PagerDuty - Critical",
    "type": "pagerduty",
    "settings": {
      "integrationKey": "YOUR_PAGERDUTY_KEY",
      "severity": "critical"
    }
  }'

# 4. Verify channels created
curl -s -u admin:foxhunt123 http://localhost:3000/api/alert-notifications | jq '.[] | {name: .name, type: .type}'
# Expected: 3 channels (Slack, Email, PagerDuty)

Alert Testing

# 1. Test Slack notification
curl -X POST http://localhost:9090/api/v1/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "component": "deployment_test"
    },
    "annotations": {
      "summary": "Wave D deployment test alert"
    }
  }]'
# Expected: Alert in Slack within 30 seconds

# 2. Silence test alerts
curl -X POST http://localhost:9090/api/v1/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      {"name": "component", "value": "deployment_test", "isRegex": false}
    ],
    "startsAt": "2025-10-19T00:00:00Z",
    "endsAt": "2025-10-19T23:59:59Z",
    "comment": "Silencing deployment test alerts"
  }'

Alert Escalation Matrix

Severity	Channels	Response Time	Escalation
Critical	Slack + PagerDuty	5 minutes	On-call -> Lead -> CTO
Warning	Slack + Email	1 hour	Team channel -> On-call
Info	Grafana dashboard	24 hours	Daily review

Monitoring Success Criteria

• 10 Prometheus alerts loaded
• 3 notification channels configured
• Test alerts delivered successfully
• Alert history queryable
• Grafana dashboards show alert status

---

Phase 6: Post-Deployment Validation (60 minutes)

T+5 Minutes: Service Health Baseline

# 1. Verify all services healthy
for service in api_gateway trading_service ml_training_service trading_agent_service backtesting_service; do
  echo "=== $service ==="
  grpc_health_probe -addr=localhost:$PORT
  curl -f http://localhost:$HTTP_PORT/health
  ps aux | grep $service | grep -v grep
done
# Expected: All SERVING, HTTP 200, processes running

# 2. Check logs for errors
for log in /var/log/foxhunt/*.log; do
  tail -n 100 $log | grep -i "error\|fatal\|panic"
done
# Expected: Zero critical errors

# 3. Verify database connections
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT COUNT(*) FROM pg_stat_activity WHERE datname = 'foxhunt';"
# Expected: >= 5 connections

# 4. Check Redis connectivity
redis-cli PING
redis-cli INFO clients
# Expected: PONG, >= 5 clients

T+10 Minutes: Wave D Feature Validation

# 1. Verify regime detection operational
for symbol in ES.FUT NQ.FUT 6E.FUT ZN.FUT; do
  echo "=== $symbol ==="
  curl -s -X GET "http://localhost:50051/api/v1/ml/regime?symbol=$symbol" \
    -H "Authorization: Bearer $JWT_TOKEN" | jq '.'
done
# Expected: Valid regime, confidence >0.7 for all symbols

# 2. Verify 225 features extracted
curl -s -X GET "http://localhost:50051/api/v1/ml/features?symbol=ES.FUT" \
  -H "Authorization: Bearer $JWT_TOKEN" | jq '.features | length'
# Expected: 225

# 3. Check Wave D features (201-224)
curl -s -X GET "http://localhost:50051/api/v1/ml/features?symbol=ES.FUT&indices=201-224" \
  -H "Authorization: Bearer $JWT_TOKEN" | jq '.features | keys | length'
# Expected: 24

# 4. Verify database persistence
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -c "SELECT COUNT(*), symbol FROM regime_states GROUP BY symbol;"
# Expected: >= 1 row per symbol

T+20 Minutes: Trading Functionality Validation

# 1. Submit test order
ORDER_ID=$(curl -s -X POST http://localhost:50051/api/v1/trading/order \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"symbol":"ES.FUT","side":"BUY","quantity":1,"order_type":"MARKET"}' | jq -r '.order_id')

# 2. Wait for fill
for i in {1..30}; do
  STATUS=$(curl -s -X GET "http://localhost:50051/api/v1/trading/order/$ORDER_ID" \
    -H "Authorization: Bearer $JWT_TOKEN" | jq -r '.status')
  [[ "$STATUS" == "FILLED" ]] && break
  sleep 1
done

# 3. Verify position with adaptive sizing
curl -s -X GET http://localhost:50051/api/v1/trading/positions \
  -H "Authorization: Bearer $JWT_TOKEN" | jq '.[] | select(.symbol=="ES.FUT")'
# Expected: Position with dynamic stop-loss

# 4. Verify stop-loss is ATR-based
STOP_LOSS=$(curl -s -X GET http://localhost:50051/api/v1/trading/positions \
  -H "Authorization: Bearer $JWT_TOKEN" | jq -r '.[] | select(.symbol=="ES.FUT") | .stop_loss')
echo "Stop Loss: $STOP_LOSS"
# Expected: Non-null, within 1.5x-4.0x ATR range

T+30 Minutes: Performance Metrics Baseline

# 1. Capture metrics snapshot
curl -s http://localhost:9091/metrics > /tmp/metrics_t30.txt

# 2. Check key metrics
echo "Regime Detection P99:"
curl -s http://localhost:9094/metrics | grep regime_detection_duration_seconds | grep 0.99
# Expected: < 50μs (0.00005s)

echo "ML Inference P99:"
curl -s http://localhost:9094/metrics | grep ml_inference_duration_seconds | grep 0.99
# Expected: < 5ms (0.005s)

echo "Order Submission P99:"
curl -s http://localhost:9092/metrics | grep order_submission_duration_seconds | grep 0.99
# Expected: < 100ms (0.1s)

# 3. Check error rates
for service in api_gateway trading_service ml_training_service trading_agent_service; do
  ERROR_RATE=$(curl -s http://localhost:909{1,2,4,2}/metrics | grep "${service}_errors_total" | awk '{sum+=$2} END {print sum}')
  echo "$service errors: $ERROR_RATE"
done
# Expected: All = 0 or very low (<5)

T+45 Minutes: Monitoring Stack Validation

# 1. Verify Prometheus scraping
ACTIVE_TARGETS=$(curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets | map(select(.health=="up")) | length')
echo "Active targets: $ACTIVE_TARGETS"
# Expected: >= 5

# 2. Check firing alerts
FIRING_ALERTS=$(curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts | map(select(.state=="firing")) | length')
echo "Firing alerts: $FIRING_ALERTS"
# Expected: 0

# 3. Verify Grafana dashboards
curl -s -u admin:foxhunt123 http://localhost:3000/api/dashboards/db/wave-d-regime-detection | jq '.dashboard.title'
# Expected: "Wave D - Regime Detection"

# 4. Check InfluxDB ingestion
curl -s http://localhost:8086/query?db=foxhunt&q=SELECT%20COUNT%28*%29%20FROM%20regime_states%20WHERE%20time%20%3E%20now%28%29%20-%201h
# Expected: >= 1 data point

T+60 Minutes: Initial Performance Assessment

# 1. Calculate regime transition rate
REGIME_TRANSITIONS=$(psql -t -c "SELECT COUNT(*) FROM regime_transitions WHERE timestamp > NOW() - INTERVAL '1 hour';")
echo "Transitions (1h): $REGIME_TRANSITIONS"
# Expected: 0-10 (normal), >50 indicates flip-flopping

# 2. Check adaptive position sizing
psql -c "SELECT symbol, AVG(position_multiplier), MIN(position_multiplier), MAX(position_multiplier)
         FROM adaptive_strategy_metrics WHERE timestamp > NOW() - INTERVAL '1 hour' GROUP BY symbol;"
# Expected: avg between 0.2-1.5

# 3. Check stop-loss multipliers
psql -c "SELECT symbol, AVG(stop_loss_multiplier), MIN(stop_loss_multiplier), MAX(stop_loss_multiplier)
         FROM adaptive_strategy_metrics WHERE timestamp > NOW() - INTERVAL '1 hour' GROUP BY symbol;"
# Expected: avg between 1.5-4.0

# 4. Generate deployment report
cat > /tmp/wave_d_deployment_report_t60.txt <<EOF
=== Wave D Deployment - T+60 Report ===
Timestamp: $(date -Iseconds)

SERVICE HEALTH: [All healthy/Issues found]
REGIME DETECTION: 4 symbols monitored, $REGIME_TRANSITIONS transitions
TRADING ACTIVITY: [Orders/Positions counts]
PERFORMANCE: [Latencies within targets]
ALERTS: $FIRING_ALERTS firing
STATUS: [SUCCESSFUL/REQUIRES ATTENTION]

Next review: T+120 minutes
EOF

cat /tmp/wave_d_deployment_report_t60.txt

Post-Deployment Validation Success Criteria

All criteria MUST pass:

• All 5 services healthy for 60 minutes
• Zero critical errors in logs
• Regime detection operational (4 symbols)
• 225 features extracted successfully
• Trading functionality operational
• Adaptive position sizing active (0.2x-1.5x)
• Dynamic stop-loss active (1.5x-4.0x)
• Performance targets met
• Monitoring stack operational
• Zero firing alerts

If ANY fails: Investigate (15 min), rollback if unresolvable

---

Phase 7: 24-Hour Observation Period

Hour 2-6: Intensive Monitoring (Every 30 minutes)

Automated Health Check Script: /opt/foxhunt/scripts/wave_d_health_check.sh

#!/bin/bash
# Wave D Health Check - Run every 30 minutes

TIMESTAMP=$(date -Iseconds)
REPORT="/var/log/foxhunt/health_checks/wave_d_health_${TIMESTAMP}.log"

mkdir -p /var/log/foxhunt/health_checks
echo "=== Wave D Health Check - $TIMESTAMP ===" | tee $REPORT

# 1. Service Health
for port in 50051 50052 50053 50054 50055; do
  if grpc_health_probe -addr=localhost:$port &>/dev/null; then
    echo "Port $port: HEALTHY" | tee -a $REPORT
  else
    echo "Port $port: UNHEALTHY - ALERT!" | tee -a $REPORT
  fi
done

# 2. Regime Detection
JWT_TOKEN=$(curl -s -X POST http://localhost:50051/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"test123"}' | jq -r '.token')

for symbol in ES.FUT NQ.FUT 6E.FUT ZN.FUT; do
  REGIME=$(curl -s -X GET "http://localhost:50051/api/v1/ml/regime?symbol=$symbol" \
    -H "Authorization: Bearer $JWT_TOKEN" | jq -r '.regime')
  CONFIDENCE=$(curl -s -X GET "http://localhost:50051/api/v1/ml/regime?symbol=$symbol" \
    -H "Authorization: Bearer $JWT_TOKEN" | jq -r '.confidence')
  echo "$symbol: $REGIME (confidence: $CONFIDENCE)" | tee -a $REPORT
done

# 3. Regime Transition Rate (flip-flopping check)
TRANSITIONS_5MIN=$(psql -t -c "SELECT COUNT(*) FROM regime_transitions WHERE timestamp > NOW() - INTERVAL '5 minutes';")
echo "Transitions (5 min): $TRANSITIONS_5MIN" | tee -a $REPORT
[ "$TRANSITIONS_5MIN" -gt 10 ] && echo "CRITICAL: Flip-flopping!" | tee -a $REPORT

# 4. Performance Metrics
REGIME_LATENCY=$(curl -s http://localhost:9094/metrics | grep regime_detection_duration_seconds | grep "0.99" | awk '{print $2}')
ML_LATENCY=$(curl -s http://localhost:9094/metrics | grep ml_inference_duration_seconds | grep "0.99" | awk '{print $2}')
echo "Regime P99: ${REGIME_LATENCY}s (target <50μs)" | tee -a $REPORT
echo "ML P99: ${ML_LATENCY}s (target <5ms)" | tee -a $REPORT

# 5. Firing Alerts
FIRING=$(curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts | map(select(.state=="firing")) | length')
echo "Firing alerts: $FIRING" | tee -a $REPORT

# 6. Summary
[ "$FIRING" -eq 0 ] && [ "$TRANSITIONS_5MIN" -lt 10 ] && echo "STATUS: HEALTHY" | tee -a $REPORT || echo "STATUS: REQUIRES ATTENTION" | tee -a $REPORT

Schedule:

# Add to crontab
*/30 * * * * /opt/foxhunt/scripts/wave_d_health_check.sh

Hour 7-24: Regular Monitoring (Every 2 hours)

Daily Check Script: /opt/foxhunt/scripts/wave_d_daily_check.sh

#!/bin/bash
# Wave D Daily Check - Run every 2 hours

TIMESTAMP=$(date -Iseconds)
REPORT="/var/log/foxhunt/daily_checks/wave_d_daily_${TIMESTAMP}.log"

mkdir -p /var/log/foxhunt/daily_checks
echo "=== Wave D Daily Check - $TIMESTAMP ===" | tee $REPORT

# 1. Quick health check
ALL_HEALTHY=true
for port in 50051 50052 50053 50054 50055; do
  grpc_health_probe -addr=localhost:$port &>/dev/null || ALL_HEALTHY=false
done
echo "All services: $([ "$ALL_HEALTHY" = true ] && echo HEALTHY || echo UNHEALTHY)" | tee -a $REPORT

# 2. Performance (2h window)
ORDERS=$(psql -t -c "SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '2 hours';")
POSITIONS=$(psql -t -c "SELECT COUNT(*) FROM positions WHERE opened_at > NOW() - INTERVAL '2 hours';")
TRANSITIONS=$(psql -t -c "SELECT COUNT(*) FROM regime_transitions WHERE timestamp > NOW() - INTERVAL '2 hours';")
echo "Orders: $ORDERS, Positions: $POSITIONS, Transitions: $TRANSITIONS" | tee -a $REPORT

# 3. Alerts
FIRING=$(curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts | map(select(.state=="firing")) | length')
echo "Firing alerts: $FIRING" | tee -a $REPORT

# 4. Status
echo "STATUS: $([ "$FIRING" -eq 0 ] && [ "$ALL_HEALTHY" = true ] && echo HEALTHY || echo INVESTIGATE)" | tee -a $REPORT

24-Hour Monitoring Checklist

T+0 to T+6 hours (Intensive):

• Health check every 30 minutes
• Monitor Grafana continuously
• All alerts (critical + warning) to Slack
• On-call engineer available
• Manual log review every 2 hours

T+6 to T+24 hours (Regular):

• Health check every 2 hours
• Grafana review every 4 hours
• Critical alerts only to Slack
• On-call engineer for escalation
• Manual log review every 6 hours

Key Metrics to Monitor

Metric	Target	Alert	Action
Service Uptime	100%	<99.5%	Investigate restarts
Regime Transitions	5-10/h	>50/h	Increase confidence
Regime Confidence	>0.7	<0.5	Review CUSUM params
Position Multiplier	0.2-1.5	Outside	Check regime
Stop Multiplier	1.5-4.0	Outside	Review volatility
Regime Latency	<50μs	>100μs	Profile performance
ML Latency	<5ms	>10ms	Check GPU
Order Latency	<100ms	>200ms	Check Trading
Error Rate	0	>5/h	Review logs
Firing Alerts	0	>0 critical	Immediate action

24-Hour Completion Criteria

All criteria MUST pass:

• All services >99.5% uptime
• Zero critical alerts fired
• Regime transitions 5-10/hour
• Performance targets met
• No unexplained restarts
• Database growth normal
• GPU memory stable (<500MB)
• All smoke tests pass

If met: Proceed to Phase 8 (Certification) If not met: Extend to 48 hours, investigate

---

Phase 8: Production Certification (30 minutes)

Production Certification Checklist

25 items - 100% required for approval

A. System Health & Stability (8 items)

• 1. All 5 services >99.5% uptime (24h observation)
• 2. Zero critical alerts fired
• 3. Zero unexplained restarts/crashes
• 4. All health checks passing
• 5. Database connections stable
• 6. Redis connectivity maintained
• 7. Vault connectivity maintained
• 8. All logs free of critical errors

B. Wave D Feature Validation (6 items)

• 9. Regime detection operational (4 symbols)
• 10. Regime confidence >0.7 consistently
• 11. Transitions 5-10/hour (no flip-flop)
• 12. 225 features extracted successfully
• 13. Adaptive sizing active (0.2x-1.5x)
• 14. Dynamic stops active (1.5x-4.0x ATR)

C. Performance & Latency (4 items)

• 15. Regime detection P99 <50μs
• 16. ML inference P99 <5ms
• 17. Order submission P99 <100ms
• 18. API Gateway P99 <1ms

D. Data Persistence & Monitoring (4 items)

• 19. Migration 045 applied successfully
• 20. All 3 Wave D tables populated
• 21. Prometheus scraping all services
• 22. Grafana dashboards live

E. Testing & Functionality (3 items)

• 23. All 5 smoke tests passing
• 24. Paper trading operational
• 25. TLI commands operational

Certification Report Template

#!/bin/bash
# Generate certification report

cat > /tmp/wave_d_certification_$(date +%Y%m%d).md <<'EOF'
# Wave D Production Certification Report

**Date**: [FILL: Date]
**Deployment Time**: [FILL: Start timestamp]
**Observation**: 24 hours
**Engineer**: [FILL: Name]

## Executive Summary

System status: **[READY/NOT READY]** for production with real capital.

**Key Metrics**:
- Uptime: [FILL]%
- Alerts: [FILL]
- Performance: [FILL]%
- Tests: [FILL]/5

## Certification Score

**Total**: [FILL]/25 ([FILL]%)
**Requirement**: 25/25 (100%)
**Status**: [PASS/FAIL]

## Decision

### GO (if 25/25):
CERTIFIED FOR PRODUCTION

Actions:
1. Enable real capital trading
2. Set risk limits
3. Enable 24/7 monitoring
4. Schedule daily reviews
5. Plan ML retraining (Week 2-6)

**Approved**: [FILL: Name, Date]

### NO-GO (if <25/25):
NOT CERTIFIED

Failed items: [FILL]
Required actions: [FILL]
Timeline: [FILL]

**Reviewed**: [FILL: Name, Date]

## Sign-Off

**Deployment Engineer**: [FILL] / [DATE]
**QA Engineer**: [FILL] / [DATE]
**Technical Lead**: [FILL] / [DATE]
**CTO Approval**: [FILL] / [DATE]

EOF

Post-Certification Actions

If CERTIFIED (GO):

1. Update CLAUDE.md status to "LIVE"
2. Enable real capital trading
3. Configure risk limits
4. Schedule daily performance reviews
5. Plan ML retraining (90-180 days data)
6. Monitor Wave D vs Wave C performance
7. Prepare Wave E planning (if +25-50% Sharpe achieved)

If NOT CERTIFIED (NO-GO):

1. Document all failures
2. Create remediation plan
3. Fix blocking issues
4. Re-run 24h observation
5. Re-execute certification
6. Consider rollback if >72h remediation

---

Rollback Procedures

Per-Service Rollback (5-8 minutes each)

General Process:

1. Stop failed service (kill process)
2. Revert to previous binary (git checkout e1834ac4)
3. Rebuild (cargo build -p --release)
4. Restart service
5. Verify health check passes
6. Monitor for 10 minutes

Service 1: API Gateway Rollback (5 min)

kill $(cat /var/run/foxhunt/api_gateway.pid)
rm /var/run/foxhunt/api_gateway.pid
lsof -i :50051  # Verify port released

cd /home/jgrusewski/Work/foxhunt
git stash
git checkout e1834ac4
cargo build -p api_gateway --release

nohup target/release/api_gateway > /var/log/foxhunt/api_gateway_rollback.log 2>&1 &
echo $! > /var/run/foxhunt/api_gateway.pid

grpc_health_probe -addr=localhost:50051
curl -f http://localhost:8080/health
# Test authentication

Service 2: Trading Service Rollback (5 min)

kill $(cat /var/run/foxhunt/trading_service.pid)
rm /var/run/foxhunt/trading_service.pid
lsof -i :50052

cd /home/jgrusewski/Work/foxhunt
git checkout e1834ac4
cargo build -p trading_service --release

nohup target/release/trading_service > /var/log/foxhunt/trading_service_rollback.log 2>&1 &
echo $! > /var/run/foxhunt/trading_service.pid

grpc_health_probe -addr=localhost:50052
# Test order submission

Service 3: ML Training Service Rollback (7 min)

kill $(cat /var/run/foxhunt/ml_training_service.pid)
rm /var/run/foxhunt/ml_training_service.pid
nvidia-smi  # Verify GPU freed
lsof -i :50054

cd /home/jgrusewski/Work/foxhunt
git checkout e1834ac4
cargo build -p ml_training_service --release

nohup target/release/ml_training_service > /var/log/foxhunt/ml_training_service_rollback.log 2>&1 &
echo $! > /var/run/foxhunt/ml_training_service.pid

for i in {1..60}; do
  grpc_health_probe -addr=localhost:50054 && break
  sleep 1
done
# Test ML prediction (201 features, NOT 225)

Service 4: Trading Agent Service Rollback (8 min)

kill $(cat /var/run/foxhunt/trading_agent_service.pid)
rm /var/run/foxhunt/trading_agent_service.pid
lsof -i :50055

cd /home/jgrusewski/Work/foxhunt
git checkout e1834ac4
cargo build -p trading_agent_service --release

nohup target/release/trading_agent_service > /var/log/foxhunt/trading_agent_service_rollback.log 2>&1 &
echo $! > /var/run/foxhunt/trading_agent_service.pid

for i in {1..60}; do
  grpc_health_probe -addr=localhost:50055 && break
  sleep 1
done
# Verify NO regime-adaptive calls in logs

Service 5: Backtesting Service Rollback (5 min)

kill $(cat /var/run/foxhunt/backtesting_service.pid)
rm /var/run/foxhunt/backtesting_service.pid
lsof -i :50053

cd /home/jgrusewski/Work/foxhunt
git checkout e1834ac4
cargo build -p backtesting_service --release

nohup target/release/backtesting_service > /var/log/foxhunt/backtesting_service_rollback.log 2>&1 &
echo $! > /var/run/foxhunt/backtesting_service.pid

grpc_health_probe -addr=localhost:50053
# Test backtest (Wave C strategy)

Database Rollback (if tables cause issues)

# 1. Stop ALL services
killall -TERM api_gateway trading_service ml_training_service trading_agent_service backtesting_service

# 2. Drop Wave D tables
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt <<EOF
DROP TABLE IF EXISTS adaptive_strategy_metrics CASCADE;
DROP TABLE IF EXISTS regime_transitions CASCADE;
DROP TABLE IF EXISTS regime_states CASCADE;
DELETE FROM _sqlx_migrations WHERE version = 45;
EOF

# 3. Verify dropped
psql -c "\dt regime_*"
# Expected: No rows

# 4. Restore from backup (if data loss)
psql < /backup/foxhunt_pre_wave_d_*.sql

# 5. Restart all services with pre-Wave D binaries

Rollback Timeline

• Single service: 5-8 minutes
• All 5 services: 25-30 minutes
• Full system + database: 35-40 minutes

Post-Rollback Actions

1. Document failure in /var/log/foxhunt/rollback_report.txt
2. Notify stakeholders (Slack/email)
3. Schedule post-mortem (24h)
4. Create bug ticket
5. Plan remediation and re-deployment

---

Deployment Timeline Summary

Phase	Duration	Description
Phase 1	30 min	Pre-deployment checklist
Phase 2	15 min	Database migration
Phase 3	60 min	Service deployment
Phase 4	20 min	Smoke tests
Phase 5	15 min	Monitoring setup
Phase 6	60 min	Post-deployment validation
Phase 7	24 hours	Observation period
Phase 8	30 min	Certification

Total: 2-4 hours (deployment) + 24 hours (observation) + 30 min (certification) = 26-28 hours

---

Quick Reference

Service Ports

Service	gRPC	HTTP Health	Metrics
API Gateway	50051	8080	9091
Trading Service	50052	8081	9092
Backtesting	50053	8082	9093
ML Training	50054	8095	9094
Trading Agent	50055	8082	N/A

Critical Commands

# Health checks
grpc_health_probe -addr=localhost:<PORT>
curl -f http://localhost:<HTTP_PORT>/health

# Service status
ps aux | grep <service> | grep -v grep
lsof -i :<PORT>

# Logs
tail -f /var/log/foxhunt/<service>.log
grep -i error /var/log/foxhunt/<service>.log

# Database
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
psql -c "SELECT COUNT(*) FROM regime_states;"

# Monitoring
curl -s http://localhost:9090/api/v1/targets
curl -s http://localhost:9090/api/v1/alerts

Emergency Contacts

• On-call Engineer: [FILL]
• Technical Lead: [FILL]
• CTO: [FILL]
• Slack Channel: #foxhunt-alerts
• PagerDuty: [FILL]

---

Appendix: File Locations

Scripts

• Health check: /opt/foxhunt/scripts/wave_d_health_check.sh
• Daily check: /opt/foxhunt/scripts/wave_d_daily_check.sh
• Certification: /opt/foxhunt/scripts/generate_certification_report.sh

Logs

• Service logs: /var/log/foxhunt/<service>.log
• Health checks: /var/log/foxhunt/health_checks/
• Daily checks: /var/log/foxhunt/daily_checks/
• Rollback report: /var/log/foxhunt/rollback_report.txt

Configuration

• Prometheus alerts: /etc/prometheus/alerts/wave_d_alerts.yml
• Grafana dashboards: grafana/wave_d_*.json
• Service config: config/production.toml

Backups

• Database: /backup/foxhunt_pre_wave_d_*.sql
• Binary archives: target/release/ (git commit e1834ac4)

---

Document Status: Ready for Execution Last Updated: 2025-10-19 Version: 1.0 Approval Required: Technical Lead + CTO

---

END OF DEPLOYMENT PLAN