foxhunt/ENSEMBLE_PRODUCTION_DEPLOYMENT_STRATEGY.md

# Ensemble Production Deployment Strategy

**Document Version**: 1.0
**Last Updated**: 2025-10-14
**Status**: Production-Ready Design
**Target System**: Foxhunt HFT Trading System

---

## Executive Summary

This document outlines a comprehensive production deployment strategy for the ensemble ML system with zero-downtime hot-swapping, A/B testing framework, advanced monitoring, and gradual rollout with risk mitigation. The strategy leverages existing Foxhunt infrastructure (gRPC, Prometheus, PostgreSQL, Redis) and follows production-ready patterns for high-frequency trading environments.

**Key Features**:
- Zero-downtime checkpoint updates via dual-buffer hot-swapping
- A/B testing framework with statistical significance testing
- Comprehensive monitoring (10 Prometheus metrics for ensemble behavior)
- Gradual rollout: Paper trading → Small position → Full deployment
- Risk mitigation with automated rollback triggers

---

## 1. Production Architecture: Ensemble Coordinator → Trading Service → Live Markets

### 1.1 Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────────┐
│                         API Gateway (Port 50051)                     │
│                  Auth, Rate Limiting, A/B Routing                    │
└───────┬─────────────────────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Trading Service (Port 50052)                      │
│                                                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │            Ensemble Coordinator (New Component)              │   │
│  │                                                               │   │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │   │
│  │  │   DQN        │  │   PPO        │  │  MAMBA-2     │      │   │
│  │  │ Checkpoint A │  │ Checkpoint A │  │ Checkpoint A │      │   │
│  │  └──────────────┘  └──────────────┘  └──────────────┘      │   │
│  │           │                │                │                │   │
│  │           ▼                ▼                ▼                │   │
│  │  ┌────────────────────────────────────────────────────┐     │   │
│  │  │        Signal Aggregator (Weighted Voting)         │     │   │
│  │  │  - Confidence-weighted averaging                   │     │   │
│  │  │  - Model disagreement detection                    │     │   │
│  │  │  - Dynamic weight adjustment (performance-based)   │     │   │
│  │  └────────────────────────────────────────────────────┘     │   │
│  │           │                                                  │   │
│  │           ▼                                                  │   │
│  │  ┌────────────────────────────────────────────────────┐     │   │
│  │  │       Ensemble Decision (Buy/Sell/Hold)            │     │   │
│  │  │  - Signal: -1.0 to 1.0 (bearish to bullish)       │     │   │
│  │  │  - Confidence: 0.0 to 1.0 (low to high)           │     │   │
│  │  │  - Metadata: per-model votes, disagreement rate   │     │   │
│  │  └────────────────────────────────────────────────────┘     │   │
│  └──────────────────────────────────────┬───────────────────────┘   │
│                                         │                           │
│                                         ▼                           │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │               Risk Engine Integration                         │  │
│  │  - Position sizing based on ensemble confidence              │  │
│  │  - VaR calculations using ensemble volatility estimates      │  │
│  │  - Circuit breaker triggers on high disagreement            │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                                         │                           │
│                                         ▼                           │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │               Order Manager                                   │  │
│  │  - Order placement with ensemble attribution                 │  │
│  │  - Execution tracking per model contribution                 │  │
│  └──────────────────────────────────────────────────────────────┘  │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌──────────────────────────────────────────────────────────────────────┐
│                        Live Markets                                   │
│  - ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT (via Databento)                   │
│  - Real-time order execution                                         │
└──────────────────────────────────────────────────────────────────────┘

              │ Monitoring & Observability │
              ▼                             ▼
┌───────────────────────┐      ┌───────────────────────┐
│   Prometheus Metrics   │      │   PostgreSQL Audit    │
│  - Ensemble metrics    │      │  - Trade attribution  │
│  - Model performance   │      │  - P&L per model      │
│  - Disagreement rate   │      │  - Rollout progress   │
└───────────────────────┘      └───────────────────────┘
```

### 1.2 Component Responsibilities

#### Ensemble Coordinator (New Component in Trading Service)

**Location**: `/home/jgrusewski/Work/foxhunt/services/trading_service/src/ensemble_coordinator.rs`

**Responsibilities**:
1. **Model Management**: Load/unload models from safetensors checkpoints
2. **Signal Aggregation**: Weighted voting across DQN, PPO, MAMBA-2, TFT
3. **Hot-Swapping**: Zero-downtime checkpoint updates via dual-buffer pattern
4. **Health Monitoring**: Per-model latency, accuracy, drift detection
5. **A/B Testing**: Route traffic to control/treatment groups

**Key Methods**:
```rust
pub struct EnsembleCoordinator {
    active_models: Arc<RwLock<ModelRegistry>>,
    shadow_models: Arc<RwLock<ModelRegistry>>, // For hot-swapping
    aggregator: Arc<SignalAggregator>,
    ab_router: Arc<ABTestRouter>,
    metrics_collector: Arc<EnsembleMetricsCollector>,
}

impl EnsembleCoordinator {
    // Primary prediction method
    pub async fn predict(&self, features: &Tensor) -> Result<EnsembleDecision>;

    // Hot-swap checkpoint without downtime
    pub async fn swap_checkpoint(&self, model_id: &str, checkpoint_path: &Path) -> Result<()>;

    // A/B testing variant selection
    pub async fn get_prediction_for_group(&self, group: ABGroup, features: &Tensor) -> Result<EnsembleDecision>;
}
```

#### Trading Service Integration

**Existing Component**: `/home/jgrusewski/Work/foxhunt/services/trading_service/src/state.rs`

**Modifications**:
```rust
pub struct TradingServiceState {
    // ... existing fields ...

    /// NEW: Ensemble coordinator for ML predictions
    pub ensemble_coordinator: Option<Arc<EnsembleCoordinator>>,
}

// Integration point in prediction flow
impl TradingServiceState {
    pub async fn get_trading_signal(&self, symbol: &str) -> Result<TradingSignal> {
        // Extract features from market data
        let features = self.feature_extractor.extract(symbol).await?;

        // Get ensemble prediction
        let ensemble_decision = self.ensemble_coordinator
            .as_ref()
            .ok_or(Error::EnsembleNotInitialized)?
            .predict(&features)
            .await?;

        // Convert to trading signal with risk adjustments
        let signal = TradingSignal {
            action: ensemble_decision.action,
            confidence: ensemble_decision.confidence,
            position_size: self.calculate_position_size(ensemble_decision.confidence),
            metadata: ensemble_decision.metadata,
        };

        Ok(signal)
    }
}
```

---

## 2. Hot-Swapping Mechanism: Zero-Downtime Checkpoint Updates

### 2.1 Dual-Buffer Architecture

**Design Pattern**: Double-buffering with atomic pointer swap

```rust
pub struct ModelRegistry {
    models: HashMap<String, ModelBufferPair>,
}

pub struct ModelBufferPair {
    active: Arc<RwLock<Box<dyn MLModel>>>,
    shadow: Arc<RwLock<Option<Box<dyn MLModel>>>>,
    swap_lock: Arc<Mutex<()>>,
}

impl ModelRegistry {
    /// Load new checkpoint into shadow buffer
    pub async fn stage_checkpoint(&mut self, model_id: &str, checkpoint_path: &Path) -> Result<()> {
        let pair = self.models.get_mut(model_id)
            .ok_or(Error::ModelNotFound)?;

        // Load into shadow buffer (no impact on active predictions)
        let new_model = self.load_checkpoint(checkpoint_path).await?;
        *pair.shadow.write().await = Some(new_model);

        tracing::info!("Staged checkpoint for model {} in shadow buffer", model_id);
        Ok(())
    }

    /// Atomic swap: shadow becomes active, active becomes shadow
    pub async fn commit_swap(&mut self, model_id: &str) -> Result<()> {
        let pair = self.models.get_mut(model_id)
            .ok_or(Error::ModelNotFound)?;

        // Acquire swap lock to ensure atomicity
        let _guard = pair.swap_lock.lock().await;

        // Swap pointers (atomic operation)
        let mut active = pair.active.write().await;
        let mut shadow = pair.shadow.write().await;

        if let Some(new_model) = shadow.take() {
            let old_model = std::mem::replace(&mut *active, new_model);
            *shadow = Some(old_model);

            tracing::info!("Committed checkpoint swap for model {}", model_id);

            // Update Prometheus metric
            CHECKPOINT_SWAPS_TOTAL.with_label_values(&[model_id]).inc();
        }

        Ok(())
    }
}
```

### 2.2 Hot-Swap Workflow

**Step-by-Step Process** (Zero Downtime):

```
1. Training completes → New checkpoint saved to MinIO
   └─ File: s3://foxhunt/checkpoints/dqn/epoch_100_sharpe_2.4.safetensors

2. ML Training Service → Notifies Trading Service via gRPC
   └─ RPC: NotifyCheckpointReady(model_id="DQN", checkpoint_url="...")

3. Trading Service → Stages checkpoint in shadow buffer
   ├─ Active buffer continues serving predictions (no interruption)
   └─ Shadow buffer loads new checkpoint (10-50ms for 50-150MB models)

4. Shadow checkpoint validation (production traffic simulation)
   ├─ Run 1,000 inference calls with historical data
   ├─ Verify latency < 50μs P99
   ├─ Verify predictions within expected ranges
   └─ If validation fails → Abort, keep active buffer

5. Atomic swap (< 1μs)
   ├─ Acquire swap lock
   ├─ Swap active ↔ shadow pointers
   └─ Release swap lock

6. Post-swap monitoring (5 minutes canary period)
   ├─ Monitor latency, accuracy, error rate
   ├─ If degradation detected → Rollback to previous checkpoint
   └─ If stable → Commit swap permanently
```

### 2.3 Rollback Mechanism

**Automatic Rollback Triggers**:

```rust
pub struct RollbackPolicy {
    latency_threshold_us: u64,      // Default: 100μs P99
    error_rate_threshold: f64,       // Default: 5%
    accuracy_drop_threshold: f64,    // Default: 10% relative drop
    canary_duration_secs: u64,       // Default: 300 seconds (5 minutes)
}

impl EnsembleCoordinator {
    /// Monitor new checkpoint during canary period
    pub async fn monitor_canary(&self, model_id: &str) -> Result<CanaryResult> {
        let policy = self.rollback_policy.read().await;
        let start_time = Instant::now();

        while start_time.elapsed() < Duration::from_secs(policy.canary_duration_secs) {
            let metrics = self.metrics_collector.get_model_metrics(model_id).await?;

            // Check latency
            if metrics.latency_p99_us > policy.latency_threshold_us {
                return Ok(CanaryResult::Failed(
                    format!("Latency P99 {}μs exceeds threshold {}μs",
                            metrics.latency_p99_us, policy.latency_threshold_us)
                ));
            }

            // Check error rate
            if metrics.error_rate > policy.error_rate_threshold {
                return Ok(CanaryResult::Failed(
                    format!("Error rate {:.2}% exceeds threshold {:.2}%",
                            metrics.error_rate * 100.0, policy.error_rate_threshold * 100.0)
                ));
            }

            // Check accuracy drop
            let baseline_accuracy = self.get_baseline_accuracy(model_id).await?;
            let accuracy_drop = (baseline_accuracy - metrics.accuracy) / baseline_accuracy;
            if accuracy_drop > policy.accuracy_drop_threshold {
                return Ok(CanaryResult::Failed(
                    format!("Accuracy dropped {:.2}% (baseline: {:.2}%, current: {:.2}%)",
                            accuracy_drop * 100.0, baseline_accuracy * 100.0, metrics.accuracy * 100.0)
                ));
            }

            tokio::time::sleep(Duration::from_secs(10)).await;
        }

        Ok(CanaryResult::Success)
    }

    /// Rollback to previous checkpoint
    pub async fn rollback(&mut self, model_id: &str) -> Result<()> {
        tracing::warn!("Initiating rollback for model {}", model_id);

        // Swap back to shadow buffer (which contains previous checkpoint)
        self.model_registry.commit_swap(model_id).await?;

        // Update Prometheus metric
        CHECKPOINT_ROLLBACKS_TOTAL.with_label_values(&[model_id]).inc();

        // Emit alert
        self.alert_manager.emit_rollback_alert(model_id).await;

        Ok(())
    }
}
```

---

## 3. A/B Testing Framework: Compare Ensemble vs Single Model

### 3.1 A/B Test Architecture

**Design**: Stratified randomization with statistical significance testing

```rust
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ABGroup {
    Control,   // Single-model baseline (e.g., DQN only)
    Treatment, // Ensemble model
}

pub struct ABTestRouter {
    config: ABTestConfig,
    group_assignments: Arc<RwLock<HashMap<String, ABGroup>>>, // user_id -> group
    metrics_tracker: Arc<ABMetricsTracker>,
}

pub struct ABTestConfig {
    pub test_id: String,
    pub control_model: ModelVariant,    // e.g., ModelVariant::DQN
    pub treatment_model: ModelVariant,  // e.g., ModelVariant::Ensemble
    pub traffic_split: f64,             // 0.5 = 50/50 split
    pub min_sample_size: usize,         // 1,000 predictions per group
    pub significance_level: f64,        // 0.05 (95% confidence)
    pub max_duration_hours: u64,        // 168 hours (1 week)
}

impl ABTestRouter {
    /// Assign user to control or treatment group (deterministic hash)
    pub fn assign_group(&self, user_id: &str) -> ABGroup {
        let hash = self.hash_user_id(user_id);
        let assignment = if (hash % 100) < (self.config.traffic_split * 100.0) as u64 {
            ABGroup::Treatment
        } else {
            ABGroup::Control
        };

        // Cache assignment for consistency
        self.group_assignments.write().await.insert(user_id.to_string(), assignment);
        assignment
    }

    /// Route prediction request to appropriate model variant
    pub async fn route_prediction(
        &self,
        user_id: &str,
        features: &Tensor,
    ) -> Result<PredictionResult> {
        let group = self.get_or_assign_group(user_id);

        let prediction = match group {
            ABGroup::Control => {
                // Use single model (e.g., DQN only)
                self.control_model.predict(features).await?
            }
            ABGroup::Treatment => {
                // Use ensemble
                self.treatment_model.predict(features).await?
            }
        };

        // Track metrics
        self.metrics_tracker.record_prediction(group, &prediction).await;

        Ok(PredictionResult {
            prediction,
            group,
            test_id: self.config.test_id.clone(),
        })
    }
}
```

### 3.2 Statistical Significance Testing

**Metrics Comparison**:

```rust
pub struct ABMetricsTracker {
    control_metrics: Arc<RwLock<GroupMetrics>>,
    treatment_metrics: Arc<RwLock<GroupMetrics>>,
}

#[derive(Debug, Clone)]
pub struct GroupMetrics {
    pub predictions: u64,
    pub correct_predictions: u64,
    pub total_pnl: f64,
    pub sharpe_ratio: f64,
    pub win_rate: f64,
    pub avg_latency_us: f64,
    pub latency_samples: Vec<u64>,
}

impl ABMetricsTracker {
    /// Compute statistical significance using Welch's t-test
    pub async fn compute_significance(&self) -> Result<ABTestResults> {
        let control = self.control_metrics.read().await;
        let treatment = self.treatment_metrics.read().await;

        // Sharpe ratio comparison (primary metric)
        let sharpe_diff = treatment.sharpe_ratio - control.sharpe_ratio;
        let sharpe_pvalue = self.welch_t_test(
            &control.sharpe_samples(),
            &treatment.sharpe_samples(),
        );

        // Win rate comparison (secondary metric)
        let win_rate_diff = treatment.win_rate - control.win_rate;
        let win_rate_pvalue = self.proportion_z_test(
            control.correct_predictions,
            control.predictions,
            treatment.correct_predictions,
            treatment.predictions,
        );

        // P&L comparison (tertiary metric)
        let pnl_diff = treatment.total_pnl - control.total_pnl;
        let pnl_pvalue = self.mann_whitney_u_test(
            &control.pnl_samples(),
            &treatment.pnl_samples(),
        );

        Ok(ABTestResults {
            test_id: self.test_id.clone(),
            control_group: control.clone(),
            treatment_group: treatment.clone(),
            sharpe_diff,
            sharpe_pvalue,
            win_rate_diff,
            win_rate_pvalue,
            pnl_diff,
            pnl_pvalue,
            is_significant: sharpe_pvalue < 0.05 && pnl_pvalue < 0.05,
            recommendation: self.generate_recommendation(sharpe_diff, sharpe_pvalue),
        })
    }

    fn generate_recommendation(&self, sharpe_diff: f64, pvalue: f64) -> Recommendation {
        if pvalue >= 0.05 {
            Recommendation::Inconclusive("Not statistically significant, continue test".to_string())
        } else if sharpe_diff > 0.2 {
            Recommendation::RolloutTreatment("Ensemble significantly better, deploy to 100%".to_string())
        } else if sharpe_diff < -0.2 {
            Recommendation::RevertToControl("Control significantly better, revert ensemble".to_string())
        } else {
            Recommendation::Neutral("No meaningful difference, use simpler model".to_string())
        }
    }
}
```

### 3.3 A/B Test Dashboard (TLI Commands)

**New TLI Commands**:

```bash
# Start A/B test
tli ab start --control DQN --treatment Ensemble --split 50/50 --duration 7d

# Check test status
tli ab status --test-id <uuid>
# Output:
# Test ID: a1b2c3d4-...
# Status: Running (Day 3/7)
# Control Group (DQN):
#   - Predictions: 5,432
#   - Sharpe Ratio: 1.82
#   - Win Rate: 54.3%
#   - Total P&L: $12,450
# Treatment Group (Ensemble):
#   - Predictions: 5,389
#   - Sharpe Ratio: 2.14 (+17.6%)
#   - Win Rate: 58.1% (+7.0%)
#   - Total P&L: $15,200 (+22.1%)
# Statistical Significance: p=0.012 (SIGNIFICANT)
# Recommendation: Roll out ensemble to 100%

# Stop test early
tli ab stop --test-id <uuid>

# Get final results
tli ab results --test-id <uuid> --format json > ab_test_results.json
```

---

## 4. Monitoring Metrics: Ensemble-Specific Observability

### 4.1 Prometheus Metrics

**New Metrics** (10 ensemble-specific metrics):

```rust
// File: /home/jgrusewski/Work/foxhunt/services/trading_service/src/ensemble_metrics.rs

use prometheus::{register_histogram_vec, register_gauge_vec, register_counter_vec};

/// Histogram for ensemble aggregation latency
pub static ENSEMBLE_AGGREGATION_LATENCY_US: Lazy<HistogramVec> = Lazy::new(|| {
    register_histogram_vec!(
        "ensemble_aggregation_latency_microseconds",
        "Time to aggregate signals from all models",
        &["aggregation_method"], // weighted_average, majority_vote
        vec![1.0, 5.0, 10.0, 25.0, 50.0, 100.0]
    ).expect("Failed to register ensemble_aggregation_latency_microseconds")
});

/// Gauge for ensemble confidence score
pub static ENSEMBLE_CONFIDENCE: Lazy<GaugeVec> = Lazy::new(|| {
    register_gauge_vec!(
        "ensemble_confidence_score",
        "Ensemble prediction confidence (0-1)",
        &["symbol"]
    ).expect("Failed to register ensemble_confidence_score")
});

/// Gauge for model disagreement rate (0-1)
pub static ENSEMBLE_DISAGREEMENT_RATE: Lazy<GaugeVec> = Lazy::new(|| {
    register_gauge_vec!(
        "ensemble_disagreement_rate",
        "Percentage of models disagreeing with ensemble decision",
        &["symbol"]
    ).expect("Failed to register ensemble_disagreement_rate")
});

/// Counter for ensemble predictions by action type
pub static ENSEMBLE_PREDICTIONS_TOTAL: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "ensemble_predictions_total",
        "Total ensemble predictions by action",
        &["action", "symbol"] // buy, sell, hold
    ).expect("Failed to register ensemble_predictions_total")
});

/// Gauge for per-model contribution to ensemble decision
pub static ENSEMBLE_MODEL_WEIGHT: Lazy<GaugeVec> = Lazy::new(|| {
    register_gauge_vec!(
        "ensemble_model_weight",
        "Dynamic weight assigned to each model in ensemble",
        &["model_id", "symbol"]
    ).expect("Failed to register ensemble_model_weight")
});

/// Counter for high-disagreement events (>50% models disagree)
pub static ENSEMBLE_HIGH_DISAGREEMENT_TOTAL: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "ensemble_high_disagreement_total",
        "Count of predictions with high model disagreement",
        &["symbol", "threshold"] // 0.5, 0.7, 0.9
    ).expect("Failed to register ensemble_high_disagreement_total")
});

/// Histogram for P&L attribution per model
pub static ENSEMBLE_MODEL_PNL_CONTRIBUTION: Lazy<HistogramVec> = Lazy::new(|| {
    register_histogram_vec!(
        "ensemble_model_pnl_contribution_dollars",
        "P&L contribution from each model in ensemble",
        &["model_id", "symbol"],
        vec![-1000.0, -500.0, -100.0, 0.0, 100.0, 500.0, 1000.0, 5000.0]
    ).expect("Failed to register ensemble_model_pnl_contribution_dollars")
});

/// Counter for checkpoint swaps (hot-swapping)
pub static CHECKPOINT_SWAPS_TOTAL: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "checkpoint_swaps_total",
        "Total checkpoint hot-swaps",
        &["model_id", "status"] // success, failed, rollback
    ).expect("Failed to register checkpoint_swaps_total")
});

/// Counter for A/B test assignments
pub static AB_TEST_ASSIGNMENTS_TOTAL: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "ab_test_assignments_total",
        "Total A/B test group assignments",
        &["test_id", "group"] // control, treatment
    ).expect("Failed to register ab_test_assignments_total")
});

/// Gauge for A/B test metric difference (treatment - control)
pub static AB_TEST_METRIC_DIFF: Lazy<GaugeVec> = Lazy::new(|| {
    register_gauge_vec!(
        "ab_test_metric_difference",
        "Difference in metrics between treatment and control",
        &["test_id", "metric"] // sharpe_ratio, win_rate, pnl
    ).expect("Failed to register ab_test_metric_difference")
});
```

### 4.2 Grafana Dashboard Panels

**Dashboard Name**: "Ensemble ML Production Monitoring"

**Panel 1: Ensemble Confidence & Disagreement**
- **Metric**: `ensemble_confidence_score` (line graph)
- **Metric**: `ensemble_disagreement_rate` (line graph)
- **Alert**: Red when disagreement > 0.5 (high uncertainty)

**Panel 2: Model Weights (Dynamic Contribution)**
- **Metric**: `ensemble_model_weight` (stacked area chart)
- **Visualization**: Shows relative contribution of DQN, PPO, MAMBA-2, TFT over time
- **Insight**: Identify which models dominate decisions

**Panel 3: Per-Model P&L Attribution**
- **Metric**: `ensemble_model_pnl_contribution_dollars` (heatmap)
- **Aggregation**: Sum P&L per model over 1h, 1d, 1w windows
- **Insight**: Which models generate most profit?

**Panel 4: Aggregation Latency**
- **Metric**: `ensemble_aggregation_latency_microseconds` (histogram)
- **Target**: P99 < 25μs
- **Alert**: Red when P99 > 50μs

**Panel 5: High Disagreement Events**
- **Metric**: `ensemble_high_disagreement_total` (counter)
- **Visualization**: Rate of high-disagreement predictions per hour
- **Insight**: Market regime shifts correlate with high disagreement

**Panel 6: Checkpoint Swap Health**
- **Metric**: `checkpoint_swaps_total{status="success"}` (counter)
- **Metric**: `checkpoint_swaps_total{status="rollback"}` (counter)
- **Alert**: Red when rollback rate > 10%

**Panel 7: A/B Test Progress**
- **Metric**: `ab_test_metric_difference{metric="sharpe_ratio"}` (gauge)
- **Metric**: `ab_test_assignments_total` (counter, grouped by group)
- **Visualization**: Real-time Sharpe ratio lift (treatment vs control)

### 4.3 PostgreSQL Audit Schema

**Table: `ensemble_predictions`**
```sql
CREATE TABLE ensemble_predictions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    symbol VARCHAR(20) NOT NULL,
    ensemble_action VARCHAR(10) NOT NULL, -- BUY, SELL, HOLD
    ensemble_confidence DOUBLE PRECISION NOT NULL,
    disagreement_rate DOUBLE PRECISION NOT NULL,

    -- Per-model votes
    dqn_signal DOUBLE PRECISION,
    dqn_confidence DOUBLE PRECISION,
    dqn_weight DOUBLE PRECISION,

    ppo_signal DOUBLE PRECISION,
    ppo_confidence DOUBLE PRECISION,
    ppo_weight DOUBLE PRECISION,

    mamba2_signal DOUBLE PRECISION,
    mamba2_confidence DOUBLE PRECISION,
    mamba2_weight DOUBLE PRECISION,

    tft_signal DOUBLE PRECISION,
    tft_confidence DOUBLE PRECISION,
    tft_weight DOUBLE PRECISION,

    -- Execution tracking
    order_id UUID REFERENCES orders(id),
    executed_price DOUBLE PRECISION,
    pnl DOUBLE PRECISION,

    -- A/B testing
    ab_test_id UUID,
    ab_group VARCHAR(20), -- control, treatment

    INDEX idx_timestamp (timestamp DESC),
    INDEX idx_symbol_timestamp (symbol, timestamp DESC),
    INDEX idx_ab_test (ab_test_id, ab_group)
);

-- Hypertable for time-series optimization (TimescaleDB)
SELECT create_hypertable('ensemble_predictions', 'timestamp');
```

**Table: `model_performance_attribution`**
```sql
CREATE TABLE model_performance_attribution (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    model_id VARCHAR(50) NOT NULL,
    symbol VARCHAR(20) NOT NULL,

    -- Performance metrics
    total_predictions INTEGER NOT NULL,
    correct_predictions INTEGER NOT NULL,
    accuracy DOUBLE PRECISION NOT NULL,
    total_pnl DOUBLE PRECISION NOT NULL,
    sharpe_ratio DOUBLE PRECISION,

    -- Contribution to ensemble
    avg_weight DOUBLE PRECISION NOT NULL,
    avg_confidence DOUBLE PRECISION NOT NULL,

    -- Rolling window
    window_hours INTEGER NOT NULL, -- 1, 24, 168 (1h, 1d, 1w)

    INDEX idx_model_timestamp (model_id, timestamp DESC),
    INDEX idx_symbol_timestamp (symbol, timestamp DESC)
);

SELECT create_hypertable('model_performance_attribution', 'timestamp');
```

---

## 5. Gradual Rollout Strategy: Paper Trading → Small Position → Full Deployment

### 5.1 Rollout Phases

**Phase 0: Pre-Production Validation** (1-2 days)
- **Environment**: Staging (real market data, simulated execution)
- **Traffic**: 0% production
- **Validation**:
  - ✅ All models load successfully (DQN, PPO, MAMBA-2, TFT)
  - ✅ Inference latency < 50μs P99
  - ✅ Hot-swap mechanism functional (test with dummy checkpoints)
  - ✅ Prometheus metrics reporting correctly
  - ✅ PostgreSQL audit logs persisting
- **Exit Criteria**: 10,000 predictions with zero errors

**Phase 1: Paper Trading** (1 week)
- **Environment**: Production (real market data, simulated execution)
- **Traffic**: 0% real capital, 100% shadow mode
- **Validation**:
  - ✅ Ensemble predictions tracked alongside existing system
  - ✅ Compare P&L of ensemble vs current production strategy
  - ✅ Monitor disagreement rate (expect 20-40% in volatile markets)
  - ✅ Verify A/B testing infrastructure
- **Exit Criteria**:
  - Sharpe ratio > 1.5 (paper trading)
  - Win rate > 52%
  - No critical errors or rollbacks

**Phase 2: Small Position (1% Capital)** (1 week)
- **Environment**: Production (real execution, limited capital)
- **Traffic**: 1% of trading capital allocated to ensemble
- **Risk Limits**:
  - Max position size: $10,000 per symbol
  - Max daily loss: $5,000
  - Circuit breaker on 3 consecutive losses
- **Validation**:
  - ✅ Real P&L positive after transaction costs
  - ✅ Slippage within acceptable range (<5 bps)
  - ✅ No execution errors or order rejections
- **Exit Criteria**:
  - Sharpe ratio > 1.5 (real trading)
  - Total P&L > $5,000
  - Max drawdown < 10%

**Phase 3: Medium Position (10% Capital)** (2 weeks)
- **Environment**: Production (real execution, scaled capital)
- **Traffic**: 10% of trading capital
- **Risk Limits**:
  - Max position size: $100,000 per symbol
  - Max daily loss: $50,000
  - Dynamic position sizing based on ensemble confidence
- **Validation**:
  - ✅ Consistent profitability across multiple symbols
  - ✅ Ensemble outperforms baseline strategy (A/B test)
  - ✅ Model weights stabilize (no wild swings)
- **Exit Criteria**:
  - Sharpe ratio > 1.8 (2 weeks of real trading)
  - A/B test shows statistically significant improvement (p < 0.05)
  - No rollbacks in checkpoint swaps

**Phase 4: Full Deployment (100% Capital)** (Ongoing)
- **Environment**: Production (full capital allocation)
- **Traffic**: 100% of trading capital managed by ensemble
- **Risk Limits**:
  - Max position size: $1,000,000 per symbol
  - Max daily loss: $250,000
  - VaR-based position sizing (confidence-adjusted)
- **Ongoing Monitoring**:
  - Daily P&L attribution per model
  - Weekly checkpoint updates with hot-swapping
  - Monthly A/B tests for new model variants
  - Quarterly retraining with latest market data

### 5.2 Rollout Automation (TLI Commands)

```bash
# Phase 1: Enable paper trading
tli rollout start --phase paper-trading --duration 7d
# Output: Ensemble predictions logged, no real execution

# Check paper trading results
tli rollout status
# Output:
# Phase: Paper Trading (Day 5/7)
# Predictions: 15,432
# Simulated P&L: $23,450 (+18.3% vs baseline)
# Sharpe Ratio: 1.92
# Win Rate: 56.2%
# Disagreement Rate: 28.5% (normal)
# Exit Criteria: ✅ All met, ready for Phase 2

# Phase 2: Small position (1% capital)
tli rollout advance --phase small-position --capital-pct 1
# Confirm: "Allocate 1% capital ($50,000) to ensemble? [y/N]"

# Emergency rollback
tli rollout rollback --reason "High slippage detected"
# Reverts to baseline strategy, logs incident

# Phase 4: Full deployment
tli rollout advance --phase full-deployment --capital-pct 100
# Confirm: "Deploy ensemble to 100% capital ($5,000,000)? [y/N]"
```

### 5.3 Risk Mitigation Strategies

#### 5.3.1 Automated Circuit Breakers

```rust
pub struct EnsembleCircuitBreaker {
    config: CircuitBreakerConfig,
    state: Arc<RwLock<CircuitBreakerState>>,
}

pub struct CircuitBreakerConfig {
    max_consecutive_losses: u32,        // Default: 3
    max_daily_drawdown: f64,            // Default: 0.05 (5%)
    max_disagreement_rate: f64,         // Default: 0.70 (70%)
    latency_threshold_us: u64,          // Default: 100μs P99
    min_confidence_threshold: f64,      // Default: 0.60
}

impl EnsembleCircuitBreaker {
    /// Check if ensemble should halt trading
    pub async fn should_halt(&self) -> Result<HaltDecision> {
        let state = self.state.read().await;

        // Check consecutive losses
        if state.consecutive_losses >= self.config.max_consecutive_losses {
            return Ok(HaltDecision::Halt(
                format!("{} consecutive losses exceeded threshold", state.consecutive_losses)
            ));
        }

        // Check daily drawdown
        let daily_pnl = state.get_daily_pnl();
        let drawdown = -daily_pnl / state.starting_capital;
        if drawdown > self.config.max_daily_drawdown {
            return Ok(HaltDecision::Halt(
                format!("Daily drawdown {:.2}% exceeds {:.2}%",
                        drawdown * 100.0, self.config.max_daily_drawdown * 100.0)
            ));
        }

        // Check high disagreement (market regime shift?)
        let disagreement_rate = state.get_recent_disagreement_rate(100); // Last 100 predictions
        if disagreement_rate > self.config.max_disagreement_rate {
            return Ok(HaltDecision::ReduceExposure(
                format!("High disagreement rate {:.2}% suggests regime shift",
                        disagreement_rate * 100.0)
            ));
        }

        // Check inference latency
        let p99_latency = state.get_latency_p99();
        if p99_latency > self.config.latency_threshold_us {
            return Ok(HaltDecision::Halt(
                format!("Inference latency P99 {}μs exceeds {}μs",
                        p99_latency, self.config.latency_threshold_us)
            ));
        }

        Ok(HaltDecision::Continue)
    }
}
```

#### 5.3.2 Dynamic Position Sizing

**Confidence-Based Position Sizing**:

```rust
impl TradingServiceState {
    /// Calculate position size based on ensemble confidence
    pub fn calculate_position_size(&self, ensemble_decision: &EnsembleDecision) -> f64 {
        let base_position = self.config.base_position_size; // e.g., $100,000
        let max_position = self.config.max_position_size;   // e.g., $1,000,000

        // Scale position by confidence (Kelly Criterion approximation)
        let confidence_multiplier = (ensemble_decision.confidence - 0.5) * 2.0; // 0.5 -> 0.0, 1.0 -> 1.0
        let kelly_fraction = confidence_multiplier * 0.25; // Max 25% of capital per trade

        // Adjust for disagreement (reduce size on high disagreement)
        let disagreement_penalty = 1.0 - ensemble_decision.disagreement_rate;

        let position_size = base_position * kelly_fraction * disagreement_penalty;
        position_size.clamp(0.0, max_position)
    }
}
```

#### 5.3.3 Real-Time Alert System

**Alert Channels**:
- **PagerDuty**: Critical alerts (circuit breaker triggered, checkpoint rollback)
- **Slack**: Warnings (high disagreement, latency spike)
- **Email**: Daily summary (P&L attribution, model performance)

```rust
pub struct AlertManager {
    pagerduty_client: PagerDutyClient,
    slack_client: SlackClient,
    email_client: EmailClient,
}

impl AlertManager {
    pub async fn emit_rollback_alert(&self, model_id: &str) {
        // Critical: Notify on-call engineer
        self.pagerduty_client.trigger_incident(
            format!("Ensemble checkpoint rollback for {}", model_id),
            Severity::Critical,
        ).await;

        // Slack notification
        self.slack_client.send_message(
            "#trading-alerts",
            format!("🚨 Checkpoint rollback: {} failed canary validation", model_id),
        ).await;
    }

    pub async fn emit_high_disagreement_alert(&self, symbol: &str, rate: f64) {
        // Warning: Possible market regime shift
        self.slack_client.send_message(
            "#trading-alerts",
            format!("⚠️ High disagreement {:.1}% on {} (possible regime shift)", rate * 100.0, symbol),
        ).await;
    }
}
```

---

## 6. Implementation Roadmap

### Week 1: Core Infrastructure
- [ ] Implement `EnsembleCoordinator` in `/services/trading_service/src/ensemble_coordinator.rs`
- [ ] Add dual-buffer hot-swapping mechanism (`ModelRegistry`)
- [ ] Integrate with existing `TradingServiceState`
- [ ] Add 10 Prometheus metrics for ensemble observability
- [ ] Create PostgreSQL audit tables (`ensemble_predictions`, `model_performance_attribution`)

### Week 2: A/B Testing Framework
- [ ] Implement `ABTestRouter` with stratified randomization
- [ ] Add statistical significance testing (Welch's t-test, proportion z-test)
- [ ] Create TLI commands: `tli ab start/status/stop/results`
- [ ] Build Grafana dashboard for A/B test monitoring

### Week 3: Rollout Automation
- [ ] Implement rollout phases (paper trading → small position → full)
- [ ] Add `CircuitBreaker` with automated halt triggers
- [ ] Create TLI commands: `tli rollout start/advance/rollback/status`
- [ ] Integrate with PagerDuty/Slack for alerts

### Week 4: Testing & Validation
- [ ] Integration tests for hot-swapping (zero-downtime validation)
- [ ] A/B test simulation with historical data
- [ ] Canary rollout simulation (inject failures, verify rollback)
- [ ] Load testing: 10,000 predictions/second with ensemble latency < 50μs P99

### Week 5: Phase 0 (Pre-Production)
- [ ] Deploy to staging environment
- [ ] Run 10,000 predictions with real market data
- [ ] Validate all metrics reporting correctly
- [ ] Execute 10 checkpoint swaps (success rate > 90%)

### Week 6-12: Gradual Rollout
- Week 6: Phase 1 (Paper Trading)
- Week 7: Phase 2 (1% Capital)
- Week 8-9: Phase 3 (10% Capital)
- Week 10-12: Phase 4 (Full Deployment with continuous monitoring)

---

## 7. Success Metrics

### Technical Metrics
- **Inference Latency**: P99 < 50μs (ensemble aggregation)
- **Hot-Swap Success Rate**: > 95% (with < 5% rollbacks)
- **A/B Test Statistical Power**: > 80% (detect 10% Sharpe improvement)
- **Uptime**: > 99.9% (zero unplanned downtime)

### Business Metrics
- **Sharpe Ratio**: > 1.8 (vs baseline 1.5)
- **Win Rate**: > 55% (vs baseline 52%)
- **Max Drawdown**: < 15% (vs baseline 20%)
- **Total P&L**: > $1M/year (after transaction costs)

### Operational Metrics
- **Checkpoint Update Frequency**: Weekly (automated)
- **Rollback Rate**: < 5% (high-quality checkpoints)
- **Alert Noise**: < 10 false positives/week
- **Time to Rollback**: < 5 minutes (automated circuit breaker)

---

## 8. Risk Assessment & Mitigation

| Risk Category | Likelihood | Impact | Mitigation |
|---------------|------------|--------|------------|
| **Hot-swap failure** | Medium | High | Dual-buffer + rollback automation |
| **Model degradation** | Medium | High | 5-minute canary + automated rollback |
| **A/B test contamination** | Low | Medium | Deterministic user hashing + audit logs |
| **High disagreement** | High | Medium | Circuit breaker reduces exposure |
| **Latency spike** | Low | High | Circuit breaker halts trading if P99 > 100μs |
| **Data poisoning** | Low | Critical | Checkpoint validation before staging |
| **Cascading failures** | Low | Critical | Per-model circuit breakers + fallback to baseline |

---

## 9. Disaster Recovery Plan

### Scenario 1: Ensemble Catastrophic Failure
**Trigger**: All models fail simultaneously (e.g., CUDA driver crash)

**Response**:
1. Circuit breaker halts trading (< 1 second)
2. Fallback to single-model baseline (DQN)
3. PagerDuty alert to on-call engineer
4. Investigate root cause (check GPU health, CUDA logs)
5. Hot-swap to last known good checkpoints
6. Resume trading after 5-minute canary validation

**Recovery Time Objective (RTO)**: < 5 minutes

### Scenario 2: Checkpoint Corruption
**Trigger**: New checkpoint fails validation (accuracy < 40%)

**Response**:
1. Abort staging, keep active buffer running
2. Log corruption details to PostgreSQL
3. Alert model training team (Slack)
4. Revert to previous checkpoint (automatic)
5. Retrain model with validated dataset

**RTO**: Zero downtime (active buffer unaffected)

### Scenario 3: High Disagreement Event
**Trigger**: Disagreement rate > 70% for 10 consecutive predictions

**Response**:
1. Reduce position sizes by 50% (dynamic position sizing)
2. Log event as potential regime shift
3. Continue trading with reduced exposure
4. Alert trading desk for manual review
5. If disagreement persists > 1 hour, halt trading

**RTO**: No halt (graceful degradation)

---

## 10. Conclusion

This production deployment strategy provides a comprehensive, battle-tested approach to deploying ensemble ML models in high-frequency trading environments. Key strengths:

1. **Zero Downtime**: Dual-buffer hot-swapping eliminates deployment windows
2. **Data-Driven Decisions**: A/B testing with statistical significance ensures only profitable models reach production
3. **Risk Mitigation**: Automated circuit breakers, gradual rollout, and dynamic position sizing minimize capital risk
4. **Observability**: 10 Prometheus metrics + PostgreSQL audit logs provide full visibility into ensemble behavior
5. **Operational Excellence**: TLI commands automate rollout, rollback, and A/B testing workflows

**Next Steps**:
1. Review and approve deployment plan with trading desk and risk management
2. Execute Week 1-4 implementation roadmap (4 weeks)
3. Begin Phase 0 pre-production validation (1-2 days)
4. Launch Phase 1 paper trading (1 week)
5. Graduate to real capital (Phases 2-4, 4-6 weeks)

**Estimated Timeline to Full Deployment**: 10-12 weeks (conservative, includes 4 weeks development + 6-8 weeks gradual rollout)

---

**Document Status**: Ready for Implementation
**Approval Required**: Trading Desk, Risk Management, Engineering Lead
**Contact**: ML Engineering Team (ml-team@foxhunt.trading)