foxhunt/ENSEMBLE_PRODUCTION_DEPLOYMENT_STRATEGY.md

Ensemble Production Deployment Strategy

Document Version: 1.0 Last Updated: 2025-10-14 Status: Production-Ready Design Target System: Foxhunt HFT Trading System

---

Executive Summary

This document outlines a comprehensive production deployment strategy for the ensemble ML system with zero-downtime hot-swapping, A/B testing framework, advanced monitoring, and gradual rollout with risk mitigation. The strategy leverages existing Foxhunt infrastructure (gRPC, Prometheus, PostgreSQL, Redis) and follows production-ready patterns for high-frequency trading environments.

Key Features:

• Zero-downtime checkpoint updates via dual-buffer hot-swapping
• A/B testing framework with statistical significance testing
• Comprehensive monitoring (10 Prometheus metrics for ensemble behavior)
• Gradual rollout: Paper trading → Small position → Full deployment
• Risk mitigation with automated rollback triggers

---

1. Production Architecture: Ensemble Coordinator → Trading Service → Live Markets

1.1 Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                         API Gateway (Port 50051)                     │
│                  Auth, Rate Limiting, A/B Routing                    │
└───────┬─────────────────────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Trading Service (Port 50052)                      │
│                                                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │            Ensemble Coordinator (New Component)              │   │
│  │                                                               │   │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │   │
│  │  │   DQN        │  │   PPO        │  │  MAMBA-2     │      │   │
│  │  │ Checkpoint A │  │ Checkpoint A │  │ Checkpoint A │      │   │
│  │  └──────────────┘  └──────────────┘  └──────────────┘      │   │
│  │           │                │                │                │   │
│  │           ▼                ▼                ▼                │   │
│  │  ┌────────────────────────────────────────────────────┐     │   │
│  │  │        Signal Aggregator (Weighted Voting)         │     │   │
│  │  │  - Confidence-weighted averaging                   │     │   │
│  │  │  - Model disagreement detection                    │     │   │
│  │  │  - Dynamic weight adjustment (performance-based)   │     │   │
│  │  └────────────────────────────────────────────────────┘     │   │
│  │           │                                                  │   │
│  │           ▼                                                  │   │
│  │  ┌────────────────────────────────────────────────────┐     │   │
│  │  │       Ensemble Decision (Buy/Sell/Hold)            │     │   │
│  │  │  - Signal: -1.0 to 1.0 (bearish to bullish)       │     │   │
│  │  │  - Confidence: 0.0 to 1.0 (low to high)           │     │   │
│  │  │  - Metadata: per-model votes, disagreement rate   │     │   │
│  │  └────────────────────────────────────────────────────┘     │   │
│  └──────────────────────────────────────┬───────────────────────┘   │
│                                         │                           │
│                                         ▼                           │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │               Risk Engine Integration                         │  │
│  │  - Position sizing based on ensemble confidence              │  │
│  │  - VaR calculations using ensemble volatility estimates      │  │
│  │  - Circuit breaker triggers on high disagreement            │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                                         │                           │
│                                         ▼                           │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │               Order Manager                                   │  │
│  │  - Order placement with ensemble attribution                 │  │
│  │  - Execution tracking per model contribution                 │  │
│  └──────────────────────────────────────────────────────────────┘  │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌──────────────────────────────────────────────────────────────────────┐
│                        Live Markets                                   │
│  - ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT (via Databento)                   │
│  - Real-time order execution                                         │
└──────────────────────────────────────────────────────────────────────┘

              │ Monitoring & Observability │
              ▼                             ▼
┌───────────────────────┐      ┌───────────────────────┐
│   Prometheus Metrics   │      │   PostgreSQL Audit    │
│  - Ensemble metrics    │      │  - Trade attribution  │
│  - Model performance   │      │  - P&L per model      │
│  - Disagreement rate   │      │  - Rollout progress   │
└───────────────────────┘      └───────────────────────┘

1.2 Component Responsibilities

Ensemble Coordinator (New Component in Trading Service)

Location: /home/jgrusewski/Work/foxhunt/services/trading_service/src/ensemble_coordinator.rs

Responsibilities:

1. Model Management: Load/unload models from safetensors checkpoints
2. Signal Aggregation: Weighted voting across DQN, PPO, MAMBA-2, TFT
3. Hot-Swapping: Zero-downtime checkpoint updates via dual-buffer pattern
4. Health Monitoring: Per-model latency, accuracy, drift detection
5. A/B Testing: Route traffic to control/treatment groups

Key Methods:

pub struct EnsembleCoordinator {
    active_models: Arc<RwLock<ModelRegistry>>,
    shadow_models: Arc<RwLock<ModelRegistry>>, // For hot-swapping
    aggregator: Arc<SignalAggregator>,
    ab_router: Arc<ABTestRouter>,
    metrics_collector: Arc<EnsembleMetricsCollector>,
}

impl EnsembleCoordinator {
    // Primary prediction method
    pub async fn predict(&self, features: &Tensor) -> Result<EnsembleDecision>;

    // Hot-swap checkpoint without downtime
    pub async fn swap_checkpoint(&self, model_id: &str, checkpoint_path: &Path) -> Result<()>;

    // A/B testing variant selection
    pub async fn get_prediction_for_group(&self, group: ABGroup, features: &Tensor) -> Result<EnsembleDecision>;
}

Trading Service Integration

Existing Component: /home/jgrusewski/Work/foxhunt/services/trading_service/src/state.rs

Modifications:

pub struct TradingServiceState {
    // ... existing fields ...

    /// NEW: Ensemble coordinator for ML predictions
    pub ensemble_coordinator: Option<Arc<EnsembleCoordinator>>,
}

// Integration point in prediction flow
impl TradingServiceState {
    pub async fn get_trading_signal(&self, symbol: &str) -> Result<TradingSignal> {
        // Extract features from market data
        let features = self.feature_extractor.extract(symbol).await?;

        // Get ensemble prediction
        let ensemble_decision = self.ensemble_coordinator
            .as_ref()
            .ok_or(Error::EnsembleNotInitialized)?
            .predict(&features)
            .await?;

        // Convert to trading signal with risk adjustments
        let signal = TradingSignal {
            action: ensemble_decision.action,
            confidence: ensemble_decision.confidence,
            position_size: self.calculate_position_size(ensemble_decision.confidence),
            metadata: ensemble_decision.metadata,
        };

        Ok(signal)
    }
}

---

2. Hot-Swapping Mechanism: Zero-Downtime Checkpoint Updates

2.1 Dual-Buffer Architecture

Design Pattern: Double-buffering with atomic pointer swap

pub struct ModelRegistry {
    models: HashMap<String, ModelBufferPair>,
}

pub struct ModelBufferPair {
    active: Arc<RwLock<Box<dyn MLModel>>>,
    shadow: Arc<RwLock<Option<Box<dyn MLModel>>>>,
    swap_lock: Arc<Mutex<()>>,
}

impl ModelRegistry {
    /// Load new checkpoint into shadow buffer
    pub async fn stage_checkpoint(&mut self, model_id: &str, checkpoint_path: &Path) -> Result<()> {
        let pair = self.models.get_mut(model_id)
            .ok_or(Error::ModelNotFound)?;

        // Load into shadow buffer (no impact on active predictions)
        let new_model = self.load_checkpoint(checkpoint_path).await?;
        *pair.shadow.write().await = Some(new_model);

        tracing::info!("Staged checkpoint for model {} in shadow buffer", model_id);
        Ok(())
    }

    /// Atomic swap: shadow becomes active, active becomes shadow
    pub async fn commit_swap(&mut self, model_id: &str) -> Result<()> {
        let pair = self.models.get_mut(model_id)
            .ok_or(Error::ModelNotFound)?;

        // Acquire swap lock to ensure atomicity
        let _guard = pair.swap_lock.lock().await;

        // Swap pointers (atomic operation)
        let mut active = pair.active.write().await;
        let mut shadow = pair.shadow.write().await;

        if let Some(new_model) = shadow.take() {
            let old_model = std::mem::replace(&mut *active, new_model);
            *shadow = Some(old_model);

            tracing::info!("Committed checkpoint swap for model {}", model_id);

            // Update Prometheus metric
            CHECKPOINT_SWAPS_TOTAL.with_label_values(&[model_id]).inc();
        }

        Ok(())
    }
}

2.2 Hot-Swap Workflow

Step-by-Step Process (Zero Downtime):

1. Training completes → New checkpoint saved to MinIO
   └─ File: s3://foxhunt/checkpoints/dqn/epoch_100_sharpe_2.4.safetensors

2. ML Training Service → Notifies Trading Service via gRPC
   └─ RPC: NotifyCheckpointReady(model_id="DQN", checkpoint_url="...")

3. Trading Service → Stages checkpoint in shadow buffer
   ├─ Active buffer continues serving predictions (no interruption)
   └─ Shadow buffer loads new checkpoint (10-50ms for 50-150MB models)

4. Shadow checkpoint validation (production traffic simulation)
   ├─ Run 1,000 inference calls with historical data
   ├─ Verify latency < 50μs P99
   ├─ Verify predictions within expected ranges
   └─ If validation fails → Abort, keep active buffer

5. Atomic swap (< 1μs)
   ├─ Acquire swap lock
   ├─ Swap active ↔ shadow pointers
   └─ Release swap lock

6. Post-swap monitoring (5 minutes canary period)
   ├─ Monitor latency, accuracy, error rate
   ├─ If degradation detected → Rollback to previous checkpoint
   └─ If stable → Commit swap permanently

2.3 Rollback Mechanism

Automatic Rollback Triggers:

pub struct RollbackPolicy {
    latency_threshold_us: u64,      // Default: 100μs P99
    error_rate_threshold: f64,       // Default: 5%
    accuracy_drop_threshold: f64,    // Default: 10% relative drop
    canary_duration_secs: u64,       // Default: 300 seconds (5 minutes)
}

impl EnsembleCoordinator {
    /// Monitor new checkpoint during canary period
    pub async fn monitor_canary(&self, model_id: &str) -> Result<CanaryResult> {
        let policy = self.rollback_policy.read().await;
        let start_time = Instant::now();

        while start_time.elapsed() < Duration::from_secs(policy.canary_duration_secs) {
            let metrics = self.metrics_collector.get_model_metrics(model_id).await?;

            // Check latency
            if metrics.latency_p99_us > policy.latency_threshold_us {
                return Ok(CanaryResult::Failed(
                    format!("Latency P99 {}μs exceeds threshold {}μs",
                            metrics.latency_p99_us, policy.latency_threshold_us)
                ));
            }

            // Check error rate
            if metrics.error_rate > policy.error_rate_threshold {
                return Ok(CanaryResult::Failed(
                    format!("Error rate {:.2}% exceeds threshold {:.2}%",
                            metrics.error_rate * 100.0, policy.error_rate_threshold * 100.0)
                ));
            }

            // Check accuracy drop
            let baseline_accuracy = self.get_baseline_accuracy(model_id).await?;
            let accuracy_drop = (baseline_accuracy - metrics.accuracy) / baseline_accuracy;
            if accuracy_drop > policy.accuracy_drop_threshold {
                return Ok(CanaryResult::Failed(
                    format!("Accuracy dropped {:.2}% (baseline: {:.2}%, current: {:.2}%)",
                            accuracy_drop * 100.0, baseline_accuracy * 100.0, metrics.accuracy * 100.0)
                ));
            }

            tokio::time::sleep(Duration::from_secs(10)).await;
        }

        Ok(CanaryResult::Success)
    }

    /// Rollback to previous checkpoint
    pub async fn rollback(&mut self, model_id: &str) -> Result<()> {
        tracing::warn!("Initiating rollback for model {}", model_id);

        // Swap back to shadow buffer (which contains previous checkpoint)
        self.model_registry.commit_swap(model_id).await?;

        // Update Prometheus metric
        CHECKPOINT_ROLLBACKS_TOTAL.with_label_values(&[model_id]).inc();

        // Emit alert
        self.alert_manager.emit_rollback_alert(model_id).await;

        Ok(())
    }
}

---

3. A/B Testing Framework: Compare Ensemble vs Single Model

3.1 A/B Test Architecture

Design: Stratified randomization with statistical significance testing

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ABGroup {
    Control,   // Single-model baseline (e.g., DQN only)
    Treatment, // Ensemble model
}

pub struct ABTestRouter {
    config: ABTestConfig,
    group_assignments: Arc<RwLock<HashMap<String, ABGroup>>>, // user_id -> group
    metrics_tracker: Arc<ABMetricsTracker>,
}

pub struct ABTestConfig {
    pub test_id: String,
    pub control_model: ModelVariant,    // e.g., ModelVariant::DQN
    pub treatment_model: ModelVariant,  // e.g., ModelVariant::Ensemble
    pub traffic_split: f64,             // 0.5 = 50/50 split
    pub min_sample_size: usize,         // 1,000 predictions per group
    pub significance_level: f64,        // 0.05 (95% confidence)
    pub max_duration_hours: u64,        // 168 hours (1 week)
}

impl ABTestRouter {
    /// Assign user to control or treatment group (deterministic hash)
    pub fn assign_group(&self, user_id: &str) -> ABGroup {
        let hash = self.hash_user_id(user_id);
        let assignment = if (hash % 100) < (self.config.traffic_split * 100.0) as u64 {
            ABGroup::Treatment
        } else {
            ABGroup::Control
        };

        // Cache assignment for consistency
        self.group_assignments.write().await.insert(user_id.to_string(), assignment);
        assignment
    }

    /// Route prediction request to appropriate model variant
    pub async fn route_prediction(
        &self,
        user_id: &str,
        features: &Tensor,
    ) -> Result<PredictionResult> {
        let group = self.get_or_assign_group(user_id);

        let prediction = match group {
            ABGroup::Control => {
                // Use single model (e.g., DQN only)
                self.control_model.predict(features).await?
            }
            ABGroup::Treatment => {
                // Use ensemble
                self.treatment_model.predict(features).await?
            }
        };

        // Track metrics
        self.metrics_tracker.record_prediction(group, &prediction).await;

        Ok(PredictionResult {
            prediction,
            group,
            test_id: self.config.test_id.clone(),
        })
    }
}

3.2 Statistical Significance Testing

Metrics Comparison:

pub struct ABMetricsTracker {
    control_metrics: Arc<RwLock<GroupMetrics>>,
    treatment_metrics: Arc<RwLock<GroupMetrics>>,
}

#[derive(Debug, Clone)]
pub struct GroupMetrics {
    pub predictions: u64,
    pub correct_predictions: u64,
    pub total_pnl: f64,
    pub sharpe_ratio: f64,
    pub win_rate: f64,
    pub avg_latency_us: f64,
    pub latency_samples: Vec<u64>,
}

impl ABMetricsTracker {
    /// Compute statistical significance using Welch's t-test
    pub async fn compute_significance(&self) -> Result<ABTestResults> {
        let control = self.control_metrics.read().await;
        let treatment = self.treatment_metrics.read().await;

        // Sharpe ratio comparison (primary metric)
        let sharpe_diff = treatment.sharpe_ratio - control.sharpe_ratio;
        let sharpe_pvalue = self.welch_t_test(
            &control.sharpe_samples(),
            &treatment.sharpe_samples(),
        );

        // Win rate comparison (secondary metric)
        let win_rate_diff = treatment.win_rate - control.win_rate;
        let win_rate_pvalue = self.proportion_z_test(
            control.correct_predictions,
            control.predictions,
            treatment.correct_predictions,
            treatment.predictions,
        );

        // P&L comparison (tertiary metric)
        let pnl_diff = treatment.total_pnl - control.total_pnl;
        let pnl_pvalue = self.mann_whitney_u_test(
            &control.pnl_samples(),
            &treatment.pnl_samples(),
        );

        Ok(ABTestResults {
            test_id: self.test_id.clone(),
            control_group: control.clone(),
            treatment_group: treatment.clone(),
            sharpe_diff,
            sharpe_pvalue,
            win_rate_diff,
            win_rate_pvalue,
            pnl_diff,
            pnl_pvalue,
            is_significant: sharpe_pvalue < 0.05 && pnl_pvalue < 0.05,
            recommendation: self.generate_recommendation(sharpe_diff, sharpe_pvalue),
        })
    }

    fn generate_recommendation(&self, sharpe_diff: f64, pvalue: f64) -> Recommendation {
        if pvalue >= 0.05 {
            Recommendation::Inconclusive("Not statistically significant, continue test".to_string())
        } else if sharpe_diff > 0.2 {
            Recommendation::RolloutTreatment("Ensemble significantly better, deploy to 100%".to_string())
        } else if sharpe_diff < -0.2 {
            Recommendation::RevertToControl("Control significantly better, revert ensemble".to_string())
        } else {
            Recommendation::Neutral("No meaningful difference, use simpler model".to_string())
        }
    }
}

3.3 A/B Test Dashboard (TLI Commands)

New TLI Commands:

# Start A/B test
tli ab start --control DQN --treatment Ensemble --split 50/50 --duration 7d

# Check test status
tli ab status --test-id <uuid>
# Output:
# Test ID: a1b2c3d4-...
# Status: Running (Day 3/7)
# Control Group (DQN):
#   - Predictions: 5,432
#   - Sharpe Ratio: 1.82
#   - Win Rate: 54.3%
#   - Total P&L: $12,450
# Treatment Group (Ensemble):
#   - Predictions: 5,389
#   - Sharpe Ratio: 2.14 (+17.6%)
#   - Win Rate: 58.1% (+7.0%)
#   - Total P&L: $15,200 (+22.1%)
# Statistical Significance: p=0.012 (SIGNIFICANT)
# Recommendation: Roll out ensemble to 100%

# Stop test early
tli ab stop --test-id <uuid>

# Get final results
tli ab results --test-id <uuid> --format json > ab_test_results.json

---

4. Monitoring Metrics: Ensemble-Specific Observability

4.1 Prometheus Metrics

New Metrics (10 ensemble-specific metrics):

// File: /home/jgrusewski/Work/foxhunt/services/trading_service/src/ensemble_metrics.rs

use prometheus::{register_histogram_vec, register_gauge_vec, register_counter_vec};

/// Histogram for ensemble aggregation latency
pub static ENSEMBLE_AGGREGATION_LATENCY_US: Lazy<HistogramVec> = Lazy::new(|| {
    register_histogram_vec!(
        "ensemble_aggregation_latency_microseconds",
        "Time to aggregate signals from all models",
        &["aggregation_method"], // weighted_average, majority_vote
        vec![1.0, 5.0, 10.0, 25.0, 50.0, 100.0]
    ).expect("Failed to register ensemble_aggregation_latency_microseconds")
});

/// Gauge for ensemble confidence score
pub static ENSEMBLE_CONFIDENCE: Lazy<GaugeVec> = Lazy::new(|| {
    register_gauge_vec!(
        "ensemble_confidence_score",
        "Ensemble prediction confidence (0-1)",
        &["symbol"]
    ).expect("Failed to register ensemble_confidence_score")
});

/// Gauge for model disagreement rate (0-1)
pub static ENSEMBLE_DISAGREEMENT_RATE: Lazy<GaugeVec> = Lazy::new(|| {
    register_gauge_vec!(
        "ensemble_disagreement_rate",
        "Percentage of models disagreeing with ensemble decision",
        &["symbol"]
    ).expect("Failed to register ensemble_disagreement_rate")
});

/// Counter for ensemble predictions by action type
pub static ENSEMBLE_PREDICTIONS_TOTAL: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "ensemble_predictions_total",
        "Total ensemble predictions by action",
        &["action", "symbol"] // buy, sell, hold
    ).expect("Failed to register ensemble_predictions_total")
});

/// Gauge for per-model contribution to ensemble decision
pub static ENSEMBLE_MODEL_WEIGHT: Lazy<GaugeVec> = Lazy::new(|| {
    register_gauge_vec!(
        "ensemble_model_weight",
        "Dynamic weight assigned to each model in ensemble",
        &["model_id", "symbol"]
    ).expect("Failed to register ensemble_model_weight")
});

/// Counter for high-disagreement events (>50% models disagree)
pub static ENSEMBLE_HIGH_DISAGREEMENT_TOTAL: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "ensemble_high_disagreement_total",
        "Count of predictions with high model disagreement",
        &["symbol", "threshold"] // 0.5, 0.7, 0.9
    ).expect("Failed to register ensemble_high_disagreement_total")
});

/// Histogram for P&L attribution per model
pub static ENSEMBLE_MODEL_PNL_CONTRIBUTION: Lazy<HistogramVec> = Lazy::new(|| {
    register_histogram_vec!(
        "ensemble_model_pnl_contribution_dollars",
        "P&L contribution from each model in ensemble",
        &["model_id", "symbol"],
        vec![-1000.0, -500.0, -100.0, 0.0, 100.0, 500.0, 1000.0, 5000.0]
    ).expect("Failed to register ensemble_model_pnl_contribution_dollars")
});

/// Counter for checkpoint swaps (hot-swapping)
pub static CHECKPOINT_SWAPS_TOTAL: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "checkpoint_swaps_total",
        "Total checkpoint hot-swaps",
        &["model_id", "status"] // success, failed, rollback
    ).expect("Failed to register checkpoint_swaps_total")
});

/// Counter for A/B test assignments
pub static AB_TEST_ASSIGNMENTS_TOTAL: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "ab_test_assignments_total",
        "Total A/B test group assignments",
        &["test_id", "group"] // control, treatment
    ).expect("Failed to register ab_test_assignments_total")
});

/// Gauge for A/B test metric difference (treatment - control)
pub static AB_TEST_METRIC_DIFF: Lazy<GaugeVec> = Lazy::new(|| {
    register_gauge_vec!(
        "ab_test_metric_difference",
        "Difference in metrics between treatment and control",
        &["test_id", "metric"] // sharpe_ratio, win_rate, pnl
    ).expect("Failed to register ab_test_metric_difference")
});

4.2 Grafana Dashboard Panels

Dashboard Name: "Ensemble ML Production Monitoring"

Panel 1: Ensemble Confidence & Disagreement

• Metric: ensemble_confidence_score (line graph)
• Metric: ensemble_disagreement_rate (line graph)
• Alert: Red when disagreement > 0.5 (high uncertainty)

Panel 2: Model Weights (Dynamic Contribution)

• Metric: ensemble_model_weight (stacked area chart)
• Visualization: Shows relative contribution of DQN, PPO, MAMBA-2, TFT over time
• Insight: Identify which models dominate decisions

Panel 3: Per-Model P&L Attribution

• Metric: ensemble_model_pnl_contribution_dollars (heatmap)
• Aggregation: Sum P&L per model over 1h, 1d, 1w windows
• Insight: Which models generate most profit?

Panel 4: Aggregation Latency

• Metric: ensemble_aggregation_latency_microseconds (histogram)
• Target: P99 < 25μs
• Alert: Red when P99 > 50μs

Panel 5: High Disagreement Events

• Metric: ensemble_high_disagreement_total (counter)
• Visualization: Rate of high-disagreement predictions per hour
• Insight: Market regime shifts correlate with high disagreement

Panel 6: Checkpoint Swap Health

• Metric: checkpoint_swaps_total{status="success"} (counter)
• Metric: checkpoint_swaps_total{status="rollback"} (counter)
• Alert: Red when rollback rate > 10%

Panel 7: A/B Test Progress

• Metric: ab_test_metric_difference{metric="sharpe_ratio"} (gauge)
• Metric: ab_test_assignments_total (counter, grouped by group)
• Visualization: Real-time Sharpe ratio lift (treatment vs control)

4.3 PostgreSQL Audit Schema

Table: ensemble_predictions

CREATE TABLE ensemble_predictions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    symbol VARCHAR(20) NOT NULL,
    ensemble_action VARCHAR(10) NOT NULL, -- BUY, SELL, HOLD
    ensemble_confidence DOUBLE PRECISION NOT NULL,
    disagreement_rate DOUBLE PRECISION NOT NULL,

    -- Per-model votes
    dqn_signal DOUBLE PRECISION,
    dqn_confidence DOUBLE PRECISION,
    dqn_weight DOUBLE PRECISION,

    ppo_signal DOUBLE PRECISION,
    ppo_confidence DOUBLE PRECISION,
    ppo_weight DOUBLE PRECISION,

    mamba2_signal DOUBLE PRECISION,
    mamba2_confidence DOUBLE PRECISION,
    mamba2_weight DOUBLE PRECISION,

    tft_signal DOUBLE PRECISION,
    tft_confidence DOUBLE PRECISION,
    tft_weight DOUBLE PRECISION,

    -- Execution tracking
    order_id UUID REFERENCES orders(id),
    executed_price DOUBLE PRECISION,
    pnl DOUBLE PRECISION,

    -- A/B testing
    ab_test_id UUID,
    ab_group VARCHAR(20), -- control, treatment

    INDEX idx_timestamp (timestamp DESC),
    INDEX idx_symbol_timestamp (symbol, timestamp DESC),
    INDEX idx_ab_test (ab_test_id, ab_group)
);

-- Hypertable for time-series optimization (TimescaleDB)
SELECT create_hypertable('ensemble_predictions', 'timestamp');

Table: model_performance_attribution

CREATE TABLE model_performance_attribution (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    model_id VARCHAR(50) NOT NULL,
    symbol VARCHAR(20) NOT NULL,

    -- Performance metrics
    total_predictions INTEGER NOT NULL,
    correct_predictions INTEGER NOT NULL,
    accuracy DOUBLE PRECISION NOT NULL,
    total_pnl DOUBLE PRECISION NOT NULL,
    sharpe_ratio DOUBLE PRECISION,

    -- Contribution to ensemble
    avg_weight DOUBLE PRECISION NOT NULL,
    avg_confidence DOUBLE PRECISION NOT NULL,

    -- Rolling window
    window_hours INTEGER NOT NULL, -- 1, 24, 168 (1h, 1d, 1w)

    INDEX idx_model_timestamp (model_id, timestamp DESC),
    INDEX idx_symbol_timestamp (symbol, timestamp DESC)
);

SELECT create_hypertable('model_performance_attribution', 'timestamp');

---

5. Gradual Rollout Strategy: Paper Trading → Small Position → Full Deployment

5.1 Rollout Phases

Phase 0: Pre-Production Validation (1-2 days)

• Environment: Staging (real market data, simulated execution)
• Traffic: 0% production
• Validation:
  • ✅ All models load successfully (DQN, PPO, MAMBA-2, TFT)
  • ✅ Inference latency < 50μs P99
  • ✅ Hot-swap mechanism functional (test with dummy checkpoints)
  • ✅ Prometheus metrics reporting correctly
  • ✅ PostgreSQL audit logs persisting
• Exit Criteria: 10,000 predictions with zero errors

Phase 1: Paper Trading (1 week)

• Environment: Production (real market data, simulated execution)
• Traffic: 0% real capital, 100% shadow mode
• Validation:
  • ✅ Ensemble predictions tracked alongside existing system
  • ✅ Compare P&L of ensemble vs current production strategy
  • ✅ Monitor disagreement rate (expect 20-40% in volatile markets)
  • ✅ Verify A/B testing infrastructure
• Exit Criteria:
  • Sharpe ratio > 1.5 (paper trading)
  • Win rate > 52%
  • No critical errors or rollbacks

Phase 2: Small Position (1% Capital) (1 week)

• Environment: Production (real execution, limited capital)
• Traffic: 1% of trading capital allocated to ensemble
• Risk Limits:
  • Max position size: $10,000 per symbol
  • Max daily loss: $5,000
  • Circuit breaker on 3 consecutive losses
• Validation:
  • ✅ Real P&L positive after transaction costs
  • ✅ Slippage within acceptable range (<5 bps)
  • ✅ No execution errors or order rejections
• Exit Criteria:
  • Sharpe ratio > 1.5 (real trading)
  • Total P&L > $5,000
  • Max drawdown < 10%

Phase 3: Medium Position (10% Capital) (2 weeks)

• Environment: Production (real execution, scaled capital)
• Traffic: 10% of trading capital
• Risk Limits:
  • Max position size: $100,000 per symbol
  • Max daily loss: $50,000
  • Dynamic position sizing based on ensemble confidence
• Validation:
  • ✅ Consistent profitability across multiple symbols
  • ✅ Ensemble outperforms baseline strategy (A/B test)
  • ✅ Model weights stabilize (no wild swings)
• Exit Criteria:
  • Sharpe ratio > 1.8 (2 weeks of real trading)
  • A/B test shows statistically significant improvement (p < 0.05)
  • No rollbacks in checkpoint swaps

Phase 4: Full Deployment (100% Capital) (Ongoing)

• Environment: Production (full capital allocation)
• Traffic: 100% of trading capital managed by ensemble
• Risk Limits:
  • Max position size: $1,000,000 per symbol
  • Max daily loss: $250,000
  • VaR-based position sizing (confidence-adjusted)
• Ongoing Monitoring:
  • Daily P&L attribution per model
  • Weekly checkpoint updates with hot-swapping
  • Monthly A/B tests for new model variants
  • Quarterly retraining with latest market data

5.2 Rollout Automation (TLI Commands)

# Phase 1: Enable paper trading
tli rollout start --phase paper-trading --duration 7d
# Output: Ensemble predictions logged, no real execution

# Check paper trading results
tli rollout status
# Output:
# Phase: Paper Trading (Day 5/7)
# Predictions: 15,432
# Simulated P&L: $23,450 (+18.3% vs baseline)
# Sharpe Ratio: 1.92
# Win Rate: 56.2%
# Disagreement Rate: 28.5% (normal)
# Exit Criteria: ✅ All met, ready for Phase 2

# Phase 2: Small position (1% capital)
tli rollout advance --phase small-position --capital-pct 1
# Confirm: "Allocate 1% capital ($50,000) to ensemble? [y/N]"

# Emergency rollback
tli rollout rollback --reason "High slippage detected"
# Reverts to baseline strategy, logs incident

# Phase 4: Full deployment
tli rollout advance --phase full-deployment --capital-pct 100
# Confirm: "Deploy ensemble to 100% capital ($5,000,000)? [y/N]"

5.3 Risk Mitigation Strategies

5.3.1 Automated Circuit Breakers

pub struct EnsembleCircuitBreaker {
    config: CircuitBreakerConfig,
    state: Arc<RwLock<CircuitBreakerState>>,
}

pub struct CircuitBreakerConfig {
    max_consecutive_losses: u32,        // Default: 3
    max_daily_drawdown: f64,            // Default: 0.05 (5%)
    max_disagreement_rate: f64,         // Default: 0.70 (70%)
    latency_threshold_us: u64,          // Default: 100μs P99
    min_confidence_threshold: f64,      // Default: 0.60
}

impl EnsembleCircuitBreaker {
    /// Check if ensemble should halt trading
    pub async fn should_halt(&self) -> Result<HaltDecision> {
        let state = self.state.read().await;

        // Check consecutive losses
        if state.consecutive_losses >= self.config.max_consecutive_losses {
            return Ok(HaltDecision::Halt(
                format!("{} consecutive losses exceeded threshold", state.consecutive_losses)
            ));
        }

        // Check daily drawdown
        let daily_pnl = state.get_daily_pnl();
        let drawdown = -daily_pnl / state.starting_capital;
        if drawdown > self.config.max_daily_drawdown {
            return Ok(HaltDecision::Halt(
                format!("Daily drawdown {:.2}% exceeds {:.2}%",
                        drawdown * 100.0, self.config.max_daily_drawdown * 100.0)
            ));
        }

        // Check high disagreement (market regime shift?)
        let disagreement_rate = state.get_recent_disagreement_rate(100); // Last 100 predictions
        if disagreement_rate > self.config.max_disagreement_rate {
            return Ok(HaltDecision::ReduceExposure(
                format!("High disagreement rate {:.2}% suggests regime shift",
                        disagreement_rate * 100.0)
            ));
        }

        // Check inference latency
        let p99_latency = state.get_latency_p99();
        if p99_latency > self.config.latency_threshold_us {
            return Ok(HaltDecision::Halt(
                format!("Inference latency P99 {}μs exceeds {}μs",
                        p99_latency, self.config.latency_threshold_us)
            ));
        }

        Ok(HaltDecision::Continue)
    }
}

5.3.2 Dynamic Position Sizing

Confidence-Based Position Sizing:

impl TradingServiceState {
    /// Calculate position size based on ensemble confidence
    pub fn calculate_position_size(&self, ensemble_decision: &EnsembleDecision) -> f64 {
        let base_position = self.config.base_position_size; // e.g., $100,000
        let max_position = self.config.max_position_size;   // e.g., $1,000,000

        // Scale position by confidence (Kelly Criterion approximation)
        let confidence_multiplier = (ensemble_decision.confidence - 0.5) * 2.0; // 0.5 -> 0.0, 1.0 -> 1.0
        let kelly_fraction = confidence_multiplier * 0.25; // Max 25% of capital per trade

        // Adjust for disagreement (reduce size on high disagreement)
        let disagreement_penalty = 1.0 - ensemble_decision.disagreement_rate;

        let position_size = base_position * kelly_fraction * disagreement_penalty;
        position_size.clamp(0.0, max_position)
    }
}

5.3.3 Real-Time Alert System

Alert Channels:

• PagerDuty: Critical alerts (circuit breaker triggered, checkpoint rollback)
• Slack: Warnings (high disagreement, latency spike)
• Email: Daily summary (P&L attribution, model performance)

pub struct AlertManager {
    pagerduty_client: PagerDutyClient,
    slack_client: SlackClient,
    email_client: EmailClient,
}

impl AlertManager {
    pub async fn emit_rollback_alert(&self, model_id: &str) {
        // Critical: Notify on-call engineer
        self.pagerduty_client.trigger_incident(
            format!("Ensemble checkpoint rollback for {}", model_id),
            Severity::Critical,
        ).await;

        // Slack notification
        self.slack_client.send_message(
            "#trading-alerts",
            format!("🚨 Checkpoint rollback: {} failed canary validation", model_id),
        ).await;
    }

    pub async fn emit_high_disagreement_alert(&self, symbol: &str, rate: f64) {
        // Warning: Possible market regime shift
        self.slack_client.send_message(
            "#trading-alerts",
            format!("⚠️ High disagreement {:.1}% on {} (possible regime shift)", rate * 100.0, symbol),
        ).await;
    }
}

---

6. Implementation Roadmap

Week 1: Core Infrastructure

• Implement EnsembleCoordinator in /services/trading_service/src/ensemble_coordinator.rs
• Add dual-buffer hot-swapping mechanism (ModelRegistry)
• Integrate with existing TradingServiceState
• Add 10 Prometheus metrics for ensemble observability
• Create PostgreSQL audit tables (ensemble_predictions, model_performance_attribution)

Week 2: A/B Testing Framework

• Implement ABTestRouter with stratified randomization
• Add statistical significance testing (Welch's t-test, proportion z-test)
• Create TLI commands: tli ab start/status/stop/results
• Build Grafana dashboard for A/B test monitoring

Week 3: Rollout Automation

• Implement rollout phases (paper trading → small position → full)
• Add CircuitBreaker with automated halt triggers
• Create TLI commands: tli rollout start/advance/rollback/status
• Integrate with PagerDuty/Slack for alerts

Week 4: Testing & Validation

• Integration tests for hot-swapping (zero-downtime validation)
• A/B test simulation with historical data
• Canary rollout simulation (inject failures, verify rollback)
• Load testing: 10,000 predictions/second with ensemble latency < 50μs P99

Week 5: Phase 0 (Pre-Production)

• Deploy to staging environment
• Run 10,000 predictions with real market data
• Validate all metrics reporting correctly
• Execute 10 checkpoint swaps (success rate > 90%)

Week 6-12: Gradual Rollout

• Week 6: Phase 1 (Paper Trading)
• Week 7: Phase 2 (1% Capital)
• Week 8-9: Phase 3 (10% Capital)
• Week 10-12: Phase 4 (Full Deployment with continuous monitoring)

---

7. Success Metrics

Technical Metrics

• Inference Latency: P99 < 50μs (ensemble aggregation)
• Hot-Swap Success Rate: > 95% (with < 5% rollbacks)
• A/B Test Statistical Power: > 80% (detect 10% Sharpe improvement)
• Uptime: > 99.9% (zero unplanned downtime)

Business Metrics

• Sharpe Ratio: > 1.8 (vs baseline 1.5)
• Win Rate: > 55% (vs baseline 52%)
• Max Drawdown: < 15% (vs baseline 20%)
• Total P&L: > $1M/year (after transaction costs)

Operational Metrics

• Checkpoint Update Frequency: Weekly (automated)
• Rollback Rate: < 5% (high-quality checkpoints)
• Alert Noise: < 10 false positives/week
• Time to Rollback: < 5 minutes (automated circuit breaker)

---

8. Risk Assessment & Mitigation

Risk Category	Likelihood	Impact	Mitigation
Hot-swap failure	Medium	High	Dual-buffer + rollback automation
Model degradation	Medium	High	5-minute canary + automated rollback
A/B test contamination	Low	Medium	Deterministic user hashing + audit logs
High disagreement	High	Medium	Circuit breaker reduces exposure
Latency spike	Low	High	Circuit breaker halts trading if P99 > 100μs
Data poisoning	Low	Critical	Checkpoint validation before staging
Cascading failures	Low	Critical	Per-model circuit breakers + fallback to baseline

---

9. Disaster Recovery Plan

Scenario 1: Ensemble Catastrophic Failure

Trigger: All models fail simultaneously (e.g., CUDA driver crash)

Response:

1. Circuit breaker halts trading (< 1 second)
2. Fallback to single-model baseline (DQN)
3. PagerDuty alert to on-call engineer
4. Investigate root cause (check GPU health, CUDA logs)
5. Hot-swap to last known good checkpoints
6. Resume trading after 5-minute canary validation

Recovery Time Objective (RTO): < 5 minutes

Scenario 2: Checkpoint Corruption

Trigger: New checkpoint fails validation (accuracy < 40%)

Response:

1. Abort staging, keep active buffer running
2. Log corruption details to PostgreSQL
3. Alert model training team (Slack)
4. Revert to previous checkpoint (automatic)
5. Retrain model with validated dataset

RTO: Zero downtime (active buffer unaffected)

Scenario 3: High Disagreement Event

Trigger: Disagreement rate > 70% for 10 consecutive predictions

Response:

1. Reduce position sizes by 50% (dynamic position sizing)
2. Log event as potential regime shift
3. Continue trading with reduced exposure
4. Alert trading desk for manual review
5. If disagreement persists > 1 hour, halt trading

RTO: No halt (graceful degradation)

---

10. Conclusion

This production deployment strategy provides a comprehensive, battle-tested approach to deploying ensemble ML models in high-frequency trading environments. Key strengths:

1. Zero Downtime: Dual-buffer hot-swapping eliminates deployment windows
2. Data-Driven Decisions: A/B testing with statistical significance ensures only profitable models reach production
3. Risk Mitigation: Automated circuit breakers, gradual rollout, and dynamic position sizing minimize capital risk
4. Observability: 10 Prometheus metrics + PostgreSQL audit logs provide full visibility into ensemble behavior
5. Operational Excellence: TLI commands automate rollout, rollback, and A/B testing workflows

Next Steps:

1. Review and approve deployment plan with trading desk and risk management
2. Execute Week 1-4 implementation roadmap (4 weeks)
3. Begin Phase 0 pre-production validation (1-2 days)
4. Launch Phase 1 paper trading (1 week)
5. Graduate to real capital (Phases 2-4, 4-6 weeks)

Estimated Timeline to Full Deployment: 10-12 weeks (conservative, includes 4 weeks development + 6-8 weeks gradual rollout)

---

Document Status: Ready for Implementation Approval Required: Trading Desk, Risk Management, Engineering Lead Contact: ML Engineering Team (ml-team@foxhunt.trading)