foxhunt/docs/CHECKPOINT_SELECTION_FRAMEWORK.md

Checkpoint Selection Framework

Author: Agent Analysis Date: 2025-10-14 Status: Design Specification Related: Wave 160 ML Training Infrastructure, GPU Training Benchmark System

---

Executive Summary

Production-ready framework for systematic ML checkpoint selection and ensemble composition in the Foxhunt HFT system. Addresses the critical problem of selecting high-performing checkpoints from 100+ epoch training runs and composing optimal ensemble models for trading signal generation.

Key Metrics:

• Target Trade Frequency: 2-5% of bars (vs. current 0.01% baseline)
• Minimum Sharpe Ratio: >1.0 (annualized, risk-adjusted)
• Minimum Win Rate: >50% (statistically significant)
• Maximum Drawdown: <20% (capital preservation)
• Selection Latency: <100ms (backtest evaluation per checkpoint)

---

1. Problem Analysis

1.1 Current Training Pipeline

DQN/PPO Training (100 epochs, RTX 3050 Ti)
    ↓
Checkpoint saves every 10 epochs → MinIO/S3
    ↓
Result: 10 checkpoints × 2 models = 20+ checkpoints
    ↓
Problem: Which checkpoints to deploy?

1.2 What Makes a "Good" Checkpoint?

Primary Quality Metrics

1. Trade Frequency (Weight: 0.2)

   • Target: 2-5% of total bars (400-1000 trades per 20K bar dataset)
   • Current Baseline: 0.01% (2 trades per 20K bars) ❌ TOO LOW
   • Why it matters: Undertrading indicates overly conservative model (low PnL potential)
   • Data source: TradeStatistics.total_trades from backtesting metrics

2. Sharpe Ratio (Weight: 0.4)

   • Target: >1.0 (annualized risk-adjusted returns)
   • World-class: >2.0 (elite quant funds)
   • Why it matters: Risk-adjusted profitability (primary alpha measure)
   • Data source: RiskMetrics.sharpe_ratio from backtesting
   • Formula: (mean_return - risk_free_rate) / std_dev * sqrt(252)

3. Win Rate (Weight: 0.3)

   • Target: >50% (statistically significant edge)
   • Elite: >55% (top decile quant strategies)
   • Why it matters: Indicates model prediction accuracy
   • Data source: TradeStatistics.win_rate
   • Formula: winning_trades / total_trades

4. Maximum Drawdown (Weight: 0.1)

   • Target: <20% (capital preservation)
   • Excellent: <10% (institutional grade)
   • Why it matters: Risk of ruin, investor confidence
   • Data source: DrawdownMetrics.max_drawdown
   • Formula: max((peak - trough) / peak)

Secondary Quality Metrics (Tiebreakers)

• Profit Factor: gross_profit / gross_loss (target >1.5)
• Sortino Ratio: Downside-focused risk measure (target >1.5)
• Calmar Ratio: Return / max_drawdown (target >1.0)
• Average Trade Duration: Alignment with HFT strategy (minutes to hours)
• Convergence Stability: Low variance in last 20% of training epochs

1.3 Current Infrastructure (Available)

From codebase analysis:

1. Checkpoint System (ml/src/checkpoint/mod.rs):

   • ✅ Metadata storage (CheckpointMetadata with metrics HashMap)
   • ✅ Version management (semantic versioning)
   • ✅ Storage backends (FileSystem, S3/MinIO)
   • ✅ Compression (LZ4/Zstd)
   • ✅ Model-specific implementations (DQN, PPO, MAMBA-2, TFT)

2. Backtesting Engine (backtesting/src/metrics.rs):

   • ✅ Comprehensive metrics (PerformanceAnalytics struct)
   • ✅ Sharpe ratio calculation (calculate_sharpe_ratio)
   • ✅ Drawdown analysis (DrawdownMetrics)
   • ✅ Trade statistics (TradeStatistics with win rate, profit factor)
   • ✅ Time-based analysis (monthly/yearly performance)

3. Ensemble System (ml/src/ensemble/model.rs):

   • ✅ Model registration (register_model)
   • ✅ Signal aggregation (weighted average, majority vote, adaptive)
   • ✅ Health monitoring (health_check, HealthInfo)
   • ⚠️ Missing: Checkpoint selection logic
   • ⚠️ Missing: Performance-based model weighting

---

2. Checkpoint Selection Algorithm

2.1 Scoring Function

/// Composite score for checkpoint quality (0.0 to 100.0)
///
/// Weighted combination of 4 primary metrics:
/// - Sharpe ratio: 40% (risk-adjusted returns)
/// - Win rate: 30% (prediction accuracy)
/// - Trade frequency: 20% (activity level)
/// - Drawdown: 10% (risk management)
pub fn calculate_checkpoint_score(metrics: &PerformanceAnalytics) -> f64 {
    const SHARPE_WEIGHT: f64 = 0.4;
    const WIN_RATE_WEIGHT: f64 = 0.3;
    const TRADE_FREQ_WEIGHT: f64 = 0.2;
    const DRAWDOWN_WEIGHT: f64 = 0.1;

    // Component scores (normalized 0-100)
    let sharpe_score = normalize_sharpe_ratio(
        metrics.risk.sharpe_ratio.to_f64()
    );
    let win_rate_score = normalize_win_rate(
        metrics.trade_stats.win_rate.to_f64()
    );
    let trade_freq_score = normalize_trade_frequency(
        metrics.trade_stats.total_trades,
        metrics.time_analysis.total_days
    );
    let drawdown_score = normalize_drawdown(
        metrics.drawdown.max_drawdown.to_f64()
    );

    // Weighted composite
    let composite_score =
        SHARPE_WEIGHT * sharpe_score +
        WIN_RATE_WEIGHT * win_rate_score +
        TRADE_FREQ_WEIGHT * trade_freq_score +
        DRAWDOWN_WEIGHT * drawdown_score;

    composite_score
}

2.2 Normalization Functions

Sharpe Ratio (Target: >1.0, World-class: >2.0)

fn normalize_sharpe_ratio(sharpe: f64) -> f64 {
    // Piecewise linear mapping
    if sharpe < 0.0 {
        0.0  // Negative Sharpe = 0 score
    } else if sharpe < 1.0 {
        sharpe * 50.0  // 0-1.0 → 0-50 points
    } else if sharpe < 2.0 {
        50.0 + (sharpe - 1.0) * 30.0  // 1.0-2.0 → 50-80 points
    } else {
        80.0 + (sharpe - 2.0).min(1.0) * 20.0  // 2.0-3.0 → 80-100 points
    }
}

Win Rate (Target: >50%, Elite: >55%)

fn normalize_win_rate(win_rate: f64) -> f64 {
    // Below 50% = losing strategy
    if win_rate < 0.5 {
        win_rate * 100.0  // 0-0.5 → 0-50 points (harsh penalty)
    } else if win_rate < 0.55 {
        50.0 + (win_rate - 0.5) * 600.0  // 0.5-0.55 → 50-80 points
    } else {
        80.0 + (win_rate - 0.55).min(0.05) * 400.0  // 0.55-0.6 → 80-100 points
    }
}

Trade Frequency (Target: 2-5% of bars)

fn normalize_trade_frequency(total_trades: u64, total_days: i64) -> f64 {
    // Assume ~20 bars per day (1-hour bars for ES futures)
    let estimated_bars = total_days * 20;
    let trade_freq_pct = (total_trades as f64 / estimated_bars as f64) * 100.0;

    // Scoring curve (bell-shaped around 3.5%)
    if trade_freq_pct < 0.1 {
        0.0  // Severe undertrading (<0.1%)
    } else if trade_freq_pct < 2.0 {
        trade_freq_pct * 25.0  // 0.1-2.0% → 2.5-50 points
    } else if trade_freq_pct <= 5.0 {
        50.0 + (trade_freq_pct - 2.0) / 3.0 * 50.0  // 2-5% → 50-100 points (optimal)
    } else {
        100.0 - (trade_freq_pct - 5.0).min(5.0) * 10.0  // >5% → penalize overtrading
    }
}

Maximum Drawdown (Target: <20%, Elite: <10%)

fn normalize_drawdown(max_drawdown_pct: f64) -> f64 {
    // Lower drawdown = higher score (inverted metric)
    if max_drawdown_pct < 0.05 {
        100.0  // <5% drawdown = perfect score
    } else if max_drawdown_pct < 0.10 {
        100.0 - (max_drawdown_pct - 0.05) * 400.0  // 5-10% → 100-80 points
    } else if max_drawdown_pct < 0.20 {
        80.0 - (max_drawdown_pct - 0.10) * 400.0  // 10-20% → 80-40 points
    } else if max_drawdown_pct < 0.30 {
        40.0 - (max_drawdown_pct - 0.20) * 200.0  // 20-30% → 40-20 points
    } else {
        (0.5 - max_drawdown_pct).max(0.0) * 40.0  // >30% → 0-20 points
    }
}

2.3 Statistical Significance Filter

Before scoring, filter out checkpoints with insufficient data:

fn is_statistically_significant(metrics: &PerformanceAnalytics) -> bool {
    // Minimum sample size requirements
    const MIN_TRADES: u64 = 30;  // Statistical power for 50% win rate test
    const MIN_DAYS: i64 = 30;    // 1 month minimum backtest period

    metrics.trade_stats.total_trades >= MIN_TRADES &&
    metrics.time_analysis.total_days >= MIN_DAYS
}

---

3. Automatic Checkpoint Ranking

3.1 Ranking Pipeline

Checkpoint Discovery (MinIO/S3 scan)
    ↓
Load Metadata (CheckpointMetadata for each checkpoint)
    ↓
Backtest Evaluation (parallel, GPU-accelerated)
    ↓
Statistical Filter (min 30 trades, 30 days)
    ↓
Score Calculation (composite metric 0-100)
    ↓
Ranking (descending by score)
    ↓
Top-K Selection (top 5 DQN + top 5 PPO)
    ↓
Ensemble Composition

3.2 Implementation Structure

// New module: ml/src/checkpoint/selection.rs
pub struct CheckpointSelector {
    backtesting_engine: Arc<BacktestingEngine>,
    checkpoint_manager: Arc<CheckpointManager>,
    config: SelectionConfig,
}

pub struct SelectionConfig {
    /// Minimum trades for statistical significance
    pub min_trades: u64,
    /// Minimum backtest period (days)
    pub min_days: i64,
    /// Number of top checkpoints to select per model
    pub top_k: usize,
    /// Parallel backtest workers
    pub num_workers: usize,
    /// DBN data source for backtesting
    pub dbn_data_dir: PathBuf,
}

pub struct RankedCheckpoint {
    pub checkpoint_id: String,
    pub checkpoint_path: String,
    pub model_type: ModelType,
    pub epoch: u64,
    pub score: f64,
    pub metrics: PerformanceAnalytics,
    pub rank: usize,  // 1 = best
}

impl CheckpointSelector {
    /// Discover and rank all checkpoints for a model
    pub async fn rank_checkpoints(
        &self,
        model_type: ModelType,
    ) -> Result<Vec<RankedCheckpoint>, MLError> {
        // 1. Discover checkpoints from storage
        let checkpoints = self.checkpoint_manager
            .list_checkpoints(model_type, "")
            .await;

        // 2. Parallel backtest evaluation
        let mut evaluated_checkpoints = Vec::new();
        for checkpoint in checkpoints {
            let metrics = self.backtest_checkpoint(&checkpoint).await?;

            // 3. Statistical significance filter
            if !is_statistically_significant(&metrics) {
                continue;
            }

            // 4. Score calculation
            let score = calculate_checkpoint_score(&metrics);

            evaluated_checkpoints.push((checkpoint, metrics, score));
        }

        // 5. Sort by score (descending)
        evaluated_checkpoints.sort_by(|a, b|
            b.2.partial_cmp(&a.2).unwrap_or(std::cmp::Ordering::Equal)
        );

        // 6. Assign ranks
        let ranked: Vec<RankedCheckpoint> = evaluated_checkpoints
            .into_iter()
            .enumerate()
            .map(|(rank, (checkpoint, metrics, score))| RankedCheckpoint {
                checkpoint_id: checkpoint.checkpoint_id.clone(),
                checkpoint_path: checkpoint.generate_filename(),
                model_type: checkpoint.model_type,
                epoch: checkpoint.epoch.unwrap_or(0),
                score,
                metrics,
                rank: rank + 1,
            })
            .collect();

        Ok(ranked)
    }

    /// Select top-K checkpoints for production ensemble
    pub async fn select_top_k(
        &self,
        model_type: ModelType,
        k: usize,
    ) -> Result<Vec<RankedCheckpoint>, MLError> {
        let ranked = self.rank_checkpoints(model_type).await?;
        Ok(ranked.into_iter().take(k).collect())
    }

    /// Backtest a single checkpoint
    async fn backtest_checkpoint(
        &self,
        checkpoint: &CheckpointMetadata,
    ) -> Result<PerformanceAnalytics, MLError> {
        // Load model from checkpoint
        let model = self.load_model(checkpoint).await?;

        // Run backtest on DBN data
        let metrics = self.backtesting_engine
            .run_backtest(
                model,
                &self.config.dbn_data_dir,
                BacktestConfig::default(),
            )
            .await?;

        Ok(metrics)
    }
}

3.3 Performance Optimization

Parallel Backtesting

use tokio::task::JoinSet;

pub async fn rank_checkpoints_parallel(
    &self,
    model_type: ModelType,
) -> Result<Vec<RankedCheckpoint>, MLError> {
    let checkpoints = self.checkpoint_manager
        .list_checkpoints(model_type, "")
        .await;

    // Spawn parallel backtest tasks
    let mut join_set = JoinSet::new();
    for checkpoint in checkpoints {
        let selector_clone = self.clone();
        join_set.spawn(async move {
            let metrics = selector_clone.backtest_checkpoint(&checkpoint).await?;
            let score = calculate_checkpoint_score(&metrics);
            Ok::<_, MLError>((checkpoint, metrics, score))
        });
    }

    // Collect results
    let mut evaluated = Vec::new();
    while let Some(result) = join_set.join_next().await {
        if let Ok(Ok((checkpoint, metrics, score))) = result {
            if is_statistically_significant(&metrics) {
                evaluated.push((checkpoint, metrics, score));
            }
        }
    }

    // Sort and rank
    evaluated.sort_by(|a, b| b.2.partial_cmp(&a.2).unwrap());
    Ok(self.to_ranked_checkpoints(evaluated))
}

Caching Strategy

use std::collections::HashMap;
use std::sync::RwLock;

pub struct CheckpointCache {
    // Cache: (checkpoint_id, dbn_data_hash) -> PerformanceAnalytics
    cache: Arc<RwLock<HashMap<(String, String), PerformanceAnalytics>>>,
}

impl CheckpointCache {
    pub fn get(
        &self,
        checkpoint_id: &str,
        data_hash: &str,
    ) -> Option<PerformanceAnalytics> {
        self.cache
            .read()
            .ok()?
            .get(&(checkpoint_id.to_string(), data_hash.to_string()))
            .cloned()
    }

    pub fn insert(
        &self,
        checkpoint_id: String,
        data_hash: String,
        metrics: PerformanceAnalytics,
    ) {
        if let Ok(mut cache) = self.cache.write() {
            cache.insert((checkpoint_id, data_hash), metrics);
        }
    }
}

---

4. Ensemble Composition

4.1 Composition Strategy

Principle: Combine top-performing checkpoints from multiple models to diversify prediction strategies.

Top 5 DQN Checkpoints (reinforcement learning)
    +
Top 5 PPO Checkpoints (policy optimization)
    +
Optional: Top 3 MAMBA-2 (state space models)
    +
Optional: Top 2 TFT (transformers)
    =
15-model Ensemble (diverse, robust)

4.2 Model Weighting Strategies

Strategy 1: Score-Based Weighting

pub fn calculate_score_based_weights(
    checkpoints: &[RankedCheckpoint],
) -> Vec<f64> {
    let total_score: f64 = checkpoints.iter().map(|c| c.score).sum();
    checkpoints
        .iter()
        .map(|c| c.score / total_score)
        .collect()
}

Strategy 2: Inverse-Rank Weighting

pub fn calculate_inverse_rank_weights(
    checkpoints: &[RankedCheckpoint],
) -> Vec<f64> {
    // Higher-ranked models get higher weights
    let weights: Vec<f64> = checkpoints
        .iter()
        .map(|c| 1.0 / c.rank as f64)
        .collect();

    let total: f64 = weights.iter().sum();
    weights.into_iter().map(|w| w / total).collect()
}

Strategy 3: Equal Weighting (Baseline)

pub fn calculate_equal_weights(
    checkpoints: &[RankedCheckpoint],
) -> Vec<f64> {
    vec![1.0 / checkpoints.len() as f64; checkpoints.len()]
}

Strategy 4: Sharpe-Optimized Weighting (Advanced)

/// Mean-variance optimization for ensemble weights
/// Maximizes Sharpe ratio of ensemble predictions
pub fn calculate_sharpe_optimized_weights(
    checkpoints: &[RankedCheckpoint],
    correlation_matrix: &[Vec<f64>],
) -> Vec<f64> {
    // Requires covariance matrix of model returns
    // Use quadratic programming to solve:
    // max: (w^T * μ - r_f) / sqrt(w^T * Σ * w)
    // s.t.: sum(w) = 1, w >= 0

    // Placeholder: Use score-based for now
    calculate_score_based_weights(checkpoints)
}

4.3 Ensemble Integration

// Update ml/src/ensemble/model.rs
impl EnsembleModel {
    /// Register top-K checkpoints as ensemble members
    pub async fn register_top_checkpoints(
        &self,
        selector: &CheckpointSelector,
        model_types: Vec<ModelType>,
        k_per_model: usize,
    ) -> Result<(), MLError> {
        for model_type in model_types {
            let top_k = selector.select_top_k(model_type, k_per_model).await?;

            for ranked in top_k {
                self.register_model(
                    &ranked.checkpoint_id,
                    &format!("{:?}", ranked.model_type),
                    &format!("epoch_{}", ranked.epoch),
                    vec!["price".to_string(), "volume".to_string()],
                    100,  // Expected latency
                ).await?;
            }
        }
        Ok(())
    }

    /// Set model weights based on checkpoint scores
    pub fn set_model_weights(
        &self,
        weights: HashMap<String, f64>,
    ) -> Result<(), MLError> {
        // Store weights for signal aggregation
        // (Implementation TBD in aggregation methods)
        Ok(())
    }
}

---

5. Periodic Re-evaluation

5.1 Re-evaluation Strategy

Frequency: Weekly (every Sunday 00:00 UTC)

Triggers:

1. New training epochs completed (automatic)
2. Performance degradation detected (>10% Sharpe drop)
3. Market regime change (volatility spike, correlation breakdown)
4. Manual trigger (operator command)

5.2 Implementation

pub struct RevaluationScheduler {
    selector: Arc<CheckpointSelector>,
    ensemble: Arc<EnsembleModel>,
    schedule: RevaluationSchedule,
}

pub struct RevaluationSchedule {
    /// Cron expression (e.g., "0 0 * * SUN")
    pub cron: String,
    /// Minimum Sharpe ratio before triggering re-evaluation
    pub min_sharpe_threshold: f64,
    /// Performance window (days)
    pub performance_window_days: i64,
}

impl RevaluationScheduler {
    pub async fn run_periodic_reevaluation(&self) -> Result<(), MLError> {
        loop {
            // Wait for next scheduled time
            self.wait_for_next_schedule().await;

            // Re-rank all checkpoints
            let dqn_ranked = self.selector
                .rank_checkpoints(ModelType::DQN)
                .await?;
            let ppo_ranked = self.selector
                .rank_checkpoints(ModelType::PPO)
                .await?;

            // Detect if top performers changed
            let current_top_dqn = self.ensemble.get_active_models(ModelType::DQN);
            let new_top_dqn = dqn_ranked.iter().take(5).collect::<Vec<_>>();

            if self.should_update_ensemble(&current_top_dqn, &new_top_dqn) {
                // Unregister old models
                for model_id in current_top_dqn {
                    self.ensemble.unregister_model(&model_id).await?;
                }

                // Register new top performers
                for ranked in new_top_dqn {
                    self.ensemble.register_model(
                        &ranked.checkpoint_id,
                        &format!("{:?}", ranked.model_type),
                        &format!("epoch_{}", ranked.epoch),
                        vec![],
                        100,
                    ).await?;
                }

                info!("Ensemble updated with new top performers");
            }
        }
    }

    fn should_update_ensemble(
        &self,
        current: &[String],
        new: &[&RankedCheckpoint],
    ) -> bool {
        // Update if any of top 3 changed
        let current_top3: HashSet<_> = current.iter().take(3).collect();
        let new_top3: HashSet<_> = new.iter().take(3)
            .map(|r| &r.checkpoint_id)
            .collect();

        current_top3 != new_top3
    }
}

5.3 Performance Monitoring

pub struct EnsemblePerformanceMonitor {
    ensemble: Arc<EnsembleModel>,
    metrics_history: Arc<RwLock<Vec<EnsembleMetrics>>>,
}

impl EnsemblePerformanceMonitor {
    /// Track ensemble Sharpe ratio in production
    pub async fn monitor_sharpe_ratio(&self) -> Result<f64, MLError> {
        let recent_metrics = self.get_recent_metrics(30)?;  // 30-day window

        let sharpe = self.calculate_sharpe_from_signals(&recent_metrics);

        // Alert if Sharpe drops below threshold
        if sharpe < self.config.min_sharpe_threshold {
            self.trigger_reevaluation().await?;
        }

        Ok(sharpe)
    }

    /// Trigger immediate re-evaluation
    async fn trigger_reevaluation(&self) -> Result<(), MLError> {
        warn!("Performance degradation detected, triggering re-evaluation");
        // Send notification to RevaluationScheduler
        Ok(())
    }
}

---

6. Production Integration

6.1 TLI Commands (User Interface)

# Rank all checkpoints for a model
tli checkpoint rank --model DQN --output ranked_dqn.json

# Select top-K checkpoints
tli checkpoint select --model DQN --top 5

# View checkpoint scores
tli checkpoint list --model DQN --sort score --limit 10

# Deploy ensemble with top performers
tli ensemble deploy --models DQN:5,PPO:5,MAMBA2:3

# Check ensemble health
tli ensemble health

# Force re-evaluation
tli ensemble reevaluate

6.2 ML Training Service Integration

Update services/ml_training_service/src/main.rs:

// New gRPC method
rpc RankCheckpoints(RankCheckpointsRequest) returns (RankCheckpointsResponse);

message RankCheckpointsRequest {
    ModelType model_type = 1;
    uint32 top_k = 2;  // Optional: limit results
}

message RankCheckpointsResponse {
    repeated RankedCheckpoint checkpoints = 1;
}

message RankedCheckpoint {
    string checkpoint_id = 1;
    string checkpoint_path = 2;
    uint64 epoch = 3;
    double score = 4;
    uint32 rank = 5;
    CheckpointMetrics metrics = 6;
}

message CheckpointMetrics {
    double sharpe_ratio = 1;
    double win_rate = 2;
    double max_drawdown = 3;
    uint64 total_trades = 4;
    double trade_frequency_pct = 5;
}

6.3 CI/CD Pipeline Integration

# .github/workflows/ml_checkpoint_selection.yml
name: ML Checkpoint Selection

on:
  schedule:
    - cron: '0 0 * * SUN'  # Weekly on Sunday
  workflow_dispatch:  # Manual trigger

jobs:
  reevaluate_checkpoints:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Rust
        uses: actions-rs/toolchain@v1

      - name: Download DBN test data
        run: aws s3 sync s3://foxhunt-ml-data/dbn/ test_data/

      - name: Run checkpoint ranking
        run: cargo run -p ml --example checkpoint_ranking --release

      - name: Update ensemble configuration
        run: |
          tli ensemble deploy \
            --models DQN:5,PPO:5 \
            --config production

      - name: Validate ensemble health
        run: tli ensemble health --fail-on-degraded

---

7. Testing Strategy

7.1 Unit Tests

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_normalize_sharpe_ratio() {
        assert_eq!(normalize_sharpe_ratio(0.0), 0.0);
        assert_eq!(normalize_sharpe_ratio(1.0), 50.0);
        assert_eq!(normalize_sharpe_ratio(2.0), 80.0);
        assert!(normalize_sharpe_ratio(3.0) >= 90.0);
    }

    #[test]
    fn test_normalize_win_rate() {
        assert!(normalize_win_rate(0.45) < 50.0);
        assert_eq!(normalize_win_rate(0.50), 50.0);
        assert!(normalize_win_rate(0.55) >= 80.0);
    }

    #[test]
    fn test_calculate_checkpoint_score() {
        let metrics = create_test_metrics(
            1.5,  // sharpe
            0.55, // win_rate
            500,  // trades
            0.15, // drawdown
        );

        let score = calculate_checkpoint_score(&metrics);
        assert!(score > 70.0);  // Good checkpoint
    }

    #[test]
    fn test_statistical_significance_filter() {
        let metrics = create_test_metrics_with_trades(10);  // Too few trades
        assert!(!is_statistically_significant(&metrics));

        let metrics = create_test_metrics_with_trades(50);
        assert!(is_statistically_significant(&metrics));
    }
}

7.2 Integration Tests

#[tokio::test]
async fn test_checkpoint_selection_end_to_end() {
    // 1. Setup test environment
    let temp_dir = tempfile::tempdir().unwrap();
    let checkpoint_manager = create_test_checkpoint_manager(&temp_dir);
    let backtesting_engine = create_test_backtesting_engine();

    let selector = CheckpointSelector::new(
        backtesting_engine,
        checkpoint_manager,
        SelectionConfig::default(),
    );

    // 2. Create synthetic checkpoints
    create_test_checkpoints(&checkpoint_manager, 10).await;

    // 3. Rank checkpoints
    let ranked = selector.rank_checkpoints(ModelType::DQN).await.unwrap();

    // 4. Assertions
    assert_eq!(ranked.len(), 10);
    assert!(ranked[0].score >= ranked[1].score);  // Descending order
    assert!(ranked[0].metrics.risk.sharpe_ratio > Decimal::ZERO);
}

#[tokio::test]
async fn test_ensemble_composition() {
    let ensemble = create_test_ensemble();
    let selector = create_test_selector();

    // Register top 5 DQN + top 5 PPO
    ensemble.register_top_checkpoints(
        &selector,
        vec![ModelType::DQN, ModelType::PPO],
        5,
    ).await.unwrap();

    // Verify 10 models registered
    let models = ensemble.list_models();
    assert_eq!(models.len(), 10);
}

7.3 Performance Benchmarks

#[tokio::test]
async fn benchmark_checkpoint_ranking_latency() {
    let selector = create_test_selector();

    let start = std::time::Instant::now();
    let _ranked = selector.rank_checkpoints(ModelType::DQN).await.unwrap();
    let duration = start.elapsed();

    // Target: <10ms per checkpoint (excluding backtest)
    assert!(duration.as_millis() < 100);
}

#[tokio::test]
async fn benchmark_parallel_backtesting() {
    let selector = create_test_selector_with_workers(4);

    let start = std::time::Instant::now();
    let _ranked = selector.rank_checkpoints_parallel(ModelType::DQN).await.unwrap();
    let duration = start.elapsed();

    // Target: 4x speedup with 4 workers
    // (Actual backtest time depends on DBN data size)
}

---

8. Deployment Checklist

Phase 1: Development (Week 1-2)

• Implement scoring functions (checkpoint/selection.rs)
• Add backtesting integration
• Write unit tests (100% coverage for scoring logic)
• Test with synthetic checkpoints

Phase 2: Integration (Week 3)

• Integrate with checkpoint manager
• Add parallel backtesting support
• Implement caching layer
• Integration tests with real DBN data

Phase 3: Production (Week 4)

• Add TLI commands (tli checkpoint rank/select)
• Implement re-evaluation scheduler
• Add ensemble composition logic
• Performance monitoring dashboard

Phase 4: Validation (Week 5-6)

• Backtest ensemble on 90-day ES/NQ data
• Compare against individual models
• Measure Sharpe ratio improvement
• Validate trade frequency targets (2-5%)

Phase 5: Deployment (Week 7)

• Deploy to production ML Training Service
• Enable weekly re-evaluation cron job
• Monitor ensemble health metrics
• A/B test against baseline strategy

---

9. Success Metrics

Immediate (Week 1-4)

• ✅ Checkpoint ranking completes in <100ms per checkpoint
• ✅ Statistical significance filter removes <20% false positives
• ✅ Score function correlates >0.8 with manual expert ranking

Short-term (Month 1-2)

• ✅ Ensemble Sharpe ratio >1.5 (vs. 1.0 baseline)
• ✅ Trade frequency in 2-5% range (vs. 0.01% baseline)
• ✅ Win rate >52% (statistically significant)
• ✅ Max drawdown <15%

Long-term (Month 3-6)

• ✅ Ensemble outperforms best individual model by >20%
• ✅ Re-evaluation detects performance degradation <7 days
• ✅ Production uptime >99.5% (ensemble health monitoring)
• ✅ User adoption: 100% of ML training jobs use checkpoint selection

---

10. References

Codebase Files

• /home/jgrusewski/Work/foxhunt/ml/src/checkpoint/mod.rs - Checkpoint infrastructure
• /home/jgrusewski/Work/foxhunt/backtesting/src/metrics.rs - Performance analytics (1664 lines)
• /home/jgrusewski/Work/foxhunt/ml/src/ensemble/model.rs - Ensemble management
• /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs - DQN training pipeline

Related Documentation

• CLAUDE.md - System architecture (ML training status)
• GPU_TRAINING_BENCHMARK.md - Wave 152 benchmark system
• ML_TRAINING_ROADMAP.md - 4-6 week training plan
• TESTING_PLAN.md - ML testing strategy

Academic References

• Sharpe Ratio: "The Sharpe Ratio" (Sharpe, 1994)
• Ensemble Methods: "Ensemble Machine Learning" (Zhang & Ma, 2012)
• Quantitative Trading: "Advances in Financial Machine Learning" (López de Prado, 2018)

---

Status: Design Complete - Ready for Implementation Next Action: Implement ml/src/checkpoint/selection.rs (Phase 1) Estimated Effort: 7 weeks (1 developer, full-time) Priority: HIGH (blocks production ML deployment)