# Checkpoint Selection Framework

**Author**: Agent Analysis
**Date**: 2025-10-14
**Status**: Design Specification
**Related**: Wave 160 ML Training Infrastructure, GPU Training Benchmark System

---

## Executive Summary

Production-ready framework for systematic ML checkpoint selection and ensemble composition in the Foxhunt HFT system. Addresses the critical problem of selecting high-performing checkpoints from 100+ epoch training runs and composing optimal ensemble models for trading signal generation.

**Key Metrics**:
- **Target Trade Frequency**: 2-5% of bars (vs. current 0.01% baseline)
- **Minimum Sharpe Ratio**: >1.0 (annualized, risk-adjusted)
- **Minimum Win Rate**: >50% (statistically significant)
- **Maximum Drawdown**: <20% (capital preservation)
- **Selection Latency**: <100ms (backtest evaluation per checkpoint)

---

## 1. Problem Analysis

### 1.1 Current Training Pipeline

```
DQN/PPO Training (100 epochs, RTX 3050 Ti)
    ↓
Checkpoint saves every 10 epochs → MinIO/S3
    ↓
Result: 10 checkpoints × 2 models = 20+ checkpoints
    ↓
Problem: Which checkpoints to deploy?
```

### 1.2 What Makes a "Good" Checkpoint?

#### Primary Quality Metrics

1. **Trade Frequency** (Weight: 0.2)
   - **Target**: 2-5% of total bars (400-1000 trades per 20K bar dataset)
   - **Current Baseline**: 0.01% (2 trades per 20K bars) ❌ TOO LOW
   - **Why it matters**: Undertrading indicates overly conservative model (low PnL potential)
   - **Data source**: `TradeStatistics.total_trades` from backtesting metrics

2. **Sharpe Ratio** (Weight: 0.4)
   - **Target**: >1.0 (annualized risk-adjusted returns)
   - **World-class**: >2.0 (elite quant funds)
   - **Why it matters**: Risk-adjusted profitability (primary alpha measure)
   - **Data source**: `RiskMetrics.sharpe_ratio` from backtesting
   - **Formula**: `(mean_return - risk_free_rate) / std_dev * sqrt(252)`

3. **Win Rate** (Weight: 0.3)
   - **Target**: >50% (statistically significant edge)
   - **Elite**: >55% (top decile quant strategies)
   - **Why it matters**: Indicates model prediction accuracy
   - **Data source**: `TradeStatistics.win_rate`
   - **Formula**: `winning_trades / total_trades`

4. **Maximum Drawdown** (Weight: 0.1)
   - **Target**: <20% (capital preservation)
   - **Excellent**: <10% (institutional grade)
   - **Why it matters**: Risk of ruin, investor confidence
   - **Data source**: `DrawdownMetrics.max_drawdown`
   - **Formula**: `max((peak - trough) / peak)`

#### Secondary Quality Metrics (Tiebreakers)

- **Profit Factor**: `gross_profit / gross_loss` (target >1.5)
- **Sortino Ratio**: Downside-focused risk measure (target >1.5)
- **Calmar Ratio**: Return / max_drawdown (target >1.0)
- **Average Trade Duration**: Alignment with HFT strategy (minutes to hours)
- **Convergence Stability**: Low variance in last 20% of training epochs

### 1.3 Current Infrastructure (Available)

From codebase analysis:

1. **Checkpoint System** (`ml/src/checkpoint/mod.rs`):
   - ✅ Metadata storage (`CheckpointMetadata` with metrics HashMap)
   - ✅ Version management (semantic versioning)
   - ✅ Storage backends (FileSystem, S3/MinIO)
   - ✅ Compression (LZ4/Zstd)
   - ✅ Model-specific implementations (DQN, PPO, MAMBA-2, TFT)

2. **Backtesting Engine** (`backtesting/src/metrics.rs`):
   - ✅ Comprehensive metrics (`PerformanceAnalytics` struct)
   - ✅ Sharpe ratio calculation (`calculate_sharpe_ratio`)
   - ✅ Drawdown analysis (`DrawdownMetrics`)
   - ✅ Trade statistics (`TradeStatistics` with win rate, profit factor)
   - ✅ Time-based analysis (monthly/yearly performance)

3. **Ensemble System** (`ml/src/ensemble/model.rs`):
   - ✅ Model registration (`register_model`)
   - ✅ Signal aggregation (weighted average, majority vote, adaptive)
   - ✅ Health monitoring (`health_check`, `HealthInfo`)
   - ⚠️ Missing: Checkpoint selection logic
   - ⚠️ Missing: Performance-based model weighting

---

## 2. Checkpoint Selection Algorithm

### 2.1 Scoring Function

```rust
/// Composite score for checkpoint quality (0.0 to 100.0)
///
/// Weighted combination of 4 primary metrics:
/// - Sharpe ratio: 40% (risk-adjusted returns)
/// - Win rate: 30% (prediction accuracy)
/// - Trade frequency: 20% (activity level)
/// - Drawdown: 10% (risk management)
pub fn calculate_checkpoint_score(metrics: &PerformanceAnalytics) -> f64 {
    const SHARPE_WEIGHT: f64 = 0.4;
    const WIN_RATE_WEIGHT: f64 = 0.3;
    const TRADE_FREQ_WEIGHT: f64 = 0.2;
    const DRAWDOWN_WEIGHT: f64 = 0.1;

    // Component scores (normalized 0-100)
    let sharpe_score = normalize_sharpe_ratio(
        metrics.risk.sharpe_ratio.to_f64()
    );
    let win_rate_score = normalize_win_rate(
        metrics.trade_stats.win_rate.to_f64()
    );
    let trade_freq_score = normalize_trade_frequency(
        metrics.trade_stats.total_trades,
        metrics.time_analysis.total_days
    );
    let drawdown_score = normalize_drawdown(
        metrics.drawdown.max_drawdown.to_f64()
    );

    // Weighted composite
    let composite_score =
        SHARPE_WEIGHT * sharpe_score +
        WIN_RATE_WEIGHT * win_rate_score +
        TRADE_FREQ_WEIGHT * trade_freq_score +
        DRAWDOWN_WEIGHT * drawdown_score;

    composite_score
}
```

### 2.2 Normalization Functions

#### Sharpe Ratio (Target: >1.0, World-class: >2.0)
```rust
fn normalize_sharpe_ratio(sharpe: f64) -> f64 {
    // Piecewise linear mapping
    if sharpe < 0.0 {
        0.0  // Negative Sharpe = 0 score
    } else if sharpe < 1.0 {
        sharpe * 50.0  // 0-1.0 → 0-50 points
    } else if sharpe < 2.0 {
        50.0 + (sharpe - 1.0) * 30.0  // 1.0-2.0 → 50-80 points
    } else {
        80.0 + (sharpe - 2.0).min(1.0) * 20.0  // 2.0-3.0 → 80-100 points
    }
}
```

#### Win Rate (Target: >50%, Elite: >55%)
```rust
fn normalize_win_rate(win_rate: f64) -> f64 {
    // Below 50% = losing strategy
    if win_rate < 0.5 {
        win_rate * 100.0  // 0-0.5 → 0-50 points (harsh penalty)
    } else if win_rate < 0.55 {
        50.0 + (win_rate - 0.5) * 600.0  // 0.5-0.55 → 50-80 points
    } else {
        80.0 + (win_rate - 0.55).min(0.05) * 400.0  // 0.55-0.6 → 80-100 points
    }
}
```

#### Trade Frequency (Target: 2-5% of bars)
```rust
fn normalize_trade_frequency(total_trades: u64, total_days: i64) -> f64 {
    // Assume ~20 bars per day (1-hour bars for ES futures)
    let estimated_bars = total_days * 20;
    let trade_freq_pct = (total_trades as f64 / estimated_bars as f64) * 100.0;

    // Scoring curve (bell-shaped around 3.5%)
    if trade_freq_pct < 0.1 {
        0.0  // Severe undertrading (<0.1%)
    } else if trade_freq_pct < 2.0 {
        trade_freq_pct * 25.0  // 0.1-2.0% → 2.5-50 points
    } else if trade_freq_pct <= 5.0 {
        50.0 + (trade_freq_pct - 2.0) / 3.0 * 50.0  // 2-5% → 50-100 points (optimal)
    } else {
        100.0 - (trade_freq_pct - 5.0).min(5.0) * 10.0  // >5% → penalize overtrading
    }
}
```

#### Maximum Drawdown (Target: <20%, Elite: <10%)
```rust
fn normalize_drawdown(max_drawdown_pct: f64) -> f64 {
    // Lower drawdown = higher score (inverted metric)
    if max_drawdown_pct < 0.05 {
        100.0  // <5% drawdown = perfect score
    } else if max_drawdown_pct < 0.10 {
        100.0 - (max_drawdown_pct - 0.05) * 400.0  // 5-10% → 100-80 points
    } else if max_drawdown_pct < 0.20 {
        80.0 - (max_drawdown_pct - 0.10) * 400.0  // 10-20% → 80-40 points
    } else if max_drawdown_pct < 0.30 {
        40.0 - (max_drawdown_pct - 0.20) * 200.0  // 20-30% → 40-20 points
    } else {
        (0.5 - max_drawdown_pct).max(0.0) * 40.0  // >30% → 0-20 points
    }
}
```

### 2.3 Statistical Significance Filter

Before scoring, filter out checkpoints with insufficient data:

```rust
fn is_statistically_significant(metrics: &PerformanceAnalytics) -> bool {
    // Minimum sample size requirements
    const MIN_TRADES: u64 = 30;  // Statistical power for 50% win rate test
    const MIN_DAYS: i64 = 30;    // 1 month minimum backtest period

    metrics.trade_stats.total_trades >= MIN_TRADES &&
    metrics.time_analysis.total_days >= MIN_DAYS
}
```

---

## 3. Automatic Checkpoint Ranking

### 3.1 Ranking Pipeline

```
Checkpoint Discovery (MinIO/S3 scan)
    ↓
Load Metadata (CheckpointMetadata for each checkpoint)
    ↓
Backtest Evaluation (parallel, GPU-accelerated)
    ↓
Statistical Filter (min 30 trades, 30 days)
    ↓
Score Calculation (composite metric 0-100)
    ↓
Ranking (descending by score)
    ↓
Top-K Selection (top 5 DQN + top 5 PPO)
    ↓
Ensemble Composition
```

### 3.2 Implementation Structure

```rust
// New module: ml/src/checkpoint/selection.rs
pub struct CheckpointSelector {
    backtesting_engine: Arc<BacktestingEngine>,
    checkpoint_manager: Arc<CheckpointManager>,
    config: SelectionConfig,
}

pub struct SelectionConfig {
    /// Minimum trades for statistical significance
    pub min_trades: u64,
    /// Minimum backtest period (days)
    pub min_days: i64,
    /// Number of top checkpoints to select per model
    pub top_k: usize,
    /// Parallel backtest workers
    pub num_workers: usize,
    /// DBN data source for backtesting
    pub dbn_data_dir: PathBuf,
}

pub struct RankedCheckpoint {
    pub checkpoint_id: String,
    pub checkpoint_path: String,
    pub model_type: ModelType,
    pub epoch: u64,
    pub score: f64,
    pub metrics: PerformanceAnalytics,
    pub rank: usize,  // 1 = best
}

impl CheckpointSelector {
    /// Discover and rank all checkpoints for a model
    pub async fn rank_checkpoints(
        &self,
        model_type: ModelType,
    ) -> Result<Vec<RankedCheckpoint>, MLError> {
        // 1. Discover checkpoints from storage
        let checkpoints = self.checkpoint_manager
            .list_checkpoints(model_type, "")
            .await;

        // 2. Parallel backtest evaluation
        let mut evaluated_checkpoints = Vec::new();
        for checkpoint in checkpoints {
            let metrics = self.backtest_checkpoint(&checkpoint).await?;

            // 3. Statistical significance filter
            if !is_statistically_significant(&metrics) {
                continue;
            }

            // 4. Score calculation
            let score = calculate_checkpoint_score(&metrics);

            evaluated_checkpoints.push((checkpoint, metrics, score));
        }

        // 5. Sort by score (descending)
        evaluated_checkpoints.sort_by(|a, b|
            b.2.partial_cmp(&a.2).unwrap_or(std::cmp::Ordering::Equal)
        );

        // 6. Assign ranks
        let ranked: Vec<RankedCheckpoint> = evaluated_checkpoints
            .into_iter()
            .enumerate()
            .map(|(rank, (checkpoint, metrics, score))| RankedCheckpoint {
                checkpoint_id: checkpoint.checkpoint_id.clone(),
                checkpoint_path: checkpoint.generate_filename(),
                model_type: checkpoint.model_type,
                epoch: checkpoint.epoch.unwrap_or(0),
                score,
                metrics,
                rank: rank + 1,
            })
            .collect();

        Ok(ranked)
    }

    /// Select top-K checkpoints for production ensemble
    pub async fn select_top_k(
        &self,
        model_type: ModelType,
        k: usize,
    ) -> Result<Vec<RankedCheckpoint>, MLError> {
        let ranked = self.rank_checkpoints(model_type).await?;
        Ok(ranked.into_iter().take(k).collect())
    }

    /// Backtest a single checkpoint
    async fn backtest_checkpoint(
        &self,
        checkpoint: &CheckpointMetadata,
    ) -> Result<PerformanceAnalytics, MLError> {
        // Load model from checkpoint
        let model = self.load_model(checkpoint).await?;

        // Run backtest on DBN data
        let metrics = self.backtesting_engine
            .run_backtest(
                model,
                &self.config.dbn_data_dir,
                BacktestConfig::default(),
            )
            .await?;

        Ok(metrics)
    }
}
```

### 3.3 Performance Optimization

#### Parallel Backtesting
```rust
use tokio::task::JoinSet;

pub async fn rank_checkpoints_parallel(
    &self,
    model_type: ModelType,
) -> Result<Vec<RankedCheckpoint>, MLError> {
    let checkpoints = self.checkpoint_manager
        .list_checkpoints(model_type, "")
        .await;

    // Spawn parallel backtest tasks
    let mut join_set = JoinSet::new();
    for checkpoint in checkpoints {
        let selector_clone = self.clone();
        join_set.spawn(async move {
            let metrics = selector_clone.backtest_checkpoint(&checkpoint).await?;
            let score = calculate_checkpoint_score(&metrics);
            Ok::<_, MLError>((checkpoint, metrics, score))
        });
    }

    // Collect results
    let mut evaluated = Vec::new();
    while let Some(result) = join_set.join_next().await {
        if let Ok(Ok((checkpoint, metrics, score))) = result {
            if is_statistically_significant(&metrics) {
                evaluated.push((checkpoint, metrics, score));
            }
        }
    }

    // Sort and rank
    evaluated.sort_by(|a, b| b.2.partial_cmp(&a.2).unwrap());
    Ok(self.to_ranked_checkpoints(evaluated))
}
```

#### Caching Strategy
```rust
use std::collections::HashMap;
use std::sync::RwLock;

pub struct CheckpointCache {
    // Cache: (checkpoint_id, dbn_data_hash) -> PerformanceAnalytics
    cache: Arc<RwLock<HashMap<(String, String), PerformanceAnalytics>>>,
}

impl CheckpointCache {
    pub fn get(
        &self,
        checkpoint_id: &str,
        data_hash: &str,
    ) -> Option<PerformanceAnalytics> {
        self.cache
            .read()
            .ok()?
            .get(&(checkpoint_id.to_string(), data_hash.to_string()))
            .cloned()
    }

    pub fn insert(
        &self,
        checkpoint_id: String,
        data_hash: String,
        metrics: PerformanceAnalytics,
    ) {
        if let Ok(mut cache) = self.cache.write() {
            cache.insert((checkpoint_id, data_hash), metrics);
        }
    }
}
```

---

## 4. Ensemble Composition

### 4.1 Composition Strategy

**Principle**: Combine top-performing checkpoints from multiple models to diversify prediction strategies.

```
Top 5 DQN Checkpoints (reinforcement learning)
    +
Top 5 PPO Checkpoints (policy optimization)
    +
Optional: Top 3 MAMBA-2 (state space models)
    +
Optional: Top 2 TFT (transformers)
    =
15-model Ensemble (diverse, robust)
```

### 4.2 Model Weighting Strategies

#### Strategy 1: Score-Based Weighting
```rust
pub fn calculate_score_based_weights(
    checkpoints: &[RankedCheckpoint],
) -> Vec<f64> {
    let total_score: f64 = checkpoints.iter().map(|c| c.score).sum();
    checkpoints
        .iter()
        .map(|c| c.score / total_score)
        .collect()
}
```

#### Strategy 2: Inverse-Rank Weighting
```rust
pub fn calculate_inverse_rank_weights(
    checkpoints: &[RankedCheckpoint],
) -> Vec<f64> {
    // Higher-ranked models get higher weights
    let weights: Vec<f64> = checkpoints
        .iter()
        .map(|c| 1.0 / c.rank as f64)
        .collect();

    let total: f64 = weights.iter().sum();
    weights.into_iter().map(|w| w / total).collect()
}
```

#### Strategy 3: Equal Weighting (Baseline)
```rust
pub fn calculate_equal_weights(
    checkpoints: &[RankedCheckpoint],
) -> Vec<f64> {
    vec![1.0 / checkpoints.len() as f64; checkpoints.len()]
}
```

#### Strategy 4: Sharpe-Optimized Weighting (Advanced)
```rust
/// Mean-variance optimization for ensemble weights
/// Maximizes Sharpe ratio of ensemble predictions
pub fn calculate_sharpe_optimized_weights(
    checkpoints: &[RankedCheckpoint],
    correlation_matrix: &[Vec<f64>],
) -> Vec<f64> {
    // Requires covariance matrix of model returns
    // Use quadratic programming to solve:
    // max: (w^T * μ - r_f) / sqrt(w^T * Σ * w)
    // s.t.: sum(w) = 1, w >= 0

    // Placeholder: Use score-based for now
    calculate_score_based_weights(checkpoints)
}
```

### 4.3 Ensemble Integration

```rust
// Update ml/src/ensemble/model.rs
impl EnsembleModel {
    /// Register top-K checkpoints as ensemble members
    pub async fn register_top_checkpoints(
        &self,
        selector: &CheckpointSelector,
        model_types: Vec<ModelType>,
        k_per_model: usize,
    ) -> Result<(), MLError> {
        for model_type in model_types {
            let top_k = selector.select_top_k(model_type, k_per_model).await?;

            for ranked in top_k {
                self.register_model(
                    &ranked.checkpoint_id,
                    &format!("{:?}", ranked.model_type),
                    &format!("epoch_{}", ranked.epoch),
                    vec!["price".to_string(), "volume".to_string()],
                    100,  // Expected latency
                ).await?;
            }
        }
        Ok(())
    }

    /// Set model weights based on checkpoint scores
    pub fn set_model_weights(
        &self,
        weights: HashMap<String, f64>,
    ) -> Result<(), MLError> {
        // Store weights for signal aggregation
        // (Implementation TBD in aggregation methods)
        Ok(())
    }
}
```

---

## 5. Periodic Re-evaluation

### 5.1 Re-evaluation Strategy

**Frequency**: Weekly (every Sunday 00:00 UTC)

**Triggers**:
1. New training epochs completed (automatic)
2. Performance degradation detected (>10% Sharpe drop)
3. Market regime change (volatility spike, correlation breakdown)
4. Manual trigger (operator command)

### 5.2 Implementation

```rust
pub struct RevaluationScheduler {
    selector: Arc<CheckpointSelector>,
    ensemble: Arc<EnsembleModel>,
    schedule: RevaluationSchedule,
}

pub struct RevaluationSchedule {
    /// Cron expression (e.g., "0 0 * * SUN")
    pub cron: String,
    /// Minimum Sharpe ratio before triggering re-evaluation
    pub min_sharpe_threshold: f64,
    /// Performance window (days)
    pub performance_window_days: i64,
}

impl RevaluationScheduler {
    pub async fn run_periodic_reevaluation(&self) -> Result<(), MLError> {
        loop {
            // Wait for next scheduled time
            self.wait_for_next_schedule().await;

            // Re-rank all checkpoints
            let dqn_ranked = self.selector
                .rank_checkpoints(ModelType::DQN)
                .await?;
            let ppo_ranked = self.selector
                .rank_checkpoints(ModelType::PPO)
                .await?;

            // Detect if top performers changed
            let current_top_dqn = self.ensemble.get_active_models(ModelType::DQN);
            let new_top_dqn = dqn_ranked.iter().take(5).collect::<Vec<_>>();

            if self.should_update_ensemble(&current_top_dqn, &new_top_dqn) {
                // Unregister old models
                for model_id in current_top_dqn {
                    self.ensemble.unregister_model(&model_id).await?;
                }

                // Register new top performers
                for ranked in new_top_dqn {
                    self.ensemble.register_model(
                        &ranked.checkpoint_id,
                        &format!("{:?}", ranked.model_type),
                        &format!("epoch_{}", ranked.epoch),
                        vec![],
                        100,
                    ).await?;
                }

                info!("Ensemble updated with new top performers");
            }
        }
    }

    fn should_update_ensemble(
        &self,
        current: &[String],
        new: &[&RankedCheckpoint],
    ) -> bool {
        // Update if any of top 3 changed
        let current_top3: HashSet<_> = current.iter().take(3).collect();
        let new_top3: HashSet<_> = new.iter().take(3)
            .map(|r| &r.checkpoint_id)
            .collect();

        current_top3 != new_top3
    }
}
```

### 5.3 Performance Monitoring

```rust
pub struct EnsemblePerformanceMonitor {
    ensemble: Arc<EnsembleModel>,
    metrics_history: Arc<RwLock<Vec<EnsembleMetrics>>>,
}

impl EnsemblePerformanceMonitor {
    /// Track ensemble Sharpe ratio in production
    pub async fn monitor_sharpe_ratio(&self) -> Result<f64, MLError> {
        let recent_metrics = self.get_recent_metrics(30)?;  // 30-day window

        let sharpe = self.calculate_sharpe_from_signals(&recent_metrics);

        // Alert if Sharpe drops below threshold
        if sharpe < self.config.min_sharpe_threshold {
            self.trigger_reevaluation().await?;
        }

        Ok(sharpe)
    }

    /// Trigger immediate re-evaluation
    async fn trigger_reevaluation(&self) -> Result<(), MLError> {
        warn!("Performance degradation detected, triggering re-evaluation");
        // Send notification to RevaluationScheduler
        Ok(())
    }
}
```

---

## 6. Production Integration

### 6.1 TLI Commands (User Interface)

```bash
# Rank all checkpoints for a model
tli checkpoint rank --model DQN --output ranked_dqn.json

# Select top-K checkpoints
tli checkpoint select --model DQN --top 5

# View checkpoint scores
tli checkpoint list --model DQN --sort score --limit 10

# Deploy ensemble with top performers
tli ensemble deploy --models DQN:5,PPO:5,MAMBA2:3

# Check ensemble health
tli ensemble health

# Force re-evaluation
tli ensemble reevaluate
```

### 6.2 ML Training Service Integration

Update `services/ml_training_service/src/main.rs`:

```rust
// New gRPC method
rpc RankCheckpoints(RankCheckpointsRequest) returns (RankCheckpointsResponse);

message RankCheckpointsRequest {
    ModelType model_type = 1;
    uint32 top_k = 2;  // Optional: limit results
}

message RankCheckpointsResponse {
    repeated RankedCheckpoint checkpoints = 1;
}

message RankedCheckpoint {
    string checkpoint_id = 1;
    string checkpoint_path = 2;
    uint64 epoch = 3;
    double score = 4;
    uint32 rank = 5;
    CheckpointMetrics metrics = 6;
}

message CheckpointMetrics {
    double sharpe_ratio = 1;
    double win_rate = 2;
    double max_drawdown = 3;
    uint64 total_trades = 4;
    double trade_frequency_pct = 5;
}
```

### 6.3 CI/CD Pipeline Integration

```yaml
# .github/workflows/ml_checkpoint_selection.yml
name: ML Checkpoint Selection

on:
  schedule:
    - cron: '0 0 * * SUN'  # Weekly on Sunday
  workflow_dispatch:  # Manual trigger

jobs:
  reevaluate_checkpoints:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Rust
        uses: actions-rs/toolchain@v1

      - name: Download DBN test data
        run: aws s3 sync s3://foxhunt-ml-data/dbn/ test_data/

      - name: Run checkpoint ranking
        run: cargo run -p ml --example checkpoint_ranking --release

      - name: Update ensemble configuration
        run: |
          tli ensemble deploy \
            --models DQN:5,PPO:5 \
            --config production

      - name: Validate ensemble health
        run: tli ensemble health --fail-on-degraded
```

---

## 7. Testing Strategy

### 7.1 Unit Tests

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_normalize_sharpe_ratio() {
        assert_eq!(normalize_sharpe_ratio(0.0), 0.0);
        assert_eq!(normalize_sharpe_ratio(1.0), 50.0);
        assert_eq!(normalize_sharpe_ratio(2.0), 80.0);
        assert!(normalize_sharpe_ratio(3.0) >= 90.0);
    }

    #[test]
    fn test_normalize_win_rate() {
        assert!(normalize_win_rate(0.45) < 50.0);
        assert_eq!(normalize_win_rate(0.50), 50.0);
        assert!(normalize_win_rate(0.55) >= 80.0);
    }

    #[test]
    fn test_calculate_checkpoint_score() {
        let metrics = create_test_metrics(
            1.5,  // sharpe
            0.55, // win_rate
            500,  // trades
            0.15, // drawdown
        );

        let score = calculate_checkpoint_score(&metrics);
        assert!(score > 70.0);  // Good checkpoint
    }

    #[test]
    fn test_statistical_significance_filter() {
        let metrics = create_test_metrics_with_trades(10);  // Too few trades
        assert!(!is_statistically_significant(&metrics));

        let metrics = create_test_metrics_with_trades(50);
        assert!(is_statistically_significant(&metrics));
    }
}
```

### 7.2 Integration Tests

```rust
#[tokio::test]
async fn test_checkpoint_selection_end_to_end() {
    // 1. Setup test environment
    let temp_dir = tempfile::tempdir().unwrap();
    let checkpoint_manager = create_test_checkpoint_manager(&temp_dir);
    let backtesting_engine = create_test_backtesting_engine();

    let selector = CheckpointSelector::new(
        backtesting_engine,
        checkpoint_manager,
        SelectionConfig::default(),
    );

    // 2. Create synthetic checkpoints
    create_test_checkpoints(&checkpoint_manager, 10).await;

    // 3. Rank checkpoints
    let ranked = selector.rank_checkpoints(ModelType::DQN).await.unwrap();

    // 4. Assertions
    assert_eq!(ranked.len(), 10);
    assert!(ranked[0].score >= ranked[1].score);  // Descending order
    assert!(ranked[0].metrics.risk.sharpe_ratio > Decimal::ZERO);
}

#[tokio::test]
async fn test_ensemble_composition() {
    let ensemble = create_test_ensemble();
    let selector = create_test_selector();

    // Register top 5 DQN + top 5 PPO
    ensemble.register_top_checkpoints(
        &selector,
        vec![ModelType::DQN, ModelType::PPO],
        5,
    ).await.unwrap();

    // Verify 10 models registered
    let models = ensemble.list_models();
    assert_eq!(models.len(), 10);
}
```

### 7.3 Performance Benchmarks

```rust
#[tokio::test]
async fn benchmark_checkpoint_ranking_latency() {
    let selector = create_test_selector();

    let start = std::time::Instant::now();
    let _ranked = selector.rank_checkpoints(ModelType::DQN).await.unwrap();
    let duration = start.elapsed();

    // Target: <10ms per checkpoint (excluding backtest)
    assert!(duration.as_millis() < 100);
}

#[tokio::test]
async fn benchmark_parallel_backtesting() {
    let selector = create_test_selector_with_workers(4);

    let start = std::time::Instant::now();
    let _ranked = selector.rank_checkpoints_parallel(ModelType::DQN).await.unwrap();
    let duration = start.elapsed();

    // Target: 4x speedup with 4 workers
    // (Actual backtest time depends on DBN data size)
}
```

---

## 8. Deployment Checklist

### Phase 1: Development (Week 1-2)
- [ ] Implement scoring functions (`checkpoint/selection.rs`)
- [ ] Add backtesting integration
- [ ] Write unit tests (100% coverage for scoring logic)
- [ ] Test with synthetic checkpoints

### Phase 2: Integration (Week 3)
- [ ] Integrate with checkpoint manager
- [ ] Add parallel backtesting support
- [ ] Implement caching layer
- [ ] Integration tests with real DBN data

### Phase 3: Production (Week 4)
- [ ] Add TLI commands (`tli checkpoint rank/select`)
- [ ] Implement re-evaluation scheduler
- [ ] Add ensemble composition logic
- [ ] Performance monitoring dashboard

### Phase 4: Validation (Week 5-6)
- [ ] Backtest ensemble on 90-day ES/NQ data
- [ ] Compare against individual models
- [ ] Measure Sharpe ratio improvement
- [ ] Validate trade frequency targets (2-5%)

### Phase 5: Deployment (Week 7)
- [ ] Deploy to production ML Training Service
- [ ] Enable weekly re-evaluation cron job
- [ ] Monitor ensemble health metrics
- [ ] A/B test against baseline strategy

---

## 9. Success Metrics

### Immediate (Week 1-4)
- ✅ Checkpoint ranking completes in <100ms per checkpoint
- ✅ Statistical significance filter removes <20% false positives
- ✅ Score function correlates >0.8 with manual expert ranking

### Short-term (Month 1-2)
- ✅ Ensemble Sharpe ratio >1.5 (vs. 1.0 baseline)
- ✅ Trade frequency in 2-5% range (vs. 0.01% baseline)
- ✅ Win rate >52% (statistically significant)
- ✅ Max drawdown <15%

### Long-term (Month 3-6)
- ✅ Ensemble outperforms best individual model by >20%
- ✅ Re-evaluation detects performance degradation <7 days
- ✅ Production uptime >99.5% (ensemble health monitoring)
- ✅ User adoption: 100% of ML training jobs use checkpoint selection

---

## 10. References

### Codebase Files
- `/home/jgrusewski/Work/foxhunt/ml/src/checkpoint/mod.rs` - Checkpoint infrastructure
- `/home/jgrusewski/Work/foxhunt/backtesting/src/metrics.rs` - Performance analytics (1664 lines)
- `/home/jgrusewski/Work/foxhunt/ml/src/ensemble/model.rs` - Ensemble management
- `/home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs` - DQN training pipeline

### Related Documentation
- `CLAUDE.md` - System architecture (ML training status)
- `GPU_TRAINING_BENCHMARK.md` - Wave 152 benchmark system
- `ML_TRAINING_ROADMAP.md` - 4-6 week training plan
- `TESTING_PLAN.md` - ML testing strategy

### Academic References
- Sharpe Ratio: "The Sharpe Ratio" (Sharpe, 1994)
- Ensemble Methods: "Ensemble Machine Learning" (Zhang & Ma, 2012)
- Quantitative Trading: "Advances in Financial Machine Learning" (López de Prado, 2018)

---

**Status**: Design Complete - Ready for Implementation
**Next Action**: Implement `ml/src/checkpoint/selection.rs` (Phase 1)
**Estimated Effort**: 7 weeks (1 developer, full-time)
**Priority**: HIGH (blocks production ML deployment)