foxhunt/docs/CHECKPOINT_SELECTION_QUICKSTART.md

Checkpoint Selection Framework - Quickstart Guide

Last Updated: 2025-10-14 Status: Implementation Ready Related: CHECKPOINT_SELECTION_FRAMEWORK.md (full specification)

---

TL;DR

Problem: After training 100 epochs, we have 10+ checkpoints per model. Which ones are production-ready?

Solution: Systematic scoring algorithm that evaluates:

1. Sharpe Ratio (40% weight) - Risk-adjusted profitability
2. Win Rate (30% weight) - Prediction accuracy
3. Trade Frequency (20% weight) - Activity level (target: 2-5% of bars)
4. Max Drawdown (10% weight) - Risk management

Output: Ranked list of checkpoints (score 0-100) → Top 5 DQN + Top 5 PPO → Production ensemble

---

Core Scoring Formula

score = 0.4 × sharpe_score +
        0.3 × win_rate_score +
        0.2 × trade_freq_score +
        0.1 × drawdown_score

// Example: Checkpoint with Sharpe=1.5, WinRate=0.55, TradePct=3.5%, Drawdown=12%
// → score = 0.4×65 + 0.3×80 + 0.2×90 + 0.1×68 = 77.8 (GOOD)

Target Ranges

Metric	Threshold	Target	Elite	Weight
Sharpe Ratio	>0.5	>1.0	>2.0	40%
Win Rate	>50%	>52%	>55%	30%
Trade Frequency	0.5-10%	2-5%	3-4%	20%
Max Drawdown	<30%	<20%	<10%	10%

---

Implementation Roadmap

Phase 1: Core Scoring (Week 1-2)

# New file: ml/src/checkpoint/selection.rs
cargo new-module ml/src/checkpoint/selection

# Implement:
- normalize_sharpe_ratio()
- normalize_win_rate()
- normalize_trade_frequency()
- normalize_drawdown()
- calculate_checkpoint_score()
- is_statistically_significant()

# Tests:
cargo test -p ml checkpoint::selection

Key Functions:

pub fn calculate_checkpoint_score(metrics: &PerformanceAnalytics) -> f64 {
    let sharpe = normalize_sharpe_ratio(metrics.risk.sharpe_ratio.to_f64());
    let win_rate = normalize_win_rate(metrics.trade_stats.win_rate.to_f64());
    let trade_freq = normalize_trade_frequency(
        metrics.trade_stats.total_trades,
        metrics.time_analysis.total_days
    );
    let drawdown = normalize_drawdown(metrics.drawdown.max_drawdown.to_f64());

    0.4 * sharpe + 0.3 * win_rate + 0.2 * trade_freq + 0.1 * drawdown
}

Phase 2: Backtest Integration (Week 3)

pub struct CheckpointSelector {
    backtesting_engine: Arc<BacktestingEngine>,
    checkpoint_manager: Arc<CheckpointManager>,
    config: SelectionConfig,
}

impl CheckpointSelector {
    pub async fn rank_checkpoints(
        &self,
        model_type: ModelType,
    ) -> Result<Vec<RankedCheckpoint>, MLError> {
        // 1. List all checkpoints from MinIO/S3
        let checkpoints = self.checkpoint_manager
            .list_checkpoints(model_type, "")
            .await;

        // 2. Backtest each checkpoint (parallel)
        let mut evaluated = Vec::new();
        for checkpoint in checkpoints {
            let metrics = self.backtest_checkpoint(&checkpoint).await?;

            // 3. Filter: min 30 trades, 30 days
            if !is_statistically_significant(&metrics) {
                continue;
            }

            // 4. Score
            let score = calculate_checkpoint_score(&metrics);
            evaluated.push((checkpoint, metrics, score));
        }

        // 5. Sort by score (descending)
        evaluated.sort_by(|a, b| b.2.partial_cmp(&a.2).unwrap());

        // 6. Return ranked
        Ok(self.to_ranked_checkpoints(evaluated))
    }
}

Phase 3: Ensemble Composition (Week 4)

// Update ml/src/ensemble/model.rs
impl EnsembleModel {
    pub async fn deploy_top_checkpoints(
        &self,
        selector: &CheckpointSelector,
    ) -> Result<(), MLError> {
        // Top 5 DQN
        let dqn_top5 = selector.select_top_k(ModelType::DQN, 5).await?;
        for ranked in dqn_top5 {
            self.register_model(&ranked.checkpoint_id, ...);
        }

        // Top 5 PPO
        let ppo_top5 = selector.select_top_k(ModelType::PPO, 5).await?;
        for ranked in ppo_top5 {
            self.register_model(&ranked.checkpoint_id, ...);
        }

        // Set weights (score-based)
        let weights = calculate_score_based_weights(&all_checkpoints);
        self.set_model_weights(weights)?;

        Ok(())
    }
}

Phase 4: TLI Commands (Week 5)

# New commands in tli/src/main.rs
tli checkpoint rank --model DQN
tli checkpoint select --model DQN --top 5
tli checkpoint list --sort score
tli ensemble deploy --models DQN:5,PPO:5
tli ensemble health

Phase 5: Production Deployment (Week 6-7)

# Weekly cron job (Sunday 00:00 UTC)
tli ensemble reevaluate

# CI/CD integration (.github/workflows/checkpoint_selection.yml)
- Download DBN data
- Run checkpoint ranking
- Update production ensemble
- Validate health

---

Example Usage

Scenario: Select Best DQN Checkpoint

# 1. List all DQN checkpoints
tli checkpoint list --model DQN
# Output:
# checkpoint_id                 epoch  score   sharpe  win_rate  trades
# dqn_e80_20251014_120000      80     85.4    1.8     0.56      450
# dqn_e70_20251014_120000      70     78.2    1.5     0.54      420
# dqn_e90_20251014_120000      90     72.1    1.3     0.52      380
# ...

# 2. Rank checkpoints by score
tli checkpoint rank --model DQN --output dqn_ranked.json

# 3. Select top 5 for ensemble
tli checkpoint select --model DQN --top 5

# 4. Deploy ensemble
tli ensemble deploy \
  --models DQN:5,PPO:5 \
  --weighting score-based \
  --config production

# 5. Verify ensemble health
tli ensemble health
# Output:
# Status: Healthy
# Total models: 10 (5 DQN, 5 PPO)
# Ensemble Sharpe: 1.9
# Ensemble Win Rate: 0.57
# Last signal: 2025-10-14 12:00:00 UTC

---

Scoring Examples

Example 1: Elite Checkpoint (Score: 92.5)

Sharpe Ratio: 2.3 → 85 points (world-class)
Win Rate: 0.58 → 92 points (elite)
Trade Frequency: 3.8% → 95 points (optimal)
Max Drawdown: 8% → 98 points (excellent)

Composite Score: 0.4×85 + 0.3×92 + 0.2×95 + 0.1×98 = 92.5

Example 2: Good Checkpoint (Score: 75.0)

Sharpe Ratio: 1.5 → 65 points (good)
Win Rate: 0.54 → 74 points (above threshold)
Trade Frequency: 4.2% → 92 points (optimal)
Max Drawdown: 15% → 70 points (acceptable)

Composite Score: 0.4×65 + 0.3×74 + 0.2×92 + 0.1×70 = 75.0

Example 3: Borderline Checkpoint (Score: 58.2)

Sharpe Ratio: 0.8 → 40 points (below target)
Win Rate: 0.51 → 53 points (barely profitable)
Trade Frequency: 6.5% → 85 points (overtrading penalty)
Max Drawdown: 22% → 36 points (high risk)

Composite Score: 0.4×40 + 0.3×53 + 0.2×85 + 0.1×36 = 58.2

Example 4: Failed Checkpoint (Score: 25.0)

Sharpe Ratio: -0.2 → 0 points (negative)
Win Rate: 0.45 → 45 points (losing strategy)
Trade Frequency: 0.05% → 1.25 points (severe undertrading)
Max Drawdown: 35% → 10 points (catastrophic)

Composite Score: 0.4×0 + 0.3×45 + 0.2×1.25 + 0.1×10 = 14.75

---

Statistical Significance Filter

Rule: Reject checkpoints with <30 trades OR <30 days backtest period

fn is_statistically_significant(metrics: &PerformanceAnalytics) -> bool {
    metrics.trade_stats.total_trades >= 30 &&
    metrics.time_analysis.total_days >= 30
}

Why 30?

• 30 trades: Minimum sample size for 50% win rate hypothesis test (power=0.8, α=0.05)
• 30 days: Sufficient for monthly seasonality, excludes weekly noise

Example:

• Checkpoint with 450 trades, 90 days → ✅ PASS
• Checkpoint with 15 trades, 90 days → ❌ REJECT (insufficient trades)
• Checkpoint with 100 trades, 10 days → ❌ REJECT (insufficient period)

---

Performance Targets

Latency

• Scoring: <10ms per checkpoint (pure calculation)
• Backtest: <5 seconds per checkpoint (20K bars, GPU-accelerated)
• Ranking: <60 seconds for 10 checkpoints (parallel backtesting)

Accuracy

• Score Correlation: >0.8 with manual expert ranking
• False Positive Rate: <10% (bad checkpoints with high scores)
• False Negative Rate: <5% (good checkpoints with low scores)

Production Ensemble

• Sharpe Ratio: >1.5 (vs. 1.0 baseline)
• Win Rate: >52% (vs. 50% baseline)
• Trade Frequency: 2-5% (vs. 0.01% baseline)
• Max Drawdown: <15% (vs. 20% baseline)

---

Testing Checklist

# Unit tests (scoring functions)
cargo test -p ml normalize_sharpe_ratio
cargo test -p ml normalize_win_rate
cargo test -p ml calculate_checkpoint_score

# Integration tests (end-to-end ranking)
cargo test -p ml checkpoint_selection_end_to_end

# Benchmarks (performance)
cargo bench -p ml checkpoint_ranking_latency

# Validation (with real DBN data)
cargo run -p ml --example validate_checkpoint_selection

# Production readiness
cargo test --workspace --release
cargo clippy --workspace -- -D warnings

---

Troubleshooting

Issue: Low Scores Across All Checkpoints

Symptom: All scores <50, even for visually "good" checkpoints

Diagnosis:

1. Check trade frequency: Is it <0.1%? (Severe undertrading)
2. Check Sharpe ratio: Is it <0.5? (Low profitability)
3. Check win rate: Is it <50%? (Losing strategy)

Fix:

• Retrain with more aggressive trading signals
• Reduce reward sparsity in DQN/PPO
• Tune epsilon decay for more exploration

Issue: Backtest Hangs/Timeout

Symptom: rank_checkpoints() takes >5 minutes

Diagnosis:

1. Check DBN data size: >100K bars?
2. Check GPU utilization: <50%?
3. Check parallelization: Single-threaded?

Fix:

// Enable parallel backtesting
let selector = CheckpointSelector::new_with_workers(4);
selector.rank_checkpoints_parallel(ModelType::DQN).await?;

Issue: Statistical Significance Filter Rejects All

Symptom: 0 checkpoints pass the filter

Diagnosis:

1. Check trade counts: All <30 trades?
2. Check backtest period: <30 days?

Fix:

• Extend backtest period (download 90-day DBN data)
• Lower threshold temporarily: min_trades: 10 (for testing only)

---

Next Steps

1. Review Full Spec: Read CHECKPOINT_SELECTION_FRAMEWORK.md (10K words)
2. Implement Phase 1: Start with scoring functions (selection.rs)
3. Test with Synthetic Data: Validate scoring logic before backtest integration
4. Integrate with Backtesting: Connect to existing metrics.rs infrastructure
5. Deploy to Production: TLI commands + weekly cron job

---

Status: Ready for Implementation Estimated Effort: 7 weeks (1 developer) Priority: HIGH (blocks production ML deployment) Blockers: None (all dependencies in place)