foxhunt/docs/CHECKPOINT_SELECTION_QUICKSTART.md

# Checkpoint Selection Framework - Quickstart Guide

**Last Updated**: 2025-10-14
**Status**: Implementation Ready
**Related**: `CHECKPOINT_SELECTION_FRAMEWORK.md` (full specification)

---

## TL;DR

**Problem**: After training 100 epochs, we have 10+ checkpoints per model. Which ones are production-ready?

**Solution**: Systematic scoring algorithm that evaluates:
1. **Sharpe Ratio** (40% weight) - Risk-adjusted profitability
2. **Win Rate** (30% weight) - Prediction accuracy
3. **Trade Frequency** (20% weight) - Activity level (target: 2-5% of bars)
4. **Max Drawdown** (10% weight) - Risk management

**Output**: Ranked list of checkpoints (score 0-100) → Top 5 DQN + Top 5 PPO → Production ensemble

---

## Core Scoring Formula

```rust
score = 0.4 × sharpe_score +
        0.3 × win_rate_score +
        0.2 × trade_freq_score +
        0.1 × drawdown_score

// Example: Checkpoint with Sharpe=1.5, WinRate=0.55, TradePct=3.5%, Drawdown=12%
// → score = 0.4×65 + 0.3×80 + 0.2×90 + 0.1×68 = 77.8 (GOOD)
```

### Target Ranges

| Metric | Threshold | Target | Elite | Weight |
|--------|-----------|--------|-------|--------|
| Sharpe Ratio | >0.5 | >1.0 | >2.0 | 40% |
| Win Rate | >50% | >52% | >55% | 30% |
| Trade Frequency | 0.5-10% | 2-5% | 3-4% | 20% |
| Max Drawdown | <30% | <20% | <10% | 10% |

---

## Implementation Roadmap

### Phase 1: Core Scoring (Week 1-2)
```bash
# New file: ml/src/checkpoint/selection.rs
cargo new-module ml/src/checkpoint/selection

# Implement:
- normalize_sharpe_ratio()
- normalize_win_rate()
- normalize_trade_frequency()
- normalize_drawdown()
- calculate_checkpoint_score()
- is_statistically_significant()

# Tests:
cargo test -p ml checkpoint::selection
```

**Key Functions**:
```rust
pub fn calculate_checkpoint_score(metrics: &PerformanceAnalytics) -> f64 {
    let sharpe = normalize_sharpe_ratio(metrics.risk.sharpe_ratio.to_f64());
    let win_rate = normalize_win_rate(metrics.trade_stats.win_rate.to_f64());
    let trade_freq = normalize_trade_frequency(
        metrics.trade_stats.total_trades,
        metrics.time_analysis.total_days
    );
    let drawdown = normalize_drawdown(metrics.drawdown.max_drawdown.to_f64());

    0.4 * sharpe + 0.3 * win_rate + 0.2 * trade_freq + 0.1 * drawdown
}
```

### Phase 2: Backtest Integration (Week 3)
```rust
pub struct CheckpointSelector {
    backtesting_engine: Arc<BacktestingEngine>,
    checkpoint_manager: Arc<CheckpointManager>,
    config: SelectionConfig,
}

impl CheckpointSelector {
    pub async fn rank_checkpoints(
        &self,
        model_type: ModelType,
    ) -> Result<Vec<RankedCheckpoint>, MLError> {
        // 1. List all checkpoints from MinIO/S3
        let checkpoints = self.checkpoint_manager
            .list_checkpoints(model_type, "")
            .await;

        // 2. Backtest each checkpoint (parallel)
        let mut evaluated = Vec::new();
        for checkpoint in checkpoints {
            let metrics = self.backtest_checkpoint(&checkpoint).await?;

            // 3. Filter: min 30 trades, 30 days
            if !is_statistically_significant(&metrics) {
                continue;
            }

            // 4. Score
            let score = calculate_checkpoint_score(&metrics);
            evaluated.push((checkpoint, metrics, score));
        }

        // 5. Sort by score (descending)
        evaluated.sort_by(|a, b| b.2.partial_cmp(&a.2).unwrap());

        // 6. Return ranked
        Ok(self.to_ranked_checkpoints(evaluated))
    }
}
```

### Phase 3: Ensemble Composition (Week 4)
```rust
// Update ml/src/ensemble/model.rs
impl EnsembleModel {
    pub async fn deploy_top_checkpoints(
        &self,
        selector: &CheckpointSelector,
    ) -> Result<(), MLError> {
        // Top 5 DQN
        let dqn_top5 = selector.select_top_k(ModelType::DQN, 5).await?;
        for ranked in dqn_top5 {
            self.register_model(&ranked.checkpoint_id, ...);
        }

        // Top 5 PPO
        let ppo_top5 = selector.select_top_k(ModelType::PPO, 5).await?;
        for ranked in ppo_top5 {
            self.register_model(&ranked.checkpoint_id, ...);
        }

        // Set weights (score-based)
        let weights = calculate_score_based_weights(&all_checkpoints);
        self.set_model_weights(weights)?;

        Ok(())
    }
}
```

### Phase 4: TLI Commands (Week 5)
```bash
# New commands in tli/src/main.rs
tli checkpoint rank --model DQN
tli checkpoint select --model DQN --top 5
tli checkpoint list --sort score
tli ensemble deploy --models DQN:5,PPO:5
tli ensemble health
```

### Phase 5: Production Deployment (Week 6-7)
```bash
# Weekly cron job (Sunday 00:00 UTC)
tli ensemble reevaluate

# CI/CD integration (.github/workflows/checkpoint_selection.yml)
- Download DBN data
- Run checkpoint ranking
- Update production ensemble
- Validate health
```

---

## Example Usage

### Scenario: Select Best DQN Checkpoint

```bash
# 1. List all DQN checkpoints
tli checkpoint list --model DQN
# Output:
# checkpoint_id                 epoch  score   sharpe  win_rate  trades
# dqn_e80_20251014_120000      80     85.4    1.8     0.56      450
# dqn_e70_20251014_120000      70     78.2    1.5     0.54      420
# dqn_e90_20251014_120000      90     72.1    1.3     0.52      380
# ...

# 2. Rank checkpoints by score
tli checkpoint rank --model DQN --output dqn_ranked.json

# 3. Select top 5 for ensemble
tli checkpoint select --model DQN --top 5

# 4. Deploy ensemble
tli ensemble deploy \
  --models DQN:5,PPO:5 \
  --weighting score-based \
  --config production

# 5. Verify ensemble health
tli ensemble health
# Output:
# Status: Healthy
# Total models: 10 (5 DQN, 5 PPO)
# Ensemble Sharpe: 1.9
# Ensemble Win Rate: 0.57
# Last signal: 2025-10-14 12:00:00 UTC
```

---

## Scoring Examples

### Example 1: Elite Checkpoint (Score: 92.5)
```
Sharpe Ratio: 2.3 → 85 points (world-class)
Win Rate: 0.58 → 92 points (elite)
Trade Frequency: 3.8% → 95 points (optimal)
Max Drawdown: 8% → 98 points (excellent)

Composite Score: 0.4×85 + 0.3×92 + 0.2×95 + 0.1×98 = 92.5
```

### Example 2: Good Checkpoint (Score: 75.0)
```
Sharpe Ratio: 1.5 → 65 points (good)
Win Rate: 0.54 → 74 points (above threshold)
Trade Frequency: 4.2% → 92 points (optimal)
Max Drawdown: 15% → 70 points (acceptable)

Composite Score: 0.4×65 + 0.3×74 + 0.2×92 + 0.1×70 = 75.0
```

### Example 3: Borderline Checkpoint (Score: 58.2)
```
Sharpe Ratio: 0.8 → 40 points (below target)
Win Rate: 0.51 → 53 points (barely profitable)
Trade Frequency: 6.5% → 85 points (overtrading penalty)
Max Drawdown: 22% → 36 points (high risk)

Composite Score: 0.4×40 + 0.3×53 + 0.2×85 + 0.1×36 = 58.2
```

### Example 4: Failed Checkpoint (Score: 25.0)
```
Sharpe Ratio: -0.2 → 0 points (negative)
Win Rate: 0.45 → 45 points (losing strategy)
Trade Frequency: 0.05% → 1.25 points (severe undertrading)
Max Drawdown: 35% → 10 points (catastrophic)

Composite Score: 0.4×0 + 0.3×45 + 0.2×1.25 + 0.1×10 = 14.75
```

---

## Statistical Significance Filter

**Rule**: Reject checkpoints with <30 trades OR <30 days backtest period

```rust
fn is_statistically_significant(metrics: &PerformanceAnalytics) -> bool {
    metrics.trade_stats.total_trades >= 30 &&
    metrics.time_analysis.total_days >= 30
}
```

**Why 30?**
- 30 trades: Minimum sample size for 50% win rate hypothesis test (power=0.8, α=0.05)
- 30 days: Sufficient for monthly seasonality, excludes weekly noise

**Example**:
- Checkpoint with 450 trades, 90 days → ✅ PASS
- Checkpoint with 15 trades, 90 days → ❌ REJECT (insufficient trades)
- Checkpoint with 100 trades, 10 days → ❌ REJECT (insufficient period)

---

## Performance Targets

### Latency
- **Scoring**: <10ms per checkpoint (pure calculation)
- **Backtest**: <5 seconds per checkpoint (20K bars, GPU-accelerated)
- **Ranking**: <60 seconds for 10 checkpoints (parallel backtesting)

### Accuracy
- **Score Correlation**: >0.8 with manual expert ranking
- **False Positive Rate**: <10% (bad checkpoints with high scores)
- **False Negative Rate**: <5% (good checkpoints with low scores)

### Production Ensemble
- **Sharpe Ratio**: >1.5 (vs. 1.0 baseline)
- **Win Rate**: >52% (vs. 50% baseline)
- **Trade Frequency**: 2-5% (vs. 0.01% baseline)
- **Max Drawdown**: <15% (vs. 20% baseline)

---

## Testing Checklist

```bash
# Unit tests (scoring functions)
cargo test -p ml normalize_sharpe_ratio
cargo test -p ml normalize_win_rate
cargo test -p ml calculate_checkpoint_score

# Integration tests (end-to-end ranking)
cargo test -p ml checkpoint_selection_end_to_end

# Benchmarks (performance)
cargo bench -p ml checkpoint_ranking_latency

# Validation (with real DBN data)
cargo run -p ml --example validate_checkpoint_selection

# Production readiness
cargo test --workspace --release
cargo clippy --workspace -- -D warnings
```

---

## Troubleshooting

### Issue: Low Scores Across All Checkpoints
**Symptom**: All scores <50, even for visually "good" checkpoints

**Diagnosis**:
1. Check trade frequency: Is it <0.1%? (Severe undertrading)
2. Check Sharpe ratio: Is it <0.5? (Low profitability)
3. Check win rate: Is it <50%? (Losing strategy)

**Fix**:
- Retrain with more aggressive trading signals
- Reduce reward sparsity in DQN/PPO
- Tune epsilon decay for more exploration

### Issue: Backtest Hangs/Timeout
**Symptom**: `rank_checkpoints()` takes >5 minutes

**Diagnosis**:
1. Check DBN data size: >100K bars?
2. Check GPU utilization: <50%?
3. Check parallelization: Single-threaded?

**Fix**:
```rust
// Enable parallel backtesting
let selector = CheckpointSelector::new_with_workers(4);
selector.rank_checkpoints_parallel(ModelType::DQN).await?;
```

### Issue: Statistical Significance Filter Rejects All
**Symptom**: 0 checkpoints pass the filter

**Diagnosis**:
1. Check trade counts: All <30 trades?
2. Check backtest period: <30 days?

**Fix**:
- Extend backtest period (download 90-day DBN data)
- Lower threshold temporarily: `min_trades: 10` (for testing only)

---

## Next Steps

1. **Review Full Spec**: Read `CHECKPOINT_SELECTION_FRAMEWORK.md` (10K words)
2. **Implement Phase 1**: Start with scoring functions (`selection.rs`)
3. **Test with Synthetic Data**: Validate scoring logic before backtest integration
4. **Integrate with Backtesting**: Connect to existing `metrics.rs` infrastructure
5. **Deploy to Production**: TLI commands + weekly cron job

---

**Status**: Ready for Implementation
**Estimated Effort**: 7 weeks (1 developer)
**Priority**: HIGH (blocks production ML deployment)
**Blockers**: None (all dependencies in place)