foxhunt/BACKTEST_REPORT_QUICK_REF.md

# Backtesting Report Generator - Quick Reference

**Status**: ✅ PRODUCTION READY  
**Module**: `ml/src/backtesting/report.rs`  
**Example**: `ml/examples/generate_backtest_report.rs`  
**Last Updated**: 2025-11-04

---

## Overview

Automated markdown report generation for DQN model backtesting with deployment recommendations based on production criteria.

## Features

- ✅ **Production Criteria Validation**: Automated APPROVE/REJECT/REVIEW decisions
- ✅ **Baseline Comparison**: Side-by-side metrics vs Trial #35 or custom baseline
- ✅ **Comprehensive Metrics**: Returns, Sharpe, drawdown, win rate, alpha, trades
- ✅ **Professional Markdown**: Ready for documentation and CI/CD integration
- ✅ **Unit Tested**: 3/3 tests passing

---

## Quick Start

### Generate Report (Default Example)

```bash
cargo run -p ml --example generate_backtest_report --release
```

**Output**: `backtest_comparison_report.md`

### Generate with Custom Metrics

```bash
cargo run -p ml --example generate_backtest_report --release -- \
  --new-model "DQN-Wave3-Entropy" \
  --total-return 18.5 \
  --sharpe 2.3 \
  --drawdown 12.5 \
  --win-rate 0.58 \
  --alpha 3.2 \
  --total-trades 150 \
  --avg-trade-return 0.123 \
  --output-file my_model_report.md
```

### Generate Multiple Examples (Verbose)

```bash
cargo run -p ml --example generate_backtest_report --release --verbose
```

**Generates 3 Reports**:
- `backtest_comparison_report.md` (Strong model - APPROVE)
- `backtest_marginal_example.md` (Marginal model - REVIEW)
- `backtest_weak_example.md` (Weak model - REJECT)

---

## Production Criteria

A model receives **APPROVE** if it passes ≥4 of these criteria:

| Criterion | Target | Status |
|-----------|--------|--------|
| Total Return | >0% | ✅ Profitable |
| Sharpe Ratio | >1.5 | ✅ Risk-adjusted |
| Max Drawdown | <20% | ✅ Acceptable risk |
| Win Rate | >50% | ✅ Consistent |
| Alpha vs B&H | >0% | ✅ Outperforms |

**Recommendation Thresholds**:
- **4-5 criteria passed**: ✅ APPROVE - Ready for Production
- **2-3 criteria passed**: ⚠️ REVIEW - Marginal Performance
- **0-1 criteria passed**: ❌ REJECT - Not Production Ready

---

## Usage in Code

### Create Report Programmatically

```rust
use ml::backtesting::report::{BacktestReport, PerformanceMetrics};

let new_results = PerformanceMetrics {
    total_return_pct: 18.5,
    sharpe_ratio: 2.5,
    max_drawdown_pct: 10.2,
    win_rate: 0.62,
    alpha: 4.5,
    total_trades: 150,
    avg_trade_return_pct: 0.123,
};

let baseline = Some(PerformanceMetrics {
    total_return_pct: 12.1,
    sharpe_ratio: 1.8,
    max_drawdown_pct: 18.3,
    win_rate: 0.52,
    alpha: 1.5,
    total_trades: 138,
    avg_trade_return_pct: 0.088,
});

let report = BacktestReport {
    model_name: "DQN-MyModel".to_string(),
    baseline_name: "DQN-Trial35".to_string(),
    new_results,
    baseline_results: baseline,
};

// Generate markdown
let markdown = report.generate_markdown();
std::fs::write("my_report.md", markdown)?;

// Get deployment recommendation
let recommendation = report.get_recommendation();
println!("Status: {}", recommendation.status);
```

### Check Recommendation Programmatically

```rust
let recommendation = report.get_recommendation();

match recommendation.status.as_str() {
    s if s.contains("APPROVE") => {
        println!("✅ Deploy to production");
    }
    s if s.contains("REVIEW") => {
        println!("⚠️  Manual review required");
    }
    s if s.contains("REJECT") => {
        println!("❌ Do NOT deploy - retrain needed");
    }
    _ => unreachable!()
}
```

---

## Report Sections

Each generated report contains:

1. **Header**: Model name, baseline, timestamp
2. **Performance Summary**: 5 production criteria with targets and status
3. **Baseline Comparison**: Side-by-side metrics with change arrows
4. **Trade Statistics**: Total trades, avg return, win rate
5. **Deployment Recommendation**: APPROVE/REVIEW/REJECT with reasoning
6. **Production Criteria Checklist**: Detailed pass/fail for each criterion

---

## Example Reports

### Strong Model (APPROVE)

```markdown
## Performance Summary

| Metric | Value | Target | Status |
|--------|-------|--------|--------|
| Total Return | 18.50% | >0% | ✅ |
| Sharpe Ratio | 2.50 | >1.5 | ✅ |
| Max Drawdown | 10.20% | <20% | ✅ |
| Win Rate | 62.0% | >50% | ✅ |
| Alpha vs B&H | 4.50% | >0% | ✅ |

**Status**: ✅ APPROVE - Ready for Production

Model passes 5/5 production criteria. Strong performance with 18.50% return...
```

### Marginal Model (REVIEW)

```markdown
**Status**: ⚠️ REVIEW - Marginal Performance

Model passes 2/5 production criteria. Performance is marginal and requires careful review.

**Concerns**:
- ❌ Low Sharpe ratio (1.20 < 1.5)
- ❌ Excessive drawdown (22.80% > 20%)
- ❌ Poor win rate (48.0% < 50%)
```

### Weak Model (REJECT)

```markdown
**Status**: ❌ REJECT - Not Production Ready

Model only passes 0/5 production criteria. Performance is insufficient for production deployment.

**Critical Issues**:
- ❌ Negative total return (-3.20%)
- ❌ Low Sharpe ratio (0.60 < 1.5)
- ❌ Excessive drawdown (35.40% > 20%)
- ❌ Poor win rate (38.0% < 50%)
- ❌ Negative alpha (-2.10%)
```

---

## Integration with Backtesting Pipeline

### From DQN Evaluation Results

```bash
# Step 1: Run DQN evaluation
cargo run -p ml --example evaluate_dqn --release --features cuda -- \
  --output-json /tmp/dqn_eval_results.json

# Step 2: Parse JSON and generate report (future enhancement)
# TODO: Add JSON parsing to generate_backtest_report
```

### Manual Entry (Current Method)

```bash
# Extract metrics from evaluation output and pass via CLI
cargo run -p ml --example generate_backtest_report --release -- \
  --new-model "DQN-Wave3" \
  --total-return 15.2 \
  --sharpe 2.1 \
  --drawdown 14.3 \
  --win-rate 0.56 \
  --alpha 2.8
```

---

## Trial #35 Baseline Metrics

**Reference Model**: DQN-Trial35-Baseline (Hyperopt best model)

| Metric | Value |
|--------|-------|
| Total Return | 12.1% |
| Sharpe Ratio | 1.8 |
| Max Drawdown | 18.3% |
| Win Rate | 52.0% |
| Alpha | 1.5% |
| Total Trades | 138 |
| Avg Trade Return | 0.088% |

**Note**: Update these values in `ml/examples/generate_backtest_report.rs::get_trial35_baseline()` when actual Trial #35 backtesting results are available.

---

## Testing

### Run Unit Tests

```bash
cargo test -p ml --lib backtesting::report --release
```

**Expected Output**:
```
running 3 tests
test backtesting::report::tests::test_report_generation_reject ... ok
test backtesting::report::tests::test_baseline_comparison ... ok
test backtesting::report::tests::test_report_generation_approve ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured
```

---

## Files Created

| File | Description |
|------|-------------|
| `ml/src/backtesting/report.rs` | Core report generation module |
| `ml/examples/generate_backtest_report.rs` | CLI example for report generation |
| `backtest_comparison_report.md` | Default output file |
| `BACKTEST_REPORT_QUICK_REF.md` | This file |

---

## Future Enhancements

1. **JSON Input Support**: Parse evaluation results directly from JSON files
2. **CI/CD Integration**: Auto-generate reports in GitLab pipeline
3. **Multi-Model Comparison**: Compare >2 models in a single report
4. **Equity Curve Plotting**: Generate performance charts (requires plotting library)
5. **Risk Metrics**: Add VaR, CVaR, Calmar ratio from backtesting/metrics.rs
6. **HTML Export**: Generate interactive HTML reports

---

## Troubleshooting

### Issue: Report shows incorrect baseline

**Solution**: Verify Trial #35 metrics in `get_trial35_baseline()` function.

### Issue: Recommendation seems wrong

**Solution**: Check production criteria thresholds - they may need adjustment based on strategy type.

### Issue: Floating point precision issues

**Solution**: All percentages formatted to 2 decimal places. Exact comparisons may fail due to rounding.

---

## References

- **Backtesting Metrics**: `/home/jgrusewski/Work/foxhunt/backtesting/src/metrics.rs`
- **DQN Evaluation**: `/home/jgrusewski/Work/foxhunt/ml/examples/evaluate_dqn.rs`
- **Production Criteria**: Based on CLAUDE.md Wave D backtest targets
- **Trial #35**: Placeholder metrics (update when actual results available)

---

*Generated: 2025-11-04*  
*Author: Claude Code Agent*  
*Status: Ready for Production Use*