gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
ee9f8f6fec181e080e4e91ef65ab9309e81672e9
foxhunt/docs/plans/2026-02-20-validation-stack-implementation.md
jgrusewski c3b5e124f0 chore: update .gitignore and add design plan docs 
Ignore ML checkpoints, trained model safetensors, stray ml/ml/ dir,
and .claude/worktrees/. Clean up duplicate hive-mind-prompt entries.
Add 17 design/implementation plan docs from 2026-02-20 to 2026-02-22.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 00:30:15 +01:00

2262 lines
72 KiB
Markdown
Raw Blame History

# Statistical Validation Stack Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Build an algorithm-agnostic statistical validation harness (walk-forward, DSR, PBO, permutation tests, per-regime analysis) that proves whether trading strategies are statistically real or overfit noise.
**Architecture:** Layered pipeline in `ml/src/validation/` — five modules (walk_forward, statistical, regime_analysis, harness) composed by an orchestrator, all behind a `ValidatableStrategy` trait. Existing `validation.rs` becomes `validation/financial.rs` inside the new directory module.
**Tech Stack:** Rust, Candle ML, chrono (timestamps), rand/rand_chacha (permutation RNG). No new dependencies needed — all already in `ml/Cargo.toml`.
**Safety gate for every task:** `SQLX_OFFLINE=true cargo check --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml` must pass.
---
### Task 1: Convert validation.rs to directory module
Convert the existing `ml/src/validation.rs` (financial validation) into a directory module so we can add new submodules alongside it.
**Files:**
- Move: `ml/src/validation.rs` → `ml/src/validation/financial.rs`
- Create: `ml/src/validation/mod.rs`
- Delete orphan: `ml/src/validation/numerical_tests.rs` (unreferenced file in existing validation dir)
**Step 1: Move validation.rs to validation/financial.rs**
Move the existing file. The `validation/` directory already exists (contains orphan `numerical_tests.rs`).
```bash
mv /home/jgrusewski/Work/foxhunt/ml/src/validation.rs /home/jgrusewski/Work/foxhunt/ml/src/validation/financial.rs
```
**Step 2: Delete the orphan numerical_tests.rs**
This file is unreferenced — it imports types that don't exist in the current module tree (`MLModel`, `ModelType`, etc.) and was never compiled.
```bash
rm /home/jgrusewski/Work/foxhunt/ml/src/validation/numerical_tests.rs
```
**Step 3: Create validation/mod.rs**
```rust
//! Statistical and Financial Validation Module
//!
//! Provides two categories of validation:
//! - Financial validation (price/volume/quantity sanity checks)
//! - Statistical validation (walk-forward, DSR, PBO, permutation tests)
pub mod financial;
// Re-export financial validation for backward compatibility
pub use financial::{
validate_model_basic, validate_model_comprehensive, validate_type_conversions,
FinancialValidationResult, ValidationResult,
};
```
**Step 4: Verify compilation**
Run: `SQLX_OFFLINE=true cargo check --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml`
Expected: Compiles clean (0 errors)
**Step 5: Run existing tests**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --lib 2>&1 | tail -5`
Expected: All tests pass (same count as before: 1823+)
**Step 6: Commit**
```bash
git add ml/src/validation/mod.rs ml/src/validation/financial.rs
git add ml/src/validation.rs # stages deletion
git add ml/src/validation/numerical_tests.rs # stages deletion
git commit -m "refactor(validation): convert to directory module for validation stack"
```
---
### Task 2: TimeSeriesData struct + ValidatableStrategy trait
Create the core data type and trait that all validation modules depend on.
**Files:**
- Modify: `ml/src/validation/mod.rs`
- Create: `ml/src/validation/types.rs`
- Test: inline `#[cfg(test)]` in `types.rs`
**Step 1: Write the failing test**
Create `ml/src/validation/types.rs` with tests that reference structs/traits that don't exist yet:
```rust
//! Core types for the statistical validation stack.
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use crate::MLError;
/// Time-series data for validation — features, prices, and returns aligned by timestamp.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TimeSeriesData {
/// Timestamps for each bar (must be monotonically increasing)
pub timestamps: Vec<DateTime<Utc>>,
/// Feature matrix: `features[i]` is the feature vector for bar `i`
pub features: Vec<Vec<f32>>,
/// Close prices for PnL computation
pub prices: Vec<f64>,
/// Log returns derived from prices: `returns[i] = ln(prices[i+1] / prices[i])`
/// Length is `prices.len() - 1`
pub returns: Vec<f64>,
}
impl TimeSeriesData {
/// Create TimeSeriesData from raw components, computing returns automatically.
///
/// # Errors
/// Returns `MLError::InvalidInput` if lengths don't match or prices has < 2 elements.
pub fn new(
timestamps: Vec<DateTime<Utc>>,
features: Vec<Vec<f32>>,
prices: Vec<f64>,
) -> Result<Self, MLError> {
if timestamps.len() != features.len() || timestamps.len() != prices.len() {
return Err(MLError::InvalidInput(format!(
"Length mismatch: timestamps={}, features={}, prices={}",
timestamps.len(),
features.len(),
prices.len()
)));
}
if prices.len() < 2 {
return Err(MLError::InvalidInput(
"Need at least 2 price points to compute returns".to_string(),
));
}
// Compute log returns
let returns: Vec<f64> = prices
.windows(2)
.map(|w| {
if w[0] > 0.0 {
(w[1] / w[0]).ln()
} else {
0.0
}
})
.collect();
Ok(Self {
timestamps,
features,
prices,
returns,
})
}
/// Number of bars in this dataset.
pub fn len(&self) -> usize {
self.timestamps.len()
}
/// Whether the dataset is empty.
pub fn is_empty(&self) -> bool {
self.timestamps.is_empty()
}
/// Slice this dataset to a given index range (inclusive start, exclusive end).
///
/// # Errors
/// Returns `MLError::InvalidInput` if range is out of bounds or too small.
pub fn slice(&self, start: usize, end: usize) -> Result<Self, MLError> {
if end > self.len() || start >= end {
return Err(MLError::InvalidInput(format!(
"Invalid slice range {}..{} for data of length {}",
start,
end,
self.len()
)));
}
if end - start < 2 {
return Err(MLError::InvalidInput(
"Slice must contain at least 2 bars".to_string(),
));
}
Self::new(
self.timestamps.get(start..end).unwrap_or_default().to_vec(),
self.features.get(start..end).unwrap_or_default().to_vec(),
self.prices.get(start..end).unwrap_or_default().to_vec(),
)
}
}
/// Trait for any model/strategy that can be validated via walk-forward analysis.
///
/// Implement this for each algorithm (DQN, PPO, TFT, etc.) to enable
/// statistical validation through the `ValidationHarness`.
pub trait ValidatableStrategy: Send {
/// Train the model on the given time-series data.
fn train(&mut self, data: &TimeSeriesData) -> Result<(), MLError>;
/// Evaluate the model on test data, returning daily PnL returns.
///
/// The returned vector should have one entry per bar in `data`
/// representing the strategy's return for that bar.
fn evaluate(&self, data: &TimeSeriesData) -> Result<Vec<f64>, MLError>;
/// Human-readable name for reporting.
fn name(&self) -> &str;
/// Reset model weights for fresh fold training.
fn reset(&mut self) -> Result<(), MLError>;
}
#[cfg(test)]
mod tests {
use super::*;
use chrono::{Duration, Utc};
fn make_test_data(n: usize) -> TimeSeriesData {
let start = Utc::now();
let timestamps: Vec<_> = (0..n)
.map(|i| start + Duration::hours(i as i64))
.collect();
let features: Vec<Vec<f32>> = (0..n).map(|i| vec![i as f32; 4]).collect();
let prices: Vec<f64> = (0..n).map(|i| 100.0 + i as f64 * 0.5).collect();
TimeSeriesData::new(timestamps, features, prices).expect("valid test data")
}
#[test]
fn test_time_series_data_new_computes_returns() {
let data = make_test_data(10);
assert_eq!(data.len(), 10);
assert_eq!(data.returns.len(), 9); // n-1 returns
// First return: ln(100.5 / 100.0)
let expected = (100.5_f64 / 100.0).ln();
assert!((data.returns[0] - expected).abs() < 1e-10);
}
#[test]
fn test_time_series_data_length_mismatch_errors() {
let start = Utc::now();
let ts = vec![start, start + Duration::hours(1)];
let features = vec![vec![1.0f32]; 3]; // wrong length
let prices = vec![100.0, 101.0];
let result = TimeSeriesData::new(ts, features, prices);
assert!(result.is_err());
}
#[test]
fn test_time_series_data_too_short_errors() {
let start = Utc::now();
let result = TimeSeriesData::new(vec![start], vec![vec![1.0]], vec![100.0]);
assert!(result.is_err());
}
#[test]
fn test_slice_valid_range() {
let data = make_test_data(20);
let sliced = data.slice(5, 15).expect("valid slice");
assert_eq!(sliced.len(), 10);
assert_eq!(sliced.returns.len(), 9);
}
#[test]
fn test_slice_out_of_bounds_errors() {
let data = make_test_data(10);
assert!(data.slice(0, 20).is_err());
assert!(data.slice(8, 8).is_err());
}
}
```
**Step 2: Register the module in mod.rs**
Add to `ml/src/validation/mod.rs`:
```rust
pub mod types;
pub use types::{TimeSeriesData, ValidatableStrategy};
```
**Step 3: Run test to verify it passes**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --lib validation::types -- --nocapture 2>&1 | tail -10`
Expected: 5 tests pass
**Step 4: Verify full compilation**
Run: `SQLX_OFFLINE=true cargo check --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml`
Expected: Compiles clean
**Step 5: Commit**
```bash
git add ml/src/validation/types.rs ml/src/validation/mod.rs
git commit -m "feat(validation): add TimeSeriesData struct and ValidatableStrategy trait"
```
---
### Task 3: Walk-forward splitter
Implement the walk-forward cross-validation splitter with embargo periods. This replaces the stubbed `walk_forward_split()` that currently returns `vec![]`.
**Files:**
- Create: `ml/src/validation/walk_forward.rs`
- Modify: `ml/src/validation/mod.rs` (add module declaration)
**Step 1: Write walk_forward.rs with tests first, then implementation**
```rust
//! Walk-forward cross-validation splitter with embargo periods.
//!
//! Produces non-overlapping folds that maintain strict temporal ordering:
//! `[train] [embargo] [test] → step → [train] [embargo] [test] → ...`
//!
//! The embargo gap prevents information leakage from features with lookback windows
//! (e.g., a 20-bar moving average would need embargo_bars >= 20).
use std::ops::Range;
use serde::{Deserialize, Serialize};
/// Configuration for walk-forward cross-validation.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct WalkForwardConfig {
/// Number of bars in each training window.
pub train_bars: usize,
/// Number of bars in each test window.
pub test_bars: usize,
/// Number of bars in the embargo gap between train and test.
/// Must be >= the maximum lookback window of any feature.
pub embargo_bars: usize,
/// How far to advance the window each fold (in bars).
pub step_bars: usize,
/// Minimum number of training bars required (folds with fewer are skipped).
pub min_train_samples: usize,
}
impl Default for WalkForwardConfig {
fn default() -> Self {
Self {
train_bars: 252 * 5, // ~5 years of daily bars
test_bars: 252, // ~1 year
embargo_bars: 20, // 20-bar lookback gap
step_bars: 126, // ~6 months step
min_train_samples: 252, // at least 1 year of training data
}
}
}
/// A single fold in walk-forward cross-validation.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Fold {
/// Zero-based fold index.
pub fold_index: usize,
/// Index range for training data (into the original TimeSeriesData).
pub train_range: Range<usize>,
/// Index range for the embargo gap (excluded from both train and test).
pub embargo_range: Range<usize>,
/// Index range for test data.
pub test_range: Range<usize>,
}
/// Generate walk-forward folds from a dataset of `num_bars` bars.
///
/// Returns an empty vec if the data is too short for even one valid fold.
pub fn walk_forward_split(num_bars: usize, config: &WalkForwardConfig) -> Vec<Fold> {
let mut folds = Vec::new();
let min_fold_size = config.train_bars + config.embargo_bars + config.test_bars;
if num_bars < min_fold_size || config.train_bars < config.min_train_samples {
return folds;
}
let mut fold_index = 0;
let mut train_start = 0;
loop {
let train_end = train_start + config.train_bars;
let embargo_end = train_end + config.embargo_bars;
let test_end = embargo_end + config.test_bars;
// Stop if test window exceeds data
if test_end > num_bars {
break;
}
// Only include fold if training window meets minimum
if config.train_bars >= config.min_train_samples {
folds.push(Fold {
fold_index,
train_range: train_start..train_end,
embargo_range: train_end..embargo_end,
test_range: embargo_end..test_end,
});
fold_index += 1;
}
train_start += config.step_bars;
}
folds
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_basic_walk_forward_split() {
let config = WalkForwardConfig {
train_bars: 100,
test_bars: 30,
embargo_bars: 5,
step_bars: 30,
min_train_samples: 50,
};
let folds = walk_forward_split(300, &config);
assert!(!folds.is_empty(), "Should produce at least one fold");
// Verify first fold
assert_eq!(folds[0].train_range, 0..100);
assert_eq!(folds[0].embargo_range, 100..105);
assert_eq!(folds[0].test_range, 105..135);
// Verify second fold steps forward
assert_eq!(folds[1].train_range, 30..130);
assert_eq!(folds[1].embargo_range, 130..135);
assert_eq!(folds[1].test_range, 135..165);
}
#[test]
fn test_no_overlap_between_train_and_test() {
let config = WalkForwardConfig {
train_bars: 50,
test_bars: 20,
embargo_bars: 10,
step_bars: 20,
min_train_samples: 10,
};
let folds = walk_forward_split(200, &config);
for fold in &folds {
// Train must end before embargo starts
assert_eq!(fold.train_range.end, fold.embargo_range.start);
// Embargo must end before test starts
assert_eq!(fold.embargo_range.end, fold.test_range.start);
// No overlap within a fold
assert!(fold.train_range.end <= fold.embargo_range.start);
assert!(fold.embargo_range.end <= fold.test_range.start);
}
}
#[test]
fn test_temporal_ordering_across_folds() {
let config = WalkForwardConfig {
train_bars: 50,
test_bars: 20,
embargo_bars: 5,
step_bars: 25,
min_train_samples: 10,
};
let folds = walk_forward_split(300, &config);
// Each fold's test start should be >= previous fold's test start
for window in folds.windows(2) {
assert!(
window[1].test_range.start >= window[0].test_range.start,
"Folds must advance in time"
);
}
}
#[test]
fn test_data_too_short_returns_empty() {
let config = WalkForwardConfig {
train_bars: 100,
test_bars: 50,
embargo_bars: 10,
step_bars: 50,
min_train_samples: 100,
};
// Need 160 bars minimum, only have 100
let folds = walk_forward_split(100, &config);
assert!(folds.is_empty());
}
#[test]
fn test_fold_indices_are_sequential() {
let config = WalkForwardConfig {
train_bars: 30,
test_bars: 10,
embargo_bars: 2,
step_bars: 10,
min_train_samples: 10,
};
let folds = walk_forward_split(100, &config);
for (i, fold) in folds.iter().enumerate() {
assert_eq!(fold.fold_index, i);
}
}
#[test]
fn test_test_range_within_bounds() {
let num_bars = 200;
let config = WalkForwardConfig {
train_bars: 50,
test_bars: 20,
embargo_bars: 5,
step_bars: 20,
min_train_samples: 10,
};
let folds = walk_forward_split(num_bars, &config);
for fold in &folds {
assert!(fold.test_range.end <= num_bars, "Test range exceeds data bounds");
}
}
#[test]
fn test_default_config() {
let config = WalkForwardConfig::default();
assert_eq!(config.embargo_bars, 20);
assert_eq!(config.train_bars, 252 * 5);
}
}
```
**Step 2: Register in mod.rs**
Add to `ml/src/validation/mod.rs`:
```rust
pub mod walk_forward;
pub use walk_forward::{walk_forward_split, Fold, WalkForwardConfig};
```
**Step 3: Run tests**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --lib validation::walk_forward -- --nocapture 2>&1 | tail -10`
Expected: 7 tests pass
**Step 4: Verify full compilation**
Run: `SQLX_OFFLINE=true cargo check --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml`
**Step 5: Commit**
```bash
git add ml/src/validation/walk_forward.rs ml/src/validation/mod.rs
git commit -m "feat(validation): implement walk-forward splitter with embargo periods"
```
---
### Task 4: Statistical math helpers (normal CDF, Sharpe, skewness, kurtosis)
Build the mathematical primitives needed by DSR, PBO, and permutation tests.
**Files:**
- Create: `ml/src/validation/statistical.rs`
- Modify: `ml/src/validation/mod.rs`
**Step 1: Write statistical.rs with helpers and their tests**
```rust
//! Statistical tests for strategy validation.
//!
//! Implements:
//! - Deflated Sharpe Ratio (Bailey & Lopez de Prado, 2014)
//! - Probability of Backtest Overfitting via CSCV (Bailey et al., 2017)
//! - Monte Carlo permutation tests
//!
//! All statistical math (normal CDF, Sharpe, moments) is self-contained
//! with no external dependencies beyond `rand`.
use rand::seq::SliceRandom;
use rand::SeedableRng;
use rand_chacha::ChaCha8Rng;
use serde::{Deserialize, Serialize};
// ══════════════════════════════════════════════════════
// Math helpers
// ══════════════════════════════════════════════════════
/// Standard normal CDF using Abramowitz & Stegun rational approximation (error < 7.5e-8).
pub fn normal_cdf(x: f64) -> f64 {
if x < -8.0 {
return 0.0;
}
if x > 8.0 {
return 1.0;
}
let a1 = 0.254829592;
let a2 = -0.284496736;
let a3 = 1.421413741;
let a4 = -1.453152027;
let a5 = 1.061405429;
let p = 0.3275911;
let sign = if x < 0.0 { -1.0 } else { 1.0 };
let x_abs = x.abs() / std::f64::consts::SQRT_2;
let t = 1.0 / (1.0 + p * x_abs);
let t2 = t * t;
let t3 = t2 * t;
let t4 = t3 * t;
let t5 = t4 * t;
let y = 1.0 - (a1 * t + a2 * t2 + a3 * t3 + a4 * t4 + a5 * t5) * (-x_abs * x_abs).exp();
0.5 * (1.0 + sign * y)
}
/// Inverse normal CDF (probit function) via rational approximation.
/// Accurate to ~1e-5 for p in (0.0003, 0.9997).
pub fn normal_ppf(p: f64) -> f64 {
if p <= 0.0 {
return f64::NEG_INFINITY;
}
if p >= 1.0 {
return f64::INFINITY;
}
if (p - 0.5).abs() < 1e-15 {
return 0.0;
}
// Rational approximation (Peter Acklam's algorithm)
let a = [
-3.969_683_028_665_376e1,
2.209_460_984_245_205e2,
-2.759_285_104_469_687e2,
1.383_577_518_672_690e2,
-3.066_479_806_614_716e1,
2.506_628_277_459_239,
];
let b = [
-5.447_609_879_822_406e1,
1.615_858_368_580_409e2,
-1.556_989_798_598_866e2,
6.680_131_188_771_972e1,
-1.328_068_155_288_572e1,
];
let c = [
-7.784_894_002_430_293e-3,
-3.223_964_580_411_365e-1,
-2.400_758_277_161_838,
-2.549_732_539_343_734,
4.374_664_141_464_968,
2.938_163_982_698_783,
];
let d = [
7.784_695_709_041_462e-3,
3.224_671_290_700_398e-1,
2.445_134_137_142_996,
3.754_408_661_907_416,
];
let p_low = 0.02425;
let p_high = 1.0 - p_low;
if p < p_low {
let q = (-2.0 * p.ln()).sqrt();
(((((c[0] * q + c[1]) * q + c[2]) * q + c[3]) * q + c[4]) * q + c[5])
/ ((((d[0] * q + d[1]) * q + d[2]) * q + d[3]) * q + 1.0)
} else if p <= p_high {
let q = p - 0.5;
let r = q * q;
(((((a[0] * r + a[1]) * r + a[2]) * r + a[3]) * r + a[4]) * r + a[5]) * q
/ (((((b[0] * r + b[1]) * r + b[2]) * r + b[3]) * r + b[4]) * r + 1.0)
} else {
let q = (-2.0 * (1.0 - p).ln()).sqrt();
-(((((c[0] * q + c[1]) * q + c[2]) * q + c[3]) * q + c[4]) * q + c[5])
/ ((((d[0] * q + d[1]) * q + d[2]) * q + d[3]) * q + 1.0)
}
}
/// Compute Sharpe ratio from a series of returns (not annualized).
pub fn sharpe_ratio(returns: &[f64]) -> f64 {
if returns.len() < 2 {
return 0.0;
}
let n = returns.len() as f64;
let mean = returns.iter().sum::<f64>() / n;
let variance = returns.iter().map(|&r| (r - mean).powi(2)).sum::<f64>() / (n - 1.0);
let std_dev = variance.sqrt();
if std_dev < 1e-15 {
return 0.0;
}
mean / std_dev
}
/// Compute skewness of a return series.
pub fn skewness(returns: &[f64]) -> f64 {
if returns.len() < 3 {
return 0.0;
}
let n = returns.len() as f64;
let mean = returns.iter().sum::<f64>() / n;
let variance = returns.iter().map(|&r| (r - mean).powi(2)).sum::<f64>() / (n - 1.0);
let std_dev = variance.sqrt();
if std_dev < 1e-15 {
return 0.0;
}
let m3 = returns.iter().map(|&r| ((r - mean) / std_dev).powi(3)).sum::<f64>();
(n / ((n - 1.0) * (n - 2.0))) * m3
}
/// Compute excess kurtosis of a return series.
pub fn excess_kurtosis(returns: &[f64]) -> f64 {
if returns.len() < 4 {
return 0.0;
}
let n = returns.len() as f64;
let mean = returns.iter().sum::<f64>() / n;
let m2 = returns.iter().map(|&r| (r - mean).powi(2)).sum::<f64>() / n;
if m2 < 1e-30 {
return 0.0;
}
let m4 = returns.iter().map(|&r| (r - mean).powi(4)).sum::<f64>() / n;
(m4 / (m2 * m2)) - 3.0
}
// ══════════════════════════════════════════════════════
// Deflated Sharpe Ratio
// ══════════════════════════════════════════════════════
/// Result of the Deflated Sharpe Ratio test.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DsrResult {
/// The observed (best) Sharpe ratio being tested.
pub observed_sharpe: f64,
/// Expected maximum Sharpe under the null hypothesis (no skill).
pub expected_max_sharpe: f64,
/// Standard error of the Sharpe ratio estimator.
pub sharpe_std_error: f64,
/// The DSR test statistic.
pub deflated_sharpe: f64,
/// p-value: probability of observing this Sharpe by chance.
pub pvalue: f64,
}
/// Compute the Deflated Sharpe Ratio (Bailey & Lopez de Prado, 2014).
///
/// # Arguments
/// * `observed_sharpe` - Sharpe ratio of the best strategy found
/// * `num_trials` - Number of strategies/hyperopt trials tested
/// * `sharpe_variance` - Variance of Sharpe ratios across trials
/// * `skew` - Skewness of the return series
/// * `kurt` - Excess kurtosis of the return series
/// * `num_observations` - Total number of return observations
pub fn deflated_sharpe_ratio(
observed_sharpe: f64,
num_trials: usize,
sharpe_variance: f64,
skew: f64,
kurt: f64,
num_observations: usize,
) -> DsrResult {
let n = num_observations as f64;
let num_trials_f = num_trials as f64;
// Expected maximum Sharpe ratio under null (Euler-Mascheroni approximation)
let gamma = 0.5772156649015329; // Euler-Mascheroni constant
let sharpe_std = sharpe_variance.sqrt().max(1e-15);
let expected_max_sharpe = if num_trials <= 1 {
0.0
} else {
let z = normal_ppf(1.0 - 1.0 / num_trials_f);
let z_e = normal_ppf(1.0 - 1.0 / (num_trials_f * std::f64::consts::E));
sharpe_std * ((1.0 - gamma) * z + gamma * z_e)
};
// Standard error of the Sharpe ratio (adjusted for non-normality)
let sr = observed_sharpe;
let se_sr = if n > 1.0 {
((1.0 - skew * sr + ((kurt - 1.0) / 4.0) * sr * sr) / (n - 1.0))
.max(0.0)
.sqrt()
} else {
1.0
};
// DSR statistic
let deflated_sharpe = if se_sr > 1e-15 {
(sr - expected_max_sharpe) / se_sr
} else {
0.0
};
// p-value (one-sided: probability observed SR exceeds expected max by chance)
let pvalue = 1.0 - normal_cdf(deflated_sharpe);
DsrResult {
observed_sharpe,
expected_max_sharpe,
sharpe_std_error: se_sr,
deflated_sharpe,
pvalue,
}
}
// ══════════════════════════════════════════════════════
// Monte Carlo Permutation Test
// ══════════════════════════════════════════════════════
/// Result of a Monte Carlo permutation test.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PermutationResult {
/// Observed Sharpe ratio of the actual return series.
pub observed_sharpe: f64,
/// p-value: fraction of permuted Sharpes >= observed Sharpe.
pub pvalue: f64,
/// Number of permutations run.
pub num_permutations: usize,
/// Mean of the null distribution.
pub null_mean: f64,
/// Std dev of the null distribution.
pub null_std: f64,
}
/// Run a Monte Carlo permutation test on daily returns.
///
/// Shuffles the return series (destroying temporal structure / any signal),
/// recomputes Sharpe on each permutation, and reports what fraction of
/// permuted Sharpes are >= the observed Sharpe.
///
/// # Arguments
/// * `daily_returns` - The actual return series to test
/// * `num_permutations` - Number of shuffles (default: 10_000)
/// * `seed` - RNG seed for reproducibility
pub fn permutation_test(
daily_returns: &[f64],
num_permutations: usize,
seed: u64,
) -> PermutationResult {
let observed = sharpe_ratio(daily_returns);
if daily_returns.len() < 2 || num_permutations == 0 {
return PermutationResult {
observed_sharpe: observed,
pvalue: 1.0,
num_permutations,
null_mean: 0.0,
null_std: 0.0,
};
}
let mut rng = ChaCha8Rng::seed_from_u64(seed);
let mut shuffled = daily_returns.to_vec();
let mut count_ge = 0usize;
let mut null_sharpes = Vec::with_capacity(num_permutations);
for _ in 0..num_permutations {
shuffled.shuffle(&mut rng);
let perm_sharpe = sharpe_ratio(&shuffled);
null_sharpes.push(perm_sharpe);
if perm_sharpe >= observed {
count_ge += 1;
}
// Restore original order for next shuffle
shuffled.copy_from_slice(daily_returns);
}
let null_mean = null_sharpes.iter().sum::<f64>() / null_sharpes.len() as f64;
let null_std = if null_sharpes.len() > 1 {
let var = null_sharpes
.iter()
.map(|&s| (s - null_mean).powi(2))
.sum::<f64>()
/ (null_sharpes.len() - 1) as f64;
var.sqrt()
} else {
0.0
};
PermutationResult {
observed_sharpe: observed,
pvalue: count_ge as f64 / num_permutations as f64,
num_permutations,
null_mean,
null_std,
}
}
// ══════════════════════════════════════════════════════
// Probability of Backtest Overfitting (PBO)
// ══════════════════════════════════════════════════════
/// Result of the PBO analysis.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PboResult {
/// Probability of backtest overfitting [0, 1].
/// PBO > 0.5 = likely overfit, PBO < 0.25 = strong evidence of real signal.
pub pbo: f64,
/// Number of CSCV combinations tested.
pub num_combinations: usize,
/// Distribution of logit(rank) values for detailed analysis.
pub logit_distribution: Vec<f64>,
}
/// Compute Probability of Backtest Overfitting via CSCV.
///
/// Uses Combinatorially Symmetric Cross-Validation:
/// 1. Given N fold Sharpes, generate C(N, N/2) combinations
/// 2. For each: split into IS/OOS halves, find IS-best, check OOS rank
/// 3. PBO = fraction where IS-best underperforms OOS median
///
/// # Arguments
/// * `per_fold_sharpes` - Sharpe ratios for each walk-forward fold (N folds).
/// N must be even (>=4) for CSCV to work.
///
/// # Note
/// For large N, this uses sampling instead of exhaustive enumeration
/// to keep computation tractable (C(16,8) = 12870 combinations).
pub fn probability_of_backtest_overfitting(per_fold_sharpes: &[f64]) -> PboResult {
let n = per_fold_sharpes.len();
// CSCV requires even N >= 4
if n < 4 || n % 2 != 0 {
return PboResult {
pbo: 0.5, // uninformative prior
num_combinations: 0,
logit_distribution: vec![],
};
}
let half = n / 2;
let combinations = generate_combinations(n, half);
let mut logits = Vec::with_capacity(combinations.len());
let mut overfit_count = 0usize;
for combo in &combinations {
// IS = indices in combo, OOS = indices not in combo
let is_sharpes: Vec<f64> = combo.iter().map(|&i| per_fold_sharpes[i]).collect();
let oos_indices: Vec<usize> = (0..n).filter(|i| !combo.contains(i)).collect();
let oos_sharpes: Vec<f64> = oos_indices.iter().map(|&i| per_fold_sharpes[i]).collect();
// Find IS-best fold index (within the combo)
let is_best_combo_idx = is_sharpes
.iter()
.enumerate()
.max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal))
.map(|(i, _)| i)
.unwrap_or(0);
// The IS-best fold's original index
let is_best_original_idx = combo[is_best_combo_idx];
// What is this fold's Sharpe in OOS context?
// (it's the same value — we're checking how its rank compares)
let is_best_sharpe = per_fold_sharpes[is_best_original_idx];
// Rank among OOS sharpes: how many OOS sharpes beat it?
let rank = oos_sharpes.iter().filter(|&&s| s > is_best_sharpe).count();
let relative_rank = rank as f64 / half as f64;
// Logit: ln(rank / (1 - rank)), clamped to avoid infinity
let clamped_rank = relative_rank.clamp(0.01, 0.99);
let logit = (clamped_rank / (1.0 - clamped_rank)).ln();
logits.push(logit);
// IS-best underperforms OOS median if rank > N/4 (worse than middle)
if relative_rank > 0.5 {
overfit_count += 1;
}
}
let num_combinations = combinations.len();
let pbo = if num_combinations > 0 {
overfit_count as f64 / num_combinations as f64
} else {
0.5
};
PboResult {
pbo,
num_combinations,
logit_distribution: logits,
}
}
/// Generate all C(n, k) combinations of indices 0..n.
/// For large n, caps at 10_000 random samples to keep computation tractable.
fn generate_combinations(n: usize, k: usize) -> Vec<Vec<usize>> {
// Estimate C(n, k) — if too large, sample instead
let max_combinations = 10_000;
let estimated = binomial_coefficient(n, k);
if estimated <= max_combinations {
// Exhaustive enumeration
let mut result = Vec::new();
let mut current = Vec::with_capacity(k);
enumerate_combinations(n, k, 0, &mut current, &mut result);
result
} else {
// Random sampling
let mut rng = ChaCha8Rng::seed_from_u64(42);
let indices: Vec<usize> = (0..n).collect();
let mut result = Vec::with_capacity(max_combinations);
let mut seen = std::collections::HashSet::new();
while result.len() < max_combinations {
let mut sample = indices.clone();
sample.shuffle(&mut rng);
let mut combo: Vec<usize> = sample.into_iter().take(k).collect();
combo.sort_unstable();
if seen.insert(combo.clone()) {
result.push(combo);
}
}
result
}
}
fn enumerate_combinations(
n: usize,
k: usize,
start: usize,
current: &mut Vec<usize>,
result: &mut Vec<Vec<usize>>,
) {
if current.len() == k {
result.push(current.clone());
return;
}
let remaining = k - current.len();
for i in start..=(n - remaining) {
current.push(i);
enumerate_combinations(n, k, i + 1, current, result);
current.pop();
}
}
fn binomial_coefficient(n: usize, k: usize) -> usize {
if k > n {
return 0;
}
let k = k.min(n - k);
let mut result: usize = 1;
for i in 0..k {
result = result.saturating_mul(n - i) / (i + 1);
}
result
}
#[cfg(test)]
mod tests {
use super::*;
// ── Math helpers ──
#[test]
fn test_normal_cdf_known_values() {
assert!((normal_cdf(0.0) - 0.5).abs() < 1e-6);
assert!((normal_cdf(1.96) - 0.975).abs() < 1e-3);
assert!((normal_cdf(-1.96) - 0.025).abs() < 1e-3);
}
#[test]
fn test_normal_ppf_known_values() {
assert!(normal_ppf(0.5).abs() < 1e-5);
assert!((normal_ppf(0.975) - 1.96).abs() < 0.01);
assert!((normal_ppf(0.025) + 1.96).abs() < 0.01);
}
#[test]
fn test_normal_cdf_ppf_roundtrip() {
for &p in &[0.01, 0.05, 0.25, 0.5, 0.75, 0.95, 0.99] {
let x = normal_ppf(p);
let p_back = normal_cdf(x);
assert!(
(p - p_back).abs() < 1e-4,
"Roundtrip failed for p={}: got {}",
p,
p_back
);
}
}
#[test]
fn test_sharpe_ratio_positive_trend() {
// Consistently positive returns → positive Sharpe
let returns = vec![0.01, 0.02, 0.015, 0.01, 0.025, 0.012, 0.018];
let sr = sharpe_ratio(&returns);
assert!(sr > 0.0, "Positive returns should give positive Sharpe: {}", sr);
}
#[test]
fn test_sharpe_ratio_zero_returns() {
let returns = vec![0.0, 0.0, 0.0, 0.0];
let sr = sharpe_ratio(&returns);
assert!((sr - 0.0).abs() < 1e-10);
}
#[test]
fn test_skewness_symmetric() {
// Roughly symmetric returns → skewness near 0
let returns = vec![-0.02, -0.01, 0.0, 0.01, 0.02];
let skew = skewness(&returns);
assert!(skew.abs() < 0.5, "Symmetric returns should have near-zero skewness: {}", skew);
}
#[test]
fn test_excess_kurtosis_normal() {
// Normal-ish returns → excess kurtosis near 0
let returns: Vec<f64> = (0..1000)
.map(|i| ((i as f64 * 0.01).sin()) * 0.01)
.collect();
let kurt = excess_kurtosis(&returns);
// Sinusoidal returns won't be exactly 0, but should be bounded
assert!(kurt.abs() < 5.0, "Kurtosis too extreme: {}", kurt);
}
// ── DSR ──
#[test]
fn test_dsr_single_trial_not_penalized() {
let result = deflated_sharpe_ratio(2.0, 1, 1.0, 0.0, 3.0, 252);
// With 1 trial, expected max = 0, so DSR should be favorable
assert!(result.pvalue < 0.05, "Single trial with SR=2 should be significant");
}
#[test]
fn test_dsr_many_trials_penalized() {
let result = deflated_sharpe_ratio(1.0, 1000, 1.0, 0.0, 3.0, 252);
// With 1000 trials, SR=1.0 is likely just the best random draw
assert!(
result.pvalue > 0.05,
"SR=1.0 with 1000 trials should NOT be significant: p={}",
result.pvalue
);
}
#[test]
fn test_dsr_higher_sharpe_more_significant() {
let low = deflated_sharpe_ratio(1.0, 100, 1.0, 0.0, 3.0, 252);
let high = deflated_sharpe_ratio(3.0, 100, 1.0, 0.0, 3.0, 252);
assert!(high.pvalue < low.pvalue, "Higher Sharpe should be more significant");
}
// ── Permutation test ──
#[test]
fn test_permutation_test_random_returns_high_pvalue() {
// Random returns with no signal → high p-value
let returns: Vec<f64> = (0..100)
.map(|i| if i % 2 == 0 { 0.01 } else { -0.01 })
.collect();
let result = permutation_test(&returns, 1000, 42);
assert!(
result.pvalue > 0.05,
"Random returns should have high p-value: {}",
result.pvalue
);
}
#[test]
fn test_permutation_test_strong_signal_low_pvalue() {
// Strong upward trend → low p-value
let returns: Vec<f64> = (0..200).map(|_| 0.02).collect();
let result = permutation_test(&returns, 1000, 42);
// All-positive returns: any shuffle preserves the mean, so Sharpe stays the same
// This tests that the code runs — the p-value for constant returns is ~1.0
// because every permutation has the same Sharpe
assert!(result.num_permutations == 1000);
}
#[test]
fn test_permutation_test_reproducible() {
let returns = vec![0.01, -0.005, 0.02, -0.01, 0.015, -0.002, 0.008];
let r1 = permutation_test(&returns, 500, 123);
let r2 = permutation_test(&returns, 500, 123);
assert!(
(r1.pvalue - r2.pvalue).abs() < 1e-10,
"Same seed should give same result"
);
}
// ── PBO ──
#[test]
fn test_pbo_random_sharpes_high() {
// Random per-fold Sharpes with no consistency → PBO should be high
let sharpes = vec![0.5, -0.3, 1.2, -0.8, 0.1, -0.5, 0.9, -0.2];
let result = probability_of_backtest_overfitting(&sharpes);
assert!(result.num_combinations > 0, "Should produce combinations");
// With random Sharpes, PBO should be around 0.5
assert!(
result.pbo > 0.2,
"Random Sharpes should have elevated PBO: {}",
result.pbo
);
}
#[test]
fn test_pbo_consistent_sharpes_low() {
// All folds have similar positive Sharpe → low PBO
let sharpes = vec![1.5, 1.6, 1.4, 1.5, 1.7, 1.3, 1.6, 1.4];
let result = probability_of_backtest_overfitting(&sharpes);
assert!(
result.pbo < 0.5,
"Consistent positive Sharpes should have low PBO: {}",
result.pbo
);
}
#[test]
fn test_pbo_too_few_folds() {
let sharpes = vec![1.0, 2.0]; // Need >= 4 even folds
let result = probability_of_backtest_overfitting(&sharpes);
assert_eq!(result.num_combinations, 0);
assert!((result.pbo - 0.5).abs() < 1e-10); // uninformative prior
}
#[test]
fn test_pbo_odd_folds() {
let sharpes = vec![1.0, 2.0, 3.0, 4.0, 5.0]; // Odd N
let result = probability_of_backtest_overfitting(&sharpes);
assert_eq!(result.num_combinations, 0); // CSCV requires even N
}
#[test]
fn test_binomial_coefficient() {
assert_eq!(binomial_coefficient(8, 4), 70);
assert_eq!(binomial_coefficient(6, 3), 20);
assert_eq!(binomial_coefficient(4, 2), 6);
}
}
```
**Step 2: Register in mod.rs**
Add to `ml/src/validation/mod.rs`:
```rust
pub mod statistical;
pub use statistical::{
deflated_sharpe_ratio, excess_kurtosis, normal_cdf, normal_ppf, permutation_test,
probability_of_backtest_overfitting, sharpe_ratio, skewness, DsrResult, PboResult,
PermutationResult,
};
```
**Step 3: Run tests**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --lib validation::statistical -- --nocapture 2>&1 | tail -15`
Expected: 14 tests pass
**Step 4: Verify full compilation**
Run: `SQLX_OFFLINE=true cargo check --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml`
**Step 5: Commit**
```bash
git add ml/src/validation/statistical.rs ml/src/validation/mod.rs
git commit -m "feat(validation): add DSR, PBO, permutation test, and math helpers"
```
---
### Task 5: Per-regime analysis
Implements per-regime Sharpe breakdown using the existing `RegimeType` enum from `dqn::regime_conditional`.
**Files:**
- Create: `ml/src/validation/regime_analysis.rs`
- Modify: `ml/src/validation/mod.rs`
**Step 1: Write regime_analysis.rs**
```rust
//! Per-regime performance analysis.
//!
//! Labels each bar with a market regime (Trending, Ranging, Volatile) using
//! the existing regime detection infrastructure, then computes per-regime
//! Sharpe ratio, win rate, and average return.
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
use crate::dqn::RegimeType;
use super::statistical::sharpe_ratio;
/// Performance metrics for a single market regime.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RegimeMetrics {
/// Which regime these metrics describe.
pub regime: RegimeType,
/// Sharpe ratio for bars in this regime.
pub sharpe: f64,
/// Number of bars classified into this regime.
pub num_bars: usize,
/// Fraction of bars with positive returns.
pub win_rate: f64,
/// Mean return across all bars in this regime.
pub avg_return: f64,
}
/// Compute per-regime performance breakdown.
///
/// # Arguments
/// * `daily_returns` - Daily returns (length N)
/// * `features` - Feature vectors for each bar (length N, used for regime classification)
///
/// Regime classification uses `RegimeType::classify_from_features()` which looks at
/// ADX and entropy values within the feature vector.
pub fn per_regime_breakdown(
daily_returns: &[f64],
features: &[Vec<f32>],
) -> HashMap<RegimeType, RegimeMetrics> {
let mut regime_returns: HashMap<RegimeType, Vec<f64>> = HashMap::new();
let n = daily_returns.len().min(features.len());
for i in 0..n {
let regime = RegimeType::classify_from_features(&features[i]);
regime_returns
.entry(regime)
.or_default()
.push(daily_returns[i]);
}
let mut result = HashMap::new();
for (regime, returns) in &regime_returns {
let num_bars = returns.len();
let avg_return = if num_bars > 0 {
returns.iter().sum::<f64>() / num_bars as f64
} else {
0.0
};
let win_rate = if num_bars > 0 {
returns.iter().filter(|&&r| r > 0.0).count() as f64 / num_bars as f64
} else {
0.0
};
result.insert(
*regime,
RegimeMetrics {
regime: *regime,
sharpe: sharpe_ratio(returns),
num_bars,
win_rate,
avg_return,
},
);
}
result
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_per_regime_breakdown_groups_correctly() {
// With insufficient features (< 211), classify_from_features returns Ranging (safe default)
let returns = vec![0.01, -0.005, 0.02, 0.015, -0.01];
let features: Vec<Vec<f32>> = (0..5).map(|i| vec![i as f32; 10]).collect();
let breakdown = per_regime_breakdown(&returns, &features);
// All should be classified as Ranging (features too short for real classification)
assert!(breakdown.contains_key(&RegimeType::Ranging));
let ranging = &breakdown[&RegimeType::Ranging];
assert_eq!(ranging.num_bars, 5);
}
#[test]
fn test_per_regime_metrics_computation() {
let returns = vec![0.01, 0.02, 0.03];
let features: Vec<Vec<f32>> = (0..3).map(|_| vec![0.0; 10]).collect();
let breakdown = per_regime_breakdown(&returns, &features);
let metrics = breakdown.values().next().expect("should have one regime");
assert_eq!(metrics.num_bars, 3);
assert!((metrics.win_rate - 1.0).abs() < 1e-10); // all positive
assert!((metrics.avg_return - 0.02).abs() < 1e-10);
}
#[test]
fn test_empty_returns() {
let breakdown = per_regime_breakdown(&[], &[]);
assert!(breakdown.is_empty());
}
}
```
**Step 2: Register in mod.rs**
Add to `ml/src/validation/mod.rs`:
```rust
pub mod regime_analysis;
pub use regime_analysis::{per_regime_breakdown, RegimeMetrics};
```
**Step 3: Run tests**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --lib validation::regime_analysis -- --nocapture 2>&1 | tail -10`
Expected: 3 tests pass
**Step 4: Verify full compilation**
Run: `SQLX_OFFLINE=true cargo check --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml`
**Step 5: Commit**
```bash
git add ml/src/validation/regime_analysis.rs ml/src/validation/mod.rs
git commit -m "feat(validation): add per-regime performance analysis"
```
---
### Task 6: Validation harness orchestrator
Ties everything together: runs a strategy through walk-forward folds, computes DSR, PBO, permutation test, and regime breakdown, then produces a `ValidationReport`.
**Files:**
- Create: `ml/src/validation/harness.rs`
- Modify: `ml/src/validation/mod.rs`
**Step 1: Write harness.rs**
```rust
//! Validation harness orchestrator.
//!
//! Runs a `ValidatableStrategy` through the complete validation pipeline:
//! 1. Walk-forward cross-validation with embargo
//! 2. Per-fold Sharpe computation
//! 3. Deflated Sharpe Ratio (adjusts for multiple testing)
//! 4. Probability of Backtest Overfitting (CSCV)
//! 5. Monte Carlo permutation test
//! 6. Per-regime performance breakdown
//! 7. Overall verdict (Pass / Marginal / Fail)
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
use crate::dqn::RegimeType;
use crate::MLError;
use super::regime_analysis::{per_regime_breakdown, RegimeMetrics};
use super::statistical::{
deflated_sharpe_ratio, excess_kurtosis, permutation_test, probability_of_backtest_overfitting,
sharpe_ratio, skewness, DsrResult, PboResult, PermutationResult,
};
use super::types::{TimeSeriesData, ValidatableStrategy};
use super::walk_forward::{walk_forward_split, WalkForwardConfig};
/// Overall validation verdict.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum ValidationVerdict {
/// All tests pass: DSR p < 0.05, PBO < 0.25, MC p < 0.05
Pass,
/// At least one test passes at relaxed threshold
Marginal,
/// No statistical evidence of a real signal
Fail,
}
impl std::fmt::Display for ValidationVerdict {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
ValidationVerdict::Pass => write!(f, "PASS"),
ValidationVerdict::Marginal => write!(f, "MARGINAL"),
ValidationVerdict::Fail => write!(f, "FAIL"),
}
}
}
/// Complete validation report from the harness.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ValidationReport {
pub strategy_name: String,
// Walk-forward results
pub per_fold_sharpes: Vec<f64>,
pub aggregate_sharpe: f64,
pub num_folds: usize,
// DSR
pub dsr: DsrResult,
// PBO
pub pbo: PboResult,
// Monte Carlo
pub permutation: PermutationResult,
// Regime breakdown
pub per_regime_metrics: HashMap<RegimeType, RegimeMetrics>,
// Overall
pub verdict: ValidationVerdict,
}
/// Configuration for the validation harness.
#[derive(Debug, Clone)]
pub struct ValidationHarnessConfig {
/// Walk-forward configuration.
pub wf_config: WalkForwardConfig,
/// Number of Monte Carlo permutations.
pub num_permutations: usize,
/// Number of hyperopt trials tested (for DSR).
pub num_trials: usize,
/// RNG seed for reproducibility.
pub seed: u64,
}
impl Default for ValidationHarnessConfig {
fn default() -> Self {
Self {
wf_config: WalkForwardConfig::default(),
num_permutations: 10_000,
num_trials: 1,
seed: 42,
}
}
}
/// The validation harness: runs a strategy through the complete pipeline.
pub struct ValidationHarness {
config: ValidationHarnessConfig,
}
impl ValidationHarness {
pub fn new(config: ValidationHarnessConfig) -> Self {
Self { config }
}
/// Run full validation on a strategy.
///
/// # Arguments
/// * `strategy` - A model implementing `ValidatableStrategy`
/// * `data` - The full time-series dataset to validate against
///
/// # Returns
/// A `ValidationReport` with all statistical test results and a verdict.
pub fn validate<S: ValidatableStrategy>(
&self,
strategy: &mut S,
data: &TimeSeriesData,
) -> Result<ValidationReport, MLError> {
// 1. Walk-forward split
let folds = walk_forward_split(data.len(), &self.config.wf_config);
if folds.is_empty() {
return Err(MLError::ValidationError {
message: format!(
"Not enough data for walk-forward validation: {} bars, need at least {}",
data.len(),
self.config.wf_config.train_bars
+ self.config.wf_config.embargo_bars
+ self.config.wf_config.test_bars
),
});
}
// 2. Run strategy through each fold
let mut per_fold_sharpes = Vec::with_capacity(folds.len());
let mut all_test_returns = Vec::new();
let mut all_test_features = Vec::new();
for fold in &folds {
// Reset strategy for fresh training
strategy.reset()?;
// Train on training data
let train_data = data.slice(fold.train_range.start, fold.train_range.end)?;
strategy.train(&train_data)?;
// Evaluate on test data
let test_data = data.slice(fold.test_range.start, fold.test_range.end)?;
let fold_returns = strategy.evaluate(&test_data)?;
// Collect per-fold Sharpe
let fold_sharpe = sharpe_ratio(&fold_returns);
per_fold_sharpes.push(fold_sharpe);
// Accumulate returns and features for aggregate stats
all_test_returns.extend_from_slice(&fold_returns);
all_test_features.extend(test_data.features.iter().take(fold_returns.len()).cloned());
}
// 3. Aggregate Sharpe
let aggregate_sharpe = sharpe_ratio(&all_test_returns);
// 4. DSR
let sharpe_variance = if per_fold_sharpes.len() > 1 {
let mean = per_fold_sharpes.iter().sum::<f64>() / per_fold_sharpes.len() as f64;
per_fold_sharpes
.iter()
.map(|&s| (s - mean).powi(2))
.sum::<f64>()
/ (per_fold_sharpes.len() - 1) as f64
} else {
1.0
};
let skew = skewness(&all_test_returns);
let kurt = excess_kurtosis(&all_test_returns);
let dsr = deflated_sharpe_ratio(
aggregate_sharpe,
self.config.num_trials,
sharpe_variance,
skew,
kurt,
all_test_returns.len(),
);
// 5. PBO (requires even number of folds)
let pbo_sharpes = if per_fold_sharpes.len() % 2 != 0 && per_fold_sharpes.len() > 1 {
// Drop last fold to make even
per_fold_sharpes[..per_fold_sharpes.len() - 1].to_vec()
} else {
per_fold_sharpes.clone()
};
let pbo = probability_of_backtest_overfitting(&pbo_sharpes);
// 6. Monte Carlo permutation test
let permutation = permutation_test(
&all_test_returns,
self.config.num_permutations,
self.config.seed,
);
// 7. Per-regime breakdown
let per_regime_metrics = per_regime_breakdown(&all_test_returns, &all_test_features);
// 8. Verdict
let verdict = determine_verdict(&dsr, &pbo, &permutation);
Ok(ValidationReport {
strategy_name: strategy.name().to_string(),
per_fold_sharpes,
aggregate_sharpe,
num_folds: folds.len(),
dsr,
pbo,
permutation,
per_regime_metrics,
verdict,
})
}
}
fn determine_verdict(
dsr: &DsrResult,
pbo: &PboResult,
permutation: &PermutationResult,
) -> ValidationVerdict {
let dsr_pass = dsr.pvalue < 0.05;
let pbo_pass = pbo.pbo < 0.25;
let mc_pass = permutation.pvalue < 0.05;
if dsr_pass && pbo_pass && mc_pass {
ValidationVerdict::Pass
} else if dsr.pvalue < 0.10 || pbo.pbo < 0.50 || permutation.pvalue < 0.10 {
ValidationVerdict::Marginal
} else {
ValidationVerdict::Fail
}
}
#[cfg(test)]
mod tests {
use super::*;
use chrono::{Duration, Utc};
/// A mock strategy that always returns fixed returns (for testing the harness).
struct MockStrategy {
name: String,
returns_per_bar: f64,
}
impl MockStrategy {
fn new(name: &str, returns_per_bar: f64) -> Self {
Self {
name: name.to_string(),
returns_per_bar,
}
}
}
impl ValidatableStrategy for MockStrategy {
fn train(&mut self, _data: &TimeSeriesData) -> Result<(), MLError> {
Ok(())
}
fn evaluate(&self, data: &TimeSeriesData) -> Result<Vec<f64>, MLError> {
Ok(vec![self.returns_per_bar; data.len()])
}
fn name(&self) -> &str {
&self.name
}
fn reset(&mut self) -> Result<(), MLError> {
Ok(())
}
}
fn make_data(n: usize) -> TimeSeriesData {
let start = Utc::now();
let timestamps: Vec<_> = (0..n)
.map(|i| start + Duration::hours(i as i64))
.collect();
let features: Vec<Vec<f32>> = (0..n).map(|i| vec![i as f32; 10]).collect();
let prices: Vec<f64> = (0..n).map(|i| 100.0 + i as f64 * 0.1).collect();
TimeSeriesData::new(timestamps, features, prices).expect("valid data")
}
#[test]
fn test_harness_runs_end_to_end() {
let data = make_data(500);
let config = ValidationHarnessConfig {
wf_config: WalkForwardConfig {
train_bars: 100,
test_bars: 50,
embargo_bars: 5,
step_bars: 50,
min_train_samples: 50,
},
num_permutations: 100, // small for test speed
num_trials: 1,
seed: 42,
};
let harness = ValidationHarness::new(config);
let mut strategy = MockStrategy::new("test-mock", 0.01);
let report = harness.validate(&mut strategy, &data);
assert!(report.is_ok(), "Harness should complete: {:?}", report.err());
let report = report.expect("valid report");
assert_eq!(report.strategy_name, "test-mock");
assert!(!report.per_fold_sharpes.is_empty());
assert!(report.num_folds > 0);
}
#[test]
fn test_harness_insufficient_data_errors() {
let data = make_data(20); // way too short
let config = ValidationHarnessConfig {
wf_config: WalkForwardConfig {
train_bars: 100,
test_bars: 50,
embargo_bars: 5,
step_bars: 50,
min_train_samples: 50,
},
..Default::default()
};
let harness = ValidationHarness::new(config);
let mut strategy = MockStrategy::new("too-short", 0.01);
let report = harness.validate(&mut strategy, &data);
assert!(report.is_err());
}
#[test]
fn test_verdict_all_pass() {
let dsr = DsrResult {
observed_sharpe: 3.0,
expected_max_sharpe: 1.0,
sharpe_std_error: 0.5,
deflated_sharpe: 4.0,
pvalue: 0.001,
};
let pbo = PboResult {
pbo: 0.1,
num_combinations: 70,
logit_distribution: vec![],
};
let perm = PermutationResult {
observed_sharpe: 3.0,
pvalue: 0.001,
num_permutations: 1000,
null_mean: 0.0,
null_std: 0.5,
};
assert_eq!(determine_verdict(&dsr, &pbo, &perm), ValidationVerdict::Pass);
}
#[test]
fn test_verdict_fail() {
let dsr = DsrResult {
observed_sharpe: 0.5,
expected_max_sharpe: 2.0,
sharpe_std_error: 0.5,
deflated_sharpe: -3.0,
pvalue: 0.99,
};
let pbo = PboResult {
pbo: 0.8,
num_combinations: 70,
logit_distribution: vec![],
};
let perm = PermutationResult {
observed_sharpe: 0.5,
pvalue: 0.6,
num_permutations: 1000,
null_mean: 0.0,
null_std: 0.5,
};
assert_eq!(determine_verdict(&dsr, &pbo, &perm), ValidationVerdict::Fail);
}
}
```
**Step 2: Register in mod.rs**
Add to `ml/src/validation/mod.rs`:
```rust
pub mod harness;
pub use harness::{ValidationHarness, ValidationHarnessConfig, ValidationReport, ValidationVerdict};
```
**Step 3: Run tests**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --lib validation::harness -- --nocapture 2>&1 | tail -10`
Expected: 4 tests pass
**Step 4: Verify full test suite**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --lib validation -- --nocapture 2>&1 | tail -10`
Expected: All validation tests pass (5 + 7 + 14 + 3 + 4 = 33 tests)
**Step 5: Commit**
```bash
git add ml/src/validation/harness.rs ml/src/validation/mod.rs
git commit -m "feat(validation): add validation harness orchestrator with verdict system"
```
---
### Task 7: DQN strategy adapter
Implement `ValidatableStrategy` for DQN so it can be validated through the harness.
**Files:**
- Create: `ml/src/validation/adapters.rs`
- Modify: `ml/src/validation/mod.rs`
**Step 1: Write adapters.rs**
This is a thin wrapper that translates between DQN's native API and the `ValidatableStrategy` trait.
```rust
//! Strategy adapters for the validation harness.
//!
//! Each adapter wraps a model's native API to implement `ValidatableStrategy`,
//! enabling walk-forward validation, DSR, PBO, and permutation tests.
use crate::dqn::{DQNConfig, Experience, DQN};
use crate::MLError;
use super::types::{TimeSeriesData, ValidatableStrategy};
/// Adapter that makes a DQN model validatable through the harness.
pub struct DqnStrategy {
config: DQNConfig,
dqn: DQN,
}
impl DqnStrategy {
/// Create a new DQN strategy adapter.
pub fn new(config: DQNConfig) -> Result<Self, MLError> {
let dqn = DQN::new(config.clone())?;
Ok(Self { config, dqn })
}
}
impl ValidatableStrategy for DqnStrategy {
fn train(&mut self, data: &TimeSeriesData) -> Result<(), MLError> {
// Feed experiences into the replay buffer
let num_bars = data.len();
if num_bars < 2 {
return Ok(());
}
for i in 0..num_bars.saturating_sub(1) {
let state = data.features.get(i).cloned().unwrap_or_default();
let next_state = data.features.get(i + 1).cloned().unwrap_or_default();
// Select action using current policy
let action = self.dqn.select_action(&state)?;
// Reward = next bar's return (simple reward shaping)
let reward = data.returns.get(i).copied().unwrap_or(0.0) as f32;
let done = i == num_bars - 2;
let exp = Experience::new(
state,
action.to_index() as u8,
reward,
next_state,
done,
);
self.dqn.store_experience(exp)?;
}
// Train for multiple steps if enough data in replay buffer
let train_steps = (num_bars / self.config.batch_size).min(100);
for _ in 0..train_steps {
match self.dqn.train_step(None) {
Ok(_) => {}
Err(MLError::TrainingError(msg)) if msg.contains("Not enough") => break,
Err(e) => return Err(e),
}
}
Ok(())
}
fn evaluate(&self, data: &TimeSeriesData) -> Result<Vec<f64>, MLError> {
let mut returns = Vec::with_capacity(data.len());
for i in 0..data.len() {
let state = data.features.get(i).cloned().unwrap_or_default();
let action = self.dqn.select_action(&state)?;
// Simple PnL model: Buy=+return, Hold=0, Sell=-return
let bar_return = data.returns.get(i).copied().unwrap_or(0.0);
let pnl = match action.to_index() {
0 => bar_return, // Buy → long exposure
2 => -bar_return, // Sell → short exposure
_ => 0.0, // Hold → no exposure
};
returns.push(pnl);
}
Ok(returns)
}
fn name(&self) -> &str {
"DQN"
}
fn reset(&mut self) -> Result<(), MLError> {
self.dqn = DQN::new(self.config.clone())?;
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use chrono::{Duration, Utc};
fn make_test_config() -> DQNConfig {
let mut config = DQNConfig::default();
config.state_dim = 10;
config.num_actions = 3;
config.hidden_dims = vec![16, 8];
config.batch_size = 4;
config.min_replay_size = 4;
config.warmup_steps = 0;
config.use_noisy_nets = false;
config.use_iqn = false;
config.use_distributional = false;
config.use_dueling = false;
config.use_per = false;
config.epsilon_start = 0.5;
config
}
fn make_test_data(n: usize, feature_dim: usize) -> TimeSeriesData {
let start = Utc::now();
let timestamps: Vec<_> = (0..n)
.map(|i| start + Duration::hours(i as i64))
.collect();
let features: Vec<Vec<f32>> = (0..n)
.map(|i| vec![0.1 * i as f32; feature_dim])
.collect();
let prices: Vec<f64> = (0..n).map(|i| 100.0 + i as f64 * 0.5).collect();
TimeSeriesData::new(timestamps, features, prices).expect("valid data")
}
#[test]
fn test_dqn_strategy_creation() {
let config = make_test_config();
let strategy = DqnStrategy::new(config);
assert!(strategy.is_ok());
}
#[test]
fn test_dqn_strategy_train_and_evaluate() {
let config = make_test_config();
let mut strategy = DqnStrategy::new(config).expect("valid strategy");
let data = make_test_data(30, 10);
let train_result = strategy.train(&data);
assert!(train_result.is_ok(), "Training failed: {:?}", train_result.err());
let eval_result = strategy.evaluate(&data);
assert!(eval_result.is_ok(), "Evaluation failed: {:?}", eval_result.err());
let returns = eval_result.expect("valid returns");
assert_eq!(returns.len(), 30);
}
#[test]
fn test_dqn_strategy_reset() {
let config = make_test_config();
let mut strategy = DqnStrategy::new(config).expect("valid strategy");
let data = make_test_data(20, 10);
strategy.train(&data).expect("training ok");
strategy.reset().expect("reset ok");
// After reset, should be able to train again cleanly
strategy.train(&data).expect("retrain ok");
}
#[test]
fn test_dqn_strategy_name() {
let config = make_test_config();
let strategy = DqnStrategy::new(config).expect("valid strategy");
assert_eq!(strategy.name(), "DQN");
}
}
```
**Step 2: Register in mod.rs**
Add to `ml/src/validation/mod.rs`:
```rust
pub mod adapters;
pub use adapters::DqnStrategy;
```
**Step 3: Run tests**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --lib validation::adapters -- --nocapture 2>&1 | tail -10`
Expected: 4 tests pass
**Step 4: Verify full compilation**
Run: `SQLX_OFFLINE=true cargo check --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml`
**Step 5: Commit**
```bash
git add ml/src/validation/adapters.rs ml/src/validation/mod.rs
git commit -m "feat(validation): add DQN strategy adapter for ValidatableStrategy trait"
```
---
### Task 8: Integration test — DQN through full validation pipeline
End-to-end test that runs a DQN model through the entire validation harness.
**Files:**
- Create: `ml/tests/validation_harness_integration_test.rs`
**Step 1: Write the integration test**
```rust
//! Integration test: Full validation pipeline with DQN model.
//!
//! Runs DQN through walk-forward validation, DSR, PBO, permutation test,
//! and per-regime analysis to verify the complete pipeline works end-to-end.
use chrono::{Duration, Utc};
use ml::dqn::DQNConfig;
use ml::validation::{
DqnStrategy, TimeSeriesData, ValidationHarness, ValidationHarnessConfig,
WalkForwardConfig, ValidatableStrategy,
};
fn make_synthetic_data(num_bars: usize, feature_dim: usize) -> TimeSeriesData {
let start = Utc::now();
let timestamps: Vec<_> = (0..num_bars)
.map(|i| start + Duration::hours(i as i64))
.collect();
let features: Vec<Vec<f32>> = (0..num_bars)
.map(|i| {
let base = (i as f32) * 0.01;
vec![base; feature_dim]
})
.collect();
let prices: Vec<f64> = (0..num_bars)
.map(|i| 100.0 + (i as f64 * 0.1).sin() * 5.0 + i as f64 * 0.01)
.collect();
TimeSeriesData::new(timestamps, features, prices).expect("valid synthetic data")
}
#[test]
fn test_full_validation_pipeline_with_dqn() {
let mut config = DQNConfig::default();
config.state_dim = 8;
config.num_actions = 3;
config.hidden_dims = vec![16, 8];
config.batch_size = 4;
config.min_replay_size = 4;
config.warmup_steps = 0;
config.use_noisy_nets = false;
config.use_iqn = false;
config.use_distributional = false;
config.use_dueling = false;
config.use_per = false;
config.epsilon_start = 0.3;
let mut strategy = DqnStrategy::new(config).expect("DQN creation");
let data = make_synthetic_data(300, 8);
let harness_config = ValidationHarnessConfig {
wf_config: WalkForwardConfig {
train_bars: 50,
test_bars: 30,
embargo_bars: 5,
step_bars: 30,
min_train_samples: 20,
},
num_permutations: 100, // small for test speed
num_trials: 1,
seed: 42,
};
let harness = ValidationHarness::new(harness_config);
let report = harness.validate(&mut strategy, &data);
assert!(report.is_ok(), "Validation pipeline failed: {:?}", report.err());
let report = report.expect("valid report");
assert_eq!(report.strategy_name, "DQN");
assert!(report.num_folds >= 2, "Should have multiple folds: {}", report.num_folds);
assert!(report.aggregate_sharpe.is_finite(), "Sharpe must be finite");
assert!(report.dsr.pvalue >= 0.0 && report.dsr.pvalue <= 1.0, "DSR p-value out of range");
assert!(report.pbo.pbo >= 0.0 && report.pbo.pbo <= 1.0, "PBO out of range");
assert!(report.permutation.pvalue >= 0.0 && report.permutation.pvalue <= 1.0, "MC p-value out of range");
// Regime breakdown should have at least one regime
assert!(!report.per_regime_metrics.is_empty(), "Should have at least one regime");
println!("=== Validation Report ===");
println!("Strategy: {}", report.strategy_name);
println!("Folds: {}", report.num_folds);
println!("Aggregate Sharpe: {:.4}", report.aggregate_sharpe);
println!("DSR p-value: {:.4}", report.dsr.pvalue);
println!("PBO: {:.4}", report.pbo.pbo);
println!("MC p-value: {:.4}", report.permutation.pvalue);
println!("Verdict: {}", report.verdict);
for (regime, metrics) in &report.per_regime_metrics {
println!(" {:?}: Sharpe={:.3}, WinRate={:.1}%, Bars={}",
regime, metrics.sharpe, metrics.win_rate * 100.0, metrics.num_bars);
}
}
#[test]
fn test_walk_forward_split_standalone() {
use ml::validation::{walk_forward_split, WalkForwardConfig};
let config = WalkForwardConfig {
train_bars: 50,
test_bars: 20,
embargo_bars: 5,
step_bars: 20,
min_train_samples: 20,
};
let folds = walk_forward_split(200, &config);
assert!(!folds.is_empty());
// Verify no test range exceeds data bounds
for fold in &folds {
assert!(fold.test_range.end <= 200);
assert!(fold.train_range.end <= fold.embargo_range.start);
assert!(fold.embargo_range.end <= fold.test_range.start);
}
}
#[test]
fn test_dsr_and_pbo_standalone() {
use ml::validation::{deflated_sharpe_ratio, probability_of_backtest_overfitting};
// DSR: high Sharpe with few trials should be significant
let dsr = deflated_sharpe_ratio(2.5, 5, 0.5, 0.1, 3.5, 500);
assert!(dsr.pvalue < 0.1, "Strong Sharpe with few trials should be significant");
// PBO: consistent fold Sharpes should show low overfitting probability
let sharpes = vec![1.0, 1.2, 0.8, 1.1, 0.9, 1.3, 1.0, 0.95];
let pbo = probability_of_backtest_overfitting(&sharpes);
assert!(pbo.num_combinations > 0);
}
```
**Step 2: Run the integration test**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --test validation_harness_integration_test -- --nocapture 2>&1 | tail -20`
Expected: 3 tests pass
**Step 3: Verify all existing tests still pass**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --lib 2>&1 | tail -5`
Expected: 1823+ tests pass (same as before plus ~37 new validation tests)
**Step 4: Commit**
```bash
git add ml/tests/validation_harness_integration_test.rs
git commit -m "test(validation): add end-to-end integration test for full validation pipeline"
```
---
### Task 9: Final verification
Full test suite run and compilation check to ensure nothing is broken.
**Step 1: Full workspace compilation**
Run: `SQLX_OFFLINE=true cargo check --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml`
Expected: Compiles clean
**Step 2: All lib tests**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --lib 2>&1 | tail -5`
Expected: All tests pass (1823 existing + ~37 new validation = ~1860)
**Step 3: All integration tests**
Run: `SQLX_OFFLINE=true cargo test --manifest-path /home/jgrusewski/Work/foxhunt/ml/Cargo.toml --tests 2>&1 | tail -10`
Expected: All integration tests pass
**Step 4: Summary of new files**
```
ml/src/validation/
├── mod.rs # Re-exports + module declarations
├── financial.rs # Existing financial validation (moved from validation.rs)
├── types.rs # TimeSeriesData + ValidatableStrategy trait
├── walk_forward.rs # Walk-forward splitter with embargo
├── statistical.rs # DSR + PBO + Monte Carlo + math helpers
├── regime_analysis.rs # Per-regime Sharpe breakdown
├── harness.rs # Orchestrator + ValidationReport + Verdict
└── adapters.rs # DqnStrategy (ValidatableStrategy for DQN)
ml/tests/
└── validation_harness_integration_test.rs # End-to-end pipeline test
```
**Step 5: Verify git status is clean**
Run: `git status`
Expected: All changes committed, working tree clean.
Reference in New Issue View Git Blame Copy Permalink