c318ffa30a
1. Walk-forward windows: replaced 3 non-overlapping with sliding (50% overlap, ~5 windows). Aggregation changed from mean-0.5*std to median-0.5*IQR for outlier robustness. 2. Composite score: tanh normalization prevents Calmar ratio scale dominance (0.02% drawdowns → values in thousands drowning out Sharpe/Sortino). 3. Q-value overestimation: new Prometheus gauge foxhunt_training_q_overestimation_ratio, warning log when ratio>10 or q_mean>5, adaptive tau doubles when Q-mean growth>0.5/epoch (capped at 0.01), decays back when stable. 2742 tests pass, 0 failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
52 lines
2.2 KiB
Markdown
52 lines
2.2 KiB
Markdown
# DQN Hyperopt Objective — 3 Bug Fixes
|
|
|
|
Date: 2026-03-07
|
|
Status: Approved
|
|
|
|
## Problem
|
|
|
|
Live hyperopt run (hyperopt-dqn-v2-hj5k8) on H100 revealed three issues in the DQN hyperopt objective function:
|
|
|
|
1. **Walk-forward windows**: Only 3 non-overlapping windows — `mean - 0.5*std` from 3 samples is statistically meaningless. PSO optimizes noise.
|
|
2. **Composite score scale dominance**: `0.4*sortino + 0.3*calmar + 0.2*sharpe + 0.1*omega` mixes incompatible scales. Calmar with 0.02% drawdown produces values in thousands, drowning out everything else. Agent biased toward holding (minimizes drawdown → maximizes Calmar).
|
|
3. **Q-value overestimation**: Q-mean drifted 0.12→0.14, Q-max hit 0.50. Double DQN + tau=0.001 already in place but insufficient for weak reward signals.
|
|
|
|
## Fix 1: Sliding Walk-Forward Windows
|
|
|
|
**File**: `crates/ml/src/hyperopt/adapters/dqn.rs`
|
|
|
|
- Remove `WINDOW_COUNT = 3`, replace with sliding windows (50% overlap)
|
|
- Window size stays at `total_bars / 3` (preserves per-window statistical power)
|
|
- Stride = `window_size / 2`, producing ~5 windows from same data
|
|
- Aggregation: `median(sharpes) - 0.5 * IQR(sharpes)` (robust to outliers)
|
|
|
|
## Fix 2: tanh Normalization of Composite Score
|
|
|
|
**File**: `crates/ml/src/hyperopt/adapters/dqn.rs`
|
|
|
|
Replace raw ratio weighting with tanh-normalized values:
|
|
|
|
```rust
|
|
0.4 * (sortino / 3.0).tanh() + // linear in [-3,3]
|
|
0.3 * (calmar / 5.0).tanh() + // tames explosion
|
|
0.2 * (sharpe / 2.0).tanh() + // linear in [-2,2]
|
|
0.1 * (omega / 2.0).tanh() // saturates beyond 2
|
|
```
|
|
|
|
Composite now bounded to [-1, 1] regardless of input magnitudes.
|
|
|
|
## Fix 3: Q-Value Overestimation — Monitoring + Adaptive Tau
|
|
|
|
**Files**: `crates/ml/src/trainers/dqn/trainer.rs`, `crates/ml/src/training_metrics.rs`
|
|
|
|
Part A: New Prometheus gauge `foxhunt_training_q_overestimation_ratio` + log warning when ratio > 10 or q_mean > 5.
|
|
|
|
Part B: Adaptive tau — when Q-mean growth per epoch > 0.5, double tau (capped at 0.01). When stable, decay back to base tau.
|
|
|
|
## Test Updates
|
|
|
|
- Update existing `test_extract_objective_*` expected values for tanh normalization
|
|
- Add `test_sliding_window_aggregation` with known inputs
|
|
- Add `test_tanh_normalization_bounds`
|
|
- Add `test_adaptive_tau_triggers`
|