Files

jgrusewski c318ffa30a fix(ml): 3 hyperopt objective bugs — sliding windows, tanh normalization, adaptive tau

1. Walk-forward windows: replaced 3 non-overlapping with sliding (50% overlap, ~5 windows).
   Aggregation changed from mean-0.5*std to median-0.5*IQR for outlier robustness.

2. Composite score: tanh normalization prevents Calmar ratio scale dominance
   (0.02% drawdowns → values in thousands drowning out Sharpe/Sortino).

3. Q-value overestimation: new Prometheus gauge foxhunt_training_q_overestimation_ratio,
   warning log when ratio>10 or q_mean>5, adaptive tau doubles when Q-mean growth>0.5/epoch
   (capped at 0.01), decays back when stable.

2742 tests pass, 0 failures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-07 02:14:31 +01:00

2.2 KiB

Raw Blame History

DQN Hyperopt Objective — 3 Bug Fixes

Date: 2026-03-07 Status: Approved

Problem

Live hyperopt run (hyperopt-dqn-v2-hj5k8) on H100 revealed three issues in the DQN hyperopt objective function:

Walk-forward windows: Only 3 non-overlapping windows — mean - 0.5*std from 3 samples is statistically meaningless. PSO optimizes noise.
Composite score scale dominance: 0.4*sortino + 0.3*calmar + 0.2*sharpe + 0.1*omega mixes incompatible scales. Calmar with 0.02% drawdown produces values in thousands, drowning out everything else. Agent biased toward holding (minimizes drawdown → maximizes Calmar).
Q-value overestimation: Q-mean drifted 0.12→0.14, Q-max hit 0.50. Double DQN + tau=0.001 already in place but insufficient for weak reward signals.

Fix 1: Sliding Walk-Forward Windows

File: crates/ml/src/hyperopt/adapters/dqn.rs

Remove WINDOW_COUNT = 3, replace with sliding windows (50% overlap)
Window size stays at total_bars / 3 (preserves per-window statistical power)
Stride = window_size / 2, producing ~5 windows from same data
Aggregation: median(sharpes) - 0.5 * IQR(sharpes) (robust to outliers)

Fix 2: tanh Normalization of Composite Score

File: crates/ml/src/hyperopt/adapters/dqn.rs

Replace raw ratio weighting with tanh-normalized values:

0.4 * (sortino / 3.0).tanh() +   // linear in [-3,3]
0.3 * (calmar / 5.0).tanh() +    // tames explosion
0.2 * (sharpe / 2.0).tanh() +    // linear in [-2,2]
0.1 * (omega / 2.0).tanh()       // saturates beyond 2

Composite now bounded to [-1, 1] regardless of input magnitudes.

Fix 3: Q-Value Overestimation — Monitoring + Adaptive Tau

Files: crates/ml/src/trainers/dqn/trainer.rs, crates/ml/src/training_metrics.rs

Part A: New Prometheus gauge foxhunt_training_q_overestimation_ratio + log warning when ratio > 10 or q_mean > 5.

Part B: Adaptive tau — when Q-mean growth per epoch > 0.5, double tau (capped at 0.01). When stable, decay back to base tau.

Test Updates

Update existing test_extract_objective_* expected values for tanh normalization
Add test_sliding_window_aggregation with known inputs
Add test_tanh_normalization_bounds
Add test_adaptive_tau_triggers

2.2 KiB Raw Blame History