# DQN Hyperopt Objective — 3 Bug Fixes

Date: 2026-03-07
Status: Approved

## Problem

Live hyperopt run (hyperopt-dqn-v2-hj5k8) on H100 revealed three issues in the DQN hyperopt objective function:

1. **Walk-forward windows**: Only 3 non-overlapping windows — `mean - 0.5*std` from 3 samples is statistically meaningless. PSO optimizes noise.
2. **Composite score scale dominance**: `0.4*sortino + 0.3*calmar + 0.2*sharpe + 0.1*omega` mixes incompatible scales. Calmar with 0.02% drawdown produces values in thousands, drowning out everything else. Agent biased toward holding (minimizes drawdown → maximizes Calmar).
3. **Q-value overestimation**: Q-mean drifted 0.12→0.14, Q-max hit 0.50. Double DQN + tau=0.001 already in place but insufficient for weak reward signals.

## Fix 1: Sliding Walk-Forward Windows

**File**: `crates/ml/src/hyperopt/adapters/dqn.rs`

- Remove `WINDOW_COUNT = 3`, replace with sliding windows (50% overlap)
- Window size stays at `total_bars / 3` (preserves per-window statistical power)
- Stride = `window_size / 2`, producing ~5 windows from same data
- Aggregation: `median(sharpes) - 0.5 * IQR(sharpes)` (robust to outliers)

## Fix 2: tanh Normalization of Composite Score

**File**: `crates/ml/src/hyperopt/adapters/dqn.rs`

Replace raw ratio weighting with tanh-normalized values:

```rust
0.4 * (sortino / 3.0).tanh() +   // linear in [-3,3]
0.3 * (calmar / 5.0).tanh() +    // tames explosion
0.2 * (sharpe / 2.0).tanh() +    // linear in [-2,2]
0.1 * (omega / 2.0).tanh()       // saturates beyond 2
```

Composite now bounded to [-1, 1] regardless of input magnitudes.

## Fix 3: Q-Value Overestimation — Monitoring + Adaptive Tau

**Files**: `crates/ml/src/trainers/dqn/trainer.rs`, `crates/ml/src/training_metrics.rs`

Part A: New Prometheus gauge `foxhunt_training_q_overestimation_ratio` + log warning when ratio > 10 or q_mean > 5.

Part B: Adaptive tau — when Q-mean growth per epoch > 0.5, double tau (capped at 0.01). When stable, decay back to base tau.

## Test Updates

- Update existing `test_extract_objective_*` expected values for tanh normalization
- Add `test_sliding_window_aggregation` with known inputs
- Add `test_tanh_normalization_bounds`
- Add `test_adaptive_tau_triggers`