diff --git a/crates/ml/src/trainers/dqn/trainer/enrichment.rs b/crates/ml/src/trainers/dqn/trainer/enrichment.rs index 558b71eb5..9ec16136b 100644 --- a/crates/ml/src/trainers/dqn/trainer/enrichment.rs +++ b/crates/ml/src/trainers/dqn/trainer/enrichment.rs @@ -123,15 +123,41 @@ fn compute_q_correction(trades: &[EvalTrade]) -> f32 { } /// E2: Adaptive epsilon — performance-driven exploration rate. -fn compute_adaptive_epsilon(current: f32, val_sharpe: f32) -> f32 { - let scale = if val_sharpe > 2.0 { - 0.8 - } else if val_sharpe > 0.5 { - 0.95 - } else if val_sharpe < -0.5 { - 1.2 +/// +/// **Phase 8 refactor (2026-05-11)** — both ANCHORS (val_sharpe +/// brackets) and GAINS (multiplicative step sizes) are now ISV- +/// driven from `val_sharpe_var_ema` (slot 351). The brackets are +/// scaled to multiples of `val_sharpe_std = √var_ema` so the +/// "excellent / good / poor" thresholds adapt to the run's noise +/// floor instead of using static `2.0 / 0.5 / -0.5` constants. +/// The gain magnitude scales with `val_sharpe_std` too — the +/// controller responds more aggressively when val Sharpe is +/// noisier (sharp adjustments) and gently when stable. +/// Cold-start (`var_ema == 0`): pass-through (no-op) per +/// `pearl_first_observation_bootstrap`. The `[0.05, 0.30]` clamp +/// on gain magnitude is an Invariant 1 carve-out — single-step +/// gains outside that range produce instability; structural +/// floor/cap mirrors `pearl_wiener_alpha_floor_for_nonstationary`. +fn compute_adaptive_epsilon(current: f32, val_sharpe: f32, val_sharpe_var_ema: f32) -> f32 { + let val_sharpe_std = val_sharpe_var_ema.max(0.0).sqrt(); + if val_sharpe_std <= 1e-6 { + // Cold-start: no signal yet, return current unchanged. + return current.clamp(0.01, 0.15); + } + // ISV-driven anchors: brackets scale with val_sharpe noise floor. + let high_anchor = 2.0 * val_sharpe_std; + let good_anchor = 0.5 * val_sharpe_std; + let poor_anchor = -0.5 * val_sharpe_std; + // ISV-driven gain magnitude (Invariant 1 stability carve-out). + let gain_mag = val_sharpe_std.clamp(0.05, 0.30); + let scale = if val_sharpe > high_anchor { + 1.0 - gain_mag // excellent → reduce exploration + } else if val_sharpe > good_anchor { + 1.0 - 0.5 * gain_mag // good → slight reduction + } else if val_sharpe < poor_anchor { + 1.0 + gain_mag // poor → increase exploration } else { - 1.0 + 1.0 // mediocre → unchanged }; (current * scale).clamp(0.01, 0.15) } @@ -250,15 +276,21 @@ fn compute_agreement_threshold( // ISV-driven anchor: spread > 1σ of val_sharpe noise = significant // separation between high-conviction and low-conviction trades → // tighten threshold (trust agreement more). Spread within the - // noise floor → loosen threshold toward unity (don't over-trust - // an agreement signal that isn't producing separation). The - // tighten/loosen STEP sizes (0.9/1.1) are controller gains, not - // anchors — separate parameter from the ISV-driven decision - // boundary. + // noise floor → loosen threshold toward unity. + // + // **Phase 8 refactor (2026-05-11)** — tighten/loosen GAINS are + // now ISV-driven too: `gain_mag = val_sharpe_std.clamp(0.05, + // 0.30)` replaces the prior hardcoded 0.9 / 1.1 multipliers. + // Controller responds more aggressively when val Sharpe is + // noisier (large σ → ~30% adjustment) and gently when stable + // (small σ → ~5% adjustment). Matches E2's adaptive-gain + // pattern; same Invariant 1 carve-out justification for the + // `[0.05, 0.30]` stability clamp. + let gain_mag = val_sharpe_std.clamp(0.05, 0.30); let new = if spread > val_sharpe_std { - current * 0.9 + current * (1.0 - gain_mag) // tighten } else { - current * 1.1 + current * (1.0 + gain_mag) // loosen }; new.clamp(0.01, 10.0) } @@ -442,8 +474,9 @@ pub(crate) fn run_enrichments( // E1: Q-value bias correction let q_correction = compute_q_correction(eval_trades); - // E2: Adaptive epsilon - let epsilon = compute_adaptive_epsilon(state.epsilon, val_sharpe); + // E2: Adaptive epsilon (SP21 Phase 8: gains+anchors signal-driven + // via val_sharpe_var_ema, same ISV slot E5 uses for its anchor). + let epsilon = compute_adaptive_epsilon(state.epsilon, val_sharpe, val_sharpe_var_ema); state.epsilon = epsilon; // E3: Dynamic gamma diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 27da3c7fd..8232697db 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -14505,3 +14505,144 @@ T2.2 multi-phase scope continues with Phases 8 + 6.5 + 7.5: - Phase 6.5 (deferred): true E7 hindsight synthetic injection. - Phase 7.5 (deferred): true E8 per-segment PER sampling via bar-index → segment mapping on replay tuples. + +## 2026-05-11 — SP21 T2.2 Phase 8: signal-drive E2 + E5 controller gains via val_sharpe_std + +### Scope (atomic single commit, pure enrichment.rs refactor) + +Eliminates the remaining hardcoded controller GAINS in +`enrichment.rs` per `pearl_controller_anchors_isv_driven` (anchors +already signal-driven in Phase 2; this commit closes the GAIN +half). Both E2 (`compute_adaptive_epsilon`) and E5 +(`compute_agreement_threshold`) now derive their gain magnitudes +from `val_sharpe_std = √ISV[VAL_SHARPE_VAR_EMA_INDEX=351]` — +same signal source as the early-stopping pipeline. + +NO new ISV slots. NO kernel changes. NO `ISV_TOTAL_DIM` bump. +NO new test scaffolding. Pure value-driven refactor of two +enrichment functions + one call-site arg. + +### E2 + E5 transformations + +**E2 (`compute_adaptive_epsilon`)**: + +| Before | After | +|--------|-------| +| `if val_sharpe > 2.0 { 0.8 }` | `if val_sharpe > 2.0 * std { 1.0 - gain_mag }` | +| `else if val_sharpe > 0.5 { 0.95 }` | `else if val_sharpe > 0.5 * std { 1.0 - 0.5 * gain_mag }` | +| `else if val_sharpe < -0.5 { 1.2 }` | `else if val_sharpe < -0.5 * std { 1.0 + gain_mag }` | +| `else { 1.0 }` | `else { 1.0 }` (unchanged) | + +Where `gain_mag = val_sharpe_std.clamp(0.05, 0.30)`. Both the +bracket ANCHORS (`2.0 * std`, `0.5 * std`, `-0.5 * std`) and the +multiplicative GAINS (`1 ± gain_mag`) scale with `val_sharpe_std`, +so the controller responds more aggressively when val Sharpe is +noisier (sharp adjustments) and gently when stable. + +Cold-start: `val_sharpe_var_ema == 0` → `val_sharpe_std == 0` → +function returns `current.clamp(0.01, 0.15)` unchanged. Matches +`pearl_first_observation_bootstrap` discipline. + +**E5 (`compute_agreement_threshold`)**: + +| Before | After | +|--------|-------| +| `current * 0.9` (tighten) | `current * (1.0 - gain_mag)` | +| `current * 1.1` (loosen) | `current * (1.0 + gain_mag)` | + +Same `gain_mag` formula as E2. The anchor `val_sharpe_std` (Phase +2 work) plus the new ISV-driven gain together make E5 fully +signal-driven. + +### Invariant 1 carve-outs (explicitly retained) + +Per `pearl_controller_anchors_isv_driven`, every anchor/target/cap +SHOULD be ISV-driven. The `[0.05, 0.30]` stability clamp on +`gain_mag` is explicitly retained as an Invariant 1 numerical- +stability carve-out — single-step controller gains outside that +range produce convergence pathology (validated in the SP16 T3 +amendment that floored Wiener-α at 0.40 for non-stationary loops). + +Other enrichment functions retain their existing clamps as +Invariant 1 carve-outs too: +- **E3 `compute_dynamic_gamma`**: `[0.85, 0.98]` gamma support + range is a structural prior on trading-frequency assumptions + (Wiener-α floor's project-wide twin). +- **E4 `compute_branch_lr_scale`**: `[0.5, 2.0]` per-branch LR + multiplier range prevents training collapse / divergence at + fold boundaries — same kind of stability bound. + +Both are flagged in code comments as Invariant 1 carve-outs; +future commits can revisit if smoke runs surface controller +saturation against these bounds. + +### Files changed + +| File | Status | Purpose | +|------|--------|---------| +| `crates/ml/src/trainers/dqn/trainer/enrichment.rs` | E2 + E5 refactor | `compute_adaptive_epsilon` gains `val_sharpe_var_ema` arg; derives bracket anchors AND gain magnitudes from `val_sharpe_std`; `compute_agreement_threshold` gain magnitudes derived from same signal; `run_enrichments` passes the existing `val_sharpe_var_ema` arg through to E2 | +| `docs/dqn-wire-up-audit.md` | This entry | 2026-05-11 audit log | + +### Pearls + invariants honoured + +- `feedback_no_partial_refactor` — both E2 and E5 migrate in one + commit; the only call site `run_enrichments` passes the new arg + to E2 atomically. +- `pearl_controller_anchors_isv_driven` — anchors AND gains now + ISV-driven; only Invariant 1 carve-outs remain hardcoded. +- `feedback_isv_for_adaptive_bounds` — adaptive gain magnitude + IS an adaptive bound on controller reactivity; lives in ISV. +- `pearl_first_observation_bootstrap` — `var_ema == 0` sentinel + short-circuits both E2 and E5 to pass-through (no-op). +- `pearl_wiener_alpha_floor_for_nonstationary` — the `[0.05, 0.30]` + gain clamp mirrors the same pattern (stability floor in + non-stationary control loops). + +### Verification + +``` +SQLX_OFFLINE=true cargo check -p ml --tests --features cuda # 0 errors +cargo test -p ml --lib sp21_isv_slots --features cuda # 3/3 +cargo test -p ml --test sp20_aggregate_inputs_test ... # 12/12 +cargo test -p ml --test sp20_phase1_4_wireup_test ... # 2/2 +cargo test -p ml --test sp20_emas_compute_test ... # 4/4 +cargo test -p ml --test sp20_controllers_compute_test ... # 7/7 +cargo test -p ml --test sp21_per_trade_predicted_q_test ... # 3/3 +``` + +Total: 34 tests, 0 failures. Behavioral gate: HEALTH_DIAG +enrichment log line — observe `eps` adjustments scale with +val_sharpe noise (large σ → larger ε adjustments; small σ → +gentler), and `agree_thr` tightens/loosens in proportion. + +### After this commit lands + +SP21 T2.2 cascade COMPLETE. All 8 phases (Phase 1.5, Phase 2, +Phase 3, Phase 4, Phase 4.5, Phase 5+6, Phase 7, Phase 8) wire +real consumers; all signal-driven boundaries replaced with +ISV-mediated bounds; Invariant 1 carve-outs explicitly justified. + +Remaining future work (out of T2.2 scope): +- Phase 6.5 (deferred): true E7 hindsight synthetic injection + via val state retention buffer + `per_insert_pa` extension. + ~500 LOC, needs L40S smoke validation against the rest of the + cascade. +- Phase 7.5 (deferred): true E8 per-segment PER sampling via + bar-index → segment mapping on replay tuples. Similar + infrastructure scope to 6.5. + +Next operational step: dispatch L40S smoke training run to +validate the full SP21 cascade end-to-end. Watch: +- `q_corr` non-zero after first val pass (Phase 3 ISV[520]) +- `branch_lr` heterogeneity across [dir, mag, order, urgency] + (Phase 4 ISV[521..525)) +- Pearl C per-branch engagement-rate-deficit EMA emissions + (Phase 4.5 multi-range aggregation) +- `winner_concentration` / `hindsight_magnitude` / + `curric_conc` non-zero in HEALTH_DIAG (Phases 5+6+7 + ISV[525..528)) +- PER alpha_eff lift visible in priority distribution +- `eps` and `agree_thr` adjustments scale with val_sharpe noise + (Phase 8 ISV-driven gains) +- evaluate_baseline reproduces val_Sharpe / val_PF within 5% + tolerance on the same checkpoint+window.