From 65c3083dec173d0688fff84d74ab7211eaccf42e Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 3 May 2026 19:00:26 +0200 Subject: [PATCH] =?UTF-8?q?fix(sp7):=20flip=20CQL=20target=5Fratio=20direc?= =?UTF-8?q?tion=20=E2=80=94=20death=20spiral=20on=20magnitude?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `target_ratio[CQL][b] = ANCHOR_CQL_RATIO * (1 - flatness[b])` had the sign inverted: it pushed CQL HIGH when Q was flat (the collapse state) and LOW when Q had variance. Per pearl_2_budget_kernel.cu, `flatness = var_q / σ²`, so flatness HIGH means Q has variance — exactly when overconfidence-risk-driven CQL pressure should kick in. Symptom (T10-v3 train-multi-seed-x7sl2 logs): Once IQN Fix 34 woke IQN-mag/ord/urg branches and produced real Q-targets, the controller's inverted formula sent cql_budget to MAX_BUDGET=1.0 saturation on every branch, the conservative pull kept Q flat, the model collapsed to single-Flat-action eval (val_active_frac=0, dir_entropy=0, trade_count=1, sharpe=0). Fix is one operational character: `(1 - flatness)` → `flatness`. C51 formula was already correctly aligned (`flatness * ANCHOR_C51_RATIO`) and unchanged. Per pearl_controller_anchors_isv_driven.md: the inverted formula was masked for months because the IQN-dead regime (Fix 34) suppressed cql_raw on mag/ord/urg, never letting the controller engage with the inverted target. Fixing the IQN regime exposed the dormant formula bug. ANCHOR_CQL_RATIO=2.0 and MAX_BUDGET=1.0 remain hardcoded; Fix 36 will make those ISV-driven via a GPU train_active_frac canary signal per the pearl's "pick the canary that fires under the pathology" rule. --- .../loss_balance_controller_kernel.cu | 23 +++++++- docs/dqn-wire-up-audit.md | 55 +++++++++++++++++++ 2 files changed, 76 insertions(+), 2 deletions(-) diff --git a/crates/ml/src/cuda_pipeline/loss_balance_controller_kernel.cu b/crates/ml/src/cuda_pipeline/loss_balance_controller_kernel.cu index fda09bc1c..f0c40ebae 100644 --- a/crates/ml/src/cuda_pipeline/loss_balance_controller_kernel.cu +++ b/crates/ml/src/cuda_pipeline/loss_balance_controller_kernel.cu @@ -20,8 +20,21 @@ // `apply_pearls_ad_kernel` smooths into ISV). // // Math (per branch b ∈ {dir, mag, ord, urg}, per head h ∈ {CQL, C51}): -// target_ratio[CQL][b] = ANCHOR_CQL_RATIO * (1 - flatness[b]) +// target_ratio[CQL][b] = ANCHOR_CQL_RATIO * flatness[b] (Fix 35, 2026-05-03) // target_ratio[C51][b] = ANCHOR_C51_RATIO * flatness[b] +// +// Fix 35 (formula direction flip): the original CQL formula used +// `(1 - flatness[b])`, which inverted the relationship between Q-distribution +// variance and CQL strength. With flatness HIGH meaning "Q has variance" +// (per pearl_2_budget_kernel definition `var_q / σ²`), the original formula +// pushed CQL HIGH when Q was flat — exactly when there is no overconfidence +// for CQL to penalise — creating a death spiral on the magnitude head: +// flat Q → CQL boosted → conservative pull keeps Q flat → repeat. Validated +// 2026-05-03 in T10-v3 train-multi-seed-x7sl2 logs (cql_budget saturated to +// MAX_BUDGET=1.0 on every branch by epoch 1 once IQN-mag/ord/urg woke from +// Fix 34, and val collapsed to active_frac=0). The corrected formula +// scales CQL with flatness: more variance → more overconfidence risk → more +// CQL gradient. C51 formula was already correctly aligned and unchanged. // slice_idx = (b == 0) ? 1 : (b == 1) ? 0 : 2 // dir, mag, trunk // actual_ratio = h_norm[slice] / iqn_norm[slice] (both > EPS_DIV via outer guard) // correction = target_ratio[h][b] / max(actual_ratio, EPS_DIV) @@ -143,8 +156,14 @@ extern "C" __global__ void loss_balance_controller_update( flat_b = fminf(1.0f, fmaxf(0.0f, flat_b)); // safety clamp // ── Compute per-branch flatness-modulated target ratio ─────────── + // Fix 35 (2026-05-03): both heads now scale UP with flatness. When Q has + // variance (flatness ≈ 1) overconfidence risk is high → both CQL and C51 + // budgets grow. When Q is flat (flatness ≈ 0) there is nothing to + // penalise / nothing to distribute → both budgets shrink. The prior CQL + // formula `(1 - flat_b)` created a death spiral on magnitude (flat Q → + // boosted CQL → pulled Q flatter). See kernel header. float target_ratio = (head == 0) - ? ANCHOR_CQL_RATIO * (1.0f - flat_b) + ? ANCHOR_CQL_RATIO * flat_b : ANCHOR_C51_RATIO * flat_b; // ── Read prior budget + per-branch Wiener state + activation flag ─ diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 1a69628cc..1b81dd7a4 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -4308,3 +4308,58 @@ Files: `trainers/dqn/fused_training.rs` (+12 LOC, −1 LOC), this audit entry. No `StateResetRegistry` changes — `cached_target_h_s2_ptr` is trainer-internal scratch state cleared by `.take()` on every IQN call, not an ISV slot. + +## Fix 35 — SP7 controller CQL target_ratio direction flip (2026-05-03) + +**Symptom:** T10-v3 train-multi-seed-x7sl2 (commit 9b5296b2f, IQN Fix 34 +applied): IQN warnings disappeared (Fix 34 working); BUT post-fix the model +flat-collapsed from epoch 2 onward. Per-epoch HEALTH_DIAG showed: + - cql_budget_per_branch saturated to MAX_BUDGET=1.0 on every branch + - val_active_frac = 0.0 (no trades) + - val_dir_entropy = 0.0 + - val_sharpe = 0.0 + - q_var_per_branch[mag] = 0.0003 (mag head not differentiating) + - cql_raw_mag = 0.5179 (~50× pre-fix value of 0.0157) + +**Root cause:** `loss_balance_controller_kernel.cu` line 23 spec / line ~146 +implementation had the CQL formula: + `target_ratio[CQL][b] = ANCHOR_CQL_RATIO * (1 - flatness[b])` + +In this codebase, `flatness` = `var_q[b] / σ²` per +`pearl_2_budget_kernel.cu`. So flatness HIGH means "Q has variance" and +flatness LOW means "Q is flat." With the `(1 - flatness)` factor, the +formula pushed CQL HIGHER when Q was flat — exactly when there is no +overconfidence to penalise. The result was a death spiral on the magnitude +head: + + flat Q → CQL boosted → CQL pull keeps Q close to mean → Q stays flat + → controller sees flat Q → boosts CQL further → ad infinitum + +Pre-fix the spiral was masked because IQN-mag/ord/urg silently no-op'd +(Fix 34) — no real Q-targets meant no overconfidence to penalise meant +small cql_raw, and the controller never engaged on those branches. Once +Fix 34 woke IQN-mag/ord/urg, real Q-targets produced large cql_raw, the +controller engaged with the inverted formula, and the spiral fired +immediately. + +**Fix:** flip the formula direction so CQL target scales WITH flatness: + `target_ratio[CQL][b] = ANCHOR_CQL_RATIO * flatness[b]` + +When Q has variance (flatness ≈ 1) overconfidence risk is real → grow CQL. +When Q is flat (flatness ≈ 0) there is nothing to penalise → shrink CQL. +This aligns CQL's controller direction with what CQL semantically does +(penalise overconfident Q estimates). + +**C51 unchanged:** the C51 formula `flatness * ANCHOR_C51_RATIO` was already +correctly aligned (distributional learning is informative when the +distribution has structure to learn from). + +**Per `pearl_controller_anchors_isv_driven.md`:** the deeper lesson is +that controller anchors silently encode the regime in which they were +tuned. The pre-Fix-34 IQN-dead regime hid the formula's incorrect sign; +fixing the regime exposed it. ANCHOR_CQL_RATIO=2.0 and MAX_BUDGET=1.0 +remain hardcoded constants — these are addressed in Fix 36 (ISV-driven +adaptive cap based on GPU-computed train_active_frac canary). + +Files: `cuda_pipeline/loss_balance_controller_kernel.cu` (+15 LOC +comments, 1 line behavior change), this audit entry.