fix(sp7): flip CQL target_ratio direction — death spiral on magnitude
`target_ratio[CQL][b] = ANCHOR_CQL_RATIO * (1 - flatness[b])` had the sign inverted: it pushed CQL HIGH when Q was flat (the collapse state) and LOW when Q had variance. Per pearl_2_budget_kernel.cu, `flatness = var_q / σ²`, so flatness HIGH means Q has variance — exactly when overconfidence-risk-driven CQL pressure should kick in. Symptom (T10-v3 train-multi-seed-x7sl2 logs): Once IQN Fix 34 woke IQN-mag/ord/urg branches and produced real Q-targets, the controller's inverted formula sent cql_budget to MAX_BUDGET=1.0 saturation on every branch, the conservative pull kept Q flat, the model collapsed to single-Flat-action eval (val_active_frac=0, dir_entropy=0, trade_count=1, sharpe=0). Fix is one operational character: `(1 - flatness)` → `flatness`. C51 formula was already correctly aligned (`flatness * ANCHOR_C51_RATIO`) and unchanged. Per pearl_controller_anchors_isv_driven.md: the inverted formula was masked for months because the IQN-dead regime (Fix 34) suppressed cql_raw on mag/ord/urg, never letting the controller engage with the inverted target. Fixing the IQN regime exposed the dormant formula bug. ANCHOR_CQL_RATIO=2.0 and MAX_BUDGET=1.0 remain hardcoded; Fix 36 will make those ISV-driven via a GPU train_active_frac canary signal per the pearl's "pick the canary that fires under the pathology" rule.
This commit is contained in:
@@ -20,8 +20,21 @@
|
||||
// `apply_pearls_ad_kernel` smooths into ISV).
|
||||
//
|
||||
// Math (per branch b ∈ {dir, mag, ord, urg}, per head h ∈ {CQL, C51}):
|
||||
// target_ratio[CQL][b] = ANCHOR_CQL_RATIO * (1 - flatness[b])
|
||||
// target_ratio[CQL][b] = ANCHOR_CQL_RATIO * flatness[b] (Fix 35, 2026-05-03)
|
||||
// target_ratio[C51][b] = ANCHOR_C51_RATIO * flatness[b]
|
||||
//
|
||||
// Fix 35 (formula direction flip): the original CQL formula used
|
||||
// `(1 - flatness[b])`, which inverted the relationship between Q-distribution
|
||||
// variance and CQL strength. With flatness HIGH meaning "Q has variance"
|
||||
// (per pearl_2_budget_kernel definition `var_q / σ²`), the original formula
|
||||
// pushed CQL HIGH when Q was flat — exactly when there is no overconfidence
|
||||
// for CQL to penalise — creating a death spiral on the magnitude head:
|
||||
// flat Q → CQL boosted → conservative pull keeps Q flat → repeat. Validated
|
||||
// 2026-05-03 in T10-v3 train-multi-seed-x7sl2 logs (cql_budget saturated to
|
||||
// MAX_BUDGET=1.0 on every branch by epoch 1 once IQN-mag/ord/urg woke from
|
||||
// Fix 34, and val collapsed to active_frac=0). The corrected formula
|
||||
// scales CQL with flatness: more variance → more overconfidence risk → more
|
||||
// CQL gradient. C51 formula was already correctly aligned and unchanged.
|
||||
// slice_idx = (b == 0) ? 1 : (b == 1) ? 0 : 2 // dir, mag, trunk
|
||||
// actual_ratio = h_norm[slice] / iqn_norm[slice] (both > EPS_DIV via outer guard)
|
||||
// correction = target_ratio[h][b] / max(actual_ratio, EPS_DIV)
|
||||
@@ -143,8 +156,14 @@ extern "C" __global__ void loss_balance_controller_update(
|
||||
flat_b = fminf(1.0f, fmaxf(0.0f, flat_b)); // safety clamp
|
||||
|
||||
// ── Compute per-branch flatness-modulated target ratio ───────────
|
||||
// Fix 35 (2026-05-03): both heads now scale UP with flatness. When Q has
|
||||
// variance (flatness ≈ 1) overconfidence risk is high → both CQL and C51
|
||||
// budgets grow. When Q is flat (flatness ≈ 0) there is nothing to
|
||||
// penalise / nothing to distribute → both budgets shrink. The prior CQL
|
||||
// formula `(1 - flat_b)` created a death spiral on magnitude (flat Q →
|
||||
// boosted CQL → pulled Q flatter). See kernel header.
|
||||
float target_ratio = (head == 0)
|
||||
? ANCHOR_CQL_RATIO * (1.0f - flat_b)
|
||||
? ANCHOR_CQL_RATIO * flat_b
|
||||
: ANCHOR_C51_RATIO * flat_b;
|
||||
|
||||
// ── Read prior budget + per-branch Wiener state + activation flag ─
|
||||
|
||||
@@ -4308,3 +4308,58 @@ Files: `trainers/dqn/fused_training.rs` (+12 LOC, −1 LOC), this audit
|
||||
entry. No `StateResetRegistry` changes — `cached_target_h_s2_ptr` is
|
||||
trainer-internal scratch state cleared by `.take()` on every IQN call,
|
||||
not an ISV slot.
|
||||
|
||||
## Fix 35 — SP7 controller CQL target_ratio direction flip (2026-05-03)
|
||||
|
||||
**Symptom:** T10-v3 train-multi-seed-x7sl2 (commit 9b5296b2f, IQN Fix 34
|
||||
applied): IQN warnings disappeared (Fix 34 working); BUT post-fix the model
|
||||
flat-collapsed from epoch 2 onward. Per-epoch HEALTH_DIAG showed:
|
||||
- cql_budget_per_branch saturated to MAX_BUDGET=1.0 on every branch
|
||||
- val_active_frac = 0.0 (no trades)
|
||||
- val_dir_entropy = 0.0
|
||||
- val_sharpe = 0.0
|
||||
- q_var_per_branch[mag] = 0.0003 (mag head not differentiating)
|
||||
- cql_raw_mag = 0.5179 (~50× pre-fix value of 0.0157)
|
||||
|
||||
**Root cause:** `loss_balance_controller_kernel.cu` line 23 spec / line ~146
|
||||
implementation had the CQL formula:
|
||||
`target_ratio[CQL][b] = ANCHOR_CQL_RATIO * (1 - flatness[b])`
|
||||
|
||||
In this codebase, `flatness` = `var_q[b] / σ²` per
|
||||
`pearl_2_budget_kernel.cu`. So flatness HIGH means "Q has variance" and
|
||||
flatness LOW means "Q is flat." With the `(1 - flatness)` factor, the
|
||||
formula pushed CQL HIGHER when Q was flat — exactly when there is no
|
||||
overconfidence to penalise. The result was a death spiral on the magnitude
|
||||
head:
|
||||
|
||||
flat Q → CQL boosted → CQL pull keeps Q close to mean → Q stays flat
|
||||
→ controller sees flat Q → boosts CQL further → ad infinitum
|
||||
|
||||
Pre-fix the spiral was masked because IQN-mag/ord/urg silently no-op'd
|
||||
(Fix 34) — no real Q-targets meant no overconfidence to penalise meant
|
||||
small cql_raw, and the controller never engaged on those branches. Once
|
||||
Fix 34 woke IQN-mag/ord/urg, real Q-targets produced large cql_raw, the
|
||||
controller engaged with the inverted formula, and the spiral fired
|
||||
immediately.
|
||||
|
||||
**Fix:** flip the formula direction so CQL target scales WITH flatness:
|
||||
`target_ratio[CQL][b] = ANCHOR_CQL_RATIO * flatness[b]`
|
||||
|
||||
When Q has variance (flatness ≈ 1) overconfidence risk is real → grow CQL.
|
||||
When Q is flat (flatness ≈ 0) there is nothing to penalise → shrink CQL.
|
||||
This aligns CQL's controller direction with what CQL semantically does
|
||||
(penalise overconfident Q estimates).
|
||||
|
||||
**C51 unchanged:** the C51 formula `flatness * ANCHOR_C51_RATIO` was already
|
||||
correctly aligned (distributional learning is informative when the
|
||||
distribution has structure to learn from).
|
||||
|
||||
**Per `pearl_controller_anchors_isv_driven.md`:** the deeper lesson is
|
||||
that controller anchors silently encode the regime in which they were
|
||||
tuned. The pre-Fix-34 IQN-dead regime hid the formula's incorrect sign;
|
||||
fixing the regime exposed it. ANCHOR_CQL_RATIO=2.0 and MAX_BUDGET=1.0
|
||||
remain hardcoded constants — these are addressed in Fix 36 (ISV-driven
|
||||
adaptive cap based on GPU-computed train_active_frac canary).
|
||||
|
||||
Files: `cuda_pipeline/loss_balance_controller_kernel.cu` (+15 LOC
|
||||
comments, 1 line behavior change), this audit entry.
|
||||
|
||||
Reference in New Issue
Block a user