fix(sp7): flip CQL target_ratio direction — death spiral on magnitude

`target_ratio[CQL][b] = ANCHOR_CQL_RATIO * (1 - flatness[b])` had the
sign inverted: it pushed CQL HIGH when Q was flat (the collapse state)
and LOW when Q had variance. Per pearl_2_budget_kernel.cu, `flatness =
var_q / σ²`, so flatness HIGH means Q has variance — exactly when
overconfidence-risk-driven CQL pressure should kick in.

Symptom (T10-v3 train-multi-seed-x7sl2 logs): Once IQN Fix 34 woke
IQN-mag/ord/urg branches and produced real Q-targets, the controller's
inverted formula sent cql_budget to MAX_BUDGET=1.0 saturation on every
branch, the conservative pull kept Q flat, the model collapsed to
single-Flat-action eval (val_active_frac=0, dir_entropy=0,
trade_count=1, sharpe=0).

Fix is one operational character: `(1 - flatness)` → `flatness`. C51
formula was already correctly aligned (`flatness * ANCHOR_C51_RATIO`)
and unchanged.

Per pearl_controller_anchors_isv_driven.md: the inverted formula was
masked for months because the IQN-dead regime (Fix 34) suppressed
cql_raw on mag/ord/urg, never letting the controller engage with the
inverted target. Fixing the IQN regime exposed the dormant formula bug.

ANCHOR_CQL_RATIO=2.0 and MAX_BUDGET=1.0 remain hardcoded; Fix 36 will
make those ISV-driven via a GPU train_active_frac canary signal per
the pearl's "pick the canary that fires under the pathology" rule.
This commit is contained in:
jgrusewski
2026-05-03 19:00:26 +02:00
parent 9b5296b2f0
commit 65c3083dec
2 changed files with 76 additions and 2 deletions

View File

@@ -20,8 +20,21 @@
// `apply_pearls_ad_kernel` smooths into ISV).
//
// Math (per branch b ∈ {dir, mag, ord, urg}, per head h ∈ {CQL, C51}):
// target_ratio[CQL][b] = ANCHOR_CQL_RATIO * (1 - flatness[b])
// target_ratio[CQL][b] = ANCHOR_CQL_RATIO * flatness[b] (Fix 35, 2026-05-03)
// target_ratio[C51][b] = ANCHOR_C51_RATIO * flatness[b]
//
// Fix 35 (formula direction flip): the original CQL formula used
// `(1 - flatness[b])`, which inverted the relationship between Q-distribution
// variance and CQL strength. With flatness HIGH meaning "Q has variance"
// (per pearl_2_budget_kernel definition `var_q / σ²`), the original formula
// pushed CQL HIGH when Q was flat — exactly when there is no overconfidence
// for CQL to penalise — creating a death spiral on the magnitude head:
// flat Q → CQL boosted → conservative pull keeps Q flat → repeat. Validated
// 2026-05-03 in T10-v3 train-multi-seed-x7sl2 logs (cql_budget saturated to
// MAX_BUDGET=1.0 on every branch by epoch 1 once IQN-mag/ord/urg woke from
// Fix 34, and val collapsed to active_frac=0). The corrected formula
// scales CQL with flatness: more variance → more overconfidence risk → more
// CQL gradient. C51 formula was already correctly aligned and unchanged.
// slice_idx = (b == 0) ? 1 : (b == 1) ? 0 : 2 // dir, mag, trunk
// actual_ratio = h_norm[slice] / iqn_norm[slice] (both > EPS_DIV via outer guard)
// correction = target_ratio[h][b] / max(actual_ratio, EPS_DIV)
@@ -143,8 +156,14 @@ extern "C" __global__ void loss_balance_controller_update(
flat_b = fminf(1.0f, fmaxf(0.0f, flat_b)); // safety clamp
// ── Compute per-branch flatness-modulated target ratio ───────────
// Fix 35 (2026-05-03): both heads now scale UP with flatness. When Q has
// variance (flatness ≈ 1) overconfidence risk is high → both CQL and C51
// budgets grow. When Q is flat (flatness ≈ 0) there is nothing to
// penalise / nothing to distribute → both budgets shrink. The prior CQL
// formula `(1 - flat_b)` created a death spiral on magnitude (flat Q →
// boosted CQL → pulled Q flatter). See kernel header.
float target_ratio = (head == 0)
? ANCHOR_CQL_RATIO * (1.0f - flat_b)
? ANCHOR_CQL_RATIO * flat_b
: ANCHOR_C51_RATIO * flat_b;
// ── Read prior budget + per-branch Wiener state + activation flag ─

View File

@@ -4308,3 +4308,58 @@ Files: `trainers/dqn/fused_training.rs` (+12 LOC, 1 LOC), this audit
entry. No `StateResetRegistry` changes — `cached_target_h_s2_ptr` is
trainer-internal scratch state cleared by `.take()` on every IQN call,
not an ISV slot.
## Fix 35 — SP7 controller CQL target_ratio direction flip (2026-05-03)
**Symptom:** T10-v3 train-multi-seed-x7sl2 (commit 9b5296b2f, IQN Fix 34
applied): IQN warnings disappeared (Fix 34 working); BUT post-fix the model
flat-collapsed from epoch 2 onward. Per-epoch HEALTH_DIAG showed:
- cql_budget_per_branch saturated to MAX_BUDGET=1.0 on every branch
- val_active_frac = 0.0 (no trades)
- val_dir_entropy = 0.0
- val_sharpe = 0.0
- q_var_per_branch[mag] = 0.0003 (mag head not differentiating)
- cql_raw_mag = 0.5179 (~50× pre-fix value of 0.0157)
**Root cause:** `loss_balance_controller_kernel.cu` line 23 spec / line ~146
implementation had the CQL formula:
`target_ratio[CQL][b] = ANCHOR_CQL_RATIO * (1 - flatness[b])`
In this codebase, `flatness` = `var_q[b] / σ²` per
`pearl_2_budget_kernel.cu`. So flatness HIGH means "Q has variance" and
flatness LOW means "Q is flat." With the `(1 - flatness)` factor, the
formula pushed CQL HIGHER when Q was flat — exactly when there is no
overconfidence to penalise. The result was a death spiral on the magnitude
head:
flat Q → CQL boosted → CQL pull keeps Q close to mean → Q stays flat
→ controller sees flat Q → boosts CQL further → ad infinitum
Pre-fix the spiral was masked because IQN-mag/ord/urg silently no-op'd
(Fix 34) — no real Q-targets meant no overconfidence to penalise meant
small cql_raw, and the controller never engaged on those branches. Once
Fix 34 woke IQN-mag/ord/urg, real Q-targets produced large cql_raw, the
controller engaged with the inverted formula, and the spiral fired
immediately.
**Fix:** flip the formula direction so CQL target scales WITH flatness:
`target_ratio[CQL][b] = ANCHOR_CQL_RATIO * flatness[b]`
When Q has variance (flatness ≈ 1) overconfidence risk is real → grow CQL.
When Q is flat (flatness ≈ 0) there is nothing to penalise → shrink CQL.
This aligns CQL's controller direction with what CQL semantically does
(penalise overconfident Q estimates).
**C51 unchanged:** the C51 formula `flatness * ANCHOR_C51_RATIO` was already
correctly aligned (distributional learning is informative when the
distribution has structure to learn from).
**Per `pearl_controller_anchors_isv_driven.md`:** the deeper lesson is
that controller anchors silently encode the regime in which they were
tuned. The pre-Fix-34 IQN-dead regime hid the formula's incorrect sign;
fixing the regime exposed it. ANCHOR_CQL_RATIO=2.0 and MAX_BUDGET=1.0
remain hardcoded constants — these are addressed in Fix 36 (ISV-driven
adaptive cap based on GPU-computed train_active_frac canary).
Files: `cuda_pipeline/loss_balance_controller_kernel.cu` (+15 LOC
comments, 1 line behavior change), this audit entry.