Cluster smoke `alpha-rl-nqd68` showed the signal-driven LR controller
working mechanically but destabilising the π head: lr_pi swung
MIN→MAX (1000×) over ~10k steps, then got stuck at MAX after the
catastrophic Adam updates wrecked the policy weights. Aggregate
l_pi max = 2.4e17, l_v max = 2,083,330 (no NaN abort, but useless
for learning). Q head was fine (lr_q correctly stuck at MIN throughout,
l_q dropped 34% vs fixed-LR).
## Two fixes
### 1. Per-step rate-of-change cap
The original target formula `target = lr_prev × (TARGET/observed)`
allows arbitrary swing magnitude. When `observed` is tiny (e.g.
quiescent π grad-norm between trade closes), target = lr_prev × 1000,
which the Wiener α=0.4 blend drags toward LR_MAX in a few steps.
Once at MAX, the next real reward signal applies catastrophic Adam
updates → policy explodes → grad-norm spikes 10⁵× → controller
sees this and tries to shrink, but the damage is done.
Adds `target_lr ∈ [lr_prev × 0.5, lr_prev × 2.0]` constraint
post-formula, pre-clamp. The controller can now at most halve or
double LR per step, taking ~10 steps to traverse the full
[LR_MIN, LR_MAX] range. Downstream gradient signal has time to
react before LR overshoots.
Same pattern as `rl_rollout_steps_controller`'s
`scale ∈ [0.5, 2.0]` cap (which was added for the same class of
multiplicative-controller instability).
### 2. Per-head TARGET_GRAD_NORM
The single `TARGET_GRAD_NORM = 1.0` anchor was wrong for π and V:
those heads have far fewer parameters than Q (1,152 and 128 vs
24,192). A "well-tuned" grad-norm magnitude scales with √n_params
(so per-parameter grad magnitude stays Adam-friendly ≈ 1e-2).
Q head w_d: 9 × 21 × 128 = 24,192 params → √ ≈ 156 → target 1.5
π head w_d: 9 × 128 = 1,152 params → √ ≈ 34 → target 0.3
V head w_d: 128 = 128 params → √ ≈ 11 → target 0.1
Without this scaling, the controller was pushing π LR up because
its grad-norm (typically 0.1-0.3) was always "below the 1.0 target"
— interpreted as "model coasting, grow LR" when really the smaller
grad-norm just reflected the smaller parameter count.
`update_lr_with_signal` now takes `head_target_grad_norm` as a
parameter. BCE and AUX heads (owned by perception, signal_slot=-1)
get target=1.0 but it's unused because the early-return at
`signal_slot < 0` short-circuits past the target derivation.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers_emit ✅
G4 target_soft_update ✅
G6 r7d_per_wiring ✅
smoke ✅ all losses finite
## Expected effect on next 50k smoke
* lr_q stays near MIN (already worked — Q grad-norm > target_Q
typically) — unchanged.
* lr_pi should NOT runaway to MAX — rate cap limits 1000× swing
to at most 2× per step; per-head π target 0.3 puts the
multiplicative ratio closer to 1.0 (no extreme target).
* lr_v should also stabilise via the V-specific target 0.1.
* Aggregate l_pi / l_v max values should drop from the 1e17 / 2e6
range to O(1).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
e074c91fb2