Smoke smoke-test-ftdjz (commit d9a4d98a3, Mech 6 + Mech 7) regressed
both F0 (44 -> 38.82, per-element cap clipped legitimate outliers) and
F1 (grad-collapse at step 2040 vs 3720 with Mech 6 alone). Slots 36-42
STILL fired — Mech 7's per-element clip didn't prevent Adam EMA
saturation, just slowed training to grad-collapse.
Re-analysis: Mech 6's 100x multiplier on the upper-bound formula was
mismatched with the slot 36 threshold ratio. Per-element gradient max
<= adaptive_clip = 100 x slow_ema x isv ~= 200. Adam m_X steady-state
reaches 200, exceeding slot 36 threshold of 100*isv = 100. Mech 6 was
doing its job but the bound was wider than the diagnostic threshold.
Two coordinated changes (per feedback_no_partial_refactor):
1. REVERT Mech 7 (per-element clip in dqn_adam_update_kernel). The
per-element approach was misdiagnosis — clipping post-global-clip
gradients tighter than legitimate per-element variance harms F0
training without addressing Adam saturation root cause. Kernel
returns to its post-Mech-6 state (blob 546feee48).
2. TIGHTEN Mech 6's upper-bound multiplier from 100x to 5x. Standard
DL practice (5-10x steady-state grad norm). Per-element gradient
max becomes <= 5 x slow_ema x isv ~= 10, well below slot 36
threshold of 100. Adam m_X EMA stays bounded <= 10 — slots 36-42
should not fire.
Why 5x and not 10x: slot 36 threshold is 100 x isv. 5x slow_ema
(~=10 absolute) leaves 10x headroom against the threshold, providing
robust margin. 10x slow_ema would be 20 absolute, only 5x margin —
risk of fluctuations triggering slot 36.
Why not tighter (e.g., 2x): per-step gradient norms can legitimately
spike to 5x slow_ema in normal training (e.g., gradient resumption
after warmup); tighter bounds would over-clip.
F0 risk: low — F0 typical adaptive_clip values are bounded by Mech 6's
upper anchor only when the EMA-driven clip exceeds 5x slow_ema, which
is rare in steady F0 training. Cap should be a no-op for F0.
F1 risk: prevents the saturation pathway diagnosed by smoke
smoke-test-ftdjz. Validates by running the next smoke at this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
f67ede94fb