Task 0.4's grad_ratio_mag_dir returns 0.0 whenever dir_norm < 1e-9, so
the epoch-end reading of 0.0000 doesn't disambiguate "magnitude
starved" vs "direction starved". Task 2.0's per-component data showed
CQL and C51 each sending 100-400x more gradient to magnitude than
direction — implying direction is the starved one, not magnitude.
This adds a HEALTH_DIAG field exposing the raw absolute norms:
grad_abs [dir=<sci-notation> mag=<sci-notation>]
Along the way uncovered + fixed two latent bugs that had been silently
zeroing the ratio signal since Task 0.4 landed:
1. `per_branch_grad_norms` read `grad_buf.len()` = total_params +
cutlass_tile_pad (~4096 elements of GEMM tile padding) but the
pinned readback slot was sized at construction to total_params
exactly. The size check `grad_len > grad_readback_pinned_capacity`
was always true, so the accessor returned Err on every call — and
FusedTrainingCtx's proxy coerced Err to 0.0 via `.unwrap_or(0.0)`.
Root cause for the "always 0.0000" grad_ratio_mag_dir. Fix: read
only the first `total_params` prefix of grad_buf (the tail is pure
GEMM padding, never holds gradient values).
2. Readback timing: process_epoch_boundary calls
estimate_avg_q_value_with_early_stopping early, which replays
`eval_forward_exec` — the SAME captured graph as forward_child
whose first op is `cuMemsetD32Async(grad_buf, 0, total_params)`.
Any grad_buf readback AFTER the avg_q call sees all zeros. Fix:
snapshot grad_dir_abs / grad_mag_abs / grad_ratio_mag_dir at the
TOP of process_epoch_boundary, before avg_q runs, and consume the
cached values in the HEALTH_DIAG block.
Last 5 epochs of fold 3 on the magnitude_distribution baseline smoke
(FOXHUNT_TEST_DATA=test_data/futures-baseline):
HEALTH_DIAG[15] ratio=42.85 grad_abs [dir=6.804900e0 mag=3.989081e2]
HEALTH_DIAG[16] ratio=232.66 grad_abs [dir=6.500046e0 mag=3.603353e0]
HEALTH_DIAG[17] ratio=318.34 grad_abs [dir=2.720317e-2 mag=1.713861e1]
HEALTH_DIAG[18] ratio=367.59 grad_abs [dir=5.180866e-2 mag=8.076681e0]
HEALTH_DIAG[19] ratio=298.55 grad_abs [dir=1.989553e-2 mag=6.498848e0]
Across all 60 epoch-boundary readings (3 folds × 20 epochs) dir ∈
[~4e-3, ~6e0] and mag ∈ [~5e-2, ~5e2], with ratio mag/dir consistently
50-400× (matching Task 2.0's per-component ratios). Neither branch is
near float precision — direction is PROPORTIONALLY starved, not
numerically zero.
Scenario confirmed: direction is starved relative to magnitude, NOT
the reverse. Phase 2's Task 2.1 (architectural fix on magnitude
branch) should pivot toward increasing direction's gradient flow
instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
41b0c559c9