Root cause from train-v8ztm 10-ep validation (commit 1396b62ec): HEALTH_DIAG
showed alpha_smoothed=0.0002 (vs ~0.5 expected steady-state), gate1=closed,
var_aux:var_q ratio 290:1 — symptoms of an EGF producer chain firing < 1%
as often as the consumer.
The original B.11 wire-up (commit 857722e77) placed `launch_sp14_q_disagreement_update`,
`launch_sp14_alpha_grad_compute`, and the prerequisite `launch_sp13_aux_dir_metrics`
in `process_epoch_boundary` — which runs ONCE per epoch (single call site at
training_loop.rs:780, called from the per-epoch loop, not from the per-step
loop in `run_training_steps_slices`). The captured backward consumer
`launch_sp14_scale_wire_col` (inside launch_cublas_backward_to, replays every
training step via parent graph) reads ISV[ALPHA_GRAD_SMOOTHED=393] per step,
but the producer was firing only at epoch boundary — every step inside the
epoch observed (steps_per_epoch − 1)-step-stale alpha values, with the EMA
chain barely accumulating past sentinel between rare per-epoch updates. The
plan §2550 explicitly specifies per-step cadence; the existing wire violated
the plan.
Fix (atomic, graph-capture-safe):
- MOVED launch_sp13_aux_dir_metrics, launch_sp14_q_disagreement_update,
launch_sp14_alpha_grad_compute from process_epoch_boundary into
fused_training.rs:submit_aux_ops, immediately after populate_q_out.
submit_aux_ops captures into the aux_child sub-graph, so each parent-graph
replay re-fires the full producer chain — restoring per-step cadence.
- launch_sp13_aux_dir_metrics had to migrate alongside the SP14 launches:
alpha_grad_compute_kernel consumes its outputs (ISV[373/374]); leaving
sp13 per-epoch while moving SP14 per-step would re-introduce the same
staleness bug for aux_dir_acc reads (atomic dependency migration per
feedback_no_partial_refactor).
- Per-epoch launch_sp14_gradient_hack_detect circuit breaker stays in
process_epoch_boundary — its lockout decrement IS one-per-epoch by design.
- Forward consumers (6 launch_sp14_dir_concat_qaux sites) and backward
consumers (2 launch_sp14_scale_wire_col sites) unchanged — they read the
same ISV[393], but now see live per-step values instead of per-epoch
staleness.
Verification:
- cargo check -p ml --tests --all-targets clean (no errors, no new warnings).
- All 6 SP14 oracle GPU tests pass (alpha_grad_adaptive_beta,
alpha_grad_schmitt_hysteresis, dir_concat_qaux_correct,
gradient_hack_circuit_breaker_fires, q_disagreement_all_hold_no_contribution,
q_disagreement_k4_k2_mapping).
- HEALTH_DIAG validation pending L40S re-dispatch — expect alpha_smoothed
to track real EGF-driven values (~0.5 in steady state).
Invariants:
- pearl_no_host_branches_in_captured_graph (kernels are pure GPU state
machines using launch_builder + pre-loaded CudaFunction; no per-call
load_cubin)
- feedback_no_partial_refactor (sp13 + 2 SP14 launches migrated atomically)
- feedback_wire_everything_up (all 3 producers now production hot-path,
re-fire on every parent-graph replay)
- feedback_isv_for_adaptive_bounds (no warmup_gate parameter — variance-
driven k_aux/k_q in alpha_grad_compute_kernel handles cold-start
adaptively, per c0fc28e45)
Refs: train-v8ztm trajectory analysis 2026-05-07T15:59:49 HEALTH_DIAG[10]
showed dir_entropy=0.6545 kill-fast breach with model converging to 64%
Hold + 84% Quarter magnitude — exactly the pathology B.11 was designed
to prevent by routing aux's directional signal into Q.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
200f05fcef