ROOT CAUSE: grad_norm kernel was inside graph_adam, but grad_buf was
zeroed by the gradient vaccine between graph_forward and graph_adam.
The vaccine zeros grad_buf to compute g_val, fails with INVALID_VALUE,
and the non-fatal error handler skips the restore — leaving grad_buf
zeroed when graph_adam's grad_norm kernel reads it.
TWO FIXES:
1. Move grad_norm computation OUT of graph_adam entirely:
- New compute_grad_norm_outside_graph() method runs between graphs
- Zeros grad_norm_f32_buf, launches grad_norm kernel, launches finalize
- Called AFTER all gradient injections, BEFORE replay_adam()
- Reads the ACTUAL modified grad_buf (C51 + CQL + IQN + ensemble)
- All 3 replay paths updated (train_step_gpu, execute_train_scalars_only,
replay_adam_and_readback)
2. Vaccine grad_buf restore on failure:
- New restore_grad_from_vaccine_save() method
- When vaccine fails after zeroing grad_buf, copies saved g_train back
- Prevents grad_buf from staying zeroed after vaccine error
RESULT: grad_norm is now stable and nonzero across all epochs:
Epoch 1: 0.6969
Epoch 2: 0.6982
Epoch 3: 0.6998
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bbf9de3ea1