target_params_buf was initialized once via DtoD copy at first train
step and then only moved toward online via slow Polyak EMA (tau≈0.005).
At fold boundaries the online weights are shrink-and-perturb'd with
alpha=0.8, which modifies params_buf in-place — but target_params_buf
still held the end-of-previous-fold values. The Bellman target would
then use stale weights against freshly perturbed online predictions,
producing a large TD error gap in the first fold-N+1 training steps.
Polyak averaging at tau=0.005 is far too slow to close that gap before
the oversized gradients compound through Adam into runaway updates —
one of the drivers of the fold-1 gradient explosion observed in both
train-7rgqd and train-5gzpn.
- Add GpuDqnTrainer::sync_target_from_online() — DtoD memcpy of the
full params_buf into target_params_buf.
- Call it from FusedTraining::reset_for_fold right after shrink-and-
perturb and before reset_adam_state, so target = perturbed online
and Adam moments zero out from the same starting point.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
9c3ddf8b37