Activates DuelingQHead's V output as the PPO advantage baseline,
replacing scalar value_head's contribution. Also wires soft-update
of DuelingQHead's target net (was deferred from 4.2).
Two changes:
1. DuelingQHead.soft_update_target(): reuses dqn_target_soft_update
kernel (generic element-wise blend with τ from ISV[401]) across
all 4 weight tensors (w_v, b_v, w_a, b_a). Called once per training
step after Adam, mirroring DQN's pattern.
2. compute_advantage_return call: v_pred_d → dueling_v_d,
v_pred_tp1_d → dueling_v_tp1_d. PPO advantage is now:
A(s_t, a_t) = E_Q_C51(s_t, a_taken) − V_dq(s_t)
returns(s_t) = r_t + γ(1−done) × V_dq(s_{t+1})
value_head still trains via MSE on returns for diagnostic
comparison; can be retired in a future commit if V_dq proves
itself.
Phase 4.2 validated structurally at b=1024 5k that DuelingQHead's
training does not perturb Plan A v2's dynamics (qpa tracked within
0.02 at every milestone). Phase 4.3 is the smallest possible commit
that USES V_dq downstream — fully de-risked by 4.2's structural test.
Cluster expectation: qpa trajectory roughly matches Plan A v2 (V_dq
calibrated by joint training on same Bellman reward signal as C51,
so cross-architecture scale issue from Phase 3.2 doesn't apply here).
Validated:
- cargo build --release clean
- integrated_trainer_smoke 1 step passes
- alpha_rl_train --steps 3 --b 128 under compute-sanitizer: 0 errors,
l_q=0.024, l_v=0.0003 (slightly higher than Plan A v2's 0.0001
because V_scalar now sees different gradient prop now that V_dq
drives advantage — still healthy)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
25f5ce99b6