Replace V-advantage (returns - V) with Q-advantage (E_Q(s,a) - V(s))
in compute_advantage_return kernel. Q's C51 distributional expected
value for the taken action feeds directly into PPO's advantage signal.
Root cause: with V-advantage, negative rewards pushed π AWAY from
Q-Thompson's preferred actions (q_pi_agree → -0.84). With Q-advantage,
the advantage is positive when Q rates the taken action above V's
baseline, aligning PPO and Q objectives.
Result: q_pi_agree goes from -0.84 (anti-correlated) to +0.31
(aligned) in 500 steps.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
008ea14a82