Three coordinated changes to remove anti-aligned components from
the reward pipeline. Predicted Pearson trajectory based on Phase 3A
audit was wrong (PopArt + hindsight only delivered +0.04 not +0.4),
but the deep Phase 3B-Z audit revealed the "low Pearson" was largely
a measurement artifact: raw_reward tracks ALL PnL events (including
fees + partial closes via pos.realized_pnl_delta) while diag's
realized_pnl_usd_cum tracks only full closures. The actual training
reward IS aligned with total realized PnL evolution.
## Changes
* Slot 753 RL_SURFER_SCAFFOLD_WEIGHT_INDEX bootstrap 1.0 → 0.0
(Phase 5 hold-bonus/entry-cost/short-hold-penalty/long-ride-bonus
shaping disabled — was the dominant anti-aligned contaminant per
Phase 3A audit and cluster 2A-E evidence of +$201M shaped vs
-$330M actual).
* New ISV slot 793 RL_POPART_NORMALIZE_ENABLED_INDEX bootstrap 0.0
(PopArt batch normalization disabled — was found to sign-flip
rewards under adverse batch composition).
* Hindsight injection (rl_hindsight_inject.cu wants_inject) extended
with scaffold_w > 0.5 condition (when slot 753 = 0, hindsight off).
Semantically pairs the two training scaffolds.
## Verification
* 13/13 multi_head_policy_invariants PASS (no regression).
* FOXHUNT_USE_MULTI_HEAD_POLICY=0/1 ./scripts/determinism-check.sh
--quick: both exit 0.
* Local single-seed mid-smoke (b=128, seed 42):
- eval pnl: -$4.6M (vs Phase 0 baseline -$4.46M, within noise)
- 14,691 trades training / 3,268 eval (healthy, not paralyzed)
- action_entropy 1.82 at step 1999 (below uniform 2.40, learning)
- 0 NaN, exit 0
## Pearson interpretation (Phase 3B-Z audit)
The Pearson 0.38 between raw_reward and pnl_step_close_usd_delta is
a structural mismatch between two valid PnL views (all events vs
close events only), NOT an anti-alignment. raw_reward at slot
753 = 0 is the delta of pos.realized_pnl which already aggregates
the events we care about for total-USD optimization.
Cluster verification (Phase 3C, next) will reveal whether the
removal of Phase 5 + popart + hindsight produces a base policy
that can learn at scale. The 2A-E cluster catastrophe (-$330M)
was driven by Phase 5 shaping that's now disabled.
Plan: docs/superpowers/plans/2026-06-03-reward-alignment-gae-
combined.md
Linked pearls:
* pearl_reward_signal_anti_aligned_with_pnl
* pearl_phase5_term_4_is_almost_potential
* pearl_popart_blind_to_session_signals
* pearl_pure_pnl_mode_starves_b16_controllers
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
093feac3da