feat(rl): reward-policy alignment — pure-pnl mode default (spec 2026-06-01)
Empirical diagnosis (local analysis of two 20k+5k cluster runs on PVC,
SHAs 22e6ddbca alpha-rl-6kghr and 87a8259c6 alpha-rl-8gtk2):
Pearson(rewards.sum, Δrealized_pnl_cum_usd) = 0.16 train / 0.31 eval
sign-agreement (rewards.sum vs Δpnl direction) = 23-32% across both
(random baseline 50%)
The shaped reward fed to PPO/Q/V losses is systematically anti-aligned
with profitable trade direction. wr_max=0.418 in train of 87a8259c6
proves the architecture can find profitable patterns; the misaligned
gradient un-learns them. THE single load-bearing bug behind 64 prior
negative-eval commits — controllers were rebalancing around a wrong
optimization target, not a wrong solver.
Mechanism: Phase 5 step 4 of rl_fused_reward_pipeline (per-step hold
bonus = hold_bonus × √hold_time, with hold_bonus=2.0) adds +20/step at
hold_time=100, dominating realized pnl (~$1k scale) by ~2600× per held
trade. V-regression learns inflated baseline → close-event advantages
wrong-signed → PPO un-trains profitable closes.
Implementation (ONE atomic commit, gated on a single ISV slot):
- RL_REWARD_PURE_PNL_MODE_INDEX (slot 753), RL_SLOTS_END 753→754
- Bootstrap = 1.0 (NEW path default; mode=0 preserves legacy shaping
for ONE regression cluster smoke, deleted in 24-48h follow-up per
feedback_no_feature_flags + feedback_single_source_of_truth)
- rl_fused_reward_pipeline.cu: Phase 5 steps 1-4 AND Phase 5b
inventory penalty gated by pure_pnl_mode > 0.5
- rewards.pure_pnl_mode emitted in diag.jsonl (679 leaves)
- tests/reward_alignment_invariants.rs: 250-row GPU-oracle invariant
test (mode==1.0 propagation + abs_max < $25k inflation fence)
Fees: LobSimCuda apply_fill_to_pos deducts cost_per_lot_per_side per
fill into pos.realized_pnl (resting_orders.cu:213-219). realized_pnl_delta
is already net-of-fee; mode=1 cleanly delegates execution costs to
LobSim, inventory limits to Layers 1/2/4 (CMDP + IQN τ + Kelly sizing).
Local validation:
reward_alignment_pure_pnl_mode_invariants OK: 250 rows validated
eval_diag_emission OK: 679 leaves train + eval (schema parity)
Falsification (cluster smoke, fold-1 walk-forward 20k+5k b=1024):
PASS: Pearson(rewards.sum, Δrealized_pnl) ≥ 0.7 train @ steps 1500-2000
AND ≥ 0.5 eval; AND wr ≥ 0.30 in train post step 1000
FAIL: any below → controller transient extended to step 3000-4000,
re-check; if still fails, abandon reward path within 1 retry
Risk A acknowledged (HIGH): 12 adaptive controllers will rebalance
under new sparse-reward distribution. Dominant time constant is
RL_REWARD_CLAMP_V_BOUND_EWMA_ALPHA = 0.001 (τ=1000 steps; ~3τ to 95%
convergence). §2 robustness clause extends verdict window if clamp
bounds haven't stabilized.
Reviewer (sp-critical-reviewer 2 passes): FIX-AND-PROCEED — all v3
BLOCKERs structurally resolved; v4 + 5 LOW edits approved.
Refs:
spec: docs/superpowers/specs/2026-06-01-reward-policy-alignment-investigation.md
pearl: pearl_reward_signal_anti_aligned_with_pnl
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
fa347e4812