foxhunt

Files

jgrusewski fa347e4812 feat(rl): reward-policy alignment — pure-pnl mode default (spec 2026-06-01)

Empirical diagnosis (local analysis of two 20k+5k cluster runs on PVC,
SHAs 22e6ddbca alpha-rl-6kghr and 87a8259c6 alpha-rl-8gtk2):

  Pearson(rewards.sum, Δrealized_pnl_cum_usd) = 0.16 train / 0.31 eval
  sign-agreement (rewards.sum vs Δpnl direction)   = 23-32% across both
  (random baseline 50%)

The shaped reward fed to PPO/Q/V losses is systematically anti-aligned
with profitable trade direction. wr_max=0.418 in train of 87a8259c6
proves the architecture can find profitable patterns; the misaligned
gradient un-learns them. THE single load-bearing bug behind 64 prior
negative-eval commits — controllers were rebalancing around a wrong
optimization target, not a wrong solver.

Mechanism: Phase 5 step 4 of rl_fused_reward_pipeline (per-step hold
bonus = hold_bonus × √hold_time, with hold_bonus=2.0) adds +20/step at
hold_time=100, dominating realized pnl (~$1k scale) by ~2600× per held
trade. V-regression learns inflated baseline → close-event advantages
wrong-signed → PPO un-trains profitable closes.

Implementation (ONE atomic commit, gated on a single ISV slot):
  - RL_REWARD_PURE_PNL_MODE_INDEX (slot 753), RL_SLOTS_END 753→754
  - Bootstrap = 1.0 (NEW path default; mode=0 preserves legacy shaping
    for ONE regression cluster smoke, deleted in 24-48h follow-up per
    feedback_no_feature_flags + feedback_single_source_of_truth)
  - rl_fused_reward_pipeline.cu: Phase 5 steps 1-4 AND Phase 5b
    inventory penalty gated by pure_pnl_mode > 0.5
  - rewards.pure_pnl_mode emitted in diag.jsonl (679 leaves)
  - tests/reward_alignment_invariants.rs: 250-row GPU-oracle invariant
    test (mode==1.0 propagation + abs_max < $25k inflation fence)

Fees: LobSimCuda apply_fill_to_pos deducts cost_per_lot_per_side per
fill into pos.realized_pnl (resting_orders.cu:213-219). realized_pnl_delta
is already net-of-fee; mode=1 cleanly delegates execution costs to
LobSim, inventory limits to Layers 1/2/4 (CMDP + IQN τ + Kelly sizing).

Local validation:
  reward_alignment_pure_pnl_mode_invariants OK: 250 rows validated
  eval_diag_emission OK: 679 leaves train + eval (schema parity)

Falsification (cluster smoke, fold-1 walk-forward 20k+5k b=1024):
  PASS: Pearson(rewards.sum, Δrealized_pnl) ≥ 0.7 train @ steps 1500-2000
        AND ≥ 0.5 eval; AND wr ≥ 0.30 in train post step 1000
  FAIL: any below → controller transient extended to step 3000-4000,
        re-check; if still fails, abandon reward path within 1 retry

Risk A acknowledged (HIGH): 12 adaptive controllers will rebalance
under new sparse-reward distribution. Dominant time constant is
RL_REWARD_CLAMP_V_BOUND_EWMA_ALPHA = 0.001 (τ=1000 steps; ~3τ to 95%
convergence). §2 robustness clause extends verdict window if clamp
bounds haven't stabilized.

Reviewer (sp-critical-reviewer 2 passes): FIX-AND-PROCEED — all v3
BLOCKERs structurally resolved; v4 + 5 LOW edits approved.

Refs:
  spec: docs/superpowers/specs/2026-06-01-reward-policy-alignment-investigation.md
  pearl: pearl_reward_signal_anti_aligned_with_pnl

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2026-06-01 22:38:47 +02:00

2026-03-10-zero-cpu-hotpath-design.md

…

2026-03-11-dqn-factored-actions-design.md

…

2026-03-13-gpu-test-workflow-design.md

…

2026-03-14-kanidm-sso-design.md

…

2026-03-15-fused-cuda-training-design.md

…

2026-03-21-h100-epoch-optimization-design.md