# Reward–PnL Alignment Restoration Spec

> **For agentic workers:** This spec defines WHY and WHAT for restoring reward↔pnl alignment as the next intervention, displacing F4 oracle pretraining. It does NOT itself implement the fix; an implementation plan follows after approval.

**Status:** Draft 1 (2026-06-05)
**Displaces (sequencing):** `docs/superpowers/specs/2026-06-05-f4-oracle-pretraining-infrastructure.md` — F4 is GATED behind this spec per the foundation-reshape spec's own §5.4 fork (alignment-branch must be ruled out before encoder-branch).
**Predecessors:** `docs/superpowers/specs/2026-06-01-reward-policy-alignment-investigation.md` (diagnosis), `docs/superpowers/specs/2026-06-02-trainer-rollout-buffer-gae.md` (the rollout/GAE scaffold this spec completes), `docs/superpowers/specs/2026-06-04-bellman-target-foundation-reshape.md` (§5.4 escalation fork).
**Branch target:** off `7376b1c67` on `ml-alpha-regime-observer` (or a fresh branch once approved).
**Estimated wall-clock:** Phase A+B (force-pin + cheap diagnostic gate) ~1-2 days; Phase C (1B-C credit-assignment completion) ~3-6 days IF Phase B passes. Honest range 1-2 days (if Phase B STOPs) to ~8 days (full build + cluster). Less than F4's 7-14 days, and on the **confirmed** lever.

**Linked pearls (load-bearing):**
- `pearl_reward_misalign_blocks_f4_slot753_override` — the measurement + mechanism this spec acts on
- `pearl_reward_signal_anti_aligned_with_pnl` — the original diagnosis (4 ADDENDA)
- `pearl_advantage_kernel_is_done_gated_td_not_gae` — why aligned-but-sparse reward starves at T_eff=1
- `pearl_foxhunt_trainer_is_genuinely_single_step` — why the credit-assignment fix is real work, not a flag flip
- `pearl_phase5_term_4_is_almost_potential` — the non-potential shaping terms slot 753 re-enables
- `pearl_diag_pnl_fields_are_shaped_reward_not_usd` — `realized_pnl_usd_delta` is the authoritative USD signal the gates use
- `pearl_pure_pnl_mode_starves_b16_controllers` — pure-pnl at b<1024 starves controllers (use b=1024 for reward-distribution changes)
- `feedback_investigation_first_falsification_methodology` — the discipline; cheapest-informative-step-first

---

## §1. Why this, why now

### §1.1 Three measured findings (2026-06-05)

1. **Reward is misaligned, measured.** `Pearson(rewards.sum, trading.realized_pnl_usd_delta) = 0.28` on the mid-smoke diag (`/tmp/foxhunt-mid-smoke/diag.jsonl`, b=128 seed=42, 2000 rows). Gate for trustworthy Q learning is ≥ 0.70. Sign-agreement = 0.655 (improved from the original 0.23-0.32; the dominant failure has shifted from sign-flips to **magnitude decoupling** — among worst-5% pnl steps losing $42k-$155k, reward range was [-1509, +591], i.e. some catastrophic-loss steps received positive reward).

2. **The "pure-pnl" bootstrap is cosmetic — verified against live code.** `integrated.rs:4293` sets `RL_SURFER_SCAFFOLD_WEIGHT_INDEX` (753) bootstrap = 0.0, but `rl_surfer_scaffold_controller.cu:101` writes `isv[753] = w` **unconditionally** with **no force-pin gate** (grep-confirmed absent). Runtime diag shows `surfer_scaffold_weight` pinned at **0.966 → 0.974** across all 2000 rows. All four Phase-5 non-potential shaping terms (`rl_fused_reward_pipeline.cu:344-373`, each gated `w*…`) are therefore live at ~97% strength regardless of the bootstrap.

3. **Falsification: the value function is NOT the fault — the reward target is.** value-error-vs-policy-error, eval/train median ratios: `loss.v` = 0.91×, `loss.q` = 0.83×, `loss.pi` = 1.05×. The value function generalizes fine (eval error ≈ train, even lower), the policy agrees with Q (`q_pi_agree_ema=0.95`, `q_value_abs_max=4.92` by end of train) — yet eval pnl = -$4.96M, wr 0.317, PF 0.64. *Low value error + high Q-π agreement + negative pnl on both folds* ⇒ the agent **faithfully optimizes a misaligned objective**. This simultaneously falsifies (a) F4 (Q-init is not the fault; Q learns its target fine) and (b) generalization machinery / IDAAC decoupling (value does not overfit). It confirms the **reward target** is the lever.

### §1.2 Why not F4

Per `pearl_reward_misalign_blocks_f4_slot753_override` and four independent omnisearch research agents (2026-06-05): F4 would label its oracle Q against this same `r_shaped` (math-identity is load-bearing in the F4 spec §3.1.B), warm-starting Q onto a 0.28-aligned target. F4-G1 cannot detect this (both sides of its Pearson are in shaped units — tautology). External literature is decisive: value-pretraining is an optimization lever, not a generalization lever (IDAAC, Raileanu & Fergus 2021); one-epoch-MSE-then-finetune triggers the WSRL (ICLR 2025) recalibration collapse; argmax-on-realized-path is look-ahead leakage. **Fix the target before warm-starting onto it.**

### §1.3 The Phase 1A landmine (why this is not a 10-LOC fix)

Force-pinning slot 753 to 0 gives reward ≈ pnl-delta → Pearson → ~1.0 **trivially** (the metric becomes near-tautological). But pure-pnl ALONE was already tried (Phase 1A, `pearl_reward_signal_anti_aligned_with_pnl` ADDENDUM 2026-06-02d) and **regressed eval −15.5%**, because aligned reward is *sparse* (pnl realizes at close events only) and the advantage kernel is done-gated 1-step TD (`compute_advantage_return.cu:43`, `advantages[b] = is_done ? (ret-vt) : 0`, T_eff=1) — only ~6.75 non-zero advantages per b=128 step. Aligned-but-sparse reward without multi-step credit assignment starves the gradient. **Alignment and credit-assignment are co-required.** This spec sequences a cheap diagnostic gate (Phase B) to confirm the co-fix will work *before* committing to the credit-assignment build (Phase C).

---

## §2. The fault mechanism

```
slot 753 bootstrap = 0.0 (cosmetic)
   ↓  rl_surfer_scaffold_controller.cu:101 writes isv[753] = w UNCONDITIONALLY (no pin gate)
   ↓  w = max(1 − competence, w_decay); cold-start n_trades=0 → competence≈0 → w≈0.97 every step
   ↓  rl_fused_reward_pipeline.cu:344-373: 4 non-potential Phase-5 shaping terms live at 0.97
   ↓  r_shaped only 0.28-Pearson-aligned with realized USD pnl (magnitude decoupled)
   ↓  V regresses r_shaped (loss.v fits fine on train AND eval); Q distills from V/Bellman
   ↓  π distills from Q (q_pi_agree 0.95) → π faithfully optimizes the misaligned objective
   ↓  eval pnl negative DESPITE low value error and high Q-π agreement
```

The two independent defects:
- **D1 (alignment):** the controller silently re-enables anti-aligned shaping; the 0.0 bootstrap never takes effect at runtime.
- **D2 (credit assignment):** even with aligned (sparse) reward, the done-gated 1-step TD advantage kernel cannot propagate close-event pnl back to the opening decision. The GAE rollout scaffold exists (`rollout_buffer.rs`, `rollout.rs`, `rollout_collection.rs`, GAE kernel, commits `5cd2f8703`+`779c03b9d`) but is **collection-only** (Phase 1B-B): training still runs per-step TD; the multi-epoch PPO update over the buffer (Phase 1B-C) was deferred per STOP-on-unexpected-finding.

---

## §3. What to build

### §3.1 Force-pin for slot 753 (D1 fix) — bounded

A pin gate so the bootstrap value is honored at runtime. Two design options (resolve in the plan):

| Option | Mechanism | Pros | Cons |
|---|---|---|---|
| §3.1.A New ISV pin slot | `RL_SURFER_SCAFFOLD_FORCE_PIN_INDEX` (next free slot); controller reads it and `if (pin > 0.5) return;` before writing `isv[753]` | ISV-native; A/B-toggleable; determinism-preserving | one new slot |
| §3.1.B Reuse bootstrap-as-pin | controller skips the write when an existing "design constant locked" flag is set | no new slot | conflates two concerns; less explicit |

**Recommendation:** §3.1.A. Mirror the F2/F5 slot-allocation pattern (`isv_slots.rs`, bootstrap table `integrated.rs:4382-4417`). Bootstrap pin = 0.0 (OFF → bit-equal to HEAD); set to 1.0 only in the alignment experiment config. The pinned *value* of slot 753 is then whatever its bootstrap says (0.0 = pure-pnl, or a chosen partial-shaping weight for ablation).

### §3.2 Cheap diagnostic gate (decides whether D2 build is justified) — bounded

With slot 753 pinned (aligned reward) AND `FOXHUNT_USE_ROLLOUT=1` (GAE computed as side-effect, training still TD), add a diagnostic that measures **`Pearson(GAE_advantage, realized_pnl_at_close)`** in the rollout buffer. This answers: *does multi-step GAE credit assignment on the aligned reward produce advantages that align with realized pnl?* — WITHOUT the expensive 1B-C training rearchitecture. This is the load-bearing investigation-first gate.

### §3.3 Complete Phase 1B-C (D2 fix) — substantive, CONDITIONAL on §3.2 passing

Multi-epoch PPO update over the GAE rollout buffer, replacing the per-step done-gated TD update path. Per `rollout_collection.rs:24-37` this is where per-step training is split. Scope this in the plan against the existing scaffold; do NOT rebuild the buffer/GAE/collection (already done in 1B-B).

### §3.4 Re-measure harness

A reusable script (`scripts/measure_reward_alignment.py`, NOT in root) that computes `Pearson(rewards.sum, realized_pnl_usd_delta)` and sign-agreement from any diag.jsonl, so every smoke reports the alignment number as a first-class gate.

---

## §4. Scope boundary

### §4.1 In scope (MVP)
- Force-pin slot 753 (D1).
- Diagnostic: GAE-advantage↔pnl Pearson (the gate).
- IF gate passes: complete 1B-C multi-epoch PPO over rollout buffer (D2).
- Re-measure harness + falsification gates at smoke and cluster.

### §4.2 Out of scope
- F4 oracle pretraining (gated behind this spec; revisit only if aligned reward + GAE still fails eval).
- Magnitude-clamp / popart redesign (the secondary finding; separate spec if Pearson stalls after D1+D2).
- New reward shaping terms (this spec REMOVES misaligned shaping, adds none).
- Generalization machinery (IDAAC decoupling, ensembles, augmentation) — only if D1+D2 succeed and a *residual* train→eval gap remains.
- A reward redesign beyond pinning existing weights (e.g. potential-based reshaping) — F4-Full-style; separate spec.

---

## §5. Falsification gates

| Gate | Metric | Pass | When | Falsifies |
|---|---|---|---|---|
| RA-G1 (pin works) | runtime `surfer_scaffold_weight` with pin=1.0, bootstrap=0.0 | == 0.0 across all rows | Phase A smoke | If fails: pin gate not wired to the controller write |
| RA-G2 (determinism) | `determinism-check.sh --quick` seeds 42/43/44 | exit 0, eval_summary bit-equal | Phase A | If fails: pin/diag introduced nondeterminism |
| RA-G3 (alignment restored) | `Pearson(rewards.sum, realized_pnl_usd_delta)` with pin=1.0 bootstrap=0.0 | ≥ 0.70 | Phase A smoke | (near-tautological with pure-pnl; confirms the pin removed the shaping) |
| **RA-G4 (credit-assignment viability — the load-bearing gate)** | `Pearson(GAE_advantage, realized_pnl_at_close)` in rollout buffer, aligned reward + USE_ROLLOUT=1 | ≥ 0.70 | Phase B smoke | **If fails: GAE on aligned reward still does not align advantages with pnl → STOP; the fault is deeper than credit assignment (reward shape itself / regime). Do NOT build 1B-C.** |
| RA-G5 (no eval regression) | smoke eval pnl with D1+D2 | ≥ F2+F5 SOTA −$415,237 (not worse than current) | Phase C smoke | If fails: D2 build did not deliver; diagnose before cluster (recall Phase 1A regressed here) |
| RA-G6 (cluster eval) | `eval_summary.total_pnl_usd` at Tier 2 b=1024 20k+5k fold 1 | > F2+F5 sk2z9 cluster baseline | Phase D cluster | If fails AND RA-G4 passed: aligned reward+GAE insufficient; THEN encoder/generalization or F4 is the next fork |

**Stop-on-fail discipline:**
- RA-G4 is the spec's economic linchpin: it costs ~1 day (no training rearchitecture) and decides whether the ~3-6 day Phase C build is justified. **If RA-G4 < 0.70, STOP — do not build 1B-C.** A failure here means even perfect multi-step credit assignment on the aligned reward does not recover pnl-aligned advantages, which would point at deeper reward-shape or regime-shift causes (and would, notably, also predict F4 failure — so it is decisive either way).
- RA-G6 failure with RA-G4 pass cleanly isolates the residual to encoder/generalization or genuine OOS regime shift — the foundation-reshape §5.4 "encoder branch," now legitimately reached.

---

## §6. Implementation phases

### Phase A — Force-pin slot 753 (~0.5-1 day)
Add pin slot + controller gate + bootstrap-table entry. Smoke with pin=1.0 bootstrap=0.0. **Gates:** RA-G1, RA-G2, RA-G3. **STOP if** RA-G1/G2 fail.

### Phase B — Credit-assignment viability diagnostic (~0.5-1 day)
Add `Pearson(GAE_advantage, realized_pnl_at_close)` diag to the rollout-buffer path. Smoke with pin=1.0 + `FOXHUNT_USE_ROLLOUT=1` (training unchanged). **Gate:** RA-G4. **STOP if** RA-G4 < 0.70 — escalate to reward-shape/regime investigation, NOT 1B-C, NOT F4.

### Phase C — Complete 1B-C multi-epoch PPO over rollout buffer (~3-6 days) — CONDITIONAL on RA-G4
Replace per-step done-gated TD update with multi-epoch PPO over the GAE buffer. Mid-smoke. **Gates:** RA-G5, RA-G2 (determinism preserved). **STOP if** RA-G5 regresses.

### Phase D — Cluster verify (~80 min + analysis)
Tier 2 b=1024, 20k+5k, fold 1, pin=1.0 + USE_ROLLOUT=1 + 1B-C. **Gate:** RA-G6. Branch per §5.

---

## §7. Risks

1. **RA-G3 is near-tautological** — pinning to pure-pnl makes reward≈pnl by construction. That is why RA-G4 (advantage↔pnl, NOT reward↔pnl) is the real gate. Do not over-read RA-G3.
2. **Phase 1A precedent** — pure-pnl regressed eval −15.5%. This spec's whole bet is that the regression was a credit-assignment (D2) artifact, not an alignment (D1) artifact. RA-G4 tests that bet cheaply before the Phase C spend. If RA-G4 fails, the bet is wrong and we stop early — cheaply.
3. **Magnitude decoupling may persist after D1** — if removing shaping still leaves Pearson < 0.70 because of clamp/popart magnitude crushing (`pearl_popart_blind_to_session_signals`), a clamp/popart fix becomes the next sub-spec. RA-G3 catches this.
4. **1B-C cost is the dominant uncertainty** — `pearl_foxhunt_trainer_is_genuinely_single_step` warns multi-step training is real work. The buffer+GAE+collection exist; the multi-epoch PPO update path does not. Plan must scope 1B-C against the existing scaffold and STOP if it balloons beyond the estimate.
5. **b<1024 controller starvation** — per `pearl_pure_pnl_mode_starves_b16_controllers`, validate reward-distribution changes at b=1024; treat local b=128 smoke as directional only (`pearl_cluster_log_is_ground_truth`).

---

## §8. Decision log entry

```
Decision: Adopt Reward–PnL Alignment Restoration as the next intervention, GATING F4 oracle pretraining behind it. Sequenced: force-pin slot 753 (D1) → cheap GAE-advantage↔pnl diagnostic (RA-G4) → conditional 1B-C credit-assignment completion (D2) → cluster.
Rejected (for now): F4 oracle pretraining — falsified as the lever (value generalizes; Q learns its target fine; the fault is the misaligned reward TARGET, not Q-init). External lit (Cal-QL/WSRL/IDAAC, 2023-2025) concurs value-pretrain is the wrong tool here.
Rationale: measured Pearson(reward,pnl)=0.28 (gate 0.70); slot-753 controller override makes the pure-pnl bootstrap cosmetic; value-error-vs-policy-error falsification (0.91×/0.83×/1.05× eval-vs-train) shows value generalizes while the agent loses money on both folds — the objective is misaligned, not the estimator.
Date: 2026-06-05
Falsification trigger (viability): RA-G4 < 0.70 → STOP before 1B-C build.
Going-in-circles guard: this is the alignment branch of foundation-reshape §5.4; if D1+D2 pass RA-G4/G5 but RA-G6 fails at cluster, the encoder/generalization branch is legitimately isolated — NOT another reward reshape.
```

---

## §9. Phase A verdict (2026-06-05, executed on clean data post-L0-fix)

Executed on top of the MBP-10 L0 decode fix (`6df3284d0`) with freshly-regenerated
clean sidecars, mid-smoke seed 42 b=128, `FOXHUNT_PIN_SURFER_SCAFFOLD=1`:

- **RA-G1 PASS** — `surfer_scaffold_weight` = 0.0 across all 2000 rows; `surfer_scaffold_force_pin` = 1.0. The D1 force-pin mechanism is correctly wired.
- **RA-G2** — DEFERRED to Phase C (the D2 trainer change needs its own determinism check; pin code committed at `d8447475c`).
- **RA-G3 INCONCLUSIVE (starvation-limited, NOT a clean fail)** — Pearson 0.016, but only because the agent **flat-collapsed**: TRAIN flat fraction 0.80→1.00, EVAL 100% flat, **0 eval trades, eval pnl $0**. No trades → no pnl variance → degenerate correlation. This is the documented b=128 pure-pnl starvation (`pearl_pure_pnl_mode_starves_b16_controllers`; spec §7 risk 5). Cannot be validated on the RTX 3050 (no b=1024).

**Interpretation:** D1-alone produces flat-collapse, exactly as the spec predicted (Phase 1A regressed −15.5%). Notably, pure-pnl flat ($0) beats the misaligned-shaping run (−$10.7M) — removing the anti-aligned hold-bonus stopped the active bleeding, but with no credit assignment (and/or no real edge) the agent simply stops trading. RA-G3/RA-G4 are not decidable locally.

**Decision (user-directed 2026-06-05):** proceed to build **D2 (Phase C, GAE multi-epoch PPO over the existing rollout buffer)** rather than gate on the locally-unmeasurable RA-G4. RA-G4 viability is consciously deferred to the eventual b=1024 cluster run (Phase D). Phase C begins with the C1 rollout-scaffold audit (no speculative code).
