# Reward-Policy Realignment — Drop Hold-Time Shaping; Direct Realized-PnL Reward

**Date:** 2026-06-01 (v4 — after 1st sp-critical-reviewer pass identified 3 BLOCKERs + 6 HIGHs + 4 MEDIUMs in v3)
**Status:** Ready for 2nd sp-critical-reviewer pass
**Branch target:** `ml-alpha-regime-observer`

---

## 0. Diagnosis (mechanism + magnitude)

### 0.1 Empirical evidence (existing PVC data)

Two 20k+5k full cluster runs, sampled 100× / 50×, analyzed locally:

| Run | SHA | Phase | Pearson(rewards.sum, Δrealized_pnl) | wr final | wr max |
|---|---|---|---|---|---|
| alpha-rl-6kghr | 22e6ddbca | train | N/A (realized_pnl field empty era) | 0.312 | 0.313 |
| alpha-rl-6kghr | 22e6ddbca | eval | N/A | 0.291 | 0.296 |
| alpha-rl-8gtk2 | 87a8259c6 | train | **0.157** | 0.343 | **0.418** |
| alpha-rl-8gtk2 | 87a8259c6 | eval | **0.314** | 0.340 | 0.380 |

`Pearson(rewards.sum, Δrealized_pnl_cum_usd)` is the clean reference because `realized_pnl_cum_usd` comes from `pos_d[b].realized_pnl` (line 88 of `rl_fused_reward_pipeline.cu`) which is written by `LobSimCuda` *independently of the RL reward path*. Pearson ≈ 0.16 train / 0.31 eval is well below 1.0 — the gradient signal fed to the loss kernels is only weakly correlated with the trade-level pnl reality.

`alpha-rl-8gtk2` train wr_max = **0.418** is the architecture-can-learn proof: encoder + heads find profitable patterns. wr_final = 0.343 means those patterns got un-trained back to a sub-break-even attractor.

### 0.2 Kernel evidence (`rl_fused_reward_pipeline.cu`)

Phase 5 of the fused pipeline applies four shaping terms on top of Phase 1's realized-pnl delta. **Bootstrap values (verified from `integrated.rs:3566-3569`)**:
- `RL_ENTRY_COST_INDEX = 15.0`
- `RL_SHORT_HOLD_MIN_STEPS_INDEX = 100.0`
- `RL_SHORT_HOLD_PENALTY_INDEX = 0.5`
- `RL_HOLD_BONUS_INDEX = 2.0`
- `RL_INVENTORY_PENALTY_BETA_INDEX = 0.0` (sentinel; Phase 5b is a no-op in current bootstrap)

Phase 5 step 4 (per-step hold bonus while holding profitable position):
```c
if (prev_lots != 0 && current_lots != 0 && r > 0.0f && hold_time > 0) {
    r += hold_bonus * sqrtf((float)hold_time);   // hold_bonus = 2.0
}
```

With `hold_bonus = 2.0` and `hold_time = 100`, this adds **+20.0 per step** for a held profitable position. Over a 100-step hold, accumulated shaping reward is `sum_{t=1..100} 2 × √t ≈ 2 × (2/3) × 100^1.5 = 1333` — versus typical realized pnl on close of perhaps **+$10** ($10/σ_welford ≈ +0.5 in σ-normalized units after popart).

**Domination ratio**: ~2600× per held trade (1333 σ-units of shaping vs 0.5 σ-units of realized pnl).

Phase 5 step 2 (short-hold penalty `r *= penalty`, penalty=0.5):
- Multiplies the closing reward by 0.5 when `hold_time < min_hold = 100`
- For a winning close: reduces magnitude (less reward)
- For a losing close: **makes the loss less negative** (less negative ≈ "less bad" in gradient terms)
- The sign-magnitude relationship is distorted: a $100 loss closed early registers as -$50 of reward, indistinguishable from a $50 loss closed at full hold

Phase 5 step 3 (long-ride bonus `r *= (1 + hold_bonus × √hold_time)`):
- Only fires on profitable close
- With hold_bonus=2.0 and hold_time=100, multiplier = 1 + 2 × 10 = **21×**
- A +$10 winning close becomes +$210 of shaping reward

### 0.3 Mechanism — V-regression-mediated, not direct PPO

(per reviewer HIGH H1, corrected from v3)

The advantage flowing into PPO surrogate is computed at `compute_advantage_return.cu:41-43`:
```c
advantages[b] = is_done ? (ret - vt) : 0.0f
```
**Non-done advantages are ALREADY zero.** So per-step hold-bonus rewards (which fire on non-done held-position steps) do NOT flow into PPO's policy gradient directly. They flow into V regression via `returns[b] = r + γ × v_{t+1}`.

The broken chain is:
1. Per-step hold bonus inflates `returns[b]` on held-position steps
2. V regression learns to predict the inflated return: V baseline becomes **artificially high** during held profitable positions
3. At a done event: `advantage_done = (realized_close_reward - V_baseline_inflated)` is **small or wrong-signed** — even a +$100 close looks "below baseline" if baseline expected the held-trajectory-bonus
4. PPO surrogate trained on these advantages **un-learns** profitable closing actions (they get smaller policy updates than the inflated baseline expected)

Simultaneously, Q-distill via `rl_q_pi_distill_grad` uses `q_target` from `compute_bellman_target` which sees the same inflated returns. So Q learns to predict the bonus, not the close.

**Mode=1 fix**: skip Phase 5 steps 2-4. Reward stream becomes:
- Held positions: `reward = 0`
- Close events: `reward = realized_pnl_delta` (no entry-cost since we drop that too — see §1.2)

Then V learns the *correct* return target (γ-discounted close-event pnl), advantage at close events reflects actual edge, PPO/Q learn closing actions on real reward.

## 1. The intervention — pure realized-pnl reward

### 1.1 ISV slot + bootstrap

```rust
// crates/ml-alpha/src/rl/isv_slots.rs
//
// Reward-policy realignment (2026-06-01). When > 0.5, rl_fused_reward_pipeline
// Phase 5 skips ALL hold-time shaping (steps 2-4) AND entry-cost (step 1).
// Reward = realized_pnl_delta (Phase 1 output). Pure pnl alignment.
//
// Bootstrap = 1.0 (NEW path default). The legacy mode=0 path is preserved
// for ONE regression cluster smoke (verifying baseline reproduction at same
// SHA), then DELETED in a follow-up commit per
// [[feedback_no_feature_flags]] + [[feedback_single_source_of_truth_no_duplicates]].
// This is a permanent reward redesign, not a permanent toggle.
pub const RL_REWARD_PURE_PNL_MODE_INDEX: usize = 753;
pub const RL_SLOTS_END: usize = 754;
```

Bootstrap entry:
```rust
(crate::rl::isv_slots::RL_REWARD_PURE_PNL_MODE_INDEX, 1.0_f32),
```
Bootstrap array `[(usize, f32); 240]` → `[..; 241]`.

### 1.2 Kernel modification (rl_fused_reward_pipeline.cu)

```c
// At top of file:
#define RL_REWARD_PURE_PNL_MODE_INDEX 753

// At kernel entry (alongside other ISV reads):
const float pure_pnl_mode = isv[RL_REWARD_PURE_PNL_MODE_INDEX];

// PHASE 5 modification:
float r = reward;  // (from Phase 1: r = current_realized - prev_realized)

if (pure_pnl_mode < 0.5f) {
    // Legacy shaping (existing code unchanged):
    // step 1: entry cost on flat → positioned
    if (prev_lots == 0 && current_lots != 0) {
        r -= entry_cost;
    }
    // step 2: short-hold penalty
    if (done > 0.5f && (float)hold_time < min_hold) {
        r *= penalty;
    }
    // step 3: long-ride bonus
    if (done > 0.5f && r > 0.0f && hold_time > 0) {
        r *= (1.0f + hold_bonus * sqrtf((float)hold_time));
    }
    // step 4: per-step hold bonus (SMOKING GUN per §0.2)
    if (prev_lots != 0 && current_lots != 0 && r > 0.0f && hold_time > 0) {
        r += hold_bonus * sqrtf((float)hold_time);
    }
}
// Mode=1: r stays at Phase 1's realized_pnl_delta unchanged.
// Entry cost dropped because LobSim's apply_fill_to_pos
// (crates/ml-backtesting/cuda/resting_orders.cu:213-219) deducts
// `filled_lots × cost_per_lot_per_side_per_b[b]` from pos.realized_pnl
// per fill. So realized_pnl_delta is already net-of-fee — adding
// entry_cost in reward shaping would double-count broker fees.

// PHASE 5b inventory penalty: ALSO gated by pure_pnl_mode.
// Default β = 0 (sentinel since 30+ commits) so this is a no-op now anyway,
// but the gate keeps mode=1 semantically pure. Layers 1 (CMDP) / 2 (IQN τ)
// / 4 (Kelly fraction) own inventory limits via action overrides + sizing
// controllers; mode=1 cleanly delegates inventory risk out of the reward path.
const float inv_beta = isv[RL_INVENTORY_PENALTY_BETA_INDEX];
if (pure_pnl_mode < 0.5f && inv_beta > 0.0f) {
    const float net_pos_mag = (float)((current_lots < 0) ? -current_lots : current_lots);
    r -= inv_beta * net_pos_mag;
}

// Phase 5 finalize:
rewards[b] = r;

// Phase 6 unchanged: raw_rewards[b] = r  (now correctly named — "raw"
// means pre-scale post-shaping in legacy mode, equals realized_pnl in mode=1).

// Phase 7 (outcome EMA) — uses r at close events.
// In mode=1, r = realized_pnl_delta, so outcome = sign(realized_pnl_delta)
// which is the cleanest possible signal for the aux head consumer.
// In mode=0, r is shaped, so outcome reflects shaped sign (legacy behavior).
// No code change needed.
```

### 1.3 Diag emit

Add ONE leaf:
```rust
// In integrated.rs::build_diag_value, under the existing rewards block:
"pure_pnl_mode": isv[crate::rl::isv_slots::RL_REWARD_PURE_PNL_MODE_INDEX],
```
Path: `rewards.pure_pnl_mode` (per reviewer M4 — keeps namespace coherent).

`EXPECTED_LEAVES`: 678 → 679 (+1).

### 1.4 Invariant test

`crates/ml-alpha/tests/reward_alignment_invariants.rs` — 3 invariants on local b=16 200+50 smoke:

1. **Mode default**: `rewards.pure_pnl_mode == 1.0` across all diag rows.
2. **Zero on non-close held steps**: when `dones[b] < 0.5` and the kernel branch did NOT execute the legacy entry-cost path, `rewards[b] == 0.0f` for that batch element.
3. **Close-event reward equals realized_pnl_delta**: at done events in mode=1, `rewards[b] == realized_pnl_current - realized_pnl_prev` within 1e-5 float epsilon (computed from `trading.realized_pnl_cum_usd` field which now correctly tracks the raw delta).

**Local smoke validates ONLY wiring** (per reviewer H4). Behavior validation (Pearson, wr) requires cluster scale — see §2.

## 2. Falsification criteria (predetermined, cluster smoke only)

**Cluster smoke**: 2k+500 b=1024 fold-1 L40S at mode=1 (default after bootstrap).

**Reference signal for Pearson computation**: `trading.realized_pnl_cum_usd` (NOT `trading.pnl_cum_usd` — that's reward-derived per reviewer B2). Delta computed from sampled diag rows; aggregated to per-100-step windows for stability.

**Verdict table**:

| Signal | Window | Target | Verdict if met | Verdict if missed |
|---|---|---|---|---|
| Pearson(rewards.sum, Δrealized_pnl) | train steps 1500-2000 | **≥ 0.7** | reward aligned | iterate (1 retry max) |
| Sign agreement rate (rewards.sum sign vs Δrealized_pnl sign) | train steps 1500-2000 | **≥ 0.65** | reward aligned | iterate |
| wr at end of train | step 2000 | **≥ 0.30** (R=2.37 break-even) | training works | architecture problem (Phase 2 spec) |
| Pearson at eval steps 100-500 | post warmup | **≥ 0.5** | reward stable OOS | eval-boundary popart shock dominates (per H3) |

**Window justification**: per reviewer's reward-controller audit, the 12 adaptive controllers (reward_scale, reward_clamp_win/loss, popart σ, etc.) will rebalance under the new sparse-reward distribution. Dominant time constant is `RL_REWARD_CLAMP_V_BOUND_EWMA_ALPHA = 0.001` → τ=1000 steps (~3000 steps to 95% convergence). The falsification window is steps 1500-2000 as the primary check.

**Robustness clause (per reviewer Finding 2)**: if `reward_clamp.v_bound_ema` (diag-emitted) is still > 5× its step 1500-2000 running average at step 2000, the clamp is still in transient — extend the verdict window to step 3000-4000 and re-evaluate. Diag emits `reward_clamp.win_bound` and `reward_clamp.loss_bound`; verify both are within ±20% of their step-1500 → step-2000 average before declaring a Pearson verdict.

**Theoretical Pearson ceiling** (per reviewer M1): with popart per-batch whitening at b=1024 and ~5% close-rate per step, theoretical max Pearson ≈ 0.85-0.95 (not 1.0 — popart subtracts batch mean which decorrelates from the close-event signal). Target 0.7 is achievable but tight.

**Maximum 1 iteration** of reward-shape variants before abandonment. Specifically: if Pearson < 0.7, retry with R-multiple reward (`r = realized_pnl_delta / stop_loss_distance_estimate`) instead of $-units. If still < 0.7, declare reward path closed and proceed to architecture simplification (Phase 2 spec, separate doc).

**Eval falsification scope** (per reviewer H3): Pearson target ≥ 0.5 (lower than train 0.7) acknowledges the unfixed eval-boundary popart-shock bug documented in `pearl_popart_reset_at_eval_boundary_shocks_normalization`. Eval-time popart σ resets and spikes 1100× on first tail event, contaminating the first ~1000 eval steps. Train-side Pearson is the primary criterion; eval-side is informational until that bug is separately addressed.

## 3. Validation plan

### V1 — Local smoke (wiring only)
- Build release; run 200+50 b=16 fold-1.
- Verify 3 invariants from §1.4.
- Expected: invariants pass; observed close-event rewards ≈ realized_pnl deltas; non-close rewards = 0.

### V2 — Cluster smoke run 1 (mode=1, the fix)
- Submit alpha-rl smoke at new commit SHA, mode=1 (default), 2k+500 b=1024 fold-1 L40S.
- Pull diag JSONL post-completion.
- Compute Pearson + sign agreement post-hoc against `realized_pnl_cum_usd`.
- Apply §2 verdict table.

### V3 — Cluster smoke run 2 (mode=0 regression)
- Per reviewer M2: re-seed mode=0 via ISV write at trainer init, same commit SHA, same params.
- Verify diag fields match the prior baseline run (alpha-rl-glv6n at a5e0d0079).
- Confirms the legacy path is bit-equivalent (or float-equality within 1e-4) to baseline.
- This is the regression run; if mode=1 succeeds AND mode=0 is bit-baseline, we have confidence the gate is correct.

### V4 — Determinism
Train phase actions/dones/rewards in mode=1 will NOT match prior runs (reward signal changed → gradient changed → policy changed). This is intended.

In mode=0 (V3 run), should match the baseline byte-for-byte modulo the new ISV read overhead (which doesn't affect numeric output).

## 4. Implementation phases

| Phase | LOC | File |
|---|---|---|
| 1 | +6 | `isv_slots.rs`: 1 const + doc comment + bump RL_SLOTS_END to 754 |
| 2 | +4 | `integrated.rs`: 1 bootstrap entry; array 240 → 241 |
| 3 | +20 | `rl_fused_reward_pipeline.cu`: #define + ISV read + Phase 5 gate + Phase 5b gate |
| 4 | +3 | `integrated.rs::build_diag_value`: 1 new leaf under `rewards` |
| 5 | +1 | `tests/eval_diag_emission.rs`: EXPECTED_LEAVES 678 → 679 |
| 6 | +100 | `tests/reward_alignment_invariants.rs`: 3 invariants |
| 7 | — | build release; local smoke; invariant test |
| 8 | — | atomic commit on `ml-alpha-regime-observer` |

Total: **~135 LOC across 5 files + 1 test**.

## 5. Risks

- **Risk A (HIGH)**: 12 adaptive controllers (reward_scale, reward_clamp_win/loss, popart σ_welford, etc.) will rebalance under the new sparse-reward distribution. Expected transient: ~500-3000 steps, dominated by `RL_REWARD_CLAMP_V_BOUND_EWMA_ALPHA = 0.001` (τ=1000 steps; ~3τ to 95% convergence). Mitigation: §2 robustness clause extends verdict window to steps 3000-4000 if clamp bounds haven't stabilized by step 2000. If controllers diverge or oscillate, diag will show; may require explicit bootstrap of those controllers to known-good mode=1 values in a follow-up.
- **Risk B (HIGH unmitigated)**: popart eval-boundary shock per `pearl_popart_reset_at_eval_boundary_shocks_normalization` is NOT fixed in current code. Mode=1 inherits this bug. Eval-side Pearson target is relaxed to 0.5 (not 0.7). Train-side is primary; eval-side improvement requires the separate popart-preservation fix.
- **Risk C (MEDIUM)**: Removing entry_cost may cause the agent to learn "free entries are good" → open spuriously. Mitigation: realized_pnl_delta naturally penalizes round-trips (spread + fees if LobSim accounts for them). If wr trajectory shows hyperactivity post-fix, re-introduce entry_cost or add a fixed open-event penalty.
- **Risk D (MEDIUM)**: Sparse reward (zero most steps) means V regression learns target ≈ γ × V_{t+1} chained until close-event. With γ=0.99 and 30-step hold, credit assignment to position-open step is attenuated by γ^30 = 0.74. Standard sparse-RL convergence; foxhunt uses 1-step TD bootstrap per `compute_advantage_return.cu` so this is the standard sparse-pnl learning regime. If learning is too slow at 2k smoke, extend to 5k or 10k cluster.
- **Risk E (LOW)**: 1-step TD is the actual algorithm (NOT GAE as my v3 claimed). With sparse rewards 1-step TD has known longer mixing time vs GAE-λ. Mitigation: standard, not problematic at 2k+ training steps.
- **Risk F (LOW)**: outcome_ema in Phase 7 reads `r` after Phase 5+5b. In mode=1 r = realized_pnl_delta, so outcome_ema = sign(realized_pnl_delta) which is the canonical correct signal. No bias introduced (reviewer H6 resolved by dropping entry_cost in mode=1).

## 6. Deletion timeline (the "permanent redesign" commitment)

This is a permanent reward redesign, not a permanent ISV toggle.

- **Commit A** (this spec): ship mode=1 default + mode=0 legacy path.
- **Cluster validation V2 + V3**: mode=1 produces verdict; mode=0 confirms baseline reproduction.
- **Commit B** (follow-up, 24-48 hours after V2/V3): if mode=1 succeeded → delete the mode=0 branch entirely from the kernel + delete `RL_REWARD_PURE_PNL_MODE_INDEX` + reduce ISV slot count back. This satisfies `feedback_no_feature_flags` + `feedback_single_source_of_truth_no_duplicates`.
- **If V2 falsified**: mode=0 is the surviving path; revert this commit (`git revert <Commit A>`).

The ISV-toggle pattern is here ONLY for the regression run (V3). Not a permanent feature flag.

## 7. What this is NOT

- NOT Phase 2 of edge-decay (deferred until reward is aligned)
- NOT architecture simplification (separate spec, conditional on this one's outcome)
- NOT another controller (zero new EMAs, no new feedback loops)
- NOT a permanent toggle (mode=0 path deletes after V2/V3 verdict)

## 8. Why this is the essential 65th-commit fix

**Data**: 2 cluster runs at b=1024 × 20k+5k steps. Pearson(rewards.sum, Δrealized_pnl) = 0.16 train / 0.31 eval. wr_max = 0.418 momentary, wr_final = 0.343 — architecture finds patterns then un-learns them.

**Kernel**: Phase 5 step 4 adds `hold_bonus × √hold_time = 2.0 × 10 = +20 per held step` for 100-step holds. Realized close pnl ≈ +$10 ≈ +0.5 σ-units. **Domination ratio ~40× per step, ~2600× per held trade.**

**Mechanism**: V regression learns to predict the inflated return target; close-event advantages become small or wrong-signed; PPO and Q distill un-learn closing actions on actual edge.

**Fix**: ISV-gated single branch removing Phase 5 steps 1-4 (legacy preserved for regression). 20-line kernel modification.

**Falsification**: Pearson(rewards.sum, Δrealized_pnl) ≥ 0.7 in train steps 1500-2000 (post-controller-rebalance transient). Sign agreement ≥ 0.65. wr ≥ 0.30 at step 2000. Cluster smoke produces verdict.

**Abandonment**: 1 retry with R-multiple variant. If still < 0.7, declare reward path closed and proceed to architecture simplification.

**Deletion**: mode=0 branch deleted within 24-48 hours of V2/V3 verdict per `feedback_no_feature_flags`.

## 9. Linked memory

- [[pearl_reward_signal_anti_aligned_with_pnl]] — data finding this spec acts on (updated to reference `realized_pnl_cum_usd` as clean reference)
- [[pearl_popart_reset_at_eval_boundary_shocks_normalization]] — Risk B (unmitigated; eval Pearson target relaxed accordingly)
- [[feedback_no_feature_flags]] — satisfied via §6 deletion timeline
- [[feedback_single_source_of_truth_no_duplicates]] — same
- [[feedback_going_in_circles_pattern]] — this spec is the antidote: load-bearing root cause from existing data, surgical fix, predetermined falsification with abandonment timeline
- [[pearl_edge_decay_detector_phase1_validated_train_also_decays]] — explains why train ph_mean rises: shaped reward un-trains profitable patterns continuously during training
- [[feedback_trust_code_not_docs]] — v3 spec failed this by inventing magnitudes; v4 verified actual bootstrap values
