# GPU-Native Hindsight Experience Replay (HER)

**Goal:** Accelerate wave-riding exit timing by showing the agent "where you SHOULD have exited" on every closed trade. Integrates into the GPU PER pipeline with zero host involvement.

**Prerequisite:** GPU PER (Tasks 1-4 of the GPU PER plan) must be wired first.

---

## Motivation: Why HER for the Surfer

The surfer philosophy: wait for the wave, ride it out, accept wipeouts.

The core learning challenge: the agent closes a trade with +$50. Was that good? Without context, yes. But the mid price during that trade peaked at +$300 before reversing — the agent missed 83% of the wave by exiting late (after the reversal). Without HER, the +$50 reward is all the agent sees. It takes thousands of similar trades to learn the exit timing pattern.

**With HER:** On every trade close, we compute the peak unrealized P&L during the trade's lifetime. If peak >> actual, we inject a synthetic "optimal exit" transition into the replay buffer with boosted priority. The Q-network now sees both:
- "At state h_entry, holding to actual_exit gave +$50"  
- "At state h_peak, exiting at peak gave +$300" (synthetic, high priority)

The agent learns exit timing 10-100× faster because it sees the counterfactual on EVERY trade, not just the realized outcome.

---

## Architecture: Bidirectional HER (backward + forward)

Two complementary hindsight signals, both GPU-native:

**Backward HER (immediate at close):** Peak unrealized PnL DURING the trade. "You held past the peak — should have exited at step Z." Teaches anti-wipeout timing.

**Forward HER (delayed by LOOKAHEAD steps):** What happened AFTER exit. "Price kept rising 10 steps after you exited — should have held." Teaches patience / wave-riding.

Both are available in the sequential training loop:
- Backward: fires at trade close (peak is in the mid_ring)
- Forward: fires LOOKAHEAD steps after a trade close (subsequent steps reveal post-exit price action)

### Data flow

```
Step N (trade open):
  mid_ring[b][0] = current_mid   ← start tracking

Step N+1 to N+K (holding):
  mid_ring[b][step_in_trade] = current_mid   ← accumulate

Step N+K (trade close, done=1):
  peak_mid = max(mid_ring[b][0..K])
  peak_pnl = |peak_mid - entry_price| × lots   ← best possible PnL
  actual_pnl = reward (from extract_realized_pnl_delta)
  
  IF peak_pnl > actual_pnl × HINDSIGHT_THRESHOLD:
    → inject synthetic transition with peak_pnl as reward
    → set priority = peak_pnl × HINDSIGHT_PRIORITY_BOOST
```

### Why this works for the surfer

| Trade outcome | HER signal |
|---------------|-----------|
| Exit too early (wave still rising) | peak >> actual → "hold longer next time" |
| Exit too late (wave reversed) | peak >> actual → "exit at the peak" |
| Good ride (exited near peak) | peak ≈ actual → no synthetic (already optimal) |
| Wipeout (wave crashed, trailed out) | peak may be small → modest synthetic |

The agent learns: "states where mid was rising rapidly → keep holding. States where mid peaked and reversed → exit NOW."

---

## Device-Resident Data Structures

```rust
pub struct GpuHindsight {
    // ── Backward HER (peak during trade) ────────────────────────
    /// Per-batch mid-price ring during current trade.
    pub mid_ring_d: CudaSlice<f32>,        // [B × HINDSIGHT_RING_LEN]
    /// Per-batch write index into mid_ring.
    pub ring_write_idx_d: CudaSlice<u32>,  // [B]
    /// Per-batch running peak mid during current trade.
    pub peak_mid_d: CudaSlice<f32>,        // [B]
    /// Per-batch entry mid (captured at trade open).
    pub entry_mid_d: CudaSlice<f32>,       // [B]

    // ── Forward HER (post-exit continuation) ────────────────────
    /// Recently-closed trades waiting for forward evaluation.
    /// Ring of (entry_price, exit_step, direction, actual_pnl, batch_idx).
    pub closed_ring_d: CudaSlice<f32>,     // [CLOSED_RING_SIZE × 5]
    /// h_t at entry for each closed trade (for synthetic transition).
    pub closed_h_t_d: CudaSlice<f32>,      // [CLOSED_RING_SIZE × HIDDEN_DIM]
    /// Write head into closed ring.
    pub closed_write_head_d: CudaSlice<u32>, // [1]
    /// Count of valid entries in closed ring.
    pub closed_count_d: CudaSlice<u32>,    // [1]
}

pub const HINDSIGHT_RING_LEN: usize = 512;
pub const CLOSED_RING_SIZE: usize = 256;  // last 256 closed trades
pub const FORWARD_LOOKAHEAD: usize = 50;  // evaluate 50 steps after close
```

The ring length of 512 covers trades up to 512 steps long. At the surfer's min-hold of 100 steps, typical trades run 100-300 steps. 512 is generous.

Memory: `B × 512 × 4B = 256 × 512 × 4 = 512 KB` — negligible.

---

## Kernels

### `rl_hindsight_track.cu` — Per-step mid-price accumulation

Runs every step AFTER the lobsim fill (has current bid/ask). Accumulates mid into ring for positioned batches.

```
Grid=(ceil(B/32)), Block=(32)
```

Per thread (one per batch):
1. If position_lots == 0: ring_write_idx[b] = 0, return (flat, no tracking)
2. mid = 0.5 × (bid_px[0] + ask_px[0])
3. If ring_write_idx[b] == 0: entry_mid[b] = mid (first step of trade)
4. idx = min(ring_write_idx[b], HINDSIGHT_RING_LEN - 1)
5. mid_ring[b × RING_LEN + idx] = mid
6. peak_mid[b] = fmaxf(peak_mid[b], mid) (running max for long) or fminf for short — direction from sign(position_lots)
7. ring_write_idx[b]++

### `rl_hindsight_inject.cu` — Synthetic transition injection on trade close

Runs AFTER `rl_per_push` (which already handled the real transition). Only fires for batch elements where done=1 AND peak was significantly better than actual.

```
Grid=(1), Block=(B) — single block for write_head coordination (same pattern as rl_per_push)
```

Per thread (one per batch):
1. If dones[b] <= 0.5: return (no trade close, skip)
2. entry = entry_mid[b]
3. peak = peak_mid[b]
4. actual_reward = rewards_raw[b] (the actual trade P&L)
5. lots_sign = (position was long) ? +1 : -1
6. peak_pnl = lots_sign × (peak - entry) × lot_value (or just `peak - entry` if in normalized space)
7. IF peak_pnl <= actual_reward × HINDSIGHT_THRESHOLD: return (trade was already near-optimal)
8. **Inject synthetic transition:**
   - h_t = replay's h_t at the ENTRY of this trade (we need to know where it was pushed)
   - Actually simpler: use the CURRENT h_t (from the close step) — the Q-network learns "at states like this, the optimal reward was peak_pnl"
   - action = same action as taken (the timing info is in the reward signal, not the action)
   - reward = peak_pnl (the counterfactual optimal)
   - priority = HINDSIGHT_PRIORITY_BOOST × |peak_pnl|^α
9. Write to replay at write_head (shared-mem prefix-sum for multi-batch coordination)
10. Reset: ring_write_idx[b] = 0, peak_mid[b] = 0

### `rl_hindsight_forward.cu` — Forward HER: "should have held longer"

Runs every step. Scans the closed_ring for trades that closed FORWARD_LOOKAHEAD steps ago. For each, computes what the PnL would be NOW if the trade had been held.

```
Grid=(ceil(CLOSED_RING_SIZE/32)), Block=(32)
```

Per thread (one per closed_ring entry):
1. If entry is empty or age < FORWARD_LOOKAHEAD: return
2. Read entry_price, direction from closed_ring
3. current_mid = 0.5 × (bid_px[0] + ask_px[0])
4. forward_pnl = direction × (current_mid - entry_price)
5. actual_pnl = closed_ring[i].actual_pnl
6. IF forward_pnl > actual_pnl × HINDSIGHT_THRESHOLD:
   - Synthetic transition: h_t = closed_h_t[i], action = Hold, reward = forward_pnl
   - Push to replay with boosted priority
   - Mark entry as consumed (clear slot)
7. IF age > FORWARD_LOOKAHEAD × 2: expire entry (clear slot regardless)

This teaches: "at this entry state, holding past your exit point would have yielded +X more."

### ISV Slots

```
RL_HINDSIGHT_THRESHOLD_INDEX = 549     // default 1.5 (inject if peak > 1.5× actual)
RL_HINDSIGHT_PRIORITY_BOOST_INDEX = 550 // default 3.0 (3× normal max_priority)
RL_HINDSIGHT_INJECT_COUNT_INDEX = 551  // diag: count of injections this step
```

---

## Integration into Step Pipeline

```
step_with_lobsim:
  ...
  [LobSim fill]
  [rl_fused_reward_pipeline]     ← extracts reward, done
  [rl_hindsight_track]           ← NEW: accumulate mid into ring
  [rl_per_push]                  ← push real transition
  [rl_hindsight_inject]          ← NEW: push synthetic if peak >> actual
  [rl_per_tree_rebuild]          ← rebuild after both pushes
  ...
  [rl_per_sample]                ← samples from both real + synthetic
  [step_synthetic]               ← trains on both
  [rl_per_update_priority]       
  [rl_per_tree_rebuild]          ← final rebuild
```

---

## Surfer Validation Metrics

In the diag JSONL:
```json
{
  "hindsight": {
    "inject_count_step": 5,
    "inject_count_total": 12847,
    "mean_peak_ratio": 2.8,
    "mean_missed_pnl": 142.50
  }
}
```

- `inject_count`: how many synthetic transitions added per step (should be ~30% of trades at threshold=1.5)
- `mean_peak_ratio`: average peak_pnl / actual_pnl for injected trades (how much the agent is leaving on the table)
- `mean_missed_pnl`: average |peak - actual| in USD (the dollar opportunity being taught)

As the agent learns wave-riding:
- `inject_count` should DECREASE (fewer trades where peak >> actual)
- `mean_peak_ratio` should approach 1.0 (agent exits closer to peak)
- `avg_hold_steps` should INCREASE toward the wave timescale (50-100 steps)

---

## Constraints

- Per `feedback_no_atomicadd`: write_head coordination via prefix-sum (same as rl_per_push)
- Per `feedback_cpu_is_read_only`: all computation on device
- Per `feedback_no_stubs`: kernel wired in same commit as struct allocation
- Per `pearl_one_unbounded_signal_per_reward`: the synthetic reward IS bounded by peak_pnl (which is bounded by the actual price move within 512 steps — finite by construction)
- Per `pearl_event_driven_reward_density_alignment`: synthetic transitions only inject on done events (event-driven, same density as trade closes)

---

## Expected Impact

At current avg_hold=13 (broken — trail fix will lift to ~100+):
- Without HER: agent needs ~200k trades to learn optimal exit timing from sparse reward alone
- With HER: ~20k trades (10× acceleration) — each losing trade produces a counterfactual showing the peak

For the surfer philosophy specifically:
- The agent currently exits at min_hold (100 steps) or trail stop
- HER shows it "if you held 30 more steps past min_hold, the wave peaked at +$300"
- This teaches the VALUE OF PATIENCE beyond the min_hold floor
- Within 50k steps of trading (at b=256 with 7% done rate = ~900k trade events), the agent should show avg_hold >> min_hold and decreasing inject_count

---

## Kill Criteria

- Synthetic transitions appear in replay (inject_count > 0 on steps with dones)
- mean_peak_ratio > 1.0 (confirms genuine missed opportunity detection)
- After 100k steps: inject_count trend downward (agent improving)
- After 200k steps: avg_hold_steps > min_hold × 1.5 (agent holding beyond minimum)
- Q-loss trajectory identical or better vs non-HER baseline (no destabilization)
