diff --git a/docs/superpowers/specs/2026-05-26-gpu-hindsight-replay.md b/docs/superpowers/specs/2026-05-26-gpu-hindsight-replay.md new file mode 100644 index 000000000..a9a842152 --- /dev/null +++ b/docs/superpowers/specs/2026-05-26-gpu-hindsight-replay.md @@ -0,0 +1,218 @@ +# GPU-Native Hindsight Experience Replay (HER) + +**Goal:** Accelerate wave-riding exit timing by showing the agent "where you SHOULD have exited" on every closed trade. Integrates into the GPU PER pipeline with zero host involvement. + +**Prerequisite:** GPU PER (Tasks 1-4 of the GPU PER plan) must be wired first. + +--- + +## Motivation: Why HER for the Surfer + +The surfer philosophy: wait for the wave, ride it out, accept wipeouts. + +The core learning challenge: the agent closes a trade with +$50. Was that good? Without context, yes. But the mid price during that trade peaked at +$300 before reversing — the agent missed 83% of the wave by exiting late (after the reversal). Without HER, the +$50 reward is all the agent sees. It takes thousands of similar trades to learn the exit timing pattern. + +**With HER:** On every trade close, we compute the peak unrealized P&L during the trade's lifetime. If peak >> actual, we inject a synthetic "optimal exit" transition into the replay buffer with boosted priority. The Q-network now sees both: +- "At state h_entry, holding to actual_exit gave +$50" +- "At state h_peak, exiting at peak gave +$300" (synthetic, high priority) + +The agent learns exit timing 10-100× faster because it sees the counterfactual on EVERY trade, not just the realized outcome. + +--- + +## Architecture: Backward-Looking HER (no future data needed) + +The old DQN trainer used FORWARD-looking hindsight (lookahead into future bars). This requires future price data that doesn't exist at push time in the step-by-step RL loop. + +**This design uses BACKWARD-looking HER:** at trade close, we look at the prices DURING the trade (from entry to exit) and find the peak. This is always available — we just need a per-batch mid-price ring buffer that accumulates during the trade. + +### Data flow + +``` +Step N (trade open): + mid_ring[b][0] = current_mid ← start tracking + +Step N+1 to N+K (holding): + mid_ring[b][step_in_trade] = current_mid ← accumulate + +Step N+K (trade close, done=1): + peak_mid = max(mid_ring[b][0..K]) + peak_pnl = |peak_mid - entry_price| × lots ← best possible PnL + actual_pnl = reward (from extract_realized_pnl_delta) + + IF peak_pnl > actual_pnl × HINDSIGHT_THRESHOLD: + → inject synthetic transition with peak_pnl as reward + → set priority = peak_pnl × HINDSIGHT_PRIORITY_BOOST +``` + +### Why this works for the surfer + +| Trade outcome | HER signal | +|---------------|-----------| +| Exit too early (wave still rising) | peak >> actual → "hold longer next time" | +| Exit too late (wave reversed) | peak >> actual → "exit at the peak" | +| Good ride (exited near peak) | peak ≈ actual → no synthetic (already optimal) | +| Wipeout (wave crashed, trailed out) | peak may be small → modest synthetic | + +The agent learns: "states where mid was rising rapidly → keep holding. States where mid peaked and reversed → exit NOW." + +--- + +## Device-Resident Data Structures + +```rust +// In GpuReplayBuffer or as separate HER struct: +pub struct GpuHindsight { + /// Per-batch mid-price ring during current trade. + /// Ring fills from entry to exit; resets on done. + pub mid_ring_d: CudaSlice, // [B × HINDSIGHT_RING_LEN] + + /// Per-batch write index into mid_ring (0-based from trade entry). + pub ring_write_idx_d: CudaSlice, // [B] + + /// Per-batch peak mid during current trade (running max). + pub peak_mid_d: CudaSlice, // [B] + + /// Per-batch entry mid (captured at trade open). + pub entry_mid_d: CudaSlice, // [B] +} + +pub const HINDSIGHT_RING_LEN: usize = 512; +``` + +The ring length of 512 covers trades up to 512 steps long. At the surfer's min-hold of 100 steps, typical trades run 100-300 steps. 512 is generous. + +Memory: `B × 512 × 4B = 256 × 512 × 4 = 512 KB` — negligible. + +--- + +## Kernels + +### `rl_hindsight_track.cu` — Per-step mid-price accumulation + +Runs every step AFTER the lobsim fill (has current bid/ask). Accumulates mid into ring for positioned batches. + +``` +Grid=(ceil(B/32)), Block=(32) +``` + +Per thread (one per batch): +1. If position_lots == 0: ring_write_idx[b] = 0, return (flat, no tracking) +2. mid = 0.5 × (bid_px[0] + ask_px[0]) +3. If ring_write_idx[b] == 0: entry_mid[b] = mid (first step of trade) +4. idx = min(ring_write_idx[b], HINDSIGHT_RING_LEN - 1) +5. mid_ring[b × RING_LEN + idx] = mid +6. peak_mid[b] = fmaxf(peak_mid[b], mid) (running max for long) or fminf for short — direction from sign(position_lots) +7. ring_write_idx[b]++ + +### `rl_hindsight_inject.cu` — Synthetic transition injection on trade close + +Runs AFTER `rl_per_push` (which already handled the real transition). Only fires for batch elements where done=1 AND peak was significantly better than actual. + +``` +Grid=(1), Block=(B) — single block for write_head coordination (same pattern as rl_per_push) +``` + +Per thread (one per batch): +1. If dones[b] <= 0.5: return (no trade close, skip) +2. entry = entry_mid[b] +3. peak = peak_mid[b] +4. actual_reward = rewards_raw[b] (the actual trade P&L) +5. lots_sign = (position was long) ? +1 : -1 +6. peak_pnl = lots_sign × (peak - entry) × lot_value (or just `peak - entry` if in normalized space) +7. IF peak_pnl <= actual_reward × HINDSIGHT_THRESHOLD: return (trade was already near-optimal) +8. **Inject synthetic transition:** + - h_t = replay's h_t at the ENTRY of this trade (we need to know where it was pushed) + - Actually simpler: use the CURRENT h_t (from the close step) — the Q-network learns "at states like this, the optimal reward was peak_pnl" + - action = same action as taken (the timing info is in the reward signal, not the action) + - reward = peak_pnl (the counterfactual optimal) + - priority = HINDSIGHT_PRIORITY_BOOST × |peak_pnl|^α +9. Write to replay at write_head (shared-mem prefix-sum for multi-batch coordination) +10. Reset: ring_write_idx[b] = 0, peak_mid[b] = 0 + +### ISV Slots + +``` +RL_HINDSIGHT_THRESHOLD_INDEX = 549 // default 1.5 (inject if peak > 1.5× actual) +RL_HINDSIGHT_PRIORITY_BOOST_INDEX = 550 // default 3.0 (3× normal max_priority) +RL_HINDSIGHT_INJECT_COUNT_INDEX = 551 // diag: count of injections this step +``` + +--- + +## Integration into Step Pipeline + +``` +step_with_lobsim: + ... + [LobSim fill] + [rl_fused_reward_pipeline] ← extracts reward, done + [rl_hindsight_track] ← NEW: accumulate mid into ring + [rl_per_push] ← push real transition + [rl_hindsight_inject] ← NEW: push synthetic if peak >> actual + [rl_per_tree_rebuild] ← rebuild after both pushes + ... + [rl_per_sample] ← samples from both real + synthetic + [step_synthetic] ← trains on both + [rl_per_update_priority] + [rl_per_tree_rebuild] ← final rebuild +``` + +--- + +## Surfer Validation Metrics + +In the diag JSONL: +```json +{ + "hindsight": { + "inject_count_step": 5, + "inject_count_total": 12847, + "mean_peak_ratio": 2.8, + "mean_missed_pnl": 142.50 + } +} +``` + +- `inject_count`: how many synthetic transitions added per step (should be ~30% of trades at threshold=1.5) +- `mean_peak_ratio`: average peak_pnl / actual_pnl for injected trades (how much the agent is leaving on the table) +- `mean_missed_pnl`: average |peak - actual| in USD (the dollar opportunity being taught) + +As the agent learns wave-riding: +- `inject_count` should DECREASE (fewer trades where peak >> actual) +- `mean_peak_ratio` should approach 1.0 (agent exits closer to peak) +- `avg_hold_steps` should INCREASE toward the wave timescale (50-100 steps) + +--- + +## Constraints + +- Per `feedback_no_atomicadd`: write_head coordination via prefix-sum (same as rl_per_push) +- Per `feedback_cpu_is_read_only`: all computation on device +- Per `feedback_no_stubs`: kernel wired in same commit as struct allocation +- Per `pearl_one_unbounded_signal_per_reward`: the synthetic reward IS bounded by peak_pnl (which is bounded by the actual price move within 512 steps — finite by construction) +- Per `pearl_event_driven_reward_density_alignment`: synthetic transitions only inject on done events (event-driven, same density as trade closes) + +--- + +## Expected Impact + +At current avg_hold=13 (broken — trail fix will lift to ~100+): +- Without HER: agent needs ~200k trades to learn optimal exit timing from sparse reward alone +- With HER: ~20k trades (10× acceleration) — each losing trade produces a counterfactual showing the peak + +For the surfer philosophy specifically: +- The agent currently exits at min_hold (100 steps) or trail stop +- HER shows it "if you held 30 more steps past min_hold, the wave peaked at +$300" +- This teaches the VALUE OF PATIENCE beyond the min_hold floor +- Within 50k steps of trading (at b=256 with 7% done rate = ~900k trade events), the agent should show avg_hold >> min_hold and decreasing inject_count + +--- + +## Kill Criteria + +- Synthetic transitions appear in replay (inject_count > 0 on steps with dones) +- mean_peak_ratio > 1.0 (confirms genuine missed opportunity detection) +- After 100k steps: inject_count trend downward (agent improving) +- After 200k steps: avg_hold_steps > min_hold × 1.5 (agent holding beyond minimum) +- Q-loss trajectory identical or better vs non-HER baseline (no destabilization)