spec: GPU-native Hindsight Experience Replay for wave-exit timing
Backward-looking HER: on trade close, compute peak unrealized PnL during the trade's lifetime. If peak >> actual, inject synthetic transition with peak_pnl and boosted priority. Teaches the agent optimal wave-exit timing 10× faster than sparse reward alone. Two kernels: rl_hindsight_track (per-step mid accumulation) and rl_hindsight_inject (synthetic push on done with coordination). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
218
docs/superpowers/specs/2026-05-26-gpu-hindsight-replay.md
Normal file
218
docs/superpowers/specs/2026-05-26-gpu-hindsight-replay.md
Normal file
@@ -0,0 +1,218 @@
|
||||
# GPU-Native Hindsight Experience Replay (HER)
|
||||
|
||||
**Goal:** Accelerate wave-riding exit timing by showing the agent "where you SHOULD have exited" on every closed trade. Integrates into the GPU PER pipeline with zero host involvement.
|
||||
|
||||
**Prerequisite:** GPU PER (Tasks 1-4 of the GPU PER plan) must be wired first.
|
||||
|
||||
---
|
||||
|
||||
## Motivation: Why HER for the Surfer
|
||||
|
||||
The surfer philosophy: wait for the wave, ride it out, accept wipeouts.
|
||||
|
||||
The core learning challenge: the agent closes a trade with +$50. Was that good? Without context, yes. But the mid price during that trade peaked at +$300 before reversing — the agent missed 83% of the wave by exiting late (after the reversal). Without HER, the +$50 reward is all the agent sees. It takes thousands of similar trades to learn the exit timing pattern.
|
||||
|
||||
**With HER:** On every trade close, we compute the peak unrealized P&L during the trade's lifetime. If peak >> actual, we inject a synthetic "optimal exit" transition into the replay buffer with boosted priority. The Q-network now sees both:
|
||||
- "At state h_entry, holding to actual_exit gave +$50"
|
||||
- "At state h_peak, exiting at peak gave +$300" (synthetic, high priority)
|
||||
|
||||
The agent learns exit timing 10-100× faster because it sees the counterfactual on EVERY trade, not just the realized outcome.
|
||||
|
||||
---
|
||||
|
||||
## Architecture: Backward-Looking HER (no future data needed)
|
||||
|
||||
The old DQN trainer used FORWARD-looking hindsight (lookahead into future bars). This requires future price data that doesn't exist at push time in the step-by-step RL loop.
|
||||
|
||||
**This design uses BACKWARD-looking HER:** at trade close, we look at the prices DURING the trade (from entry to exit) and find the peak. This is always available — we just need a per-batch mid-price ring buffer that accumulates during the trade.
|
||||
|
||||
### Data flow
|
||||
|
||||
```
|
||||
Step N (trade open):
|
||||
mid_ring[b][0] = current_mid ← start tracking
|
||||
|
||||
Step N+1 to N+K (holding):
|
||||
mid_ring[b][step_in_trade] = current_mid ← accumulate
|
||||
|
||||
Step N+K (trade close, done=1):
|
||||
peak_mid = max(mid_ring[b][0..K])
|
||||
peak_pnl = |peak_mid - entry_price| × lots ← best possible PnL
|
||||
actual_pnl = reward (from extract_realized_pnl_delta)
|
||||
|
||||
IF peak_pnl > actual_pnl × HINDSIGHT_THRESHOLD:
|
||||
→ inject synthetic transition with peak_pnl as reward
|
||||
→ set priority = peak_pnl × HINDSIGHT_PRIORITY_BOOST
|
||||
```
|
||||
|
||||
### Why this works for the surfer
|
||||
|
||||
| Trade outcome | HER signal |
|
||||
|---------------|-----------|
|
||||
| Exit too early (wave still rising) | peak >> actual → "hold longer next time" |
|
||||
| Exit too late (wave reversed) | peak >> actual → "exit at the peak" |
|
||||
| Good ride (exited near peak) | peak ≈ actual → no synthetic (already optimal) |
|
||||
| Wipeout (wave crashed, trailed out) | peak may be small → modest synthetic |
|
||||
|
||||
The agent learns: "states where mid was rising rapidly → keep holding. States where mid peaked and reversed → exit NOW."
|
||||
|
||||
---
|
||||
|
||||
## Device-Resident Data Structures
|
||||
|
||||
```rust
|
||||
// In GpuReplayBuffer or as separate HER struct:
|
||||
pub struct GpuHindsight {
|
||||
/// Per-batch mid-price ring during current trade.
|
||||
/// Ring fills from entry to exit; resets on done.
|
||||
pub mid_ring_d: CudaSlice<f32>, // [B × HINDSIGHT_RING_LEN]
|
||||
|
||||
/// Per-batch write index into mid_ring (0-based from trade entry).
|
||||
pub ring_write_idx_d: CudaSlice<u32>, // [B]
|
||||
|
||||
/// Per-batch peak mid during current trade (running max).
|
||||
pub peak_mid_d: CudaSlice<f32>, // [B]
|
||||
|
||||
/// Per-batch entry mid (captured at trade open).
|
||||
pub entry_mid_d: CudaSlice<f32>, // [B]
|
||||
}
|
||||
|
||||
pub const HINDSIGHT_RING_LEN: usize = 512;
|
||||
```
|
||||
|
||||
The ring length of 512 covers trades up to 512 steps long. At the surfer's min-hold of 100 steps, typical trades run 100-300 steps. 512 is generous.
|
||||
|
||||
Memory: `B × 512 × 4B = 256 × 512 × 4 = 512 KB` — negligible.
|
||||
|
||||
---
|
||||
|
||||
## Kernels
|
||||
|
||||
### `rl_hindsight_track.cu` — Per-step mid-price accumulation
|
||||
|
||||
Runs every step AFTER the lobsim fill (has current bid/ask). Accumulates mid into ring for positioned batches.
|
||||
|
||||
```
|
||||
Grid=(ceil(B/32)), Block=(32)
|
||||
```
|
||||
|
||||
Per thread (one per batch):
|
||||
1. If position_lots == 0: ring_write_idx[b] = 0, return (flat, no tracking)
|
||||
2. mid = 0.5 × (bid_px[0] + ask_px[0])
|
||||
3. If ring_write_idx[b] == 0: entry_mid[b] = mid (first step of trade)
|
||||
4. idx = min(ring_write_idx[b], HINDSIGHT_RING_LEN - 1)
|
||||
5. mid_ring[b × RING_LEN + idx] = mid
|
||||
6. peak_mid[b] = fmaxf(peak_mid[b], mid) (running max for long) or fminf for short — direction from sign(position_lots)
|
||||
7. ring_write_idx[b]++
|
||||
|
||||
### `rl_hindsight_inject.cu` — Synthetic transition injection on trade close
|
||||
|
||||
Runs AFTER `rl_per_push` (which already handled the real transition). Only fires for batch elements where done=1 AND peak was significantly better than actual.
|
||||
|
||||
```
|
||||
Grid=(1), Block=(B) — single block for write_head coordination (same pattern as rl_per_push)
|
||||
```
|
||||
|
||||
Per thread (one per batch):
|
||||
1. If dones[b] <= 0.5: return (no trade close, skip)
|
||||
2. entry = entry_mid[b]
|
||||
3. peak = peak_mid[b]
|
||||
4. actual_reward = rewards_raw[b] (the actual trade P&L)
|
||||
5. lots_sign = (position was long) ? +1 : -1
|
||||
6. peak_pnl = lots_sign × (peak - entry) × lot_value (or just `peak - entry` if in normalized space)
|
||||
7. IF peak_pnl <= actual_reward × HINDSIGHT_THRESHOLD: return (trade was already near-optimal)
|
||||
8. **Inject synthetic transition:**
|
||||
- h_t = replay's h_t at the ENTRY of this trade (we need to know where it was pushed)
|
||||
- Actually simpler: use the CURRENT h_t (from the close step) — the Q-network learns "at states like this, the optimal reward was peak_pnl"
|
||||
- action = same action as taken (the timing info is in the reward signal, not the action)
|
||||
- reward = peak_pnl (the counterfactual optimal)
|
||||
- priority = HINDSIGHT_PRIORITY_BOOST × |peak_pnl|^α
|
||||
9. Write to replay at write_head (shared-mem prefix-sum for multi-batch coordination)
|
||||
10. Reset: ring_write_idx[b] = 0, peak_mid[b] = 0
|
||||
|
||||
### ISV Slots
|
||||
|
||||
```
|
||||
RL_HINDSIGHT_THRESHOLD_INDEX = 549 // default 1.5 (inject if peak > 1.5× actual)
|
||||
RL_HINDSIGHT_PRIORITY_BOOST_INDEX = 550 // default 3.0 (3× normal max_priority)
|
||||
RL_HINDSIGHT_INJECT_COUNT_INDEX = 551 // diag: count of injections this step
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration into Step Pipeline
|
||||
|
||||
```
|
||||
step_with_lobsim:
|
||||
...
|
||||
[LobSim fill]
|
||||
[rl_fused_reward_pipeline] ← extracts reward, done
|
||||
[rl_hindsight_track] ← NEW: accumulate mid into ring
|
||||
[rl_per_push] ← push real transition
|
||||
[rl_hindsight_inject] ← NEW: push synthetic if peak >> actual
|
||||
[rl_per_tree_rebuild] ← rebuild after both pushes
|
||||
...
|
||||
[rl_per_sample] ← samples from both real + synthetic
|
||||
[step_synthetic] ← trains on both
|
||||
[rl_per_update_priority]
|
||||
[rl_per_tree_rebuild] ← final rebuild
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Surfer Validation Metrics
|
||||
|
||||
In the diag JSONL:
|
||||
```json
|
||||
{
|
||||
"hindsight": {
|
||||
"inject_count_step": 5,
|
||||
"inject_count_total": 12847,
|
||||
"mean_peak_ratio": 2.8,
|
||||
"mean_missed_pnl": 142.50
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `inject_count`: how many synthetic transitions added per step (should be ~30% of trades at threshold=1.5)
|
||||
- `mean_peak_ratio`: average peak_pnl / actual_pnl for injected trades (how much the agent is leaving on the table)
|
||||
- `mean_missed_pnl`: average |peak - actual| in USD (the dollar opportunity being taught)
|
||||
|
||||
As the agent learns wave-riding:
|
||||
- `inject_count` should DECREASE (fewer trades where peak >> actual)
|
||||
- `mean_peak_ratio` should approach 1.0 (agent exits closer to peak)
|
||||
- `avg_hold_steps` should INCREASE toward the wave timescale (50-100 steps)
|
||||
|
||||
---
|
||||
|
||||
## Constraints
|
||||
|
||||
- Per `feedback_no_atomicadd`: write_head coordination via prefix-sum (same as rl_per_push)
|
||||
- Per `feedback_cpu_is_read_only`: all computation on device
|
||||
- Per `feedback_no_stubs`: kernel wired in same commit as struct allocation
|
||||
- Per `pearl_one_unbounded_signal_per_reward`: the synthetic reward IS bounded by peak_pnl (which is bounded by the actual price move within 512 steps — finite by construction)
|
||||
- Per `pearl_event_driven_reward_density_alignment`: synthetic transitions only inject on done events (event-driven, same density as trade closes)
|
||||
|
||||
---
|
||||
|
||||
## Expected Impact
|
||||
|
||||
At current avg_hold=13 (broken — trail fix will lift to ~100+):
|
||||
- Without HER: agent needs ~200k trades to learn optimal exit timing from sparse reward alone
|
||||
- With HER: ~20k trades (10× acceleration) — each losing trade produces a counterfactual showing the peak
|
||||
|
||||
For the surfer philosophy specifically:
|
||||
- The agent currently exits at min_hold (100 steps) or trail stop
|
||||
- HER shows it "if you held 30 more steps past min_hold, the wave peaked at +$300"
|
||||
- This teaches the VALUE OF PATIENCE beyond the min_hold floor
|
||||
- Within 50k steps of trading (at b=256 with 7% done rate = ~900k trade events), the agent should show avg_hold >> min_hold and decreasing inject_count
|
||||
|
||||
---
|
||||
|
||||
## Kill Criteria
|
||||
|
||||
- Synthetic transitions appear in replay (inject_count > 0 on steps with dones)
|
||||
- mean_peak_ratio > 1.0 (confirms genuine missed opportunity detection)
|
||||
- After 100k steps: inject_count trend downward (agent improving)
|
||||
- After 200k steps: avg_hold_steps > min_hold × 1.5 (agent holding beyond minimum)
|
||||
- Q-loss trajectory identical or better vs non-HER baseline (no destabilization)
|
||||
Reference in New Issue
Block a user