# Trainer Rollout-Buffer + GAE Refactor — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Each phase is a separate atomic commit with predetermined falsification gates.

**Goal:** Refactor foxhunt's single-step PPO trainer into a rollout-based architecture with true GAE backward sweep, enabling proper credit assignment + V regression on all states. Empirically required after Phase 1A regression confirmed math-agent's atomicity claim (drop Phase 5 shaping AND add GAE must ship together).

**Architecture:** `[B × T_rollout]` rollout buffer (T=256 default) populated by env-step collection loop without training, then GAE backward sweep kernel produces advantages + returns, then multi-epoch PPO update over minibatches drawn from the rollout. PER continues serving off-policy distributional Q + aux heads. Determinism preserved via seeded shuffling + sequential backward sweep + no atomicAdd.

**Tech Stack:** Rust 1.85, Tokio 1.40, cudarc, CUDA 12.4, sm_86 (RTX 3050 local) / sm_89 (L40S) / sm_90 (H100). Existing crates: `ml-alpha`, `ml-backtesting`. Spec: `docs/superpowers/specs/2026-06-02-trainer-rollout-buffer-gae.md`.

**Branch target:** `ml-alpha-regime-observer` off `cfaa420bd` (Phase 0). Each phase is a separate commit; Phase 1B-D is the cluster-verification gate.

**Prerequisites (must verify before dispatching):**
- `git rev-parse HEAD` returns `cfaa420bd`
- Working tree clean (Phase 1A was reverted)
- `./scripts/determinism-check.sh --quick` exits 0 at HEAD
- `eval_summary.total_pnl_usd ≈ −$4.46M` is the baseline number for Phase 1B-D comparison

---

## Phase 1B-A — ISV slots + RolloutBuffer struct + GAE kernel + GPU-oracle invariant tests

**Goal:** Foundation. New ISV slots allocated, RolloutBuffer struct + GAE backward sweep kernel implemented as standalone components with GPU-oracle tests. No integration with the trainer yet.

**Files:**
- Create: `crates/ml-alpha/src/trainer/rollout_buffer.rs`
- Create: `crates/ml-alpha/cuda/gae_backward_sweep.cu`
- Create: `crates/ml-alpha/tests/rollout_buffer_invariants.rs`
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs` (3 new slots + RL_SLOTS_END bump 758 → 761)
- Modify: `crates/ml-alpha/build.rs` (register new .cu)
- Modify: `crates/ml-alpha/src/trainer/mod.rs` (pub mod rollout_buffer)

### Task A.1 — Allocate ISV slots

- [ ] **Step 1: Add 3 new ISV slots in `isv_slots.rs`**

After the existing `RL_SURFER_SCAFFOLD_*` block (slot 757), insert:

```rust
// ─────────────────────────────────────────────────────────────────────
// Phase 1B (2026-06-02) — Trainer rollout-buffer + GAE refactor.
// Spec: docs/superpowers/specs/2026-06-02-trainer-rollout-buffer-gae.md
// ─────────────────────────────────────────────────────────────────────

/// T_rollout — number of env steps the collection loop runs before
/// invoking GAE + training pass. Bootstrap 256. Higher → more
/// independent advantage observations per epoch, lower → smaller
/// rollout buffer (less GPU memory). Must be a multiple of
/// RL_PPO_MINIBATCH_FRACTION_INDEX's reciprocal so minibatches divide
/// evenly.
pub const RL_PPO_ROLLOUT_HORIZON_INDEX: usize = 758;

/// K_ppo — number of PPO update epochs per rollout. Bootstrap 4
/// (Schulman 2017 default range 4-10). Higher → more data efficiency
/// per env step, but encoder frozen during all epochs (stale features
/// risk).
pub const RL_PPO_N_EPOCHS_INDEX: usize = 759;

/// Minibatches per epoch — divide rollout (B × T) into this many
/// shuffled minibatches per epoch. Bootstrap 8 (standard PPO default).
/// Each minibatch gets one Adam step.
pub const RL_PPO_N_MINIBATCHES_INDEX: usize = 760;
```

Update `RL_SLOTS_END = 761` (was 758 after Phase 1A's pin slot 757; if Phase 1A was reverted, was 757 — verify and adjust).

- [ ] **Step 2: Build sanity**

```bash
SQLX_OFFLINE=true cargo check -p ml-alpha
```
Expected: exit 0.

- [ ] **Step 3: Commit**

```bash
git add crates/ml-alpha/src/rl/isv_slots.rs
# NOT YET committed — bundled into A.4
```

### Task A.2 — Write `gae_backward_sweep.cu` kernel

- [ ] **Step 1: Create kernel file**

Path: `crates/ml-alpha/cuda/gae_backward_sweep.cu`

```c
// gae_backward_sweep.cu — Generalized Advantage Estimation backward sweep.
//
// Spec: docs/superpowers/specs/2026-06-02-trainer-rollout-buffer-gae.md §1.3
//
// Per batch b, sequentially compute (backward):
//   A_T   = 0
//   for t in (T-1)..0:
//     non_terminal = 1.0f - dones[b][t]
//     v_tp1 = (t == T-1) ? v_T_bootstrap[b] : v_t[b][t+1]
//     δ_t = r[b][t] + γ * v_tp1 * non_terminal - v_t[b][t]
//     A_t = δ_t + γ * λ * A_next * non_terminal
//     A_next = A_t
//   returns[b][t] = A_t + v_t[b][t]
//
// Single-thread-per-batch kernel. Sequential backward sweep guarantees
// determinism (no parallel reductions). Memory access is strided over T
// for each thread — coalescing suboptimal but functional. If profiling
// shows bottleneck, can be tiled with warp-cooperative reduction in a
// follow-up.
//
// Per feedback_no_atomicadd: no atomic ops.
// Per feedback_cpu_is_read_only: pure device-side.
// Per feedback_no_nvrtc: pre-compiled cubin.

#include <stdint.h>

extern "C" __global__ void gae_backward_sweep(
    const float* __restrict__ rewards,      // [B × T]
    const uint8_t* __restrict__ dones,      // [B × T]
    const float* __restrict__ v_t,          // [B × T]
    const float* __restrict__ v_T_bootstrap,// [B]   — V(s_T) for trajectory end
    float* __restrict__ advantages,         // [B × T] (output)
    float* __restrict__ returns,            // [B × T] (output)
    const int B,
    const int T,
    const float gamma,
    const float lambda
) {
    const int b = blockIdx.x * blockDim.x + threadIdx.x;
    if (b >= B) return;

    float a_next = 0.0f;
    for (int t = T - 1; t >= 0; t--) {
        const int idx = b * T + t;
        const float r_t = rewards[idx];
        const float v_curr = v_t[idx];
        const float non_terminal = 1.0f - (float)dones[idx];
        const float v_tp1 = (t == T - 1) ? v_T_bootstrap[b] : v_t[idx + 1];
        const float delta = r_t + gamma * v_tp1 * non_terminal - v_curr;
        const float a_t = delta + gamma * lambda * a_next * non_terminal;
        advantages[idx] = a_t;
        returns[idx] = a_t + v_curr;
        a_next = a_t * non_terminal;  // reset GAE on done
    }
}
```

- [ ] **Step 2: Register kernel in `build.rs`**

Find the existing `compile_cuda_kernel` chain in `crates/ml-alpha/build.rs` and add:

```rust
compile_cuda_kernel("gae_backward_sweep")?;
```

Adjacent to existing kernel registrations (e.g., near `compute_advantage_return`).

- [ ] **Step 3: Build sanity**

```bash
SQLX_OFFLINE=true cargo build -p ml-alpha
```
Expected: exit 0, `target/.../gae_backward_sweep.cubin` exists.

### Task A.3 — Create `RolloutBuffer` struct

- [ ] **Step 1: Create file**

Path: `crates/ml-alpha/src/trainer/rollout_buffer.rs`

```rust
//! Rollout buffer for on-policy PPO with GAE.
//!
//! Spec: docs/superpowers/specs/2026-06-02-trainer-rollout-buffer-gae.md §1.1
//!
//! Stores `[B × T_rollout]` trajectories of (rewards, dones, v_t, v_tp1,
//! actions, log_pi_old, h_t encoder snapshot) collected by the
//! rollout-collection loop. The trainer then invokes
//! `gae_backward_sweep` to compute advantages + returns, then runs
//! multi-epoch minibatch updates.
//!
//! Per `feedback_no_htod_htoh_only_mapped_pinned`: all CPU↔GPU paths
//! use `read_slice_d_into<T>` (mapped-pinned). All device-side ops are
//! stream-ordered.

use anyhow::{Context, Result};
use cudarc::driver::{CudaSlice, CudaStream, CudaFunction, CudaModule};
use std::sync::Arc;

pub struct RolloutBuffer {
    pub b_size: usize,
    pub t_rollout: usize,
    pub hidden_dim: usize,

    // Per-step trajectory buffers ([B × T])
    pub rewards_d: CudaSlice<f32>,
    pub dones_d: CudaSlice<u8>,
    pub v_t_d: CudaSlice<f32>,
    pub actions_d: CudaSlice<i32>,
    pub log_pi_old_d: CudaSlice<f32>,
    pub h_t_d: CudaSlice<f32>,  // [B × T × HIDDEN_DIM]

    // Trajectory-end bootstrap V(s_T) ([B])
    pub v_T_bootstrap_d: CudaSlice<f32>,

    // Output buffers (filled by GAE) ([B × T])
    pub advantages_d: CudaSlice<f32>,
    pub returns_d: CudaSlice<f32>,

    // GAE kernel
    pub gae_kernel: CudaFunction,
    _gae_module: Arc<CudaModule>,

    // Current write position (set by collection loop)
    pub current_t: usize,
}

impl RolloutBuffer {
    pub fn new(
        ctx: &Arc<cudarc::driver::CudaContext>,
        stream: &Arc<CudaStream>,
        b_size: usize,
        t_rollout: usize,
        hidden_dim: usize,
    ) -> Result<Self> {
        let bt = b_size * t_rollout;
        let bth = bt * hidden_dim;

        let rewards_d = stream.alloc_zeros::<f32>(bt).context("rewards_d")?;
        let dones_d = stream.alloc_zeros::<u8>(bt).context("dones_d")?;
        let v_t_d = stream.alloc_zeros::<f32>(bt).context("v_t_d")?;
        let actions_d = stream.alloc_zeros::<i32>(bt).context("actions_d")?;
        let log_pi_old_d = stream.alloc_zeros::<f32>(bt).context("log_pi_old_d")?;
        let h_t_d = stream.alloc_zeros::<f32>(bth).context("h_t_d")?;
        let v_T_bootstrap_d = stream.alloc_zeros::<f32>(b_size).context("v_T_bootstrap_d")?;
        let advantages_d = stream.alloc_zeros::<f32>(bt).context("advantages_d")?;
        let returns_d = stream.alloc_zeros::<f32>(bt).context("returns_d")?;

        let gae_module = ctx
            .load_module(include_bytes!(concat!(env!("OUT_DIR"), "/gae_backward_sweep.cubin")))
            .context("load gae_backward_sweep cubin")?;
        let gae_kernel = gae_module.load_function("gae_backward_sweep")
            .context("load gae_backward_sweep function")?;

        Ok(Self {
            b_size, t_rollout, hidden_dim,
            rewards_d, dones_d, v_t_d, actions_d, log_pi_old_d, h_t_d,
            v_T_bootstrap_d, advantages_d, returns_d,
            gae_kernel, _gae_module: gae_module,
            current_t: 0,
        })
    }

    /// Reset write position. Called at start of each rollout.
    pub fn reset(&mut self) {
        self.current_t = 0;
    }

    /// Invoke GAE backward sweep. Caller must have set v_T_bootstrap_d.
    pub fn compute_gae(
        &self,
        stream: &Arc<CudaStream>,
        gamma: f32,
        lambda: f32,
    ) -> Result<()> {
        let b = self.b_size as i32;
        let t = self.t_rollout as i32;
        let threads = 256u32;
        let blocks = ((b as u32) + threads - 1) / threads;
        use cudarc::driver::{LaunchConfig, PushKernelArg};
        let cfg = LaunchConfig {
            grid_dim: (blocks, 1, 1),
            block_dim: (threads, 1, 1),
            shared_mem_bytes: 0,
        };
        let mut launch = stream.launch_builder(&self.gae_kernel);
        unsafe {
            launch
                .arg(&self.rewards_d)
                .arg(&self.dones_d)
                .arg(&self.v_t_d)
                .arg(&self.v_T_bootstrap_d)
                .arg(&self.advantages_d)
                .arg(&self.returns_d)
                .arg(&b)
                .arg(&t)
                .arg(&gamma)
                .arg(&lambda)
                .launch(cfg)?;
        }
        Ok(())
    }
}
```

- [ ] **Step 2: Register module in `trainer/mod.rs`**

Add `pub mod rollout_buffer;` to `crates/ml-alpha/src/trainer/mod.rs`.

- [ ] **Step 3: Build sanity**

```bash
SQLX_OFFLINE=true cargo build -p ml-alpha
```
Expected: exit 0.

### Task A.4 — GPU-oracle invariant tests

- [ ] **Step 1: Create test file**

Path: `crates/ml-alpha/tests/rollout_buffer_invariants.rs`

```rust
//! GPU-oracle invariant tests for RolloutBuffer + gae_backward_sweep kernel.
//!
//! Tests run on actual GPU per `feedback_no_cpu_test_fallbacks`. Each test
//! constructs a tiny synthetic trajectory, computes expected GAE values
//! by hand (in pure Rust with f64 accumulation for reference precision),
//! invokes the kernel, and asserts bit-equality within 1e-5 relative
//! tolerance.

use anyhow::Result;
use cudarc::driver::{CudaContext, DevicePtrMut};
use ml_alpha::trainer::rollout_buffer::RolloutBuffer;

const TOLERANCE: f32 = 1e-5;

#[test]
#[ignore = "requires CUDA"]
fn gae_terminal_only_matches_close_event_pnl() -> Result<()> {
    // Setup: B=2, T=4, all dones=0 except dones[b][T-1]=1
    // rewards = [[0, 0, 0, 10], [0, 0, 0, -5]]
    // v_t   = [[1, 1, 1, 0], [-1, -1, -1, 0]]
    // v_T_bootstrap = [0, 0]  (terminal, ignored since done=1)
    // γ=0.99, λ=0.95
    //
    // Expected: t=3 (done=1):
    //   non_terminal = 0
    //   v_tp1 = bootstrap (but non_terminal=0 zeros it)
    //   δ_3 = r_3 + γ·v_tp1·0 - v_3 = r_3 - v_3 = 10 - 0 = 10
    //   A_3 = δ_3 + γλ·0·0 = 10
    //   a_next = A_3 · 0 = 0 (reset on done)
    // t=2 (done=0):
    //   v_tp1 = v_3 = 0
    //   δ_2 = 0 + 0.99·0 - 1 = -1
    //   A_2 = -1 + 0.99·0.95·0 = -1
    // t=1: same structure, A_1 = -1
    // t=0: same, A_0 = -1
    //
    // Returns_t = A_t + V_t
    let ctx = CudaContext::new(0)?;
    let stream = ctx.default_stream();
    let b = 2usize; let t = 4usize; let hidden_dim = 4usize;
    let mut rb = RolloutBuffer::new(&ctx, &stream, b, t, hidden_dim)?;
    // ... populate rewards_d, dones_d, v_t_d via mapped-pinned write helpers
    // ... invoke compute_gae(γ=0.99, λ=0.95)
    // ... read back advantages_d, returns_d
    // ... assert close to expected
    Ok(())
}

#[test]
#[ignore = "requires CUDA"]
fn gae_dense_reward_geometric_decay() -> Result<()> {
    // Setup: B=1, T=10, all dones=0, all r=1, all v_t=0
    // Expected: A_t = Σ (γλ)^k from k=0 to T-t-1
    // For T=10, γ=0.99, λ=0.95: λγ = 0.9405
    //   A_9 = 1; A_8 = 1 + 0.9405 = 1.9405; A_7 = 1 + 0.9405·1.9405 = ...
    // Asserts geometric series matches kernel output to 1e-5.
    Ok(())
}

#[test]
#[ignore = "requires CUDA"]
fn gae_done_resets_credit() -> Result<()> {
    // Setup: B=1, T=6, dones=[0,0,1,0,0,0], all r=1, all v_t=0
    // Expected: A_2 (done=1) should NOT pass credit to A_1, A_0.
    // A_3..5 form one trajectory: A_5=1, A_4=1+λγ, A_3=...
    // A_2 = 1 (terminal), A_2·non_terminal=0 → A_1 starts fresh
    // A_1 = 1+λγ·A_2_non_term = 1+0; A_0 = 1+λγ·A_1·non_terminal=1
    // Verifies the done-reset in the kernel.
    Ok(())
}

#[test]
#[ignore = "requires CUDA"]
fn gae_deterministic_across_runs() -> Result<()> {
    // Run the same input through the kernel TWICE with fresh context.
    // Verify bit-equality on advantages_d + returns_d (per
    // pearl_determinism_achieved contract — new kernel must not break it).
    Ok(())
}

#[test]
#[ignore = "requires CUDA"]
fn rollout_buffer_alloc_sizes_match_spec() -> Result<()> {
    let ctx = CudaContext::new(0)?;
    let stream = ctx.default_stream();
    let b = 128; let t = 256; let h = 256;
    let rb = RolloutBuffer::new(&ctx, &stream, b, t, h)?;
    assert_eq!(rb.rewards_d.len(), b*t);
    assert_eq!(rb.dones_d.len(), b*t);
    assert_eq!(rb.h_t_d.len(), b*t*h);
    assert_eq!(rb.v_T_bootstrap_d.len(), b);
    Ok(())
}
```

- [ ] **Step 2: Implement the test helper bodies**

(Translate the math-comments into actual mapped-pinned writes + assertions. Use `read_slice_d_into<T>` for readback.)

- [ ] **Step 3: Run tests**

```bash
SQLX_OFFLINE=true cargo test -p ml-alpha --test rollout_buffer_invariants --release -- --ignored --nocapture
```
Expected: 5/5 PASS.

### Task A.5 — Atomic commit + verify determinism not broken

- [ ] **Step 1: Stage Phase 1B-A changes**

```bash
git add \
  crates/ml-alpha/src/rl/isv_slots.rs \
  crates/ml-alpha/src/trainer/rollout_buffer.rs \
  crates/ml-alpha/src/trainer/mod.rs \
  crates/ml-alpha/cuda/gae_backward_sweep.cu \
  crates/ml-alpha/build.rs \
  crates/ml-alpha/tests/rollout_buffer_invariants.rs
```

- [ ] **Step 2: Verify determinism unchanged**

```bash
./scripts/determinism-check.sh --quick
```
Expected: exit 0. The new struct/kernel are not wired into the trainer yet, so existing pipeline must be bit-equal.

- [ ] **Step 3: Pre-commit hook simulation**

```bash
git diff --cached -U0 | grep -E '^\+[^+]' | grep -E '\.memcpy_(htod|dtoh)\(' | grep -v 'gpu-ok:' | wc -l
# expect 0
git diff --cached -U0 | grep -E '^\+[^+]' | grep -E '\batomicAdd\(' | wc -l
# expect 0
```

- [ ] **Step 4: Commit**

```bash
git commit -m "$(cat <<'EOF'
feat(ml-alpha): Phase 1B-A — RolloutBuffer + GAE kernel foundation

Spec: docs/superpowers/specs/2026-06-02-trainer-rollout-buffer-gae.md

Adds the foundation for Phase 1B (trainer rollout-buffer + GAE refactor).
No trainer integration yet — this commit only establishes the components
and validates them with GPU-oracle tests. Subsequent phases (1B-B through
1B-E) wire the components into the actual training loop.

New components:
- 3 ISV slots (758-760): RL_PPO_ROLLOUT_HORIZON_INDEX (T=256 default),
  RL_PPO_N_EPOCHS_INDEX (K_ppo=4), RL_PPO_N_MINIBATCHES_INDEX (8).
  RL_SLOTS_END 758 -> 761.
- crates/ml-alpha/cuda/gae_backward_sweep.cu: single-thread-per-batch
  sequential backward sweep computing A_t = δ_t + γλ·A_{t+1} (1-non_terminal),
  returns_t = A_t + V_t. Deterministic by construction (no parallel
  reductions), no atomicAdd, no nvrtc.
- crates/ml-alpha/src/trainer/rollout_buffer.rs: RolloutBuffer struct with
  [B × T] device buffers (rewards, dones, v_t, actions, log_pi_old, h_t),
  v_T_bootstrap, and output advantages/returns. compute_gae() invokes
  the kernel.
- crates/ml-alpha/tests/rollout_buffer_invariants.rs: 5 GPU-oracle
  invariant tests covering terminal-only credit, geometric decay,
  done-reset, determinism, and allocation sizes.

Validation:
- cargo build -p ml-alpha: exit 0
- cargo test rollout_buffer_invariants --release: 5/5 PASS
- ./scripts/determinism-check.sh --quick: exit 0 (pipeline unchanged)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
EOF
)"
```

---

## Phase 1B-B — Rollout collection loop behind env flag

**Goal:** Add a parallel rollout-collection code path that runs T env steps without training, populating the RolloutBuffer. Gated behind `FOXHUNT_USE_ROLLOUT=1` env var so default behavior is unchanged. Allows side-by-side validation.

**Note on `FOXHUNT_USE_ROLLOUT` env flag**: per `feedback_no_feature_flags`, permanent feature flags are forbidden. This env var is a TEMPORARY VALIDATION FLAG, removed in Phase 1B-E. The pattern is the same as `FOXHUNT_DETERMINISTIC` (Phase 2.3 of determinism foundation) which used an env toggle for dev vs prod validation.

**Files:**
- Create: `crates/ml-alpha/src/trainer/rollout_collection.rs`
- Modify: `crates/ml-alpha/src/trainer/integrated.rs` (gated branch into rollout collection)
- Modify: `crates/ml-alpha/examples/alpha_rl_train.rs` (route through rollout when flag set)

### Tasks B.1-B.4 (outline — expand when 1B-A passes)

1. B.1 — Implement `RolloutCollection::collect()` that runs T env steps, snapshots V_t/log_pi_old/h_t after each step, populates RolloutBuffer
2. B.2 — Bootstrap V(s_T) at end of collection
3. B.3 — Invoke `RolloutBuffer::compute_gae()` after collection completes
4. B.4 — Gate behind `FOXHUNT_USE_ROLLOUT=1`; if not set, fall through to existing per-step path

**Falsification at 1B-B**:
- `FOXHUNT_USE_ROLLOUT=0` (default): mid-smoke identical to HEAD (bit-equal output for same seed)
- `FOXHUNT_USE_ROLLOUT=1`: smoke runs without error; rollout buffer populated correctly; GAE outputs sensible distributions
- Determinism preserved in both modes

---

## Phase 1B-C — Multi-epoch PPO update over rollout

**Goal:** With `FOXHUNT_USE_ROLLOUT=1`, run K_ppo=4 epochs of minibatch PPO updates over the rollout buffer instead of the existing per-step K-loop. Distributional Q + aux heads continue from PER.

**Files:**
- Create: `crates/ml-alpha/src/trainer/ppo_minibatch.rs`
- Modify: `crates/ml-alpha/src/trainer/integrated.rs` (call into multi-epoch PPO)

### Tasks C.1-C.5 (outline — expand when 1B-B passes)

1. C.1 — Implement minibatch shuffling with deterministic seeded RNG
2. C.2 — Implement single-minibatch forward (re-evaluate V + π on h_t from rollout, compute PPO surrogate + V loss)
3. C.3 — Backprop through head losses (encoder frozen during multi-epoch — standard PPO impl)
4. C.4 — Adam step per minibatch
5. C.5 — Loop K_ppo epochs

**Falsification at 1B-C** (mid-smoke b=128, 2000+500, seed=42, with `FOXHUNT_USE_ROLLOUT=1`):
- **G1**: `Pearson(rewards.sum, Δpnl_USD)` train last 500 ≥ 0.7 (relaxed from math agent's 0.85 due to existing clamp+popart nonlinearity)
- **G2**: `Pearson(v_pred_t, returns_t)` train last 500 ≥ 0.5 (V regression quality — new measurement enabled by GAE)
- **G3**: wr_train ≥ 0.40 (sample efficiency gain)
- **G4**: Determinism preserved
- **G5**: Cross-source consistency $50
- **G6 (smoke-level)**: eval_summary.total_pnl_usd less negative than Phase 0 baseline −$4.46M

If G1-G5 pass but G6 marginal: proceed to 1B-D cluster verification (smoke isn't the final word on eval pnl direction at limited budget).
If G6 strongly negative (eval pnl below baseline by > 20%): STOP, this is a Phase 1A-style regression. Diagnose before continuing.

---

## Phase 1B-D — Cluster verification 20k+5k

**Goal:** Validate Phase 1B-C at cluster scale. This is the LOAD-BEARING test — local mid-smoke can't capture regime-shift effects across the 24-month dataset.

**Cluster smoke**: `./scripts/argo-alpha-rl.sh --branch ml-alpha-regime-observer --instrument-mode front-month --n-folds 3 --fold-idx 1` with the current `FOXHUNT_USE_ROLLOUT=1` binary.

### Tasks D.1-D.3 (outline — expand when 1B-C passes)

1. D.1 — Submit cluster job at the 1B-C commit SHA
2. D.2 — Monitor for completion (~70 min on L40S)
3. D.3 — Analyze eval verdict against G6 cluster gate

**Falsification at 1B-D (cluster)**:
- `eval_summary.total_pnl_usd` strictly better than Phase 0 baseline cluster value (TBD: re-run baseline at HEAD `cfaa420bd` cluster to get number)
- Determinism preserved (same-seed cluster run is bit-equal)
- popart.sigma CV ≤ 0.96 (no regression from Phase 0)
- max_drawdown_usd not worse than Phase 0 by > 20%

If G6 passes at cluster: 1B-E (cleanup) is the next step.
If G6 fails at cluster: the GAE refactor + pure-pnl combination ALONE is not sufficient for eval-collapse. Pivot to architectural fixes (multi-head policy or regime-invariance via DANN) per the strategic plan's Phase 2/3.

---

## Phase 1B-E — Default flag=1, remove legacy path, cleanup

**Goal:** Once 1B-D validates, make rollout-buffer + GAE the default. Remove `FOXHUNT_USE_ROLLOUT` env flag, delete the legacy `compute_advantage_return.cu` 1-step TD path.

### Tasks E.1-E.5 (outline — expand when 1B-D passes)

1. E.1 — Default `FOXHUNT_USE_ROLLOUT` semantics inverted (flag=0 → opt-in to legacy, but warn deprecated)
2. E.2 — Remove the gated branch in `integrated.rs`; rollout-buffer is the only path
3. E.3 — Delete `compute_advantage_return.cu` (or keep as docs-only reference noting it was replaced)
4. E.4 — Remove `FOXHUNT_USE_ROLLOUT` env var; rollout-buffer is unconditional
5. E.5 — Update `pearl_advantage_kernel_is_done_gated_td_not_gae` to mark RESOLVED; update `pearl_foxhunt_trainer_is_genuinely_single_step` to note the refactor is done

**Falsification at 1B-E**:
- Mid-smoke + cluster sanity: same output as 1B-D (no behavior change)
- All existing tests pass
- Pre-commit hook passes (no atomicAdd, no raw memcpy_dtoh)

---

## Self-review (controller, before dispatch)

### Spec coverage
✅ Spec §0 (why) — addressed by motivation + Phase 1A regression citation
✅ Spec §1 (architecture decisions) — RolloutBuffer struct, GAE kernel, K_ppo epochs, PER reconciliation, K-loop semantics, CUDA Graph, determinism — all in plan
✅ Spec §2 (diagnostics) — deferred to 1B-C (when rollout integration produces the new fields)
✅ Spec §3 (falsification criteria) — G1-G6 in 1B-C and 1B-D
✅ Spec §4 (file design) — all files mapped to phases
✅ Spec §5 (sequencing) — 5-phase delivery preserved
✅ Spec §6 (risks) — addressed in falsification gates
✅ Spec §7 (decision tree from 1A) — N/A here (1A was the input; this plan IS the response)

### Placeholders
- Task bodies for B.1-B.4, C.1-C.5, D.1-D.3, E.1-E.5 are OUTLINES, not full code. Expand each when its prerequisite phase passes (this prevents premature design lock-in and lets each subsequent phase incorporate learnings from the previous).
- Phase 0 baseline cluster value referenced in 1B-D is TBD — need to run the baseline at cluster scale, separate dispatch.

### Type consistency
- `RolloutBuffer` fields use consistent `_d` suffix for device slices
- `T_rollout`, `K_ppo`, `n_minibatches` named consistently in spec, plan, and code samples

## Execution handoff

**Plan complete and saved to `docs/superpowers/plans/2026-06-02-trainer-rollout-buffer-gae-implementation.md`.**

Per the methodology in `feedback_investigation_first_falsification_methodology`:

1. Dispatch Phase 1B-A coder NOW (foundation: ISV slots + RolloutBuffer + GAE kernel + tests; no trainer integration).
2. After 1B-A commits + passes determinism check: expand 1B-B task bodies, dispatch 1B-B coder.
3. After 1B-B mid-smoke validates: expand 1B-C task bodies, dispatch 1B-C coder.
4. After 1B-C smoke passes G1-G5: dispatch 1B-D cluster verification.
5. After 1B-D cluster validates G6: dispatch 1B-E cleanup.

Each phase is one commit. Each phase's verdict feeds into the next phase's scope.

This sequencing follows `feedback_investigation_first_falsification_methodology` — predetermined falsification gates at every phase, STOP-on-unexpected-finding required of every coder agent, no rationalization-by-shipping permitted.
