feat(ml-alpha): Phase 1B-A — RolloutBuffer + GAE kernel foundation
Spec: docs/superpowers/specs/2026-06-02-trainer-rollout-buffer-gae.md
Plan: docs/superpowers/plans/2026-06-02-trainer-rollout-buffer-gae-implementation.md
Foundation phase of the trainer rollout-buffer + GAE refactor (Phase 1B).
Empirically required after Phase 1A regression confirmed math-agent's
atomicity claim: dropping Phase 5 shaping WITHOUT GAE makes things worse
(eval_pnl regressed -$691k from -$4.46M to -$5.15M, popart sigma CV
0.96 -> 1.58, sign agreement 60% -> 46%). See pearl_reward_signal_
anti_aligned_with_pnl ADDENDUM 2026-06-02d for the mechanism analysis.
This commit establishes the components WITHOUT trainer integration —
subsequent phases (1B-B through 1B-E) wire them into the actual training
loop. Subdivides the multi-week refactor into atomically-verifiable
phases per feedback_investigation_first_falsification_methodology.
New components:
* 3 ISV slots (758-760):
RL_PPO_ROLLOUT_HORIZON_INDEX = 758 (T_rollout, bootstrap 256)
RL_PPO_N_EPOCHS_INDEX = 759 (K_ppo, bootstrap 4)
RL_PPO_N_MINIBATCHES_INDEX = 760 (minibatches/epoch, bootstrap 8)
RL_SLOTS_END 757 -> 761
* crates/ml-alpha/cuda/gae_backward_sweep.cu (58 LOC):
Single-thread-per-batch sequential backward sweep computing
A_t = δ_t + γλ·A_{t+1}·(1-done), returns_t = A_t + V_t.
Deterministic by construction (no parallel reductions, no atomicAdd,
no nvrtc). Reset on done. v_T_bootstrap parameter for trajectory-end
V estimate.
* crates/ml-alpha/src/trainer/rollout_buffer.rs (210 LOC):
RolloutBuffer struct with [B × T] device buffers for rewards, dones,
v_t, actions, log_pi_old, h_t; plus [B] v_T_bootstrap; plus output
advantages, returns. compute_gae(γ, λ) invokes the kernel using
cudarc raw-pointer pattern (`device_ptr(stream).0` resolved into
local before .arg() — idiomatic for codebase). Loaded module +
kernel handle stay private; struct fields exposed per spec.
* crates/ml-alpha/tests/rollout_buffer_invariants.rs (466 LOC):
5 GPU-oracle invariant tests (#[ignore = "requires CUDA"]):
1. gae_terminal_only_matches_close_event_pnl — single done at T-1
2. gae_dense_reward_geometric_decay — Σ(γλ)^k geometric series
3. gae_done_resets_credit — done reset propagation
4. gae_deterministic_across_runs — bit-equality across contexts
5. rollout_buffer_alloc_sizes_match_spec — alloc verification
All 5 PASS in 2.15s.
Validation gates (Phase 1B-A complete when all pass):
* cargo build --release --example alpha_rl_train -p ml-alpha: exit 0
(1m 00s release build, gae_backward_sweep.cubin compiled with -O3
--use_fast_math --ftz=true --fmad=true for sm_86)
* cargo test -p ml-alpha --test rollout_buffer_invariants --release: 5/5 PASS
* ./scripts/determinism-check.sh --quick: exit 0 — DETERMINISTIC: all
checksums.* leaves match across all 200 rows (rel-tol=1e-5, abs-tol=1e-7).
Pipeline output bit-equal to baseline since new struct/kernel are
loaded but unused in the training loop.
* Pre-commit hook on staged diff: PASS (0 memcpy_dtoh, 0 atomicAdd).
Next: Phase 1B-B (rollout collection loop behind FOXHUNT_USE_ROLLOUT env
flag) per plan §Phase 1B-B. Dispatch after this commit lands.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
5cd2f87039