docs(plans): integrated RL trainer GPU-pure rebuild

Replaces the flawed Phase F + G arc preserved on branch
`ml-alpha-phase-f-g-flawed` (commits 99a125cdb..b3808a5ac). The prior
attempt shipped a trainer with multiple production-blocking defects:

1. ISV[400..406] uninitialised → kernels read γ=0, ε=0, entropy=0
2. rl_reward_scale_controller drifted to 1e3 on no-trade steps
3. 6 controllers exist as .cu but never launched
4. Target net never soft-updated (τ has no consumer)
5. step_with_lobsim violated feedback_cpu_is_read_only with host
   Thompson sampling + EMA tracking + advantage/return loops
6. "toy" framing leaked into production (alpha_rl_train.rs shipped
   with next_snapshots=snapshots — the F.4 next-state code path was
   a no-op until that one issue got caught mid-review)
7. No NaN abort in production CLI

The convergence-gate fixtures (dqn_toy/ppo_toy → renamed
dqn_reward_signal/ppo_reward_signal on the flawed branch) hid every
defect because MockLobEnv is state-invariant with horizon=1.

The rebuild is GPU-pure: kernel-driven action sampling, kernel-driven
EMA tracking, kernel-driven advantage/return, ISV bootstrap at trainer
construction, all 7 controllers wired with device-resident EMA inputs,
target-net soft update consumer, NaN abort, no LobEnv trait (drives
LobSimCuda via the existing decision-policy kernel pattern).

Sequenced as R1..R9 with falsifiability gates G1..G7 that exercise
the specific failure modes the convergence-gate fixtures couldn't
catch. Calendar ~8.5 dev days.

Also adds memory pearl
feedback_extending_existing_code_audits_for_existing_violations
capturing the lesson: extending pre-existing CPU/orphan-controller
violations is how this happened.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md

@@ -0,0 +1,381 @@
+# Integrated RL Trainer — GPU-Pure Rebuild
+
+> **Status:** Rebuild plan, replaces the flawed Phase F + G arc preserved
+> on branch `ml-alpha-phase-f-g-flawed` (commits 99a125cdb..b3808a5ac).
+> Baseline: `ml-alpha-phase-a` reset to commit `9114374d2`
+> (Phase E.3b activation).
+
+## Why this rebuild
+
+The prior Phase F + G attempt landed 9 commits that shipped a trainer
+with multiple production-blocking defects. The defects were not bugs in
+the new code so much as **propagation of pre-existing violations** in
+`step_with_lobsim` plus **incomplete wiring of the ISV adaptive-control
+infrastructure**. The convergence-gate fixtures (`dqn_reward_signal`,
+`ppo_reward_signal`, then named `dqn_toy` / `ppo_toy`) hid every defect
+because their `MockLobEnv` has `done = true` every step, reward in
+`[-1, +1]`, and horizon = 1. The cluster-bound smoke was the only
+mechanism that would have exposed the problems — at $5–20 / 30–120 min
+each.
+
+### Defects in the flawed branch
+
+1. **ISV[400..406] uninitialised in production paths.** The trainer
+   `alloc_zeros` the ISV array; only `rl_lr_controller` was launched.
+   Production consumers read sentinel `0`:
+   - `bellman_target_projection.cu:133` reads γ = 0 → Bellman target
+     reduces to `r + 0·V(s_{t+1}) = r`. Q-head learns immediate-reward
+     only, zero credit assignment.
+   - `ppo_clipped_surrogate.cu:205,305` reads ε = 0 → `clip(r, 1, 1) = 1`
+     always. PPO trust region collapses.
+   - `ppo_clipped_surrogate.cu:210` reads entropy coef = 0 → no entropy
+     bonus, policy collapses to deterministic immediately.
+
+2. **`rl_reward_scale_controller` drifts to `1e3` on no-trade steps.**
+   The launcher fired every step regardless of `mean_abs_pnl_ema`
+   readiness; with `ema = 0` and `prev = REWARD_SCALE_BOOTSTRAP = 1.0`,
+   the kernel computed `target = 1/max(0, 1e-3) = 1000`, Wiener-blended
+   toward it, and ramped the scale to `1e3` over the first ~10 hold
+   steps. Toy fixture hid this because `done = true` every step.
+
+3. **6 controllers exist as .cu files but never launched.**
+   `rl_gamma`, `rl_target_tau`, `rl_ppo_clip`, `rl_entropy_coef`,
+   `rl_rollout_steps`, `rl_per_alpha`. The build system compiles their
+   cubins; no Rust launcher fires them. ISV bootstrap (defect #1)
+   would have masked this even if the controllers had been wired.
+
+4. **Target-net soft update missing.** `DqnHead` owns `target_w_d /
+   target_b_d` but no kernel updates them. The "target network" is
+   actually the initial random init for the entire run — no soft Polyak
+   update, no hard sync. Double-DQN Bellman target uses an
+   untrained-static target.
+
+5. **`feedback_cpu_is_read_only` violation in `step_with_lobsim`.**
+   Host code:
+   - 5 DtoH copies per step (Q / V / π logits at h_t, V / Q at h_{t+1})
+   - Host-side softmax + Thompson categorical sample + argmax loops
+     over `[B][N_ACTIONS][Q_N_ATOMS]`
+   - Host EMA updates for `mean_abs_pnl_ema`
+   - Host advantage / return computation
+   - HtoD copies for `actions`, `rewards`, `dones`
+   The pre-existing Phase E.3b code already had this pattern; the
+   flawed Phase F extended it instead of rebuilding it as GPU-resident
+   per `pearl_cold_path_no_exception_to_gpu_drives`.
+
+6. **"Toy" framing in production code.** Test files `dqn_toy.rs` /
+   `ppo_toy.rs`, fixture constructor `MockLobEnv::toy_bandit()`, and
+   pervasive "toy bandit" / "for the toy bandit smoke" comments leaked
+   into production source. Worse: `alpha_rl_train.rs` shipped with
+   `next_snapshots = snapshots` — production code applying a
+   toy-fixture simplification — until that one issue got caught
+   mid-review.
+
+7. **No NaN / divergence abort in `alpha_rl_train.rs`.** A NaN-poisoned
+   run would burn the full 14400-second `activeDeadlineSeconds` budget
+   silently.
+
+### Pattern lesson
+
+The convergence-gate fixtures verified the gradient mechanics
+(`reward_signal_drives_argmax_to_rewarded_action`) but their
+state-invariant fixture intrinsically *can't* test:
+- Sparse-reward credit assignment (defect #1's γ = 0 effect)
+- Reward-magnitude drift (defect #2)
+- Trust-region clipping (defect #1's ε = 0 effect)
+- Target-network staleness (defect #4)
+
+**Pre-cluster falsifiability requires fixtures that exercise the same
+failure modes the cluster will see.** A short bull-market loop with
+state-dependent rewards + per-step holds + multi-step trades is the
+minimum.
+
+## Architectural decisions
+
+### A1. GPU-pure action submission via the existing decision-policy path
+
+The pre-existing production decision pipeline (`step_decision_with_latency`
+in `crates/ml-backtesting/src/sim/mod.rs`) already submits actions
+GPU-pure: `alpha_probs_d` (device) feeds `decision_policy_kernel`
+which writes `market_targets_d` (device) which the fill kernel reads.
+No host roundtrip.
+
+**Decision:** the integrated RL trainer drives the same kernel path.
+Action sampling happens IN THE DECISION KERNEL (or a new
+`rl_action_kernel` that lives parallel to it), reading Q / π logits +
+ISV[entropy, ε] from device + writing to `market_targets_d`. The
+`LobEnv` trait + `LobSimEnvAdapter` from the flawed branch are
+DELETED. The trainer interacts with `LobSimCuda` via the existing
+decision-policy entry point.
+
+**Consequence:** the trait-abstraction layer that let `MockLobEnv` mock
+the env is removed. Convergence-gate tests must run against a real
+`LobSimCuda` with synthetic book data (still feasible — fixtures
+inject snapshots).
+
+### A2. All EMAs live in ISV
+
+Currently `mean_abs_pnl_ema` lives as a host `f32` field on
+`IntegratedTrainer`. **Decision:** every EMA the controllers consume
+lives in an ISV slot, updated by a dedicated per-step kernel that reads
+the source signal from device buffers (rewards_d, dones_d, etc.).
+
+New ISV slots needed for the EMA inputs:
+- `RL_MEAN_TRADE_DURATION_EMA_INDEX` (input to `rl_gamma_controller`)
+- `RL_Q_DIVERGENCE_EMA_INDEX` (input to `rl_target_tau_controller`)
+- `RL_KL_PI_EMA_INDEX` (input to `rl_ppo_clip_controller`)
+- `RL_ENTROPY_OBSERVED_EMA_INDEX` (input to `rl_entropy_coef_controller`)
+- `RL_ADVANTAGE_VAR_RATIO_EMA_INDEX` (input to `rl_rollout_steps_controller`)
+- `RL_TD_KURTOSIS_EMA_INDEX` (input to `rl_per_alpha_controller`)
+- `RL_MEAN_ABS_PNL_EMA_INDEX` (input to `rl_reward_scale_controller`)
+
+That's 7 new slots → `RL_SLOTS_END = 424`.
+
+### A3. ISV bootstrap at trainer construction
+
+Defect #1 and the Wiener-blend behavior of the controllers both
+require slots to be at known-good values before any kernel reads them.
+**Decision:** trainer `new()` does ONE memcpy_htod writing canonical
+defaults to ISV[400..406]:
+
+| Slot | Default | Source |
+|------|---------|--------|
+| RL_GAMMA_INDEX (400) | 0.99 | DQN convention |
+| RL_TARGET_TAU_INDEX (401) | 0.005 | DQN convention |
+| RL_PPO_CLIP_INDEX (402) | 0.2 | PPO original paper |
+| RL_ENTROPY_COEF_INDEX (403) | 0.01 | PPO convention |
+| RL_N_ROLLOUT_STEPS_INDEX (404) | 2048 | PPO convention |
+| RL_PER_ALPHA_INDEX (405) | 0.6 | Schaul et al. 2016 |
+| RL_REWARD_SCALE_INDEX (406) | 1.0 | identity pass-through |
+
+Controllers then ADAPT from these defaults rather than bootstrapping
+from sentinel zero. The sentinel-zero bootstrap path in each kernel
+stays as a safety net but is never the production cold-start path.
+
+### A4. Target-net soft update consumer for ISV[401]
+
+New `dqn_target_soft_update` kernel: per-step element-wise
+`target_w[i] = (1-τ)·target_w[i] + τ·w[i]` reading τ from `ISV[401]`.
+Launched from `step_after_encoder_forward` after the Q-head Adam
+update.
+
+### A5. GPU-resident action sampling
+
+New kernels:
+- `thompson_sample_categorical` — input: Q logits, ISV[entropy_coef],
+  device PRNG state. Output: per-batch action index in `market_targets`
+  format directly.
+- `argmax_expected_q` — input: Q logits. Output: per-batch
+  next-action index.
+- `log_pi_at_action` — input: π logits + action_d. Output: per-batch
+  log π(a_t) — feeds the PPO importance-ratio path.
+
+Device PRNG via a per-batch `curandStatePhilox4_32_10_t` allocated at
+trainer init + advanced per step.
+
+### A6. GPU-resident advantage / return / TD computation
+
+New `compute_advantage_return` kernel: per-batch `r + γ(1-done)·V_tp1`
+and `r_t - V_t`, reading γ from `ISV[400]`. Replaces the host loop in
+the flawed branch's `step_with_lobsim`.
+
+### A7. Fee model
+
+The flawed F.3a's approach (`order_match.cu` deduct fee + new
+`upload_cost_per_lot_per_side` API) was correct in principle, mirroring
+the existing `apply_fill_to_pos` pattern from `resting_orders.cu`. The
+rebuild keeps this change.
+
+### A8. Loader pair API
+
+The flawed Phase G's `MultiHorizonLoader::next_sequence_pair` was
+correct (true `(s_t, s_{t+1})` consecutive sampling). The rebuild keeps
+this addition.
+
+## Falsifiability gates — pre-cluster
+
+Each gate is a local CUDA test that exercises the failure mode it
+covers. None of these existed in the flawed branch.
+
+| Gate | Asserts | Catches defect |
+|------|---------|----------------|
+| G1 | ISV[400..406] held at canonical defaults after `IntegratedTrainer::new` | #1 |
+| G2 | `RL_REWARD_SCALE_INDEX` stays at 1.0 for first N hold-only steps | #2 |
+| G3 | Each controller's launcher visibly mutates its ISV slot after first non-sentinel input | #3 |
+| G4 | `target_w` Frobenius norm drifts from `w` only after the soft-update kernel fires | #4 |
+| G5 | `cudaMemcpy` count per `step_with_lobsim` is 0 host-initiated (other than ISV diagnostic readback) | #5 |
+| G6 | Bull-market forced-long 50-trade loop: mean reward per trade > 0 (already in flawed F.3b's `reward_calibration.rs`) | (regression-coverage for fee + scale) |
+| G7 | Run for N steps; abort + non-zero exit code if any per-head loss is non-finite | #7 |
+
+G5 is the load-bearing gate for the GPU-pure decision (A1). Tooling:
+hook `cudaMemcpyAsync` via a CUPTI counter or count Rust-side
+`memcpy_htod` / `memcpy_dtoh` calls during the step.
+
+## Required new files
+
+| File | Purpose |
+|------|---------|
+| `crates/ml-alpha/cuda/thompson_sample_categorical.cu` | A5 |
+| `crates/ml-alpha/cuda/argmax_expected_q.cu` | A5 |
+| `crates/ml-alpha/cuda/log_pi_at_action.cu` | A5 |
+| `crates/ml-alpha/cuda/compute_advantage_return.cu` | A6 |
+| `crates/ml-alpha/cuda/dqn_target_soft_update.cu` | A4 |
+| `crates/ml-alpha/cuda/ema_update_on_done.cu` | A2 — generic done-gated EMA (parameterised slot) |
+| `crates/ml-alpha/cuda/ema_update_per_step.cu` | A2 — generic per-step EMA (parameterised slot) |
+
+Reused / extended:
+- `rl_reward_scale_controller.cu` (port from flawed branch)
+- `rl_gamma_controller.cu`, `rl_target_tau_controller.cu`,
+  `rl_ppo_clip_controller.cu`, `rl_entropy_coef_controller.cu`,
+  `rl_rollout_steps_controller.cu`, `rl_per_alpha_controller.cu`
+  (already in baseline tree — port their Rust launchers from the
+  flawed branch concept, but feed device-resident EMA inputs)
+- `crates/ml-backtesting/cuda/order_match.cu` (port the F.3a fee
+  deduction)
+
+Deleted:
+- `crates/ml-alpha/src/rl/reward.rs::LobEnv` trait
+- `crates/ml-backtesting/src/sim/mod.rs::LobSimEnvAdapter`
+
+## Phased execution
+
+Each phase = one commit. Each phase has a green-build + green-gate
+requirement before the next phase starts.
+
+### R1 — ISV bootstrap + slot extension (1 commit)
+
+- Add new ISV slot constants for the 7 EMA inputs (A2) → `RL_SLOTS_END = 424`.
+- In `IntegratedTrainer::new`, one memcpy_htod writing canonical
+  defaults to ISV[400..406] (A3).
+- Gate G1.
+
+### R2 — Fee model + loader pair API (2 commits)
+
+- R2a: port `order_match.cu` fee deduction + `upload_cost_per_lot_per_side`
+  host API (from F.3a). Gate: existing `lob_env_adapter_buy_close_cycle`
+  passes with zero fees; new with-fees variant lands.
+- R2b: port `MultiHorizonLoader::next_sequence_pair` (from G fix).
+
+### R3 — GPU-pure action sampling (3 commits)
+
+- R3a: `thompson_sample_categorical` kernel + Rust launcher. Local CUDA
+  test: sample 10000 actions from a known Q distribution, assert
+  empirical frequency matches softmax probabilities within tolerance.
+- R3b: `argmax_expected_q` kernel. Local CUDA test: feed adversarial
+  Q distributions, assert argmax matches host reference.
+- R3c: `log_pi_at_action` kernel. Local CUDA test: feed known π
+  distribution, assert log-prob within 1e-5 of host softmax.
+
+### R4 — GPU-resident EMA + advantage / return (2 commits)
+
+- R4a: `ema_update_on_done.cu` + `ema_update_per_step.cu` generic
+  kernels. Local CUDA test: feed deterministic reward stream, assert
+  EMA tracks the analytic Wiener-α trajectory.
+- R4b: `compute_advantage_return.cu`. Local CUDA test: feed known
+  rewards / dones / V values, assert advantage / return match host
+  reference.
+
+### R5 — Wire all 7 controllers + target-net soft update (1 commit)
+
+- Add launcher fields + helpers for all 7 controllers (gamma,
+  target_tau, ppo_clip, entropy_coef, rollout_steps, per_alpha,
+  reward_scale). Each launcher pulls its input from the corresponding
+  EMA slot (A2 from R4a). Gate G3.
+- Add `dqn_target_soft_update` kernel + launcher (A4). Gate G4.
+
+### R6 — Replace `LobEnv` with kernel-driven decision path (3 commits)
+
+- R6a: delete `LobEnv` trait + `LobSimEnvAdapter`. Add direct
+  `LobSimCuda` integration on `IntegratedTrainer` (no trait).
+- R6b: wire action submission via the decision-policy kernel pattern
+  (mirror `step_decision_with_latency`). Adapt `market_targets_d` to
+  accept the Thompson-sampled action index directly.
+- R6c: rewrite the trainer's convergence-gate fixtures (formerly
+  `dqn_toy.rs` / `ppo_toy.rs`) to drive a real `LobSimCuda` with
+  synthetic book data instead of a `MockLobEnv`. Reward signal becomes
+  "synthetic monotonic uptick → forced long → trade close pays > 0".
+- Gate G5 (no host memcpy in step_with_lobsim other than ISV
+  diagnostics).
+
+### R7 — Reward composition (1 commit)
+
+- Per-step reward = `realized_pnl_delta` (already net of fees + slippage
+  from R2a) scaled by `ISV[RL_REWARD_SCALE_INDEX]`. Optionally subtract
+  exposure penalty (deferred unless smoke shows it's needed).
+- Gate G6.
+
+### R8 — CLI binary + Argo template + dispatcher (1 commit)
+
+- `crates/ml-alpha/examples/alpha_rl_train.rs` driving the rebuilt
+  trainer. Uses `next_sequence_pair` for true `(s_t, s_{t+1})` windows.
+- `infra/k8s/argo/alpha-rl-template.yaml` + `scripts/argo-alpha-rl.sh`
+  (port from flawed branch).
+- NaN abort: per-step `is_finite()` check on `l_total`, exit code 2 on
+  divergence. Gate G7.
+
+### R9 — Pre-cluster smoke + first cluster smoke (out-of-band)
+
+- Run all R1..R8 local CUDA tests on a GPU host. Confirm green.
+- `git push`. `kubectl apply -f` template. `./scripts/argo-alpha-rl.sh
+  --n-steps 1000` for a 5-min validation smoke. If green, full 50k.
+
+## What we keep from the flawed branch
+
+These specific elements landed correctly and are ported directly:
+- `rl_reward_scale_controller.cu` kernel (Phase F.2)
+- `order_match.cu` fee deduction pattern (Phase F.3a)
+- `MultiHorizonLoader::next_sequence_pair` (Phase G fix)
+- The 6 unlaunched controller .cu files (Phases C, D, E) — they're
+  fine kernels, just need launchers
+- F.4's true-h_{t+1} architectural insight (Bellman target on
+  h_{t+1}, V bootstrap on h_{t+1}) — the structure becomes simpler in
+  the GPU-pure rebuild because state lives on device
+
+## What we discard
+
+- `LobEnv` trait + `LobSimEnvAdapter` (replaced by kernel-driven path)
+- `MockLobEnv` "toy bandit" fixture (replaced by synthetic-book
+  `LobSimCuda` fixture)
+- Host-side action sampling, EMA tracking, advantage / return loops
+- The `dqn_toy.rs` / `ppo_toy.rs` test file names (renamed to
+  `dqn_reward_signal.rs` / `ppo_reward_signal.rs` in the rebuild)
+
+## Pearls honored
+
+- `feedback_cpu_is_read_only` — A1, A5, A6 are direct compliance.
+- `feedback_isv_for_adaptive_bounds` — A2, A3, R5.
+- `feedback_no_partial_refactor` — each R-phase is atomic.
+- `feedback_no_stubs` — every controller has a launcher AND a
+  consumer at landing time.
+- `feedback_no_atomicadd` — new kernels use single-writer-per-block
+  or per-batch scratch + reduce_axis0.
+- `feedback_no_htod_htoh_only_mapped_pinned` — applies to the
+  remaining ISV diagnostic readback.
+- `pearl_first_observation_bootstrap` — combined with A3, the host
+  bootstrap takes precedence; kernel sentinel-zero path stays as a
+  safety net.
+- `pearl_wiener_alpha_floor_for_nonstationary` — every EMA uses
+  α-floor = 0.4.
+- `feedback_no_hiding` — `step_event`'s `_ts_ns` / `_trade_signed_vol`
+  go away with the `LobEnv` trait deletion.
+- `feedback_no_feature_flags` — controllers are unconditional.
+- `feedback_registry_entries_need_dispatch_arms` — controllers go in
+  the state-reset registry (if applicable to ml-alpha).
+- `pearl_tests_must_prove_not_lock_observations` — convergence-gate
+  fixtures assert invariants (reward signal flows, sign correctness),
+  not specific learned-checkpoint values.
+
+## Calendar estimate
+
+- R1: 0.5 days
+- R2: 1 day
+- R3: 1.5 days
+- R4: 1 day
+- R5: 1 day
+- R6: 2 days
+- R7: 0.5 days
+- R8: 0.5 days
+- R9: 0.5 days + 1h cluster smoke
+
+Total ~8.5 dev days. Roughly 2× the flawed Phase F + G calendar
+because the rebuild does the work the flawed branch deferred or
+simplified.