From e4c3cc60d2349f4fa53a8c6597ec80afaf6c7238 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sat, 23 May 2026 09:25:03 +0200 Subject: [PATCH] docs(plans): integrated RL trainer GPU-pure rebuild MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the flawed Phase F + G arc preserved on branch `ml-alpha-phase-f-g-flawed` (commits 99a125cdb..b3808a5ac). The prior attempt shipped a trainer with multiple production-blocking defects: 1. ISV[400..406] uninitialised → kernels read γ=0, ε=0, entropy=0 2. rl_reward_scale_controller drifted to 1e3 on no-trade steps 3. 6 controllers exist as .cu but never launched 4. Target net never soft-updated (τ has no consumer) 5. step_with_lobsim violated feedback_cpu_is_read_only with host Thompson sampling + EMA tracking + advantage/return loops 6. "toy" framing leaked into production (alpha_rl_train.rs shipped with next_snapshots=snapshots — the F.4 next-state code path was a no-op until that one issue got caught mid-review) 7. No NaN abort in production CLI The convergence-gate fixtures (dqn_toy/ppo_toy → renamed dqn_reward_signal/ppo_reward_signal on the flawed branch) hid every defect because MockLobEnv is state-invariant with horizon=1. The rebuild is GPU-pure: kernel-driven action sampling, kernel-driven EMA tracking, kernel-driven advantage/return, ISV bootstrap at trainer construction, all 7 controllers wired with device-resident EMA inputs, target-net soft update consumer, NaN abort, no LobEnv trait (drives LobSimCuda via the existing decision-policy kernel pattern). Sequenced as R1..R9 with falsifiability gates G1..G7 that exercise the specific failure modes the convergence-gate fixtures couldn't catch. Calendar ~8.5 dev days. Also adds memory pearl feedback_extending_existing_code_audits_for_existing_violations capturing the lesson: extending pre-existing CPU/orphan-controller violations is how this happened. Co-Authored-By: Claude Opus 4.7 --- ...026-05-23-integrated-rl-trainer-rebuild.md | 381 ++++++++++++++++++ 1 file changed, 381 insertions(+) create mode 100644 docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md diff --git a/docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md b/docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md new file mode 100644 index 000000000..894744eea --- /dev/null +++ b/docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md @@ -0,0 +1,381 @@ +# Integrated RL Trainer — GPU-Pure Rebuild + +> **Status:** Rebuild plan, replaces the flawed Phase F + G arc preserved +> on branch `ml-alpha-phase-f-g-flawed` (commits 99a125cdb..b3808a5ac). +> Baseline: `ml-alpha-phase-a` reset to commit `9114374d2` +> (Phase E.3b activation). + +## Why this rebuild + +The prior Phase F + G attempt landed 9 commits that shipped a trainer +with multiple production-blocking defects. The defects were not bugs in +the new code so much as **propagation of pre-existing violations** in +`step_with_lobsim` plus **incomplete wiring of the ISV adaptive-control +infrastructure**. The convergence-gate fixtures (`dqn_reward_signal`, +`ppo_reward_signal`, then named `dqn_toy` / `ppo_toy`) hid every defect +because their `MockLobEnv` has `done = true` every step, reward in +`[-1, +1]`, and horizon = 1. The cluster-bound smoke was the only +mechanism that would have exposed the problems — at $5–20 / 30–120 min +each. + +### Defects in the flawed branch + +1. **ISV[400..406] uninitialised in production paths.** The trainer + `alloc_zeros` the ISV array; only `rl_lr_controller` was launched. + Production consumers read sentinel `0`: + - `bellman_target_projection.cu:133` reads γ = 0 → Bellman target + reduces to `r + 0·V(s_{t+1}) = r`. Q-head learns immediate-reward + only, zero credit assignment. + - `ppo_clipped_surrogate.cu:205,305` reads ε = 0 → `clip(r, 1, 1) = 1` + always. PPO trust region collapses. + - `ppo_clipped_surrogate.cu:210` reads entropy coef = 0 → no entropy + bonus, policy collapses to deterministic immediately. + +2. **`rl_reward_scale_controller` drifts to `1e3` on no-trade steps.** + The launcher fired every step regardless of `mean_abs_pnl_ema` + readiness; with `ema = 0` and `prev = REWARD_SCALE_BOOTSTRAP = 1.0`, + the kernel computed `target = 1/max(0, 1e-3) = 1000`, Wiener-blended + toward it, and ramped the scale to `1e3` over the first ~10 hold + steps. Toy fixture hid this because `done = true` every step. + +3. **6 controllers exist as .cu files but never launched.** + `rl_gamma`, `rl_target_tau`, `rl_ppo_clip`, `rl_entropy_coef`, + `rl_rollout_steps`, `rl_per_alpha`. The build system compiles their + cubins; no Rust launcher fires them. ISV bootstrap (defect #1) + would have masked this even if the controllers had been wired. + +4. **Target-net soft update missing.** `DqnHead` owns `target_w_d / + target_b_d` but no kernel updates them. The "target network" is + actually the initial random init for the entire run — no soft Polyak + update, no hard sync. Double-DQN Bellman target uses an + untrained-static target. + +5. **`feedback_cpu_is_read_only` violation in `step_with_lobsim`.** + Host code: + - 5 DtoH copies per step (Q / V / π logits at h_t, V / Q at h_{t+1}) + - Host-side softmax + Thompson categorical sample + argmax loops + over `[B][N_ACTIONS][Q_N_ATOMS]` + - Host EMA updates for `mean_abs_pnl_ema` + - Host advantage / return computation + - HtoD copies for `actions`, `rewards`, `dones` + The pre-existing Phase E.3b code already had this pattern; the + flawed Phase F extended it instead of rebuilding it as GPU-resident + per `pearl_cold_path_no_exception_to_gpu_drives`. + +6. **"Toy" framing in production code.** Test files `dqn_toy.rs` / + `ppo_toy.rs`, fixture constructor `MockLobEnv::toy_bandit()`, and + pervasive "toy bandit" / "for the toy bandit smoke" comments leaked + into production source. Worse: `alpha_rl_train.rs` shipped with + `next_snapshots = snapshots` — production code applying a + toy-fixture simplification — until that one issue got caught + mid-review. + +7. **No NaN / divergence abort in `alpha_rl_train.rs`.** A NaN-poisoned + run would burn the full 14400-second `activeDeadlineSeconds` budget + silently. + +### Pattern lesson + +The convergence-gate fixtures verified the gradient mechanics +(`reward_signal_drives_argmax_to_rewarded_action`) but their +state-invariant fixture intrinsically *can't* test: +- Sparse-reward credit assignment (defect #1's γ = 0 effect) +- Reward-magnitude drift (defect #2) +- Trust-region clipping (defect #1's ε = 0 effect) +- Target-network staleness (defect #4) + +**Pre-cluster falsifiability requires fixtures that exercise the same +failure modes the cluster will see.** A short bull-market loop with +state-dependent rewards + per-step holds + multi-step trades is the +minimum. + +## Architectural decisions + +### A1. GPU-pure action submission via the existing decision-policy path + +The pre-existing production decision pipeline (`step_decision_with_latency` +in `crates/ml-backtesting/src/sim/mod.rs`) already submits actions +GPU-pure: `alpha_probs_d` (device) feeds `decision_policy_kernel` +which writes `market_targets_d` (device) which the fill kernel reads. +No host roundtrip. + +**Decision:** the integrated RL trainer drives the same kernel path. +Action sampling happens IN THE DECISION KERNEL (or a new +`rl_action_kernel` that lives parallel to it), reading Q / π logits + +ISV[entropy, ε] from device + writing to `market_targets_d`. The +`LobEnv` trait + `LobSimEnvAdapter` from the flawed branch are +DELETED. The trainer interacts with `LobSimCuda` via the existing +decision-policy entry point. + +**Consequence:** the trait-abstraction layer that let `MockLobEnv` mock +the env is removed. Convergence-gate tests must run against a real +`LobSimCuda` with synthetic book data (still feasible — fixtures +inject snapshots). + +### A2. All EMAs live in ISV + +Currently `mean_abs_pnl_ema` lives as a host `f32` field on +`IntegratedTrainer`. **Decision:** every EMA the controllers consume +lives in an ISV slot, updated by a dedicated per-step kernel that reads +the source signal from device buffers (rewards_d, dones_d, etc.). + +New ISV slots needed for the EMA inputs: +- `RL_MEAN_TRADE_DURATION_EMA_INDEX` (input to `rl_gamma_controller`) +- `RL_Q_DIVERGENCE_EMA_INDEX` (input to `rl_target_tau_controller`) +- `RL_KL_PI_EMA_INDEX` (input to `rl_ppo_clip_controller`) +- `RL_ENTROPY_OBSERVED_EMA_INDEX` (input to `rl_entropy_coef_controller`) +- `RL_ADVANTAGE_VAR_RATIO_EMA_INDEX` (input to `rl_rollout_steps_controller`) +- `RL_TD_KURTOSIS_EMA_INDEX` (input to `rl_per_alpha_controller`) +- `RL_MEAN_ABS_PNL_EMA_INDEX` (input to `rl_reward_scale_controller`) + +That's 7 new slots → `RL_SLOTS_END = 424`. + +### A3. ISV bootstrap at trainer construction + +Defect #1 and the Wiener-blend behavior of the controllers both +require slots to be at known-good values before any kernel reads them. +**Decision:** trainer `new()` does ONE memcpy_htod writing canonical +defaults to ISV[400..406]: + +| Slot | Default | Source | +|------|---------|--------| +| RL_GAMMA_INDEX (400) | 0.99 | DQN convention | +| RL_TARGET_TAU_INDEX (401) | 0.005 | DQN convention | +| RL_PPO_CLIP_INDEX (402) | 0.2 | PPO original paper | +| RL_ENTROPY_COEF_INDEX (403) | 0.01 | PPO convention | +| RL_N_ROLLOUT_STEPS_INDEX (404) | 2048 | PPO convention | +| RL_PER_ALPHA_INDEX (405) | 0.6 | Schaul et al. 2016 | +| RL_REWARD_SCALE_INDEX (406) | 1.0 | identity pass-through | + +Controllers then ADAPT from these defaults rather than bootstrapping +from sentinel zero. The sentinel-zero bootstrap path in each kernel +stays as a safety net but is never the production cold-start path. + +### A4. Target-net soft update consumer for ISV[401] + +New `dqn_target_soft_update` kernel: per-step element-wise +`target_w[i] = (1-τ)·target_w[i] + τ·w[i]` reading τ from `ISV[401]`. +Launched from `step_after_encoder_forward` after the Q-head Adam +update. + +### A5. GPU-resident action sampling + +New kernels: +- `thompson_sample_categorical` — input: Q logits, ISV[entropy_coef], + device PRNG state. Output: per-batch action index in `market_targets` + format directly. +- `argmax_expected_q` — input: Q logits. Output: per-batch + next-action index. +- `log_pi_at_action` — input: π logits + action_d. Output: per-batch + log π(a_t) — feeds the PPO importance-ratio path. + +Device PRNG via a per-batch `curandStatePhilox4_32_10_t` allocated at +trainer init + advanced per step. + +### A6. GPU-resident advantage / return / TD computation + +New `compute_advantage_return` kernel: per-batch `r + γ(1-done)·V_tp1` +and `r_t - V_t`, reading γ from `ISV[400]`. Replaces the host loop in +the flawed branch's `step_with_lobsim`. + +### A7. Fee model + +The flawed F.3a's approach (`order_match.cu` deduct fee + new +`upload_cost_per_lot_per_side` API) was correct in principle, mirroring +the existing `apply_fill_to_pos` pattern from `resting_orders.cu`. The +rebuild keeps this change. + +### A8. Loader pair API + +The flawed Phase G's `MultiHorizonLoader::next_sequence_pair` was +correct (true `(s_t, s_{t+1})` consecutive sampling). The rebuild keeps +this addition. + +## Falsifiability gates — pre-cluster + +Each gate is a local CUDA test that exercises the failure mode it +covers. None of these existed in the flawed branch. + +| Gate | Asserts | Catches defect | +|------|---------|----------------| +| G1 | ISV[400..406] held at canonical defaults after `IntegratedTrainer::new` | #1 | +| G2 | `RL_REWARD_SCALE_INDEX` stays at 1.0 for first N hold-only steps | #2 | +| G3 | Each controller's launcher visibly mutates its ISV slot after first non-sentinel input | #3 | +| G4 | `target_w` Frobenius norm drifts from `w` only after the soft-update kernel fires | #4 | +| G5 | `cudaMemcpy` count per `step_with_lobsim` is 0 host-initiated (other than ISV diagnostic readback) | #5 | +| G6 | Bull-market forced-long 50-trade loop: mean reward per trade > 0 (already in flawed F.3b's `reward_calibration.rs`) | (regression-coverage for fee + scale) | +| G7 | Run for N steps; abort + non-zero exit code if any per-head loss is non-finite | #7 | + +G5 is the load-bearing gate for the GPU-pure decision (A1). Tooling: +hook `cudaMemcpyAsync` via a CUPTI counter or count Rust-side +`memcpy_htod` / `memcpy_dtoh` calls during the step. + +## Required new files + +| File | Purpose | +|------|---------| +| `crates/ml-alpha/cuda/thompson_sample_categorical.cu` | A5 | +| `crates/ml-alpha/cuda/argmax_expected_q.cu` | A5 | +| `crates/ml-alpha/cuda/log_pi_at_action.cu` | A5 | +| `crates/ml-alpha/cuda/compute_advantage_return.cu` | A6 | +| `crates/ml-alpha/cuda/dqn_target_soft_update.cu` | A4 | +| `crates/ml-alpha/cuda/ema_update_on_done.cu` | A2 — generic done-gated EMA (parameterised slot) | +| `crates/ml-alpha/cuda/ema_update_per_step.cu` | A2 — generic per-step EMA (parameterised slot) | + +Reused / extended: +- `rl_reward_scale_controller.cu` (port from flawed branch) +- `rl_gamma_controller.cu`, `rl_target_tau_controller.cu`, + `rl_ppo_clip_controller.cu`, `rl_entropy_coef_controller.cu`, + `rl_rollout_steps_controller.cu`, `rl_per_alpha_controller.cu` + (already in baseline tree — port their Rust launchers from the + flawed branch concept, but feed device-resident EMA inputs) +- `crates/ml-backtesting/cuda/order_match.cu` (port the F.3a fee + deduction) + +Deleted: +- `crates/ml-alpha/src/rl/reward.rs::LobEnv` trait +- `crates/ml-backtesting/src/sim/mod.rs::LobSimEnvAdapter` + +## Phased execution + +Each phase = one commit. Each phase has a green-build + green-gate +requirement before the next phase starts. + +### R1 — ISV bootstrap + slot extension (1 commit) + +- Add new ISV slot constants for the 7 EMA inputs (A2) → `RL_SLOTS_END = 424`. +- In `IntegratedTrainer::new`, one memcpy_htod writing canonical + defaults to ISV[400..406] (A3). +- Gate G1. + +### R2 — Fee model + loader pair API (2 commits) + +- R2a: port `order_match.cu` fee deduction + `upload_cost_per_lot_per_side` + host API (from F.3a). Gate: existing `lob_env_adapter_buy_close_cycle` + passes with zero fees; new with-fees variant lands. +- R2b: port `MultiHorizonLoader::next_sequence_pair` (from G fix). + +### R3 — GPU-pure action sampling (3 commits) + +- R3a: `thompson_sample_categorical` kernel + Rust launcher. Local CUDA + test: sample 10000 actions from a known Q distribution, assert + empirical frequency matches softmax probabilities within tolerance. +- R3b: `argmax_expected_q` kernel. Local CUDA test: feed adversarial + Q distributions, assert argmax matches host reference. +- R3c: `log_pi_at_action` kernel. Local CUDA test: feed known π + distribution, assert log-prob within 1e-5 of host softmax. + +### R4 — GPU-resident EMA + advantage / return (2 commits) + +- R4a: `ema_update_on_done.cu` + `ema_update_per_step.cu` generic + kernels. Local CUDA test: feed deterministic reward stream, assert + EMA tracks the analytic Wiener-α trajectory. +- R4b: `compute_advantage_return.cu`. Local CUDA test: feed known + rewards / dones / V values, assert advantage / return match host + reference. + +### R5 — Wire all 7 controllers + target-net soft update (1 commit) + +- Add launcher fields + helpers for all 7 controllers (gamma, + target_tau, ppo_clip, entropy_coef, rollout_steps, per_alpha, + reward_scale). Each launcher pulls its input from the corresponding + EMA slot (A2 from R4a). Gate G3. +- Add `dqn_target_soft_update` kernel + launcher (A4). Gate G4. + +### R6 — Replace `LobEnv` with kernel-driven decision path (3 commits) + +- R6a: delete `LobEnv` trait + `LobSimEnvAdapter`. Add direct + `LobSimCuda` integration on `IntegratedTrainer` (no trait). +- R6b: wire action submission via the decision-policy kernel pattern + (mirror `step_decision_with_latency`). Adapt `market_targets_d` to + accept the Thompson-sampled action index directly. +- R6c: rewrite the trainer's convergence-gate fixtures (formerly + `dqn_toy.rs` / `ppo_toy.rs`) to drive a real `LobSimCuda` with + synthetic book data instead of a `MockLobEnv`. Reward signal becomes + "synthetic monotonic uptick → forced long → trade close pays > 0". +- Gate G5 (no host memcpy in step_with_lobsim other than ISV + diagnostics). + +### R7 — Reward composition (1 commit) + +- Per-step reward = `realized_pnl_delta` (already net of fees + slippage + from R2a) scaled by `ISV[RL_REWARD_SCALE_INDEX]`. Optionally subtract + exposure penalty (deferred unless smoke shows it's needed). +- Gate G6. + +### R8 — CLI binary + Argo template + dispatcher (1 commit) + +- `crates/ml-alpha/examples/alpha_rl_train.rs` driving the rebuilt + trainer. Uses `next_sequence_pair` for true `(s_t, s_{t+1})` windows. +- `infra/k8s/argo/alpha-rl-template.yaml` + `scripts/argo-alpha-rl.sh` + (port from flawed branch). +- NaN abort: per-step `is_finite()` check on `l_total`, exit code 2 on + divergence. Gate G7. + +### R9 — Pre-cluster smoke + first cluster smoke (out-of-band) + +- Run all R1..R8 local CUDA tests on a GPU host. Confirm green. +- `git push`. `kubectl apply -f` template. `./scripts/argo-alpha-rl.sh + --n-steps 1000` for a 5-min validation smoke. If green, full 50k. + +## What we keep from the flawed branch + +These specific elements landed correctly and are ported directly: +- `rl_reward_scale_controller.cu` kernel (Phase F.2) +- `order_match.cu` fee deduction pattern (Phase F.3a) +- `MultiHorizonLoader::next_sequence_pair` (Phase G fix) +- The 6 unlaunched controller .cu files (Phases C, D, E) — they're + fine kernels, just need launchers +- F.4's true-h_{t+1} architectural insight (Bellman target on + h_{t+1}, V bootstrap on h_{t+1}) — the structure becomes simpler in + the GPU-pure rebuild because state lives on device + +## What we discard + +- `LobEnv` trait + `LobSimEnvAdapter` (replaced by kernel-driven path) +- `MockLobEnv` "toy bandit" fixture (replaced by synthetic-book + `LobSimCuda` fixture) +- Host-side action sampling, EMA tracking, advantage / return loops +- The `dqn_toy.rs` / `ppo_toy.rs` test file names (renamed to + `dqn_reward_signal.rs` / `ppo_reward_signal.rs` in the rebuild) + +## Pearls honored + +- `feedback_cpu_is_read_only` — A1, A5, A6 are direct compliance. +- `feedback_isv_for_adaptive_bounds` — A2, A3, R5. +- `feedback_no_partial_refactor` — each R-phase is atomic. +- `feedback_no_stubs` — every controller has a launcher AND a + consumer at landing time. +- `feedback_no_atomicadd` — new kernels use single-writer-per-block + or per-batch scratch + reduce_axis0. +- `feedback_no_htod_htoh_only_mapped_pinned` — applies to the + remaining ISV diagnostic readback. +- `pearl_first_observation_bootstrap` — combined with A3, the host + bootstrap takes precedence; kernel sentinel-zero path stays as a + safety net. +- `pearl_wiener_alpha_floor_for_nonstationary` — every EMA uses + α-floor = 0.4. +- `feedback_no_hiding` — `step_event`'s `_ts_ns` / `_trade_signed_vol` + go away with the `LobEnv` trait deletion. +- `feedback_no_feature_flags` — controllers are unconditional. +- `feedback_registry_entries_need_dispatch_arms` — controllers go in + the state-reset registry (if applicable to ml-alpha). +- `pearl_tests_must_prove_not_lock_observations` — convergence-gate + fixtures assert invariants (reward signal flows, sign correctness), + not specific learned-checkpoint values. + +## Calendar estimate + +- R1: 0.5 days +- R2: 1 day +- R3: 1.5 days +- R4: 1 day +- R5: 1 day +- R6: 2 days +- R7: 0.5 days +- R8: 0.5 days +- R9: 0.5 days + 1h cluster smoke + +Total ~8.5 dev days. Roughly 2× the flawed Phase F + G calendar +because the rebuild does the work the flawed branch deferred or +simplified.