# Integrated RL Trainer — GPU-Pure Rebuild

> **Status:** Rebuild plan, replaces the flawed Phase F + G arc preserved
> on branch `ml-alpha-phase-f-g-flawed` (commits 99a125cdb..b3808a5ac).
> Baseline: `ml-alpha-phase-a` reset to commit `9114374d2`
> (Phase E.3b activation).
>
> **Discipline:** Smokes are expensive ($5–20 / 30–120 min L40S each)
> and producing G8 numbers from a buggy trainer wastes both cluster
> credit and gate-evaluation cycles. Every rule below is non-negotiable.
> Every gate is falsifiable locally before any `argo submit`.

## Why this rebuild

The prior Phase F + G attempt landed 9 commits with multiple
production-blocking defects. None of the defects were exotic bugs;
all of them were direct violations of pre-existing project memory
that I missed because the `MockLobEnv` "toy bandit" fixture has
`done = true` every step, reward ∈ `[-1, +1]`, and horizon = 1 —
which hides every failure mode the production cluster will see.

### Defect table — each row maps to a memory rule the flawed branch violated

| # | Defect | Memory rule violated |
|---|--------|----------------------|
| 1 | ISV[400..406] uninitialised in production paths (γ=0, ε=0, entropy=0 in kernels) | `feedback_wire_everything_up`, `feedback_isv_for_adaptive_bounds` |
| 2 | `rl_reward_scale_controller` drifts to 1e3 on no-trade steps | `pearl_first_observation_bootstrap` |
| 3 | 6 controllers exist as .cu files but never launched | `feedback_wire_everything_up`, `feedback_no_partial_refactor` |
| 4 | Target net never soft-updated (τ has no consumer) | `feedback_no_stubs`, `feedback_wire_everything_up` |
| 5 | `step_with_lobsim` has 5 DtoH + host argmax + host EMA + host advantage loops | `feedback_cpu_is_read_only`, `pearl_cold_path_no_exception_to_gpu_drives` |
| 6 | PER buffer exists in `src/rl/replay.rs` but trainer never samples from it | `feedback_always_per` |
| 7 | "Toy bandit" framing leaked into production (`alpha_rl_train.rs` shipped with `next_snapshots = snapshots` until caught mid-review) | `feedback_no_stubs`, `feedback_no_quickfixes`, `feedback_extending_existing_code_audits_for_existing_violations` |
| 8 | No NaN / divergence abort in production CLI | `feedback_stop_on_anomaly`, `feedback_kill_runs_on_anomaly_quickly` |
| 9 | Convergence-gate fixtures (`dqn_toy.rs`, `ppo_toy.rs`) used `MockLobEnv` with state-invariant rewards — intrinsically can't catch defects #1, #2, #5 | `pearl_tests_must_prove_not_lock_observations`, `feedback_no_cpu_test_fallbacks` |

### Root-cause pattern

Defects #1, #3, #5, #6 are all the same shape: **the pre-existing
Phase E.3b code already violated current memory rules, and the
flawed Phase F extended the pattern instead of flagging it.** The new
memory pearl `feedback_extending_existing_code_audits_for_existing_violations`
exists specifically to prevent this in future work.

## Strict memory rules this rebuild MUST honor

Explicit catalog, every rule named below is gating. If the rebuild
diverges from any of these the work does NOT ship.

### GPU/CPU contract
- `feedback_cpu_is_read_only` — no CPU compute, no CPU forwards, no GPU↔CPU roundtrips on hot path
- `feedback_no_htod_htoh_only_mapped_pinned` — mapped-pinned only; tests not exempt
- `feedback_no_atomicadd` — block tree-reduce / per-batch scratch + `reduce_axis0`, never atomicAdd
- `feedback_no_cpu_test_fallbacks` — GPU oracle tests only; no CPU reference implementations
- `feedback_no_nvrtc` — pre-compiled cubins via `build.rs`
- `pearl_no_host_branches_in_captured_graph` — host branches inside captured graphs break recording
- `pearl_cold_path_no_exception_to_gpu_drives` — "cold path" is not a license for CPU compute
- `pearl_fused_per_group_statistics_oracle` — K groups × N stats = ONE fused kernel
- `pearl_build_rs_rerun_if_env_changed` — every `std::env::var()` paired with `cargo:rerun-if-env-changed`

### Code quality
- `feedback_no_stubs` — wire it for real or delete
- `feedback_no_todo_fixme` — no TODO/FIXME/XXX markers
- `feedback_no_hiding` — never suppress with `_` prefix or `#[allow]`
- `feedback_no_legacy_aliases` — no deprecated wrappers
- `feedback_no_feature_flags` — no `enable_*` / `use_*` booleans
- `feedback_no_quickfixes` — every issue gets the proper fix
- `feedback_no_functionality_removal` — never delete features; fix what's broken
- `feedback_fix_everything_aggressively` — zero tolerance, no deferrals
- `feedback_extending_existing_code_audits_for_existing_violations` — when extending existing code, audit it FIRST against current rules

### Architecture discipline
- `feedback_single_source_of_truth_no_duplicates` — no `v2_` / parallel impls
- `feedback_no_partial_refactor` — contract change = every consumer migrates atomically
- `feedback_wire_everything_up` — every kernel/module wired in same commit; no orphans
- `feedback_isv_for_adaptive_bounds` — adaptive bounds live in ISV, not hardcoded constants
- `feedback_adaptive_not_tuned` — signal-driven (EMA/windowed), not tuned constants
- `feedback_magnitude_must_be_useful` — DQN magnitude branch never deleted

### Training discipline
- `feedback_always_per` — PER always enabled; non-PER paths are dead code
- `feedback_mbp10_mandatory` — `--mbp10-data-dir` mandatory; trades-data-dir paired for OFI completeness
- `feedback_default_to_l40s_pool` — defaults to `ci-training-l40s`, not H100
- `feedback_stop_on_anomaly` — terminate on training anomaly, diagnose, fix, re-run
- `feedback_kill_runs_on_anomaly_quickly` — kill smoke at first useful signal

### Workflow
- `feedback_push_before_deploy` — always `git push` before `argo submit`
- `feedback_argo_template_must_apply` — `kubectl apply -f <template>` before `argo submit --from=wftmpl/<name>`
- `feedback_no_concurrent_agents_shared_tree` — worktrees or serial dispatch
- `feedback_registry_entries_need_dispatch_arms` — registry entries require matching reset arms in same commit
- `feedback_smoke_validation_before_structural_priors` — validate wiring with neutral config first

### Reward + controller pearls
- `pearl_first_observation_bootstrap` — sentinel=0; first observation replaces directly
- `pearl_wiener_alpha_floor_for_nonstationary` — α-floor 0.4 in control loops
- `pearl_wiener_optimal_adaptive_alpha` — α = diff_var / (diff_var + sample_var + ε); no tuned α
- `pearl_blend_formulas_must_have_permanent_floor` — `max(real, floor)`, not blend
- `pearl_one_unbounded_signal_per_reward` — exactly ONE unbounded multiplicand per reward term
- `pearl_symmetric_clamp_audit` — kernel-derived bounded scalars use bilateral `fmaxf(lo, fminf(x, hi))`
- `pearl_audit_unboundedness_for_implicit_asymmetry` — symmetric bounds may erase implicit asymmetry; use asymmetric bounded clamp
- `pearl_event_driven_reward_density_alignment` — event-driven objectives need event-density reward
- `pearl_bounded_modifier_outputs_require_structural_activation` — bounded outputs feeding multiplicative modifiers MUST use structural activation (sigmoid/tanh), not runtime clamps
- `pearl_controller_anchors_isv_driven` — every controller anchor/target/cap is ISV-driven
- `pearl_thompson_for_distributional_action_selection` — Thompson at rollout, argmax for Bellman target only

### Backtest / OOS discipline
- `pearl_single_window_oos_is_not_oos` — ≥3 walk-forward folds, report mean ± SD
- `pearl_q_spread_kill_criterion_misaligned_for_distributional_q` — DON'T use Q_SPREAD as kill metric (spikes signal GOOD balance under C51+Thompson)
- `pearl_tests_must_prove_not_lock_observations` — assert invariants, not observed values
- `pearl_action_pruning_falsified` — KEEP 9 actions (pruning 9→4 regressed Sharpe 2.4×)
- `pearl_c51_thompson_closed_phase_e3_gap` — C51 + Thompson is the proven combo

### Kernel hygiene pearls
- `pearl_kernel_input_filter_must_match_validation_surface` — per-depth validation
- `pearl_behavioral_sentinel_values_poison_downstream_consumers` — per-emission-site counters localize bugs
- `pearl_cooperative_staging_eliminates_redundant_reads` — stage per-batch rows once
- `pearl_coalesce_via_thread_role_swap` — output-row → output-column thread role for non-coalesced writes
- `pearl_reduce_axis0_column_tile` — block-per-column 32-wide column tile

## Architectural decisions

### A1 — Delete `LobEnv` trait; drive `LobSimCuda` via kernel-driven decision path

The production decision pipeline (`step_decision_with_latency` in
`crates/ml-backtesting/src/sim/mod.rs`) already submits actions
GPU-pure: `alpha_probs_d` (device) feeds `decision_policy_kernel`
which writes `market_targets_d` (device). No host roundtrip.

**Decision:** the RL trainer drives the same kernel path. Action
sampling lives in a new `rl_action_kernel` (Thompson over Q logits)
parallel to the production decision kernel; both write to
`market_targets_d`. `LobEnv` trait, `LobSimEnvAdapter`, and
`MockLobEnv` are DELETED. Convergence-gate fixtures use real
`LobSimCuda` with synthetic book data.

Rules honored: `feedback_cpu_is_read_only`,
`pearl_cold_path_no_exception_to_gpu_drives`,
`feedback_single_source_of_truth_no_duplicates` (no duplicate
action-submission paths), `feedback_no_legacy_aliases` (no trait
adapter left as a shim).

### A2 — All EMAs live in ISV

Every controller-input EMA lives in an ISV slot, updated by a
dedicated per-step kernel that reads the source signal from device
buffers (rewards_d, dones_d, actions_d, etc.).

New ISV slots:
- `RL_MEAN_TRADE_DURATION_EMA_INDEX` (input to `rl_gamma_controller`)
- `RL_Q_DIVERGENCE_EMA_INDEX` (input to `rl_target_tau_controller`)
- `RL_KL_PI_EMA_INDEX` (input to `rl_ppo_clip_controller`)
- `RL_ENTROPY_OBSERVED_EMA_INDEX` (input to `rl_entropy_coef_controller`)
- `RL_ADVANTAGE_VAR_RATIO_EMA_INDEX` (input to `rl_rollout_steps_controller`)
- `RL_TD_KURTOSIS_EMA_INDEX` (input to `rl_per_alpha_controller`)
- `RL_MEAN_ABS_PNL_EMA_INDEX` (input to `rl_reward_scale_controller`)

7 new slots → `RL_SLOTS_END = 424`.

All EMA updates use `pearl_wiener_optimal_adaptive_alpha` (adaptive α
from observed diff/sample variance ratio) with floor 0.4 per
`pearl_wiener_alpha_floor_for_nonstationary`. Sentinel-zero bootstrap
per `pearl_first_observation_bootstrap`.

Rules honored: `feedback_isv_for_adaptive_bounds`,
`pearl_controller_anchors_isv_driven`, `feedback_adaptive_not_tuned`.

### A3 — Bootstrap controllers via launch-once at init, NOT host memcpy_htod

The prior plan said "memcpy_htod canonical defaults to ISV[400..406]".
That violated `feedback_no_htod_htoh_only_mapped_pinned` (tests not
exempt) and short-circuited `pearl_first_observation_bootstrap`.

**Decision:** in `IntegratedTrainer::new`, after the ISV alloc_zeros,
launch each controller ONCE with a zero (sentinel) input EMA. Each
kernel sees `prev = 0` (sentinel), executes its first-observation
bootstrap path, and writes its canonical value (γ=0.99, τ=0.005, ε=0.2,
entropy_coef=0.01, n_rollout_steps=2048, per_α=0.6, reward_scale=1.0).

Subsequent in-loop launches see non-sentinel ISV → execute Wiener
blend with real EMA input.

No host write to ISV. Bootstrap path is canonical, observable in HEALTH
diagnostics, and follows the same `pearl_first_observation_bootstrap`
pattern as every other adaptive controller in the codebase.

Rules honored: `pearl_first_observation_bootstrap`,
`feedback_isv_for_adaptive_bounds`,
`feedback_no_htod_htoh_only_mapped_pinned`,
`feedback_wire_everything_up`.

### A4 — Target-net soft update consumer for ISV[401]

New `dqn_target_soft_update` kernel: element-wise per-weight
`target_w[i] = (1-τ)·target_w[i] + τ·w[i]` reading τ from `ISV[401]`.
Parallel across weight elements (no contention; no atomicAdd needed).
Launched from `step_after_encoder_forward` after the Q-head Adam
update.

Rules honored: `feedback_wire_everything_up`,
`feedback_no_atomicadd`, `feedback_no_stubs`.

### A5 — GPU-resident action sampling kernels

New kernels:
- `thompson_sample_categorical` — per-batch: softmax over atoms,
  sample one atom per action via device PRNG, pick argmax over
  sampled per-action returns. Writes action index to `market_targets_d`
  directly. Single-writer-per-block. Device PRNG state =
  `curandStatePhilox4_32_10_t` allocated at trainer init, advanced per
  step. Reads ISV[entropy_coef] if exploration-temperature scaling
  applies.
- `argmax_expected_q` — per-batch: softmax + expected value + argmax.
  Writes next_actions for the Bellman target argmax. Single-writer.
- `log_pi_at_action` — per-batch: softmax + log-prob lookup at action.
  Writes log_pi_old for PPO importance ratio. Single-writer.

Per `pearl_thompson_for_distributional_action_selection`: Thompson at
rollout (rl_action_kernel uses `thompson_sample_categorical`), argmax
only for Bellman target (uses `argmax_expected_q`).

Rules honored: `feedback_cpu_is_read_only`, `feedback_no_atomicadd`,
`pearl_thompson_for_distributional_action_selection`,
`pearl_no_host_branches_in_captured_graph`.

### A6 — GPU-resident advantage / return / TD computation

New `compute_advantage_return` kernel: per-batch
`returns[b] = r_b + γ(1 − done_b) · v_tp1[b]`,
`advantages[b] = returns[b] − v_t[b]`. Reads γ from `ISV[400]`,
v_t / v_tp1 from device buffers. Element-wise; trivially parallel.

Replaces the host advantage/return loop in the flawed branch's
`step_with_lobsim`.

Rules honored: `feedback_cpu_is_read_only`,
`pearl_controller_anchors_isv_driven`.

### A7 — Fee model: port F.3a unchanged

`order_match.cu::submit_market_immediate` adds per-fill cost
deduction mirroring `apply_fill_to_pos` from `resting_orders.cu:213-219`.
`upload_cost_per_lot_per_side` API on `LobSimCuda`. Default zero,
opt-in (CLI defaults to ES $1.25/side, $2.50 round-trip).

Single-writer-per-block plain `+=` for `total_fees_per_b[b]` (line 79
of `submit_market_immediate` has `threadIdx.x != 0 → return`).

Rules honored: `feedback_no_atomicadd` (single-writer),
`feedback_single_source_of_truth_no_duplicates` (mirrors existing
`apply_fill_to_pos` pattern).

### A8 — Loader pair API: port G fix unchanged

`MultiHorizonLoader::next_sequence_pair` returns
`(LabeledSequence, LabeledSequence)` at adjacent anchors in the same
file. Refactor `next_sequence`'s body into `build_sequence_at(file_idx,
anchor)` helper shared by both API surfaces.

Rules honored: `feedback_no_partial_refactor` (helper shared by both
call sites; no duplicate windowing logic).

### A9 — Wire PER replay into the training step

`feedback_always_per` says PER is always on; the flawed branch had
the `ReplayBuffer` struct but never sampled from it in
`step_with_lobsim`. The rebuild WIRES the buffer:

- Every step pushes the current transition to the replay buffer.
- Every step samples a batch from the buffer (priority^α) for the
  off-policy Q-head update.
- TD-error magnitudes from the Q backward feed
  `update_priorities`.
- `RL_PER_ALPHA_INDEX` (already in ISV[405], adaptive via
  `rl_per_alpha_controller`) drives the priority exponent.

PER buffer storage: device-resident `h_t`/`next_h_t` per
`pearl_cold_path_no_exception_to_gpu_drives`. Sum-tree starts
naive O(N) per `replay.rs` doc (acceptable at capacity ≤ 4096); GPU
sum-tree is a Phase R-future enhancement.

Rules honored: `feedback_always_per`, `feedback_no_stubs` (replay
buffer becomes load-bearing instead of orphan), `feedback_wire_everything_up`.

## Falsifiability gates — local CUDA BEFORE cluster

Each gate has a local CUDA test (`#[ignore = "requires CUDA"]`,
runnable on the dev RTX 3050 Ti per CLAUDE.md) that exercises the
failure mode the convergence-gate fixtures couldn't catch. Per
`feedback_no_cpu_test_fallbacks`: every test oracle is either
analytical synthetic, property-based, or cross-kernel GPU oracle.
NEVER a CPU reference implementation.

| Gate | Asserts (invariant, not observed value) | Test design (GPU oracle only) | Catches defect |
|------|----------------------------------------|-------------------------------|----------------|
| G1 | ISV[400..406] equal canonical values after `IntegratedTrainer::new` | Construct trainer, read ISV slice via HEALTH_DIAG mapped-pinned, assert each slot equals its bootstrap value | #1 |
| G2 | `RL_REWARD_SCALE_INDEX` stays in `[1e-3, 1e3]` and remains at 1.0 across N hold-only env steps (no closed trades) | Drive trainer for 100 steps with synthetic book where market never crosses (no fills); assert ISV[406] never moves from 1.0 | #2 |
| G3 | Each controller's launcher visibly mutates its ISV slot after first non-sentinel EMA input | Construct trainer, manually launch one controller with a non-zero scalar input via the EMA-update kernel, assert ISV slot moved off bootstrap value | #3 |
| G4 | `target_w` Frobenius distance from `w` is zero before `dqn_target_soft_update` ever fires, and >0 after one fire | Construct trainer, snapshot `target_w_d`, fire 1 step (which fires soft-update once), snapshot again, assert any element changed | #4 |
| G5 | Zero host-initiated `cudaMemcpyAsync` calls in `step_with_lobsim`'s hot path (other than HEALTH_DIAG readback) | Hook `cudaMemcpyAsync` via CUPTI counter OR count Rust-side `memcpy_htod`/`memcpy_dtoh` calls; assert count = expected ISV-readback count | #5 |
| G6 | PER buffer's `len()` grows by exactly 1 per `step_with_lobsim` call; `sample_indices` returns batch of correct size | Drive trainer for 100 steps, assert `replay.len() == min(100, capacity)`; sample, assert batch size | #6 |
| G7 | Bull-market 50-cycle forced-long produces mean per-trade reward > 0 on synthetic `LobSimCuda` with ES-realistic fees | Port the flawed branch's `reward_calibration.rs` to drive `LobSimCuda` directly (no `LobEnv` adapter) | (regression coverage for fee + scale wiring) |
| G8 | NaN abort: trainer exits with non-zero code if any per-head loss is non-finite | Inject NaN into a synthetic reward stream, assert process exits 2 within 1 step | #8 |

G5 is the load-bearing gate for the GPU-pure decision (A1).
Implementation: a thin Rust wrapper around `CudaStream::memcpy_*`
that increments a counter; tests assert the counter after one
`step_with_lobsim` call. CUPTI is a follow-up enhancement.

## Cluster smoke discipline (Phase H equivalent)

Per `pearl_single_window_oos_is_not_oos`: the G8 backtest gate
(`profit_factor > 1.0`) MUST be evaluated across ≥3 walk-forward
folds, reporting mean ± SD. A single-window pass is NOT G8 pass.

Per `feedback_kill_runs_on_anomaly_quickly`: the cluster smoke
script tails logs and kills the run if any of the following fire:
- Any per-head loss non-finite (G8)
- `RL_REWARD_SCALE_INDEX` clipped at `1e-3` or `1e3` (instrumentation
  saturation indicator)
- Per-step wall time > 10× the rolling median (kernel hang indicator)

DO NOT kill on Q-distribution spread (per
`pearl_q_spread_kill_criterion_misaligned_for_distributional_q`:
Q_SPREAD spikes signal GOOD balance under C51+Thompson).

Per `feedback_push_before_deploy`: dispatcher script verifies
`git push` succeeded before `argo submit`. Hard-error if local HEAD
differs from `origin/<branch>` after push.

Per `feedback_argo_template_must_apply`: dispatcher prints a
top-of-output reminder `kubectl apply -n foxhunt -f
infra/k8s/argo/alpha-rl-template.yaml` and refuses to submit if the
template has changed since last apply (compare git SHA of the
template file vs the cluster CRD's `app.kubernetes.io/version` label
— or just hard-print the reminder).

Per `feedback_default_to_l40s_pool`: dispatcher defaults to
`--gpu-pool ci-training-l40s` (NOT H100). H100 pool is opt-in for
production scale-up, not smoke. Per `feedback_h100_gpu`: when H100
IS chosen, max-utilization config; hard-error on under-utilization.

## Required new files

### Kernels
- `crates/ml-alpha/cuda/rl_action_kernel.cu` — A5 Thompson sampler writing market_targets
- `crates/ml-alpha/cuda/argmax_expected_q.cu` — A5
- `crates/ml-alpha/cuda/log_pi_at_action.cu` — A5
- `crates/ml-alpha/cuda/compute_advantage_return.cu` — A6
- `crates/ml-alpha/cuda/dqn_target_soft_update.cu` — A4
- `crates/ml-alpha/cuda/ema_update_on_done.cu` — A2 (generic done-gated EMA, parameterised by slot)
- `crates/ml-alpha/cuda/ema_update_per_step.cu` — A2 (generic per-step EMA, parameterised by slot)

### Ported from flawed branch
- `crates/ml-alpha/cuda/rl_reward_scale_controller.cu`
- `crates/ml-backtesting/cuda/order_match.cu` modifications (A7)

### Already in baseline, need Rust launchers added (R5)
- `rl_gamma_controller.cu`, `rl_target_tau_controller.cu`,
  `rl_ppo_clip_controller.cu`, `rl_entropy_coef_controller.cu`,
  `rl_rollout_steps_controller.cu`, `rl_per_alpha_controller.cu`

### Rust modules
- Extensions to `crates/ml-alpha/src/trainer/integrated.rs` (rebuild
  `step_with_lobsim` as GPU-pure, add launchers)
- Extensions to `crates/ml-alpha/src/rl/isv_slots.rs` (7 new EMA slots)
- New `crates/ml-alpha/src/rl/per_step.rs` (wires PER buffer into the step)
- Extensions to `crates/ml-alpha/src/data/loader.rs` (A8 port)
- Extensions to `crates/ml-backtesting/src/sim/mod.rs` (A7 port)

### Deleted
- `crates/ml-alpha/src/rl/reward.rs::LobEnv` trait + `MockLobEnv`
- (Flawed branch's `LobSimEnvAdapter` was on `ml-backtesting`; not in
  baseline since baseline reset removed it)

### Renamed (per the "no toys" feedback)
- `crates/ml-alpha/tests/dqn_toy.rs` → `dqn_reward_signal.rs`
  (rewritten to drive synthetic-book `LobSimCuda` directly)
- `crates/ml-alpha/tests/ppo_toy.rs` → `ppo_reward_signal.rs`
  (same)
- All "toy bandit" / "bandit" comments in production source rewritten
  to "reward-signal fixture" or equivalent

## Phased execution

Each R-phase = one atomic commit. Each phase has a green-build +
green local-CUDA-gate requirement before the next phase starts. No
phase is allowed to defer work to a later phase (per
`feedback_no_partial_refactor` and the lesson from the flawed
branch).

### R1 — ISV slot extension + controller bootstrap launches (1 commit)
- Add 7 new ISV slot constants (`RL_*_EMA_INDEX`) → `RL_SLOTS_END = 424`.
- In `IntegratedTrainer::new`: alloc_zeros ISV, then launch each of
  the 7 controllers ONCE with sentinel-zero EMA input. Each kernel's
  first-observation-bootstrap path fires → ISV slot gets canonical
  value (A3).
- **Gate G1** local CUDA test.

### R2 — Fee model + loader pair API (1 commit, was 2)
- Port `order_match.cu` fee deduction + `upload_cost_per_lot_per_side`
  host API (A7).
- Port `MultiHorizonLoader::next_sequence_pair` + `build_sequence_at`
  refactor (A8).
- Existing tests pass; no new tests in this phase (covered by G7 later).

### R3 — GPU-resident EMA + advantage/return kernels (1 commit)
- `ema_update_on_done.cu` + `ema_update_per_step.cu` generic kernels
  (A2).
- `compute_advantage_return.cu` (A6).
- Local CUDA tests for each kernel via GPU-oracle pattern:
  - EMA: feed constant input, assert ISV converges to input (property:
    EMA of constant = constant).
  - EMA bootstrap: feed sentinel-zero start + one observation, assert
    slot equals observation (per `pearl_first_observation_bootstrap`).
  - Advantage/return: feed all-zero rewards + done=0 + v_t=v_tp1=k,
    assert returns = γ·k and advantages = γ·k − k = (γ−1)·k (property).

### R4 — GPU-resident action sampling kernels (1 commit)
- `rl_action_kernel.cu` (Thompson) + `argmax_expected_q.cu` +
  `log_pi_at_action.cu` + curand PRNG state allocation.
- Local CUDA tests via GPU-oracle:
  - `thompson_sample_categorical`: sample 10000 actions from a Q
    distribution where atom probability mass is concentrated on a
    known atom; assert empirical action frequency matches the
    softmax-weighted expected value within tolerance. Cross-kernel
    oracle: compare against `argmax_expected_q` on the same input
    when the distribution is sharp enough that Thompson ≈ argmax.
  - `argmax_expected_q`: feed Q distribution where one action has
    strictly higher expected value than all others by a known margin;
    assert that action is chosen for every batch.
  - `log_pi_at_action`: feed π logits where one action has logit
    massively above others; assert log π(that action) ≈ 0
    (probability ≈ 1).

### R5 — Wire all 7 controllers + target-net soft update (1 commit)
- Launcher fields + helpers for all 7 controllers (gamma, target_tau,
  ppo_clip, entropy_coef, rollout_steps, per_alpha, reward_scale).
  Each pulls input from the corresponding EMA slot (R3a outputs).
- `dqn_target_soft_update.cu` + launcher (A4).
- **Gate G3** local CUDA test (per-controller).
- **Gate G4** local CUDA test (soft-update fires + moves target_w).

### R6 — Replace `LobEnv` with kernel-driven decision path (2 commits)
- R6a: delete `LobEnv` trait + `MockLobEnv` from
  `src/rl/reward.rs`. Direct `LobSimCuda` integration on
  `IntegratedTrainer` (no trait abstraction). Wire action submission
  via the existing decision-policy kernel pattern; `market_targets_d`
  receives the Thompson-sampled index from `rl_action_kernel`.
- R6b: rewrite convergence-gate fixtures (`dqn_reward_signal.rs`,
  `ppo_reward_signal.rs`, `integrated_trainer_smoke.rs`) to drive a
  real `LobSimCuda` with synthetic book data instead of MockLobEnv.
  Reward signal: "synthetic monotonic uptick → forced long action →
  trade close pays > 0". Per `pearl_tests_must_prove_not_lock_observations`,
  asserts invariants (reward signal flows, sign correctness), not
  specific Q-value magnitudes.
- **Gate G5** local CUDA test (zero host memcpy in step_with_lobsim
  hot path).
- **Gate G7** local CUDA test (bull-market mean per-trade reward > 0).

### R7 — PER wiring (1 commit)
- Push every step's transition to the replay buffer.
- Sample `batch_size` indices per step for the off-policy Q update.
- Update priorities from per-sample TD-error magnitudes after Q
  backward.
- **Gate G6** local CUDA test (buffer grows, sample returns correct
  size).

### R8 — CLI binary + Argo template + dispatcher (1 commit)
- `crates/ml-alpha/examples/alpha_rl_train.rs` driving the rebuilt
  trainer. Uses `next_sequence_pair` for true `(s_t, s_{t+1})` windows.
  - Mandatory CLI args: `--mbp10-data-dir`. Plan-time reconciliation
    with `feedback_mbp10_mandatory`: ml-alpha's `MultiHorizonLoader`
    does not currently consume `--trades-data-dir` (the OFI features
    in `Mbp10RawInput` are derived from MBP-10 snapshot deltas, not
    from a separate trades stream). If the project chooses to wire a
    trades stream into ml-alpha's feature path, that's a separate
    plan; this rebuild ships with mbp10-only and flags the gap.
  - NaN abort: per-step `is_finite()` check on `l_total`; exit code 2
    on divergence. (**Gate G8**)
- `infra/k8s/argo/alpha-rl-template.yaml` + `scripts/argo-alpha-rl.sh`
  (port from flawed branch).
  - Per `feedback_default_to_l40s_pool`: defaults to
    `ci-training-l40s`, not H100.
  - Per `feedback_argo_template_must_apply`: top-of-output reminder
    + refuse-to-submit if template not applied since last edit.
  - Per `feedback_push_before_deploy`: verify `git push` succeeded
    before `argo submit`; hard-error if local HEAD ≠ origin HEAD.

### R9 — Pre-cluster validation + first cluster smoke (out-of-band)
- Run all R1..R8 local CUDA tests on a GPU host (the dev RTX 3050 Ti
  per CLAUDE.md or any L40S). Confirm green: G1, G2, G3, G4, G5, G6,
  G7, G8.
- `git push`. `kubectl apply -n foxhunt -f
  infra/k8s/argo/alpha-rl-template.yaml`.
- `./scripts/argo-alpha-rl.sh --n-steps 1000 --instrument-mode
  front-month` (5-min validation smoke). If green: scale to full 50k.
- For the G8 backtest gate: per `pearl_single_window_oos_is_not_oos`,
  run ≥3 walk-forward lob-sweep folds, report mean ± SD. A single
  pass is NOT G8 pass.

## Pre-cluster validation checklist (R9 prerequisite)

Per `feedback_smoke_validation_before_structural_priors`, validate the
wiring with neutral config BEFORE shipping production scale. The
following MUST be green on a local GPU host (RTX 3050 Ti is
sufficient for everything except wall-time benchmarks):

```bash
# Build clean
SQLX_OFFLINE=true cargo build --release --workspace 2>&1 | tail -10
SQLX_OFFLINE=true cargo build --release -p ml-alpha --example alpha_rl_train

# All local-CUDA tests green (R1..R8 gates)
SQLX_OFFLINE=true cargo test -p ml-alpha --test isv_bootstrap -- --ignored --nocapture            # G1
SQLX_OFFLINE=true cargo test -p ml-alpha --test reward_scale_hold_steady -- --ignored --nocapture # G2
SQLX_OFFLINE=true cargo test -p ml-alpha --test controllers_emit -- --ignored --nocapture         # G3
SQLX_OFFLINE=true cargo test -p ml-alpha --test target_soft_update -- --ignored --nocapture      # G4
SQLX_OFFLINE=true cargo test -p ml-alpha --test no_host_memcpy -- --ignored --nocapture          # G5
SQLX_OFFLINE=true cargo test -p ml-alpha --test per_wired -- --ignored --nocapture               # G6
SQLX_OFFLINE=true cargo test -p ml-backtesting --test reward_calibration -- --ignored --nocapture # G7
SQLX_OFFLINE=true cargo test -p ml-alpha --test nan_abort -- --ignored --nocapture               # G8

# Convergence-gate fixtures green
SQLX_OFFLINE=true cargo test -p ml-alpha --test dqn_reward_signal -- --ignored --nocapture
SQLX_OFFLINE=true cargo test -p ml-alpha --test ppo_reward_signal -- --ignored --nocapture
SQLX_OFFLINE=true cargo test -p ml-alpha --test integrated_trainer_smoke -- --ignored --nocapture
```

Only after ALL of these pass green: `git push` → `kubectl apply` →
`argo submit --n-steps 1000`.

Only after the 1000-step smoke completes Succeeded with finite losses
+ ISV slots in healthy ranges + > 0 trades closed: scale to 50k.

Only after the 50k run produces a checkpoint that achieves G6
calibration + survives 3 walk-forward lob-sweep folds: claim Phase H
G8 pass.

## What we keep from the flawed branch

Direct ports:
- `rl_reward_scale_controller.cu` kernel
- `order_match.cu` fee deduction + `upload_cost_per_lot_per_side` API
- `MultiHorizonLoader::next_sequence_pair` + `build_sequence_at`
  refactor

Architectural insights that survive the rebuild:
- True h_{t+1} Bellman target (F.4 close-out idea) — in the rebuild
  this falls out for free because action sampling is GPU-resident, so
  the next-state Q forward is just another kernel call with the next
  snapshot, no host roundtrip needed.
- V-bootstrap from V(s_{t+1}) — same.
- Toy rename (`dqn_toy.rs` → `dqn_reward_signal.rs`, etc.) — applied
  in R6b on top of the rewritten fixtures.

## What we discard

- `LobEnv` trait + `LobSimEnvAdapter` (kernel-driven path replaces
  the trait abstraction entirely)
- `MockLobEnv` "toy bandit" fixture (replaced by synthetic-book
  `LobSimCuda` fixture in R6b)
- Host-side action sampling, EMA tracking, advantage/return loops
- Host `mean_abs_pnl_ema: f32` field on `IntegratedTrainer` (moved
  to ISV slot, updated by `ema_update_on_done` kernel)
- Single-window OOS as a G8 pass criterion (replaced by ≥3
  walk-forward folds per `pearl_single_window_oos_is_not_oos`)

## Memory pearls honored

Catalog of pearls the rebuild explicitly applies (line numbers and
specific implementation pointers will be filled in at R-phase commit
time per `feedback_no_partial_refactor`):

- `feedback_cpu_is_read_only` → A1, A5, A6, G5
- `feedback_isv_for_adaptive_bounds` → A2, A3, R5
- `feedback_wire_everything_up` → A3, A4, A9, R1, R5, R7
- `feedback_no_partial_refactor` → every R-phase atomic; controllers
  AND their consumers land together
- `feedback_no_stubs` → A4 (target soft update), A9 (PER wired)
- `feedback_no_atomicadd` → all new kernels single-writer-per-block
  or `reduce_axis0`
- `feedback_no_cpu_test_fallbacks` → every gate's test uses GPU
  oracles only (analytical synthetic, property assertion, or
  cross-kernel)
- `feedback_no_htod_htoh_only_mapped_pinned` → A3 (no host memcpy
  bootstrap), mapped-pinned for any HEALTH_DIAG readback
- `feedback_always_per` → A9
- `feedback_mbp10_mandatory` → R8 (CLI requires `--mbp10-data-dir`;
  reconciles trades-data-dir gap explicitly)
- `feedback_default_to_l40s_pool` → R8 dispatcher
- `feedback_push_before_deploy` → R8 dispatcher
- `feedback_argo_template_must_apply` → R8 dispatcher
- `feedback_kill_runs_on_anomaly_quickly` → cluster smoke discipline
- `feedback_extending_existing_code_audits_for_existing_violations`
  → the lesson driving this entire rebuild
- `pearl_first_observation_bootstrap` → A3, R3 EMA bootstrap tests
- `pearl_wiener_alpha_floor_for_nonstationary` → all EMA controllers
- `pearl_wiener_optimal_adaptive_alpha` → A2 EMA design
- `pearl_thompson_for_distributional_action_selection` → A5 split
  (Thompson at rollout, argmax for Bellman)
- `pearl_action_pruning_falsified` → 9-action grid preserved
- `pearl_c51_thompson_closed_phase_e3_gap` → C51 + Thompson combo
  preserved
- `pearl_single_window_oos_is_not_oos` → cluster smoke discipline
  (≥3 walk-forward folds for G8)
- `pearl_q_spread_kill_criterion_misaligned_for_distributional_q` →
  cluster smoke kill criteria explicitly exclude Q_SPREAD
- `pearl_tests_must_prove_not_lock_observations` → every gate
  asserts invariants, not specific learned-checkpoint values
- `pearl_cold_path_no_exception_to_gpu_drives` → A1, A2, A5, A6 are
  all GPU-resident even for "cold path" diagnostic computations
- `pearl_no_host_branches_in_captured_graph` → A5 PRNG state lives
  on device; no host RNG advance inside any captured kernel chain
- `pearl_one_unbounded_signal_per_reward` → reward = scale × pnl_delta
  has scale bounded `[1e-3, 1e3]` (R5 reward_scale controller) and
  pnl_delta as the single unbounded multiplicand
- `pearl_symmetric_clamp_audit` → reward_scale clamp is bilateral
  `fmaxf(REWARD_SCALE_MIN, fminf(x, REWARD_SCALE_MAX))`
- `pearl_event_driven_reward_density_alignment` → per-trade reward
  fires on `done=true` (event density = trade close density)

## Calendar estimate

Roughly 2× the flawed Phase F+G calendar because the rebuild does the
work the flawed branch deferred or simplified.

- R1: 0.5 days (controller bootstrap launches + ISV slot extension)
- R2: 1 day (port F.3a + G fixes)
- R3: 1 day (EMA + advantage/return kernels + GPU-oracle tests)
- R4: 1.5 days (action sampling kernels + GPU-oracle tests)
- R5: 1 day (controller wiring + soft update)
- R6: 2 days (LobEnv removal + fixture rewrite)
- R7: 0.5 days (PER wiring)
- R8: 0.5 days (CLI + Argo)
- R9: 0.5 days local validation + 5-min smoke + 1.5h × 3 folds = ~5h cluster

Total ~8.5 dev days + ~6h cluster smoke wall.

## Execution discipline

- Per `feedback_no_concurrent_agents_shared_tree`: R-phases execute
  serially in this branch (`ml-alpha-phase-a`). If parallel execution
  is needed, use git worktrees.
- Per `feedback_fix_everything_aggressively`: no defer-to-later. Each
  R-phase ships its work complete.
- Per `feedback_extending_existing_code_audits_for_existing_violations`:
  when each R-phase touches pre-existing code, the audit comes FIRST.
  If the existing code already violates a rule, the R-phase's scope
  expands to include the fix, not silently extends the violation.

End of plan.
