foxhunt/docs/dqn-gpu-hot-path-audit.md

# DQN GPU Hot-Path Audit

**Invariant 3 enforcement.** Every cross-boundary call (DtoH, HtoD, memcpy between host and device) in the DQN code path is classified below. No hot-path `memcpy` DtoH/HtoD is allowed — zero tolerance. Pinned + device-mapped memory is the ONLY permitted mechanism for per-step host-visible data.

**Hot path definition:** every function called from `fused_training.rs::run_full_step` or its descendants that executes once per training step (up to 178 times per epoch at batch size 8192), including kernel invocations inside the captured CUDA graph (`adam_grad_child`, `forward_child`, `aux_grad_child`, `aux_child`, `maintenance_child`) and host-side orchestration between them.

**Classification:**
- `OK-pinned` — uses the zero-copy pinned + device-mapped pattern (`cuMemAllocHost_v2` + `cuMemHostGetDevicePointer_v2`); allowed on hot path.
- `MIGRATE` — was a plain `memcpy` but has been converted to pinned in this commit.
- `COLD-PATH` — acceptable placement (constructor init, checkpoint save/load, fold transition, epoch-boundary); annotated with reason.

| File:line | Call signature | Path class | Annotation | Status |
|---|---|---|---|---|
| `gpu_dqn_trainer.rs:6586` | `malloc_host` → `total_loss_pinned` | OK-pinned | Constructor: `cuMemAllocHost+GetDevicePointer`; GPU writes via dev ptr, CPU reads via `readback_training_scalars()` | OK |
| `gpu_dqn_trainer.rs:6598` | `malloc_host` → `mse_loss_pinned` | OK-pinned | Constructor: same pattern as `total_loss_pinned` | OK |
| `gpu_dqn_trainer.rs:6611` | `malloc_host` → `q_divergence_pinned` | OK-pinned | Constructor: same pattern | OK |
| `gpu_dqn_trainer.rs:6624` | `malloc_host` → `iqn_readiness_pinned` | OK-pinned | Constructor: same pattern | OK |
| `gpu_dqn_trainer.rs:6662` | `malloc_host` → `grad_norm_pinned` | OK-pinned | Constructor: same pattern | OK |
| `gpu_dqn_trainer.rs:6707` | `malloc_host` (plain, not DEVICEMAP) → readback scratch | COLD-PATH | Constructor: plain pinned for per-epoch `flush_readback()`; not written by GPU kernel directly | OK |
| `gpu_dqn_trainer.rs:6788` | `malloc_host` → monitoring summary pinned | OK-pinned | Constructor: monitoring kernel writes via dev ptr each step | OK |
| `gpu_dqn_trainer.rs:6895` | `malloc_host` → `eval_v_range_pinned` | OK-pinned | Constructor: GPU writes; host reads only at fold/epoch boundary via `eval_v_range()` | OK |
| `gpu_dqn_trainer.rs:6911` | `malloc_host` → `nan_flag_pinned` | OK-pinned | Constructor: GPU writes on NaN; host reads only in fault path (`read_nan_flags`) | OK |
| `gpu_dqn_trainer.rs:6929` | `malloc_host` → ISV `isv_signals_pinned` | OK-pinned | Constructor: GPU writes ISV bus each step; host reads via `read_isv_health_and_regime()` in `compute_adaptive_budgets` | OK |
| `gpu_dqn_trainer.rs:6949` | `malloc_host` → `lr_pinned` | OK-pinned | Constructor: CPU writes LR; GPU reads via dev ptr | OK |
| `gpu_dqn_trainer.rs:6968` | `malloc_host` → `adaptive_clip_pinned` | OK-pinned | Constructor: CPU writes clip; GPU reads via dev ptr | OK |
| `gpu_dqn_trainer.rs:6988` | `malloc_host` → `popart_pinned` | OK-pinned | Constructor: GPU writes PopArt stats; host reads at epoch boundary | OK |
| `gpu_dqn_trainer.rs:7099` | `malloc_host` → `q_stats_pinned` | OK-pinned | Constructor: GPU writes Q stats; host reads via `reduce_current_q_stats()` at epoch boundary | OK |
| `gpu_dqn_trainer.rs:7127` | `malloc_host` → `q_readback_pinned` | OK-pinned | Constructor: GPU writes per-step Q readback; host reads via `last_readback_scalars()` in training guard | OK |
| `gpu_dqn_trainer.rs:7206` | `cuMemAllocHost_v2` → `var_ema` pinned (ISV subsystem) | OK-pinned | Constructor: ISV variance EMA — GPU writes each step; host reads via ISV bus | OK |
| `gpu_dqn_trainer.rs:7243` | `cuMemAllocHost_v2` → `regime_util` pinned (ISV subsystem) | OK-pinned | Constructor: ISV regime utilization — GPU writes each step | OK |
| `gpu_dqn_trainer.rs:7289` | `malloc_host` → `popart_var_pinned` | OK-pinned | Constructor: GPU writes PopArt variance; host reads at epoch boundary | OK |
| `gpu_dqn_trainer.rs:7312` | `cuMemcpyHtoDAsync_v2` (constructor) | COLD-PATH | Constructor: one-shot HtoD to seed deterministic-mask buffer; never replayed | OK |
| `gpu_dqn_trainer.rs:7878` | `cuMemAllocHost_v2` → `q_mean_scratch` | OK-pinned | Constructor: GPU scratch for Q-mean reduction kernel | OK |
| `gpu_dqn_trainer.rs:7898` | `cuMemAllocHost_v2` → `q_mean_ema` | OK-pinned | Constructor: Q-mean EMA — GPU writes, host reads at epoch boundary | OK |
| `gpu_dqn_trainer.rs:7919` | `cuMemAllocHost_v2` → `td_error_scratch` | OK-pinned | Constructor: TD-error scratch for reduce kernel | OK |
| `gpu_dqn_trainer.rs:7939` | `cuMemAllocHost_v2` → `q_var_scratch` | OK-pinned | Constructor: Q-variance scratch for reduce kernel | OK |
| `gpu_dqn_trainer.rs:2300` | `stream.memcpy_htod` → `atom_positions_buf` | COLD-PATH | `set_atom_positions()`: called at fold boundary to reload C51 atom grid — not on hot path | OK |
| `gpu_dqn_trainer.rs:2606` | `stream.memcpy_dtoh` → grad buf readback | COLD-PATH | `per_branch_grad_norms()`: per-epoch call only; drains grad buffer for logging | OK |
| `gpu_dqn_trainer.rs:2925` | `stream.memcpy_dtoh` (online params) | COLD-PATH | `per_branch_target_drift()`: per-epoch drift measurement | OK |
| `gpu_dqn_trainer.rs:2928` | `stream.memcpy_dtoh` (target params) | COLD-PATH | `per_branch_target_drift()`: per-epoch drift measurement | OK |
| `gpu_dqn_trainer.rs:2991` | `stream.memcpy_dtoh` → weight inspection | COLD-PATH | `per_branch_vsn_mean()`: per-epoch VSN mean read | OK |
| `gpu_dqn_trainer.rs:3072` | `stream.memcpy_dtoh` → Q-out inspection | COLD-PATH | `compute_q_stats()`: per-epoch Q-stats; not called per-step | OK |
| `gpu_dqn_trainer.rs:7541` | `stream.memcpy_htod` → descriptor buffer | COLD-PATH | Constructor: one-shot HtoD for GEMM descriptor upload | OK |
| `gpu_dqn_trainer.rs:7680` | `stream.memcpy_htod` → stochastic depth scale | COLD-PATH | Constructor: one-shot HtoD for stochastic depth kernel seed | OK |
| `gpu_dqn_trainer.rs:7686` | `stream.memcpy_htod` → stochastic depth RNG | COLD-PATH | Constructor: one-shot HtoD for stochastic depth RNG | OK |
| `gpu_dqn_trainer.rs` (removed) | `cuMemcpyDtoHAsync_v2` in `run_causal_intervention_unconditional` | **MIGRATED** | **Fix 1**: removed dead copy — result `causal_mean_scratch` was never consumed by any caller; result stays on GPU | FIXED |
| `gpu_iqn_head.rs` (removed) | `clone_to_device_f32_via_pinned` → `cos_features` / `online_taus` / `target_taus` (3 CudaSlice fields) | **MIGRATED** | **SP6 fix**: converted to `[MappedF32Buffer; 4]` per-branch arrays (`online_taus_per_branch`, `target_taus_per_branch`, `cos_features_per_branch`). Constructor writes via `write_from_slice` (host_ptr direct); IQN kernels read via `dev_ptr`. `mem::swap` activate/deactivate replaced by `active_branch_idx` selection. `upload_f32_via_pinned` (which did an intermediate DtoD inside graph capture → `STREAM_CAPTURE_INVALIDATED`) fully eliminated. | FIXED |
| `gpu_iqn_head.rs:498` | `malloc_host` → `total_loss_pinned` | OK-pinned | Constructor: GPU writes IQN loss; host reads via `read_total_loss()` | OK |
| `gpu_iqn_head.rs:534` | `malloc_host` → `t_pinned` | OK-pinned | Constructor: Adam step counter — CPU increments, GPU reads via dev ptr | OK |
| `gpu_iqn_head.rs:548` | `malloc_host` → `tau_pinned` | OK-pinned | Constructor (Fix 3): tau scalar — CPU writes, GPU reads via `tau_dev_ptr`; replaces `CudaSlice<f32>` + `cuMemcpyHtoDAsync_v2` | OK |
| `gpu_iqn_head.rs:807` | `cuMemcpyDtoDAsync_v2` (DtoD) | OK-pinned | Device-to-device copy inside graph-captured target-H forward cache; no host crossing | OK |
| `gpu_iqn_head.rs:991` | `cuMemcpyDtoDAsync_v2` (DtoD) | OK-pinned | Device-to-device copy for cached target states; no host crossing | OK |
| `gpu_iqn_head.rs:1564` | `stream.clone_dtoh` → `total_loss` | COLD-PATH | `read_loss()`: synchronous fallback for epoch-end logging only; `read_total_loss()` (pinned) is the hot-path equivalent | OK |
| `gpu_iqn_head.rs:1602` | `cuMemcpyDtoH_v2` (sync) in `compute_iqr` | **MIGRATED** | **Fix 2A**: `compute_iqr()` removed from `submit_aux_ops`; now called at epoch boundary via `refresh_iqn_iqr()` only; sync DtoH cannot be captured in CUDA graph — was silently a no-op on replayed steps | FIXED |
| `gpu_iqn_head.rs:1622` | `cuMemcpyHtoDAsync_v2` in `compute_iqr` | **MIGRATED** | **Fix 2B**: same relocation — async HtoD with stale IQR data was being replayed; now runs once at epoch boundary | FIXED |
| `gpu_iqn_head.rs` (removed) | `cuMemcpyHtoDAsync_v2` in `target_ema_update` for `tau_buf` | **MIGRATED** | **Fix 3**: replaced `CudaSlice<f32>` + async copy with pinned+device-mapped page; CPU writes `*tau_pinned = tau`; kernel reads via `tau_dev_ptr` — zero PCIe overhead | FIXED |
| `gpu_iqn_head.rs:2058` | `stream.clone_htod` → `params_buf` constructor | COLD-PATH | `init_iqn_head_from_weights()`: one-shot weight upload at construction/load; not on hot path | OK |
| `gpu_iqn_head.rs:2107` | `stream.clone_dtoh` + `stream.clone_htod` in `clone_cuda_slice` | COLD-PATH | `clone_cuda_slice()`: called during checkpoint save/load only | OK |
| `fused_training.rs:476` | `stream.memcpy_htod` → `src_gpu` (PER source indices) | COLD-PATH | `prepare_per_batch()`: per-epoch PER batch preparation — not per-step | OK |
| `fused_training.rs:482` | `stream.memcpy_htod` → `ones_gpu` | COLD-PATH | `prepare_per_batch()`: per-epoch importance weights init | OK |
| `fused_training.rs:2914` | `stream.clone_dtoh` → state inspection | COLD-PATH | `eval_state_debug()`: debug helper, no hot-path caller | OK |
| `fused_training.rs:2928` | `stream.clone_dtoh` → param inspection | COLD-PATH | `eval_state_debug()`: debug helper | OK |
| `fused_training.rs:2955` | `cuMemcpyDtoH_v2` (sync) | COLD-PATH | `eval_v_range()`: called only at fold boundary / epoch-0 init; explicit sync is documented and acceptable at boundary | OK |
| `training_loop.rs:220` | `collector.stream().memcpy_dtoh` → PER scores | COLD-PATH | Pre-epoch PER priority init — runs once before the step loop starts | OK |
| `training_loop.rs:409` | `stream.memcpy_htod` → `episode_starts_buf` | COLD-PATH | Per-epoch experience setup — before `run_training_steps_slices` begins | OK |
| `training_loop.rs:1003` | `clone_htod_f32` → targets device buffer | COLD-PATH | Data loading: DBN targets uploaded before step loop | OK |
| `training_loop.rs:1007` | `clone_htod_f32` → features device buffer | COLD-PATH | Data loading: feature matrix uploaded before step loop | OK |
| `tau_update_kernel.cu` (Plan 1 Task 13) | single-thread kernel: reads ISV[39,40,12]; writes ISV[42] | COLD-PATH | Per-epoch boundary call from `training_loop.rs`; 1-thread, 1-block; zero hot-path overhead | OK |
| `epsilon_update_kernel.cu` (Plan 1 Task 14) | single-thread kernel: reads ISV[39,40,12]; writes ISV[41] | COLD-PATH | Per-epoch boundary call from `training_loop.rs`; 1-thread, 1-block; zero hot-path overhead | OK |
| `gamma_update_kernel.cu` (Plan 1 Task 10) | single-thread kernel: reads ISV[12]; writes ISV[43] | COLD-PATH | Per-epoch boundary call from `training_loop.rs`; 1-thread, 1-block; ISV[43] read by fill_gamma_buf on hot path via read_isv_signal_at (pinned, zero-copy) | OK |
| `kelly_cap_update_kernel.cu` (Plan 1 Task 11) | single-thread kernel: reads ISV[12] + portfolio_states[n_envs, PS_STRIDE]; aggregates mean half-Kelly × health-safety; writes ISV[44] | COLD-PATH | Per-epoch boundary call from `training_loop.rs`; 1-thread, 1-block; portfolio_states dev ptr passed from GpuExperienceCollector without CPU roundtrip | OK |
| `atoms_update_kernel.cu` (Plan 1 Task 9) | 4-block kernel (one per branch): reads ISV[23..31) + spacing_raw params[52..56); writes atom_positions_buf[4 * num_atoms] via softmax + cumsum | COLD-PATH | Per-epoch boundary + SGD step (every 50 steps); grid=(4,1,1), block=(256,1,1); replaces CPU loop that launched adaptive_atom_positions 4× separately | OK |
| `gpu_portfolio.rs:174` | `actions_buf` upload (per simulate_batch_gpu) | **MIGRATED** | Promoted `actions_buf` from `CudaSlice<i32>` to `MappedI32Buffer(MAX_BATCH_SIZE)` allocated once at constructor. Per-call host writes go through `host_ptr`; kernel arg switched to `dev_ptr` u64. Zero HtoD copy on the simulate hot path. | FIXED |
| `gpu_portfolio.rs:122,296` | `portfolio_state_buf` init/reset | **MIGRATED** | env_step kernel writes back to `portfolio_state_buf`, so the destination must remain a `CudaSlice<f32>`. Init/reset now stage via a temporary `MappedF32Buffer` + `memcpy_dtod_async` (new helper `upload_portfolio_state`). | FIXED |
| `decision_transformer.rs:913` | `scratch.adam_t` upload per `train_step_gpu` | **MIGRATED** | HOT path: 4-byte step counter was uploaded via `stream.memcpy_htod` every backward pass. Promoted `DtScratch.adam_t` from `CudaSlice<i32>[1]` to `MappedI32Buffer(1)`. Per-step host write goes through `host_ptr`; Adam kernel reads via `dev_ptr`. | FIXED |
| `decision_transformer.rs:478` | DT params init upload (constructor) | **MIGRATED** | params is mutated by Adam, so destination must remain `CudaSlice<f32>`. Xavier-init upload now stages via `MappedF32Buffer` + `memcpy_dtod_async` through new helper `upload_host_to_cuda_f32`. | FIXED |
| `decision_transformer.rs:501` | DT wd_mask init (constructor) | **MIGRATED** | wd_mask is read-only by GPU. Promoted from `CudaSlice<f32>` to `MappedF32Buffer`. CPU writes 1.0s through `host_ptr`; kernel reads via `dev_ptr`. | FIXED |
| `decision_transformer.rs:1105` | trajectory build episode_starts | **MIGRATED** | One-time per pre-training data build (cold path). Replaced `clone_htod` with `MappedI32Buffer` + direct `host_ptr` write; kernel reads via `dev_ptr`. | FIXED |
| `decision_transformer.rs:1147` | trajectory build ep_rewards | **MIGRATED** | Same trajectory builder cold path. Replaced `htod_f32` with `MappedF32Buffer`; CPU writes via `host_ptr`, return_to_go kernel reads via `dev_ptr`. | FIXED |
| `gpu_experience_collector.rs:2992` | hindsight `bar_indices` upload (per `collect_experiences_gpu`) | **MIGRATED** | HOT path: alloc + memcpy_htod ran every collect when hindsight active. Added persistent `bar_indices_pinned: MappedI32Buffer(alloc_episodes * alloc_timesteps * 2)` allocated at constructor. Per-call host writes go through `host_ptr`; kernel arg switched to `dev_ptr` u64. | FIXED |
| `gpu_experience_collector.rs:3186` | feature_mask upload (per-collect at t==0) | **MIGRATED** | HOT path: was alloc + memcpy_htod every epoch. Promoted `feature_mask_buf` from `Option<CudaSlice<f32>>` to `Option<MappedF32Buffer>`. Reallocates only when mask size changes (still a cold-path event); CPU writes via `host_ptr`, state_gather reads via `dev_ptr`. | FIXED |
| `gpu_experience_collector.rs:3097` | episode_starts upload (per-collect) | **MIGRATED** | episode_starts_buf is GPU-mutated by `domain_rand_episode_starts`, so the destination must remain a CudaSlice. Stage via `MappedI32Buffer` + `memcpy_dtod_async` through `upload_host_to_cuda_i32_via_pinned`. | FIXED |
| `gpu_experience_collector.rs:2009` | `upload_expert_actions` (warm) | **MIGRATED** | Promoted `expert_actions_gpu` from `Option<CudaSlice<i32>>` to `Option<MappedI32Buffer>`. Direct host_ptr write replaces the alloc + memcpy_htod pair. | FIXED |
| `gpu_experience_collector.rs:3816` | `update_per_sample_support` (per-epoch) | **MIGRATED** | per_sample_support_buf is read-only by GPU; promoted from `CudaSlice<f32>` to `MappedF32Buffer`. Compute_expected_q + quantile_q_select now receive `dev_ptr` u64 kernel args. | FIXED |
| `gpu_experience_collector.rs:1076,3848` | portfolio_states init/reset (cold) | **MIGRATED** | portfolio_states is GPU-mutated by env_step. Init/reset stage via `MappedF32Buffer` + `memcpy_dtod_async` through `upload_host_to_cuda_f32_via_pinned`. | FIXED |
| `gpu_experience_collector.rs:1093` | epoch_state init (cold) | **MIGRATED** | epoch_state is GPU-mutated. Replaced `clone_htod_f32` with explicit `alloc_zeros` + `upload_host_to_cuda_f32_via_pinned`. | FIXED |
| `gpu_experience_collector.rs:1341` | saboteur_base init (cold) | **MIGRATED** | saboteur_base is GPU-mutated by `saboteur_select`. Replaced `memcpy_htod` with `upload_host_to_cuda_f32_via_pinned`. | FIXED |
| `gpu_experience_collector.rs:2784` | trade_stats_buf zero (per-epoch) | **MIGRATED** | The htod_f32 of an all-zeros vector was needlessly going through PCIe. Replaced with `stream.memset_zeros` — fully GPU-side. | FIXED |

## Summary

| Class | Count |
|---|---|
| OK-pinned | 31 |
| COLD-PATH | 20 |
| MIGRATED (fixed in this commit) | 4 |
| Remaining MIGRATE (unfixed) | 0 |

**All MIGRATE sites resolved.** Zero hot-path `memcpy` calls remain in the DQN training loop.

## Fixes Applied (this commit)

### Fix 1 — Remove dead async DtoH in `run_causal_intervention_unconditional`
`gpu_dqn_trainer.rs`: `cuMemcpyDtoHAsync_v2` from `causal_mean_scratch_ptr` to `readback_pinned[10]` was removed. The only reader (`run_causal_intervention`, which has no callers) was never invoked, making this copy dead code. The result now stays on the GPU device buffer where the kernel leaves it.

### Fix 2 — Relocate `compute_iqr()` from hot path to epoch boundary
`fused_training.rs / training_loop.rs`: `iqn.compute_iqr()` was being called inside `submit_aux_ops`, which is captured in the CUDA graph (`aux_child`). The sync `cuMemcpyDtoH_v2` inside `compute_iqr` cannot be captured in a CUDA graph — it ran during capture only and was silently skipped on every replayed step. The subsequent `cuMemcpyHtoDAsync_v2` (uploading IQR back) was therefore replaying stale data. Moved to `refresh_iqn_iqr()` called at epoch boundary in `process_epoch_boundary`.

### Fix 3 — Convert `tau_buf` from device VRAM to pinned+device-mapped
`gpu_iqn_head.rs`: `tau_buf: CudaSlice<f32>` + `tau_host: f32` + `cuMemcpyHtoDAsync_v2` replaced by `tau_pinned: *mut f32` + `tau_dev_ptr: u64` allocated via `cuMemAllocHost_v2(DEVICEMAP)` + `cuMemHostGetDevicePointer_v2`. CPU writes `*tau_pinned = tau` directly; the EMA kernel reads via `tau_dev_ptr` with no PCIe copy. Drop impl updated to free `tau_pinned`.

### Fix 4 (perf) — Async-eval pipeline: split `launch_metrics_and_download` (2026-04-28)
`gpu_backtest_evaluator.rs` + `trainer/metrics.rs` + `trainer/training_loop.rs`: the prior merge `3c0d26292` (async-diag — eval on dedicated stream + spawn_blocking checkpoint, building on `d9cb14f1b` + `673b04a8d`) wired the **GPU side** of cross-stream pipelining correctly (training stream `cuda_stream`, eval stream `validation_stream`, `cuStreamWaitEvent` barrier on `train_done_event`) but left a host-side `stream.synchronize()` inside `launch_metrics_and_download` that blocked the CPU thread for the FULL eval drain (~25-30s/epoch on L40S). The host thread is the same one that submits the next epoch's training kernels via `run_full_step` — so until eval drained, training submission was gated on it, and the dedicated `validation_stream` provided no actual wall-clock benefit.

Split into two record/consume halves at the buffer-readback layer:

* `launch_metrics_and_record_event(&mut self) -> Result<(), MLError>` — submits the metrics kernel + DtoD-async copies of `actions_history_buf` / `intent_mag_buf` / `picked_action_history_buf` into mapped-pinned mirrors (`actions_history_pinned: MappedI32Buffer` allocated in `new()`, the other two lazy-allocated alongside their device buffers in `ensure_action_select_ready`). Records `eval_done_event` (lazy-created on first launch via `stream.context().new_event(None)`, re-recorded each epoch via `event.record(stream)`). **Returns immediately** — no host wait.
* `consume_metrics_after_event(&self) -> Result<Vec<WindowMetrics>, MLError>` — calls `event.synchronize()` (the SOLE host-side wait per epoch boundary), then `read_volatile`s the mapped-pinned `metrics_buf` for the per-window metrics. After the event syncs, the four action-distribution helpers (`read_eval_action_distribution_per_*`, `read_eval_intent_magnitude_distribution`, `read_chunked_actions_direction_distribution`) read directly from the now-coherent mapped-pinned mirrors — no `memcpy_dtoh` per call (the prior implementation allocated a fresh `Vec<i32>` of size `n_windows * max_len` and did a synchronous DtoH each call, which violates `feedback_no_htod_htoh_only_mapped_pinned.md` AND added 4 extra host-side stalls beyond the metrics one). All four mirrors land in the same event-sync barrier as the metrics readback.

Caller migration (`feedback_no_partial_refactor.md`):
* `evaluate_dqn_graphed` (existing public API, used by hyperopt + smoke test) becomes a thin synchronous wrapper: launch_async → consume. Public ABI unchanged.
* New `evaluate_dqn_graphed_async` — launch-only sibling for the pipelined path. Returns `()`.
* `evaluate` / `evaluate_ppo` / `evaluate_supervised` migrated inline to launch+consume — no caller pipelines them, so no behavioural change.
* `compute_validation_loss` removed; replaced by `launch_validation_loss` (returns `()`) + `consume_validation_loss` (returns `f64` and emits the `HEALTH_DIAG val [...]` / `val_dir_dist` / `val_picked_dir_dist` lines).
* `training_loop.rs`'s validation reporting block now: epoch-start consumes any pending launch into `cached_async_val_loss`, epoch-end reports the cached value and submits a fresh launch with `pending_val_loss = Some(0.0)` as the "launched" sentinel. Bootstrap (epoch 0 / no validation_stream) does sync launch+consume. After the for-epoch loop terminates, a final consume drains the LAST epoch's launch so smoke tests / hyperopt that read `last_eval_direction_dist()` see the most recent epoch's data, not the second-to-last.

Mathematical identity preserved: every consumed val_sharpe is bit-identical to what the prior `compute_validation_loss` would have produced for the same epoch — the only change is *when* the host parses it (one epoch later, which matches the existing `cached_async_val_loss` lag semantics; HEALTH_DIAG `val [...]` emit moves with the consume). Expected wall-clock saving: ~25-30s/epoch on L40S.

Out-of-scope (remaining host-side stream syncs in eval, not part of this fix): `gpu_backtest_evaluator.rs` lines ~1596 + ~1863 — periodic `self.stream.synchronize()` calls every 10 chunks inside `submit_dqn_step_loop_cublas` for kernel-error detection. They block the host CPU thread but only for the chunked rollout duration of ~5 calls × per-chunk-residency-time per epoch (each catches deferred CUDA errors within 10 chunks of origin). Smaller in aggregate than the `launch_metrics_and_download` final wait this fix removed; future work to convert to async error-polling (e.g., `eval_done_event.is_complete()` polling from training submissions). Not adjacent to the perf-fix scope.

### Fix 5 — Add mapped-pinned staging helpers in `mapped_pinned.rs`
`mapped_pinned.rs`: added `clone_to_device_{f32,i32}_via_pinned` and `upload_{f32,i32}_via_pinned` helpers that allocate a temporary `MappedXxxBuffer` (cuMemHostAlloc DEVICEMAP), write the payload via `host_ptr`, then async DtoD into a target `CudaSlice<T>`. Used to migrate cold-path constructor uploads off explicit `cudaMemcpy` HtoD per `feedback_no_htod_htoh_only_mapped_pinned.md` while preserving struct field types (e.g. `portfolio_buf`, weight buffers consumed by Adam) where the kernel mutates the buffer every step and device-resident memory is correct.

### Fix 6 — Migrate `cuda_pipeline/mod.rs` callers to mapped-pinned
`cuda_pipeline/mod.rs`: 7 in-file callers of `clone_htod_f32` in `DqnGpuData::upload_slices`, `DqnGpuData::upload_ofi`, `DqnGpuData::build_batch_states` (portfolio), `DqnGpuData::htod_copy_into`, and `PpoGpuData::upload` rewritten to use `mapped_pinned::clone_to_device_f32_via_pinned`. The `htod_f32` / `clone_htod_f32` helper bodies at lines 129-145 remain in place because other crates are still in mid-migration; they will be deleted once all consumers migrate.

### Fix 7 — Migrate `gpu_walk_forward.rs` to mapped-pinned (3 sites)
`gpu_walk_forward.rs`: 3 `clone_htod_f32` calls in the walk-forward GPU data upload (features, targets, OFI) rewritten to `mapped_pinned::clone_to_device_f32_via_pinned`.

### Fix 8 — Migrate PPO trainer + hyperopt adapters to mapped-pinned (6 sites)
`trainers/ppo.rs` (2 sites: features, targets in `set_raw_market_data`), `hyperopt/adapters/ppo.rs` (3 sites: features, targets in upload path; action_indices in backtest forward closure), `hyperopt/adapters/mamba2.rs` (1 site: `gpu_to_stream`). All `clone_htod_f32` and bare `stream.clone_htod` calls rewritten to `mapped_pinned::clone_to_device_{f32,i32}_via_pinned`.

### Fix 9 — Migrate weight-upload sites in attention/tlob/iql/weights (4 sites)
`gpu_weights.rs` (1 site: `RmsNormWeightSet::ones` constructor), `gpu_attention.rs` (1 site: attention params constructor), `gpu_tlob.rs` (1 site: TLOB params constructor — `mod tests` blocks intentionally untouched per scope), `gpu_iql_trainer.rs` (2 sites: per_sample_support seed, IQL params init). All `htod_f32` calls rewritten to `mapped_pinned::upload_f32_via_pinned`; the gpu_weights.rs ones-init uses `clone_to_device_f32_via_pinned` since the destination is freshly allocated.

### Fix 10 — Migrate `gpu_iqn_head.rs` constructors + DtoD-only target clone (5 sites)
`gpu_iqn_head.rs`: 4 `stream.clone_htod` calls (cos_features precompute, online_taus + target_taus tile broadcast, init_iqn_xavier_weights upload) rewritten to `mapped_pinned::clone_to_device_f32_via_pinned`. Additionally the helper `clone_cuda_slice` was replaced — it previously did device→host (`clone_dtoh`) → host→device (`clone_htod`) round-trip, double-violating both the no-DtoH and no-HtoD rules. Now uses `clone_cuda_slice_f32` (single async DtoD), the only callsite being the target-network seed at constructor time (`target_params = clone_cuda_slice(&online_params)`).

### Fix 11 — Migrate `gpu_her.rs` to mapped-pinned + add MappedU32Buffer (3 sites)
`gpu_her.rs`: 1 cold-path constructor `stream.memcpy_htod` (rng_states u32 init) and 2 warm-path `memcpy_htod` (source_indices, donor_indices in `relabel_batch`) rewritten to `mapped_pinned::upload_{u32,i32}_via_pinned`. The warm sites re-allocate pinned staging per relabel call — acceptable on this cold path (relabel runs once per epoch). New `MappedU32Buffer` type added to `mapped_pinned.rs` mirroring `MappedI32Buffer`; carries a manual `Debug` impl so the workspace warning count stays at 13.

### Fix 12 — Migrate `gpu_backtest_evaluator.rs` to mapped-pinned (5 sites)
`gpu_backtest_evaluator.rs`: 5 HtoD sites in `GpuBacktestEvaluator::new` + `reset_evaluation_state` rewritten to mapped-pinned staging. Sites: prices_buf (f32), features_buf (f32 — was already going through `clone_htod_f32` helper), window_lens_buf (i32), portfolio_buf init (f32), and the per-evaluation reset of portfolio_buf. The fields stay typed `CudaSlice<T>` because the kernel mutates `portfolio_buf` each step (must remain device-resident); the read-only fields could become `MappedXxxBuffer` but keeping them as `CudaSlice` preserves the kernel-launch contract `.arg(&self.field)` already used at the many launch sites in the file.

### Fix 13 — Migrate `gpu_ppo_collector.rs` to mapped-pinned (6 sites + persistent staging)
`gpu_ppo_collector.rs`: all 6 HtoD sites migrated.
- Constructor (cold): `portfolio_states` init (htod_f32) → `upload_f32_via_pinned`; `rng_states` init (memcpy_htod) → `upload_u32_via_pinned`.
- `collect_experiences` (warm but per-collect): `episode_starts_buf` (memcpy_htod) → `upload_i32_via_pinned`; `barrier_config` (htod_f32) → `upload_f32_via_pinned`.
- `reset_episodes` (warm, per-epoch): replaced the `Vec<f32>` / `Vec<u32>` host staging fields with persistent `MappedF32Buffer` / `MappedU32Buffer` allocated once at construction. Resets fill via `host_slice_mut()` (zero-allocation, zero-HtoD); a single async DtoD seeds the persistent device-resident `portfolio_states` / `rng_states` (which the kernel mutates each step). Adds `host_slice_mut()` accessor on both mapped-pinned types.

### Fix 17 — Delete orphan `htod_f32` / `clone_htod_f32` helpers from `cuda_pipeline/mod.rs` (2026-04-28)
After Fix 1..16 migrated all 80+ production callers off `super::htod_f32` and `super::clone_htod_f32`, the helper bodies in `cuda_pipeline/mod.rs:129-145` were unused outside `#[cfg(test)]` modules. Deleted both function definitions per `feedback_no_legacy_aliases.md` (no deprecated wrappers).

The two surviving test-block callers in `gpu_tlob.rs::tests` (lines 1017 and 1132 — `super::super::htod_f32(&stream, &host_states, …)` and `super::super::htod_f32(&stream, &host_d_concat, …)`) were also migrated to `mapped_pinned::upload_f32_via_pinned` in the same commit per `feedback_no_partial_refactor.md` (when a contract is deleted, every consumer migrates together — including tests). The test-only callers in `signal_adapter.rs::tests` (5 sites), `gpu_action_selector.rs::tests` (1), and `cuda_pipeline/mod.rs::tests` (1) all use bare `stream.memcpy_htod` / `stream.memcpy_stod` against the cudarc handle directly, not the deleted helpers — no change needed.

A doc-block was added at the deletion site in `mod.rs` recording when and why the helpers were removed and pointing future readers at `mapped_pinned::clone_to_device_f32_via_pinned` and `mapped_pinned::upload_f32_via_pinned` as the canonical replacements.

Final state of the HtoD migration sequence:
- `grep -rEn "stream\.memcpy_htod|memcpy_stod|stream\.htod_copy|htod_async" crates/ml/src/`: production count **0**, test count 7 (5 in `signal_adapter.rs::tests`, 1 in `gpu_action_selector.rs::tests`, 1 in `cuda_pipeline/mod.rs::tests`).
- `grep -rn "htod_f32\|clone_htod_f32" crates/ml/src/`: outside test blocks **0** (only doc-comment text references remain). Helper definitions removed from `mod.rs`.

### Fix 16 — Migrate `training_loop.rs` raw features/targets uploads (2 sites, 2026-04-28)
`trainers/dqn/trainer/training_loop.rs:1250` (targets_raw) and `:1254` (features_raw): both `crate::cuda_pipeline::clone_htod_f32` callsites in the data-load helper rewritten to `mapped_pinned::clone_to_device_f32_via_pinned`. These run once per fold immediately before the step loop begins (the `_raw` suffix indicates the un-normalized arrays kept on-GPU for the eval-time critic). The `.map_err(|e| anyhow::anyhow!("…raw upload: {e}"))` wrapping is preserved verbatim — the helper already returns `Result<_, String>` which feeds `anyhow::anyhow!` directly, so the change is a 1:1 path swap.

### Fix 15 — Migrate `gpu_experience_collector.rs::upload_ofi_features` (1 site, 2026-04-28)
`gpu_experience_collector.rs:4258`: warm-path `super::clone_htod_f32` upload of OFI features rewritten to `mapped_pinned::clone_to_device_f32_via_pinned`. Called from training_loop's data-load phase before each fold's training run, so the OFI tensor (4M bars × 32 dims = ~512 MB) now stages through mapped-pinned + DtoD just like all other large data uploads. Error-type shift handled with `.map_err(|e| MLError::ModelError(format!("upload_ofi_features: {e}")))?` since the helper returns `Result<_, String>` and the function returns `Result<_, MLError>`.

### Fix 14 — Migrate `gpu_dqn_trainer.rs` residual COLD ctor sites (8 sites, 2026-04-28)
`gpu_dqn_trainer.rs`: residual COLD ctor sites missed by the original Fix-1..13 audit are now migrated off explicit `super::htod_f32` per `feedback_no_htod_htoh_only_mapped_pinned.md`.
- `:9922` `sel_clip_buf` 1-element init — selectivity gate clip-norm seed (1.0).
- `:10075-10078` spectral-norm `init_u_s1` / `init_v_s1` / `init_u_s2` / `init_v_s2` (4 separate uploads of trunk-encoder spectral u/v vectors).
- `:10095-10096` `alloc_spec_pair!` macro body — both u/v init uploads inside the macro now go through `upload_f32_via_pinned`. The macro's `$lbl_u` / `$lbl_v` are reused in the error-message format so per-pair failures (`spec_u_v1`, `spec_v_v1`, `spec_u_a1`, …) stay diagnostically distinct.
- `:11276` `graph_params_host` (60 floats: cross-branch graph message passing weights, 4 edges × 15 params each, near-identity W_msg init).
- `:11330` `denoise_params_host` (1800 floats: 2-step diffusion Q-refinement Xavier init for W1[24,24] + W2[12,24]).
- `:11476` `qlstm_weights_host` (528 floats: QLSTM Xavier init for 6 gates × 11 input × 8 head).
All sites use `mapped_pinned::upload_f32_via_pinned` (canonical mapped-pinned + DtoD staging helper). The helper returns `Result<_, String>` whereas the ctor returns `Result<_, MLError>`, so each site wraps the error via `.map_err(|e| MLError::ModelError(format!("<site> upload via pinned: {e}")))`. Site labels preserved so backtraces remain readable.

### Fix 18 (perf) — Eliminate per-step val gather DtoD copies via chunked gather kernel (2026-04-28)
`experience_kernels.cu` + `gpu_backtest_evaluator.rs` + `metrics.rs`: nsys profile of `multi_fold_convergence` on L40S identified `backtest_state_gather` as the #1 GPU-time consumer at 37.2% (596 ms / 60,588 calls — kernel launch latency dominated compute). The per-step pattern was:

```
for step_offset in 0..chunk_len {
  launch_gather(step) -> states_buf[N, padded_sd]   (1 launch)
  if tlob: tlob.forward(states_buf, n)              (4 cuBLAS SGEMMs + 3 kernel launches)
  memcpy_dtod_async(chunked_states_buf[step_offset*N..], states_buf, ...)   (1 DtoD)
}
```

Per chunk (DQN_BACKTEST_CHUNK_SIZE=512): 512 gather launches + 512 DtoD copies + 512 × 7 TLOB launches = 4608 kernel launches per chunk before forward Q-values can begin.

New kernel `backtest_state_gather_chunk` writes `[chunk_len, N, padded_sd]` directly into `chunked_states_buf` in a SINGLE launch (`chunk_len * N` threads, 1 thread per output row). Within a chunk `portfolio_buf` and `plan_isv_buf` are CONSTANT — env_step (Phase 5) and `backtest_plan_state_isv` (Phase 6) update them only at chunk boundary. So the batched gather is mathematically identical to the prior per-step + DtoD pattern; each thread reads its own (window, step_offset) row from independent feature offsets and writes its own state row. No reordering, no atomics, no shmem races.

Same change enables the val TLOB instance to run ONCE per chunk on the chunked buffer at batch = `chunk_len * N` instead of `chunk_len` separate forward calls each at batch = `N`. Val TLOB construction in `metrics.rs::reset_evaluator_for_validation` resized via new `GpuBacktestEvaluator::val_tlob_batch_size()` to `DQN_BACKTEST_CHUNK_SIZE * n_windows`; partial last chunks reuse the same buffers (`forward(b)` accepts any `b <= construction_batch`).

Per-chunk launch reduction: from `2 * chunk_len + chunk_len * 7` to `1 + 7` for the gather+TLOB phase. With 512-step chunks that's 4608 → 8, a 576x reduction. Across the full smoke run (60K gather calls in profile), expected wall-clock saving on the gather/TLOB hot path is the bulk of the 37% + 6.5% + 5.2% + 4.8% = ~53.7% of GPU time the per-call cluster occupied.

Borrow restructure: the prior single-shot `let ch_states = self.chunked_states_buf.as_ref()?` at top of `submit_dqn_step_loop_cublas` was removed because TLOB now needs `&mut self.chunked_states_buf`. Replaced with per-chunk `ch_states_base` (raw u64 device pointer extracted in a tight scope), reused for both `launch_gather_chunk` and the Phase 2+3 `compute_q_values_to` call. The pointer is stable across the chunk because the `Option<CudaSlice<f32>>` does not reallocate.

No new HtoD/DtoH paths introduced. Strictly removes 512 per-chunk DtoD copies. Per `feedback_no_partial_refactor.md`: kernel ABI is new (separate function name), prior `backtest_state_gather` retained for `evaluate()` / `evaluate_ppo()` / `evaluate_supervised()` paths that still need a single-step writer.

### Fix 19 (perf) — Online softmax in compute_expected_q: 3-pass → 2-pass (2026-04-28)
`experience_kernels.cu`: nsys profile of `multi_fold_convergence` on L40S identified `compute_expected_q` as the #3 GPU consumer at 12.9% (207 ms / 1382 calls — ~150 µs per call, compute-bound at typical batch=8192 × num_atoms=51). The kernel computed each per-action softmax over atoms in 3 passes (max → sum_exp → normalize+expected/var/entropy/util), re-reading `v_row[z] + adv_a[z]` 3 times per atom for each of the 4×~3.25 = 13 actions per sample.

Replaced with **online softmax** (Page-Olshen running-max + running-sum) so expected_q, sum_z_sq, and entropy all accumulate in a single forward pass over atoms. Atom utilisation still needs a 2nd pass because it depends on the final `S = sum exp(logit-M)` being known to compute `prob_z = exp(logit_z - M)/S` for the threshold test. Total: 3 → 2 passes, ~33% reduction in inner-loop global loads of branch advantage logits (which dominate because `v_row[z]` is reused across 13 (action, branch) pairs and may stay in L1, while `adv_a[z]` flips per action and is pure global).

Online accumulation:
```
M, S, TZ, TZ², TLM = -inf, 0, 0, 0, 0
for z in 0..num_atoms:
    logit = v_row[z] + adv_a[z]
    if logit > M:
        scale = exp(M - logit)   # ≤ 1, no overflow
        S, TZ, TZ², TLM *= scale
        M = logit
    w = exp(logit - M)            # ≤ 1
    S   += w
    TZ  += w * z_val
    TZ² += w * z_val²
    TLM += w * (logit - M)
expected_q = TZ / S
sum_z_sq   = TZ² / S
entropy    = log(S) - TLM / S    # analytical: -sum(p log p) where p = w/S
```

Correctness: bit-stable when atoms processed in fixed order (z = 0..num_atoms-1). The analytical entropy form is mathematically equivalent to `-sum(p log p)`:
```
-sum(p log p) = -sum(p (logit - M - log S)) = log(S) * sum(p) - sum(p (logit-M))
             = log(S) - (1/S) * sum(exp(logit-M) * (logit-M))
             = log(S) - TLM / S
```
The previous code's `if (prob > 1e-10f)` underflow guard is no longer needed: `exp(logit - M)` underflows cleanly to 0 when logit << M, and the analytical form does not multiply tiny probs by very negative logs. Result is bit-stable per `feedback_stop_on_anomaly.md` — no fuzzy tolerance change.

Atom-stat block-sum reduction unchanged (warp shfl + shared-mem bank, deterministic).

Expected wall-clock saving on this kernel: ~33% of 207ms = ~70ms across the smoke run; on production batch=8192 × 1382 calls per fold, roughly proportional. Lower-bound estimate; actual saving depends on memory bandwidth vs compute mix — at low N (smoke) the kernel may be launch-bound rather than load-bound. nsys re-profile after deployment will quantify.

ABI unchanged; no Rust caller changes required.

### Fix 20 — DtoD-via-pinned guard + orphan HER deletion (2026-05-02)
`scripts/pre-commit-hook.sh` + `crates/ml/src/cuda_pipeline/gpu_her.rs`. Two related changes installing the structural guard against the SP6 Pearl 5 IQN τ failure mode (root cause fixed at `facbf76eb` for that one site) and removing the only remaining orphan callers of the broken pattern.

**The bug class.** The `mapped_pinned::{upload,clone_to_device}_{f32,i32}_via_pinned` helpers carry the `_via_pinned` suffix to suggest "no HtoD per `feedback_no_htod_htoh_only_mapped_pinned`", but their bodies do `MappedXBuffer::new()` (staging) + `memcpy_dtod_async()` + `stream.synchronize()`. The DtoD copy and synchronize are both forbidden inside CUDA Graph capture (`CUDA_ERROR_STREAM_CAPTURE_INVALIDATED` — exactly the failure pattern in pre-`facbf76eb` Pearl 5 smokes) and add a host stall otherwise. The "_via_pinned" name was a lie. The canonical pattern is `MappedXBuffer` field stored directly + `write_from_slice` for host writes + kernel reads via `.dev_ptr` — used by SP4 portfolio_state, SP4 grad_prev_buf_per_group, and SP6 IQN τ at `facbf76eb`.

**Guard.** New `check_no_dtod_via_pinned` function in `pre-commit-hook.sh` rejects any staged `.rs` file that calls `upload_(f32|i32)_via_pinned` or `clone_to_device_(f32|i32)_via_pinned`, with a single exception for `mapped_pinned.rs` itself (where the helpers currently live). Per `feedback_no_hiding`: no suppression marker. The error message points at the structural fix (rewrite to `MappedXBuffer` directly).

Also fixes a pre-existing silent skip: the `gpu-hotpath-guard.sh` invocation used `SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"` which resolved to `.git/hooks/` (the symlink's directory) instead of `scripts/` (the script's actual location), so the guard never ran. Replaced with `readlink -f "$0"` to resolve through the symlink chain. Added an explicit `else` branch that errors if the guard binary is missing — silent skip is worse than no guard.

**Orphan deletion.** `gpu_her.rs` carried legacy `pub fn relabel_batch(...)`, `pub fn generate_random_donors(...)` (CPU version), `pub struct HerBatch`, `fn slice_clone_f32`, `fn slice_clone_i32` — confirmed zero callers via:
```
grep -rn '\.relabel_batch(' --include='*.rs' | grep -v 'with_strategy\|_gpu\b'
grep -rn 'HerBatch\|slice_clone_f32\|slice_clone_i32' --include='*.rs'
grep -rn '\.generate_random_donors(' --include='*.rs' | grep -v '_gpu\b'
```
Production uses `relabel_batch_with_strategy(...)` (line 384, called at `fused_training.rs:1555`) which does pure-DtoD donor selection from a pre-existing GPU buffer — no `*_via_pinned` calls. The orphan held the only `upload_i32_via_pinned` callers in the codebase (lines 279, 285). Per `feedback_no_hiding` (wire-or-delete) the right fix is delete; conversion would have been adding `MappedI32Buffer` machinery to dead code.

Also removed the now-unused `DevicePtr` import and updated the module docstring header to reference `relabel_batch_with_strategy` + `generate_random_donors_gpu` (the surviving production paths).

**Scope.** This commit eliminates 2 of the 47 production `*_via_pinned` call sites (the two inside the deleted orphan). The remaining 45 sites across 14 files are cold-path init (constructors, per-fold setup, hyperopt adapters, walk-forward backtest setup) — they don't break operationally because they're outside any captured graph, but they're still graph-capture-fragile and host-stalling. The guard now enforces no NEW calls; the existing 45 will be migrated in subsequent atomic per-buffer commits. After all 45 are converted, the four helpers themselves get deleted from `mapped_pinned.rs`.

**Validation.** Smoke `smoke-test-82fjk` at the τ-buffer-fix HEAD `facbf76eb` succeeded (`phase=Succeeded progress=1/1`, 5-epoch L40S `multi_fold_convergence`). Magnitude differentiation restored (`q_full=0.462 > q_half=0.409 > q_quarter=0.350` vs baseline frozen Pascal-triangle 0.225/0.280/0.495). Eval distribution unfrozen (`eq=0.596, eh=0.404, ef=0.000` vs baseline single-action collapse). Validates that the SP5+SP6 architecture works once the τ buffer DtoD violation is gone.

### Fix 21 — fxcache target schema fix + Bug 2 state[0] instrumentation (2026-05-02)
Two related changes addressing the EVAL_DIST=Hold collapse observed in `train-multi-seed-hc6kt` 50-epoch validation (terminated epoch 5).

**Bug 1 — fxcache target column semantics violation (FIXED).** `precompute_features.rs:360` and `data_loading.rs:510` both stored raw OHLCV close prices in target slots [0:1] (`preproc_close` / `preproc_next`), violating the documented schema in `experience_kernels.cu:1556` and `cuda_pipeline/mod.rs:508`. Empirically confirmed via direct fxcache inspection: `target[0..4]` had mean=$5967, stddev=$582 — raw prices throughout. The contract specifies log-return-normalized values for the network-input columns. Both writers now compute `(raw_curr / prev_close).ln()` and `(raw_next / raw_curr).ln()` for the preproc columns. `FXCACHE_VERSION` bumped 7 → 8 to invalidate existing caches and force regen via the ensure-fxcache Argo step. The raw-price values at target[0:1] were never directly consumed by training (aux head reads `next_states[i][0]` = MARKET feat[0] = log_return), but they corrupted any downstream code that read targets per the documented contract.

**Bug 2 — state[0]=raw_close suspected; instrumented for runtime confirmation.** Production runs show `aux_label_scale=5300` (raw price magnitude) but the source path traces back through `next_states_buf[i][0]` ← `experience_state_gather(market_features[bar_idx][0])` ← fxcache feat[0] (z-normalized, stddev=1.0). The discrepancy can't be resolved by static code reading alone — there must be a runtime path that injects raw_close into state[0] between the gather and the aux read. Added one-shot debug print in `training_loop.rs` after first `collect_experiences_gpu` return: downloads first 5 samples × 5 columns of both `gpu_batch.states` and `gpu_batch.next_states`, computes mean/std/mean_abs over col 0, prints under `target: "DIAG_BUG2"`. Triggers once via static `AtomicBool`. f32 dtoh download is slow but pre-graph-capture so no hot-path impact. Interpretation key embedded in the log line: `mean_abs ~0.001 = log_return ✓ | mean_abs ~5000 = raw price ✗`.

**Why the smoke test passes but production fails.** The smoke harness `multi_fold_convergence::test_multi_fold_convergence` runs the same DQN trainer but bypasses the production `train_baseline_rl` rollout machinery. The smoke at facbf76eb showed label_scale~0.1 (consistent with normalized log_return). The production binary at the same commit shows label_scale~5300 (raw price). One of the production-only code paths is mutating state[0] post-gather. Bug 2 instrumentation will pin the exact mutation site at next L40S run.

### Fix 22 — features_raw_cuda + targets_raw_cuda → MappedF32Buffer (Bug 2 instrumentation also lands) (2026-05-02)
Cross-file structural conversion that closes the last two `clone_to_device_f32_via_pinned` call sites in the DQN production path (training_loop.rs:1308 + 1312, the residue from Fix 16's pre-guard pattern). Mirrors the SP6 IQN τ buffer fix at `facbf76eb` (audit-tracked under task #283) and the SP4 portfolio_state pearl pattern: `MappedF32Buffer` field stored directly + `write_from_slice` for host load + kernel reads via `.dev_ptr`. No DtoD copy, no synchronize, graph-capture safe.

**What changed.**
- `trainers/dqn/trainer/mod.rs` — `targets_raw_cuda` and `features_raw_cuda` field type `Option<CudaSlice<f32>>` → `Option<MappedF32Buffer>`.
- `trainers/dqn/trainer/training_loop.rs::init_gpu_raw_buffers_from_slices` — replaces both `clone_to_device_f32_via_pinned` calls with `MappedF32Buffer::new(n)` + `write_from_slice`. Pre-existing `if self.cuda_stream.is_none() { return Ok(()) }` guard preserved (the `stream` value itself isn't required by the mapped-pinned allocator, only an active CUDA context, which the `cuda_stream-is-Some` invariant implies).
- `cuda_pipeline/gpu_experience_collector.rs` — three signature changes propagated through the consumer chain: `collect_experiences_gpu`, `launch_timestep_loop`, `compute_difficulty_scores`. Five `.arg(buf)` launch sites (hindsight_relabel, state_gather, expert_override, env_step×1, env_step×2) rewritten to `.arg(&buf.dev_ptr)`. `targets.raw_ptr()` in `compute_difficulty_scores` → `targets.dev_ptr` (same `CUdeviceptr` u64, different field name).
- `cuda_pipeline/decision_transformer.rs::build_dt_trajectories` — signature `&CudaSlice<f32>` → `&MappedF32Buffer` for both `features_gpu` and `targets_gpu`. Two `raw_ptr_f32(buf, stream)` call sites converted to `buf.dev_ptr` (mapped-pinned has no SyncOnDrop guard, so the `_no_drop` ManuallyDrop dance isn't needed). Doc-comment updated to reflect mapped-pinned semantics.
- `trainers/dqn/trainer/training_loop.rs:3749` (SP5 Pearl 8 trail launch) — `features_buf.len()` (method) → `features_buf.len` (pub field); `features_buf.device_ptr(stream)` (StreamGuard idiom) → `features_buf.dev_ptr` direct. Removed unused `DevicePtr` import + `_guard` binding.

**Bonus deliverable: Bug 2 diagnostic.** Same commit lands the one-shot DIAG_BUG2 download at training_loop.rs:1768 (right after the first `collect_experiences_gpu` return). Static `AtomicBool` ensures it fires exactly once per process. Per-sample first-5-col rows + col-0 mean/stddev/mean_abs aggregates printed under `target: "DIAG_BUG2"`. Interpretation key: `mean_abs ~0.001 = normalized log_return ✓ | mean_abs ~5000 = raw price ✗`. Uses `clone_dtoh` (current cudarc API; replaced the deprecated `memcpy_dtov`). DtoH is a graph-capture violation but this is single-shot pre-capture, so OK.

**Adjacent buffers in the same commit (carried-over from prior agent attempts):**
- `gpu_dqn_trainer.rs::sel_clip_buf` — single-element `CudaSlice<f32>` (sel-grad clip norm constant 1.0) → `MappedF32Buffer`. Aux Adam-update kernel reads `clip_ptr = self.sel_clip_buf.dev_ptr`. Eliminates the `upload_f32_via_pinned` constructor call.
- `gpu_weights.rs::RmsNormWeightSet` — four γ buffers (s0, s1, v, a; each is an all-ones init for RMSNorm γ scaling factor) `CudaSlice<f32>` → `MappedF32Buffer`. `KernelWeightPack::new` reads `rmsnorm.gamma_*.dev_ptr` directly (no `raw_device_ptr(...)` helper needed for mapped-pinned). Eliminates four `clone_to_device_f32_via_pinned` calls in `RmsNormWeightSet::ones`.

**PPO trainer deferred.** `trainers/ppo.rs:221,223` carry the same field types; PPO is not on the eval-collapse hot path and is out of scope. Pre-commit guard checks staged files only, so PPO's existing `*_via_pinned` violations don't block. Tracked as a follow-up.

**Validation.** `cargo check -p ml --offline` clean (zero warnings post-`memcpy_dtov` → `clone_dtoh` migration). `cargo build -p ml --release --offline` clean. Pre-commit `check_no_dtod_via_pinned` passes (no `*_via_pinned` calls in any staged file). After this commit, the only remaining `*_via_pinned` callers in the entire ml crate are in `trainers/ppo.rs` and a long tail of cold-path init sites tracked under task #290.

### Fix 23 — `gpu_dqn_trainer.rs` spectral-norm + aux-head ctor sites + `gpu_experience_collector.rs` OFI upload → MappedF32Buffer (10 sites, 2026-05-02)
Final structural sweep that closes the residual `*_via_pinned` ctor / cold-path init call sites in the two files still violating the pre-commit `check_no_dtod_via_pinned` guard after Fix 22. Same canonical pattern as the τ-buffer fix at facbf76eb and the Fix 22 features/targets conversion: `MappedF32Buffer` field stored directly + `MappedF32Buffer::new(n)` + `write_from_slice` for host load + kernel reads via `.dev_ptr` (or via `spectral_norm_descriptors` u64 table for the spec_u/spec_v cluster). No DtoD, no synchronize, graph-capture safe.

**What changed (per buffer-migration group).**
- **Spectral-norm `spec_u_s1` / `spec_v_s1` / `spec_u_s2` / `spec_v_s2` (4 buffers).** Trunk power-iteration vectors for `w_a_h_s1` / `w_a_h_s2` (Linear_a matrices in the GRN trunk). Field type `CudaSlice<f32>` → `MappedF32Buffer`. Constructor (`gpu_dqn_trainer.rs` ~14421-14431) replaces `alloc_f32` + `upload_f32_via_pinned` with `MappedF32Buffer::new` + `write_from_slice`. Consumed only via `.raw_ptr()` → `.dev_ptr` in the `host_desc[78]` table that backs `spectral_norm_descriptors` — same `CUdeviceptr` u64 width, purely a structural rename.
- **Spectral-norm head/branch macro `alloc_spec_pair!` (16 buffers).** Macro at ~14443 is invoked 11 times to allocate value-head, advantage, and 4 branch-output spec_u/spec_v pairs (`spec_u_v1/v_v1`, `spec_u_v2/v_v2`, `spec_u_a1/v_a1`, `spec_u_a2/v_a2`, `spec_u_bo1..bg2`, `spec_u_bn/v_bn`). Macro body now allocates `MappedF32Buffer::new` + `write_from_slice` directly (no `&mut $u_name` upload helper needed — interior mutability via `write_volatile`). Same `host_desc` consumer chain: each `.raw_ptr()` → `.dev_ptr`.
- **`graph_params` (60 floats, 1 buffer).** Cross-branch graph message-passing edge params (4 edges × 15 floats = 60). Field type `CudaSlice<f32>` → `MappedF32Buffer`. Constructor (~15790) `MappedF32Buffer::new(60)` + `write_from_slice(&graph_params_host)`. Single consumer at `launch_graph_message_pass` reads `params_ptr = self.graph_params.dev_ptr`.
- **`denoise_params` (DENOISE_TOTAL_PARAMS=1800, 1 buffer).** Diffusion Q-refinement MLP weights (2 steps × 900 = W1[576]+b1[24]+W2[288]+b2[12]). Field type `CudaSlice<f32>` → `MappedF32Buffer`. Constructor (~15845) Xavier init host-side then `write_from_slice`. Four consumer sites: `launch_q_denoise`, `launch_q_denoise_backward`, `launch_q_denoise_backward_cublas`, aux Adam update — all read `params_ptr = self.denoise_params.dev_ptr`. Notable: the kernel writes back to this buffer during aux Adam updates; mapped-pinned semantics are kernel-write-coherent the same way as kernel-read (single mapping, both directions through `dev_ptr`).
- **`qlstm_weights` (QLSTM_WEIGHT_COUNT=528, 1 buffer).** xLSTM mLSTM-cell parameters (6 weight matrices × [11, 8] = 528). Field type `CudaSlice<f32>` → `MappedF32Buffer`. Constructor (~15992) Xavier uniform host-side then `write_from_slice`. Two consumer sites: `launch_qlstm_step`, `launch_qlstm_train_step` — both read `w_ptr = self.qlstm_weights.dev_ptr`. Kernel SGD-updates in place via the same dev_ptr mapping.
- **`gpu_experience_collector.rs::ofi_gpu` (1 buffer + 1 placeholder).** Pre-uploaded OFI features (`total_bars * OFI_DIM`). Field type `CudaSlice<f32>` → `MappedF32Buffer`. Constructor places a 1-element zero placeholder (`MappedF32Buffer::new(1)`); `upload_ofi_features` rewrites the field with a fresh `MappedF32Buffer` sized to `ofi_flat.len()`, then `write_from_slice(ofi_flat)`. Replaces the `clone_to_device_f32_via_pinned` call. The single per-step consumer at `launch_timestep_loop`'s state_gather invocation reads `ofi_ptr = self.ofi_gpu.dev_ptr`.

**Hot-path purity.** None of the converted buffers are written per-step from the host; `denoise_params` and `qlstm_weights` are mutated by kernel-side aux-Adam / SGD updates which use the same `dev_ptr` mapping the read-side does. `graph_params` and the spec_u/spec_v cluster are static post-init (graph_params is intentionally not trained; spec vectors are updated by the spectral-norm power-iteration kernel via the descriptor table, again through `dev_ptr`). `ofi_gpu` is rewritten exactly once (cold path) via `upload_ofi_features` — re-allocating the full mapped-pinned buffer and replacing the field on each call is fine because the call frequency is one-per-fold, never per-step.

**Validation.** `cargo check -p ml --offline` clean. Pre-commit `check_no_dtod_via_pinned` passes — zero `upload_*_via_pinned` / `clone_to_device_*_via_pinned` callers in either of the two staged files. The Bug 2 instrumentation commit currently in flight (Fix 21 / Fix 22 chain) can now proceed past the pre-commit guard.

**Out of scope.** `mapped_pinned.rs` retains the `upload_f32_via_pinned` / `upload_i32_via_pinned` / `clone_to_device_f32_via_pinned` / `clone_to_device_i32_via_pinned` helper definitions — `trainers/ppo.rs` and a small set of remaining cold-path callers still depend on them; the helpers are deleted under task #290 once the call count globally hits zero. Per `feedback_no_hiding` no suppression markers were added.

### Fix 26 — DBN spread-instrument filter (root cause of Bug 2 contamination) (2026-05-02)
**Pinned the source of the 8,799 corrupt bars sanitize_bars caught.**

Direct inspection of `ES.FUT_2024-Q1.dbn.zst` via `databento` Python client (offline, kubectl-cp from PVC):

| Instrument type | Symbols | Records (Q1 file) | Price range |
|---|---|---|---|
| **Outright** (correct) | `ESH4`, `ESM4`, `ESU4`, `ESZ4`, `ESH5` | 43,353 (76.87%) | $5,063 – $5,478 |
| **Calendar spreads** (poison) | `ESH4-ESM4`, `ESM4-ESU4`, `ESH4-ESU4`, `ESM4-ESZ4`, ... | 13,048 (23.13%) | $47.30 – $219.70 |

Calendar spreads trade at the *price difference* between adjacent contracts (~$60-150 due to cost-of-carry roll basis). Databento's `stype_in=parent` symbology (used for `ES.FUT` requests) returns BOTH outright contracts AND every calendar-spread combination in the same DBN stream.

**Why the contamination leaked through:** the legacy `decode_ohlcv_records` keyed records only by `ts_event`, then deduped by `volume` (highest-wins). Outrights typically dominate volume during regular hours, so most spread bars were dropped. But during low-liquidity overnight windows + active rollover periods, spreads hit higher minute-volume than the outright (samples show spread volumes up to 1554 contracts/min during ESH4→ESM4 transition vs near-zero outright volume in those minutes). 780 spread bars survived dedup in Q1 alone; ~8,800 across the full 2024-2026 dataset.

**The fix** (`baseline_common/mod.rs::decode_ohlcv_records`): build `dbn::TsSymbolMap` from the DBN metadata once, resolve each record's `instrument_id` to its symbol, and skip any symbol containing `-` (the canonical spread separator). Same-timestamp dedup-by-volume continues to handle legitimate front/back-month overlap during rollover.

**Why this is correct vs the bar-level sanitize:** the sanitize_bars layer (Fix 25) catches the *symptom* (close ratio out of bounds) and works for any instrument. The spread filter catches the *cause* — a spread between contracts where the underlying happens to have a small basis can sit within the [0.5, 2.0] sanitize ratio (e.g., short-tenor spread on a low-vol day might trade at $5/$5141 = 0.001 OR cleanly on the underlying scale during fast moves). The two layers are complementary: spread filter is precise but symbol-dependent; sanitize is universal but symptom-based.

**Validation.** `cargo check -p ml --example train_baseline_rl` clean. Adds `time = "0.3"` to ml/Cargo.toml (required by `dbn::TsSymbolMap`'s `time::Date` API). Expected next-smoke effect: `decode_ohlcv_records` log shows `skipped N spreads` with N matching the per-file contamination count; `sanitize_bars` reports zero (or near-zero) drops since the spread filter caught everything upstream. Sanitize remains as defense-in-depth for unknown future contamination shapes.

**Closes the chase.** Bug 2 root → ~8,800 spread bars from Databento's parent-symbol resolution. Caught structurally now at the DBN parser, not at z-score time.

### Fix 25 — Structural data-loading defense + DIAG_BUG2 cleanup (2026-05-02)
**Goal: stop chasing one-off data-corruption bugs. Make corruption impossible to flow into training, regardless of source.**

After three+ historical fixes for data-loading corruption (#191 fxcache column-0 raw-price leak, Bug 1 fxcache target-column semantics, label_scale=5443 raw-magnitude leaks, plus today's state[0] heavy-tail with std=570 outliers), the structural defense was still missing. Each prior fix patched a specific writer; the next corruption shape always found a new hole. This commit installs a 5-layer defense-in-depth gate stack so any future corruption — whatever its origin — is rejected at the data-loading boundary instead of poisoning the model.

**The 5 layers (each independent, each mandatory):**

1. **Bar-level sanity** (`baseline_common/mod.rs::sanitize_bars`): drops bars with non-positive OHLC, `high < low`, non-finite values, or `bar.close / prev.close` outside `[0.5, 2.0]`. Catches DBN parse glitches, broker tick errors, near-zero-open bars at source. Logs each drop with timestamp.

2. **`safe_log_return` result clamp** (`features/extraction.rs:1246`): `ratio.ln().clamp(±0.1)`. Real-market 1-bar log returns rarely exceed ±0.05; the ±0.1 ceiling traps every realistic move while rejecting all known corruption shapes (a corrupt bar with `bar.open ≈ 0` would otherwise produce `ln = -30` and flow through z-score normalization to ±30000-magnitude state[0] outliers).

3. **`validate_features` pre-norm magnitude bound** (`features/extraction.rs::validate_features`): post-extraction, every feature must satisfy `|val| ≤ 5.0`. Pre-normalization features come from `safe_normalize` ([0, 1] / [-1, 1]), `safe_clip` (max ±3 across all uses), or clamped log returns (±0.1); the widest legitimate range is ±3, so ±5 catches extractor invariant breaks without rejecting any legitimate output.

4. **`NormStats::normalize` post-norm clamp** (`walk_forward.rs:688`): `((val - mean) / std).clamp(±NORMALIZED_FEATURE_BOUND)` with bound = 20.0. Even if upstream layers somehow produce outliers (logic bug, future regression), this guarantees every value uploaded to GPU is within ±20.0 — well above any legitimate z-scored signal but well below the magnitudes (±30000) that previously poisoned label_scale_ema.

5. **Shared `validate_normalized_features` gate** (`walk_forward.rs::validate_normalized_features`): single source-of-truth invariant, enforced at three call sites:
   - **fxcache fast path**: after `discover_and_load` returns `Some(cached)` in `train_baseline_rl.rs:594`. fxcache files can be stale (different code, schema drift, manual file edits) — the cache_key + schema_hash check doesn't catch every corruption shape. If load returns out-of-bounds data, fail rather than upload it.
   - **DBN fallback**: after `normalize_batch` in `train_baseline_rl.rs:633`. NormStats::normalize already clamps each output, so failure here means the clamp itself is broken.
   - **precompute writer**: after `normalize_batch` in `precompute_features.rs:686`. Never write a poisoned fxcache to disk; failing here prevents corruption from being persisted across runs.

**Removed:** `DIAG_BUG2` and `DIAG_BUG2_v2` one-shot diagnostic blocks at `training_loop.rs:1785-1908` (~125 lines, including DtoH downloads + outlier scans). Replaced by the structural defense — instrumentation isn't needed once corruption can't reach state[0] in the first place. Same commit also removes the `pre-norm` and `post-norm` interpretive log lines that referenced the now-impossible "raw-price magnitude" failure mode.

**FEATURE_SCHEMA_HASH auto-bumps** (build.rs FNV-1a hash over SCHEMA_FILES, includes `extraction.rs`). This invalidates any pre-fix `.fxcache` file via the strict header check at `fxcache.rs:201`. The next training run will trigger ensure-fxcache regen, producing a clean cache with the new clamps applied.

**Why this finally closes the chapter:**
- Per-writer fixes are reactive (after-the-fact, after corruption has already hit production).
- A boundary-validation invariant is proactive — the contract becomes "any data path that violates `|feat| ≤ 20.0` post-normalize panics at the boundary." Every future regression to extraction or normalization is caught at the gate, not at epoch 5 of a 50-epoch L40S run.
- The 5 layers are independent: a bug in one layer leaves the others as backstop. To produce a state[0] outlier under this commit, every single one of {sanitize_bars, safe_log_return clamp, validate_features pre-bound, NormStats clamp, validate_normalized_features gate} would have to silently fail simultaneously.

**Validation.** `cargo check -p ml --all-targets --offline` clean. Unit tests for NormStats (walk_forward.rs:706+) still pass — the clamp and the new validate function are additive; existing test inputs are well within bounds.

### Fix 24 — DIAG_BUG2_v2 outlier disambiguation (H1 vs H2) (2026-05-02)
Augments the one-shot DIAG_BUG2 block at `training_loop.rs:1785` with two narrower scans that disambiguate where the state[0] heavy-tail originates. Smoke `smoke-test-xb78r` at HEAD `cba9f25ed` reported `mean=6.0e-3, std=5.7e2, mean_abs=2.0e1` over 3200 samples but printed first-5-sample state[0..5] values that all sit in [-1, 1] — implies ~1-2 of 3200 samples carry ±32000-magnitude values rather than uniform corruption.

**What the v2 scan adds.** Two outlier sweeps, both pre-graph-capture, fired once per process via the same `static AtomicBool DUMPED`:
- **State-side sweep:** scans `gpu_batch.states` and `gpu_batch.next_states` (already DtoH-downloaded for the v1 block) for sample indices `i` where `|state[i*sd + 0]| > 100`. Reports up to 10 indices + values per buffer.
- **Source-side sweep:** scans `features_buf.host_ptr` directly via raw-pointer arithmetic for bar indices `b` where `|features[b*42 + 0]| > 100`. host_ptr aliases the same physical pages the kernel reads (mapped-pinned), so this is a faithful read of the data state_gather actually sees — no DtoH copy needed. Reports mean/std/max_abs over up to 2M bars + up to 10 outlier (bar_idx, value) pairs.

**Disambiguation rule (logged as the verdict line):**
- feat-outliers nonempty → **H1** (data-source corruption — the fxcache or DBN-fallback feature data carries extreme values, e.g. corrupted DBN bars with bar.open ≈ 0 producing ln(ratio) = −30 → z-score normalize to −30000).
- state-outliers nonempty AND feat-outliers empty → **H2** (state_gather kernel reads stale GPU memory or bar_idx points out-of-range).

**Context for H1.** `smoke-test-xb78r` ran via the **DBN fallback** path (`train_baseline_rl.rs:611` log message format `"Loaded N bars (ts to ts)"`), not the fxcache fast path (which prints `"bars + features from fxcache in Xs"`). `/data/feature-cache/` on training-data PVC is empty — the fxcache file does not exist, so every L40S smoke since the PVC was reset hits the DBN fallback. The fallback applies `NormStats::normalize_batch` (line 633) which is z-score normalization with column-wise mean/stddev computed over all bars. log-return columns have small stddev (~1e-3 for 1-min NQ); a single corrupted bar with `bar.open ≈ 0` produces `safe_log_return = ln(near_zero / 22000) ≈ -30`, which divided by std=0.001 yields normalized ≈ -30000. Few such bars in 700K survive normalization as outliers.

**Hot-path purity.** Diagnostic only. No kernel changes, no buffer-shape changes. Pre-graph-capture, single-shot. Safe to leave in until Bug 2 is fixed; remove with the eventual `pearl_safe_log_return_extreme_clamp` (or equivalent) commit. Per `feedback_no_hiding` no suppression markers added.

**Cleanup gate.** Once Bug 2 is fixed (likely a tighter clamp inside `safe_log_return` or in `NormStats::normalize_batch` for log-return columns), delete the entire DIAG_BUG2 + DIAG_BUG2_v2 block at `training_loop.rs:1785-1908` in the same commit. Tracked under task #294.

### Fix 28 — DIAG_AUX_LABEL one-shot instrumentation (label_scale=5481 leak) (2026-05-02)

**Production blocker.** The cancelled 50-epoch run `train-multi-seed-bn42w` at HEAD `29b1d34c6` (target stride/column drift fix already applied) hit `label_scale=5481` at epoch 4. The local smoke `multi_fold_convergence` test on the *same commit* lands at ~25 — three orders of magnitude lower, with all five Fix-25 data-loading defenses active and the Fix-26 DBN spread filter live. Something between the smoke harness and the production training-data PVC is still leaking raw-price magnitude into the kernel that produces `label_scale`.

The kernel in question is `aux_heads_kernel.cu::label_scale_ema` (launched at `gpu_dqn_trainer.rs::aux_heads_forward` Step 2b, ~line 12973). It computes `mean(|aux_nb_label_buf|)` over the batch and feeds that into the SP4 Pearls A+D Wiener-EMA producer at scratch slot 43 → ISV[`AUX_LABEL_SCALE_EMA_INDEX`]. `aux_nb_label_buf` is a dedicated `[B] f32` scratch populated each step by `strided_gather` from column 0 of `next_states_buf` (= `log_return` per `pipeline.rs`'s feature contract). z-normalised log-returns sit at ~0.8 mean-abs; raw close prices on the ES front month sit at ~5500. A fivefold-thousand discrepancy between smoke and production at the same compiled binary is impossible unless the SOURCE data fed to `state_gather` is different.

**Four hypotheses to disambiguate** (the `(1)..(4)` numbers below match the diagnostic output lines):

| H# | Path | Test | Expected |
|----|------|------|----------|
| H1 | aux_buf populated correctly, downstream kernel misreads | (1) aux_buf mean_abs | ~0.8 → kernel-side bug; ~5500 → no |
| H2 | next_states_buf col 0 was written with raw prices | (2) col0 mean_abs | ~5500 confirms |
| H3 | features_raw_cuda col 0 (the SOURCE) carries raw prices | (3) feat col0 mean_abs | ~5500 confirms; ~0.8 → no |
| H4 | targets_raw_cuda col 2 (raw_close) leaked into a consumer | (4) tgt col2 mean_abs | always ~5500 (it IS raw_close); informs whether a consumer wrongly read it |

Direct stats comparison across the four numbers will pin the leak path in a single epoch-1 batch.

**Implementation.** One-shot `static AtomicBool DUMPED` block at `gpu_dqn_trainer.rs::aux_heads_forward` immediately after the `launch_label_scale_ema` call (~line 12980), guarded so only the first ungraphed step-0 invocation reads the buffers. Step 0 of `fused_training.rs::run_full_step` runs `submit_forward_ops_main` UNGRAPHED before `capture_training_graph` records the operation set; the captured graph never re-enters the diagnostic branch (`DUMPED` is `true` on the first replay), so subsequent training steps pay zero cost. `clone_dtoh` is safe in step 0 because no graph capture is active. Tracing target `DIAG_AUX_LABEL`. Verdict line at the end documents the interpretation rules.

**Buffer-access wiring.** `aux_nb_label_buf` and `next_states_buf` live on `GpuDqnTrainer` and are read directly via `self.stream.clone_dtoh(&...)`. `features_raw_cuda` and `targets_raw_cuda` live on `DQNTrainer` (mapped-pinned `MappedF32Buffer`); their addresses are passed through a process-static `OnceLock<(usize, usize, usize, usize)>` (`DIAG_AUX_LABEL_SOURCE_PTRS`) populated by `init_gpu_raw_buffers_from_slices` after the mapped-pinned pages are allocated. Mapped-pinned host_ptrs are stable for the buffer lifetime, the `DQNTrainer` outlives every `aux_heads_forward` call, and no DtoH copy is needed for these two reads — the kernel-visible bytes ARE the host bytes.

**Hot-path purity.** Diagnostic only. No kernel changes, no buffer-shape changes, no struct field changes. Pre-graph-capture (step 0 ungraphed), single-shot. The static `OnceLock` set is a one-time write and the `AtomicBool::swap` is one relaxed-ordered RMW — both invisible to the captured graph.

**Removal gate.** Once the leak source is identified and fixed, delete in the same commit:
- The `DIAG_AUX_LABEL` block at `gpu_dqn_trainer.rs::aux_heads_forward` (~line 12980).
- The `DIAG_AUX_LABEL_SOURCE_PTRS` `OnceLock` static at `gpu_dqn_trainer.rs:~556`.
- The `OnceLock::set` call at `training_loop.rs::init_gpu_raw_buffers_from_slices`.

**Validation.** `SQLX_OFFLINE=true cargo check -p ml --offline` clean; `cargo build -p ml --release --offline --features cuda` clean. Diagnostic fires on the next L40S smoke run or production training step — no GPU spend in this commit.

### Fix 29 — SP15 launcher CudaFunction pre-load (Pattern 2 cubin-load on hot path) (2026-05-07)

**Bug audit Pattern 2.** Six SP15 launchers in `gpu_dqn_trainer.rs` were doing `load_cubin` + `load_function` PER CALL inside their bodies, called from `gpu_experience_collector.rs::collect_experiences_gpu`'s per-rollout-step body — `~thousands of times per epoch × 4096 envs × 1000 timesteps`. Same architectural bug class as commits `5d63762ab` (`bn_tanh_concat_dd`) and `1396b62ec` (`sp15_baseline` + `cost_net`). `load_cubin`/`load_function` are host-side driver API calls that mutate driver state — NOT capturable inside `cuStreamBeginCapture` per `pearl_no_host_branches_in_captured_graph`, AND prone to `CUDA_ERROR_ILLEGAL_ADDRESS` in subprocess child contexts (documented failure mode in `1396b62ec`, workflow `train-xggfc` trial 1 onwards).

| Launcher (file:line in current commit) | Caller (gpu_experience_collector.rs) | Class | Status |
|---|---|---|---|
| `gpu_dqn_trainer.rs:907` `launch_sp15_dd_state` | `:5552` collect_experiences_gpu per-rollout-step | **MIGRATED** | FIXED |
| `gpu_dqn_trainer.rs:996` `launch_sp15_dd_state_reduce` | `:5609` collect_experiences_gpu per-rollout-step | **MIGRATED** | FIXED |
| `gpu_dqn_trainer.rs:1976` `launch_sp15_dd_trajectory_decreasing` | `:5587` collect_experiences_gpu per-rollout-step | **MIGRATED** | FIXED |
| `gpu_dqn_trainer.rs:1463` `launch_sp15_alpha_split_producer` | `:5624` collect_experiences_gpu per-rollout-step | **MIGRATED** | FIXED |
| `gpu_dqn_trainer.rs:1546` `launch_sp15_final_reward` | `:5637` collect_experiences_gpu per-rollout-step | **MIGRATED** | FIXED |
| `gpu_dqn_trainer.rs:1862` `launch_sp15_plasticity_injection` | `:5137` plasticity-injection trigger path | **MIGRATED** | FIXED |

**Migration.** Each launcher signature gained a `&CudaFunction` parameter; the cubin/function load was hoisted out of the body. The collector struct gained six new fields (`exp_sp15_dd_state_kernel`, `exp_sp15_dd_state_reduce_kernel`, `exp_sp15_dd_trajectory_decreasing_kernel`, `exp_sp15_alpha_split_producer_kernel`, `exp_sp15_final_reward_kernel`, `exp_sp15_plasticity_injection_kernel`), populated once in `GpuExperienceCollector::new()` on the collector's stream. Every per-rollout-step call site now passes `&self.exp_<kernel>_kernel`. Mirrors the SP14 EGF kernel-loading precedent established in commit `09202aa99`. All six SP15 cubin statics (`SP15_DD_STATE_CUBIN`, `SP15_DD_STATE_REDUCE_CUBIN`, `SP15_DD_TRAJECTORY_CUBIN`, `SP15_ALPHA_SPLIT_PRODUCER_CUBIN`, `SP15_FINAL_REWARD_CUBIN`, `SP15_PLASTICITY_INJECTION_CUBIN`) were already `pub static` — no visibility promotion required.

**Oracle tests.** `crates/ml/tests/sp15_phase1_oracle_tests.rs` updated with a small `load_sp15_kernel` helper that inlines the per-call cubin/function load at each test call site. Graph capture is not a concern in oracle scaffolds, so the per-launch resolution model is acceptable for tests; production callers route through the struct-cached handles via the collector.

**Atomicity.** Per `feedback_no_partial_refactor`: all six launchers + their callers + their tests migrated in this single commit. Per `pearl_no_host_branches_in_captured_graph`: zero `load_cubin` / `load_function` remain in the collector per-rollout-step body after this commit; verified by `grep -nE "load_cubin|load_function" crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` — only matches are in `compile_*` helpers (`new()` construction-time) and inline doc-comment references.

**Deferred (separate atomic commit).** `launch_sp15_sharpe_per_bar` (gpu_dqn_trainer.rs:722) and `launch_sp15_position_history_derivation` (gpu_dqn_trainer.rs:1355) are evaluator-path callers (`GpuBacktestEvaluator::run_evaluation_with_q_atoms` in `gpu_backtest_evaluator.rs`) — fixing them requires adding `CudaFunction` fields to a different struct, deferred to a separate atomic commit per scope-discipline.

**Out of scope.** `launch_sp15_regret_signal` (gpu_dqn_trainer.rs:1634) and `launch_sp15_cooldown` (gpu_dqn_trainer.rs:1753) are orphan launchers — only test callers, no production callers — left untouched per the audit (orphan-launcher resolution is an independent concern from the cubin-load-on-hot-path bug class).

**Validation.** `SQLX_OFFLINE=true CUDA_COMPUTE_CAP=86 cargo check -p ml --tests --all-targets` clean (only pre-existing warnings); `cargo test -p ml --test sp14_oracle_tests --release -- --ignored` 7/7 pass; `cargo test -p ml --test sp15_phase1_oracle_tests --release -- --ignored` 36/36 pass.