foxhunt/docs/dqn-gpu-hot-path-audit.md

DQN GPU Hot-Path Audit

Invariant 3 enforcement. Every cross-boundary call (DtoH, HtoD, memcpy between host and device) in the DQN code path is classified below. No hot-path memcpy DtoH/HtoD is allowed — zero tolerance. Pinned + device-mapped memory is the ONLY permitted mechanism for per-step host-visible data.

Hot path definition: every function called from fused_training.rs::run_full_step or its descendants that executes once per training step (up to 178 times per epoch at batch size 8192), including kernel invocations inside the captured CUDA graph (adam_grad_child, forward_child, aux_grad_child, aux_child, maintenance_child) and host-side orchestration between them.

Classification:

• OK-pinned — uses the zero-copy pinned + device-mapped pattern (cuMemAllocHost_v2 + cuMemHostGetDevicePointer_v2); allowed on hot path.
• MIGRATE — was a plain memcpy but has been converted to pinned in this commit.
• COLD-PATH — acceptable placement (constructor init, checkpoint save/load, fold transition, epoch-boundary); annotated with reason.

File:line	Call signature	Path class	Annotation	Status
gpu_dqn_trainer.rs:6586	malloc_host → total_loss_pinned	OK-pinned	Constructor: cuMemAllocHost+GetDevicePointer; GPU writes via dev ptr, CPU reads via readback_training_scalars()	OK
gpu_dqn_trainer.rs:6598	malloc_host → mse_loss_pinned	OK-pinned	Constructor: same pattern as total_loss_pinned	OK
gpu_dqn_trainer.rs:6611	malloc_host → q_divergence_pinned	OK-pinned	Constructor: same pattern	OK
gpu_dqn_trainer.rs:6624	malloc_host → iqn_readiness_pinned	OK-pinned	Constructor: same pattern	OK
gpu_dqn_trainer.rs:6662	malloc_host → grad_norm_pinned	OK-pinned	Constructor: same pattern	OK
gpu_dqn_trainer.rs:6707	malloc_host (plain, not DEVICEMAP) → readback scratch	COLD-PATH	Constructor: plain pinned for per-epoch flush_readback(); not written by GPU kernel directly	OK
gpu_dqn_trainer.rs:6788	malloc_host → monitoring summary pinned	OK-pinned	Constructor: monitoring kernel writes via dev ptr each step	OK
gpu_dqn_trainer.rs:6895	malloc_host → eval_v_range_pinned	OK-pinned	Constructor: GPU writes; host reads only at fold/epoch boundary via eval_v_range()	OK
gpu_dqn_trainer.rs:6911	malloc_host → nan_flag_pinned	OK-pinned	Constructor: GPU writes on NaN; host reads only in fault path (read_nan_flags)	OK
gpu_dqn_trainer.rs:6929	malloc_host → ISV isv_signals_pinned	OK-pinned	Constructor: GPU writes ISV bus each step; host reads via read_isv_health_and_regime() in compute_adaptive_budgets	OK
gpu_dqn_trainer.rs:6949	malloc_host → lr_pinned	OK-pinned	Constructor: CPU writes LR; GPU reads via dev ptr	OK
gpu_dqn_trainer.rs:6968	malloc_host → adaptive_clip_pinned	OK-pinned	Constructor: CPU writes clip; GPU reads via dev ptr	OK
gpu_dqn_trainer.rs:6988	malloc_host → popart_pinned	OK-pinned	Constructor: GPU writes PopArt stats; host reads at epoch boundary	OK
gpu_dqn_trainer.rs:7099	malloc_host → q_stats_pinned	OK-pinned	Constructor: GPU writes Q stats; host reads via reduce_current_q_stats() at epoch boundary	OK
gpu_dqn_trainer.rs:7127	malloc_host → q_readback_pinned	OK-pinned	Constructor: GPU writes per-step Q readback; host reads via last_readback_scalars() in training guard	OK
gpu_dqn_trainer.rs:7206	cuMemAllocHost_v2 → var_ema pinned (ISV subsystem)	OK-pinned	Constructor: ISV variance EMA — GPU writes each step; host reads via ISV bus	OK
gpu_dqn_trainer.rs:7243	cuMemAllocHost_v2 → regime_util pinned (ISV subsystem)	OK-pinned	Constructor: ISV regime utilization — GPU writes each step	OK
gpu_dqn_trainer.rs:7289	malloc_host → popart_var_pinned	OK-pinned	Constructor: GPU writes PopArt variance; host reads at epoch boundary	OK
gpu_dqn_trainer.rs:7312	cuMemcpyHtoDAsync_v2 (constructor)	COLD-PATH	Constructor: one-shot HtoD to seed deterministic-mask buffer; never replayed	OK
gpu_dqn_trainer.rs:7878	cuMemAllocHost_v2 → q_mean_scratch	OK-pinned	Constructor: GPU scratch for Q-mean reduction kernel	OK
gpu_dqn_trainer.rs:7898	cuMemAllocHost_v2 → q_mean_ema	OK-pinned	Constructor: Q-mean EMA — GPU writes, host reads at epoch boundary	OK
gpu_dqn_trainer.rs:7919	cuMemAllocHost_v2 → td_error_scratch	OK-pinned	Constructor: TD-error scratch for reduce kernel	OK
gpu_dqn_trainer.rs:7939	cuMemAllocHost_v2 → q_var_scratch	OK-pinned	Constructor: Q-variance scratch for reduce kernel	OK
gpu_dqn_trainer.rs:2300	stream.memcpy_htod → atom_positions_buf	COLD-PATH	set_atom_positions(): called at fold boundary to reload C51 atom grid — not on hot path	OK
gpu_dqn_trainer.rs:2606	stream.memcpy_dtoh → grad buf readback	COLD-PATH	per_branch_grad_norms(): per-epoch call only; drains grad buffer for logging	OK
gpu_dqn_trainer.rs:2925	stream.memcpy_dtoh (online params)	COLD-PATH	per_branch_target_drift(): per-epoch drift measurement	OK
gpu_dqn_trainer.rs:2928	stream.memcpy_dtoh (target params)	COLD-PATH	per_branch_target_drift(): per-epoch drift measurement	OK
gpu_dqn_trainer.rs:2991	stream.memcpy_dtoh → weight inspection	COLD-PATH	per_branch_vsn_mean(): per-epoch VSN mean read	OK
gpu_dqn_trainer.rs:3072	stream.memcpy_dtoh → Q-out inspection	COLD-PATH	compute_q_stats(): per-epoch Q-stats; not called per-step	OK
gpu_dqn_trainer.rs:7541	stream.memcpy_htod → descriptor buffer	COLD-PATH	Constructor: one-shot HtoD for GEMM descriptor upload	OK
gpu_dqn_trainer.rs:7680	stream.memcpy_htod → stochastic depth scale	COLD-PATH	Constructor: one-shot HtoD for stochastic depth kernel seed	OK
gpu_dqn_trainer.rs:7686	stream.memcpy_htod → stochastic depth RNG	COLD-PATH	Constructor: one-shot HtoD for stochastic depth RNG	OK
gpu_dqn_trainer.rs (removed)	cuMemcpyDtoHAsync_v2 in run_causal_intervention_unconditional	MIGRATED	Fix 1: removed dead copy — result causal_mean_scratch was never consumed by any caller; result stays on GPU	FIXED
gpu_iqn_head.rs (removed)	clone_to_device_f32_via_pinned → cos_features / online_taus / target_taus (3 CudaSlice fields)	MIGRATED	SP6 fix: converted to [MappedF32Buffer; 4] per-branch arrays (online_taus_per_branch, target_taus_per_branch, cos_features_per_branch). Constructor writes via write_from_slice (host_ptr direct); IQN kernels read via dev_ptr. mem::swap activate/deactivate replaced by active_branch_idx selection. upload_f32_via_pinned (which did an intermediate DtoD inside graph capture → STREAM_CAPTURE_INVALIDATED) fully eliminated.	FIXED
gpu_iqn_head.rs:498	malloc_host → total_loss_pinned	OK-pinned	Constructor: GPU writes IQN loss; host reads via read_total_loss()	OK
gpu_iqn_head.rs:534	malloc_host → t_pinned	OK-pinned	Constructor: Adam step counter — CPU increments, GPU reads via dev ptr	OK
gpu_iqn_head.rs:548	malloc_host → tau_pinned	OK-pinned	Constructor (Fix 3): tau scalar — CPU writes, GPU reads via tau_dev_ptr; replaces CudaSlice<f32> + cuMemcpyHtoDAsync_v2	OK
gpu_iqn_head.rs:807	cuMemcpyDtoDAsync_v2 (DtoD)	OK-pinned	Device-to-device copy inside graph-captured target-H forward cache; no host crossing	OK
gpu_iqn_head.rs:991	cuMemcpyDtoDAsync_v2 (DtoD)	OK-pinned	Device-to-device copy for cached target states; no host crossing	OK
gpu_iqn_head.rs:1564	stream.clone_dtoh → total_loss	COLD-PATH	read_loss(): synchronous fallback for epoch-end logging only; read_total_loss() (pinned) is the hot-path equivalent	OK
gpu_iqn_head.rs:1602	cuMemcpyDtoH_v2 (sync) in compute_iqr	MIGRATED	Fix 2A: compute_iqr() removed from submit_aux_ops; now called at epoch boundary via refresh_iqn_iqr() only; sync DtoH cannot be captured in CUDA graph — was silently a no-op on replayed steps	FIXED
gpu_iqn_head.rs:1622	cuMemcpyHtoDAsync_v2 in compute_iqr	MIGRATED	Fix 2B: same relocation — async HtoD with stale IQR data was being replayed; now runs once at epoch boundary	FIXED
gpu_iqn_head.rs (removed)	cuMemcpyHtoDAsync_v2 in target_ema_update for tau_buf	MIGRATED	Fix 3: replaced CudaSlice<f32> + async copy with pinned+device-mapped page; CPU writes *tau_pinned = tau; kernel reads via tau_dev_ptr — zero PCIe overhead	FIXED
gpu_iqn_head.rs:2058	stream.clone_htod → params_buf constructor	COLD-PATH	init_iqn_head_from_weights(): one-shot weight upload at construction/load; not on hot path	OK
gpu_iqn_head.rs:2107	stream.clone_dtoh + stream.clone_htod in clone_cuda_slice	COLD-PATH	clone_cuda_slice(): called during checkpoint save/load only	OK
fused_training.rs:476	stream.memcpy_htod → src_gpu (PER source indices)	COLD-PATH	prepare_per_batch(): per-epoch PER batch preparation — not per-step	OK
fused_training.rs:482	stream.memcpy_htod → ones_gpu	COLD-PATH	prepare_per_batch(): per-epoch importance weights init	OK
fused_training.rs:2914	stream.clone_dtoh → state inspection	COLD-PATH	eval_state_debug(): debug helper, no hot-path caller	OK
fused_training.rs:2928	stream.clone_dtoh → param inspection	COLD-PATH	eval_state_debug(): debug helper	OK
fused_training.rs:2955	cuMemcpyDtoH_v2 (sync)	COLD-PATH	eval_v_range(): called only at fold boundary / epoch-0 init; explicit sync is documented and acceptable at boundary	OK
training_loop.rs:220	collector.stream().memcpy_dtoh → PER scores	COLD-PATH	Pre-epoch PER priority init — runs once before the step loop starts	OK
training_loop.rs:409	stream.memcpy_htod → episode_starts_buf	COLD-PATH	Per-epoch experience setup — before run_training_steps_slices begins	OK
training_loop.rs:1003	clone_htod_f32 → targets device buffer	COLD-PATH	Data loading: DBN targets uploaded before step loop	OK
training_loop.rs:1007	clone_htod_f32 → features device buffer	COLD-PATH	Data loading: feature matrix uploaded before step loop	OK
tau_update_kernel.cu (Plan 1 Task 13)	single-thread kernel: reads ISV[39,40,12]; writes ISV[42]	COLD-PATH	Per-epoch boundary call from training_loop.rs; 1-thread, 1-block; zero hot-path overhead	OK
epsilon_update_kernel.cu (Plan 1 Task 14)	single-thread kernel: reads ISV[39,40,12]; writes ISV[41]	COLD-PATH	Per-epoch boundary call from training_loop.rs; 1-thread, 1-block; zero hot-path overhead	OK
gamma_update_kernel.cu (Plan 1 Task 10)	single-thread kernel: reads ISV[12]; writes ISV[43]	COLD-PATH	Per-epoch boundary call from training_loop.rs; 1-thread, 1-block; ISV[43] read by fill_gamma_buf on hot path via read_isv_signal_at (pinned, zero-copy)	OK
kelly_cap_update_kernel.cu (Plan 1 Task 11)	single-thread kernel: reads ISV[12] + portfolio_states[n_envs, PS_STRIDE]; aggregates mean half-Kelly × health-safety; writes ISV[44]	COLD-PATH	Per-epoch boundary call from training_loop.rs; 1-thread, 1-block; portfolio_states dev ptr passed from GpuExperienceCollector without CPU roundtrip	OK
atoms_update_kernel.cu (Plan 1 Task 9)	4-block kernel (one per branch): reads ISV[23..31) + spacing_raw params[52..56); writes atom_positions_buf[4 * num_atoms] via softmax + cumsum	COLD-PATH	Per-epoch boundary + SGD step (every 50 steps); grid=(4,1,1), block=(256,1,1); replaces CPU loop that launched adaptive_atom_positions 4× separately	OK
gpu_portfolio.rs:174	actions_buf upload (per simulate_batch_gpu)	MIGRATED	Promoted actions_buf from CudaSlice<i32> to MappedI32Buffer(MAX_BATCH_SIZE) allocated once at constructor. Per-call host writes go through host_ptr; kernel arg switched to dev_ptr u64. Zero HtoD copy on the simulate hot path.	FIXED
gpu_portfolio.rs:122,296	portfolio_state_buf init/reset	MIGRATED	env_step kernel writes back to portfolio_state_buf, so the destination must remain a CudaSlice<f32>. Init/reset now stage via a temporary MappedF32Buffer + memcpy_dtod_async (new helper upload_portfolio_state).	FIXED
decision_transformer.rs:913	scratch.adam_t upload per train_step_gpu	MIGRATED	HOT path: 4-byte step counter was uploaded via stream.memcpy_htod every backward pass. Promoted DtScratch.adam_t from CudaSlice<i32>[1] to MappedI32Buffer(1). Per-step host write goes through host_ptr; Adam kernel reads via dev_ptr.	FIXED
decision_transformer.rs:478	DT params init upload (constructor)	MIGRATED	params is mutated by Adam, so destination must remain CudaSlice<f32>. Xavier-init upload now stages via MappedF32Buffer + memcpy_dtod_async through new helper upload_host_to_cuda_f32.	FIXED
decision_transformer.rs:501	DT wd_mask init (constructor)	MIGRATED	wd_mask is read-only by GPU. Promoted from CudaSlice<f32> to MappedF32Buffer. CPU writes 1.0s through host_ptr; kernel reads via dev_ptr.	FIXED
decision_transformer.rs:1105	trajectory build episode_starts	MIGRATED	One-time per pre-training data build (cold path). Replaced clone_htod with MappedI32Buffer + direct host_ptr write; kernel reads via dev_ptr.	FIXED
decision_transformer.rs:1147	trajectory build ep_rewards	MIGRATED	Same trajectory builder cold path. Replaced htod_f32 with MappedF32Buffer; CPU writes via host_ptr, return_to_go kernel reads via dev_ptr.	FIXED
gpu_experience_collector.rs:2992	hindsight bar_indices upload (per collect_experiences_gpu)	MIGRATED	HOT path: alloc + memcpy_htod ran every collect when hindsight active. Added persistent bar_indices_pinned: MappedI32Buffer(alloc_episodes * alloc_timesteps * 2) allocated at constructor. Per-call host writes go through host_ptr; kernel arg switched to dev_ptr u64.	FIXED
gpu_experience_collector.rs:3186	feature_mask upload (per-collect at t==0)	MIGRATED	HOT path: was alloc + memcpy_htod every epoch. Promoted feature_mask_buf from Option<CudaSlice<f32>> to Option<MappedF32Buffer>. Reallocates only when mask size changes (still a cold-path event); CPU writes via host_ptr, state_gather reads via dev_ptr.	FIXED
gpu_experience_collector.rs:3097	episode_starts upload (per-collect)	MIGRATED	episode_starts_buf is GPU-mutated by domain_rand_episode_starts, so the destination must remain a CudaSlice. Stage via MappedI32Buffer + memcpy_dtod_async through upload_host_to_cuda_i32_via_pinned.	FIXED
gpu_experience_collector.rs:2009	upload_expert_actions (warm)	MIGRATED	Promoted expert_actions_gpu from Option<CudaSlice<i32>> to Option<MappedI32Buffer>. Direct host_ptr write replaces the alloc + memcpy_htod pair.	FIXED
gpu_experience_collector.rs:3816	update_per_sample_support (per-epoch)	MIGRATED	per_sample_support_buf is read-only by GPU; promoted from CudaSlice<f32> to MappedF32Buffer. Compute_expected_q + quantile_q_select now receive dev_ptr u64 kernel args.	FIXED
gpu_experience_collector.rs:1076,3848	portfolio_states init/reset (cold)	MIGRATED	portfolio_states is GPU-mutated by env_step. Init/reset stage via MappedF32Buffer + memcpy_dtod_async through upload_host_to_cuda_f32_via_pinned.	FIXED
gpu_experience_collector.rs:1093	epoch_state init (cold)	MIGRATED	epoch_state is GPU-mutated. Replaced clone_htod_f32 with explicit alloc_zeros + upload_host_to_cuda_f32_via_pinned.	FIXED
gpu_experience_collector.rs:1341	saboteur_base init (cold)	MIGRATED	saboteur_base is GPU-mutated by saboteur_select. Replaced memcpy_htod with upload_host_to_cuda_f32_via_pinned.	FIXED
gpu_experience_collector.rs:2784	trade_stats_buf zero (per-epoch)	MIGRATED	The htod_f32 of an all-zeros vector was needlessly going through PCIe. Replaced with stream.memset_zeros — fully GPU-side.	FIXED

Summary

Class	Count
OK-pinned	31
COLD-PATH	20
MIGRATED (fixed in this commit)	4
Remaining MIGRATE (unfixed)	0

All MIGRATE sites resolved. Zero hot-path memcpy calls remain in the DQN training loop.

Fixes Applied (this commit)

Fix 1 — Remove dead async DtoH in run_causal_intervention_unconditional

gpu_dqn_trainer.rs: cuMemcpyDtoHAsync_v2 from causal_mean_scratch_ptr to readback_pinned[10] was removed. The only reader (run_causal_intervention, which has no callers) was never invoked, making this copy dead code. The result now stays on the GPU device buffer where the kernel leaves it.

Fix 2 — Relocate compute_iqr() from hot path to epoch boundary

fused_training.rs / training_loop.rs: iqn.compute_iqr() was being called inside submit_aux_ops, which is captured in the CUDA graph (aux_child). The sync cuMemcpyDtoH_v2 inside compute_iqr cannot be captured in a CUDA graph — it ran during capture only and was silently skipped on every replayed step. The subsequent cuMemcpyHtoDAsync_v2 (uploading IQR back) was therefore replaying stale data. Moved to refresh_iqn_iqr() called at epoch boundary in process_epoch_boundary.

Fix 3 — Convert tau_buf from device VRAM to pinned+device-mapped

gpu_iqn_head.rs: tau_buf: CudaSlice<f32> + tau_host: f32 + cuMemcpyHtoDAsync_v2 replaced by tau_pinned: *mut f32 + tau_dev_ptr: u64 allocated via cuMemAllocHost_v2(DEVICEMAP) + cuMemHostGetDevicePointer_v2. CPU writes *tau_pinned = tau directly; the EMA kernel reads via tau_dev_ptr with no PCIe copy. Drop impl updated to free tau_pinned.

Fix 4 (perf) — Async-eval pipeline: split launch_metrics_and_download (2026-04-28)

gpu_backtest_evaluator.rs + trainer/metrics.rs + trainer/training_loop.rs: the prior merge 3c0d26292 (async-diag — eval on dedicated stream + spawn_blocking checkpoint, building on d9cb14f1b + 673b04a8d) wired the GPU side of cross-stream pipelining correctly (training stream cuda_stream, eval stream validation_stream, cuStreamWaitEvent barrier on train_done_event) but left a host-side stream.synchronize() inside launch_metrics_and_download that blocked the CPU thread for the FULL eval drain (~25-30s/epoch on L40S). The host thread is the same one that submits the next epoch's training kernels via run_full_step — so until eval drained, training submission was gated on it, and the dedicated validation_stream provided no actual wall-clock benefit.

Split into two record/consume halves at the buffer-readback layer:

• launch_metrics_and_record_event(&mut self) -> Result<(), MLError> — submits the metrics kernel + DtoD-async copies of actions_history_buf / intent_mag_buf / picked_action_history_buf into mapped-pinned mirrors (actions_history_pinned: MappedI32Buffer allocated in new(), the other two lazy-allocated alongside their device buffers in ensure_action_select_ready). Records eval_done_event (lazy-created on first launch via stream.context().new_event(None), re-recorded each epoch via event.record(stream)). Returns immediately — no host wait.
• consume_metrics_after_event(&self) -> Result<Vec<WindowMetrics>, MLError> — calls event.synchronize() (the SOLE host-side wait per epoch boundary), then read_volatiles the mapped-pinned metrics_buf for the per-window metrics. After the event syncs, the four action-distribution helpers (read_eval_action_distribution_per_*, read_eval_intent_magnitude_distribution, read_chunked_actions_direction_distribution) read directly from the now-coherent mapped-pinned mirrors — no memcpy_dtoh per call (the prior implementation allocated a fresh Vec<i32> of size n_windows * max_len and did a synchronous DtoH each call, which violates feedback_no_htod_htoh_only_mapped_pinned.md AND added 4 extra host-side stalls beyond the metrics one). All four mirrors land in the same event-sync barrier as the metrics readback.

Caller migration (feedback_no_partial_refactor.md):

• evaluate_dqn_graphed (existing public API, used by hyperopt + smoke test) becomes a thin synchronous wrapper: launch_async → consume. Public ABI unchanged.
• New evaluate_dqn_graphed_async — launch-only sibling for the pipelined path. Returns ().
• evaluate / evaluate_ppo / evaluate_supervised migrated inline to launch+consume — no caller pipelines them, so no behavioural change.
• compute_validation_loss removed; replaced by launch_validation_loss (returns ()) + consume_validation_loss (returns f64 and emits the HEALTH_DIAG val [...] / val_dir_dist / val_picked_dir_dist lines).
• training_loop.rs's validation reporting block now: epoch-start consumes any pending launch into cached_async_val_loss, epoch-end reports the cached value and submits a fresh launch with pending_val_loss = Some(0.0) as the "launched" sentinel. Bootstrap (epoch 0 / no validation_stream) does sync launch+consume. After the for-epoch loop terminates, a final consume drains the LAST epoch's launch so smoke tests / hyperopt that read last_eval_direction_dist() see the most recent epoch's data, not the second-to-last.

Mathematical identity preserved: every consumed val_sharpe is bit-identical to what the prior compute_validation_loss would have produced for the same epoch — the only change is when the host parses it (one epoch later, which matches the existing cached_async_val_loss lag semantics; HEALTH_DIAG val [...] emit moves with the consume). Expected wall-clock saving: ~25-30s/epoch on L40S.

Out-of-scope (remaining host-side stream syncs in eval, not part of this fix): gpu_backtest_evaluator.rs lines ~1596 + ~1863 — periodic self.stream.synchronize() calls every 10 chunks inside submit_dqn_step_loop_cublas for kernel-error detection. They block the host CPU thread but only for the chunked rollout duration of ~5 calls × per-chunk-residency-time per epoch (each catches deferred CUDA errors within 10 chunks of origin). Smaller in aggregate than the launch_metrics_and_download final wait this fix removed; future work to convert to async error-polling (e.g., eval_done_event.is_complete() polling from training submissions). Not adjacent to the perf-fix scope.

Fix 5 — Add mapped-pinned staging helpers in mapped_pinned.rs

mapped_pinned.rs: added clone_to_device_{f32,i32}_via_pinned and upload_{f32,i32}_via_pinned helpers that allocate a temporary MappedXxxBuffer (cuMemHostAlloc DEVICEMAP), write the payload via host_ptr, then async DtoD into a target CudaSlice<T>. Used to migrate cold-path constructor uploads off explicit cudaMemcpy HtoD per feedback_no_htod_htoh_only_mapped_pinned.md while preserving struct field types (e.g. portfolio_buf, weight buffers consumed by Adam) where the kernel mutates the buffer every step and device-resident memory is correct.

Fix 6 — Migrate cuda_pipeline/mod.rs callers to mapped-pinned

cuda_pipeline/mod.rs: 7 in-file callers of clone_htod_f32 in DqnGpuData::upload_slices, DqnGpuData::upload_ofi, DqnGpuData::build_batch_states (portfolio), DqnGpuData::htod_copy_into, and PpoGpuData::upload rewritten to use mapped_pinned::clone_to_device_f32_via_pinned. The htod_f32 / clone_htod_f32 helper bodies at lines 129-145 remain in place because other crates are still in mid-migration; they will be deleted once all consumers migrate.

Fix 7 — Migrate gpu_walk_forward.rs to mapped-pinned (3 sites)

gpu_walk_forward.rs: 3 clone_htod_f32 calls in the walk-forward GPU data upload (features, targets, OFI) rewritten to mapped_pinned::clone_to_device_f32_via_pinned.

Fix 8 — Migrate PPO trainer + hyperopt adapters to mapped-pinned (6 sites)

trainers/ppo.rs (2 sites: features, targets in set_raw_market_data), hyperopt/adapters/ppo.rs (3 sites: features, targets in upload path; action_indices in backtest forward closure), hyperopt/adapters/mamba2.rs (1 site: gpu_to_stream). All clone_htod_f32 and bare stream.clone_htod calls rewritten to mapped_pinned::clone_to_device_{f32,i32}_via_pinned.

Fix 9 — Migrate weight-upload sites in attention/tlob/iql/weights (4 sites)

gpu_weights.rs (1 site: RmsNormWeightSet::ones constructor), gpu_attention.rs (1 site: attention params constructor), gpu_tlob.rs (1 site: TLOB params constructor — mod tests blocks intentionally untouched per scope), gpu_iql_trainer.rs (2 sites: per_sample_support seed, IQL params init). All htod_f32 calls rewritten to mapped_pinned::upload_f32_via_pinned; the gpu_weights.rs ones-init uses clone_to_device_f32_via_pinned since the destination is freshly allocated.

Fix 10 — Migrate gpu_iqn_head.rs constructors + DtoD-only target clone (5 sites)

gpu_iqn_head.rs: 4 stream.clone_htod calls (cos_features precompute, online_taus + target_taus tile broadcast, init_iqn_xavier_weights upload) rewritten to mapped_pinned::clone_to_device_f32_via_pinned. Additionally the helper clone_cuda_slice was replaced — it previously did device→host (clone_dtoh) → host→device (clone_htod) round-trip, double-violating both the no-DtoH and no-HtoD rules. Now uses clone_cuda_slice_f32 (single async DtoD), the only callsite being the target-network seed at constructor time (target_params = clone_cuda_slice(&online_params)).

Fix 11 — Migrate gpu_her.rs to mapped-pinned + add MappedU32Buffer (3 sites)

gpu_her.rs: 1 cold-path constructor stream.memcpy_htod (rng_states u32 init) and 2 warm-path memcpy_htod (source_indices, donor_indices in relabel_batch) rewritten to mapped_pinned::upload_{u32,i32}_via_pinned. The warm sites re-allocate pinned staging per relabel call — acceptable on this cold path (relabel runs once per epoch). New MappedU32Buffer type added to mapped_pinned.rs mirroring MappedI32Buffer; carries a manual Debug impl so the workspace warning count stays at 13.

Fix 12 — Migrate gpu_backtest_evaluator.rs to mapped-pinned (5 sites)

gpu_backtest_evaluator.rs: 5 HtoD sites in GpuBacktestEvaluator::new + reset_evaluation_state rewritten to mapped-pinned staging. Sites: prices_buf (f32), features_buf (f32 — was already going through clone_htod_f32 helper), window_lens_buf (i32), portfolio_buf init (f32), and the per-evaluation reset of portfolio_buf. The fields stay typed CudaSlice<T> because the kernel mutates portfolio_buf each step (must remain device-resident); the read-only fields could become MappedXxxBuffer but keeping them as CudaSlice preserves the kernel-launch contract .arg(&self.field) already used at the many launch sites in the file.

Fix 13 — Migrate gpu_ppo_collector.rs to mapped-pinned (6 sites + persistent staging)

gpu_ppo_collector.rs: all 6 HtoD sites migrated.

• Constructor (cold): portfolio_states init (htod_f32) → upload_f32_via_pinned; rng_states init (memcpy_htod) → upload_u32_via_pinned.
• collect_experiences (warm but per-collect): episode_starts_buf (memcpy_htod) → upload_i32_via_pinned; barrier_config (htod_f32) → upload_f32_via_pinned.
• reset_episodes (warm, per-epoch): replaced the Vec<f32> / Vec<u32> host staging fields with persistent MappedF32Buffer / MappedU32Buffer allocated once at construction. Resets fill via host_slice_mut() (zero-allocation, zero-HtoD); a single async DtoD seeds the persistent device-resident portfolio_states / rng_states (which the kernel mutates each step). Adds host_slice_mut() accessor on both mapped-pinned types.

Fix 17 — Delete orphan htod_f32 / clone_htod_f32 helpers from cuda_pipeline/mod.rs (2026-04-28)

After Fix 1..16 migrated all 80+ production callers off super::htod_f32 and super::clone_htod_f32, the helper bodies in cuda_pipeline/mod.rs:129-145 were unused outside #[cfg(test)] modules. Deleted both function definitions per feedback_no_legacy_aliases.md (no deprecated wrappers).

The two surviving test-block callers in gpu_tlob.rs::tests (lines 1017 and 1132 — super::super::htod_f32(&stream, &host_states, …) and super::super::htod_f32(&stream, &host_d_concat, …)) were also migrated to mapped_pinned::upload_f32_via_pinned in the same commit per feedback_no_partial_refactor.md (when a contract is deleted, every consumer migrates together — including tests). The test-only callers in signal_adapter.rs::tests (5 sites), gpu_action_selector.rs::tests (1), and cuda_pipeline/mod.rs::tests (1) all use bare stream.memcpy_htod / stream.memcpy_stod against the cudarc handle directly, not the deleted helpers — no change needed.

A doc-block was added at the deletion site in mod.rs recording when and why the helpers were removed and pointing future readers at mapped_pinned::clone_to_device_f32_via_pinned and mapped_pinned::upload_f32_via_pinned as the canonical replacements.

Final state of the HtoD migration sequence:

• grep -rEn "stream\.memcpy_htod|memcpy_stod|stream\.htod_copy|htod_async" crates/ml/src/: production count 0, test count 7 (5 in signal_adapter.rs::tests, 1 in gpu_action_selector.rs::tests, 1 in cuda_pipeline/mod.rs::tests).
• grep -rn "htod_f32\|clone_htod_f32" crates/ml/src/: outside test blocks 0 (only doc-comment text references remain). Helper definitions removed from mod.rs.

Fix 16 — Migrate training_loop.rs raw features/targets uploads (2 sites, 2026-04-28)

trainers/dqn/trainer/training_loop.rs:1250 (targets_raw) and :1254 (features_raw): both crate::cuda_pipeline::clone_htod_f32 callsites in the data-load helper rewritten to mapped_pinned::clone_to_device_f32_via_pinned. These run once per fold immediately before the step loop begins (the _raw suffix indicates the un-normalized arrays kept on-GPU for the eval-time critic). The .map_err(|e| anyhow::anyhow!("…raw upload: {e}")) wrapping is preserved verbatim — the helper already returns Result<_, String> which feeds anyhow::anyhow! directly, so the change is a 1:1 path swap.

Fix 15 — Migrate gpu_experience_collector.rs::upload_ofi_features (1 site, 2026-04-28)

gpu_experience_collector.rs:4258: warm-path super::clone_htod_f32 upload of OFI features rewritten to mapped_pinned::clone_to_device_f32_via_pinned. Called from training_loop's data-load phase before each fold's training run, so the OFI tensor (4M bars × 32 dims = ~512 MB) now stages through mapped-pinned + DtoD just like all other large data uploads. Error-type shift handled with .map_err(|e| MLError::ModelError(format!("upload_ofi_features: {e}")))? since the helper returns Result<_, String> and the function returns Result<_, MLError>.

Fix 14 — Migrate gpu_dqn_trainer.rs residual COLD ctor sites (8 sites, 2026-04-28)

gpu_dqn_trainer.rs: residual COLD ctor sites missed by the original Fix-1..13 audit are now migrated off explicit super::htod_f32 per feedback_no_htod_htoh_only_mapped_pinned.md.

• :9922 sel_clip_buf 1-element init — selectivity gate clip-norm seed (1.0).
• :10075-10078 spectral-norm init_u_s1 / init_v_s1 / init_u_s2 / init_v_s2 (4 separate uploads of trunk-encoder spectral u/v vectors).
• :10095-10096 alloc_spec_pair! macro body — both u/v init uploads inside the macro now go through upload_f32_via_pinned. The macro's $lbl_u / $lbl_v are reused in the error-message format so per-pair failures (spec_u_v1, spec_v_v1, spec_u_a1, …) stay diagnostically distinct.
• :11276 graph_params_host (60 floats: cross-branch graph message passing weights, 4 edges × 15 params each, near-identity W_msg init).
• :11330 denoise_params_host (1800 floats: 2-step diffusion Q-refinement Xavier init for W1[24,24] + W2[12,24]).
• :11476 qlstm_weights_host (528 floats: QLSTM Xavier init for 6 gates × 11 input × 8 head). All sites use mapped_pinned::upload_f32_via_pinned (canonical mapped-pinned + DtoD staging helper). The helper returns Result<_, String> whereas the ctor returns Result<_, MLError>, so each site wraps the error via .map_err(|e| MLError::ModelError(format!("<site> upload via pinned: {e}"))). Site labels preserved so backtraces remain readable.

Fix 18 (perf) — Eliminate per-step val gather DtoD copies via chunked gather kernel (2026-04-28)

experience_kernels.cu + gpu_backtest_evaluator.rs + metrics.rs: nsys profile of multi_fold_convergence on L40S identified backtest_state_gather as the #1 GPU-time consumer at 37.2% (596 ms / 60,588 calls — kernel launch latency dominated compute). The per-step pattern was:

for step_offset in 0..chunk_len {
  launch_gather(step) -> states_buf[N, padded_sd]   (1 launch)
  if tlob: tlob.forward(states_buf, n)              (4 cuBLAS SGEMMs + 3 kernel launches)
  memcpy_dtod_async(chunked_states_buf[step_offset*N..], states_buf, ...)   (1 DtoD)
}

Per chunk (DQN_BACKTEST_CHUNK_SIZE=512): 512 gather launches + 512 DtoD copies + 512 × 7 TLOB launches = 4608 kernel launches per chunk before forward Q-values can begin.

New kernel backtest_state_gather_chunk writes [chunk_len, N, padded_sd] directly into chunked_states_buf in a SINGLE launch (chunk_len * N threads, 1 thread per output row). Within a chunk portfolio_buf and plan_isv_buf are CONSTANT — env_step (Phase 5) and backtest_plan_state_isv (Phase 6) update them only at chunk boundary. So the batched gather is mathematically identical to the prior per-step + DtoD pattern; each thread reads its own (window, step_offset) row from independent feature offsets and writes its own state row. No reordering, no atomics, no shmem races.

Same change enables the val TLOB instance to run ONCE per chunk on the chunked buffer at batch = chunk_len * N instead of chunk_len separate forward calls each at batch = N. Val TLOB construction in metrics.rs::reset_evaluator_for_validation resized via new GpuBacktestEvaluator::val_tlob_batch_size() to DQN_BACKTEST_CHUNK_SIZE * n_windows; partial last chunks reuse the same buffers (forward(b) accepts any b <= construction_batch).

Per-chunk launch reduction: from 2 * chunk_len + chunk_len * 7 to 1 + 7 for the gather+TLOB phase. With 512-step chunks that's 4608 → 8, a 576x reduction. Across the full smoke run (60K gather calls in profile), expected wall-clock saving on the gather/TLOB hot path is the bulk of the 37% + 6.5% + 5.2% + 4.8% = ~53.7% of GPU time the per-call cluster occupied.

Borrow restructure: the prior single-shot let ch_states = self.chunked_states_buf.as_ref()? at top of submit_dqn_step_loop_cublas was removed because TLOB now needs &mut self.chunked_states_buf. Replaced with per-chunk ch_states_base (raw u64 device pointer extracted in a tight scope), reused for both launch_gather_chunk and the Phase 2+3 compute_q_values_to call. The pointer is stable across the chunk because the Option<CudaSlice<f32>> does not reallocate.

No new HtoD/DtoH paths introduced. Strictly removes 512 per-chunk DtoD copies. Per feedback_no_partial_refactor.md: kernel ABI is new (separate function name), prior backtest_state_gather retained for evaluate() / evaluate_ppo() / evaluate_supervised() paths that still need a single-step writer.

Fix 19 (perf) — Online softmax in compute_expected_q: 3-pass → 2-pass (2026-04-28)

experience_kernels.cu: nsys profile of multi_fold_convergence on L40S identified compute_expected_q as the #3 GPU consumer at 12.9% (207 ms / 1382 calls — ~150 µs per call, compute-bound at typical batch=8192 × num_atoms=51). The kernel computed each per-action softmax over atoms in 3 passes (max → sum_exp → normalize+expected/var/entropy/util), re-reading v_row[z] + adv_a[z] 3 times per atom for each of the 4×~3.25 = 13 actions per sample.

Replaced with online softmax (Page-Olshen running-max + running-sum) so expected_q, sum_z_sq, and entropy all accumulate in a single forward pass over atoms. Atom utilisation still needs a 2nd pass because it depends on the final S = sum exp(logit-M) being known to compute prob_z = exp(logit_z - M)/S for the threshold test. Total: 3 → 2 passes, ~33% reduction in inner-loop global loads of branch advantage logits (which dominate because v_row[z] is reused across 13 (action, branch) pairs and may stay in L1, while adv_a[z] flips per action and is pure global).

Online accumulation:

M, S, TZ, TZ², TLM = -inf, 0, 0, 0, 0
for z in 0..num_atoms:
    logit = v_row[z] + adv_a[z]
    if logit > M:
        scale = exp(M - logit)   # ≤ 1, no overflow
        S, TZ, TZ², TLM *= scale
        M = logit
    w = exp(logit - M)            # ≤ 1
    S   += w
    TZ  += w * z_val
    TZ² += w * z_val²
    TLM += w * (logit - M)
expected_q = TZ / S
sum_z_sq   = TZ² / S
entropy    = log(S) - TLM / S    # analytical: -sum(p log p) where p = w/S

Correctness: bit-stable when atoms processed in fixed order (z = 0..num_atoms-1). The analytical entropy form is mathematically equivalent to -sum(p log p):

-sum(p log p) = -sum(p (logit - M - log S)) = log(S) * sum(p) - sum(p (logit-M))
             = log(S) - (1/S) * sum(exp(logit-M) * (logit-M))
             = log(S) - TLM / S

The previous code's if (prob > 1e-10f) underflow guard is no longer needed: exp(logit - M) underflows cleanly to 0 when logit << M, and the analytical form does not multiply tiny probs by very negative logs. Result is bit-stable per feedback_stop_on_anomaly.md — no fuzzy tolerance change.

Atom-stat block-sum reduction unchanged (warp shfl + shared-mem bank, deterministic).

Expected wall-clock saving on this kernel: ~33% of 207ms = ~70ms across the smoke run; on production batch=8192 × 1382 calls per fold, roughly proportional. Lower-bound estimate; actual saving depends on memory bandwidth vs compute mix — at low N (smoke) the kernel may be launch-bound rather than load-bound. nsys re-profile after deployment will quantify.

ABI unchanged; no Rust caller changes required.

Fix 20 — DtoD-via-pinned guard + orphan HER deletion (2026-05-02)

scripts/pre-commit-hook.sh + crates/ml/src/cuda_pipeline/gpu_her.rs. Two related changes installing the structural guard against the SP6 Pearl 5 IQN τ failure mode (root cause fixed at facbf76eb for that one site) and removing the only remaining orphan callers of the broken pattern.

The bug class. The mapped_pinned::{upload,clone_to_device}_{f32,i32}_via_pinned helpers carry the _via_pinned suffix to suggest "no HtoD per feedback_no_htod_htoh_only_mapped_pinned", but their bodies do MappedXBuffer::new() (staging) + memcpy_dtod_async() + stream.synchronize(). The DtoD copy and synchronize are both forbidden inside CUDA Graph capture (CUDA_ERROR_STREAM_CAPTURE_INVALIDATED — exactly the failure pattern in pre-facbf76eb Pearl 5 smokes) and add a host stall otherwise. The "_via_pinned" name was a lie. The canonical pattern is MappedXBuffer field stored directly + write_from_slice for host writes + kernel reads via .dev_ptr — used by SP4 portfolio_state, SP4 grad_prev_buf_per_group, and SP6 IQN τ at facbf76eb.

Guard. New check_no_dtod_via_pinned function in pre-commit-hook.sh rejects any staged .rs file that calls upload_(f32|i32)_via_pinned or clone_to_device_(f32|i32)_via_pinned, with a single exception for mapped_pinned.rs itself (where the helpers currently live). Per feedback_no_hiding: no suppression marker. The error message points at the structural fix (rewrite to MappedXBuffer directly).

Also fixes a pre-existing silent skip: the gpu-hotpath-guard.sh invocation used SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" which resolved to .git/hooks/ (the symlink's directory) instead of scripts/ (the script's actual location), so the guard never ran. Replaced with readlink -f "$0" to resolve through the symlink chain. Added an explicit else branch that errors if the guard binary is missing — silent skip is worse than no guard.

Orphan deletion. gpu_her.rs carried legacy pub fn relabel_batch(...), pub fn generate_random_donors(...) (CPU version), pub struct HerBatch, fn slice_clone_f32, fn slice_clone_i32 — confirmed zero callers via:

grep -rn '\.relabel_batch(' --include='*.rs' | grep -v 'with_strategy\|_gpu\b'
grep -rn 'HerBatch\|slice_clone_f32\|slice_clone_i32' --include='*.rs'
grep -rn '\.generate_random_donors(' --include='*.rs' | grep -v '_gpu\b'

Production uses relabel_batch_with_strategy(...) (line 384, called at fused_training.rs:1555) which does pure-DtoD donor selection from a pre-existing GPU buffer — no *_via_pinned calls. The orphan held the only upload_i32_via_pinned callers in the codebase (lines 279, 285). Per feedback_no_hiding (wire-or-delete) the right fix is delete; conversion would have been adding MappedI32Buffer machinery to dead code.

Also removed the now-unused DevicePtr import and updated the module docstring header to reference relabel_batch_with_strategy + generate_random_donors_gpu (the surviving production paths).

Scope. This commit eliminates 2 of the 47 production *_via_pinned call sites (the two inside the deleted orphan). The remaining 45 sites across 14 files are cold-path init (constructors, per-fold setup, hyperopt adapters, walk-forward backtest setup) — they don't break operationally because they're outside any captured graph, but they're still graph-capture-fragile and host-stalling. The guard now enforces no NEW calls; the existing 45 will be migrated in subsequent atomic per-buffer commits. After all 45 are converted, the four helpers themselves get deleted from mapped_pinned.rs.

Validation. Smoke smoke-test-82fjk at the τ-buffer-fix HEAD facbf76eb succeeded (phase=Succeeded progress=1/1, 5-epoch L40S multi_fold_convergence). Magnitude differentiation restored (q_full=0.462 > q_half=0.409 > q_quarter=0.350 vs baseline frozen Pascal-triangle 0.225/0.280/0.495). Eval distribution unfrozen (eq=0.596, eh=0.404, ef=0.000 vs baseline single-action collapse). Validates that the SP5+SP6 architecture works once the τ buffer DtoD violation is gone.

Fix 21 — fxcache target schema fix + Bug 2 state[0] instrumentation (2026-05-02)

Two related changes addressing the EVAL_DIST=Hold collapse observed in train-multi-seed-hc6kt 50-epoch validation (terminated epoch 5).

Bug 1 — fxcache target column semantics violation (FIXED). precompute_features.rs:360 and data_loading.rs:510 both stored raw OHLCV close prices in target slots [0:1] (preproc_close / preproc_next), violating the documented schema in experience_kernels.cu:1556 and cuda_pipeline/mod.rs:508. Empirically confirmed via direct fxcache inspection: target[0..4] had mean=$5967, stddev=$582 — raw prices throughout. The contract specifies log-return-normalized values for the network-input columns. Both writers now compute (raw_curr / prev_close).ln() and (raw_next / raw_curr).ln() for the preproc columns. FXCACHE_VERSION bumped 7 → 8 to invalidate existing caches and force regen via the ensure-fxcache Argo step. The raw-price values at target[0:1] were never directly consumed by training (aux head reads next_states[i][0] = MARKET feat[0] = log_return), but they corrupted any downstream code that read targets per the documented contract.

Bug 2 — state[0]=raw_close suspected; instrumented for runtime confirmation. Production runs show aux_label_scale=5300 (raw price magnitude) but the source path traces back through next_states_buf[i][0] ← experience_state_gather(market_features[bar_idx][0]) ← fxcache feat[0] (z-normalized, stddev=1.0). The discrepancy can't be resolved by static code reading alone — there must be a runtime path that injects raw_close into state[0] between the gather and the aux read. Added one-shot debug print in training_loop.rs after first collect_experiences_gpu return: downloads first 5 samples × 5 columns of both gpu_batch.states and gpu_batch.next_states, computes mean/std/mean_abs over col 0, prints under target: "DIAG_BUG2". Triggers once via static AtomicBool. f32 dtoh download is slow but pre-graph-capture so no hot-path impact. Interpretation key embedded in the log line: mean_abs ~0.001 = log_return ✓ | mean_abs ~5000 = raw price ✗.

Why the smoke test passes but production fails. The smoke harness multi_fold_convergence::test_multi_fold_convergence runs the same DQN trainer but bypasses the production train_baseline_rl rollout machinery. The smoke at facbf76eb showed label_scale0.1 (consistent with normalized log_return). The production binary at the same commit shows label_scale5300 (raw price). One of the production-only code paths is mutating state[0] post-gather. Bug 2 instrumentation will pin the exact mutation site at next L40S run.

Fix 22 — features_raw_cuda + targets_raw_cuda → MappedF32Buffer (Bug 2 instrumentation also lands) (2026-05-02)

Cross-file structural conversion that closes the last two clone_to_device_f32_via_pinned call sites in the DQN production path (training_loop.rs:1308 + 1312, the residue from Fix 16's pre-guard pattern). Mirrors the SP6 IQN τ buffer fix at facbf76eb (audit-tracked under task #283) and the SP4 portfolio_state pearl pattern: MappedF32Buffer field stored directly + write_from_slice for host load + kernel reads via .dev_ptr. No DtoD copy, no synchronize, graph-capture safe.

What changed.

• trainers/dqn/trainer/mod.rs — targets_raw_cuda and features_raw_cuda field type Option<CudaSlice<f32>> → Option<MappedF32Buffer>.
• trainers/dqn/trainer/training_loop.rs::init_gpu_raw_buffers_from_slices — replaces both clone_to_device_f32_via_pinned calls with MappedF32Buffer::new(n) + write_from_slice. Pre-existing if self.cuda_stream.is_none() { return Ok(()) } guard preserved (the stream value itself isn't required by the mapped-pinned allocator, only an active CUDA context, which the cuda_stream-is-Some invariant implies).
• cuda_pipeline/gpu_experience_collector.rs — three signature changes propagated through the consumer chain: collect_experiences_gpu, launch_timestep_loop, compute_difficulty_scores. Five .arg(buf) launch sites (hindsight_relabel, state_gather, expert_override, env_step×1, env_step×2) rewritten to .arg(&buf.dev_ptr). targets.raw_ptr() in compute_difficulty_scores → targets.dev_ptr (same CUdeviceptr u64, different field name).
• cuda_pipeline/decision_transformer.rs::build_dt_trajectories — signature &CudaSlice<f32> → &MappedF32Buffer for both features_gpu and targets_gpu. Two raw_ptr_f32(buf, stream) call sites converted to buf.dev_ptr (mapped-pinned has no SyncOnDrop guard, so the _no_drop ManuallyDrop dance isn't needed). Doc-comment updated to reflect mapped-pinned semantics.
• trainers/dqn/trainer/training_loop.rs:3749 (SP5 Pearl 8 trail launch) — features_buf.len() (method) → features_buf.len (pub field); features_buf.device_ptr(stream) (StreamGuard idiom) → features_buf.dev_ptr direct. Removed unused DevicePtr import + _guard binding.

Bonus deliverable: Bug 2 diagnostic. Same commit lands the one-shot DIAG_BUG2 download at training_loop.rs:1768 (right after the first collect_experiences_gpu return). Static AtomicBool ensures it fires exactly once per process. Per-sample first-5-col rows + col-0 mean/stddev/mean_abs aggregates printed under target: "DIAG_BUG2". Interpretation key: mean_abs ~0.001 = normalized log_return ✓ | mean_abs ~5000 = raw price ✗. Uses clone_dtoh (current cudarc API; replaced the deprecated memcpy_dtov). DtoH is a graph-capture violation but this is single-shot pre-capture, so OK.

Adjacent buffers in the same commit (carried-over from prior agent attempts):

• gpu_dqn_trainer.rs::sel_clip_buf — single-element CudaSlice<f32> (sel-grad clip norm constant 1.0) → MappedF32Buffer. Aux Adam-update kernel reads clip_ptr = self.sel_clip_buf.dev_ptr. Eliminates the upload_f32_via_pinned constructor call.
• gpu_weights.rs::RmsNormWeightSet — four γ buffers (s0, s1, v, a; each is an all-ones init for RMSNorm γ scaling factor) CudaSlice<f32> → MappedF32Buffer. KernelWeightPack::new reads rmsnorm.gamma_*.dev_ptr directly (no raw_device_ptr(...) helper needed for mapped-pinned). Eliminates four clone_to_device_f32_via_pinned calls in RmsNormWeightSet::ones.

PPO trainer deferred. trainers/ppo.rs:221,223 carry the same field types; PPO is not on the eval-collapse hot path and is out of scope. Pre-commit guard checks staged files only, so PPO's existing *_via_pinned violations don't block. Tracked as a follow-up.

Validation. cargo check -p ml --offline clean (zero warnings post-memcpy_dtov → clone_dtoh migration). cargo build -p ml --release --offline clean. Pre-commit check_no_dtod_via_pinned passes (no *_via_pinned calls in any staged file). After this commit, the only remaining *_via_pinned callers in the entire ml crate are in trainers/ppo.rs and a long tail of cold-path init sites tracked under task #290.

Fix 23 — gpu_dqn_trainer.rs spectral-norm + aux-head ctor sites + gpu_experience_collector.rs OFI upload → MappedF32Buffer (10 sites, 2026-05-02)

Final structural sweep that closes the residual *_via_pinned ctor / cold-path init call sites in the two files still violating the pre-commit check_no_dtod_via_pinned guard after Fix 22. Same canonical pattern as the τ-buffer fix at facbf76eb and the Fix 22 features/targets conversion: MappedF32Buffer field stored directly + MappedF32Buffer::new(n) + write_from_slice for host load + kernel reads via .dev_ptr (or via spectral_norm_descriptors u64 table for the spec_u/spec_v cluster). No DtoD, no synchronize, graph-capture safe.

What changed (per buffer-migration group).

• Spectral-norm spec_u_s1 / spec_v_s1 / spec_u_s2 / spec_v_s2 (4 buffers). Trunk power-iteration vectors for w_a_h_s1 / w_a_h_s2 (Linear_a matrices in the GRN trunk). Field type CudaSlice<f32> → MappedF32Buffer. Constructor (gpu_dqn_trainer.rs ~14421-14431) replaces alloc_f32 + upload_f32_via_pinned with MappedF32Buffer::new + write_from_slice. Consumed only via .raw_ptr() → .dev_ptr in the host_desc[78] table that backs spectral_norm_descriptors — same CUdeviceptr u64 width, purely a structural rename.
• Spectral-norm head/branch macro alloc_spec_pair! (16 buffers). Macro at ~14443 is invoked 11 times to allocate value-head, advantage, and 4 branch-output spec_u/spec_v pairs (spec_u_v1/v_v1, spec_u_v2/v_v2, spec_u_a1/v_a1, spec_u_a2/v_a2, spec_u_bo1..bg2, spec_u_bn/v_bn). Macro body now allocates MappedF32Buffer::new + write_from_slice directly (no &mut $u_name upload helper needed — interior mutability via write_volatile). Same host_desc consumer chain: each .raw_ptr() → .dev_ptr.
• graph_params (60 floats, 1 buffer). Cross-branch graph message-passing edge params (4 edges × 15 floats = 60). Field type CudaSlice<f32> → MappedF32Buffer. Constructor (~15790) MappedF32Buffer::new(60) + write_from_slice(&graph_params_host). Single consumer at launch_graph_message_pass reads params_ptr = self.graph_params.dev_ptr.
• denoise_params (DENOISE_TOTAL_PARAMS=1800, 1 buffer). Diffusion Q-refinement MLP weights (2 steps × 900 = W1[576]+b1[24]+W2[288]+b2[12]). Field type CudaSlice<f32> → MappedF32Buffer. Constructor (~15845) Xavier init host-side then write_from_slice. Four consumer sites: launch_q_denoise, launch_q_denoise_backward, launch_q_denoise_backward_cublas, aux Adam update — all read params_ptr = self.denoise_params.dev_ptr. Notable: the kernel writes back to this buffer during aux Adam updates; mapped-pinned semantics are kernel-write-coherent the same way as kernel-read (single mapping, both directions through dev_ptr).
• qlstm_weights (QLSTM_WEIGHT_COUNT=528, 1 buffer). xLSTM mLSTM-cell parameters (6 weight matrices × [11, 8] = 528). Field type CudaSlice<f32> → MappedF32Buffer. Constructor (~15992) Xavier uniform host-side then write_from_slice. Two consumer sites: launch_qlstm_step, launch_qlstm_train_step — both read w_ptr = self.qlstm_weights.dev_ptr. Kernel SGD-updates in place via the same dev_ptr mapping.
• gpu_experience_collector.rs::ofi_gpu (1 buffer + 1 placeholder). Pre-uploaded OFI features (total_bars * OFI_DIM). Field type CudaSlice<f32> → MappedF32Buffer. Constructor places a 1-element zero placeholder (MappedF32Buffer::new(1)); upload_ofi_features rewrites the field with a fresh MappedF32Buffer sized to ofi_flat.len(), then write_from_slice(ofi_flat). Replaces the clone_to_device_f32_via_pinned call. The single per-step consumer at launch_timestep_loop's state_gather invocation reads ofi_ptr = self.ofi_gpu.dev_ptr.

Hot-path purity. None of the converted buffers are written per-step from the host; denoise_params and qlstm_weights are mutated by kernel-side aux-Adam / SGD updates which use the same dev_ptr mapping the read-side does. graph_params and the spec_u/spec_v cluster are static post-init (graph_params is intentionally not trained; spec vectors are updated by the spectral-norm power-iteration kernel via the descriptor table, again through dev_ptr). ofi_gpu is rewritten exactly once (cold path) via upload_ofi_features — re-allocating the full mapped-pinned buffer and replacing the field on each call is fine because the call frequency is one-per-fold, never per-step.

Validation. cargo check -p ml --offline clean. Pre-commit check_no_dtod_via_pinned passes — zero upload_*_via_pinned / clone_to_device_*_via_pinned callers in either of the two staged files. The Bug 2 instrumentation commit currently in flight (Fix 21 / Fix 22 chain) can now proceed past the pre-commit guard.

Out of scope. mapped_pinned.rs retains the upload_f32_via_pinned / upload_i32_via_pinned / clone_to_device_f32_via_pinned / clone_to_device_i32_via_pinned helper definitions — trainers/ppo.rs and a small set of remaining cold-path callers still depend on them; the helpers are deleted under task #290 once the call count globally hits zero. Per feedback_no_hiding no suppression markers were added.

Fix 26 — DBN spread-instrument filter (root cause of Bug 2 contamination) (2026-05-02)

Pinned the source of the 8,799 corrupt bars sanitize_bars caught.

Direct inspection of ES.FUT_2024-Q1.dbn.zst via databento Python client (offline, kubectl-cp from PVC):

Instrument type	Symbols	Records (Q1 file)	Price range
Outright (correct)	ESH4, ESM4, ESU4, ESZ4, ESH5	43,353 (76.87%)	$5,063 – $5,478
Calendar spreads (poison)	ESH4-ESM4, ESM4-ESU4, ESH4-ESU4, ESM4-ESZ4, ...	13,048 (23.13%)	$47.30 – $219.70

Calendar spreads trade at the price difference between adjacent contracts (~$60-150 due to cost-of-carry roll basis). Databento's stype_in=parent symbology (used for ES.FUT requests) returns BOTH outright contracts AND every calendar-spread combination in the same DBN stream.

Why the contamination leaked through: the legacy decode_ohlcv_records keyed records only by ts_event, then deduped by volume (highest-wins). Outrights typically dominate volume during regular hours, so most spread bars were dropped. But during low-liquidity overnight windows + active rollover periods, spreads hit higher minute-volume than the outright (samples show spread volumes up to 1554 contracts/min during ESH4→ESM4 transition vs near-zero outright volume in those minutes). 780 spread bars survived dedup in Q1 alone; ~8,800 across the full 2024-2026 dataset.

The fix (baseline_common/mod.rs::decode_ohlcv_records): build dbn::TsSymbolMap from the DBN metadata once, resolve each record's instrument_id to its symbol, and skip any symbol containing - (the canonical spread separator). Same-timestamp dedup-by-volume continues to handle legitimate front/back-month overlap during rollover.

Why this is correct vs the bar-level sanitize: the sanitize_bars layer (Fix 25) catches the symptom (close ratio out of bounds) and works for any instrument. The spread filter catches the cause — a spread between contracts where the underlying happens to have a small basis can sit within the [0.5, 2.0] sanitize ratio (e.g., short-tenor spread on a low-vol day might trade at $5/$5141 = 0.001 OR cleanly on the underlying scale during fast moves). The two layers are complementary: spread filter is precise but symbol-dependent; sanitize is universal but symptom-based.

Validation. cargo check -p ml --example train_baseline_rl clean. Adds time = "0.3" to ml/Cargo.toml (required by dbn::TsSymbolMap's time::Date API). Expected next-smoke effect: decode_ohlcv_records log shows skipped N spreads with N matching the per-file contamination count; sanitize_bars reports zero (or near-zero) drops since the spread filter caught everything upstream. Sanitize remains as defense-in-depth for unknown future contamination shapes.

Closes the chase. Bug 2 root → ~8,800 spread bars from Databento's parent-symbol resolution. Caught structurally now at the DBN parser, not at z-score time.

Fix 25 — Structural data-loading defense + DIAG_BUG2 cleanup (2026-05-02)

Goal: stop chasing one-off data-corruption bugs. Make corruption impossible to flow into training, regardless of source.

After three+ historical fixes for data-loading corruption (#191 fxcache column-0 raw-price leak, Bug 1 fxcache target-column semantics, label_scale=5443 raw-magnitude leaks, plus today's state[0] heavy-tail with std=570 outliers), the structural defense was still missing. Each prior fix patched a specific writer; the next corruption shape always found a new hole. This commit installs a 5-layer defense-in-depth gate stack so any future corruption — whatever its origin — is rejected at the data-loading boundary instead of poisoning the model.

The 5 layers (each independent, each mandatory):

1. Bar-level sanity (baseline_common/mod.rs::sanitize_bars): drops bars with non-positive OHLC, high < low, non-finite values, or bar.close / prev.close outside [0.5, 2.0]. Catches DBN parse glitches, broker tick errors, near-zero-open bars at source. Logs each drop with timestamp.

2. safe_log_return result clamp (features/extraction.rs:1246): ratio.ln().clamp(±0.1). Real-market 1-bar log returns rarely exceed ±0.05; the ±0.1 ceiling traps every realistic move while rejecting all known corruption shapes (a corrupt bar with bar.open ≈ 0 would otherwise produce ln = -30 and flow through z-score normalization to ±30000-magnitude state[0] outliers).

3. validate_features pre-norm magnitude bound (features/extraction.rs::validate_features): post-extraction, every feature must satisfy |val| ≤ 5.0. Pre-normalization features come from safe_normalize ([0, 1] / [-1, 1]), safe_clip (max ±3 across all uses), or clamped log returns (±0.1); the widest legitimate range is ±3, so ±5 catches extractor invariant breaks without rejecting any legitimate output.

4. NormStats::normalize post-norm clamp (walk_forward.rs:688): ((val - mean) / std).clamp(±NORMALIZED_FEATURE_BOUND) with bound = 20.0. Even if upstream layers somehow produce outliers (logic bug, future regression), this guarantees every value uploaded to GPU is within ±20.0 — well above any legitimate z-scored signal but well below the magnitudes (±30000) that previously poisoned label_scale_ema.

5. Shared validate_normalized_features gate (walk_forward.rs::validate_normalized_features): single source-of-truth invariant, enforced at three call sites:

   • fxcache fast path: after discover_and_load returns Some(cached) in train_baseline_rl.rs:594. fxcache files can be stale (different code, schema drift, manual file edits) — the cache_key + schema_hash check doesn't catch every corruption shape. If load returns out-of-bounds data, fail rather than upload it.
   • DBN fallback: after normalize_batch in train_baseline_rl.rs:633. NormStats::normalize already clamps each output, so failure here means the clamp itself is broken.
   • precompute writer: after normalize_batch in precompute_features.rs:686. Never write a poisoned fxcache to disk; failing here prevents corruption from being persisted across runs.

Removed: DIAG_BUG2 and DIAG_BUG2_v2 one-shot diagnostic blocks at training_loop.rs:1785-1908 (~125 lines, including DtoH downloads + outlier scans). Replaced by the structural defense — instrumentation isn't needed once corruption can't reach state[0] in the first place. Same commit also removes the pre-norm and post-norm interpretive log lines that referenced the now-impossible "raw-price magnitude" failure mode.

FEATURE_SCHEMA_HASH auto-bumps (build.rs FNV-1a hash over SCHEMA_FILES, includes extraction.rs). This invalidates any pre-fix .fxcache file via the strict header check at fxcache.rs:201. The next training run will trigger ensure-fxcache regen, producing a clean cache with the new clamps applied.

Why this finally closes the chapter:

• Per-writer fixes are reactive (after-the-fact, after corruption has already hit production).
• A boundary-validation invariant is proactive — the contract becomes "any data path that violates |feat| ≤ 20.0 post-normalize panics at the boundary." Every future regression to extraction or normalization is caught at the gate, not at epoch 5 of a 50-epoch L40S run.
• The 5 layers are independent: a bug in one layer leaves the others as backstop. To produce a state[0] outlier under this commit, every single one of {sanitize_bars, safe_log_return clamp, validate_features pre-bound, NormStats clamp, validate_normalized_features gate} would have to silently fail simultaneously.

Validation. cargo check -p ml --all-targets --offline clean. Unit tests for NormStats (walk_forward.rs:706+) still pass — the clamp and the new validate function are additive; existing test inputs are well within bounds.

Fix 24 — DIAG_BUG2_v2 outlier disambiguation (H1 vs H2) (2026-05-02)

Augments the one-shot DIAG_BUG2 block at training_loop.rs:1785 with two narrower scans that disambiguate where the state[0] heavy-tail originates. Smoke smoke-test-xb78r at HEAD cba9f25ed reported mean=6.0e-3, std=5.7e2, mean_abs=2.0e1 over 3200 samples but printed first-5-sample state[0..5] values that all sit in [-1, 1] — implies ~1-2 of 3200 samples carry ±32000-magnitude values rather than uniform corruption.

What the v2 scan adds. Two outlier sweeps, both pre-graph-capture, fired once per process via the same static AtomicBool DUMPED:

• State-side sweep: scans gpu_batch.states and gpu_batch.next_states (already DtoH-downloaded for the v1 block) for sample indices i where |state[i*sd + 0]| > 100. Reports up to 10 indices + values per buffer.
• Source-side sweep: scans features_buf.host_ptr directly via raw-pointer arithmetic for bar indices b where |features[b*42 + 0]| > 100. host_ptr aliases the same physical pages the kernel reads (mapped-pinned), so this is a faithful read of the data state_gather actually sees — no DtoH copy needed. Reports mean/std/max_abs over up to 2M bars + up to 10 outlier (bar_idx, value) pairs.

Disambiguation rule (logged as the verdict line):

• feat-outliers nonempty → H1 (data-source corruption — the fxcache or DBN-fallback feature data carries extreme values, e.g. corrupted DBN bars with bar.open ≈ 0 producing ln(ratio) = −30 → z-score normalize to −30000).
• state-outliers nonempty AND feat-outliers empty → H2 (state_gather kernel reads stale GPU memory or bar_idx points out-of-range).

Context for H1. smoke-test-xb78r ran via the DBN fallback path (train_baseline_rl.rs:611 log message format "Loaded N bars (ts to ts)"), not the fxcache fast path (which prints "bars + features from fxcache in Xs"). /data/feature-cache/ on training-data PVC is empty — the fxcache file does not exist, so every L40S smoke since the PVC was reset hits the DBN fallback. The fallback applies NormStats::normalize_batch (line 633) which is z-score normalization with column-wise mean/stddev computed over all bars. log-return columns have small stddev (~1e-3 for 1-min NQ); a single corrupted bar with bar.open ≈ 0 produces safe_log_return = ln(near_zero / 22000) ≈ -30, which divided by std=0.001 yields normalized ≈ -30000. Few such bars in 700K survive normalization as outliers.

Hot-path purity. Diagnostic only. No kernel changes, no buffer-shape changes. Pre-graph-capture, single-shot. Safe to leave in until Bug 2 is fixed; remove with the eventual pearl_safe_log_return_extreme_clamp (or equivalent) commit. Per feedback_no_hiding no suppression markers added.

Cleanup gate. Once Bug 2 is fixed (likely a tighter clamp inside safe_log_return or in NormStats::normalize_batch for log-return columns), delete the entire DIAG_BUG2 + DIAG_BUG2_v2 block at training_loop.rs:1785-1908 in the same commit. Tracked under task #294.

Fix 28 — DIAG_AUX_LABEL one-shot instrumentation (label_scale=5481 leak) (2026-05-02)

Production blocker. The cancelled 50-epoch run train-multi-seed-bn42w at HEAD 29b1d34c6 (target stride/column drift fix already applied) hit label_scale=5481 at epoch 4. The local smoke multi_fold_convergence test on the same commit lands at ~25 — three orders of magnitude lower, with all five Fix-25 data-loading defenses active and the Fix-26 DBN spread filter live. Something between the smoke harness and the production training-data PVC is still leaking raw-price magnitude into the kernel that produces label_scale.

The kernel in question is aux_heads_kernel.cu::label_scale_ema (launched at gpu_dqn_trainer.rs::aux_heads_forward Step 2b, ~line 12973). It computes mean(|aux_nb_label_buf|) over the batch and feeds that into the SP4 Pearls A+D Wiener-EMA producer at scratch slot 43 → ISV[AUX_LABEL_SCALE_EMA_INDEX]. aux_nb_label_buf is a dedicated [B] f32 scratch populated each step by strided_gather from column 0 of next_states_buf (= log_return per pipeline.rs's feature contract). z-normalised log-returns sit at ~0.8 mean-abs; raw close prices on the ES front month sit at ~5500. A fivefold-thousand discrepancy between smoke and production at the same compiled binary is impossible unless the SOURCE data fed to state_gather is different.

Four hypotheses to disambiguate (the (1)..(4) numbers below match the diagnostic output lines):

H#	Path	Test	Expected
H1	aux_buf populated correctly, downstream kernel misreads	(1) aux_buf mean_abs	~0.8 → kernel-side bug; ~5500 → no
H2	next_states_buf col 0 was written with raw prices	(2) col0 mean_abs	~5500 confirms
H3	features_raw_cuda col 0 (the SOURCE) carries raw prices	(3) feat col0 mean_abs	~5500 confirms; ~0.8 → no
H4	targets_raw_cuda col 2 (raw_close) leaked into a consumer	(4) tgt col2 mean_abs	always ~5500 (it IS raw_close); informs whether a consumer wrongly read it

Direct stats comparison across the four numbers will pin the leak path in a single epoch-1 batch.

Implementation. One-shot static AtomicBool DUMPED block at gpu_dqn_trainer.rs::aux_heads_forward immediately after the launch_label_scale_ema call (~line 12980), guarded so only the first ungraphed step-0 invocation reads the buffers. Step 0 of fused_training.rs::run_full_step runs submit_forward_ops_main UNGRAPHED before capture_training_graph records the operation set; the captured graph never re-enters the diagnostic branch (DUMPED is true on the first replay), so subsequent training steps pay zero cost. clone_dtoh is safe in step 0 because no graph capture is active. Tracing target DIAG_AUX_LABEL. Verdict line at the end documents the interpretation rules.

Buffer-access wiring. aux_nb_label_buf and next_states_buf live on GpuDqnTrainer and are read directly via self.stream.clone_dtoh(&...). features_raw_cuda and targets_raw_cuda live on DQNTrainer (mapped-pinned MappedF32Buffer); their addresses are passed through a process-static OnceLock<(usize, usize, usize, usize)> (DIAG_AUX_LABEL_SOURCE_PTRS) populated by init_gpu_raw_buffers_from_slices after the mapped-pinned pages are allocated. Mapped-pinned host_ptrs are stable for the buffer lifetime, the DQNTrainer outlives every aux_heads_forward call, and no DtoH copy is needed for these two reads — the kernel-visible bytes ARE the host bytes.

Hot-path purity. Diagnostic only. No kernel changes, no buffer-shape changes, no struct field changes. Pre-graph-capture (step 0 ungraphed), single-shot. The static OnceLock set is a one-time write and the AtomicBool::swap is one relaxed-ordered RMW — both invisible to the captured graph.

Removal gate. Once the leak source is identified and fixed, delete in the same commit:

• The DIAG_AUX_LABEL block at gpu_dqn_trainer.rs::aux_heads_forward (~line 12980).
• The DIAG_AUX_LABEL_SOURCE_PTRS OnceLock static at gpu_dqn_trainer.rs:~556.
• The OnceLock::set call at training_loop.rs::init_gpu_raw_buffers_from_slices.

Validation. SQLX_OFFLINE=true cargo check -p ml --offline clean; cargo build -p ml --release --offline --features cuda clean. Diagnostic fires on the next L40S smoke run or production training step — no GPU spend in this commit.