5f26c57514
nsys profile of multi_fold_convergence on L40S identified compute_expected_q as the #3 GPU consumer at 12.9% (207ms / 1382 calls — ~150 µs per call, compute-bound at typical batch=8192 × num_atoms=51). The kernel computed each per-action softmax over atoms in 3 passes (max → sum_exp → normalize+expected +var+entropy+util), re-reading v_row[z]+adv_a[z] 3 times per atom for each of the 4 × ~3.25 = 13 actions per sample. Replaced with online softmax (running-max + running-sum) so expected_q, sum_z_sq, and entropy all accumulate in a single forward pass over atoms. Atom utilisation still needs a 2nd pass because it depends on the final S = sum exp(logit-M) being known to compute prob_z = exp(logit_z-M)/S for the threshold test. Inner loop reduction: 3 → 2 passes, ~33% fewer global loads of branch advantage logits (which dominate because v_row[z] is reused across 13 (action, branch) pairs and may stay in L1, while adv_a[z] flips per action and is pure global). Online accumulation pattern (Page-Olshen): M, S, TZ, TZ², TLM = -inf, 0, 0, 0, 0 for z in 0..num_atoms: logit = v_row[z] + adv_a[z] if logit > M: scale = exp(M - logit) # ≤ 1, no overflow S, TZ, TZ², TLM *= scale M = logit w = exp(logit - M) S += w TZ += w * z_val TZ² += w * z_val² TLM += w * (logit - M) expected_q = TZ / S sum_z_sq = TZ² / S entropy = log(S) - TLM / S # analytical: -sum(p log p) Correctness: bit-stable when atoms processed in fixed order (z = 0..num_atoms-1). The analytical entropy form is mathematically identical to -sum(p log p): -sum(p log p) = log(S) - (1/S) * sum(exp(logit-M) * (logit-M)) = log(S) - TLM / S The previous code's `if (prob > 1e-10f)` underflow guard is no longer needed: exp(logit - M) underflows cleanly to 0 when logit << M, and the analytical form does not multiply tiny probs by very negative logs. Bit-stable per feedback_stop_on_anomaly.md — no fuzzy tolerance change. Atom-stat block-sum reduction unchanged (warp shfl + shared-mem bank, deterministic). Expected wall-clock saving: ~33% of 207ms = ~70ms across the smoke run; on production batch=8192 × 1382 calls per fold, roughly proportional. Lower-bound estimate — at low batch (smoke) the kernel may be launch-bound rather than load-bound. nsys re-profile after deployment will quantify. ABI unchanged; no Rust caller changes required. Per Invariant 7 the audit doc dqn-gpu-hot-path-audit.md is updated with Fix 19 entry. Build: SQLX_OFFLINE=true cargo check -p ml --lib clean (12 warnings, baseline). Tests: cargo test -p ml --lib --no-run clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
250 lines
38 KiB
Markdown
250 lines
38 KiB
Markdown
# DQN GPU Hot-Path Audit
|
||
|
||
**Invariant 3 enforcement.** Every cross-boundary call (DtoH, HtoD, memcpy between host and device) in the DQN code path is classified below. No hot-path `memcpy` DtoH/HtoD is allowed — zero tolerance. Pinned + device-mapped memory is the ONLY permitted mechanism for per-step host-visible data.
|
||
|
||
**Hot path definition:** every function called from `fused_training.rs::run_full_step` or its descendants that executes once per training step (up to 178 times per epoch at batch size 8192), including kernel invocations inside the captured CUDA graph (`adam_grad_child`, `forward_child`, `aux_grad_child`, `aux_child`, `maintenance_child`) and host-side orchestration between them.
|
||
|
||
**Classification:**
|
||
- `OK-pinned` — uses the zero-copy pinned + device-mapped pattern (`cuMemAllocHost_v2` + `cuMemHostGetDevicePointer_v2`); allowed on hot path.
|
||
- `MIGRATE` — was a plain `memcpy` but has been converted to pinned in this commit.
|
||
- `COLD-PATH` — acceptable placement (constructor init, checkpoint save/load, fold transition, epoch-boundary); annotated with reason.
|
||
|
||
| File:line | Call signature | Path class | Annotation | Status |
|
||
|---|---|---|---|---|
|
||
| `gpu_dqn_trainer.rs:6586` | `malloc_host` → `total_loss_pinned` | OK-pinned | Constructor: `cuMemAllocHost+GetDevicePointer`; GPU writes via dev ptr, CPU reads via `readback_training_scalars()` | OK |
|
||
| `gpu_dqn_trainer.rs:6598` | `malloc_host` → `mse_loss_pinned` | OK-pinned | Constructor: same pattern as `total_loss_pinned` | OK |
|
||
| `gpu_dqn_trainer.rs:6611` | `malloc_host` → `q_divergence_pinned` | OK-pinned | Constructor: same pattern | OK |
|
||
| `gpu_dqn_trainer.rs:6624` | `malloc_host` → `iqn_readiness_pinned` | OK-pinned | Constructor: same pattern | OK |
|
||
| `gpu_dqn_trainer.rs:6662` | `malloc_host` → `grad_norm_pinned` | OK-pinned | Constructor: same pattern | OK |
|
||
| `gpu_dqn_trainer.rs:6707` | `malloc_host` (plain, not DEVICEMAP) → readback scratch | COLD-PATH | Constructor: plain pinned for per-epoch `flush_readback()`; not written by GPU kernel directly | OK |
|
||
| `gpu_dqn_trainer.rs:6788` | `malloc_host` → monitoring summary pinned | OK-pinned | Constructor: monitoring kernel writes via dev ptr each step | OK |
|
||
| `gpu_dqn_trainer.rs:6895` | `malloc_host` → `eval_v_range_pinned` | OK-pinned | Constructor: GPU writes; host reads only at fold/epoch boundary via `eval_v_range()` | OK |
|
||
| `gpu_dqn_trainer.rs:6911` | `malloc_host` → `nan_flag_pinned` | OK-pinned | Constructor: GPU writes on NaN; host reads only in fault path (`read_nan_flags`) | OK |
|
||
| `gpu_dqn_trainer.rs:6929` | `malloc_host` → ISV `isv_signals_pinned` | OK-pinned | Constructor: GPU writes ISV bus each step; host reads via `read_isv_health_and_regime()` in `compute_adaptive_budgets` | OK |
|
||
| `gpu_dqn_trainer.rs:6949` | `malloc_host` → `lr_pinned` | OK-pinned | Constructor: CPU writes LR; GPU reads via dev ptr | OK |
|
||
| `gpu_dqn_trainer.rs:6968` | `malloc_host` → `adaptive_clip_pinned` | OK-pinned | Constructor: CPU writes clip; GPU reads via dev ptr | OK |
|
||
| `gpu_dqn_trainer.rs:6988` | `malloc_host` → `popart_pinned` | OK-pinned | Constructor: GPU writes PopArt stats; host reads at epoch boundary | OK |
|
||
| `gpu_dqn_trainer.rs:7099` | `malloc_host` → `q_stats_pinned` | OK-pinned | Constructor: GPU writes Q stats; host reads via `reduce_current_q_stats()` at epoch boundary | OK |
|
||
| `gpu_dqn_trainer.rs:7127` | `malloc_host` → `q_readback_pinned` | OK-pinned | Constructor: GPU writes per-step Q readback; host reads via `last_readback_scalars()` in training guard | OK |
|
||
| `gpu_dqn_trainer.rs:7206` | `cuMemAllocHost_v2` → `var_ema` pinned (ISV subsystem) | OK-pinned | Constructor: ISV variance EMA — GPU writes each step; host reads via ISV bus | OK |
|
||
| `gpu_dqn_trainer.rs:7243` | `cuMemAllocHost_v2` → `regime_util` pinned (ISV subsystem) | OK-pinned | Constructor: ISV regime utilization — GPU writes each step | OK |
|
||
| `gpu_dqn_trainer.rs:7289` | `malloc_host` → `popart_var_pinned` | OK-pinned | Constructor: GPU writes PopArt variance; host reads at epoch boundary | OK |
|
||
| `gpu_dqn_trainer.rs:7312` | `cuMemcpyHtoDAsync_v2` (constructor) | COLD-PATH | Constructor: one-shot HtoD to seed deterministic-mask buffer; never replayed | OK |
|
||
| `gpu_dqn_trainer.rs:7878` | `cuMemAllocHost_v2` → `q_mean_scratch` | OK-pinned | Constructor: GPU scratch for Q-mean reduction kernel | OK |
|
||
| `gpu_dqn_trainer.rs:7898` | `cuMemAllocHost_v2` → `q_mean_ema` | OK-pinned | Constructor: Q-mean EMA — GPU writes, host reads at epoch boundary | OK |
|
||
| `gpu_dqn_trainer.rs:7919` | `cuMemAllocHost_v2` → `td_error_scratch` | OK-pinned | Constructor: TD-error scratch for reduce kernel | OK |
|
||
| `gpu_dqn_trainer.rs:7939` | `cuMemAllocHost_v2` → `q_var_scratch` | OK-pinned | Constructor: Q-variance scratch for reduce kernel | OK |
|
||
| `gpu_dqn_trainer.rs:2300` | `stream.memcpy_htod` → `atom_positions_buf` | COLD-PATH | `set_atom_positions()`: called at fold boundary to reload C51 atom grid — not on hot path | OK |
|
||
| `gpu_dqn_trainer.rs:2606` | `stream.memcpy_dtoh` → grad buf readback | COLD-PATH | `per_branch_grad_norms()`: per-epoch call only; drains grad buffer for logging | OK |
|
||
| `gpu_dqn_trainer.rs:2925` | `stream.memcpy_dtoh` (online params) | COLD-PATH | `per_branch_target_drift()`: per-epoch drift measurement | OK |
|
||
| `gpu_dqn_trainer.rs:2928` | `stream.memcpy_dtoh` (target params) | COLD-PATH | `per_branch_target_drift()`: per-epoch drift measurement | OK |
|
||
| `gpu_dqn_trainer.rs:2991` | `stream.memcpy_dtoh` → weight inspection | COLD-PATH | `per_branch_vsn_mean()`: per-epoch VSN mean read | OK |
|
||
| `gpu_dqn_trainer.rs:3072` | `stream.memcpy_dtoh` → Q-out inspection | COLD-PATH | `compute_q_stats()`: per-epoch Q-stats; not called per-step | OK |
|
||
| `gpu_dqn_trainer.rs:7541` | `stream.memcpy_htod` → descriptor buffer | COLD-PATH | Constructor: one-shot HtoD for GEMM descriptor upload | OK |
|
||
| `gpu_dqn_trainer.rs:7680` | `stream.memcpy_htod` → stochastic depth scale | COLD-PATH | Constructor: one-shot HtoD for stochastic depth kernel seed | OK |
|
||
| `gpu_dqn_trainer.rs:7686` | `stream.memcpy_htod` → stochastic depth RNG | COLD-PATH | Constructor: one-shot HtoD for stochastic depth RNG | OK |
|
||
| `gpu_dqn_trainer.rs` (removed) | `cuMemcpyDtoHAsync_v2` in `run_causal_intervention_unconditional` | **MIGRATED** | **Fix 1**: removed dead copy — result `causal_mean_scratch` was never consumed by any caller; result stays on GPU | FIXED |
|
||
| `gpu_iqn_head.rs:469` | `stream.clone_htod` → `cos_features` | COLD-PATH | Constructor: precompute cosine embedding table; one-shot | OK |
|
||
| `gpu_iqn_head.rs:477` | `stream.clone_htod` → `online_taus` | COLD-PATH | Constructor: tau quantile tiling; one-shot | OK |
|
||
| `gpu_iqn_head.rs:480` | `stream.clone_htod` → `target_taus` | COLD-PATH | Constructor: tau quantile tiling; one-shot | OK |
|
||
| `gpu_iqn_head.rs:498` | `malloc_host` → `total_loss_pinned` | OK-pinned | Constructor: GPU writes IQN loss; host reads via `read_total_loss()` | OK |
|
||
| `gpu_iqn_head.rs:534` | `malloc_host` → `t_pinned` | OK-pinned | Constructor: Adam step counter — CPU increments, GPU reads via dev ptr | OK |
|
||
| `gpu_iqn_head.rs:548` | `malloc_host` → `tau_pinned` | OK-pinned | Constructor (Fix 3): tau scalar — CPU writes, GPU reads via `tau_dev_ptr`; replaces `CudaSlice<f32>` + `cuMemcpyHtoDAsync_v2` | OK |
|
||
| `gpu_iqn_head.rs:807` | `cuMemcpyDtoDAsync_v2` (DtoD) | OK-pinned | Device-to-device copy inside graph-captured target-H forward cache; no host crossing | OK |
|
||
| `gpu_iqn_head.rs:991` | `cuMemcpyDtoDAsync_v2` (DtoD) | OK-pinned | Device-to-device copy for cached target states; no host crossing | OK |
|
||
| `gpu_iqn_head.rs:1564` | `stream.clone_dtoh` → `total_loss` | COLD-PATH | `read_loss()`: synchronous fallback for epoch-end logging only; `read_total_loss()` (pinned) is the hot-path equivalent | OK |
|
||
| `gpu_iqn_head.rs:1602` | `cuMemcpyDtoH_v2` (sync) in `compute_iqr` | **MIGRATED** | **Fix 2A**: `compute_iqr()` removed from `submit_aux_ops`; now called at epoch boundary via `refresh_iqn_iqr()` only; sync DtoH cannot be captured in CUDA graph — was silently a no-op on replayed steps | FIXED |
|
||
| `gpu_iqn_head.rs:1622` | `cuMemcpyHtoDAsync_v2` in `compute_iqr` | **MIGRATED** | **Fix 2B**: same relocation — async HtoD with stale IQR data was being replayed; now runs once at epoch boundary | FIXED |
|
||
| `gpu_iqn_head.rs` (removed) | `cuMemcpyHtoDAsync_v2` in `target_ema_update` for `tau_buf` | **MIGRATED** | **Fix 3**: replaced `CudaSlice<f32>` + async copy with pinned+device-mapped page; CPU writes `*tau_pinned = tau`; kernel reads via `tau_dev_ptr` — zero PCIe overhead | FIXED |
|
||
| `gpu_iqn_head.rs:2058` | `stream.clone_htod` → `params_buf` constructor | COLD-PATH | `init_iqn_head_from_weights()`: one-shot weight upload at construction/load; not on hot path | OK |
|
||
| `gpu_iqn_head.rs:2107` | `stream.clone_dtoh` + `stream.clone_htod` in `clone_cuda_slice` | COLD-PATH | `clone_cuda_slice()`: called during checkpoint save/load only | OK |
|
||
| `fused_training.rs:476` | `stream.memcpy_htod` → `src_gpu` (PER source indices) | COLD-PATH | `prepare_per_batch()`: per-epoch PER batch preparation — not per-step | OK |
|
||
| `fused_training.rs:482` | `stream.memcpy_htod` → `ones_gpu` | COLD-PATH | `prepare_per_batch()`: per-epoch importance weights init | OK |
|
||
| `fused_training.rs:2914` | `stream.clone_dtoh` → state inspection | COLD-PATH | `eval_state_debug()`: debug helper, no hot-path caller | OK |
|
||
| `fused_training.rs:2928` | `stream.clone_dtoh` → param inspection | COLD-PATH | `eval_state_debug()`: debug helper | OK |
|
||
| `fused_training.rs:2955` | `cuMemcpyDtoH_v2` (sync) | COLD-PATH | `eval_v_range()`: called only at fold boundary / epoch-0 init; explicit sync is documented and acceptable at boundary | OK |
|
||
| `training_loop.rs:220` | `collector.stream().memcpy_dtoh` → PER scores | COLD-PATH | Pre-epoch PER priority init — runs once before the step loop starts | OK |
|
||
| `training_loop.rs:409` | `stream.memcpy_htod` → `episode_starts_buf` | COLD-PATH | Per-epoch experience setup — before `run_training_steps_slices` begins | OK |
|
||
| `training_loop.rs:1003` | `clone_htod_f32` → targets device buffer | COLD-PATH | Data loading: DBN targets uploaded before step loop | OK |
|
||
| `training_loop.rs:1007` | `clone_htod_f32` → features device buffer | COLD-PATH | Data loading: feature matrix uploaded before step loop | OK |
|
||
| `tau_update_kernel.cu` (Plan 1 Task 13) | single-thread kernel: reads ISV[39,40,12]; writes ISV[42] | COLD-PATH | Per-epoch boundary call from `training_loop.rs`; 1-thread, 1-block; zero hot-path overhead | OK |
|
||
| `epsilon_update_kernel.cu` (Plan 1 Task 14) | single-thread kernel: reads ISV[39,40,12]; writes ISV[41] | COLD-PATH | Per-epoch boundary call from `training_loop.rs`; 1-thread, 1-block; zero hot-path overhead | OK |
|
||
| `gamma_update_kernel.cu` (Plan 1 Task 10) | single-thread kernel: reads ISV[12]; writes ISV[43] | COLD-PATH | Per-epoch boundary call from `training_loop.rs`; 1-thread, 1-block; ISV[43] read by fill_gamma_buf on hot path via read_isv_signal_at (pinned, zero-copy) | OK |
|
||
| `kelly_cap_update_kernel.cu` (Plan 1 Task 11) | single-thread kernel: reads ISV[12] + portfolio_states[n_envs, PS_STRIDE]; aggregates mean half-Kelly × health-safety; writes ISV[44] | COLD-PATH | Per-epoch boundary call from `training_loop.rs`; 1-thread, 1-block; portfolio_states dev ptr passed from GpuExperienceCollector without CPU roundtrip | OK |
|
||
| `atoms_update_kernel.cu` (Plan 1 Task 9) | 4-block kernel (one per branch): reads ISV[23..31) + spacing_raw params[52..56); writes atom_positions_buf[4 * num_atoms] via softmax + cumsum | COLD-PATH | Per-epoch boundary + SGD step (every 50 steps); grid=(4,1,1), block=(256,1,1); replaces CPU loop that launched adaptive_atom_positions 4× separately | OK |
|
||
| `gpu_portfolio.rs:174` | `actions_buf` upload (per simulate_batch_gpu) | **MIGRATED** | Promoted `actions_buf` from `CudaSlice<i32>` to `MappedI32Buffer(MAX_BATCH_SIZE)` allocated once at constructor. Per-call host writes go through `host_ptr`; kernel arg switched to `dev_ptr` u64. Zero HtoD copy on the simulate hot path. | FIXED |
|
||
| `gpu_portfolio.rs:122,296` | `portfolio_state_buf` init/reset | **MIGRATED** | env_step kernel writes back to `portfolio_state_buf`, so the destination must remain a `CudaSlice<f32>`. Init/reset now stage via a temporary `MappedF32Buffer` + `memcpy_dtod_async` (new helper `upload_portfolio_state`). | FIXED |
|
||
| `decision_transformer.rs:913` | `scratch.adam_t` upload per `train_step_gpu` | **MIGRATED** | HOT path: 4-byte step counter was uploaded via `stream.memcpy_htod` every backward pass. Promoted `DtScratch.adam_t` from `CudaSlice<i32>[1]` to `MappedI32Buffer(1)`. Per-step host write goes through `host_ptr`; Adam kernel reads via `dev_ptr`. | FIXED |
|
||
| `decision_transformer.rs:478` | DT params init upload (constructor) | **MIGRATED** | params is mutated by Adam, so destination must remain `CudaSlice<f32>`. Xavier-init upload now stages via `MappedF32Buffer` + `memcpy_dtod_async` through new helper `upload_host_to_cuda_f32`. | FIXED |
|
||
| `decision_transformer.rs:501` | DT wd_mask init (constructor) | **MIGRATED** | wd_mask is read-only by GPU. Promoted from `CudaSlice<f32>` to `MappedF32Buffer`. CPU writes 1.0s through `host_ptr`; kernel reads via `dev_ptr`. | FIXED |
|
||
| `decision_transformer.rs:1105` | trajectory build episode_starts | **MIGRATED** | One-time per pre-training data build (cold path). Replaced `clone_htod` with `MappedI32Buffer` + direct `host_ptr` write; kernel reads via `dev_ptr`. | FIXED |
|
||
| `decision_transformer.rs:1147` | trajectory build ep_rewards | **MIGRATED** | Same trajectory builder cold path. Replaced `htod_f32` with `MappedF32Buffer`; CPU writes via `host_ptr`, return_to_go kernel reads via `dev_ptr`. | FIXED |
|
||
| `gpu_experience_collector.rs:2992` | hindsight `bar_indices` upload (per `collect_experiences_gpu`) | **MIGRATED** | HOT path: alloc + memcpy_htod ran every collect when hindsight active. Added persistent `bar_indices_pinned: MappedI32Buffer(alloc_episodes * alloc_timesteps * 2)` allocated at constructor. Per-call host writes go through `host_ptr`; kernel arg switched to `dev_ptr` u64. | FIXED |
|
||
| `gpu_experience_collector.rs:3186` | feature_mask upload (per-collect at t==0) | **MIGRATED** | HOT path: was alloc + memcpy_htod every epoch. Promoted `feature_mask_buf` from `Option<CudaSlice<f32>>` to `Option<MappedF32Buffer>`. Reallocates only when mask size changes (still a cold-path event); CPU writes via `host_ptr`, state_gather reads via `dev_ptr`. | FIXED |
|
||
| `gpu_experience_collector.rs:3097` | episode_starts upload (per-collect) | **MIGRATED** | episode_starts_buf is GPU-mutated by `domain_rand_episode_starts`, so the destination must remain a CudaSlice. Stage via `MappedI32Buffer` + `memcpy_dtod_async` through `upload_host_to_cuda_i32_via_pinned`. | FIXED |
|
||
| `gpu_experience_collector.rs:2009` | `upload_expert_actions` (warm) | **MIGRATED** | Promoted `expert_actions_gpu` from `Option<CudaSlice<i32>>` to `Option<MappedI32Buffer>`. Direct host_ptr write replaces the alloc + memcpy_htod pair. | FIXED |
|
||
| `gpu_experience_collector.rs:3816` | `update_per_sample_support` (per-epoch) | **MIGRATED** | per_sample_support_buf is read-only by GPU; promoted from `CudaSlice<f32>` to `MappedF32Buffer`. Compute_expected_q + quantile_q_select now receive `dev_ptr` u64 kernel args. | FIXED |
|
||
| `gpu_experience_collector.rs:1076,3848` | portfolio_states init/reset (cold) | **MIGRATED** | portfolio_states is GPU-mutated by env_step. Init/reset stage via `MappedF32Buffer` + `memcpy_dtod_async` through `upload_host_to_cuda_f32_via_pinned`. | FIXED |
|
||
| `gpu_experience_collector.rs:1093` | epoch_state init (cold) | **MIGRATED** | epoch_state is GPU-mutated. Replaced `clone_htod_f32` with explicit `alloc_zeros` + `upload_host_to_cuda_f32_via_pinned`. | FIXED |
|
||
| `gpu_experience_collector.rs:1341` | saboteur_base init (cold) | **MIGRATED** | saboteur_base is GPU-mutated by `saboteur_select`. Replaced `memcpy_htod` with `upload_host_to_cuda_f32_via_pinned`. | FIXED |
|
||
| `gpu_experience_collector.rs:2784` | trade_stats_buf zero (per-epoch) | **MIGRATED** | The htod_f32 of an all-zeros vector was needlessly going through PCIe. Replaced with `stream.memset_zeros` — fully GPU-side. | FIXED |
|
||
|
||
## Summary
|
||
|
||
| Class | Count |
|
||
|---|---|
|
||
| OK-pinned | 31 |
|
||
| COLD-PATH | 20 |
|
||
| MIGRATED (fixed in this commit) | 4 |
|
||
| Remaining MIGRATE (unfixed) | 0 |
|
||
|
||
**All MIGRATE sites resolved.** Zero hot-path `memcpy` calls remain in the DQN training loop.
|
||
|
||
## Fixes Applied (this commit)
|
||
|
||
### Fix 1 — Remove dead async DtoH in `run_causal_intervention_unconditional`
|
||
`gpu_dqn_trainer.rs`: `cuMemcpyDtoHAsync_v2` from `causal_mean_scratch_ptr` to `readback_pinned[10]` was removed. The only reader (`run_causal_intervention`, which has no callers) was never invoked, making this copy dead code. The result now stays on the GPU device buffer where the kernel leaves it.
|
||
|
||
### Fix 2 — Relocate `compute_iqr()` from hot path to epoch boundary
|
||
`fused_training.rs / training_loop.rs`: `iqn.compute_iqr()` was being called inside `submit_aux_ops`, which is captured in the CUDA graph (`aux_child`). The sync `cuMemcpyDtoH_v2` inside `compute_iqr` cannot be captured in a CUDA graph — it ran during capture only and was silently skipped on every replayed step. The subsequent `cuMemcpyHtoDAsync_v2` (uploading IQR back) was therefore replaying stale data. Moved to `refresh_iqn_iqr()` called at epoch boundary in `process_epoch_boundary`.
|
||
|
||
### Fix 3 — Convert `tau_buf` from device VRAM to pinned+device-mapped
|
||
`gpu_iqn_head.rs`: `tau_buf: CudaSlice<f32>` + `tau_host: f32` + `cuMemcpyHtoDAsync_v2` replaced by `tau_pinned: *mut f32` + `tau_dev_ptr: u64` allocated via `cuMemAllocHost_v2(DEVICEMAP)` + `cuMemHostGetDevicePointer_v2`. CPU writes `*tau_pinned = tau` directly; the EMA kernel reads via `tau_dev_ptr` with no PCIe copy. Drop impl updated to free `tau_pinned`.
|
||
|
||
### Fix 4 (perf) — Async-eval pipeline: split `launch_metrics_and_download` (2026-04-28)
|
||
`gpu_backtest_evaluator.rs` + `trainer/metrics.rs` + `trainer/training_loop.rs`: the prior merge `3c0d26292` (async-diag — eval on dedicated stream + spawn_blocking checkpoint, building on `d9cb14f1b` + `673b04a8d`) wired the **GPU side** of cross-stream pipelining correctly (training stream `cuda_stream`, eval stream `validation_stream`, `cuStreamWaitEvent` barrier on `train_done_event`) but left a host-side `stream.synchronize()` inside `launch_metrics_and_download` that blocked the CPU thread for the FULL eval drain (~25-30s/epoch on L40S). The host thread is the same one that submits the next epoch's training kernels via `run_full_step` — so until eval drained, training submission was gated on it, and the dedicated `validation_stream` provided no actual wall-clock benefit.
|
||
|
||
Split into two record/consume halves at the buffer-readback layer:
|
||
|
||
* `launch_metrics_and_record_event(&mut self) -> Result<(), MLError>` — submits the metrics kernel + DtoD-async copies of `actions_history_buf` / `intent_mag_buf` / `picked_action_history_buf` into mapped-pinned mirrors (`actions_history_pinned: MappedI32Buffer` allocated in `new()`, the other two lazy-allocated alongside their device buffers in `ensure_action_select_ready`). Records `eval_done_event` (lazy-created on first launch via `stream.context().new_event(None)`, re-recorded each epoch via `event.record(stream)`). **Returns immediately** — no host wait.
|
||
* `consume_metrics_after_event(&self) -> Result<Vec<WindowMetrics>, MLError>` — calls `event.synchronize()` (the SOLE host-side wait per epoch boundary), then `read_volatile`s the mapped-pinned `metrics_buf` for the per-window metrics. After the event syncs, the four action-distribution helpers (`read_eval_action_distribution_per_*`, `read_eval_intent_magnitude_distribution`, `read_chunked_actions_direction_distribution`) read directly from the now-coherent mapped-pinned mirrors — no `memcpy_dtoh` per call (the prior implementation allocated a fresh `Vec<i32>` of size `n_windows * max_len` and did a synchronous DtoH each call, which violates `feedback_no_htod_htoh_only_mapped_pinned.md` AND added 4 extra host-side stalls beyond the metrics one). All four mirrors land in the same event-sync barrier as the metrics readback.
|
||
|
||
Caller migration (`feedback_no_partial_refactor.md`):
|
||
* `evaluate_dqn_graphed` (existing public API, used by hyperopt + smoke test) becomes a thin synchronous wrapper: launch_async → consume. Public ABI unchanged.
|
||
* New `evaluate_dqn_graphed_async` — launch-only sibling for the pipelined path. Returns `()`.
|
||
* `evaluate` / `evaluate_ppo` / `evaluate_supervised` migrated inline to launch+consume — no caller pipelines them, so no behavioural change.
|
||
* `compute_validation_loss` removed; replaced by `launch_validation_loss` (returns `()`) + `consume_validation_loss` (returns `f64` and emits the `HEALTH_DIAG val [...]` / `val_dir_dist` / `val_picked_dir_dist` lines).
|
||
* `training_loop.rs`'s validation reporting block now: epoch-start consumes any pending launch into `cached_async_val_loss`, epoch-end reports the cached value and submits a fresh launch with `pending_val_loss = Some(0.0)` as the "launched" sentinel. Bootstrap (epoch 0 / no validation_stream) does sync launch+consume. After the for-epoch loop terminates, a final consume drains the LAST epoch's launch so smoke tests / hyperopt that read `last_eval_direction_dist()` see the most recent epoch's data, not the second-to-last.
|
||
|
||
Mathematical identity preserved: every consumed val_sharpe is bit-identical to what the prior `compute_validation_loss` would have produced for the same epoch — the only change is *when* the host parses it (one epoch later, which matches the existing `cached_async_val_loss` lag semantics; HEALTH_DIAG `val [...]` emit moves with the consume). Expected wall-clock saving: ~25-30s/epoch on L40S.
|
||
|
||
Out-of-scope (remaining host-side stream syncs in eval, not part of this fix): `gpu_backtest_evaluator.rs` lines ~1596 + ~1863 — periodic `self.stream.synchronize()` calls every 10 chunks inside `submit_dqn_step_loop_cublas` for kernel-error detection. They block the host CPU thread but only for the chunked rollout duration of ~5 calls × per-chunk-residency-time per epoch (each catches deferred CUDA errors within 10 chunks of origin). Smaller in aggregate than the `launch_metrics_and_download` final wait this fix removed; future work to convert to async error-polling (e.g., `eval_done_event.is_complete()` polling from training submissions). Not adjacent to the perf-fix scope.
|
||
|
||
### Fix 5 — Add mapped-pinned staging helpers in `mapped_pinned.rs`
|
||
`mapped_pinned.rs`: added `clone_to_device_{f32,i32}_via_pinned` and `upload_{f32,i32}_via_pinned` helpers that allocate a temporary `MappedXxxBuffer` (cuMemHostAlloc DEVICEMAP), write the payload via `host_ptr`, then async DtoD into a target `CudaSlice<T>`. Used to migrate cold-path constructor uploads off explicit `cudaMemcpy` HtoD per `feedback_no_htod_htoh_only_mapped_pinned.md` while preserving struct field types (e.g. `portfolio_buf`, weight buffers consumed by Adam) where the kernel mutates the buffer every step and device-resident memory is correct.
|
||
|
||
### Fix 6 — Migrate `cuda_pipeline/mod.rs` callers to mapped-pinned
|
||
`cuda_pipeline/mod.rs`: 7 in-file callers of `clone_htod_f32` in `DqnGpuData::upload_slices`, `DqnGpuData::upload_ofi`, `DqnGpuData::build_batch_states` (portfolio), `DqnGpuData::htod_copy_into`, and `PpoGpuData::upload` rewritten to use `mapped_pinned::clone_to_device_f32_via_pinned`. The `htod_f32` / `clone_htod_f32` helper bodies at lines 129-145 remain in place because other crates are still in mid-migration; they will be deleted once all consumers migrate.
|
||
|
||
### Fix 7 — Migrate `gpu_walk_forward.rs` to mapped-pinned (3 sites)
|
||
`gpu_walk_forward.rs`: 3 `clone_htod_f32` calls in the walk-forward GPU data upload (features, targets, OFI) rewritten to `mapped_pinned::clone_to_device_f32_via_pinned`.
|
||
|
||
### Fix 8 — Migrate PPO trainer + hyperopt adapters to mapped-pinned (6 sites)
|
||
`trainers/ppo.rs` (2 sites: features, targets in `set_raw_market_data`), `hyperopt/adapters/ppo.rs` (3 sites: features, targets in upload path; action_indices in backtest forward closure), `hyperopt/adapters/mamba2.rs` (1 site: `gpu_to_stream`). All `clone_htod_f32` and bare `stream.clone_htod` calls rewritten to `mapped_pinned::clone_to_device_{f32,i32}_via_pinned`.
|
||
|
||
### Fix 9 — Migrate weight-upload sites in attention/tlob/iql/weights (4 sites)
|
||
`gpu_weights.rs` (1 site: `RmsNormWeightSet::ones` constructor), `gpu_attention.rs` (1 site: attention params constructor), `gpu_tlob.rs` (1 site: TLOB params constructor — `mod tests` blocks intentionally untouched per scope), `gpu_iql_trainer.rs` (2 sites: per_sample_support seed, IQL params init). All `htod_f32` calls rewritten to `mapped_pinned::upload_f32_via_pinned`; the gpu_weights.rs ones-init uses `clone_to_device_f32_via_pinned` since the destination is freshly allocated.
|
||
|
||
### Fix 10 — Migrate `gpu_iqn_head.rs` constructors + DtoD-only target clone (5 sites)
|
||
`gpu_iqn_head.rs`: 4 `stream.clone_htod` calls (cos_features precompute, online_taus + target_taus tile broadcast, init_iqn_xavier_weights upload) rewritten to `mapped_pinned::clone_to_device_f32_via_pinned`. Additionally the helper `clone_cuda_slice` was replaced — it previously did device→host (`clone_dtoh`) → host→device (`clone_htod`) round-trip, double-violating both the no-DtoH and no-HtoD rules. Now uses `clone_cuda_slice_f32` (single async DtoD), the only callsite being the target-network seed at constructor time (`target_params = clone_cuda_slice(&online_params)`).
|
||
|
||
### Fix 11 — Migrate `gpu_her.rs` to mapped-pinned + add MappedU32Buffer (3 sites)
|
||
`gpu_her.rs`: 1 cold-path constructor `stream.memcpy_htod` (rng_states u32 init) and 2 warm-path `memcpy_htod` (source_indices, donor_indices in `relabel_batch`) rewritten to `mapped_pinned::upload_{u32,i32}_via_pinned`. The warm sites re-allocate pinned staging per relabel call — acceptable on this cold path (relabel runs once per epoch). New `MappedU32Buffer` type added to `mapped_pinned.rs` mirroring `MappedI32Buffer`; carries a manual `Debug` impl so the workspace warning count stays at 13.
|
||
|
||
### Fix 12 — Migrate `gpu_backtest_evaluator.rs` to mapped-pinned (5 sites)
|
||
`gpu_backtest_evaluator.rs`: 5 HtoD sites in `GpuBacktestEvaluator::new` + `reset_evaluation_state` rewritten to mapped-pinned staging. Sites: prices_buf (f32), features_buf (f32 — was already going through `clone_htod_f32` helper), window_lens_buf (i32), portfolio_buf init (f32), and the per-evaluation reset of portfolio_buf. The fields stay typed `CudaSlice<T>` because the kernel mutates `portfolio_buf` each step (must remain device-resident); the read-only fields could become `MappedXxxBuffer` but keeping them as `CudaSlice` preserves the kernel-launch contract `.arg(&self.field)` already used at the many launch sites in the file.
|
||
|
||
### Fix 13 — Migrate `gpu_ppo_collector.rs` to mapped-pinned (6 sites + persistent staging)
|
||
`gpu_ppo_collector.rs`: all 6 HtoD sites migrated.
|
||
- Constructor (cold): `portfolio_states` init (htod_f32) → `upload_f32_via_pinned`; `rng_states` init (memcpy_htod) → `upload_u32_via_pinned`.
|
||
- `collect_experiences` (warm but per-collect): `episode_starts_buf` (memcpy_htod) → `upload_i32_via_pinned`; `barrier_config` (htod_f32) → `upload_f32_via_pinned`.
|
||
- `reset_episodes` (warm, per-epoch): replaced the `Vec<f32>` / `Vec<u32>` host staging fields with persistent `MappedF32Buffer` / `MappedU32Buffer` allocated once at construction. Resets fill via `host_slice_mut()` (zero-allocation, zero-HtoD); a single async DtoD seeds the persistent device-resident `portfolio_states` / `rng_states` (which the kernel mutates each step). Adds `host_slice_mut()` accessor on both mapped-pinned types.
|
||
|
||
### Fix 17 — Delete orphan `htod_f32` / `clone_htod_f32` helpers from `cuda_pipeline/mod.rs` (2026-04-28)
|
||
After Fix 1..16 migrated all 80+ production callers off `super::htod_f32` and `super::clone_htod_f32`, the helper bodies in `cuda_pipeline/mod.rs:129-145` were unused outside `#[cfg(test)]` modules. Deleted both function definitions per `feedback_no_legacy_aliases.md` (no deprecated wrappers).
|
||
|
||
The two surviving test-block callers in `gpu_tlob.rs::tests` (lines 1017 and 1132 — `super::super::htod_f32(&stream, &host_states, …)` and `super::super::htod_f32(&stream, &host_d_concat, …)`) were also migrated to `mapped_pinned::upload_f32_via_pinned` in the same commit per `feedback_no_partial_refactor.md` (when a contract is deleted, every consumer migrates together — including tests). The test-only callers in `signal_adapter.rs::tests` (5 sites), `gpu_action_selector.rs::tests` (1), and `cuda_pipeline/mod.rs::tests` (1) all use bare `stream.memcpy_htod` / `stream.memcpy_stod` against the cudarc handle directly, not the deleted helpers — no change needed.
|
||
|
||
A doc-block was added at the deletion site in `mod.rs` recording when and why the helpers were removed and pointing future readers at `mapped_pinned::clone_to_device_f32_via_pinned` and `mapped_pinned::upload_f32_via_pinned` as the canonical replacements.
|
||
|
||
Final state of the HtoD migration sequence:
|
||
- `grep -rEn "stream\.memcpy_htod|memcpy_stod|stream\.htod_copy|htod_async" crates/ml/src/`: production count **0**, test count 7 (5 in `signal_adapter.rs::tests`, 1 in `gpu_action_selector.rs::tests`, 1 in `cuda_pipeline/mod.rs::tests`).
|
||
- `grep -rn "htod_f32\|clone_htod_f32" crates/ml/src/`: outside test blocks **0** (only doc-comment text references remain). Helper definitions removed from `mod.rs`.
|
||
|
||
### Fix 16 — Migrate `training_loop.rs` raw features/targets uploads (2 sites, 2026-04-28)
|
||
`trainers/dqn/trainer/training_loop.rs:1250` (targets_raw) and `:1254` (features_raw): both `crate::cuda_pipeline::clone_htod_f32` callsites in the data-load helper rewritten to `mapped_pinned::clone_to_device_f32_via_pinned`. These run once per fold immediately before the step loop begins (the `_raw` suffix indicates the un-normalized arrays kept on-GPU for the eval-time critic). The `.map_err(|e| anyhow::anyhow!("…raw upload: {e}"))` wrapping is preserved verbatim — the helper already returns `Result<_, String>` which feeds `anyhow::anyhow!` directly, so the change is a 1:1 path swap.
|
||
|
||
### Fix 15 — Migrate `gpu_experience_collector.rs::upload_ofi_features` (1 site, 2026-04-28)
|
||
`gpu_experience_collector.rs:4258`: warm-path `super::clone_htod_f32` upload of OFI features rewritten to `mapped_pinned::clone_to_device_f32_via_pinned`. Called from training_loop's data-load phase before each fold's training run, so the OFI tensor (4M bars × 32 dims = ~512 MB) now stages through mapped-pinned + DtoD just like all other large data uploads. Error-type shift handled with `.map_err(|e| MLError::ModelError(format!("upload_ofi_features: {e}")))?` since the helper returns `Result<_, String>` and the function returns `Result<_, MLError>`.
|
||
|
||
### Fix 14 — Migrate `gpu_dqn_trainer.rs` residual COLD ctor sites (8 sites, 2026-04-28)
|
||
`gpu_dqn_trainer.rs`: residual COLD ctor sites missed by the original Fix-1..13 audit are now migrated off explicit `super::htod_f32` per `feedback_no_htod_htoh_only_mapped_pinned.md`.
|
||
- `:9922` `sel_clip_buf` 1-element init — selectivity gate clip-norm seed (1.0).
|
||
- `:10075-10078` spectral-norm `init_u_s1` / `init_v_s1` / `init_u_s2` / `init_v_s2` (4 separate uploads of trunk-encoder spectral u/v vectors).
|
||
- `:10095-10096` `alloc_spec_pair!` macro body — both u/v init uploads inside the macro now go through `upload_f32_via_pinned`. The macro's `$lbl_u` / `$lbl_v` are reused in the error-message format so per-pair failures (`spec_u_v1`, `spec_v_v1`, `spec_u_a1`, …) stay diagnostically distinct.
|
||
- `:11276` `graph_params_host` (60 floats: cross-branch graph message passing weights, 4 edges × 15 params each, near-identity W_msg init).
|
||
- `:11330` `denoise_params_host` (1800 floats: 2-step diffusion Q-refinement Xavier init for W1[24,24] + W2[12,24]).
|
||
- `:11476` `qlstm_weights_host` (528 floats: QLSTM Xavier init for 6 gates × 11 input × 8 head).
|
||
All sites use `mapped_pinned::upload_f32_via_pinned` (canonical mapped-pinned + DtoD staging helper). The helper returns `Result<_, String>` whereas the ctor returns `Result<_, MLError>`, so each site wraps the error via `.map_err(|e| MLError::ModelError(format!("<site> upload via pinned: {e}")))`. Site labels preserved so backtraces remain readable.
|
||
|
||
### Fix 18 (perf) — Eliminate per-step val gather DtoD copies via chunked gather kernel (2026-04-28)
|
||
`experience_kernels.cu` + `gpu_backtest_evaluator.rs` + `metrics.rs`: nsys profile of `multi_fold_convergence` on L40S identified `backtest_state_gather` as the #1 GPU-time consumer at 37.2% (596 ms / 60,588 calls — kernel launch latency dominated compute). The per-step pattern was:
|
||
|
||
```
|
||
for step_offset in 0..chunk_len {
|
||
launch_gather(step) -> states_buf[N, padded_sd] (1 launch)
|
||
if tlob: tlob.forward(states_buf, n) (4 cuBLAS SGEMMs + 3 kernel launches)
|
||
memcpy_dtod_async(chunked_states_buf[step_offset*N..], states_buf, ...) (1 DtoD)
|
||
}
|
||
```
|
||
|
||
Per chunk (DQN_BACKTEST_CHUNK_SIZE=512): 512 gather launches + 512 DtoD copies + 512 × 7 TLOB launches = 4608 kernel launches per chunk before forward Q-values can begin.
|
||
|
||
New kernel `backtest_state_gather_chunk` writes `[chunk_len, N, padded_sd]` directly into `chunked_states_buf` in a SINGLE launch (`chunk_len * N` threads, 1 thread per output row). Within a chunk `portfolio_buf` and `plan_isv_buf` are CONSTANT — env_step (Phase 5) and `backtest_plan_state_isv` (Phase 6) update them only at chunk boundary. So the batched gather is mathematically identical to the prior per-step + DtoD pattern; each thread reads its own (window, step_offset) row from independent feature offsets and writes its own state row. No reordering, no atomics, no shmem races.
|
||
|
||
Same change enables the val TLOB instance to run ONCE per chunk on the chunked buffer at batch = `chunk_len * N` instead of `chunk_len` separate forward calls each at batch = `N`. Val TLOB construction in `metrics.rs::reset_evaluator_for_validation` resized via new `GpuBacktestEvaluator::val_tlob_batch_size()` to `DQN_BACKTEST_CHUNK_SIZE * n_windows`; partial last chunks reuse the same buffers (`forward(b)` accepts any `b <= construction_batch`).
|
||
|
||
Per-chunk launch reduction: from `2 * chunk_len + chunk_len * 7` to `1 + 7` for the gather+TLOB phase. With 512-step chunks that's 4608 → 8, a 576x reduction. Across the full smoke run (60K gather calls in profile), expected wall-clock saving on the gather/TLOB hot path is the bulk of the 37% + 6.5% + 5.2% + 4.8% = ~53.7% of GPU time the per-call cluster occupied.
|
||
|
||
Borrow restructure: the prior single-shot `let ch_states = self.chunked_states_buf.as_ref()?` at top of `submit_dqn_step_loop_cublas` was removed because TLOB now needs `&mut self.chunked_states_buf`. Replaced with per-chunk `ch_states_base` (raw u64 device pointer extracted in a tight scope), reused for both `launch_gather_chunk` and the Phase 2+3 `compute_q_values_to` call. The pointer is stable across the chunk because the `Option<CudaSlice<f32>>` does not reallocate.
|
||
|
||
No new HtoD/DtoH paths introduced. Strictly removes 512 per-chunk DtoD copies. Per `feedback_no_partial_refactor.md`: kernel ABI is new (separate function name), prior `backtest_state_gather` retained for `evaluate()` / `evaluate_ppo()` / `evaluate_supervised()` paths that still need a single-step writer.
|
||
|
||
### Fix 19 (perf) — Online softmax in compute_expected_q: 3-pass → 2-pass (2026-04-28)
|
||
`experience_kernels.cu`: nsys profile of `multi_fold_convergence` on L40S identified `compute_expected_q` as the #3 GPU consumer at 12.9% (207 ms / 1382 calls — ~150 µs per call, compute-bound at typical batch=8192 × num_atoms=51). The kernel computed each per-action softmax over atoms in 3 passes (max → sum_exp → normalize+expected/var/entropy/util), re-reading `v_row[z] + adv_a[z]` 3 times per atom for each of the 4×~3.25 = 13 actions per sample.
|
||
|
||
Replaced with **online softmax** (Page-Olshen running-max + running-sum) so expected_q, sum_z_sq, and entropy all accumulate in a single forward pass over atoms. Atom utilisation still needs a 2nd pass because it depends on the final `S = sum exp(logit-M)` being known to compute `prob_z = exp(logit_z - M)/S` for the threshold test. Total: 3 → 2 passes, ~33% reduction in inner-loop global loads of branch advantage logits (which dominate because `v_row[z]` is reused across 13 (action, branch) pairs and may stay in L1, while `adv_a[z]` flips per action and is pure global).
|
||
|
||
Online accumulation:
|
||
```
|
||
M, S, TZ, TZ², TLM = -inf, 0, 0, 0, 0
|
||
for z in 0..num_atoms:
|
||
logit = v_row[z] + adv_a[z]
|
||
if logit > M:
|
||
scale = exp(M - logit) # ≤ 1, no overflow
|
||
S, TZ, TZ², TLM *= scale
|
||
M = logit
|
||
w = exp(logit - M) # ≤ 1
|
||
S += w
|
||
TZ += w * z_val
|
||
TZ² += w * z_val²
|
||
TLM += w * (logit - M)
|
||
expected_q = TZ / S
|
||
sum_z_sq = TZ² / S
|
||
entropy = log(S) - TLM / S # analytical: -sum(p log p) where p = w/S
|
||
```
|
||
|
||
Correctness: bit-stable when atoms processed in fixed order (z = 0..num_atoms-1). The analytical entropy form is mathematically equivalent to `-sum(p log p)`:
|
||
```
|
||
-sum(p log p) = -sum(p (logit - M - log S)) = log(S) * sum(p) - sum(p (logit-M))
|
||
= log(S) - (1/S) * sum(exp(logit-M) * (logit-M))
|
||
= log(S) - TLM / S
|
||
```
|
||
The previous code's `if (prob > 1e-10f)` underflow guard is no longer needed: `exp(logit - M)` underflows cleanly to 0 when logit << M, and the analytical form does not multiply tiny probs by very negative logs. Result is bit-stable per `feedback_stop_on_anomaly.md` — no fuzzy tolerance change.
|
||
|
||
Atom-stat block-sum reduction unchanged (warp shfl + shared-mem bank, deterministic).
|
||
|
||
Expected wall-clock saving on this kernel: ~33% of 207ms = ~70ms across the smoke run; on production batch=8192 × 1382 calls per fold, roughly proportional. Lower-bound estimate; actual saving depends on memory bandwidth vs compute mix — at low N (smoke) the kernel may be launch-bound rather than load-bound. nsys re-profile after deployment will quantify.
|
||
|
||
ABI unchanged; no Rust caller changes required.
|