gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(sp6): IQN τ buffers — MappedF32Buffer per feedback_no_htod_htoh_only_mapped_pinned

Browse Source 
Pearl 5's online_taus/target_taus/cos_features were declared as
CudaSlice<f32> (device-only), populated via upload_f32_via_pinned
which does a DtoD copy from a separate mapped-pinned staging buffer.
The DtoD inside CUDA Graph capture triggers
CUDA_ERROR_STREAM_CAPTURE_INVALIDATED and the 'continuing ungraphed'
fallback observed in smoke-test-hhr5q.

This violates feedback_no_htod_htoh_only_mapped_pinned: the rule is
mapped-pinned (cuMemHostAlloc DEVICEMAP) for ALL CPU↔GPU paths. No
DtoD copies, no HtoD copies, no exceptions.

Fix: convert all 3 buffers (online_taus, target_taus, cos_features)
to MappedF32Buffer per-branch [MappedF32Buffer; 4] arrays. Host writes
go directly to host_ptr; IQN kernel reads dev_ptr of the same memory
— no copy step at all. The mem::swap pattern is replaced with pure
selection: activate_branch_taus sets active_branch_idx; kernel launch
sites index online_taus_per_branch[active_branch_idx].dev_ptr.

Eliminates upload_f32_via_pinned calls for these buffers entirely.

Refresh becomes a host write to mapped-pinned host_ptr at fold
boundary; subsequent kernel launches see the write through the
mapped-pinned coherence guarantee after stream sync.

cargo check + cargo build --release + cargo test --lib (sp4 sp5
state_reset_registry: 13/13) all clean. Sanity grep for
upload_f32_via_pinned in gpu_iqn_head.rs returns zero.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-02 09:39:13 +02:00
parent b861567890
commit facbf76eb5
2 changed files with 106 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/dqn-gpu-hot-path-audit.md
View File

@@ -44,9 +44,7 @@
| `gpu_dqn_trainer.rs:7680` | `stream.memcpy_htod` → stochastic depth scale | COLD-PATH | Constructor: one-shot HtoD for stochastic depth kernel seed | OK |
| `gpu_dqn_trainer.rs:7686` | `stream.memcpy_htod` → stochastic depth RNG | COLD-PATH | Constructor: one-shot HtoD for stochastic depth RNG | OK |
| `gpu_dqn_trainer.rs` (removed) | `cuMemcpyDtoHAsync_v2` in `run_causal_intervention_unconditional` | **MIGRATED** | **Fix 1**: removed dead copy — result `causal_mean_scratch` was never consumed by any caller; result stays on GPU | FIXED |
| `gpu_iqn_head.rs:469` | `stream.clone_htod` → `cos_features` | COLD-PATH | Constructor: precompute cosine embedding table; one-shot | OK |
| `gpu_iqn_head.rs:477` | `stream.clone_htod` → `online_taus` | COLD-PATH | Constructor: tau quantile tiling; one-shot | OK |
| `gpu_iqn_head.rs:480` | `stream.clone_htod` → `target_taus` | COLD-PATH | Constructor: tau quantile tiling; one-shot | OK |
| `gpu_iqn_head.rs` (removed) | `clone_to_device_f32_via_pinned` → `cos_features` / `online_taus` / `target_taus` (3 CudaSlice fields) | **MIGRATED** | **SP6 fix**: converted to `[MappedF32Buffer; 4]` per-branch arrays (`online_taus_per_branch`, `target_taus_per_branch`, `cos_features_per_branch`). Constructor writes via `write_from_slice` (host_ptr direct); IQN kernels read via `dev_ptr`. `mem::swap` activate/deactivate replaced by `active_branch_idx` selection. `upload_f32_via_pinned` (which did an intermediate DtoD inside graph capture → `STREAM_CAPTURE_INVALIDATED`) fully eliminated. | FIXED |
| `gpu_iqn_head.rs:498` | `malloc_host` → `total_loss_pinned` | OK-pinned | Constructor: GPU writes IQN loss; host reads via `read_total_loss()` | OK |
| `gpu_iqn_head.rs:534` | `malloc_host` → `t_pinned` | OK-pinned | Constructor: Adam step counter — CPU increments, GPU reads via dev ptr | OK |
| `gpu_iqn_head.rs:548` | `malloc_host` → `tau_pinned` | OK-pinned | Constructor (Fix 3): tau scalar — CPU writes, GPU reads via `tau_dev_ptr`; replaces `CudaSlice<f32>` + `cuMemcpyHtoDAsync_v2` | OK |