mamba2_temporal_scan (forward) and mamba2_scan_backward are no longer
called — replaced by cuBLAS projection GEMMs + lightweight scan kernels.
Removes ~200 lines of dead CUDA code and 2 unused CUfunction handles.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Default was hardcoded as 64 in train_baseline_rl.rs and evaluate_baseline.rs,
overriding the config default of 32. Added num_quantiles=32 to production
config so it's explicit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Accumulate loss + grad_norm from pinned readback on EVERY step in
FusedTrainingCtx. Fixes smoke test failure where guard only ran
every 5th step (missing data when epoch has <5 steps).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Source: docs.nvidia.com/nsight-systems/InstallationGuide
Repo: developer.download.nvidia.com/devtools/repos/ubuntu2404/amd64/
Key: 7fa2af80.pub from CUDA repo
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Changed per_prefix_scan, per_sample, and is_weights_f32 kernels to accept
size as a const int* pointer (pinned device-mapped) instead of a baked int.
Graph replay now uses the CURRENT buffer size, not the capture-time value.
Host updates the pinned size on every insert_batch and clear.
per_sample also reads rng_step from pinned pointer (GPU-side increment).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Guard's check_host uses host-side loss/grad_norm accumulation instead of
GPU kernel. Reads from pinned device-mapped readback (one-step lag).
Fixes 8 smoke test failures from stale zero values.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Output: /data/nsys/ on training-data PVC (survives pod GC)
--show-output=true prints summary to pod logs for mid-run visibility
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Auto-sizer inflated replay buffer to 15.8M entries on H100 (45% of 80GB VRAM).
The PER prefix scan inside the CUDA graph processes ALL 15.8M entries every step,
dominating step time. With buffer_size=500K, the scan processes 500K entries instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Vaccine now uses prev_grad_buf inside maintenance_child (graphed).
The external sample() call was launching ~10 ungraphed PER kernels
on the training stream every 5th step — potential CUmodule corruption
and CPU-GPU serialization point.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
At 16384, each GEMM saturates 97% of 132 SMs — no room for multi-stream
parallelism. At 8192, GEMMs use ~50% SMs, allowing branch streams and
aux parallelism to fill idle SMs. 2x more steps/epoch but each ~2x faster.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: rng_step was passed as 0u64 (null) to increment_step_counters
kernel, causing atomicAdd on address 0 → CUDA_ERROR_ILLEGAL_ADDRESS →
cascading CUBLAS_STATUS_EXECUTION_FAILED on all subsequent operations.
Fixes:
- Replace all atomicAdd with plain writes (single-thread kernel)
- Add null guards for optional pointers (iqn_t, attn_t, rng_step)
- Allocate pinned device-mapped rng_step in GpuReplayBuffer
- Wire rng_step_dev_ptr from replay buffer to FusedTrainingCtx
- Remove stale atomicAdd comment
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move sample_proportional (prefix scan, binary search, gather x6, IS weights)
from training_loop.rs into run_full_step, captured as per_sample child graph
node. Steps 1+ replay all 13 children via single cuGraphLaunch with no
ungraphed PER kernel dispatches.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Composes 11 child graphs into single parent via cuGraphAddChildGraphNode.
Counter increments (GPU-side), IQL modulate, PER priority update all
captured as child graph nodes. PhaseEvents removed — single parent launch
makes per-child event profiling irrelevant.
Step 0 runs ungraphed + captures all 11 children + composes parent.
Steps 1+ replay via single cuGraphLaunch on the training stream.
After graph launch: only pinned scalar reads (nanoseconds, no GPU ops).
Child graph order:
counters -> spectral -> forward -> ddqn -> aux -> post_aux ->
adam_grad -> adam -> maintenance -> iql_modulate -> per_priority
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PER gather now writes directly to GpuDqnTrainer's padded buffers via
gather_f32_rows_padded, gather_f32_scalar, and gather_i32_scalar kernels
compiled into the replay buffer cubin. set_trainer_buffers() wires stable
device pointers at init; sample_proportional uses them when available,
falling back to intermediate buffers otherwise. memset_zeros calls in
the sampling hot path converted to raw cuMemsetD8Async.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds 4 GPU-native kernels replacing host-side operations on the training
hot path: gather_f32_rows_padded (row gather + 128-byte zero-pad),
gather_f32_scalar, gather_i32_scalar, and increment_step_counters (atomic
counter bumps + cosine-annealed tau — zero CPU sync per step).
Wired into build.rs (nvcc cubin compile) and gpu_dqn_trainer.rs
(GRAPH_UTILITY_CUBIN include_bytes!).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
shrink_and_perturb() is called at epoch boundaries after child graphs are
captured. shrink_perturb_kernel lives in the same CUmodule as scale_f32,
saxpy_f32, spectral_norm, adam_update, and other graphed CUfunctions.
On Hopper (sm_90), launching any CUfunction from a graphed CUmodule
ungraphed corrupts the child graph kernel state → 3100ms replay.
scale_adam_momentum() is called at cosine LR warm restarts (also after
graph capture). scale_f32_kernel is captured in forward_child — same
CUmodule violation.
Fix: add shrink_perturb_ungraphed and scale_f32_ungraphed loaded from
the existing ungraphed_module (separate CUmodule instance of the same
DQN_UTILITY_CUBIN). Both ungraphed callers now use isolated handles.
CUmodule count: 5 → 5 (ungraphed_module already existed, reused).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ROOT CAUSE: pad_states_kernel and stochastic_depth_rng_kernel were loaded from
the SAME CUmodule as graph-captured kernels (adam_update, grad_norm, saxpy, etc.)
but launched UNGRAPHED every step. On Hopper, launching ANY CUfunction from a
CUmodule that also has graph-captured CUfunctions corrupts the graph's kernel
state — causing 3100ms replay instead of ~30ms.
Fix: load from a separate ungraphed_module. Zero CUmodule contamination between
graphed and ungraphed execution contexts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Single cuGraphExecLaunch per step. PER sampling, gather, training,
priority update ALL as child graph nodes in one parent. Direct-to-trainer
gather eliminates DtoD copies. GPU-side counters eliminate host writes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Enables nsys profiling for CUDA graph node-level traces on H100.
Use --sanitizer nsys in argo-train.sh to activate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use --sanitizer nsys to wrap training binary with nsys profile.
Captures CUDA graph node-level traces + GPU metrics.
Output: /workspace/output/nsys_profile.nsys-rep
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three changes to eliminate the 3100ms replay regression on Hopper:
1. CUfunction isolation: saxpy_f32_kernel was shared across 3 child graphs
(forward, aux, adam_grad). Added saxpy_f32_adam_grad and saxpy_f32_aux from
separate CUmodules. Also isolated grad_norm_standalone for post_aux_child
(was shared with forward_child's d_logits clipping path).
2. Raw cuMemsetD8Async: replaced all cudarc memset_zeros in graph-captured
functions (submit_forward_ops_main, apply_cql_gradient,
run_causal_intervention_unconditional) with raw cuMemsetD8Async which is
properly captured by CUDA Graph. 6 call sites fixed.
3. DtoD memcpy audit: all memcpy_dtod_async calls verified — large buffer
copies (grad snapshot 2.6MB, multi-horizon blend) are correct for DtoD;
no scalar copies found that should use pinned memory.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE of 3100ms adam_child: adam_update_kernel and grad_norm_finalize_kernel
were shared between post_aux_child and adam_update/forward children. On Hopper,
CUfunction sharing across child graphs corrupts kernel state.
Fix: load utility cubin from separate CUmodule for post_aux child, giving it
isolated adam_update_post_aux, grad_norm_finalize_post_aux, and scale_f32_post_aux
handles. Zero cross-child CUfunction sharing for these kernels.
Also: convert memset_zeros to raw cuMemsetD8Async in IQN (3 calls), IQL (1),
Attention (1) — eliminates cudarc device_ptr_mut() overhead during graph capture.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds try-fused-fallback-to-separate pattern to forward_target_raw,
forward_online_f32, and forward_value_head — matching the existing
pattern in forward_online_raw. Eliminates ~10-14 separate bias+ReLU
kernel launches by fusing them into the preceding cuBLAS GEMM epilogue.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add post_aux_child (selectivity + denoise + risk_sgd + multi-horizon) and
maintenance_child (causal intervention + gradient vaccine) to the CUDA graph
pipeline. All kernel launches now run inside child graphs; only kernel-free
ops (pinned readbacks, host counters, IQL modulate_td_errors on its own
CUfunction) remain outside the graph.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add child-graph-capturable submit methods that run post-aux and
maintenance ops without conditional guards (graphs_captured, step %
interval). This enables moving all kernel launches into CUDA child
graphs, eliminating ungraphed ops that cause CUfunction corruption on
Hopper during 3100ms adam replays.
- prev_grad_buf: snapshot of grad_buf after Adam for next-step vaccine
- submit_post_aux_ops: selectivity, denoise, risk SGD, multi-horizon
- submit_maintenance_ops: causal intervention + gradient vaccine
- run_causal_intervention_unconditional: no step/graph guards
- apply_gradient_vaccine_from_prev: uses prev_grad_buf instead of
separate forward+backward pass
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: CUfunction sharing between graphed children and ungraphed
outside-graph ops corrupts kernel state on Hopper → 3100ms adam replay.
Fix: capture EVERYTHING unconditionally in child graphs. Selectivity,
denoise, causal intervention, vaccine, Q-stats all become graphed children.
Zero ungraphed launches = zero CUfunction conflicts.
7 children, ~190 kernel nodes, one capture, one replay per step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
