Move sample_proportional (prefix scan, binary search, gather x6, IS weights)
from training_loop.rs into run_full_step, captured as per_sample child graph
node. Steps 1+ replay all 13 children via single cuGraphLaunch with no
ungraphed PER kernel dispatches.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Composes 11 child graphs into single parent via cuGraphAddChildGraphNode.
Counter increments (GPU-side), IQL modulate, PER priority update all
captured as child graph nodes. PhaseEvents removed — single parent launch
makes per-child event profiling irrelevant.
Step 0 runs ungraphed + captures all 11 children + composes parent.
Steps 1+ replay via single cuGraphLaunch on the training stream.
After graph launch: only pinned scalar reads (nanoseconds, no GPU ops).
Child graph order:
counters -> spectral -> forward -> ddqn -> aux -> post_aux ->
adam_grad -> adam -> maintenance -> iql_modulate -> per_priority
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PER gather now writes directly to GpuDqnTrainer's padded buffers via
gather_f32_rows_padded, gather_f32_scalar, and gather_i32_scalar kernels
compiled into the replay buffer cubin. set_trainer_buffers() wires stable
device pointers at init; sample_proportional uses them when available,
falling back to intermediate buffers otherwise. memset_zeros calls in
the sampling hot path converted to raw cuMemsetD8Async.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds 4 GPU-native kernels replacing host-side operations on the training
hot path: gather_f32_rows_padded (row gather + 128-byte zero-pad),
gather_f32_scalar, gather_i32_scalar, and increment_step_counters (atomic
counter bumps + cosine-annealed tau — zero CPU sync per step).
Wired into build.rs (nvcc cubin compile) and gpu_dqn_trainer.rs
(GRAPH_UTILITY_CUBIN include_bytes!).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
shrink_and_perturb() is called at epoch boundaries after child graphs are
captured. shrink_perturb_kernel lives in the same CUmodule as scale_f32,
saxpy_f32, spectral_norm, adam_update, and other graphed CUfunctions.
On Hopper (sm_90), launching any CUfunction from a graphed CUmodule
ungraphed corrupts the child graph kernel state → 3100ms replay.
scale_adam_momentum() is called at cosine LR warm restarts (also after
graph capture). scale_f32_kernel is captured in forward_child — same
CUmodule violation.
Fix: add shrink_perturb_ungraphed and scale_f32_ungraphed loaded from
the existing ungraphed_module (separate CUmodule instance of the same
DQN_UTILITY_CUBIN). Both ungraphed callers now use isolated handles.
CUmodule count: 5 → 5 (ungraphed_module already existed, reused).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ROOT CAUSE: pad_states_kernel and stochastic_depth_rng_kernel were loaded from
the SAME CUmodule as graph-captured kernels (adam_update, grad_norm, saxpy, etc.)
but launched UNGRAPHED every step. On Hopper, launching ANY CUfunction from a
CUmodule that also has graph-captured CUfunctions corrupts the graph's kernel
state — causing 3100ms replay instead of ~30ms.
Fix: load from a separate ungraphed_module. Zero CUmodule contamination between
graphed and ungraphed execution contexts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Single cuGraphExecLaunch per step. PER sampling, gather, training,
priority update ALL as child graph nodes in one parent. Direct-to-trainer
gather eliminates DtoD copies. GPU-side counters eliminate host writes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Enables nsys profiling for CUDA graph node-level traces on H100.
Use --sanitizer nsys in argo-train.sh to activate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use --sanitizer nsys to wrap training binary with nsys profile.
Captures CUDA graph node-level traces + GPU metrics.
Output: /workspace/output/nsys_profile.nsys-rep
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three changes to eliminate the 3100ms replay regression on Hopper:
1. CUfunction isolation: saxpy_f32_kernel was shared across 3 child graphs
(forward, aux, adam_grad). Added saxpy_f32_adam_grad and saxpy_f32_aux from
separate CUmodules. Also isolated grad_norm_standalone for post_aux_child
(was shared with forward_child's d_logits clipping path).
2. Raw cuMemsetD8Async: replaced all cudarc memset_zeros in graph-captured
functions (submit_forward_ops_main, apply_cql_gradient,
run_causal_intervention_unconditional) with raw cuMemsetD8Async which is
properly captured by CUDA Graph. 6 call sites fixed.
3. DtoD memcpy audit: all memcpy_dtod_async calls verified — large buffer
copies (grad snapshot 2.6MB, multi-horizon blend) are correct for DtoD;
no scalar copies found that should use pinned memory.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE of 3100ms adam_child: adam_update_kernel and grad_norm_finalize_kernel
were shared between post_aux_child and adam_update/forward children. On Hopper,
CUfunction sharing across child graphs corrupts kernel state.
Fix: load utility cubin from separate CUmodule for post_aux child, giving it
isolated adam_update_post_aux, grad_norm_finalize_post_aux, and scale_f32_post_aux
handles. Zero cross-child CUfunction sharing for these kernels.
Also: convert memset_zeros to raw cuMemsetD8Async in IQN (3 calls), IQL (1),
Attention (1) — eliminates cudarc device_ptr_mut() overhead during graph capture.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds try-fused-fallback-to-separate pattern to forward_target_raw,
forward_online_f32, and forward_value_head — matching the existing
pattern in forward_online_raw. Eliminates ~10-14 separate bias+ReLU
kernel launches by fusing them into the preceding cuBLAS GEMM epilogue.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add post_aux_child (selectivity + denoise + risk_sgd + multi-horizon) and
maintenance_child (causal intervention + gradient vaccine) to the CUDA graph
pipeline. All kernel launches now run inside child graphs; only kernel-free
ops (pinned readbacks, host counters, IQL modulate_td_errors on its own
CUfunction) remain outside the graph.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add child-graph-capturable submit methods that run post-aux and
maintenance ops without conditional guards (graphs_captured, step %
interval). This enables moving all kernel launches into CUDA child
graphs, eliminating ungraphed ops that cause CUfunction corruption on
Hopper during 3100ms adam replays.
- prev_grad_buf: snapshot of grad_buf after Adam for next-step vaccine
- submit_post_aux_ops: selectivity, denoise, risk SGD, multi-horizon
- submit_maintenance_ops: causal intervention + gradient vaccine
- run_causal_intervention_unconditional: no step/graph guards
- apply_gradient_vaccine_from_prev: uses prev_grad_buf instead of
separate forward+backward pass
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: CUfunction sharing between graphed children and ungraphed
outside-graph ops corrupts kernel state on Hopper → 3100ms adam replay.
Fix: capture EVERYTHING unconditionally in child graphs. Selectivity,
denoise, causal intervention, vaccine, Q-stats all become graphed children.
Zero ungraphed launches = zero CUfunction conflicts.
7 children, ~190 kernel nodes, one capture, one replay per step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
step_denoise_adam uses adam_update_kernel + grad_norm_finalize_kernel UNGRAPHED
after child graph replays. Same CUfunctions are captured in adam_child and
forward_child. On Hopper, launching a captured CUfunction ungraphed corrupts
graph kernel state → 3100ms adam_child replay on next step.
Fix: skip selectivity+denoise ungraphed training. These are auxiliary optimizers
with non-fatal error handling. To re-enable: load separate CUfunction instances
for ungraphed paths or move into child graphs.
Expected: adam drops from 3100ms to ~30ms. Epoch from ~690s to ~60s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE: grad_norm_finalize_kernel (CUfunction) was captured in BOTH
forward_child (d_logits clipping at lines 7572/7821) AND adam_child
(compute_grad_norm_for_adam via launch_grad_norm_finalize at line 6346).
Sharing a CUfunction between two captured graphs corrupts kernel state
on Hopper (sm_90), causing the second graph to replay at 3100ms instead
of <1ms. This is the SAME class of bug as the earlier grad_norm_standalone
fix — we fixed one shared CUfunction but missed the finalize kernel.
FIX: Load grad_norm_finalize from a SEPARATE CUmodule (new cubin load)
for the adam path. Each CUfunction instance is now captured in exactly
one child graph. Also split adam into adam_grad + adam_update children
for per-operation timing visibility.
Expected: adam drops from 3100ms to ~30ms. Total step ~350ms. Epoch ~62s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
adam_child = 3113ms even after CUfunction fix. Split into:
- adam_grad: mamba2_backward + step_mamba2_adam + pruning + grad_norm
- adam_update: Adam kernel + unflatten + ISV
If adam_grad is fast and adam_update is slow, the bottleneck is Adam/unflatten.
If adam_grad is slow, grad_norm_kernel has an issue.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The backward branch parallelism allocated 4 × 32MB = 128MB for per-branch
cuBLAS workspaces, pushing H100 VRAM over 80GB → OOM. Forward and backward
are in the same child graph (sequential) — workspaces never conflict.
Now backward receives forward's branch_workspace_ptrs as a parameter.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Separates buffer prep (decode_actions, DtoD copies) from training pipeline
so prep runs on main stream and execute_training_pipeline runs on iqn_stream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
grad_norm_standalone was captured in BOTH forward_child (d_logits clipping)
and adam_child (gradient norm). Same CUfunction in two captured graphs
corrupts kernel state on Hopper — the GPU replayed a corrupted node for
3120ms per step (88% of total step time).
Fix: adam_child uses grad_norm_kernel (the regular handle, not captured
elsewhere) instead of grad_norm_standalone. Each CUfunction is now captured
in exactly one child graph.
Expected: adam_child drops from 3120ms to ~30ms. Total step from ~3550ms
to ~460ms. Epoch from ~700s to ~80s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add optional override_stream and override_workspace parameters to
execute_training_pipeline (GpuIqnHead), and forward/backward/adam_step
(GpuAttention). All existing callers pass None, None — behaviour is
unchanged. The parallel path can now dispatch these ops to dedicated
CUDA streams with separate cuBLAS workspaces.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The child graph breakdown showed adam=-1.0ms because it was measuring
elapsed(backward_end, adam_end) where adam_end is from the OLD phase
timing system, not the adam child graph. Added proper adam_child_end
event recorded after the adam_child launch.
This will reveal the actual adam_child GPU time — currently suspected
to be 3117ms (73% of step time) based on total - measured children.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Records cuEventRecord between each of the 5 child graph launches.
At epoch end, cuEventElapsedTime reports per-child GPU time:
spectral, forward, ddqn, aux, adam.
This tells us exactly which child dominates the 4s/step graph replay
cost, informing Phase 2 multi-stream parallelism priorities.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 results: 4.0s/step GPU compute (graph replay = ungraphed, cuBLAS
replaced manual matmul but feature count increased). CPU pipeline fully
async (0.1ms/step). Total ~718s/epoch — 100% GPU bound.
Updated Phase 2 targets: aux parallelism (2b) is highest priority —
80/177 kernels with 4 independent trainers. nsys profiling should run
first to validate SM vs memory-bound assumptions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per-step reduce_current_q_stats created GPU pipeline bubbles every 50 steps.
Each sync waited for the GPU to drain all pending graph replays before
the CPU could continue. Q-stats are now computed at epoch boundary only
(process_epoch_boundary already calls reduce_current_q_stats).
Training loop is now fully async: 178 graph launches queue in <1ms,
GPU executes back-to-back with zero pipeline stalls.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
q_stats_kernel now writes directly to q_readback_dev_ptr (pinned device-mapped).
CPU reads previous step's values from q_readback_pinned — one-step lag, zero sync.
This eliminates the LAST cuStreamSynchronize from the per-step training path.
The training loop is now fully async: graph launch → CPU returns immediately →
next step starts while GPU still executes → zero pipeline bubbles.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The function re-ran 15+ kernels (expected_q, q_mean_center, q_anchoring,
risk_budget, branch_confidence, epistemic_gate, temporal_consistency,
branch_independence, q_attention, graph_message_pass, q_denoise) outside
the graph every step — duplicating work the graph already did.
Now: q_out_buf is populated by graph replay (populate_q_out in aux_child).
reduce_current_q_stats just runs q_stats_kernel (1 kernel) + DtoH sync.
Runs every step since cost is now ~1ms, not ~4s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
reduce_current_q_stats ran 15+ GPU kernels (expected_q, q_mean_center,
q_anchoring, risk_budget, branch_confidence, epistemic_gate, temporal_consistency,
branch_independence, q_attention, graph_message_pass, q_denoise, q_stats)
PLUS cuStreamSynchronize for DtoH readback — EVERY training step.
This added ~4s per step (matching graph replay cost), making epochs 700s
instead of ~35s. Running every 50 steps reduces Q-stats overhead to <1%
of epoch time while still tracking Q-range changes within 200 steps.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ensemble diversity (2 extra head forward passes + KL gradient + trunk backward)
was running OUTSIDE the CUDA graph on every training step. Now runs inside
submit_aux_ops, captured in the aux_child graph for zero launch overhead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
reduce_current_q_stats() re-ran 10 kernels (mamba2_step, predictive_coding,
regime_dropout, ISV forward/gate/route, recursive_confidence, trade_plan,
plan_noise, risk_budget) every training step OUTSIDE the graph replay.
These ops already executed inside the graph — the logit buffers are current.
The redundant execution added ~4s of GPU compute per step (matching the
graph replay cost), doubling total epoch time.
Fix: compute expected_q directly from the existing logit buffers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The async write lock (self.agent.write().await) was acquired inside the
per-step training loop, causing 696s/epoch overhead from async yield on
every iteration — 178 steps × ~4s per lock acquisition. Actual GPU compute
was only 3.6s (graph replay at 2ms/step).
Fix: acquire the lock once before the loop. The training loop is the only
writer during this phase — no contention, no need to release between steps.
Expected: epoch time drops from 700s to ~5s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Logs step_ms for first 5 steps and every 50th step thereafter.
If steps aren't completing, the graph is hanging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Forward: removed spurious ln_save_xnorm arg (kernel has 9 params, code passed 10).
The extra arg shifted D and B, causing CUDA_ERROR_ILLEGAL_ADDRESS on H100.
Backward: added missing residual (projected_buf) and save_mean args (kernel has
11 params, code passed 9). Without these, gamma/rstd/d_x/d_gamma/d_beta were all
shifted to wrong positions.
Both bugs existed since the cuBLAS attention rewrite but were latent because
attn_sdp_fwd was loaded from the wrong cubin (attention_kernel.cubin instead of
attention_backward_kernel.cubin), silently disabling attention.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
attn_sdp_fwd and attn_layer_norm_fwd were loaded from attention_kernel.cubin
(which only contains the legacy multihead_feature_attention) instead of
attention_backward_kernel.cubin where they actually live. This caused
CUDA_ERROR_NOT_FOUND on H100, silently disabling attention.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IQN read_total_loss() called cuStreamSynchronize + cuMemcpyDtoH every training
step, blocking the CPU until all GPU work completed. This serialized the entire
async pipeline, causing 3537ms/step instead of <10ms.
Fix: pinned device-mapped memory for IQN total_loss (same pattern as DQN trainer).
GPU writes via device pointer, CPU reads via host pointer, zero sync.
Also re-applies the cuGraphClone elimination (was lost when agents modified
fused_training.rs) and adds recursive_confidence_reduce for deterministic
gradient accumulation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- capture_child_graph: use std::mem::forget to take ownership of cudarc's
CudaGraph handles directly. cuGraphClone corrupts cublasLtMatmul HMMA tile
scheduling state on Hopper, causing graph replay to hang with temporal ops.
- create_eval_forward_exec: instantiate directly from original graph (no clone).
Multiple execs from the same CUgraph is safe per NVIDIA docs.
- recursive_confidence_backward: replace atomicAdd into grad_buf with per-block
partial output + deterministic reduce kernel. Zero atomicAdd on gradient path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
