Temporary diagnostics between each fused training step to identify
which operation hangs after replay_forward on H100.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eliminated 5 per-step GPU allocations:
- insert_prio_buf, insert_idx_buf, insert_ep_buf: 3x cuMemAlloc per
insert_batch/insert_batch_bf16 call (up to 4M elements each)
- scratch_f32: reusable 1-element temp for flush_max_priority and
apply_max_priority_scalar (was 2x a32f per call)
- update_batch_spare: pre-allocated spare for None→Some priority swap,
replenished by flush_max_priority (was 1x a32f per epoch start)
cuMemAlloc is a synchronizing driver call that stalls the GPU pipeline.
With 8192 batch_size and multiple inserts per epoch, this was measurable
overhead on H100.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The per_prefix_scan kernel was processing capacity/256 tiles (96K on H100
with 24.7M auto-sized capacity) instead of size/256 tiles (~3200 for 819K
experiences). This caused the scan to take ~50ms per step, appearing as a
hang (6.5 min/epoch). Now computes actual_tiles from self.size at scan
time — all tiles fit in one round of 512 resident blocks (<1ms).
Removed dead num_scan_tiles field. Added regression test with 1M capacity
and 256 experiences to catch the high capacity/size ratio failure mode.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With 96K tiles on H100 (132 SMs, ~500 resident blocks), the original
per_prefix_scan launched one block per tile. Blocks waiting on
predecessors that weren't scheduled created circular deadlocks.
Fix: blocks grab tiles dynamically via atomicAdd on a global counter.
Grid size capped at 512 (max resident). Each block processes multiple
tiles in a while loop, ensuring all tiles complete without preemption.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove diagnostic eprintln calls from the H100 hang investigation:
- H100_STEP, H100_LOOP, H100_HANG4 prefixes in fused_training.rs
- adam_readback prefix + ADAM_CTR static counter in gpu_dqn_trainer.rs
- H100_LOOP + FUSED STEP ERROR in training_loop.rs
- Debug cuStreamSynchronize block after PER update in fused_training.rs
Integration tests and smoke tests already match current signatures
(ext_stream: Option<&Arc<CudaStream>> with None). seg_tree_kernel.cu
already deleted in prior commit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace segment tree sampling with prefix scan + binary search:
- sample_proportional: per_prefix_scan builds prefix sum, per_sample does
Philox RNG threshold search with binary search over prefix sum
- insert_batch/insert_batch_bf16: per_insert_pa writes priority^alpha
directly to priorities_pa (no rebuild needed, scan runs at sample time)
- Delete seg_tree_kernel.cu (no longer referenced)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The full tree rebuild (25 levels × 33M nodes) took too long on H100.
Only 8192 of 33M leaves change per step — rebuilding all nodes is
4000× wasted work.
Restore atomicAdd delta propagation in seg_tree_update_leaves:
O(B × log N) = 8192 × 25 = 200K atomics vs O(N) = 33M reads.
atomicExch for leaf writes (duplicate-index safe).
Keep seg_tree_rebuild_level for insert_batch (new slots need full rebuild).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cuModuleLoadData hangs on H100 CUDA 13 when called after CUDA graph
capture — the context state appears incompatible with module loading.
Pre-initialize the OnceLock during GpuReplayBuffer::new() when the
context is clean and no graphs have been captured.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
get_cast_kernels uses OnceLock + cuModuleLoadData which requires the
CUDA context to be bound to the current thread. When ext_stream (the
trainer's stream) was passed, its context wasn't bound — causing a hang
in cuModuleLoadData on H100.
Fix: always use self.stream (replay buffer's stream, with bound context)
for kernel compilation. Only use ext_stream for the actual kernel launches.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The atomicAdd delta propagation caused severe L2 cache contention on
H100: 8192 threads all atomicAdd to tree[1] (root) serialized through
a single cacheline, taking 10+ seconds per PER update.
Split into two phases:
1. seg_tree_update_leaves: write all leaves in parallel (zero contention)
2. seg_tree_rebuild_level: rebuild one tree level per kernel launch,
bottom-up. 20 levels × ~2µs = ~40µs total. Each level is fully
parallel — nodes at the same level are independent.
Also applies to seg_tree_insert (insert_batch path).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The replay buffer used its own CUDA stream for seg_tree_update, but
the td_errors buffer was computed on the trainer's stream. Without
synchronization, the replay stream could read incomplete data — causing
a hang on H100 where streams execute truly concurrently.
Fix: update_priorities_gpu now accepts an optional ext_stream parameter.
The fused training path passes the trainer's main stream, ensuring all
GPU work (td_errors computation + seg_tree_update) runs on the same
stream with implicit ordering.
Also pre-allocate update_td_f32, update_batch_max, update_max_merge
scratch buffers at construction — eliminates 3 cuMemAlloc calls per
training step that could trigger implicit device synchronization.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PER samples with replacement, so a batch can contain duplicate indices.
With non-atomic leaf writes, two threads processing the same index both
read the same old_pa, but only one write survives — internal nodes get
double the delta while the leaf changed once, permanently corrupting
the tree sum.
atomicExch returns the exact replaced value so each thread propagates
the correct delta regardless of concurrent duplicate writes.
Also fix readback_pinned layout doc: slot 9 is used for diversity loss.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Capture the batched spectral norm kernel (10 matrices) + bf16→f32 sync
into a CUDA graph on step 1, replayed on steps 2+ with zero launch
overhead. Spectral norm uses fixed device pointers (params_bf16,
spec_u/v descriptors) that never reallocate, making it a perfect graph
capture candidate. Graph is invalidated on fold reset.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove 3 memset_zeros calls on buffers that are fully overwritten by
their consumer kernels (no atomicAdd accumulation):
- q_stats_buf (2 sites): q_stats_reduce is a single-thread kernel that
directly assigns all 5 output floats — no accumulation pattern
- causal_mean_scratch (1 site): causal_mean_reduce writes out[0] via
direct assignment in thread 0 — no atomicAdd
All remaining memsets are documented as REQUIRED with the specific
accumulation pattern (atomicAdd / beta=1.0 GEMM) that necessitates them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Audit all grid=(1,1,1) kernels called per training step:
- grad_norm_finalize: cannot fuse into grad_norm_kernel because the
reduction uses multi-block atomicAdd — no block knows it holds the
final sum without cooperative launch. 1-thread finalize is negligible.
- stochastic_depth_rng: standalone RNG writing 3 floats, not a finalizer
after a reduction — nothing to fuse into.
- causal_reduce, causal_mean_reduce, q_stats: only called at
intervention-interval or epoch-boundary, not per training step.
Document rationale inline so future reviewers don't re-investigate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace 10 individual `spectral_norm_kernel` launches (each grid=(1,1,1))
with a single `spectral_norm_batched` launch (grid=(10,1,1)) that processes
all weight matrices in parallel across 10 blocks.
- Build descriptor buffer at construction (10 × 6 u64 entries with W/u/v
pointers, out_dim, in_dim per matrix) — pointers are stable
- Delete `spectral_norm_kernel` from dqn_utility_kernels.cu (replaced by
`spectral_norm_batched` which was already written)
- Remove spec_norm! macro and per-matrix launch loop
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Switch all 3 forward-pass cublasGemmEx calls from CUBLAS_COMPUTE_32F to
CUBLAS_COMPUTE_32F_FAST_TF32, enabling H100 TF32 tensor cores for 2-3x
throughput. Backward pass remains at full F32 precision for gradient stability.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Refactor spectral normalization to operate directly on params_bf16 via
raw u64 offset pointers (computed by bf16_weight_ptrs_from_base) instead
of separate DuelingWeightSet/BranchingWeightSet CudaSlice allocations.
This eliminates 10 memcpy_dtod_async calls per training step (~25MB/step,
25GB/epoch at 1000 steps) that previously synced spectrally-normalized
weights from per-layer slices back to the flat params_bf16 buffer.
Key changes:
- apply_spectral_norm now takes &DuelingWeightSet (immutable) instead of
&mut, only needed for first-call flatten into params_bf16
- Spectral norm kernels receive raw u64 pointers at GOFF_* offsets into
params_bf16, writing in-place — no sync copies needed
- Spectral norm u/v vectors remain separate allocations (unchanged)
- unflatten_online_weights promoted to pub(crate) for test access
- Tests updated to call unflatten after spectral norm for readback
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
seg_tree_update and seg_tree_insert used non-atomic
tree[node] = tree[2*node] + tree[2*node+1] with 8192 threads racing
on shared internal nodes. On H100 (132 SMs), all threads execute
simultaneously causing data races and hangs.
Replace with atomicAdd delta propagation: each thread computes
delta = new_leaf - old_leaf, then atomicAdd to every ancestor.
Commutative + associative = race-free. O(log n) per thread,
hardware-accelerated on SM 9.0.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace synchronous stream.synchronize() + memcpy_dtoh + CPU sum in
run_causal_intervention with a GPU reduction kernel (causal_mean_reduce)
and async DtoH to the pinned readback buffer at slot 8. The result is
double-buffered: the caller reads the previous iteration's value while
the current one transfers asynchronously.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extend readback_pinned from 3 to 16 floats and write Q-stats DtoH
into slots [3..8] instead of the non-pinned q_stats_pending field.
On H100 CUDA 13 driver, non-pinned host memory causes cuMemcpyDtoHAsync
to degrade to a synchronous copy, creating a latent hang risk.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GpuTrainResult was replaced by FusedStepResult (CPU-only f32 scalars).
F32ToU32Caster, get_cast_kernels_f32_to_u32, and f32_slice_to_gpu_tensor_gpu
were only used by the old pre-fused training path. Also removes
#[allow(dead_code)] from gather_f32 which is actively used.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the orphaned per_update_kernel call site in fused_training.rs
with agent.update_priorities_from_td() which routes through
GpuReplayBuffer::update_priorities_gpu_raw -> seg_tree_update.
This kernel correctly writes priorities AND propagates the segment tree
from leaf to root, fixing stale PER sampling.
Add update_priorities_from_td() on ReplayBufferType and DqnAgentConfig.
Remove priorities_f32_ptr() which exposed raw pointers for the deleted
kernel. Remove batch_max epoch-boundary flush since seg_tree_update
propagates max_priority internally.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The per_update_priorities_kernel scatter-wrote to the priorities[] buffer
but never updated the PER segment tree, leaving sampling priorities stale.
Remove the orphaned CUDA kernel, its cubin include, struct field,
compilation, and the update_priorities_cuda_raw / batch_max_readback_and_reset
methods from GpuDqnTrainer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GpuTrainResult::from_fused_scalars() allocated GPU memory and performed
two synchronous cuMemcpyHtoD copies after every training step. On H100
CUDA 13 driver, synchronous HtoD blocks until ALL pending async GPU work
completes — same class of bug as the DtoH pinned memory fix.
The uploaded scalars were never used: the training guard already reads
loss/grad_norm directly from GPU-resident buffers via loss_gpu_buf() /
grad_norm_gpu_buf(). Replace GpuTrainResult with lightweight
FusedStepResult (CPU-only f32 scalars, zero GPU ops).
Also adds H100_HANG4 diagnostic eprintln between PER update,
bookkeeping, varmap sync, and graph_aux capture to isolate if any
secondary hang exists.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cuMemcpyDtoHAsync with non-pinned host pointers blocks the CPU on
CUDA 13 / H100 driver 580+ until ALL pending GPU work completes.
With graph_forward + aux_ops + adam queued, this caused multi-minute
hangs between training steps.
Fix: allocate 12 bytes of pinned (page-locked) host memory via
cuMemHostAlloc(DEVICEMAP) for the 3 scalar readbacks (loss, mse_loss,
grad_norm). Pinned memory enables true async DtoH — CPU returns
immediately, GPU copies when it reaches that point in the stream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cuMemcpyDtoHAsync_v2 with non-pinned host pointer degrades to
synchronous copy on CUDA 13 / H100 driver 580+, blocking until
ALL pending GPU work completes. With graph_forward + aux_ops + adam
queued, this caused a multi-minute hang.
Fix: explicit cuStreamSynchronize + synchronous cuMemcpyDtoH.
Stream sync + log at timestep 0, 50%, and 100% to identify which
phase of experience collection hangs on H100 with 4096 episodes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Both were unused (bf16 path replaced by f32 path in #30).
build_next_states_dtod had the same CPU loop anti-pattern (795K memcpy
calls at 4096 episodes) that caused the H100 hang.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of H100 training hang: build_next_states_f32() used a CPU
for-loop launching individual memcpy_dtod per episode per fill timestep.
At gpu_n_episodes=4096 × 2 counterfactual × 97 fill steps = 795K CUDA
API calls that saturated the driver command queue.
Replaced with single build_next_states_kernel launch — all episodes
processed in parallel, zero CPU loops. One kernel, done.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Moved init_from_fxcache BEFORE fold generation so GPU features buffer
exists when generate_folds_stratified_gpu is called. Features uploaded
as temporary f32 for regime classification (freed after fold gen).
Removes TODO — GPU prefix sum path is now the default when CUDA available.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
generate_folds_stratified_gpu uses GPU prefix sums for O(1) range
queries during boundary adjustment. 1.1M bars: ~3ms vs ~1000ms CPU.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
regime_classify_kernel: parallel classification of 1.1M bars
regime_prefix_sum_kernel: cumulative counts for O(1) range queries
classify_regimes_gpu(): Rust wrapper that runs both and returns host prefix sums
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Split graph_forward into graph_forward_main (Pass 1+2+loss+backward
on main stream) and graph_forward_ddqn (Pass 3 on double_dqn_stream).
Event synchronization happens in ungraphed replay_forward() code,
not inside captured functions. Each graph uses its own CublasForward
handle — no set_stream() during capture.
Fixes: H100 training hang with gpu_n_episodes=4096.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
submit_forward_ops_main: Pass 1 + 2 + loss + grad + backward (main stream)
submit_forward_ops_ddqn: Pass 3 Double DQN (double_dqn_stream)
No set_stream() or event ops inside either function.
Graph capture will record each on its own stream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
