Force cudarc's is_managing_stream_synchronization() to false.
Eliminates ALL internal event tracking (cuStreamSynchronize,
cuEventRecord, cuStreamWaitEvent) that was adding 18.7 syncs/step.
All stream synchronization is managed manually via raw_launch events.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Capture the experience collector's bottleneck + Q-network forward pass
into a CUDA Graph on first timestep. Prevents ungraphed cublasLtMatmul
calls from contaminating cuBLAS internal state between epochs.
Root cause identified: multiple cuBLAS handles on the same CUDA stream.
NVIDIA docs: "maintain a separate cuBLAS handle for each stream" —
implying one handle per stream. We have 3 handles (experience collector,
training, DDQN) on one stream.
Proper fix (next session): refactor CublasForward to share a single
cuBLAS handle across all components. Separate the handle (shared)
from per-component cached descriptors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Route evaluation Q-values through graph_mega (the EXACT graph used for
training). Save params before replay, restore after to undo Adam's
weight update. Persist graph_mega and graph_forward across fold
boundaries — all buffer addresses are stable, kernel args use pinned
device-mapped pointers.
This guarantees training and eval use identical cublasLtMatmul kernel
configs from the same capture session. Both are fully deterministic
WITHIN a given process. Cross-process variation remains from the
initial graph capture (NVIDIA cublasLtMatmul heuristic selects
different kernel configs between process invocations).
Cleanup: removed separate eval_fwd_graph, eval_params_snapshot serves
dual purpose (fold boundary save + eval rollback).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Capture a separate eval-only CUDA Graph immediately after graph_forward
(same cublasLtMatmul internal state). The exec handle is leaked via
mem::forget to survive graph_forward invalidation between folds.
Eval replays this frozen graph instead of the recaptured training
graph_forward, giving consistent results across walk-forward folds.
Training: fully bit-identical (verified with weight checksums).
Evaluation: identical epochs 1-4 across runs. Remaining minor
variation from cublasLtMatmul internal state differences between
consecutive capture sessions — NVIDIA library limitation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: cuMemAllocAsync (stream-ordered memory pools) produces
memory that cublasLtMatmul rejects with CUBLAS_STATUS_NOT_SUPPORTED
on H100 (CUDA 13.0 driver) when the matmul runs on a different
stream than the allocation. Despite CUDA documentation stating
same-device async allocations should be accessible from all streams
with proper event synchronization, cublasLtMatmul disagrees.
Proof chain:
- C++ test with cudaMalloc: ALL dimensions pass on H100 ✓
- Rust test with cuMemAlloc: ALL dimensions pass on H100 ✓
- Rust test with cudarc alloc_zeros (cuMemAllocAsync): FAILS ✗
- Same dimensions, same handle, same workspace — only alloc differs
Fix: set has_async_alloc=false unconditionally in CudaContext::new().
This forces cuMemAlloc_v2 for ALL allocations. Zero performance impact
(allocations at construction time, not in hot loops).
Also: converted has_async_alloc from bool to AtomicBool for safety.
Removed diagnostic code (CUBLASLT_DIAG, test_lt_matmul_raw).
Reverted cuMemAlloc workspace hack (now uses normal alloc_zeros).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: cudarc uses cuMemAllocAsync (stream-ordered memory pools)
on H100 where CU_DEVICE_ATTRIBUTE_MEMORY_POOLS_SUPPORTED=1. These
allocations are only accessible from the allocating stream.
cublasLtMatmul dispatches GEMM work on branch streams (multi-stream
fork-join for parallel advantage heads), which can't access
stream-local memory from the main stream — returns NOT_SUPPORTED.
Proof: C++ test with cudaMalloc passes ALL dims on H100 ✓
Rust test_lt_matmul_raw with cuMemAlloc passes on H100 ✓
Same Rust code with cudarc alloc_zeros (cuMemAllocAsync) FAILS ✗
Fix: add CudaContext::disable_async_alloc() to vendored cudarc.
Called in DQN::new_with_stream() to force cuMemAlloc_v2 for ALL
allocations. No performance impact (allocations at construction,
not in hot loops).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added raw_ptr() method to vendor/cudarc CudaSlice — returns device
address without event tracking (no SyncOnDrop guard leak).
Replaced ALL 119 raw_device_ptr() wrapper calls with .raw_ptr().
Deleted raw_device_ptr, raw_device_ptr_i32, raw_device_ptr_u32 wrappers.
Added forward_online_raw / forward_target_raw for graph-safe forward.
All CachedPtrs used in graph-captured code paths.
Smoke test: model trains (Sharpe improves -24→+1), but Q-stats
readback still returns 1e30. The issue is NOT device_ptr event
tracking — deeper investigation needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
