perf(cudarc): enable async alloc — eliminates 934 stream syncs/200 steps

cudarc's CudaSlice::Drop checked has_async_alloc (always false) and
fell back to stream.synchronize() + free_sync() on every drop. This
blocked the host for ~12.6ms per sync × 4.7 drops/step = ~59ms/step
of pure idle waiting (35.8% of wall time per nsys).

With has_async_alloc=true, drops use cuMemFreeAsync (non-blocking,
same-stream ordering). CUDA 12.4 on sm_86+ fully supports this.

Combined with the background diag writer and fused label gather,
the training loop should now be GPU-bound at ~13ms/step → ~77 sps.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-27 21:45:44 +02:00
parent dad5656bf6
commit 29d21b3241

View File

@@ -85,7 +85,7 @@ impl CudaContext {
cu_device,
cu_ctx,
ordinal,
has_async_alloc: AtomicBool::new(false),
has_async_alloc: AtomicBool::new(true),
num_streams: AtomicUsize::new(0),
event_tracking: AtomicBool::new(true),
error_state: AtomicU32::new(0),