perf(cudarc): enable async alloc — eliminates 934 stream syncs/200 steps
cudarc's CudaSlice::Drop checked has_async_alloc (always false) and fell back to stream.synchronize() + free_sync() on every drop. This blocked the host for ~12.6ms per sync × 4.7 drops/step = ~59ms/step of pure idle waiting (35.8% of wall time per nsys). With has_async_alloc=true, drops use cuMemFreeAsync (non-blocking, same-stream ordering). CUDA 12.4 on sm_86+ fully supports this. Combined with the background diag writer and fused label gather, the training loop should now be GPU-bound at ~13ms/step → ~77 sps. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2
vendor/cudarc/src/driver/safe/core.rs
vendored
2
vendor/cudarc/src/driver/safe/core.rs
vendored
@@ -85,7 +85,7 @@ impl CudaContext {
|
||||
cu_device,
|
||||
cu_ctx,
|
||||
ordinal,
|
||||
has_async_alloc: AtomicBool::new(false),
|
||||
has_async_alloc: AtomicBool::new(true),
|
||||
num_streams: AtomicUsize::new(0),
|
||||
event_tracking: AtomicBool::new(true),
|
||||
error_state: AtomicU32::new(0),
|
||||
|
||||
Reference in New Issue
Block a user