gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix: add cuStreamSynchronize before pinned atom_stats read

Browse Source 
populate_q_out + q_stats_reduce launch async kernels that write to
pinned device-mapped memory. The CPU was reading immediately — before
the GPU finished writing. Added cuStreamSynchronize to ensure the
kernel results are visible before the pinned memory read.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-19 16:51:46 +02:00
parent a0b52b7876
commit 65604db516
1 changed files with 6 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs
View File

@@ -8219,9 +8219,13 @@ impl GpuDqnTrainer {
12 * std::mem::size_of::<f32>(), self.stream.cu_stream(),
).map_err(|e| MLError::ModelError(format!("q_out sample0 DtoD: {e}")))?;
}
// No sync — CPU reads previous step's values from pinned host pointer.
// Sync: populate_q_out + q_stats_reduce just launched — must wait for
// pinned memory writes to complete before CPU reads.
unsafe {
let rc = cudarc::driver::sys::cuStreamSynchronize(self.stream.cu_stream());
assert_eq!(rc, cudarc::driver::sys::cudaError_enum::CUDA_SUCCESS, "q_stats stream sync");
}
// Q-mean EMA update — AFTER stream sync so pinned memory has the GPU-written value.
self.update_q_mean_ema();
// Read from pinned buffer — host[0..7] = q_stats, host[7..19] = q_out sample 0