gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix: remove cuStreamSynchronize — one-step lag is fine for atom_stats

Browse Source 
The async pinned readback design intentionally reads previous epoch's
values. Epoch 1 shows zeros (lag), epoch 2+ shows real atom stats.
cuStreamSynchronize kills GPU pipeline — removed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-19 16:53:48 +02:00
parent 65604db516
commit ff7bbc7dd9
1 changed files with 3 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs
View File

@@ -8219,12 +8219,9 @@ impl GpuDqnTrainer {
12 * std::mem::size_of::<f32>(), self.stream.cu_stream(),
).map_err(|e| MLError::ModelError(format!("q_out sample0 DtoD: {e}")))?;
}
// Sync: populate_q_out + q_stats_reduce just launched — must wait for
// pinned memory writes to complete before CPU reads.
unsafe {
let rc = cudarc::driver::sys::cuStreamSynchronize(self.stream.cu_stream());
assert_eq!(rc, cudarc::driver::sys::cudaError_enum::CUDA_SUCCESS, "q_stats stream sync");
}
// No sync — one-step lag is fine for monitoring. populate_q_out + q_stats_reduce
// launched async; pinned memory becomes visible by the NEXT epoch's call.
// Epoch 1 reads zeros (expected), epoch 2+ reads real atom stats.
self.update_q_mean_ema();