jgrusewski
907b5f3355
fix: cross-stream race in PER update + pre-allocate scratch buffers
...
The replay buffer used its own CUDA stream for seg_tree_update, but
the td_errors buffer was computed on the trainer's stream. Without
synchronization, the replay stream could read incomplete data — causing
a hang on H100 where streams execute truly concurrently.
Fix: update_priorities_gpu now accepts an optional ext_stream parameter.
The fused training path passes the trainer's main stream, ensuring all
GPU work (td_errors computation + seg_tree_update) runs on the same
stream with implicit ordering.
Also pre-allocate update_td_f32, update_batch_max, update_max_merge
scratch buffers at construction — eliminates 3 cuMemAlloc calls per
training step that could trigger implicit device synchronization.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-04-06 12:36:15 +02:00
..
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-13 10:30:51 +01:00
2026-03-29 21:32:04 +02:00
2026-03-15 12:00:13 +01:00
2026-03-13 10:30:51 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00
2026-04-04 10:02:37 +02:00
2026-04-04 01:04:45 +02:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 08:55:55 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00
2026-04-04 10:02:37 +02:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-04-01 01:13:19 +02:00
2026-03-25 09:45:54 +01:00
2026-04-04 10:02:37 +02:00
2026-03-19 08:55:55 +01:00
2026-03-19 10:02:26 +01:00
2026-03-31 01:30:33 +02:00
2026-03-31 01:30:33 +02:00
2026-04-04 10:02:37 +02:00
2026-03-19 10:02:26 +01:00
2026-04-04 10:02:37 +02:00
2026-03-16 21:01:28 +01:00
2026-03-16 21:01:28 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-25 09:45:54 +01:00
2026-03-31 10:56:20 +02:00
2026-04-04 15:25:02 +02:00
2026-03-31 01:30:33 +02:00
2026-03-25 09:45:54 +01:00
2026-03-29 21:32:04 +02:00
2026-03-15 12:00:13 +01:00
2026-03-31 01:30:33 +02:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-04-04 09:48:38 +02:00
2026-03-29 21:32:04 +02:00
2026-03-29 21:32:04 +02:00
2026-04-06 12:36:15 +02:00
2026-03-25 09:45:54 +01:00
2026-03-19 10:02:26 +01:00
2026-03-31 10:56:20 +02:00
2026-03-31 01:30:33 +02:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 10:02:26 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 08:59:52 +01:00
2026-03-20 15:41:42 +01:00
2026-03-19 10:02:26 +01:00
2026-03-18 00:53:47 +01:00
2026-03-19 10:02:26 +01:00
2026-04-01 01:13:19 +02:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-31 01:30:33 +02:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 08:59:52 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 10:02:26 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 08:59:52 +01:00
2026-03-19 08:59:52 +01:00
2026-04-02 14:21:45 +02:00
2026-03-19 10:02:26 +01:00
2026-03-19 08:59:52 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 10:02:26 +01:00
2026-03-15 12:00:13 +01:00
2026-04-04 10:02:37 +02:00
2026-03-19 10:02:26 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 08:55:55 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00
2026-04-04 10:02:37 +02:00
2026-03-19 10:02:26 +01:00
2026-03-19 10:02:26 +01:00
2026-03-21 09:14:53 +01:00
2026-03-29 21:32:04 +02:00
2026-03-19 08:59:52 +01:00
2026-03-19 10:02:26 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 08:44:50 +01:00
2026-03-19 10:02:26 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 08:44:50 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 10:02:26 +01:00
2026-03-19 08:44:50 +01:00
2026-03-19 08:59:52 +01:00
2026-03-16 16:11:48 +01:00
2026-03-14 11:35:15 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-04-04 10:02:37 +02:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-15 12:00:13 +01:00
2026-03-19 10:02:26 +01:00