cudarc's EventTrackingGuard + check_err alone is NOT sufficient.
The cuStreamSynchronize is required every step because cudarc's
internal event state machine becomes corrupted without it —
training hangs indefinitely after graph capture.
This is a cudarc limitation: CUDA Graph replay generates stale
events that accumulate and eventually block kernel launches.
The sync drains the pipeline (~150µs) but prevents the hang.
Removing these syncs requires either:
1. Patching cudarc to not record events on graph-replayed buffers
2. Using raw CUDA driver API without cudarc's event tracking
3. The mega-graph refactor (all ops in one graph, no cudarc ops between)
The other perf wins (vaccine throttle, actions DtoD, HER GPU-native,
causal interval) remain active. Expected epoch: ~500s (vs 614s baseline).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
50c5e786a6