Validation backtest was serialised on the training stream by passing
self.cuda_stream as parent_stream into GpuBacktestEvaluator::new (the
forked eval-stream still depends on training-stream completion, but the
metrics + plan_diag readbacks went through clone_dtoh / memcpy_dtoh,
each forcing an implicit sync that blocked behind the training stream's
pending work). The prior comment claimed single-stream eval was needed
"for cuBLAS determinism" — incorrect: cuBLAS bit-wise reproducibility
caveats apply per-handle, not across the process, and each stream
already owns its own PerStreamCublasHandles. Costs ~30-40s/epoch on L40S.
Plan A:
- Route eval through self.validation_stream (already forked at
constructor.rs:547 but unused on the eval path) and add a fresh
cuda_stream-recorded train_done_event so validation_stream +
evaluator's internal forked stream both wait on it before launching
eval kernels. GPU-side cuStreamWaitEvent — no CPU sync.
- Replace metrics_buf and plan_diag_buf in GpuBacktestEvaluator from
CudaSlice<f32> to MappedF32Buffer (cuMemHostAlloc DEVICEMAP). Kernels
write through the buffer's dev_ptr; host reads via volatile host_ptr
after a single eval_stream.synchronize() per evaluation. Replaces
the forbidden clone_dtoh / memcpy_dtoh path
per feedback_no_htod_htoh_only_mapped_pinned.md. The single sync
blocks only the eval stream — training rolls on uninterrupted.
- Update the misleading "single-stream cuBLAS determinism" comment
to explain the actual per-stream-handle architecture.
Per feedback_no_partial_refactor: every consumer of the
metrics_buf / plan_diag_buf contract migrated in lockstep (struct
fields, alloc, kernel-arg passing, host readout). Per-stream cuBLAS
handle parity with training preserved (val TLOB still uses
PerStreamCublasHandles::new(&eval_stream); evaluator forks its own
internal cuBLAS state from parent_stream).
Verified: cargo check -p ml --lib clean (13 warnings, workspace
baseline). cargo test -p ml --lib --no-run clean. No fingerprint
change — mapped-pinned is allocation-method orthogonal to kernel
buffer layout.
Companion commit lands Plan B (async best-checkpoint serialize).
Wire-up audit entry covers Plan A here; will be extended with Plan B
edit sites in the companion commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
d9cb14f1bc