Best-checkpoint save (val Sharpe improvement, ~30% of epochs in
convergent runs) blocked the epoch loop for 20-40s on each improvement:
serialize_model() chained N (~26) per-tensor DtoH downloads via
GpuTensor::to_host → memcpy_dtoh, each forcing an implicit stream sync
on a busy training stream. The DtoH chain also violates
feedback_no_htod_htoh_only_mapped_pinned.md (only cuMemHostAlloc
DEVICEMAP allowed for CPU↔GPU paths).
Plan B:
- Introduce snapshot_model_to_pinned (mod.rs): allocate one
MappedF32Buffer sized for all named weight slices concatenated,
cuMemcpyDtoDAsync each slice into the buffer's device pointer
(which aliases the host page), single stream sync, copy bytes out
to a Send + 'static Vec<u8>. One sync per snapshot, replaces N.
- serialize_snapshot_bytes (mod.rs): pure-CPU safetensors construction
from CheckpointSnapshot. Static — callable without &self, so the
worker can move the snapshot across thread boundary.
- handle_epoch_checkpoints_and_early_stopping on val-Sharpe
improvement: save_best_gpu_params (DtoD, fast) + snapshot to pinned
+ tokio::task::spawn_blocking the safetensors construction +
checkpoint_callback invocation. JoinHandle parked on
pending_checkpoint_handles. Training loop continues immediately.
- await_pending_checkpoint_handles drains in-flight workers at
training end (success branch + early-stop branches) and before
any synchronous cold-path checkpoint write to keep disk ordering
deterministic.
- F bound on train / train_walk_forward / train_fold_from_slices
gains + 'static so the callback can be moved into the worker.
All public callers already use 'static-compatible move closures
(test fixtures with shared mutable state migrate to Arc<Mutex<T>>).
Internal pipeline uses CheckpointCallbackHandle =
Arc<std::sync::Mutex<Box<dyn FnMut + Send + 'static>>> so the
same callback flows through multi-fold walk-forward into every
fold's worker.
- serialize_model itself rewritten via the snapshot path: the
no-DtoH rule now holds across ALL checkpoint paths (best, periodic,
early-stop, plateau-exhausted). The pre-existing GpuTensor::to_host
path is no longer reachable from the DQN trainer.
The audit's spec called for an mpsc channel(1) drop-old worker, but
the multi-fold + &mut F pre-existing API made the simpler
fire-and-forget spawn_blocking pattern a cleaner fit (Mutex
serialises any concurrent invocations; Vec<JoinHandle> drain at end
guarantees disk writes complete before the trainer returns). Same
overlap benefit (training rolls while serialize+disk run on a
blocking thread); upper bound on in-flight work is one-per-improved-
epoch which approximates the spec's depth=1 in realistic training
runs.
Per feedback_no_partial_refactor: every site that constructs a
checkpoint payload migrated in lockstep — best-improvement uses
the worker; periodic / plateau-exhausted / early-stop call the
shared Arc<Mutex<F>> handle inline. All paths read params via
snapshot_model_to_pinned, so the no-DtoH rule applies uniformly.
Test fixtures (8 .rs files) updated for the + 'static bound (move
closures + cloned PathBufs / Arc<Mutex<T>> for shared mutable state).
Verified: SQLX_OFFLINE=true cargo check --workspace --tests clean
(warnings unchanged from baseline). cargo test -p ml --lib --no-run
clean. No fingerprint change.
Wire-up audit entry extended with Plan B file:line edit sites
(rides under the same Async-validation overlap section started by
the companion Plan A commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.
Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
