4c3806da9c
Systematic audit of all `memcpy_dtoh` call sites following commit5da434ab4(per_branch_grad_norms pinned-dst async-DMA read race). Found two latent sites with the same pattern where the doc claims "synchronous DtoH" but the implementation relies on cudarc's `memcpy_dtoh` — which forwards to `cuMemcpyDtoHAsync_v2` and is asynchronous when the destination is pinned. The affected functions are currently unreachable (no callers in-tree, only wrapper proxies in fused_training.rs) but accept a caller-supplied `dst: &mut [f32]` that could legitimately be pinned host memory. Add the post-copy `stream.synchronize()` so the doc-claimed "synchronous" contract holds regardless of the caller's dst memory type — matches the5da434ab4pattern exactly. Audit scope (23 `stream.memcpy_dtoh` + 8 raw `cuMemcpyDtoH*` call sites across crates/ml, crates/ml-core, crates/ml-dqn, crates/ml-ppo, crates/ml-ensemble). Full site-by-site verdict table: RACE FIXED (latent, unreachable today): - gpu_dqn_trainer.rs:8447 download_params (pinnable dst) - gpu_dqn_trainer.rs:8460 download_target_params (pinnable dst) SAFE-PAGEABLE (dst is `vec![..]` / `[T; N]` stack array — cuMemcpyDtoHAsync_v2 on pageable dst degrades to synchronous per CUDA runtime rules): - gpu_dqn_trainer.rs:2507, 2510, 2573, 2654, 9896, 10219 - gpu_experience_collector.rs:1776-1923, 2002-2011, 2099, 2185, 3364, 3392, 3394, 3396 - gpu_backtest_evaluator.rs:872 - gpu_walk_forward.rs:697 - gpu_action_selector.rs:204 - signal_adapter.rs:270, 302, 343, 393 (test code) - gpu_ppo_collector.rs:162, 200 - noisy_layers.rs:354, 357 - gpu_replay_buffer.rs:1127 (test code, explicit pre-sync) - ensemble/adapters/dqn.rs:352 - ml-core/cuda_autograd/{reductions,optimizer,var_store,gpu_tensor, linear,elementwise,init}.rs (all reduction-scalar or test paths) - ml-ppo/cuda_nn/{tensor_util,linear}.rs - ml-ensemble/stream_ensemble.rs:381 - target_update.rs:242 (test code) - training_stability.rs:159, 203 (test code) - gpu_residency.rs:84, 92 (test code) - gradient_budget.rs:105, 114, 123, 190 (test code) SAFE-SYNCED (uses the crate's sync helper `super::dtoh_f32`, which pre-syncs the stream and uses raw `cuMemcpyDtoH_v2` — the synchronous API — so no async DMA is ever queued): - gpu_dqn_trainer.rs:8841, 9210 - gpu_experience_collector.rs:2156, 2174, 3156 - gpu_monitoring.rs:129 - gpu_statistics.rs:115 - gpu_ppo_collector.rs:152, 172, 181, 190 - decision_transformer.rs:986, 1135 - trainers/tlob.rs:519 - trainers/dqn/trainer/metrics.rs:352 SAFE-SYNCED (raw `cuMemcpyDtoH_v2` — synchronous API by CUDA spec): - gpu_dqn_trainer.rs:8940, 8954, 9267 - gpu_iqn_head.rs:1572 - gpu_experience_collector.rs:1562, 1622 - fused_training.rs:2861 SAFE-BY-DESIGN (intentional async double-buffered pattern — queues new DMA, reads previous call's result from pinned offset, relying on stream-ordered lag. Documented as such; not a race): - gpu_dqn_trainer.rs:9433 causal-sensitivity readback (pinned offset +10) - gpu_dqn_trainer.rs:9556 causal-sensitivity readback unconditional ALREADY FIXED (this is the reference commit5da434ab4): - gpu_dqn_trainer.rs:2222 per_branch_grad_norms (pinned grad_readback_pinned) Validation: 6× sequential smoke runs (magnitude_distribution ignored test, fold 0 HEALTH_DIAG[0] grad_abs[dir]) at this HEAD: run 1: dir=1.381060e-2 (env blip — runs 1-3 ran during system load spike) run 2: dir=1.440200e-2 run 3: dir=1.418204e-2 run 4: dir=1.374509e-2 run 5: dir=1.375137e-2 run 6: dir=1.376674e-2 Steady-state (runs 4-6): 0.16% spread — matches the 0.2% reference floor from commit5da434ab4. download_{params,target_params} are not exercised by the smoke test (unreachable), so no measurable delta expected or observed on the reachable metrics. The fix is forward-looking correctness insurance for the declared-pub API surface. Logs: /tmp/foxhunt_smoke/dma_audit_run{1..6}.log Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ml
10-model ML ensemble for the Foxhunt HFT system, built on Candle v0.9.1.
Models
- DQN (Rainbow) — deep Q-network with prioritized replay, dueling heads, noisy nets
- PPO — proximal policy optimization with GAE, LSTM policies, clip-higher
- TFT — temporal fusion transformer for multi-horizon forecasting
- Mamba2 — state space model for sequence prediction
- Liquid Networks — biologically inspired networks for non-stationary data
- TLOB — transformer-based limit order book analysis
- KAN — Kolmogorov-Arnold networks
- xLSTM — extended LSTM architecture
- TGGN — temporal graph neural network
- Diffusion — diffusion-based generative model
Key Modules
ensemble— model ensemble coordination and confidence aggregationhyperopt— PSO-based hyperparameter optimization with per-model adapterstrainers— unified training loops (DQN, PPO, supervised)inference—InferenceAdaptertrait for predictioncheckpoint— model checkpointing and restorationevaluation— walk-forward evaluation pipeline
Usage
use ml::dqn::DQN;
use ml::ppo::PpoTrainer;