8 test files had stale types from the bf16→f32 conversion:
- gpu_smoketest: missing adam_epsilon in DQNConfig
- gpu_backtest_validation: closure params bf16→f32
- gpu_kernel_parity_test: market data, weight readback bf16→f32
- gpu_per_integration_test: weights readback bf16→f32
- target_update_tests: varstore register bf16→f32
- smoke_test_real_data: market buffers bf16→f32
- activation_tests, dropout_scheduler_tests: forward() signature change
These tests only compile with --features cuda (CI path), which is why
they passed locally with cargo test --lib.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Final cleanup:
- 61 test files + 5 example files: candle imports replaced
- 8 testing/integration files: migrated to cudarc/ml-core types
- 3 services/trading_service test files: migrated
- Root Cargo.toml: candle-core, candle-nn removed from [workspace.dependencies]
- crates/ml/Cargo.toml: candle-nn dependency removed
- testing/e2e/Cargo.toml: candle-core dependency removed
Zero active candle_core/candle_nn/candle_optimisers code references remain.
Zero candle dependency declarations in any Cargo.toml.
Remaining "candle" strings are exclusively in doc comments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: shmem_max_in_dim only included trunk dims (state_dim,
shared_h1, shared_h2) but not head dims (value_h, adv_h). When
hidden_dim_base=32 made the trunk narrow while heads stayed at 128,
the BF16 weight tile for branch output (255×128=32640 BF16 elements)
overflowed the shared memory region (12288 BF16 elements). On H100
the overflow landed in unused-but-mapped hardware shmem (silent
corruption). On RTX 3050 (48KB physical shmem) it hit unmapped
memory → CUDA_ERROR_ILLEGAL_ADDRESS.
Changes:
- gpu_dqn_trainer.rs: shmem_max_in_dim includes value_h/adv_h
- Remove all #[ignore] from smoke tests (feature_coverage,
training_stability, gpu_residency)
- Smoke tests use real .dbn data from test_data/ (hard error if missing)
- Remove synthetic_data() fallback — no fake data in tests
- GPU-direct DtoD training path (train_step_gpu, FusedTrainScalars)
- GPU-native PER priority update kernel (zero CPU readback)
- IQN dual-head integration (gpu_iqn_head.rs)
- BF16 dtype fixes across 6 model adapters
- Hyperopt 30D→31D (iqn_lambda)
- portfolio_transformer: unconditional BF16 (remove dead CPU branches)
- liquid/adapter: all tests use Cuda(0) directly
- Fix pre-existing gpu_kernel_parity_test.rs (stale args)
- Fix pre-existing evaluate_baseline.rs (removed fields)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove ALL #[cfg(feature = "cuda")] guards (~400+ occurrences)
- Remove ALL #[cfg_attr(not(feature = "cuda"), ignore)] test annotations (~250)
- Make cuda default feature in 9 ML crates (ml, ml-core, ml-dqn, ml-ppo, etc.)
- Convert nvrtc JIT compilation to precompiled nvcc (searchsorted, prefix_sum)
- Move compile_ptx_for_device() to ml-core for shared access
- Delete dead CPU code: multi_step.rs, self_supervised_pretraining.rs,
training_guard_gpu_tests.rs, CPU PER buffer paths, CPU Q-diagnostics
- Replace unwrap_or(Device::Cpu) with hard errors everywhere
- Remove dead is_cuda() else branches in DQN/PPO/hyperopt trainers
- Change config defaults from "cpu" to "cuda" (rainbow, tlob, pipeline)
- Port IQL value network to GPU kernel (5 CUDA entry points)
- Port HER goal relabeling to GPU kernel (warp-per-sample)
- Wire DSR GPU-to-CPU sync in training loop
- cfg!(feature = "cuda") → true in inference_validator
Zero warnings, zero errors across entire workspace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three root causes fixed:
1. GPU PER + experience collector (cudarc) created dual CUDA allocator
fragmentation on GPUs ≤8 GB, making training impossible even at
batch_size=1. Added `use_gpu_replay_buffer` config flag; both GPU PER
and experience collector now disabled when VRAM ≤8192 MB.
2. Search space bounds were WIDENED instead of capped — max_batch_size
returning 4096 replaced the original 512 upper bound, and
max_hidden_dim_base_full returning 3072 replaced the original 1024.
Fixed with min() to only narrow, never widen.
3. VRAM estimator assumed GPU features always active, overcharging when
they're disabled on small GPUs. Now conditional: when
replay_buffer_capacity=0 (proxy for GPU PER disabled), collector/cudarc
costs are zero and fragmentation multiplier drops from 3× to 1.5×.
Additional small-GPU guard: GPUs ≤8 GB get clamped search space
(batch≤128, hidden≤512, atoms≤51, buffer≤50K) to fit 3 regime heads
+ C51 + noisy nets + dueling in limited VRAM.
Validated: 7/7 trials complete on RTX 3050 Ti 4 GB, zero OOM, best trial
Sharpe 9.8 with 50.6% win rate. Previous runs had 100% OOM failure rate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU experience kernel had hardcoded STATE_DIM=54, MARKET_DIM=51,
NUM_ATOMS_MAX=51 but the host-side feature buffer uploads 40 features
per bar and networks use state_dim=56 with up to 100 atoms. Past ~702K
bars the stride mismatch caused out-of-bounds GPU reads → ILLEGAL_ADDRESS.
All dimension constants now use #ifndef guards so the NVRTC JIT compiler
receives the actual values via #define injection — zero runtime overhead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hyperopt varies hidden_dim_base (256–1024) but the CUDA experience
kernel hardcoded SHARED_H1=256, SHARED_H2=256. When hyperopt picked
larger dims, the kernel did matvec with wrong strides → illegal memory
access → training crash on first epoch.
Fix: wrap CUDA #defines in #ifndef guards and inject actual dims from
the dueling network config at NVRTC compile time. The kernel is now
specialized per-trial with the exact weight matrix dimensions — zero
runtime overhead, no dimension mismatch possible.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests require a CUDA GPU and are #[ignore]d by default. Run with:
cargo test -p ml --test gpu_kernel_parity_test -- --ignored
Covers:
- Standard dueling forward: finite Q-values, valid action range [0,4]
- C51 distributional forward: Q-values bounded by atom support [-25,25]
- Candle vs kernel Q-value parity: argmax action consistency
- NoisyNet exploration: noise injection produces action diversity
- Weight extraction roundtrip: VarMap → CudaSlice → sync
- Distributional weight shapes: value_out [51,128], advantage_out [255,128]
- RMSNorm gamma extraction: initialized to 1.0
- Repeated kernel launches: 5 consecutive runs all produce finite output
All 8 tests pass on RTX 3050 Ti (4.49s total).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
