Created config/gpu/{default,rtx3050,h100,a100}.toml with all GPU-specific
parameters: batch_size, num_atoms, buffer_size, hidden_dim_base,
replay_buffer_vram_fraction, gpu_n_episodes, gpu_timesteps_per_episode,
cuda_stack_bytes.
GpuProfile::load() auto-detects GPU by device name, falls back to
embedded defaults (include_str!). Override via FOXHUNT_GPU_PROFILE env.
Removed dead code:
- detect_vram_mb(), vram_scaled_hidden_dims(), vram_scaled_base_dim(),
resolve_hidden_dim_base() + 18 tests for these functions
All callers updated: train_baseline_rl, DQNTrainer constructor,
PPO trainer, smoke tests, pipeline tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace scattered VRAM-based if/else chains with a declarative TOML profile
system. GPU profiles (rtx3050, a100, h100, default) are selected by device
name and embedded at compile time via include_str! for zero-filesystem
fallback in CI/containers, with filesystem and env var overrides.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: graph capture disables event tracking, but cudarc's
record_err() stores errors from cuStreamWaitEvent on disabled events.
bind_to_thread() calls check_err() and returns the stored error,
blocking ALL subsequent cudarc API calls (launch, sync, memcpy).
Fix:
- check_err() before EMA kernel launch to consume stale errors
- Raw cuStreamSynchronize + cuMemcpyDtoH for post-graph readback
(bypasses bind_to_thread entirely)
- Raw device pointers for EMA kernel args (bypasses device_ptr event wait)
- GPU-native cat bounds check (dst_start + n <= output.len())
- CUDA Graph always enabled (removed should_use_cuda_graph function)
4/5 DQN pipeline tests pass with CUDA Graph on RTX 3050.
5th (epsilon_greedy) passes individually, flaky in sequence.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: GpuTensor::cat() called to_host() on each input tensor,
concatenated on CPU, then from_host() back to GPU. This caused:
1. Race conditions when async GPU work hadn't completed before to_host
2. CUDA_ERROR_ILLEGAL_ADDRESS when event tracking was disabled
3. Massive performance hit (2 DMA transfers per tensor per cat call)
Fix: replace with DtoD memcpy via CudaSlice::slice() views. Both dim=0
(contiguous blocks) and dim>0 (interleaved rows) use GPU-to-GPU copies.
Zero CPU involvement, zero event dependency, zero race conditions.
Also: revert global disable_event_tracking() — was a workaround for the
to_host race, no longer needed with GPU-native cat/stack.
5/5 DQN pipeline tests pass with CUDA Graph enabled on RTX 3050.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three fixes:
1. FusedTrainingCtx::new now accepts RegimeConditionalDQN by using
primary_head() — the old code rejected it, causing silent fallback
to non-fused train_step() which has action-space mismatch with
branching DQN (5-action Q-table vs 45-action factored indices →
CUDA_ERROR_ILLEGAL_ADDRESS buffer overrun)
2. ensure_fused_ctx() returns Result instead of () — fused init
failure is now a hard error (no silent CPU fallback)
3. GpuTensor::cat now supports dim>0 concatenation (was unimplemented,
caused "dim=1 > 0 not yet implemented" error in validation path)
Root cause chain:
RegimeConditional rejected by fused init
→ silent fallback to DQN::train_step()
→ compute_loss_internal() uses num_actions=5 but actions are 0-44
→ gather with out-of-bounds offsets → CUDA_ERROR_ILLEGAL_ADDRESS
→ async error poisons CUDA context
→ next stream.synchronize() deadlocks forever
Remaining: curiosity kernel crash also poisons the stream. The
curiosity training runs inside collect_gpu_experiences() and its
async error blocks the PER insert. This needs separate investigation
of curiosity_training_kernel.cu.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cuda_if_available() silently fell back to CPU when CUDA init failed,
causing DQNTrainer to crash later with "cuda_stream() called on CPU
device". This was the root cause of the CI "hang" — the test failed
immediately but the error was invisible due to buffered output.
- DQNTrainer::new() now uses MlDevice::cuda(0)? with explicit error
- cuda_if_available() now logs the CUDA error before falling back
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- ScalarValue trait: as_f32() → to_f32_scalar() (clippy wrong_self_convention)
- CI template: test-gate uses ci-builder (CUDA) instead of ci-builder-cpu
- VPC: enabled DefaultRoutePropagation for gateway DHCP
- PVCs: all set to Retain reclaim policy
- Argo CLI: configured server mode for archived log retrieval
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was ignored because debug mode (no SIMD) exceeded 10µs threshold.
Changed to 500µs which passes in debug (~11µs) and release (<1µs).
A 1024-element dot product exceeding 500µs indicates a real problem.
ml-core: 302 pass, 0 fail, 0 ignored.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
elementwise.rs: added broadcast_col_binary CUDA kernel for [N,M]*[N,1]
gpu_tensor.rs: broadcast_mul/broadcast_div now handle column broadcast
stream_ops.rs: gpu_cat_dim1 extended for 3D tensors
branching.rs: NoisyLinear weights registered in GpuVarStore
noisy_layers.rs: register_in_store() method for weight registration
distributional_dueling.rs: NoisyLinear weight registration
dqn.rs: checkpoint load wires weights into VarStore
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
test_max_shared_memory_kb_h100: was calling driver query path which
returns actual GPU's shared memory (100KB on RTX 3050, not 228KB).
Changed to test name-based heuristic directly via max_shared_memory_kb_by_name().
Added test_max_shared_memory_kb_queries_real_device with range assertion
(48-256 KB) for device-agnostic driver query test.
ml-core: 301 pass, 0 fail.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug 1: Tree reduction stopped at s>32, leaving shmem[32..63] unfolded
before warp shuffle. Added explicit fold: thread i merges shmem[i+32]
before __shfl_down_sync. Affected all 5 kernels (stats, argmax, sum,
argmax_rows, col_sum). Caused max=96 instead of 100 for N=100.
Bug 2: fused_stats_reduce count only atomicAdd'd thread 0's local_count.
Added 5th shared memory slot for count through full tree reduction.
All 5 reduction tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
elementwise.rs: added abs, neg, exp, log, le, affine CUDA kernel ops
(cases 5-10 in elementwise_unary). All GPU-native, zero host downloads.
gpu_tensor.rs: 7 new methods — abs(), neg(), exp(), log(), le(),
affine(), broadcast_sub(). All dispatch to CUDA kernels.
dqn.rs: DELETED 30+ host-side helper functions (~455 lines) that
downloaded to CPU, computed, re-uploaded. ALL replaced with direct
GpuTensor methods: affine(), clamp(), abs(), neg(), exp(), log(),
broadcast_mul(), broadcast_sub(), broadcast_as(), index_select(),
dim(), sqr(), flatten_all(), gpu_clone(), ActivationKernels.
Zero host-side math remaining in dqn.rs hot paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New ml-core/cuda_autograd/reductions.rs:
- fused_stats_reduce: min/max/sum/sum_sq/count in single pass with
warp-level __shfl_down_sync + atomicCAS (20 bytes DtoH)
- argmax_flat: single u32 result (4 bytes DtoH)
- argmax_rows: per-row argmax for 2D data
- sum_reduce: standard tree reduction
- col_sum_reduce: per-column sum along rows
DQN trainer integration:
- collect_qvalue_statistics: 4 DtoH → 1 (single stats() call)
- compute_q_diagnostics_fused: replaces Candle sort/narrow pipeline
with argmax_rows + stats + col_sums (3 kernels, 3 readbacks)
- ReductionKernels lazy-initialized in training loop
All kernels compiled via compile_ptx_for_device (native cubin + disk cache).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Test code used CudaDevice (cudarc 0.17) instead of CudaContext (0.19),
and wrapped Arc::new(CudaContext::new().new_stream()) which double-wraps
since new_stream() already returns Arc<CudaStream>.
740 tests pass across ml-core, ml-ppo, ml-supervised, ml-ensemble.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Moved ml-supervised's duplicate GpuTensor (stream-carrying) + GpuLinear +
50 free functions to ml-core/cuda_autograd/stream_ops.rs as StreamTensor
and StreamLinear. ml-supervised/gpu_tensor.rs → 53 lines of re-exports.
Two tensor flavors now canonical in ml-core:
- GpuTensor: takes &Arc<CudaStream> per-call (autograd integration)
- StreamTensor: carries Arc<CudaStream> internally (self-contained ops)
Zero consumer import changes. Both crates compile clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Final cleanup:
- 61 test files + 5 example files: candle imports replaced
- 8 testing/integration files: migrated to cudarc/ml-core types
- 3 services/trading_service test files: migrated
- Root Cargo.toml: candle-core, candle-nn removed from [workspace.dependencies]
- crates/ml/Cargo.toml: candle-nn dependency removed
- testing/e2e/Cargo.toml: candle-core dependency removed
Zero active candle_core/candle_nn/candle_optimisers code references remain.
Zero candle dependency declarations in any Cargo.toml.
Remaining "candle" strings are exclusively in doc comments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace deprecated cudarc memcpy_stod with clone_htod across all GPU
upload paths (14 call sites in trainers, cuda_pipeline, hyperopt).
Replace deprecated memcpy_dtov with clone_dtoh in ml-supervised
gpu_tensor.rs.
Bridge GpuTensor-migrated submodules (xLSTM, Liquid CfC, TFT GRN)
with Candle Tensor callers via from_candle_tensor/to_candle_tensor
conversion utilities at API boundaries. Fix CudaDevice->CudaContext
in ml-dqn distributional_dueling.rs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace VarMap/VarBuilder weight storage with GpuVarStore (native CUDA)
for RMSNorm, LayerNorm, and ResidualBlock. Linear layers in ResidualBlock
now use GpuLinear with cuBLAS sgemm. Cold-path forwards convert between
GpuTensor and Candle Tensor at the boundary since downstream ops (GELU,
LayerNorm, dropout, broadcast) still use Candle.
Changes:
- ml-core cuda_autograd: add GpuTensor::to_candle/from_candle interop,
export GpuParam/AdamWConfig/ActivationKernels/LossKernels/LossResult,
make init::upload_to_gpu public
- rmsnorm.rs: RMSNorm/LayerNorm now take (Arc<CudaStream>, Device, dim)
instead of (VarBuilder, dim). Weights stored in GpuVarStore.
- residual.rs: ResidualBlock now takes (Arc<CudaStream>, Device, config, name).
fc1/fc2 are GpuLinear with cuBLAS sgemm forward. LayerNorm params in GpuVarStore.
- distributional_dueling.rs: updated RMSNorm constructor calls to new API
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace PolicyNetwork and ValueNetwork with CudaPolicyNetwork/CudaValueNetwork
backed implementations. Each struct now stores a cuda_nn GPU-native network
(cuBLAS sgemm + CUDA kernels) alongside shadow Candle layers for autograd
training compatibility. Adds forward_cuda() inference paths that bypass Candle
entirely. Wire CudaTrajectoryTensors into TrajectoryBatch with to_cuda_tensors().
Re-export CudaLSTM from lstm_networks and all cuda_nn types from crate root.
Import GpuContext into continuous_policy, continuous_action_masking, flow_policy,
coupling_layer, and adaptive_entropy for future GPU migration. Add CudaVec::to_tensor()
bridge for cuda_nn→Candle interop. Fix upload_host_to_gpu→upload_to_gpu rename
in ml-core cuda_autograd init.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace Candle's backward()/GradStore/VarMap/VarBuilder system with CUDA-native
primitives in ml-core::cuda_autograd. This eliminates the Candle dispatch overhead
(~2100 kernel launches per batch) for models that adopt the new API.
Components:
- GpuTensor: CudaSlice<f32> wrapper with shape metadata (replaces candle Tensor)
- GpuVarStore: named parameter store with flatten/unflatten (replaces VarMap/VarBuilder)
- GpuLinear: cuBLAS sgemm forward + manual backward (replaces candle_nn::Linear)
- GpuAdamW: per-parameter CUDA kernel optimizer (replaces candle_optimisers::Adam)
- ActivationKernels: ReLU/LeakyReLU/GELU/Sigmoid/Tanh forward+backward CUDA kernels
- LossKernels: MSE/Huber loss with fused gradient computation
- init: Xavier/Kaiming/near-zero initialization via CPU generate + GPU upload
Added cublas feature to ml-core's cudarc dependency for sgemm support.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CUDA infrastructure in ml-core (cuda_compile, gpu/capabilities, gpu/l2_cache)
previously accessed cudarc types through candle_core::cuda_backend::cudarc --
a transitive re-export that coupled low-level CUDA driver calls to Candle.
Changes:
- Add cudarc 0.17 as direct optional dep (gated behind cuda feature)
- Replace all candle_core::cuda_backend::cudarc paths with direct cudarc::
- Generalize OOM detection to accept &dyn Debug (was &CandleError)
- Add generic computation_error_to_common_error() alongside legacy alias
Candle references: 163 -> 72 (remaining are autograd-dependent: Tensor,
Var, GradStore, VarBuilder -- cannot be replaced with cudarc raw ops)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove ALL #[cfg(feature = "cuda")] guards (~400+ occurrences)
- Remove ALL #[cfg_attr(not(feature = "cuda"), ignore)] test annotations (~250)
- Make cuda default feature in 9 ML crates (ml, ml-core, ml-dqn, ml-ppo, etc.)
- Convert nvrtc JIT compilation to precompiled nvcc (searchsorted, prefix_sum)
- Move compile_ptx_for_device() to ml-core for shared access
- Delete dead CPU code: multi_step.rs, self_supervised_pretraining.rs,
training_guard_gpu_tests.rs, CPU PER buffer paths, CPU Q-diagnostics
- Replace unwrap_or(Device::Cpu) with hard errors everywhere
- Remove dead is_cuda() else branches in DQN/PPO/hyperopt trainers
- Change config defaults from "cpu" to "cuda" (rainbow, tlob, pipeline)
- Port IQL value network to GPU kernel (5 CUDA entry points)
- Port HER goal relabeling to GPU kernel (warp-per-sample)
- Wire DSR GPU-to-CPU sync in training loop
- cfg!(feature = "cuda") → true in inference_validator
Zero warnings, zero errors across entire workspace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete every CPU fallback block across 14 files (-286 lines):
- dqn.rs: CPU replay buffer, tensor construction, PER fallback, gradient paths
- hyperopt/adapters/ppo.rs: CPU curiosity modules, trajectory generation
- hyperopt/adapters/dqn.rs: CPU backtest fallback, sync no-op
- trainers/ppo.rs: CPU training error stub
- l2_cache.rs: CPU stub functions (gate callers behind cuda too)
- replay_buffer_type.rs: CPU is_gpu_prioritized fallback
- validation/harness.rs: CPU regime breakdown fallback
- build.rs: cpu_only_build cfg (never referenced)
- evaluate_baseline.rs: CPU gpu_handled fallback
- testing/integration/gpu: CPU test stubs
Zero #[cfg(not(feature = "cuda"))] remains in the codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cast all Tensor::full() / Tensor::from_vec() call sites to training_dtype
instead of defaulting to F32. Fixes dtype mismatch errors (BF16 vs F32)
in PPO training on CUDA:
- tensor_ops: scalar_mul, clamp, normalize match operand dtype
- trajectories: TrajectoryBatch/MiniBatch to_tensors cast to training dtype
- continuous_ppo: ContinuousTrajectoryBatch/MiniBatch to_tensors cast
- adaptive_entropy: cast entropy to F32 for alpha multiplication boundary
- continuous_policy: forward() input cast, Tensor::full scalars match dtype
- flow_policy: sample_base_noise cast to training dtype
- hidden_state_manager: reset tensors use training_dtype
- ensemble/ppo adapter: predict input cast to training dtype
- trainable_adapter: test uses training_dtype instead of hardcoded F32
Verified: 198/198 ml-ppo tests pass, 63/63 ml PPO tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- backtest_forward_kernel: dim_overrides must precede common_device_functions.cuh
which has #error guards requiring STATE_DIM/MARKET_DIM/PORTFOLIO_DIM to be
defined before inclusion. Experience collector already had correct ordering.
- Cap MAX_EPISODES from 8192→4096 (diminishing returns above 4096, wastes walltime)
- Cap trainer .min() from 0x8000 (32768) → 4096 to match
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Cast input to weight dtype in DQN residual, rmsnorm, noisy_layers
- Set use_gpu=true in QNetworkConfig defaults and all config sites
- Resolve BF16 boundary mismatches in attention, curiosity, branching,
distributional_dueling across ml-dqn
- GPU-resident regime ops with BF16 boundary casts, eliminate .expect() in CUDA paths
- Eliminate all Device::Cpu fallbacks — GPU-only across 10 ML crates
- PPO: cast logits to F32 before softmax, cast batch tensors to training dtype
- Gradient collapse detection for RegimeConditionalDQN
- Wire halt_grad_collapse from CUDA guard kernel to halt training
- Dead neuron detection uses active network VarMap + squeeze factored readback
- Increment gradient_logging_step in GPU PER path
- Gradient collapse warmup guards use original buffer_size
- Cap training steps per epoch + tracing migration
- Replace Tensor::all() with sum_all() for pinned Candle compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two GPU bottlenecks fixed:
1. GradStore key mismatch (40,000+ warnings/run): clip_grad_norm now uses
TensorId-based iteration exclusively. The old Var-based lookup always
failed (0/16 matches) due to identity drift from BF16 dtype conversion,
then fell back to TensorId anyway. Removed the pointless Var path and
fallback warning entirely.
2. CUDA_ERROR_INVALID_PTX on H100 (sm_90): The standard per-thread kernel
(~7.5 KB stack × 256 threads) caused invalid PTX when co-compiled with
the warp kernel for compute_90. Guarded with #if __CUDA_ARCH__ < 900
so only the warp-cooperative kernel (200 bytes/lane) is compiled on
Hopper. Rust-side kernel loading restructured to query SM before
compilation and load the appropriate kernel variant directly.
Test results: ml-core=311, ml-dqn=416, ml=915 — 0 failures, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The circuit breaker (>20% drawdown) carried forward across epochs,
permanently locking out all trades once triggered. With compounding
portfolios, early drawdown in one epoch could produce zero rewards
for all subsequent epochs (constant rewards bug).
Fix: reset peak_value (high-water mark) at each epoch start. Capital
still compounds (Bug #15 preserved), but drawdown is measured fresh
per epoch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- gradient_utils: Add TensorId-based fallback when Var identity mismatch
causes 0/N vars to match GradStore (Candle Adam clones Arcs). Fallback
computes norm AND clips via insert_id. Throttled warning (1st + every
1000th). 7 unit tests including mismatch-actually-clips.
- monitoring: Track full 45-action factored space (5 exposure × 3 order
× 3 urgency). Fix validate_rewards false alarm on GPU path where single
aggregated mean_reward per epoch gives N=1 → std=0.
- trainer: GPU experience collection routes exposure actions through
route_action() for factored tracking instead of exposure-only counts.
Applied in both per-step and epoch-summary paths.
- train_baseline_rl: Auto-detect VRAM <8GB → disable GPU replay buffer
to prevent OOM on RTX 3050 Ti class GPUs.
- smoke_test_real_data: E2E DQN training test with 6 assertions (epoch
completion, loss decrease, finite losses, Q-value divergence, 45-action
space, finite gradient norms).
Validated: 1642 tests pass (ml=915, ml-core=311, ml-dqn=416), 0 clippy
warnings, baseline RL trains 10 epochs on CUDA with Sharpe +5.45.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU PER replay buffer had a hardcoded 4 GB MAX_BYTES limit that
rejected the auto-sizer's 10M-entry proposal on H100 (80 GB VRAM).
Now per_max_buffer_bytes() computes 20% of total VRAM (min 1 GB) and
flows through OptimalReplayConfig → DQNConfig → GpuReplayBufferConfig
so both subsystems agree on the budget.
Also fixes misleading regime detection log (indices 211/219 → 40/41)
and renames dqn_config_2025 → dqn_default_config.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevents experience collector output tensors from starving backward
graph and GPU PER on VRAM-constrained GPUs like RTX 3050 Ti 4GB.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
free_memory_mb is sampled after model/optimizer load, so subtract only
backward-graph + fragmentation + data upload headroom (400 MB), not the
full model/PER/data stack. Increase collector share to 50%.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Staging/GPU replay buffer: truncate batch when it exceeds ring buffer
capacity (CPU fallback can flush 100K+ experiences at once)
- GpuExperienceCollector: compute shmem tile rows dynamically to stay
under 48 KB default limit instead of hardcoded 64-row constant
- Auto batch size: subtract concurrent VRAM consumers (~530 MB model +
optimizer + PER + data) before budgeting experience collector at 40%
- Hyperopt DQN adapter: GPU PER always on, include replay buffer in
VRAM estimate for small GPUs (was excluded assuming CPU-only PER)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
