test_max_shared_memory_kb_h100: was calling driver query path which
returns actual GPU's shared memory (100KB on RTX 3050, not 228KB).
Changed to test name-based heuristic directly via max_shared_memory_kb_by_name().
Added test_max_shared_memory_kb_queries_real_device with range assertion
(48-256 KB) for device-agnostic driver query test.
ml-core: 301 pass, 0 fail.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug 1: Tree reduction stopped at s>32, leaving shmem[32..63] unfolded
before warp shuffle. Added explicit fold: thread i merges shmem[i+32]
before __shfl_down_sync. Affected all 5 kernels (stats, argmax, sum,
argmax_rows, col_sum). Caused max=96 instead of 100 for N=100.
Bug 2: fused_stats_reduce count only atomicAdd'd thread 0's local_count.
Added 5th shared memory slot for count through full tree reduction.
All 5 reduction tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
elementwise.rs: added abs, neg, exp, log, le, affine CUDA kernel ops
(cases 5-10 in elementwise_unary). All GPU-native, zero host downloads.
gpu_tensor.rs: 7 new methods — abs(), neg(), exp(), log(), le(),
affine(), broadcast_sub(). All dispatch to CUDA kernels.
dqn.rs: DELETED 30+ host-side helper functions (~455 lines) that
downloaded to CPU, computed, re-uploaded. ALL replaced with direct
GpuTensor methods: affine(), clamp(), abs(), neg(), exp(), log(),
broadcast_mul(), broadcast_sub(), broadcast_as(), index_select(),
dim(), sqr(), flatten_all(), gpu_clone(), ActivationKernels.
Zero host-side math remaining in dqn.rs hot paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PpoExperienceBatch: all 6 fields Vec<T>→CudaSlice<T>. Data stays on GPU
from kernel collection through training. Zero DtoH for batch data.
- collect_experiences(): returns CudaSlice via D2D copy, no memcpy_dtoh
- gpu_batch_to_trajectory_batch(): DELETED (was CPU conversion)
- PPO::update_gpu(): reads states via D2D mini-batch, forward on GPU
- compute_metrics_from_gpu(): downloads only scalars (returns/advantages/
actions), NOT the 123 MB state tensor
- download_*() methods for debug/checkpoint only
States (123 MB/epoch) never leave GPU. Only 2 scalar losses come to CPU.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deleted ExperienceBatch (CPU Vec<f32>) struct and collect_experiences()
method (120 lines of memcpy_dtoh + CPU next_state computation).
Production DQN training already uses collect_experiences_gpu() which
returns GpuExperienceBatch (CudaSlice, zero CPU download).
Tests rewritten as pure index-math validation without GPU types.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New ml-core/cuda_autograd/reductions.rs:
- fused_stats_reduce: min/max/sum/sum_sq/count in single pass with
warp-level __shfl_down_sync + atomicCAS (20 bytes DtoH)
- argmax_flat: single u32 result (4 bytes DtoH)
- argmax_rows: per-row argmax for 2D data
- sum_reduce: standard tree reduction
- col_sum_reduce: per-column sum along rows
DQN trainer integration:
- collect_qvalue_statistics: 4 DtoH → 1 (single stats() call)
- compute_q_diagnostics_fused: replaces Candle sort/narrow pipeline
with argmax_rows + stats + col_sums (3 kernels, 3 readbacks)
- ReductionKernels lazy-initialized in training loop
All kernels compiled via compile_ptx_for_device (native cubin + disk cache).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
branching.rs: MlDevice→Arc<CudaStream>, Candle gather/unsqueeze→host-side,
MaybeNoisyLinear::noisy_sigma_slices(), copy_weights via memcpy pattern.
regime_conditional.rs: device→stream field, all GpuTensor ops get &stream,
batch_softmax_actions via host-side Gumbel-max, gradient merge via BTreeMap.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Test code used CudaDevice (cudarc 0.17) instead of CudaContext (0.19),
and wrapped Arc::new(CudaContext::new().new_stream()) which double-wraps
since new_stream() already returns Arc<CudaStream>.
740 tests pass across ml-core, ml-ppo, ml-supervised, ml-ensemble.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaced 4 cudarc::nvrtc::compile_ptx() calls in ml-ppo cuda_nn with
compile_ptx_for_device() — native cubin via nvcc -O3 with disk cache.
Before: virtual PTX → driver JIT (no -O3, no arch targeting, ~100ms first launch)
After: native SASS for exact sm_XX, cached to disk, <10ms load
Files: lstm.rs (2 kernels), linear.rs (1), softmax.rs (1)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Experience collector: PinnedHostBuf<T> via cuMemHostAlloc(PORTABLE) for
6 DtoH buffers — ~25-30% faster transfers vs pageable memory.
NVRTC caching: 4 OnceLock additions in ml-ppo cuda_nn:
- softmax.rs: was re-compiling NVRTC on EVERY forward pass batch (!)
- linear.rs, lstm.rs: cached for multi-layer construction
Device attribute: OnceLock<u32> for max_threads in gpu_replay_buffer
pfx_sum — eliminates ~5µs driver query per call.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
reset_episodes() was allocating 5 CPU Vecs per epoch for HtoD zero-fill.
Replaced with:
- memset_zeros() for barrier/diversity buffers (async GPU-side, no HtoD)
- Pre-allocated host staging vectors for portfolio_init and RNG seeds
gpu_curiosity_trainer already optimized (fused kernel zeros internally).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Activations: ml-ppo/cuda_nn/activations.rs 208→60 lines (-71%).
Deleted 4 inline CUDA kernels, replaced with thin wrappers delegating
to ml-core::cuda_autograd::ActivationKernels via new _raw methods.
AdamW: Unified kernel source between ml-ppo and ml-core. Switched to
architecture-aware SASS compilation (compile_ptx_for_device). Fixed
weight decay from L2-regularization to true decoupled AdamW formula.
Added GpuTensor::into_parts() for zero-copy CudaSlice extraction.
Added ActivationKernels::relu_fwd_raw/bwd_raw/tanh_fwd_raw/sigmoid_fwd_raw
for direct CudaSlice operation without GpuTensor wrapping.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Moved ml-supervised's duplicate GpuTensor (stream-carrying) + GpuLinear +
50 free functions to ml-core/cuda_autograd/stream_ops.rs as StreamTensor
and StreamLinear. ml-supervised/gpu_tensor.rs → 53 lines of re-exports.
Two tensor flavors now canonical in ml-core:
- GpuTensor: takes &Arc<CudaStream> per-call (autograd integration)
- StreamTensor: carries Arc<CudaStream> internally (self-contained ops)
Zero consumer import changes. Both crates compile clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Final cleanup:
- 61 test files + 5 example files: candle imports replaced
- 8 testing/integration files: migrated to cudarc/ml-core types
- 3 services/trading_service test files: migrated
- Root Cargo.toml: candle-core, candle-nn removed from [workspace.dependencies]
- crates/ml/Cargo.toml: candle-nn dependency removed
- testing/e2e/Cargo.toml: candle-core dependency removed
Zero active candle_core/candle_nn/candle_optimisers code references remain.
Zero candle dependency declarations in any Cargo.toml.
Remaining "candle" strings are exclusively in doc comments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add CudaSlice<f32> accessor methods to GpuTrainResult and GradientResult
(loss_cuda_slice, grad_norm_cuda_slice) in ml-dqn, eliminating redundant
tensor_to_cuda_slice_f32 calls at every training guard check site.
- GpuTrainResult: add from_fused_scalars() constructor and CudaSlice extractors
- GradientResult: add CudaSlice extractors that delegate to GpuTrainResult
- fused_training.rs: use from_fused_scalars() instead of Tensor::new() wrapping
- train_step.rs: use result.loss_cuda_slice() instead of tensor_to_cuda_slice_f32
- training_loop.rs: same CudaSlice accessor pattern for both fused and accum paths
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- gpu_weights.rs: Remove slow-path Candle fallback (flatten/cast/contiguous)
from extract_one() and sync_one(). F32-contiguous is now a hard requirement
enforced by ensure_f32_contiguous() at network construction. Non-F32 or
non-contiguous tensors produce a clear error instead of silently falling
back to the Candle conversion pipeline.
- gpu_action_selector.rs: Remove unused ensure_contiguous_f32(). Keep
stream_from_device/cuda_u32_to_tensor/cuda_f32_to_tensor as boundary
converters (they have external callers).
- gpu_experience_collector.rs: Clean up doc comments referencing Candle
where only cudarc CudaSlice is used.
- mod.rs: Update module docs to clarify Candle Tensor/Device are only
at the upload boundary (DqnGpuData, PpoGpuData, tensor_to_cuda_slice).
- tlob/trainable_adapter.rs: Fix double candle_core:: path that broke build.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All VarBuilder/NoisyLinear init changed from BF16 to F32.
The fused CUDA trainer's Adam kernel operates on F32 master weights
with separate BF16 mirrors for forward kernels. BF16 VarBuilder
caused dtype mismatches in Candle forward paths and broke ensure_f32.
381 ml-dqn tests pass, 0 errors, 0 warnings workspace-wide.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace deprecated cudarc memcpy_stod with clone_htod across all GPU
upload paths (14 call sites in trainers, cuda_pipeline, hyperopt).
Replace deprecated memcpy_dtov with clone_dtoh in ml-supervised
gpu_tensor.rs.
Bridge GpuTensor-migrated submodules (xLSTM, Liquid CfC, TFT GRN)
with Candle Tensor callers via from_candle_tensor/to_candle_tensor
conversion utilities at API boundaries. Fix CudaDevice->CudaContext
in ml-dqn distributional_dueling.rs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace VarMap/VarBuilder weight storage with GpuVarStore (native CUDA)
for RMSNorm, LayerNorm, and ResidualBlock. Linear layers in ResidualBlock
now use GpuLinear with cuBLAS sgemm. Cold-path forwards convert between
GpuTensor and Candle Tensor at the boundary since downstream ops (GELU,
LayerNorm, dropout, broadcast) still use Candle.
Changes:
- ml-core cuda_autograd: add GpuTensor::to_candle/from_candle interop,
export GpuParam/AdamWConfig/ActivationKernels/LossKernels/LossResult,
make init::upload_to_gpu public
- rmsnorm.rs: RMSNorm/LayerNorm now take (Arc<CudaStream>, Device, dim)
instead of (VarBuilder, dim). Weights stored in GpuVarStore.
- residual.rs: ResidualBlock now takes (Arc<CudaStream>, Device, config, name).
fc1/fc2 are GpuLinear with cuBLAS sgemm forward. LayerNorm params in GpuVarStore.
- distributional_dueling.rs: updated RMSNorm constructor calls to new API
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace PolicyNetwork and ValueNetwork with CudaPolicyNetwork/CudaValueNetwork
backed implementations. Each struct now stores a cuda_nn GPU-native network
(cuBLAS sgemm + CUDA kernels) alongside shadow Candle layers for autograd
training compatibility. Adds forward_cuda() inference paths that bypass Candle
entirely. Wire CudaTrajectoryTensors into TrajectoryBatch with to_cuda_tensors().
Re-export CudaLSTM from lstm_networks and all cuda_nn types from crate root.
Import GpuContext into continuous_policy, continuous_action_masking, flow_policy,
coupling_layer, and adaptive_entropy for future GPU migration. Add CudaVec::to_tensor()
bridge for cuda_nn→Candle interop. Fix upload_host_to_gpu→upload_to_gpu rename
in ml-core cuda_autograd init.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VarBuilder and NoisyLinear were creating BF16 weights, then ensure_f32
tried Var::set() which rejects dtype changes. Fix: create F32 from the
start. BF16 mirrors are managed separately by GpuDqnTrainer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eliminate candle-core/candle-nn from the KAN and Diffusion model forward
paths in ml-supervised. All dense layers now use cuBLAS sgemm via the new
gpu_tensor module. Element-wise ops (SiLU, sigmoid, tanh, exp) use host
roundtrips for now; fused CUDA kernels are a follow-up.
Changes:
- Add gpu_tensor.rs: GpuTensor (CudaSlice<f32> + shape), GpuLinear
(cuBLAS sgemm), and ~30 element-wise GPU ops
- Rewrite kan/{spline,layer,network}.rs to use GpuTensor instead of
candle_core::Tensor and candle_nn::{VarBuilder,Linear}
- Rewrite diffusion/{denoiser,noise,sampler}.rs to use GpuTensor and
GpuLinear instead of candle_nn::Linear
- Update ml crate trainable adapters (kan/trainable.rs,
diffusion/trainable.rs) to bridge Candle<->GpuTensor at the
UnifiedTrainable interface boundary
- Update ensemble inference adapters for both models
- Add cudarc 0.17 with cublas feature to ml-supervised Cargo.toml
- Candle deps retained in ml-supervised for unconverted models (TFT,
Liquid, Mamba, xLSTM) -- will be removed once all 8 models are
converted
Net: -424 lines, 509 -> ~453 Candle refs remaining (KAN: 0, Diffusion: 0)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace Candle's backward()/GradStore/VarMap/VarBuilder system with CUDA-native
primitives in ml-core::cuda_autograd. This eliminates the Candle dispatch overhead
(~2100 kernel launches per batch) for models that adopt the new API.
Components:
- GpuTensor: CudaSlice<f32> wrapper with shape metadata (replaces candle Tensor)
- GpuVarStore: named parameter store with flatten/unflatten (replaces VarMap/VarBuilder)
- GpuLinear: cuBLAS sgemm forward + manual backward (replaces candle_nn::Linear)
- GpuAdamW: per-parameter CUDA kernel optimizer (replaces candle_optimisers::Adam)
- ActivationKernels: ReLU/LeakyReLU/GELU/Sigmoid/Tanh forward+backward CUDA kernels
- LossKernels: MSE/Huber loss with fused gradient computation
- init: Xavier/Kaiming/near-zero initialization via CPU generate + GPU upload
Added cublas feature to ml-core's cudarc dependency for sgemm support.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds CudaTrajectoryTensors — GPU-resident trajectory batch storage that
replaces the Candle-based TrajectoryTensors. All trajectory data (states,
actions, log_probs, advantages, returns) uploaded as CudaSlice<f32> with
zero Candle tensor overhead.
The cuda_nn module is now complete with all primitives needed to replace
Candle in the PPO training loop:
- CudaLinear (cuBLAS sgemm)
- CudaLSTM (fused gate kernel)
- CudaPolicyNetwork / CudaValueNetwork (MLP stacks)
- CudaAdam (GPU-resident optimizer)
- CudaTrajectoryTensors (GPU batch storage)
- Softmax / log-softmax / ReLU / tanh / sigmoid kernels
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add dqn_forward_only_kernel to dqn_training_kernel.cu — runs a single
branching forward pass (shared layers -> value head -> 3 branch heads ->
C51 distributional -> expected Q) and writes per-branch Q-values to
q_out[B, 11]. No loss computation, no activation saves, no backward.
~3x faster than forward_loss for pure inference.
Wire the kernel into GpuDqnTrainer via forward_only_q() which takes
CudaSlice<f32> states directly and returns &CudaSlice<f32> Q-values,
replacing the Candle BranchingDuelingQNetwork::forward_branches() call
chain (~160 Candle kernel dispatches -> 1 fused CUDA launch).
Mark all 6 ml-dqn Candle forward methods as #[cold] with documentation
that hot-path compute is handled by fused CUDA kernels:
- noisy_layers.rs: NoisyLinear::forward() -> dqn_experience_kernel.cu
- quantile_regression.rs: QuantileNetwork::forward() -> iqn_dual_head_kernel.cu
- distributional.rs: CategoricalDistribution::to_scalar() -> dqn_forward_only_kernel
- curiosity.rs: ForwardDynamicsModel::predict() -> curiosity_training_kernel.cu
- residual.rs: ResidualBlock::forward() -> dqn_forward_only_kernel
- rmsnorm.rs: RMSNorm::forward() / LayerNorm::forward() -> dqn_experience_kernel.cu
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
