Cast all Tensor::full() / Tensor::from_vec() call sites to training_dtype
instead of defaulting to F32. Fixes dtype mismatch errors (BF16 vs F32)
in PPO training on CUDA:
- tensor_ops: scalar_mul, clamp, normalize match operand dtype
- trajectories: TrajectoryBatch/MiniBatch to_tensors cast to training dtype
- continuous_ppo: ContinuousTrajectoryBatch/MiniBatch to_tensors cast
- adaptive_entropy: cast entropy to F32 for alpha multiplication boundary
- continuous_policy: forward() input cast, Tensor::full scalars match dtype
- flow_policy: sample_base_noise cast to training dtype
- hidden_state_manager: reset tensors use training_dtype
- ensemble/ppo adapter: predict input cast to training dtype
- trainable_adapter: test uses training_dtype instead of hardcoded F32
Verified: 198/198 ml-ppo tests pass, 63/63 ml PPO tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- backtest_forward_kernel: dim_overrides must precede common_device_functions.cuh
which has #error guards requiring STATE_DIM/MARKET_DIM/PORTFOLIO_DIM to be
defined before inclusion. Experience collector already had correct ordering.
- Cap MAX_EPISODES from 8192→4096 (diminishing returns above 4096, wastes walltime)
- Cap trainer .min() from 0x8000 (32768) → 4096 to match
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Cast input to weight dtype in DQN residual, rmsnorm, noisy_layers
- Set use_gpu=true in QNetworkConfig defaults and all config sites
- Resolve BF16 boundary mismatches in attention, curiosity, branching,
distributional_dueling across ml-dqn
- GPU-resident regime ops with BF16 boundary casts, eliminate .expect() in CUDA paths
- Eliminate all Device::Cpu fallbacks — GPU-only across 10 ML crates
- PPO: cast logits to F32 before softmax, cast batch tensors to training dtype
- Gradient collapse detection for RegimeConditionalDQN
- Wire halt_grad_collapse from CUDA guard kernel to halt training
- Dead neuron detection uses active network VarMap + squeeze factored readback
- Increment gradient_logging_step in GPU PER path
- Gradient collapse warmup guards use original buffer_size
- Cap training steps per epoch + tracing migration
- Replace Tensor::all() with sum_all() for pinned Candle compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The ml crate fix alone wasn't enough — ml-core, ml-dqn, ml-ppo,
ml-supervised, ml-ensemble, ml-explainability, ml-hyperopt, and
ml-labeling all had `default = ["cuda"]`, each independently pulling
in cudarc via candle-core/cuda.
Now `default = []` on all sub-crates. CUDA activates only when the
compile-and-train template passes `--features ml/cuda`, which
propagates through ml's cuda feature gate to all sub-crates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two GPU bottlenecks fixed:
1. GradStore key mismatch (40,000+ warnings/run): clip_grad_norm now uses
TensorId-based iteration exclusively. The old Var-based lookup always
failed (0/16 matches) due to identity drift from BF16 dtype conversion,
then fell back to TensorId anyway. Removed the pointless Var path and
fallback warning entirely.
2. CUDA_ERROR_INVALID_PTX on H100 (sm_90): The standard per-thread kernel
(~7.5 KB stack × 256 threads) caused invalid PTX when co-compiled with
the warp kernel for compute_90. Guarded with #if __CUDA_ARCH__ < 900
so only the warp-cooperative kernel (200 bytes/lane) is compiled on
Hopper. Rust-side kernel loading restructured to query SM before
compilation and load the appropriate kernel variant directly.
Test results: ml-core=311, ml-dqn=416, ml=915 — 0 failures, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The circuit breaker (>20% drawdown) carried forward across epochs,
permanently locking out all trades once triggered. With compounding
portfolios, early drawdown in one epoch could produce zero rewards
for all subsequent epochs (constant rewards bug).
Fix: reset peak_value (high-water mark) at each epoch start. Capital
still compounds (Bug #15 preserved), but drawdown is measured fresh
per epoch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- gradient_utils: Add TensorId-based fallback when Var identity mismatch
causes 0/N vars to match GradStore (Candle Adam clones Arcs). Fallback
computes norm AND clips via insert_id. Throttled warning (1st + every
1000th). 7 unit tests including mismatch-actually-clips.
- monitoring: Track full 45-action factored space (5 exposure × 3 order
× 3 urgency). Fix validate_rewards false alarm on GPU path where single
aggregated mean_reward per epoch gives N=1 → std=0.
- trainer: GPU experience collection routes exposure actions through
route_action() for factored tracking instead of exposure-only counts.
Applied in both per-step and epoch-summary paths.
- train_baseline_rl: Auto-detect VRAM <8GB → disable GPU replay buffer
to prevent OOM on RTX 3050 Ti class GPUs.
- smoke_test_real_data: E2E DQN training test with 6 assertions (epoch
completion, loss decrease, finite losses, Q-value divergence, 45-action
space, finite gradient norms).
Validated: 1642 tests pass (ml=915, ml-core=311, ml-dqn=416), 0 clippy
warnings, baseline RL trains 10 epochs on CUDA with Sharpe +5.45.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU PER replay buffer had a hardcoded 4 GB MAX_BYTES limit that
rejected the auto-sizer's 10M-entry proposal on H100 (80 GB VRAM).
Now per_max_buffer_bytes() computes 20% of total VRAM (min 1 GB) and
flows through OptimalReplayConfig → DQNConfig → GpuReplayBufferConfig
so both subsystems agree on the budget.
Also fixes misleading regime detection log (indices 211/219 → 40/41)
and renames dqn_config_2025 → dqn_default_config.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevents experience collector output tensors from starving backward
graph and GPU PER on VRAM-constrained GPUs like RTX 3050 Ti 4GB.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
free_memory_mb is sampled after model/optimizer load, so subtract only
backward-graph + fragmentation + data upload headroom (400 MB), not the
full model/PER/data stack. Increase collector share to 50%.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Staging/GPU replay buffer: truncate batch when it exceeds ring buffer
capacity (CPU fallback can flush 100K+ experiences at once)
- GpuExperienceCollector: compute shmem tile rows dynamically to stay
under 48 KB default limit instead of hardcoded 64-row constant
- Auto batch size: subtract concurrent VRAM consumers (~530 MB model +
optimizer + PER + data) before budgeting experience collector at 40%
- Hyperopt DQN adapter: GPU PER always on, include replay buffer in
VRAM estimate for small GPUs (was excluded assuming CPU-only PER)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GPU experience collector:
- Add GpuHardwareInfo with SM count detection from device name lookup
(H100=132, A100=108, L40S=142, RTX 4090=128, etc.)
- optimal_n_episodes() scales to GPU: sm_count × 2 warps × 32 threads,
capped by 15% free VRAM budget, 256-aligned for block scheduling
- Remove hardcoded .min(256) cap in trainer — auto-scales from 128 to 8192
- MAX_EPISODES_LIMIT raised from 256 to 4096
Numerical stability:
- BF16 cross-entropy epsilon: 1e-8 → 1e-4 (below BF16 precision floor
1e-8 rounds to zero, making log-stabilization a no-op → -inf → NaN)
- Add log_probs.clamp(-20, 0) guard against -inf × 0 = NaN in loss
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
clip_grad_norm now returns a GPU-resident Tensor instead of (f64, f64),
keeping backward → clip → optimizer.step fully pipelined on GPU with
zero cuStreamSynchronize stalls. The unconditional multiply trick
(scale = min(max_norm/(norm+eps), 1.0)) avoids the conditional branch
that previously required reading the norm to CPU.
Key changes:
- gradient_utils::clip_grad_norm: return Tensor, unconditional GPU multiply
- Handle BF16 mixed-precision via per-gradient to_dtype cast
- adam.rs: backward_step_with_monitoring returns Tensor (zero sync)
- gradient_accumulation: delegate to gradient_utils (DRY, same GPU path)
- dqn.rs: single sync boundary after ALL GPU work queued
1746 tests passing (ml-core 286, ml-dqn 388, ml-ppo 198, ml 874).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
clip_grad_norm: accumulate squared norms on GPU-resident scalar, single
to_scalar() at end (was 16-20 per-param syncs per step × 2917 steps/epoch).
check_gradients_finite: same GPU-accumulation pattern, single sync.
gpu_replay_buffer sample_proportional/rank_based: generate random targets
via rand(0,1)*total_sum on GPU, normalize weights via broadcast_div
(eliminates 2 to_vec0 syncs per sample call).
gpu_replay_buffer update_priorities_gpu: replace CPU loop of 50 individual
slice_scatter calls with single batched index_add delta trick.
dqn NaN detection (every 500 steps): accumulate 3 NaN counts on GPU,
single to_scalar for total; detailed breakdown only if NaN found.
dqn dead neuron detection (every 1000 steps): accumulate dead count on
GPU-resident scalar (was to_vec0 per parameter tensor, ~16-20 syncs).
Net: ~22 GPU→CPU syncs + 50 micro-kernels per training step → 2 syncs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bring Branching Dueling Q-Network (Tavakoli 2018) to full Rainbow parity
with the existing GPU hotpath. 3 independent advantage heads (exposure=5,
order=3, urgency=3) decompose the 45-action space into learnable branches.
H1 - CUDA fallback: gate GpuExperienceCollector when use_branching=true
(fused kernel hardcodes NUM_ACTIONS=5, incompatible with 45 factored)
H2 - Per-branch C51 distributional: each branch outputs [batch, n_d, atoms]
log-softmax, loss = avg of D cross-entropies vs projected Bellman target
M1 - NoisyNet: MaybeNoisyLinear enum in branch heads, reset_noise/disable_noise
wired through select_action, compute_loss, and set_eval_mode
M2 - Regime-conditional IS weights: Trending=1.2, Ranging=0.8, Volatile=0.6
applied to branching loss via ADX/CUSUM features at state[40:41]
M3 - State dim alignment: align_dim_for_tensor_cores() in from_dqn_params()
for H100 HMMA dispatch (8-byte alignment)
L1 - Fill simulator: splitmix64 replaces golden ratio hash (chi-squared tested)
L2 - Hyperopt 29D: branch_hidden_dim [64,256] added to PSO search space
Config plumbing: branch_hidden_dim, v_min/v_max/num_atoms, use_distributional,
use_noisy, noisy_sigma_init all flow from DQNConfig → BranchingConfig.
10 files, +3207/-125 lines, 33 branching tests + 387 ml-dqn + 284 ml-core pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
training_dtype() now returns BF16 on all CUDA devices, enabling full
tensor-core utilization. F32 is used only at boundaries (scalar
extraction, loss computation, softmax). VRAM estimator updated to
account for BF16 byte sizes, C51/QR atoms, dueling streams, NoisyNet
param doubling, and GPU PER buffer pre-allocation.
Changes:
- mixed_precision.rs: training_dtype() returns BF16 on CUDA
- curiosity.rs: F32 cast before scalar extraction
- network.rs: F32 output at NetworkLayers forward boundary
- traits.rs: estimate_trial_vram_mb_full() with BF16-aware sizing
- dqn.rs adapter: uses full estimator with worst-case architecture
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Four monitoring functions (estimate_avg_q_value_with_early_stopping,
collect_qvalue_statistics, compute_q_gap_for_epoch, compute_per_action_q_values)
iterated over batch_sample.experiences to build CPU tensors for forward passes.
When GPU PER (GpuPrioritized) is active, experiences is always vec![] — all data
lives on GPU tensors in gpu_batch. This created zero-element tensors with non-zero
shapes, triggering CUBLAS_STATUS_INVALID_VALUE on the next forward pass.
Fix: all four functions now check for gpu_batch.states and use it directly,
falling back to CPU experiences only for non-GPU buffers. Also removes debug
eprintln probes, wires new_on_device for DQN/RegimeConditionalDQN construction,
and adds GPU-aware smoke tests (33 pass, 874 total, 0 failures).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates the CPU roundtrip in PER binary search: previously the
cumsum tensor (~400KB for 100K buffer) was downloaded to CPU, searched
in a loop, then indices uploaded back. Now a CUDA kernel runs one
thread per target with O(log n) binary search — zero DMA.
- Add searchsorted_kernel.cu (40-line CUDA binary search kernel)
- Implement SearchSorted as Candle CustomOp2 with CPU fallback
- Wire into sample_proportional() and sample_rank_based()
- Fix all clippy warnings in gpu_replay_buffer.rs (const fn, shadow,
doc backticks, to_owned, div_ceil, module_name_repetitions)
- Fix wildcard_enum_match_arm in mixed_precision.rs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace expensive CPU↔GPU data transfers with GPU-native operations:
- check_gradients_finite: sum_all() scalar readback (4B per tensor)
replaces flatten_all()+to_vec1() that copied entire gradients to CPU
(up to 200MB per step across 9 call sites in DQN+PPO)
- Huber loss: affine() replaces 3× Tensor::from_vec(vec![const; N])
CPU Vec allocations in the inner loss computation loop
- update_priorities_gpu: per-index slice_scatter (~1KB DMA) replaces
full-buffer to_vec1()+from_vec() roundtrip (1.2MB DMA per step)
- PER sampling: single from_vec + GPU to_dtype replaces duplicate
from_vec calls and CPU type conversion roundtrips
- RegimeConditional: log warning on silent GPU batch drops
822 tests pass (350 ml-dqn, 274 ml-core, 198 ml-ppo), 0 clippy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for GPU PER hot path:
1. IQN quantile loss used empty CPU weights Vec instead of GPU-resident
weights_tensor_cached — caused CUDA_ERROR_ILLEGAL_ADDRESS from
uninitialized GPU memory. Now uses cached GPU tensor matching C51
and standard DQN paths.
2. GpuPrioritized add()/add_batch() replaced with StagedGpuBuffer:
add() stages on CPU (Vec::push, zero GPU ops), sample() batch-flushes
staging→GPU in one DMA before sampling. Production path (insert_batch_tensors)
bypasses staging entirely — GPU→GPU with zero CPU.
3. All 9 ML sub-crates default to cuda feature so `cargo test -p ml-dqn`
exercises GPU code paths on CUDA workstations. CI service crates use
default-features=false, unaffected.
Test results: 350 passed (was 343+7 failed), 0 failed, 1 ignored.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove cuda from default features in ml-core, ml-dqn, ml-ppo, ml
- Propagate cuda feature from ml → ml-core/ml-dqn/ml-ppo
- CI compile-training already uses --features ml/cuda explicitly
- Fix MaxDD log format: {:.1}% → {:.3}% (was rounding 0.033% to 0.0%)
- Suppress unused_labels/unused_variables warnings for cfg(cuda) code
- Add CALLBACK_ENDPOINT env to ml-training-service deployment
- Fix Grafana active_workers query to use sum() with fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move UnifiedTrainable trait, TrainingMetrics, CheckpointMetadata, and
checkpoint helpers from ml to ml-core (zero new dependencies — ml-core
already had candle-core + serde_json)
- Wrap Mamba2SSM in Mamba2TrainableAdapter to satisfy orphan rule (trait
in ml-core, type in ml-supervised — all other 9 models already used
wrapper pattern)
- Make Mamba2SSM::validate() pub for cross-crate adapter access
- Delete 5 permanently disabled deployment modules (cfg(any()) — never
compiled): registry, hot_swap, validation, monitoring, endpoints
(-5,842 lines)
- Delete 2 undeclared dead files in training/: dqn_trainer.rs,
transformer_trainer.rs (-138 lines)
- Fix pre-existing compute_loss test shape mismatch in mamba adapter
UnifiedTrainable in ml-core unblocks future trainers/ extraction (17.8K
lines) since model-specific trainers can now depend on ml-core for the
trait without pulling in the full ml monolith.
14 files changed, +128 -6,497 (net -6,369 lines)
Tests: 274 ml-core + 948 ml = 1,222 passed, 0 failed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move Sharpe ratio metrics and SIMD performance module to ml-core.
These only depend on MLError which is already in ml-core.
Add approx = "0.5" to ml-core dev-dependencies for Sharpe tests.
Skip observability/ (needs prometheus dep — stays in ml).
Test counts: 271 ml-core + 2487 ml = 2758 total, 0 failures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 6 shared modules (~2.3K lines) from ml to ml-core:
- trading_action.rs (TradingAction enum)
- action_space.rs (action masking, re-exports)
- xavier_init.rs (Xavier/Glorot weight initialization)
- mixed_precision.rs (AMP utilities, FP16/BF16)
- order_router.rs (deterministic order routing)
- portfolio_tracker.rs (portfolio state tracking)
With TradingAction now in ml-core alongside FactoredAction, the
FactoredActionExt extension trait is eliminated entirely —
from_trading_action()/to_trading_action() become inherent methods
on FactoredAction. This removes the need for `use FactoredActionExt`
imports in reward.rs, gae.rs, and trajectories.rs.
Test counts: 250 ml-core + 2508 ml = 2758 total, 0 failures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move memory_optimization/ (4.6K lines, 7 files) from ml to ml-core
- Move batch_size_resolver.rs to ml-core (now both deps in same crate)
- Add tempfile to ml-core dev-dependencies (qat tests)
- Re-export both modules from ml facade
- Keep checkpoint/ in ml (model_implementations.rs has model-specific deps)
185 ml-core tests + 2573 ml tests = 2758 total, 0 failures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move safety/ module (6.5K lines) from ml to ml-core (self-contained)
- Keep security/ in ml (depends on ensemble::EnsembleDecision)
- Move From<ProductionTrainingError> for MLSafetyError to ml (cross-crate)
- Rename Real-prefixed inference types to proper names:
RealInferenceError → InferenceError
RealInferenceConfig → InferenceConfig
RealPredictionResult → InferencePrediction
RealNeuralNetwork → NeuralNetwork
RealMLInferenceEngine → MLInferenceEngine
- Re-export safety from ml facade for backward compatibility
129 ml-core tests + 2629 ml tests = 2758 total, 0 failures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move optimizers, gradient_accumulation, gradient_utils, cuda_compat,
tensor_ops, and gpu (device config, capabilities, memory profiling) to
ml-core. These are shared compute primitives used by all models.
Also commit module files for core types (common, config, error, model,
traits, types) that were moved from ml to ml-core in task 5a but left
staged without being committed.
Notable changes:
- resolve_batch_size() stays in ml (new batch_size_resolver module)
because it depends on memory_optimization::auto_batch_size which
has not yet moved to ml-core
- FactoredAction legacy bridge converted from inherent impl to
extension trait (FactoredActionLegacy) since FactoredAction is now
defined in ml-core, not ml
- candle-optimisers added to ml-core dependencies (needed by Adam)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all inline type definitions from ml/src/lib.rs to ml-core:
MLError, MLResult, Trade, MarketRegime, HealthStatus, Features,
MLModel trait, ModelRegistry, ParallelExecutor, LatencyOptimizer,
TrainingMetrics, ValidationMetrics, InferenceResult, ModelMetadata.
Dedup: consolidate ConfigError{reason}/ConfigurationError(msg) into
single ConfigError(String) tuple variant (was 2 variants, 174 refs).
Cleanup: convert create_hft_* free functions to associated methods
(HFTPerformanceProfile::ultra_low_latency(), ParallelExecutor::hft()).
ml facade re-exports via `pub use ml_core::*`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 5 empty sub-crates (ml-core, ml-dqn, ml-ppo, ml-supervised, ml-infra)
to workspace. Modules will be moved from monolithic ml crate in subsequent
tasks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
