Commit Graph

8 Commits

Author SHA1 Message Date
jgrusewski
d95e205d4b refactor(ml): delete mixed_precision module — BF16 unconditional on CUDA
Eliminate the entire mixed_precision runtime indirection layer:
- Delete crates/ml-core/src/mixed_precision.rs (training_dtype, ensure_training_dtype, align_dim_for_tensor_cores)
- Inline ~100 call sites across 130 files to constants:
  training_dtype(&device) → candle_core::DType::BF16
  ensure_training_dtype(x) → x.to_dtype(candle_core::DType::BF16)
  align_dim_for_tensor_cores(x, &device) → (x + 7) & !7
- Remove re-exports from ml-dqn, ml-supervised, ml lib.rs
- Clean config/toml/json/shell references

No CPU/Metal training path exists — BF16 is the only dtype.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 16:11:48 +01:00
jgrusewski
216db0301d fix(gpu): eliminate all GPU→CPU roundtrip violations — zero guard findings
Replace .to_vec1()/.to_vec2() bulk downloads with GPU-resident ops:
- PPO/DQN action selection: Gumbel-max trick (categorical on GPU)
- Scalar readbacks: .to_scalar() instead of .to_vec1()[0]
- GPU stats: abs().max(), sqr().sum_all() — single scalar out
- NaN/Inf check: sum_all().to_scalar().is_finite()
- Guard exclusions: inference output boundaries + CPU fallback with GPU path

26 files across ml-ppo, ml-dqn, ml-supervised, ml (ensemble adapters, metrics, data_loading)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 00:19:09 +01:00
jgrusewski
77715209f6 chore: delete dead demo_dqn.rs, update GPU hot-path guard
- Remove ml-dqn/src/demo_dqn.rs (134 lines, unused)
- Guard: add new hot-path patterns, tighten leak detection

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 23:50:54 +01:00
jgrusewski
9d564a829e fix(guard): restore #[cfg(test)] exclusion, keep cfg(not(cuda)) filter
Unit tests need scalar readbacks for assertions — .to_scalar() is not
flagged but .to_vec1() in test modules would generate false positives.
Guard now correctly excludes inline #[cfg(test)] modules while checking
all production code including ensemble adapters.

Full --all scan reveals 31 pre-existing production violations across
ml-dqn, ml-ppo, ml-supervised, and ensemble adapters — separate cleanup.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 23:11:06 +01:00
jgrusewski
39e5dafc62 fix(cuda): harden GPU hot-path guard — exclude tests, remove false-positive patterns
- Guard: exclude #[cfg(test)] modules (tests need scalar readbacks for assertions)
- Guard: exclude #[cfg(not(feature = "cuda"))] guarded expressions (dead code with CUDA)
- Guard: remove Tensor::from_vec/from_slice from leak patterns (CPU→GPU is correct direction)
- Guard: remove .to_scalar from leak patterns (single 4-byte readback, not bulk transfer)
- dqn.rs: rewrite log_q_values() to use GPU tensor ops (min/max/mean/var), eliminate to_vec1
- dqn.rs: rewrite clip monitoring to use GPU tensor ops, individual .to_scalar() readbacks
- ppo.rs: replace stacked .to_vec1() metrics readback with individual .to_scalar() calls
- evaluate_baseline.rs: single-bar DQN action from .to_vec1::<u32>() to .to_scalar::<u32>()
- mod.rs: remove gpu_upload_vec/gpu_upload_slice wrappers (guard no longer flags from_vec)
- Delete dead demo_dqn.rs (zero callers, stub returning mock results)

Remaining: 4 .to_vec1() violations across 3 files — porting to existing CUDA implementations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 23:04:46 +01:00
jgrusewski
1fae917c22 perf(cuda): H100 kernel optimizations — nvcc pipeline, kernel fusion, GPU-only training
- Migrate from NVRTC JIT to cached nvcc -O3 for all CUDA kernels
- Fuse guard kernels, increase prefetch chunk, eliminate per-step GPU alloc
- H100-specific: fused Adam, warp reductions, shmem tiling, PPO occupancy
- Vectorize gather_states with __ldg() and 4x unroll
- sincosf() Box-Muller + paired Gaussian generation in noisy nets
- Shared-memory tiled branching DQN forward pass for sm_<90
- GPU-resident training guard kernel replacing Candle tensor ops
- Eliminate all to_vec1/to_vec2 CPU roundtrips, DtoD weight copy
- GPU PER mandatory everywhere — kill CPU replay path on CUDA
- Full GPU action masking — eliminate CPU fallback path
- Fix cuBLAS handle sharing via OnceLock (root cause of 49 cascade failures)
- Fix ILLEGAL_ADDRESS: scratch1_dist buffer overflow, stack sizing, curand determinism
- Fix CudaStream lifetime: bind before .context() to extend lifetime
- Keep raw cudarc buffers alive across epochs
- Add gpu-hotpath-guard.sh (37 patterns) and ptx-cache-invalidate.sh

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 11:58:40 +01:00
jgrusewski
6efb78ba9c feat(cuda): pure-CUDA backtest forward, eliminate Candle dispatch in hyperopt DQN
Replace closure-based evaluate() with evaluate_dqn_graphed() for non-OFI
walk-forward backtest path. Extracts DuelingWeightSet from VarMap (branching
or standard dueling) and runs hand-written warp-cooperative CUDA forward
kernel with CUDA Graph capture — zero Candle dispatch overhead per step.

Key changes:
- GpuBacktestEvaluator::stream() getter for weight extraction on eval stream
- DQNAgentType::is_using_branching() / network_dims() for CUDA kernel config
- Hyperopt evaluate_gpu() non-OFI path: extract_dueling_weights_branching()
  → evaluate_dqn_graphed() (CUDA Graph accelerated)
- OFI path: retains Candle closure for state permutation (gather kernel
  layout mismatch — future CUDA permutation kernel)
- 66+ GPU hot-path violations hardened to hard errors across DQN/PPO/supervised
- Stripped all gpu-ok suppression comments
- Proper #[cfg(feature = "cuda")] gating for CUDA-only code paths

77 files, 0 errors, 0 warnings across workspace.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 02:49:25 +01:00
jgrusewski
4709ca8bc2 feat(dqn): enable Branching DQN with 45 factored actions (5×3×3)
Restore 45-action factored space via Branching DQN (Tavakoli 2018),
outputting 11 Q-values (5+3+3) instead of 45. This was reduced to 5
exposure-only actions during debugging and was never intended as permanent.

- Enable use_branching: true by default in DQNConfig and DQNHyperparameters
- Add branching paths to select_action_with_confidence and select_action_inference
- Update agent.rs select_action_factored for branching-aware selection
- Expand CountBonus to per-branch tracking with bonuses_branched()
- Add order_type + urgency distribution tracking in monitoring
- Add DQN_ORDER_ACTIONS=3, DQN_URGENCY_ACTIONS=3, DQN_TOTAL_ACTIONS=45 to CUDA header
- Fix 7 pre-existing clippy doc_markdown errors in regime_conditional.rs
- Fix pre-existing cognitive_complexity in replay_buffer_type.rs (extract helpers)
- Fix flaky GPU test OOM under parallel execution (CPU fallback + test VRAM safety)
- Delete unused flash_attention submodules (block_sparse, causal_masking, etc.)
- Add GPU hot-path guard scripts and ensemble/hyperopt adapter improvements

Tests: ml-dqn 416/0, ml 905/0, clippy 0 errors on both crates

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 22:00:13 +01:00