jgrusewski 75f83888ca feat(bf16): mixed-precision kernels, f32 IS-weights, CUTLASS padding, fast_isnan ...

Mixed-precision loss/grad kernels:
- MSE + C51 loss: float softmax/projection/TD-error (prevents bf16 exp overflow)
- MSE + C51 grad: float arithmetic + bf16 range clamp before atomicAdd
- Shared memory: float (4 bytes/elem) for numerically stable reductions
- Bias kernels: float add+clamp ±500 (prevents bf16 Inf cascade between layers)
- Noisy bias kernel: same float clamping

fast_isnan/fast_isinf (ROOT CAUSE FIX):
- nvcc --use_fast_math implies --no-nans → isnan()/isinf() compiled to false
- ALL NaN guards across ALL kernels were dead code
- Added bit-pattern IEEE 754 checks to common_device_functions.cuh
- Replaced isnan/isinf in 7 kernel files (21 occurrences)
- ml-dqn build.rs: all kernels now get common header (no more standalone)

f32 PER IS-weights:
- GpuBatchSlices.weights: CudaSlice<u16> → CudaSlice<f32>
- GpuBatch.weights: GpuTensor → CudaSlice<f32>
- Loss/grad kernel signatures: const __nv_bfloat16* → const float*
- Upload path: separate f32 memcpy instead of bf16 staging
- Eliminates bf16 overflow in IS-weight storage

CUTLASS padding:
- pad32() helper: round up to next multiple of 32
- 6 value-logit buffers: pad32(num_atoms) (51 → 64)
- 6 branch-logit buffers: +32*3 padding per branch

895/895 unit tests, 8/9 smoke tests pass.
50-epoch convergence: NaN at step ~100-200 — backward pass produces NaN
gradients within the CUDA graph replay (same atomic execution as Adam).
Root cause: bf16 backward GemmEx inputs can overflow. Needs mixed-precision
backward pass (same pattern as loss kernels) or f32 gradient output buffers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-28 21:40:09 +01:00

..

backtesting

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

common

feat: gradient stability + 9-agent audit bug fixes

2026-03-27 19:52:44 +01:00

config

…

ctrader-openapi

fix: tune log levels across workspace — demote noisy warn to debug/trace

2026-03-14 11:35:15 +01:00

data

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

database

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

market-data

…

ml

feat(bf16): mixed-precision kernels, f32 IS-weights, CUTLASS padding, fast_isnan

2026-03-28 21:40:09 +01:00

ml-asset-selection

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-backtesting

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-checkpoint

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-core

feat(bf16): ALL tests pass — ml-core 300/300, ml-dqn 359/359, ml 890/895

2026-03-28 12:13:45 +01:00

ml-data

…

ml-data-validation

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-dqn

feat(bf16): mixed-precision kernels, f32 IS-weights, CUTLASS padding, fast_isnan

2026-03-28 21:40:09 +01:00

ml-ensemble

feat(bf16): ml-dqn, ml-ppo, ml-supervised, ml-ensemble, ml-explainability compile clean

2026-03-28 01:51:45 +01:00

ml-explainability

feat(bf16): ml-dqn, ml-ppo, ml-supervised, ml-ensemble, ml-explainability compile clean

2026-03-28 01:51:45 +01:00

ml-features

fix(ml): BF16/F32 boundary alignment, GPU-resident ops across all ML crates

2026-03-15 11:59:31 +01:00

ml-hyperopt

fix: eliminate NVRTC, fix evaluator SIGSEGV, 22D search space, trade physics

2026-03-27 00:33:05 +01:00

ml-labeling

fix(clippy): ml-supervised + ml-core + thin crates — 34 errors fixed

2026-03-19 00:39:03 +01:00

ml-observability

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-paper-trading

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-ppo

feat(bf16): 1722/1722 tests pass — ALL PASS, zero failures

2026-03-28 13:11:47 +01:00

ml-regime

refactor(cuda): eliminate all CPU fallbacks — CUDA mandatory across ML stack

2026-03-16 21:01:28 +01:00

ml-regime-detection

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-risk

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-security

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-stress-testing

…

ml-supervised

feat(bf16): ml-dqn, ml-ppo, ml-supervised, ml-ensemble, ml-explainability compile clean

2026-03-28 01:51:45 +01:00

ml-universe

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-validation

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

model_loader

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

risk

refactor: eliminate candle from entire workspace — tests, examples, Cargo.toml

2026-03-18 00:53:47 +01:00

risk-data

fix: tune log levels across workspace — demote noisy warn to debug/trace

2026-03-14 11:35:15 +01:00

storage

…

trading_engine

fix(ml): BF16/F32 boundary alignment, GPU-resident ops across all ML crates

2026-03-15 11:59:31 +01:00

trading-data

…

training_uploader

…