3 Commits

Author	SHA1	Message	Date
jgrusewski	68804a1a51	fix(cuda): remove --use_fast_math, eliminate cross-stream NaN race, f32 rewards/dones ... Three root causes of sporadic NaN during training: 1. --use_fast_math (nvcc) breaks IEEE 754 NaN semantics: fmaxf(NaN,x) returns NaN instead of x, isnan()/isinf() compile to false. Replaced with --ftz=true --fmad=true --prec-div=true --prec-sqrt=true across all 4 build.rs (ml, ml-dqn, ml-ppo, ml-core). 2. Cross-stream race: replay buffer wrote batch data on the device's original stream while the trainer read it on a forked stream. Fixed by passing the forked stream to the DQN agent via agent_device, so all GPU components share a single CUDA stream (zero sync overhead). 3. Rewards/dones stored as bf16 in replay buffer caused done=0xFFFF NaN. Converted entire rewards/dones pipeline to f32: experience collector, replay buffer storage, nstep kernel, loss/grad kernels. Also: - Removed fast_isnan/fast_isinf/fast_isfinite wrappers — standard isnan/isinf/isfinite work correctly without --use_fast_math - Updated dqn-smoketest.toml: lr=1e-4, epsilon=1e-8 (f32 Adam values) - Removed debug printfs from gather kernels - Added curiosity_weight to training profile system - Cleaned up smoke_params() inline overrides 11/11 smoke tests pass, 5/5 stress runs of 50-epoch test pass, 359/359 ml-dqn + 895/895 ml unit tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-29 16:18:37 +02:00
jgrusewski	75f83888ca	feat(bf16): mixed-precision kernels, f32 IS-weights, CUTLASS padding, fast_isnan ... Mixed-precision loss/grad kernels: - MSE + C51 loss: float softmax/projection/TD-error (prevents bf16 exp overflow) - MSE + C51 grad: float arithmetic + bf16 range clamp before atomicAdd - Shared memory: float (4 bytes/elem) for numerically stable reductions - Bias kernels: float add+clamp ±500 (prevents bf16 Inf cascade between layers) - Noisy bias kernel: same float clamping fast_isnan/fast_isinf (ROOT CAUSE FIX): - nvcc --use_fast_math implies --no-nans → isnan()/isinf() compiled to false - ALL NaN guards across ALL kernels were dead code - Added bit-pattern IEEE 754 checks to common_device_functions.cuh - Replaced isnan/isinf in 7 kernel files (21 occurrences) - ml-dqn build.rs: all kernels now get common header (no more standalone) f32 PER IS-weights: - GpuBatchSlices.weights: CudaSlice<u16> → CudaSlice<f32> - GpuBatch.weights: GpuTensor → CudaSlice<f32> - Loss/grad kernel signatures: const __nv_bfloat16* → const float* - Upload path: separate f32 memcpy instead of bf16 staging - Eliminates bf16 overflow in IS-weight storage CUTLASS padding: - pad32() helper: round up to next multiple of 32 - 6 value-logit buffers: pad32(num_atoms) (51 → 64) - 6 branch-logit buffers: +32*3 padding per branch 895/895 unit tests, 8/9 smoke tests pass. 50-epoch convergence: NaN at step ~100-200 — backward pass produces NaN gradients within the CUDA graph replay (same atomic execution as Adam). Root cause: bf16 backward GemmEx inputs can overflow. Needs mixed-precision backward pass (same pattern as loss kernels) or f32 gradient output buffers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-28 21:40:09 +01:00
jgrusewski	9a72112590	feat(bf16): precompile ml-dqn + ml-ppo inline kernels, fix cubin mappings ... - ml-dqn build.rs: compile rmsnorm, noisy, residual, cast, replay buffer, seg_tree kernels → cubins - ml-ppo build.rs: compile linear, softmax, lstm, adam kernels → cubins - Wire all constructors to load_cubin (no nvrtc) - Fix ema_kernel cubin: was DQN_UTILITY_CUBIN, should be EMA_CUBIN - Fix VRAM estimation test for BF16 (2 bytes not 4) ml-core: 300/300 | ml-dqn: 348/359 | ml: 880/895 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-28 11:39:07 +01:00