jgrusewski fc3f6f9df7 fix: 7 NaN root causes in GPU training pipeline + regression test ...

Root causes found and fixed:
- f32/bf16 type mismatch in pad_states_kernel (PER stores f32, kernel read bf16)
- States buffer stride mismatch in backward_full (lda=48 on pad128=128 buffer)
- States buffer stride mismatch in apply_iqn_trunk_gradient (same bug)
- IQN trunk forward stride mismatch in custom matvec kernel
- IQN grad_norm sqrt-per-block bug (sqrtf of partials → wrong norm + NaN)
- Uninitialized state vector slots in experience_state_gather (stale GPU memory)
- CUDA graph warmup launches producing extreme gradients from Xavier weights

Defense-in-depth guards:
- f32_to_bf16_kernel: NaN→0, clamp ±500
- Adam isfinite guards in DQN, IQN, attention, curiosity kernels
- IQN backward isfinite on d_h_s2 atomicAdd
- clipped_saxpy isfinite on auxiliary gradient input

Adds test_no_nan_after_graph_capture regression test (2s, fxcache-based).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-05 12:27:31 +02:00

..

backtesting

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

common

fix: switch tracing to stderr (unbuffered) — stdbuf has no effect on Rust

2026-04-02 16:29:55 +02:00

config

fix(ci): clippy test-gate uses --lib, fix pre-existing lint errors

2026-03-12 19:10:18 +01:00

ctrader-openapi

fix: tune log levels across workspace — demote noisy warn to debug/trace

2026-03-14 11:35:15 +01:00

data

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

database

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

market-data

docs: create/update README.md for all 17 crates

2026-03-01 22:47:39 +01:00

ml

fix: 7 NaN root causes in GPU training pipeline + regression test

2026-04-05 12:27:31 +02:00

ml-asset-selection

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-backtesting

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-checkpoint

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-core

fix: restore explicit buffer_size in GPU profiles (auto-sizing removed)

2026-04-04 01:24:19 +02:00

ml-data

docs: create/update README.md for all 17 crates

2026-03-01 22:47:39 +01:00

ml-data-validation

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-dqn

fix: remove hardcoded MIN_GPU_CAPACITY=1024, use batch_size as floor

2026-04-04 15:32:59 +02:00

ml-ensemble

feat(ensemble): GPU-native ensemble reduce kernel, remove CPU fallbacks

2026-03-29 22:35:37 +02:00

ml-explainability

feat(ensemble): GPU-native ensemble reduce kernel, remove CPU fallbacks

2026-03-29 22:35:37 +02:00

ml-features

fix: test reads threshold from dqn-smoketest.toml — no hardcoded values

2026-03-31 12:37:25 +02:00

ml-hyperopt

refactor: LHS is the correct algorithm for small budgets, not a fallback

2026-03-29 19:18:47 +02:00

ml-labeling

fix(clippy): ml-supervised + ml-core + thin crates — 34 errors fixed

2026-03-19 00:39:03 +01:00

ml-observability

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-paper-trading

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-ppo

feat: add PPO CUDA kernels and curiosity inference kernel

2026-03-31 00:39:59 +02:00

ml-regime

refactor(cuda): eliminate all CPU fallbacks — CUDA mandatory across ML stack

2026-03-16 21:01:28 +01:00

ml-regime-detection

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-risk

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-security

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-stress-testing

fix(clippy): achieve zero warnings across entire workspace

2026-03-10 13:18:57 +01:00

ml-supervised

feat(bf16): ml-dqn, ml-ppo, ml-supervised, ml-ensemble, ml-explainability compile clean

2026-03-28 01:51:45 +01:00

ml-universe

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-validation

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

model_loader

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

risk

refactor: eliminate candle from entire workspace — tests, examples, Cargo.toml

2026-03-18 00:53:47 +01:00

risk-data

fix: tune log levels across workspace — demote noisy warn to debug/trace

2026-03-14 11:35:15 +01:00

storage

docs: create/update README.md for all 17 crates

2026-03-01 22:47:39 +01:00

trading_engine

fix(ml): BF16/F32 boundary alignment, GPU-resident ops across all ML crates

2026-03-15 11:59:31 +01:00

trading-data

docs: create/update README.md for all 17 crates

2026-03-01 22:47:39 +01:00

training_uploader

feat: implement all 15 fxt CLI commands with real gRPC calls

2026-03-03 22:16:35 +01:00