jgrusewski 875f263ec9 feat(bf16): f32 backward dX scratch + bf16 staging — eliminates dX truncation ...

Backward pass dX computation now uses f32 scratch buffers with bf16
staging for the backward chain. Pattern: f32 GemmEx → relu_mask (f32) →
f32→bf16 cast → next layer reads bf16 dY.

Changes:
- 6 bw_d_h_* scratch buffers: CudaSlice<half::bf16> → CudaSlice<f32>
- New bw_dy_bf16_staging: shared bf16 buffer for layer transitions
- backward_fc_layer dX: gemmex_bf16 → gemmex_bf16_acc_f32 (f32 output)
- launch_dx_only: gemmex_bf16 → gemmex_bf16_acc_f32 (f32 with beta)
- relu_mask_kernel: reads/writes f32 (no bf16 clamp needed)
- f32_to_bf16_cast_kernel: ±500 clamp at type boundary (in backward_kernels.cu)
- cast_dx_to_staging: f32 scratch → bf16 staging per layer
- IQN/ensemble backward: bf16→f32 cast for dX input, f32→bf16 for dY output
- bw_d_h_s2_as_bf16(): attention backward receives bf16 via staging

Hyperparameters updated for f32 Adam:
- learning_rate: 1e-5 → 1e-4 (updates must exceed bf16 shadow step ~1e-3)
- adam_epsilon: 1e-3 → 1e-8 (standard Adam, bf16 workaround no longer needed)
- grad_norm NaN skip kept as defense-in-depth (source still under investigation)

895/895 unit + 359/359 ml-dqn tests pass.
7-11/11 smoke tests (intermittent NaN from unknown source — NOT backward dX).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-29 12:17:04 +02:00

..

backtesting

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

common

feat: gradient stability + 9-agent audit bug fixes

2026-03-27 19:52:44 +01:00

config

fix(ci): clippy test-gate uses --lib, fix pre-existing lint errors

2026-03-12 19:10:18 +01:00

ctrader-openapi

fix: tune log levels across workspace — demote noisy warn to debug/trace

2026-03-14 11:35:15 +01:00

data

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

database

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

market-data

…

ml

feat(bf16): f32 backward dX scratch + bf16 staging — eliminates dX truncation

2026-03-29 12:17:04 +02:00

ml-asset-selection

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-backtesting

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-checkpoint

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-core

feat(bf16): ALL tests pass — ml-core 300/300, ml-dqn 359/359, ml 890/895

2026-03-28 12:13:45 +01:00

ml-data

…

ml-data-validation

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-dqn

fix(bf16): IS-weight bf16 clamp + last NaN root cause documented

2026-03-29 09:48:44 +02:00

ml-ensemble

feat(bf16): ml-dqn, ml-ppo, ml-supervised, ml-ensemble, ml-explainability compile clean

2026-03-28 01:51:45 +01:00

ml-explainability

feat(bf16): ml-dqn, ml-ppo, ml-supervised, ml-ensemble, ml-explainability compile clean

2026-03-28 01:51:45 +01:00

ml-features

fix(ml): BF16/F32 boundary alignment, GPU-resident ops across all ML crates

2026-03-15 11:59:31 +01:00

ml-hyperopt

fix: eliminate NVRTC, fix evaluator SIGSEGV, 22D search space, trade physics

2026-03-27 00:33:05 +01:00

ml-labeling

fix(clippy): ml-supervised + ml-core + thin crates — 34 errors fixed

2026-03-19 00:39:03 +01:00

ml-observability

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-paper-trading

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-ppo

feat(bf16): 1722/1722 tests pass — ALL PASS, zero failures

2026-03-28 13:11:47 +01:00

ml-regime

refactor(cuda): eliminate all CPU fallbacks — CUDA mandatory across ML stack

2026-03-16 21:01:28 +01:00

ml-regime-detection

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-risk

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-security

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-stress-testing

fix(clippy): achieve zero warnings across entire workspace

2026-03-10 13:18:57 +01:00

ml-supervised

feat(bf16): ml-dqn, ml-ppo, ml-supervised, ml-ensemble, ml-explainability compile clean

2026-03-28 01:51:45 +01:00

ml-universe

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

ml-validation

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

model_loader

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

risk

refactor: eliminate candle from entire workspace — tests, examples, Cargo.toml

2026-03-18 00:53:47 +01:00

risk-data

fix: tune log levels across workspace — demote noisy warn to debug/trace

2026-03-14 11:35:15 +01:00

storage

…

trading_engine

fix(ml): BF16/F32 boundary alignment, GPU-resident ops across all ML crates

2026-03-15 11:59:31 +01:00

trading-data

…

training_uploader

…