jgrusewski 68e83bef0e fix: async adam_step HtoD + batch_size=0 auto — root cause of >128 hang ...

The batch_size >128 hang was caused by cudarc's synchronous
memcpy_htod for the adam_step counter. At batch_size=128 the
sync completes fast enough, but at 509+ the pipeline drain
from the sync interacts with CUDA Graph replay timing and
deadlocks the stream.

Replaced all 3 memcpy_htod(&[self.adam_step]) calls with raw
cuMemcpyHtoDAsync_v2 — zero pipeline drain, zero CPU sync.

Production TOML set to batch_size=0 (AutoBatchSizer drives it).
AutoBatchSizer caps at 8192 for RL training.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-02 08:19:56 +02:00

..

asset_selection

refactor(ml): extract universe, backtesting, asset-selection into sub-crates

2026-03-08 15:17:22 +01:00

backtesting

refactor(ml): extract universe, backtesting, asset-selection into sub-crates

2026-03-08 15:17:22 +01:00

benchmark

fix: 11 H100 training bugs — Sharpe per-trade, batch autosizing, PER/HER capacity, Q-clip

2026-04-02 00:54:12 +02:00

checkpoint

refactor(ml): split monolith into 9 sub-crates + delete dead code

2026-03-08 15:17:22 +01:00

cuda_common

…

cuda_pipeline

fix: async adam_step HtoD + batch_size=0 auto — root cause of >128 hang

2026-04-02 08:19:56 +02:00

data_loaders

cleanup: remove ALL 32 enable_* feature flags — all features unconditional

2026-03-31 10:56:20 +02:00

data_pipeline

fix(ci): mark 8 heavy data-loading tests #[ignore] for CI

2026-03-19 13:38:49 +01:00

data_validation

refactor(ml): split monolith into 9 sub-crates + delete dead code

2026-03-08 15:17:22 +01:00

deployment

refactor(ml): move UnifiedTrainable to ml-core + delete 6K dead deployment code

2026-03-08 15:17:22 +01:00

diffusion

fix: wire real GPU backprop + SPSA gradients, fix checkpoint loading, eliminate candle from examples

2026-03-20 15:41:42 +01:00

dqn

feat(bf16): FULL WORKSPACE COMPILES — zero F32 on GPU 🎉🎉🎉

2026-03-28 02:28:58 +01:00

ensemble

fix: update smoke tests for 9-action branching DQN (was 5-action)

2026-03-25 21:37:09 +01:00

evaluation

refactor(ml): split monolith into 9 sub-crates + delete dead code

2026-03-08 15:17:22 +01:00

explainability

refactor(ml): extract regime-detection, explainability, paper-trading

2026-03-08 15:17:22 +01:00

features

audit: annotate all remaining to_vec/memcpy_dtoh — categorized 158 sites

2026-03-18 19:36:41 +01:00

flash_attention

fix(clippy): ZERO errors across entire workspace — CI ready

2026-03-19 01:04:09 +01:00

gpu_benchmarks

…

hyperopt

feat: mbp10_data_dir and trades_data_dir are unconditional (no Option)

2026-04-01 01:24:30 +02:00

integration

audit: annotate all remaining to_vec/memcpy_dtoh — categorized 158 sites

2026-03-18 19:36:41 +01:00

kan

fix: wire real GPU backprop + SPSA gradients, fix checkpoint loading, eliminate candle from examples

2026-03-20 15:41:42 +01:00

labeling

refactor(ml): split monolith into 9 sub-crates + delete dead code

2026-03-08 15:17:22 +01:00

liquid

fix: wire real GPU backprop + SPSA gradients, fix checkpoint loading, eliminate candle from examples

2026-03-20 15:41:42 +01:00

mamba

fix: wire real GPU backprop + SPSA gradients, fix checkpoint loading, eliminate candle from examples

2026-03-20 15:41:42 +01:00

microstructure

refactor(ml): split monolith into 9 sub-crates + delete dead code

2026-03-08 15:17:22 +01:00

model_registry

refactor(cuda): eliminate all CPU fallbacks — CUDA mandatory across ML stack

2026-03-16 21:01:28 +01:00

observability

refactor(ml): extract observability, stress-testing, security into sub-crates

2026-03-08 15:17:22 +01:00

paper_trading

refactor(ml): extract regime-detection, explainability, paper-trading

2026-03-08 15:17:22 +01:00

ppo

fix(clippy): ZERO errors across entire workspace — CI ready

2026-03-19 01:04:09 +01:00

regime

refactor(ml): split monolith into 9 sub-crates + delete dead code

2026-03-08 15:17:22 +01:00

regime_detection

refactor(ml): extract regime-detection, explainability, paper-trading

2026-03-08 15:17:22 +01:00

registry

…

risk

refactor(ml): split monolith into 9 sub-crates + delete dead code

2026-03-08 15:17:22 +01:00

security

refactor(ml): extract observability, stress-testing, security into sub-crates

2026-03-08 15:17:22 +01:00

stress_testing

refactor(ml): extract observability, stress-testing, security into sub-crates

2026-03-08 15:17:22 +01:00

tests

fix(ml): BF16/F32 boundary alignment, GPU-resident ops across all ML crates

2026-03-15 11:59:31 +01:00

tft

fix: resolve all CI test failures — TFT, DQN pipeline, benchmarks

2026-03-21 09:09:51 +01:00

tgnn

fix: wire real GPU backprop + SPSA gradients, fix checkpoint loading, eliminate candle from examples

2026-03-20 15:41:42 +01:00

tlob

fix: wire real GPU backprop + SPSA gradients, fix checkpoint loading, eliminate candle from examples

2026-03-20 15:41:42 +01:00

trainers

fix: remove VRAM batch floor — batch_size >128 hangs CUDA graph capture

2026-04-02 03:57:19 +02:00

training

audit: annotate all remaining to_vec/memcpy_dtoh — categorized 158 sites

2026-03-18 19:36:41 +01:00

transformers

fix(ml): migrate all test code to GPU types — 125 compile errors fixed

2026-03-18 14:47:33 +01:00

universe

refactor(ml): extract universe, backtesting, asset-selection into sub-crates

2026-03-08 15:17:22 +01:00

validation

feat: DQN gems — spectral decoupling, manifold mixup, regime replay, family hyperopt

2026-03-31 00:39:23 +02:00

xlstm

fix: wire real GPU backprop + SPSA gradients, fix checkpoint loading, eliminate candle from examples

2026-03-20 15:41:42 +01:00

.gitignore

…

batch_processing.rs

refactor(ml): move core types to ml-core + dedup MLError (5a)

2026-03-08 15:13:41 +01:00

benchmarks.rs

feat: WORKSPACE COMPILES — zero candle, zero errors, GPU-native

2026-03-18 13:31:35 +01:00

bridge.rs

…

data_loader.rs

fix: data_loader finds futures-baseline + mamba2 benchmark #[ignore]

2026-03-20 11:38:52 +01:00

examples.rs

audit: annotate all remaining to_vec/memcpy_dtoh — categorized 158 sites

2026-03-18 19:36:41 +01:00

feature_cache.rs

fix: cache key uses filenames only — CWD-independent

2026-04-01 09:02:16 +02:00

fxcache.rs

feat: fxcache stores per-bar timestamps for walk-forward windowing

2026-04-01 19:29:41 +02:00

inference_validator.rs

fix: wire real GPU backprop + SPSA gradients, fix checkpoint loading, eliminate candle from examples

2026-03-20 15:41:42 +01:00

inference.rs

chore: remove last 4 candle references from ml crate (comments only)

2026-03-20 01:05:44 +01:00

lib.rs

feat: fxcache module — flat binary feature cache format (f64 + bf16)

2026-03-31 23:45:22 +02:00

model_factory.rs

…

model_loader_integration.rs

fix: tune log levels across workspace — demote noisy warn to debug/trace

2026-03-14 11:35:15 +01:00

model_registry.rs

fix: unignore 2 PostgreSQL model registry tests — docker running

2026-03-18 17:20:23 +01:00

portfolio_transformer.rs

fix(ml): migrate remaining files — hyperopt, validation, data_loaders, inference

2026-03-18 09:57:24 +01:00

preprocessing.rs

refactor(ml): eliminate candle from all 104 src files

2026-03-18 00:24:35 +01:00

training_pipeline.rs

fix(clippy): ZERO errors across entire workspace — CI ready

2026-03-19 01:04:09 +01:00

training_profile.rs

feat: mbp10_data_dir and trades_data_dir are unconditional (no Option)

2026-04-01 01:24:30 +02:00

training.rs

…

walk_forward.rs

feat: add curriculum learning with ADX-based difficulty scoring to walk-forward windows

2026-03-23 12:54:01 +01:00