Root cause of NaN at step 22: cuBLAS out-of-bounds reads in the GPU
experience collector and training forward pass, caused by buffer
dimension mismatches that produced garbage states → garbage Q-values →
NaN loss.
Bugs fixed:
1. GpuDqnTrainConfig::default() had bottleneck_dim=2, market_dim=42 —
collector computed s1_input_dim=40 instead of state_dim=80, cuBLAS
read 2x past W1 weight buffer (10KB OOB per SGEMM)
2. batch_states allocated with state_dim=80 but cuBLAS ldb=128 (CUTLASS
K-tile alignment) — 48 bf16/f32 values read past end per row
3. Validation state tensor had 59 elements/row (42+3+8+6) instead of
128 (state_dim_padded) — missing portfolio/MTF features + padding
4. CUTLASS K-tile overread on all hidden activation buffers (K=64 but
tile=128) — added 128-element padding to 14 bf16 buffers
5. init_from_fxcache never set self.ofi_features — collector OFI buffer
was 1-element placeholder, state_gather kernel read OOB
6. collect_gpu_experiences_slices missing *2 counterfactual multiplier —
half of collected experiences never inserted into PER
Dead code removed (~1228 lines):
- train_with_data_full_loop (old Vec<(FeatureVector, Vec<f64>)> loop)
- init_gpu_data, init_gpu_raw_buffers, collect_gpu_experiences,
run_training_steps (old helpers only called by above)
- train_with_preloaded_data, train_with_shared_data (old hyperopt APIs)
- train() and train_walk_forward() now convert to fixed-size arrays
and route through init_from_fxcache + train_fold_from_slices
Other fixes:
- precompute_features defaults output-dir to sibling of data-dir
- workspace_root() + feature_cache_dir() helpers for reliable path
resolution (works from any cwd, with/without CARGO_MANIFEST_DIR)
- Production smoketest uses fxcache (no slow DBN parsing)
- compute-sanitizer: 0 errors (was 2539)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
a448e03e64