fix(cuda): shmem tile overflow → CUDA_ERROR_ILLEGAL_ADDRESS on RTX 3050

Root cause: shmem_max_in_dim only included trunk dims (state_dim,
shared_h1, shared_h2) but not head dims (value_h, adv_h). When
hidden_dim_base=32 made the trunk narrow while heads stayed at 128,
the BF16 weight tile for branch output (255×128=32640 BF16 elements)
overflowed the shared memory region (12288 BF16 elements). On H100
the overflow landed in unused-but-mapped hardware shmem (silent
corruption). On RTX 3050 (48KB physical shmem) it hit unmapped
memory → CUDA_ERROR_ILLEGAL_ADDRESS.

Changes:
- gpu_dqn_trainer.rs: shmem_max_in_dim includes value_h/adv_h
- Remove all #[ignore] from smoke tests (feature_coverage,
  training_stability, gpu_residency)
- Smoke tests use real .dbn data from test_data/ (hard error if missing)
- Remove synthetic_data() fallback — no fake data in tests
- GPU-direct DtoD training path (train_step_gpu, FusedTrainScalars)
- GPU-native PER priority update kernel (zero CPU readback)
- IQN dual-head integration (gpu_iqn_head.rs)
- BF16 dtype fixes across 6 model adapters
- Hyperopt 30D→31D (iqn_lambda)
- portfolio_transformer: unconditional BF16 (remove dead CPU branches)
- liquid/adapter: all tests use Cuda(0) directly
- Fix pre-existing gpu_kernel_parity_test.rs (stale args)
- Fix pre-existing evaluate_baseline.rs (removed fields)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-03-17 08:34:51 +01:00
parent 12b9eb2436
commit ad28482a93
35 changed files with 2730 additions and 359 deletions

View File

@@ -82,12 +82,30 @@ LEAK_PATTERNS=(
# They add maintenance burden and hide the real code path.
'#\[cfg\(not\(feature\s*=\s*"cuda"\)\)\]'
# ─── CPU-side batch data mutation (GPU data wrongly processed on CPU) ───
# Functions taking mutable Vec<f32> batch arrays indicate CPU-side processing
# that should be a GPU kernel or Candle tensor ops.
# Caught: CPU HER relabeling, CPU normalization, CPU reward shaping, etc.
'mut states: Vec<f32>'
'mut next_states: Vec<f32>'
'mut rewards: Vec<f32>'
# ─── Candle tensor wrapping in per-step hot path ───
# Creating Candle Tensors from CudaSlice in the training loop adds
# Rust-side allocation + dispatch overhead. Use raw CUDA kernels instead.
# Tensor::new() creates a CPU scalar → GPU copy (HtoD for 4 bytes).
# CudaStorage::wrap_cuda_slice takes ownership — blocks reuse patterns.
'CudaStorage::wrap_cuda_slice'
# NOTE: Tensor::from_vec/from_slice are NOT flagged — they are CPU→GPU
# uploads (correct direction). Inner-loop abuse is caught by code review.
#
# NOTE: memcpy_dtoh/htod and .synchronize() are not checked here —
# they are CUDA implementation primitives in cuda_pipeline/.
# Audit those separately with: scripts/cuda-perf-audit.sh
#
# NOTE: storage_and_layout() is NOT flagged — it's a zero-cost pointer
# unwrap for extracting CudaSlice refs from existing Tensors.
)
# ── Excluded paths ────────────────────────────────────────────────────
@@ -106,6 +124,9 @@ EXCLUDE_PATHS=(
"ml-supervised/src/tft/hft_optimizations.rs"
# evaluate_cpu() is slow fallback before precompute_gpu_grid(); GPU path exists
"ml-supervised/src/kan/spline.rs"
# SearchSorted/CumsumOp CustomOp2 impls must return CudaStorage (Candle API boundary)
# Runs once per PER sampling batch, not inside per-step training kernel loop
"ml-dqn/src/gpu_replay_buffer.rs"
)
# ── Test module exclusion ─────────────────────────────────────────────