fix(cuda): shmem tile overflow → CUDA_ERROR_ILLEGAL_ADDRESS on RTX 3050
Root cause: shmem_max_in_dim only included trunk dims (state_dim, shared_h1, shared_h2) but not head dims (value_h, adv_h). When hidden_dim_base=32 made the trunk narrow while heads stayed at 128, the BF16 weight tile for branch output (255×128=32640 BF16 elements) overflowed the shared memory region (12288 BF16 elements). On H100 the overflow landed in unused-but-mapped hardware shmem (silent corruption). On RTX 3050 (48KB physical shmem) it hit unmapped memory → CUDA_ERROR_ILLEGAL_ADDRESS. Changes: - gpu_dqn_trainer.rs: shmem_max_in_dim includes value_h/adv_h - Remove all #[ignore] from smoke tests (feature_coverage, training_stability, gpu_residency) - Smoke tests use real .dbn data from test_data/ (hard error if missing) - Remove synthetic_data() fallback — no fake data in tests - GPU-direct DtoD training path (train_step_gpu, FusedTrainScalars) - GPU-native PER priority update kernel (zero CPU readback) - IQN dual-head integration (gpu_iqn_head.rs) - BF16 dtype fixes across 6 model adapters - Hyperopt 30D→31D (iqn_lambda) - portfolio_transformer: unconditional BF16 (remove dead CPU branches) - liquid/adapter: all tests use Cuda(0) directly - Fix pre-existing gpu_kernel_parity_test.rs (stale args) - Fix pre-existing evaluate_baseline.rs (removed fields) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -82,12 +82,30 @@ LEAK_PATTERNS=(
|
||||
# They add maintenance burden and hide the real code path.
|
||||
'#\[cfg\(not\(feature\s*=\s*"cuda"\)\)\]'
|
||||
|
||||
# ─── CPU-side batch data mutation (GPU data wrongly processed on CPU) ───
|
||||
# Functions taking mutable Vec<f32> batch arrays indicate CPU-side processing
|
||||
# that should be a GPU kernel or Candle tensor ops.
|
||||
# Caught: CPU HER relabeling, CPU normalization, CPU reward shaping, etc.
|
||||
'mut states: Vec<f32>'
|
||||
'mut next_states: Vec<f32>'
|
||||
'mut rewards: Vec<f32>'
|
||||
|
||||
# ─── Candle tensor wrapping in per-step hot path ───
|
||||
# Creating Candle Tensors from CudaSlice in the training loop adds
|
||||
# Rust-side allocation + dispatch overhead. Use raw CUDA kernels instead.
|
||||
# Tensor::new() creates a CPU scalar → GPU copy (HtoD for 4 bytes).
|
||||
# CudaStorage::wrap_cuda_slice takes ownership — blocks reuse patterns.
|
||||
'CudaStorage::wrap_cuda_slice'
|
||||
|
||||
# NOTE: Tensor::from_vec/from_slice are NOT flagged — they are CPU→GPU
|
||||
# uploads (correct direction). Inner-loop abuse is caught by code review.
|
||||
#
|
||||
# NOTE: memcpy_dtoh/htod and .synchronize() are not checked here —
|
||||
# they are CUDA implementation primitives in cuda_pipeline/.
|
||||
# Audit those separately with: scripts/cuda-perf-audit.sh
|
||||
#
|
||||
# NOTE: storage_and_layout() is NOT flagged — it's a zero-cost pointer
|
||||
# unwrap for extracting CudaSlice refs from existing Tensors.
|
||||
)
|
||||
|
||||
# ── Excluded paths ────────────────────────────────────────────────────
|
||||
@@ -106,6 +124,9 @@ EXCLUDE_PATHS=(
|
||||
"ml-supervised/src/tft/hft_optimizations.rs"
|
||||
# evaluate_cpu() is slow fallback before precompute_gpu_grid(); GPU path exists
|
||||
"ml-supervised/src/kan/spline.rs"
|
||||
# SearchSorted/CumsumOp CustomOp2 impls must return CudaStorage (Candle API boundary)
|
||||
# Runs once per PER sampling batch, not inside per-step training kernel loop
|
||||
"ml-dqn/src/gpu_replay_buffer.rs"
|
||||
)
|
||||
|
||||
# ── Test module exclusion ─────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user