# GPU Max Performance Phase 2 Design (L4 → H100) **Goal**: Maximize GPU utilization from ~40% (post-Phase 1) to 80%+ across all GPU tiers (L4 24GB, L40S 48GB, A100 40/80GB, H100 80GB). Covers both training throughput and inference latency. **Prior work (Phase 1)**: BF16 mixed precision (DQN+PPO), dynamic hidden_dim_base, data caching via Arc, batch experience storage, PPO GAE CPU path. All 5 wired end-to-end, verified by quality audit. **Framework**: Candle v0.9.1 (cuBLAS + cuDNN, CUDA streams via cudarc 0.17.3). --- ## 1. Dynamic Batch Size Per GPU **Problem**: DQN hard-caps batch_size at 230 (RTX 3050 Ti legacy). PPO defaults to 64. Neither adapts to available VRAM. **Design**: At trainer init, query `HardwareBudget::detect()` and compute device-specific batch size. Formula: `(gpu_vram_mb - model_overhead_mb) / mb_per_sample`, clamped to `[64, 8192]`. | GPU | VRAM | DQN batch | PPO batch | |---|---|---|---| | RTX 3050 Ti | 4GB | 230 | 64 | | L4 | 24GB | 2048 | 512 | | L40S | 48GB | 4096 | 1024 | | A100 80GB | 80GB | 8192 | 2048 | | H100 | 80GB | 8192 | 2048 | **Files**: `trainers/dqn/trainer.rs`, `trainers/ppo.rs`, `hyperopt/traits.rs` **Impact**: 8-20% throughput gain on H100 (larger batches = better GPU utilization). --- ## 2. Tensor Core Dimension Alignment **Problem**: `state_dim = 54` (51 market + 3 portfolio). Not a multiple of 8 — tensor cores on Ampere/Hopper can't fire optimally for the first Linear layer. **Design**: Pad state_dim from 54 to 56 by appending 2 zero features during tensor construction in the CUDA pipeline's `upload()` and `get_batch()` methods. Hidden dims already default to powers of 2 (256, 128, 64). Enforce rounding to nearest multiple of 8 in `HardwareBudget::max_hidden_dim_base()`. **Files**: `cuda_pipeline/mod.rs`, `hyperopt/traits.rs` **Impact**: 2-5% compute throughput (ensures cuBLAS selects tensor core paths). --- ## 3. Inference Without Gradient Tape **Problem**: `evaluate_baseline.rs` runs forward passes but Candle still builds the computation graph. Wastes ~2x VRAM on activation caches that are never used. **Design**: Add `Tensor::detach()` on forward pass outputs in evaluation and inference code. Candle has no `torch.no_grad()` equivalent, but detaching outputs prevents gradient accumulation. Key: don't call `.backward()` during inference — the VRAM savings come from not storing intermediate activations for a backward pass that never happens. **Files**: `examples/evaluate_baseline.rs`, `inference.rs`, `ppo/trainable_adapter.rs` **Impact**: ~2x VRAM savings during inference, enabling larger batch sizes for evaluation. --- ## 4. CUDA Stream Double-Buffering **Problem**: Data upload is synchronous. GPU sits idle while CPU prepares the next batch. **Design**: Use existing `cuda_dev.cuda_stream()` (already accessed in 3 trainer files) to create a secondary stream for data staging. ``` Stream A (compute): forward + backward on current batch Stream B (staging): async-copy next batch from host → device Synchronize at batch boundary, swap streams ``` Wrap this pattern in a `DoubleBufferedLoader` struct that holds two pre-allocated GPU buffers and alternates between them. **Files**: `cuda_pipeline/mod.rs` (new `DoubleBufferedLoader`), `trainers/dqn/trainer.rs`, `trainers/ppo.rs` **Impact**: 5-15% training throughput (overlap I/O with compute). --- ## 5. Epoch Data Prefetch **Problem**: Feature extraction and normalization happen synchronously before each epoch. CPU work blocks GPU. **Design**: Spawn a background `tokio::spawn` task that computes next epoch's features/normalization stats while the current epoch trains on GPU. Use `tokio::sync::oneshot` channel to pass pre-computed data to the next epoch. ``` Epoch N: GPU trains ─────────────────────> CPU prefetch (epoch N+1 features) ──> Epoch N+1: GPU trains (data ready) ─────────> ``` **Files**: `trainers/dqn/trainer.rs`, `trainers/ppo.rs` **Impact**: 5-12% training throughput (hidden CPU-side latency behind GPU compute). --- ## 6. INT8 Quantized Inference **Problem**: 28 quantization files exist in `crates/ml/` (INT8/INT4, QAT, quantized attention/GRN/LSTM). But `QuantizedTFT::from_fp32_model()` has empty scales and zero-points. The skeleton is built, the wiring is missing. **Design**: Wire existing quantization infrastructure: 1. **Post-training quantization**: After training completes, compute per-layer min/max from weight tensors to derive scale and zero-point. 2. **Quantized checkpoint**: Store INT8 weights + scale/zero-point metadata alongside FP32 checkpoints. 3. **Quantized inference**: Load quantized model, use INT8 matmul for forward passes. 4. **Priority order**: TFT first (most complete skeleton), then DQN, then PPO. **Files**: `transformers/quantization.rs`, `memory_optimization/quantization.rs`, `tft/hft_optimizations.rs` **Impact**: 2-4x inference throughput + 4x memory savings for inference workloads. --- ## 7. Memory Pinning for Host-Device Transfers **Problem**: CPU to GPU transfers use pageable memory (`Tensor::from_vec()` allocates from standard heap). DMA engine can't use the fast pinned-memory path. **Design**: For the CUDA pipeline's large transfers (`upload()` method), use cudarc's `htod_copy()` which internally uses pinned memory. For small per-step tensors (portfolio features, 3-dim), pre-allocate a pinned buffer pool and reuse it across steps. **Files**: `cuda_pipeline/mod.rs` **Impact**: 2-5% host-device transfer latency reduction. Marginal for pre-uploaded data, meaningful for per-step portfolio features. --- ## 8. Custom CUDA Kernel Optimization **Problem**: Two custom kernels (`dqn_experience_kernel.cu`, `ppo_experience_kernel.cu`) exist but haven't been optimized for L4/H100 SM counts. **Design**: Audit and optimize: 1. **Block/grid sizing**: Auto-configure based on device SM count (L4: 58 SMs, L40S: 142, A100: 108, H100: 132). 2. **Shared memory**: Use shared memory for reduction operations in experience collection. 3. **Fused ops**: Identify sequential kernel launches that can be merged. 4. **Occupancy**: Target 75%+ occupancy by tuning threads-per-block. Query SM count at init via `cudarc::driver::CudaDevice::attribute()`. **Files**: `cuda_kernels/dqn_experience_kernel.cu`, `cuda_kernels/ppo_experience_kernel.cu`, `cuda_pipeline/mod.rs` **Impact**: 5-10% for custom kernel paths. cuBLAS ops already optimized. --- ## Summary | # | Optimization | Effort | Impact | Training | Inference | |---|---|---|---|---|---| | 1 | Dynamic batch size | Low | 8-20% | Y | Y | | 2 | Dimension alignment | Low | 2-5% | Y | Y | | 3 | No-grad inference | Low | 2x VRAM | N | Y | | 4 | CUDA stream double-buffer | Medium | 5-15% | Y | N | | 5 | Epoch data prefetch | Medium | 5-12% | Y | N | | 6 | INT8 quantized inference | High | 2-4x | N | Y | | 7 | Memory pinning | Medium | 2-5% | Y | Y | | 8 | CUDA kernel optimization | High | 5-10% | Y | N | **Combined theoretical gain**: 30-70% training throughput + 2-4x inference speedup. **Target GPU utilization**: 60-80% sustained (from current ~40% post-Phase 1). **Non-goals**: Multi-GPU (single GPU per pod, Kapsule quota=1), FP8 (unsupported by Candle).