169821b3da
8 optimizations: dynamic batch size, tensor core alignment, no-grad inference, CUDA stream double-buffering, epoch prefetch, INT8 quantized inference, memory pinning, CUDA kernel optimization. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
159 lines
7.1 KiB
Markdown
159 lines
7.1 KiB
Markdown
# GPU Max Performance Phase 2 Design (L4 → H100)
|
|
|
|
**Goal**: Maximize GPU utilization from ~40% (post-Phase 1) to 80%+ across all GPU tiers (L4 24GB, L40S 48GB, A100 40/80GB, H100 80GB). Covers both training throughput and inference latency.
|
|
|
|
**Prior work (Phase 1)**: BF16 mixed precision (DQN+PPO), dynamic hidden_dim_base, data caching via Arc, batch experience storage, PPO GAE CPU path. All 5 wired end-to-end, verified by quality audit.
|
|
|
|
**Framework**: Candle v0.9.1 (cuBLAS + cuDNN, CUDA streams via cudarc 0.17.3).
|
|
|
|
---
|
|
|
|
## 1. Dynamic Batch Size Per GPU
|
|
|
|
**Problem**: DQN hard-caps batch_size at 230 (RTX 3050 Ti legacy). PPO defaults to 64. Neither adapts to available VRAM.
|
|
|
|
**Design**: At trainer init, query `HardwareBudget::detect()` and compute device-specific batch size. Formula: `(gpu_vram_mb - model_overhead_mb) / mb_per_sample`, clamped to `[64, 8192]`.
|
|
|
|
| GPU | VRAM | DQN batch | PPO batch |
|
|
|---|---|---|---|
|
|
| RTX 3050 Ti | 4GB | 230 | 64 |
|
|
| L4 | 24GB | 2048 | 512 |
|
|
| L40S | 48GB | 4096 | 1024 |
|
|
| A100 80GB | 80GB | 8192 | 2048 |
|
|
| H100 | 80GB | 8192 | 2048 |
|
|
|
|
**Files**: `trainers/dqn/trainer.rs`, `trainers/ppo.rs`, `hyperopt/traits.rs`
|
|
|
|
**Impact**: 8-20% throughput gain on H100 (larger batches = better GPU utilization).
|
|
|
|
---
|
|
|
|
## 2. Tensor Core Dimension Alignment
|
|
|
|
**Problem**: `state_dim = 54` (51 market + 3 portfolio). Not a multiple of 8 — tensor cores on Ampere/Hopper can't fire optimally for the first Linear layer.
|
|
|
|
**Design**: Pad state_dim from 54 to 56 by appending 2 zero features during tensor construction in the CUDA pipeline's `upload()` and `get_batch()` methods. Hidden dims already default to powers of 2 (256, 128, 64). Enforce rounding to nearest multiple of 8 in `HardwareBudget::max_hidden_dim_base()`.
|
|
|
|
**Files**: `cuda_pipeline/mod.rs`, `hyperopt/traits.rs`
|
|
|
|
**Impact**: 2-5% compute throughput (ensures cuBLAS selects tensor core paths).
|
|
|
|
---
|
|
|
|
## 3. Inference Without Gradient Tape
|
|
|
|
**Problem**: `evaluate_baseline.rs` runs forward passes but Candle still builds the computation graph. Wastes ~2x VRAM on activation caches that are never used.
|
|
|
|
**Design**: Add `Tensor::detach()` on forward pass outputs in evaluation and inference code. Candle has no `torch.no_grad()` equivalent, but detaching outputs prevents gradient accumulation. Key: don't call `.backward()` during inference — the VRAM savings come from not storing intermediate activations for a backward pass that never happens.
|
|
|
|
**Files**: `examples/evaluate_baseline.rs`, `inference.rs`, `ppo/trainable_adapter.rs`
|
|
|
|
**Impact**: ~2x VRAM savings during inference, enabling larger batch sizes for evaluation.
|
|
|
|
---
|
|
|
|
## 4. CUDA Stream Double-Buffering
|
|
|
|
**Problem**: Data upload is synchronous. GPU sits idle while CPU prepares the next batch.
|
|
|
|
**Design**: Use existing `cuda_dev.cuda_stream()` (already accessed in 3 trainer files) to create a secondary stream for data staging.
|
|
|
|
```
|
|
Stream A (compute): forward + backward on current batch
|
|
Stream B (staging): async-copy next batch from host → device
|
|
Synchronize at batch boundary, swap streams
|
|
```
|
|
|
|
Wrap this pattern in a `DoubleBufferedLoader` struct that holds two pre-allocated GPU buffers and alternates between them.
|
|
|
|
**Files**: `cuda_pipeline/mod.rs` (new `DoubleBufferedLoader`), `trainers/dqn/trainer.rs`, `trainers/ppo.rs`
|
|
|
|
**Impact**: 5-15% training throughput (overlap I/O with compute).
|
|
|
|
---
|
|
|
|
## 5. Epoch Data Prefetch
|
|
|
|
**Problem**: Feature extraction and normalization happen synchronously before each epoch. CPU work blocks GPU.
|
|
|
|
**Design**: Spawn a background `tokio::spawn` task that computes next epoch's features/normalization stats while the current epoch trains on GPU. Use `tokio::sync::oneshot` channel to pass pre-computed data to the next epoch.
|
|
|
|
```
|
|
Epoch N: GPU trains ─────────────────────>
|
|
CPU prefetch (epoch N+1 features) ──>
|
|
Epoch N+1: GPU trains (data ready) ─────────>
|
|
```
|
|
|
|
**Files**: `trainers/dqn/trainer.rs`, `trainers/ppo.rs`
|
|
|
|
**Impact**: 5-12% training throughput (hidden CPU-side latency behind GPU compute).
|
|
|
|
---
|
|
|
|
## 6. INT8 Quantized Inference
|
|
|
|
**Problem**: 28 quantization files exist in `crates/ml/` (INT8/INT4, QAT, quantized attention/GRN/LSTM). But `QuantizedTFT::from_fp32_model()` has empty scales and zero-points. The skeleton is built, the wiring is missing.
|
|
|
|
**Design**: Wire existing quantization infrastructure:
|
|
|
|
1. **Post-training quantization**: After training completes, compute per-layer min/max from weight tensors to derive scale and zero-point.
|
|
2. **Quantized checkpoint**: Store INT8 weights + scale/zero-point metadata alongside FP32 checkpoints.
|
|
3. **Quantized inference**: Load quantized model, use INT8 matmul for forward passes.
|
|
4. **Priority order**: TFT first (most complete skeleton), then DQN, then PPO.
|
|
|
|
**Files**: `transformers/quantization.rs`, `memory_optimization/quantization.rs`, `tft/hft_optimizations.rs`
|
|
|
|
**Impact**: 2-4x inference throughput + 4x memory savings for inference workloads.
|
|
|
|
---
|
|
|
|
## 7. Memory Pinning for Host-Device Transfers
|
|
|
|
**Problem**: CPU to GPU transfers use pageable memory (`Tensor::from_vec()` allocates from standard heap). DMA engine can't use the fast pinned-memory path.
|
|
|
|
**Design**: For the CUDA pipeline's large transfers (`upload()` method), use cudarc's `htod_copy()` which internally uses pinned memory. For small per-step tensors (portfolio features, 3-dim), pre-allocate a pinned buffer pool and reuse it across steps.
|
|
|
|
**Files**: `cuda_pipeline/mod.rs`
|
|
|
|
**Impact**: 2-5% host-device transfer latency reduction. Marginal for pre-uploaded data, meaningful for per-step portfolio features.
|
|
|
|
---
|
|
|
|
## 8. Custom CUDA Kernel Optimization
|
|
|
|
**Problem**: Two custom kernels (`dqn_experience_kernel.cu`, `ppo_experience_kernel.cu`) exist but haven't been optimized for L4/H100 SM counts.
|
|
|
|
**Design**: Audit and optimize:
|
|
|
|
1. **Block/grid sizing**: Auto-configure based on device SM count (L4: 58 SMs, L40S: 142, A100: 108, H100: 132).
|
|
2. **Shared memory**: Use shared memory for reduction operations in experience collection.
|
|
3. **Fused ops**: Identify sequential kernel launches that can be merged.
|
|
4. **Occupancy**: Target 75%+ occupancy by tuning threads-per-block.
|
|
|
|
Query SM count at init via `cudarc::driver::CudaDevice::attribute()`.
|
|
|
|
**Files**: `cuda_kernels/dqn_experience_kernel.cu`, `cuda_kernels/ppo_experience_kernel.cu`, `cuda_pipeline/mod.rs`
|
|
|
|
**Impact**: 5-10% for custom kernel paths. cuBLAS ops already optimized.
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
| # | Optimization | Effort | Impact | Training | Inference |
|
|
|---|---|---|---|---|---|
|
|
| 1 | Dynamic batch size | Low | 8-20% | Y | Y |
|
|
| 2 | Dimension alignment | Low | 2-5% | Y | Y |
|
|
| 3 | No-grad inference | Low | 2x VRAM | N | Y |
|
|
| 4 | CUDA stream double-buffer | Medium | 5-15% | Y | N |
|
|
| 5 | Epoch data prefetch | Medium | 5-12% | Y | N |
|
|
| 6 | INT8 quantized inference | High | 2-4x | N | Y |
|
|
| 7 | Memory pinning | Medium | 2-5% | Y | Y |
|
|
| 8 | CUDA kernel optimization | High | 5-10% | Y | N |
|
|
|
|
**Combined theoretical gain**: 30-70% training throughput + 2-4x inference speedup.
|
|
|
|
**Target GPU utilization**: 60-80% sustained (from current ~40% post-Phase 1).
|
|
|
|
**Non-goals**: Multi-GPU (single GPU per pod, Kapsule quota=1), FP8 (unsupported by Candle).
|