foxhunt/docs/plans/2026-03-01-gpu-max-performance-phase2-design.md

GPU Max Performance Phase 2 Design (L4 → H100)

Goal: Maximize GPU utilization from ~40% (post-Phase 1) to 80%+ across all GPU tiers (L4 24GB, L40S 48GB, A100 40/80GB, H100 80GB). Covers both training throughput and inference latency.

Prior work (Phase 1): BF16 mixed precision (DQN+PPO), dynamic hidden_dim_base, data caching via Arc, batch experience storage, PPO GAE CPU path. All 5 wired end-to-end, verified by quality audit.

Framework: Candle v0.9.1 (cuBLAS + cuDNN, CUDA streams via cudarc 0.17.3).

---

1. Dynamic Batch Size Per GPU

Problem: DQN hard-caps batch_size at 230 (RTX 3050 Ti legacy). PPO defaults to 64. Neither adapts to available VRAM.

Design: At trainer init, query HardwareBudget::detect() and compute device-specific batch size. Formula: (gpu_vram_mb - model_overhead_mb) / mb_per_sample, clamped to [64, 8192].

GPU	VRAM	DQN batch	PPO batch
RTX 3050 Ti	4GB	230	64
L4	24GB	2048	512
L40S	48GB	4096	1024
A100 80GB	80GB	8192	2048
H100	80GB	8192	2048

Files: trainers/dqn/trainer.rs, trainers/ppo.rs, hyperopt/traits.rs

Impact: 8-20% throughput gain on H100 (larger batches = better GPU utilization).

---

2. Tensor Core Dimension Alignment

Problem: state_dim = 54 (51 market + 3 portfolio). Not a multiple of 8 — tensor cores on Ampere/Hopper can't fire optimally for the first Linear layer.

Design: Pad state_dim from 54 to 56 by appending 2 zero features during tensor construction in the CUDA pipeline's upload() and get_batch() methods. Hidden dims already default to powers of 2 (256, 128, 64). Enforce rounding to nearest multiple of 8 in HardwareBudget::max_hidden_dim_base().

Files: cuda_pipeline/mod.rs, hyperopt/traits.rs

Impact: 2-5% compute throughput (ensures cuBLAS selects tensor core paths).

---

3. Inference Without Gradient Tape

Problem: evaluate_baseline.rs runs forward passes but Candle still builds the computation graph. Wastes ~2x VRAM on activation caches that are never used.

Design: Add Tensor::detach() on forward pass outputs in evaluation and inference code. Candle has no torch.no_grad() equivalent, but detaching outputs prevents gradient accumulation. Key: don't call .backward() during inference — the VRAM savings come from not storing intermediate activations for a backward pass that never happens.

Files: examples/evaluate_baseline.rs, inference.rs, ppo/trainable_adapter.rs

Impact: ~2x VRAM savings during inference, enabling larger batch sizes for evaluation.

---

4. CUDA Stream Double-Buffering

Problem: Data upload is synchronous. GPU sits idle while CPU prepares the next batch.

Design: Use existing cuda_dev.cuda_stream() (already accessed in 3 trainer files) to create a secondary stream for data staging.

Stream A (compute): forward + backward on current batch
Stream B (staging):  async-copy next batch from host → device
Synchronize at batch boundary, swap streams

Wrap this pattern in a DoubleBufferedLoader struct that holds two pre-allocated GPU buffers and alternates between them.

Files: cuda_pipeline/mod.rs (new DoubleBufferedLoader), trainers/dqn/trainer.rs, trainers/ppo.rs

Impact: 5-15% training throughput (overlap I/O with compute).

---

5. Epoch Data Prefetch

Problem: Feature extraction and normalization happen synchronously before each epoch. CPU work blocks GPU.

Design: Spawn a background tokio::spawn task that computes next epoch's features/normalization stats while the current epoch trains on GPU. Use tokio::sync::oneshot channel to pass pre-computed data to the next epoch.

Epoch N:   GPU trains  ─────────────────────>
           CPU prefetch (epoch N+1 features) ──>
Epoch N+1: GPU trains (data ready)  ─────────>

Files: trainers/dqn/trainer.rs, trainers/ppo.rs

Impact: 5-12% training throughput (hidden CPU-side latency behind GPU compute).

---

6. INT8 Quantized Inference

Problem: 28 quantization files exist in crates/ml/ (INT8/INT4, QAT, quantized attention/GRN/LSTM). But QuantizedTFT::from_fp32_model() has empty scales and zero-points. The skeleton is built, the wiring is missing.

Design: Wire existing quantization infrastructure:

1. Post-training quantization: After training completes, compute per-layer min/max from weight tensors to derive scale and zero-point.
2. Quantized checkpoint: Store INT8 weights + scale/zero-point metadata alongside FP32 checkpoints.
3. Quantized inference: Load quantized model, use INT8 matmul for forward passes.
4. Priority order: TFT first (most complete skeleton), then DQN, then PPO.

Files: transformers/quantization.rs, memory_optimization/quantization.rs, tft/hft_optimizations.rs

Impact: 2-4x inference throughput + 4x memory savings for inference workloads.

---

7. Memory Pinning for Host-Device Transfers

Problem: CPU to GPU transfers use pageable memory (Tensor::from_vec() allocates from standard heap). DMA engine can't use the fast pinned-memory path.

Design: For the CUDA pipeline's large transfers (upload() method), use cudarc's htod_copy() which internally uses pinned memory. For small per-step tensors (portfolio features, 3-dim), pre-allocate a pinned buffer pool and reuse it across steps.

Files: cuda_pipeline/mod.rs

Impact: 2-5% host-device transfer latency reduction. Marginal for pre-uploaded data, meaningful for per-step portfolio features.

---

8. Custom CUDA Kernel Optimization

Problem: Two custom kernels (dqn_experience_kernel.cu, ppo_experience_kernel.cu) exist but haven't been optimized for L4/H100 SM counts.

Design: Audit and optimize:

1. Block/grid sizing: Auto-configure based on device SM count (L4: 58 SMs, L40S: 142, A100: 108, H100: 132).
2. Shared memory: Use shared memory for reduction operations in experience collection.
3. Fused ops: Identify sequential kernel launches that can be merged.
4. Occupancy: Target 75%+ occupancy by tuning threads-per-block.

Query SM count at init via cudarc::driver::CudaDevice::attribute().

Files: cuda_kernels/dqn_experience_kernel.cu, cuda_kernels/ppo_experience_kernel.cu, cuda_pipeline/mod.rs

Impact: 5-10% for custom kernel paths. cuBLAS ops already optimized.

---

Summary

#	Optimization	Effort	Impact	Training	Inference
1	Dynamic batch size	Low	8-20%	Y	Y
2	Dimension alignment	Low	2-5%	Y	Y
3	No-grad inference	Low	2x VRAM	N	Y
4	CUDA stream double-buffer	Medium	5-15%	Y	N
5	Epoch data prefetch	Medium	5-12%	Y	N
6	INT8 quantized inference	High	2-4x	N	Y
7	Memory pinning	Medium	2-5%	Y	Y
8	CUDA kernel optimization	High	5-10%	Y	N

Combined theoretical gain: 30-70% training throughput + 2-4x inference speedup.

Target GPU utilization: 60-80% sustained (from current ~40% post-Phase 1).

Non-goals: Multi-GPU (single GPU per pod, Kapsule quota=1), FP8 (unsupported by Candle).