foxhunt/docs/superpowers/specs/2026-04-06-gpu-native-validation-design.md

GPU-Native Validation + H100 Performance Optimizations

Goal

Replace the CPU validation path with the existing GpuBacktestEvaluator, delete all dead CPU code, and apply remaining performance optimizations to push epoch time from 15.6s toward <5s.

Architecture

The training loop's compute_validation_loss() in metrics.rs currently:

1. Assembles states on CPU (Vec<f32>)
2. Uploads to GPU (clone_htod_f32_to_bf16)
3. Runs cuBLAS forward (compute_q_values)
4. Downloads Q-values to CPU (dtoh_bf16_to_f32) — 2s sync
5. CPU argmax over Q-values
6. CPU PnL computation
7. CPU Sharpe ratio

Steps 4-7 are pure CPU with a mandatory GPU→CPU sync. The GpuBacktestEvaluator (2086 lines, gpu_backtest_evaluator.rs) already does all of this on GPU — used by hyperopt adapters but NOT by the training loop.

Changes

1. Wire GpuBacktestEvaluator into Training Loop

• Add gpu_evaluator: Option<GpuBacktestEvaluator> field to DQNTrainer
• Initialize in constructor from validation data + DqnBacktestConfig
• Replace compute_validation_loss() body: call gpu_evaluator.evaluate_dqn_graphed() with current weights, extract WindowMetrics.sharpe
• The evaluator pre-uploads validation data to GPU once at construction (not per epoch)

2. Delete Dead CPU Validation Code

Remove from DQNTrainer:

• val_features_gpu: Option<Vec<f32>> field + lazy init
• val_closes_gpu: Option<(Vec<f32>, Vec<f32>)> field + lazy init
• val_ofi_gpu: Option<Vec<f32>> field + lazy init
• CPU state assembly loop (lines 485-509 in metrics.rs)
• clone_htod_f32_to_bf16 call
• dtoh_bf16_to_f32 readback
• CPU argmax loop (lines 535-555)
• CPU Sharpe computation (lines 568-584)
• All associated use imports

3. BF16 Backward TF32

Change batched_backward.rs:

• 2 GEMMs from CUBLAS_COMPUTE_32F → CUBLAS_COMPUTE_32F_FAST_TF32
• Both activation gradient (BF16→BF16) and weight gradient (BF16→F32) paths
• TF32's 10-bit mantissa is sufficient for gradients (Adam momentum smooths noise)

4. Training Loop Optimizations (already in working tree)

• Vaccine sample only every 10th step (was: every step, wasting 43ms)
• Training guard every 5th step (was: every step, 12ms overhead)
• Single write lock for sample+step (was: read→write cycling, 43ms contention)
• Causal intervention interval 10→50
• Remove all H100_DIAG tracing::info

Expected Impact

Optimization	Per-epoch savings
GPU-native validation (2s CPU → 50ms GPU)	-1.95s
BF16 backward TF32 (2-3x GEMM throughput)	-3-5s
Single-sample + guard + lock	-3.6s
Total estimated	~8-10s
Target epoch time	~5-7s

Testing

• Smoke tests: verify Sharpe computation matches between old CPU path and new GPU evaluator (within BF16 tolerance)
• H100 deployment: confirm epoch timing improvement
• Loss convergence: verify training quality unchanged