# GPU-Native Validation + H100 Performance Optimizations

## Goal

Replace the CPU validation path with the existing `GpuBacktestEvaluator`, delete all dead CPU code, and apply remaining performance optimizations to push epoch time from 15.6s toward <5s.

## Architecture

The training loop's `compute_validation_loss()` in `metrics.rs` currently:
1. Assembles states on CPU (`Vec<f32>`)
2. Uploads to GPU (`clone_htod_f32_to_bf16`)
3. Runs cuBLAS forward (`compute_q_values`)
4. Downloads Q-values to CPU (`dtoh_bf16_to_f32`) — **2s sync**
5. CPU argmax over Q-values
6. CPU PnL computation
7. CPU Sharpe ratio

Steps 4-7 are pure CPU with a mandatory GPU→CPU sync. The `GpuBacktestEvaluator` (2086 lines, `gpu_backtest_evaluator.rs`) already does all of this on GPU — used by hyperopt adapters but NOT by the training loop.

### Changes

#### 1. Wire GpuBacktestEvaluator into Training Loop

- Add `gpu_evaluator: Option<GpuBacktestEvaluator>` field to `DQNTrainer`
- Initialize in constructor from validation data + `DqnBacktestConfig`
- Replace `compute_validation_loss()` body: call `gpu_evaluator.evaluate_dqn_graphed()` with current weights, extract `WindowMetrics.sharpe`
- The evaluator pre-uploads validation data to GPU once at construction (not per epoch)

#### 2. Delete Dead CPU Validation Code

Remove from `DQNTrainer`:
- `val_features_gpu: Option<Vec<f32>>` field + lazy init
- `val_closes_gpu: Option<(Vec<f32>, Vec<f32>)>` field + lazy init
- `val_ofi_gpu: Option<Vec<f32>>` field + lazy init
- CPU state assembly loop (lines 485-509 in metrics.rs)
- `clone_htod_f32_to_bf16` call
- `dtoh_bf16_to_f32` readback
- CPU argmax loop (lines 535-555)
- CPU Sharpe computation (lines 568-584)
- All associated `use` imports

#### 3. BF16 Backward TF32

Change `batched_backward.rs`:
- 2 GEMMs from `CUBLAS_COMPUTE_32F` → `CUBLAS_COMPUTE_32F_FAST_TF32`
- Both activation gradient (BF16→BF16) and weight gradient (BF16→F32) paths
- TF32's 10-bit mantissa is sufficient for gradients (Adam momentum smooths noise)

#### 4. Training Loop Optimizations (already in working tree)

- Vaccine sample only every 10th step (was: every step, wasting 43ms)
- Training guard every 5th step (was: every step, 12ms overhead)
- Single write lock for sample+step (was: read→write cycling, 43ms contention)
- Causal intervention interval 10→50
- Remove all H100_DIAG tracing::info

## Expected Impact

| Optimization | Per-epoch savings |
|---|---|
| GPU-native validation (2s CPU → 50ms GPU) | -1.95s |
| BF16 backward TF32 (2-3x GEMM throughput) | -3-5s |
| Single-sample + guard + lock | -3.6s |
| **Total estimated** | **~8-10s** |
| **Target epoch time** | **~5-7s** |

## Testing

- Smoke tests: verify Sharpe computation matches between old CPU path and new GPU evaluator (within BF16 tolerance)
- H100 deployment: confirm epoch timing improvement
- Loss convergence: verify training quality unchanged