# GPU Blockers Validation — Design

**Date**: 2026-03-01
**Status**: Approved
**Goal**: Fix 5 blockers preventing GPU utilization from reaching 80%+ after initial dynamic-sizing changes

## Findings Summary

Post-implementation validation of the GPU dynamic-sizing changes (20 files, +353/-84) found 3 critical and 2 important blockers.

## Critical Fixes

### Fix 1: Wire BF16 Mixed Precision End-to-End

**Problem**: Detection works (`HardwareBudget::detect()` → `supports_bf16()`) but result stored in `_mixed_precision_detected` (unused). Chain broken at 3 points:
- `DQNConfig` has no `mixed_precision` field
- `DQNHyperparameters` has no `mixed_precision` field
- `DQNAgent::new()` hardcodes `mixed_precision: None`

**Fix**:
1. Add `mixed_precision: Option<MixedPrecisionConfig>` to `DQNConfig` (dqn/dqn.rs)
2. Add `mixed_precision: Option<MixedPrecisionConfig>` to `DQNHyperparameters` (trainers/dqn/config.rs)
3. In `trainer.rs`: rename `_mixed_precision_detected` → `mixed_precision_detected`, pass to DQNConfig
4. In `DQN::new()`: pass `config.mixed_precision` → `QNetworkConfig.mixed_precision`
5. In hyperopt `train_with_params()`: auto-detect and pass mixed precision

**Files**: `dqn/dqn.rs`, `trainers/dqn/config.rs`, `trainers/dqn/trainer.rs`, `dqn/agent.rs`, `hyperopt/adapters/dqn.rs`

### Fix 2: Wire `hidden_dim_base` Through Training + Eval Binaries

**Problem**: `train_baseline_rl.rs` hardcodes `hidden_dims: vec![128, 64]` in both `train_dqn_fold()` and `train_ppo_fold()`. Hyperopt discovers optimal `hidden_dim_base` but it's silently ignored at train time.

**Fix**:
1. In `train_dqn_fold()`: read `hidden_dim_base` from hyperopt JSON, compute `[base, base/2, base/4]`
2. In `train_ppo_fold()`: read `hidden_dim_base`, compute policy `[base, base/2]` and value `[base*2, base, base/2]`
3. In `evaluate_baseline.rs`: same treatment for eval DQN/PPO config construction
4. Fallback to `[128, 64]` when key absent (backward compat with old hyperopt results)

**Files**: `examples/train_baseline_rl.rs`, `examples/evaluate_baseline.rs`

### Fix 3: Reduce GPU Sync Points (5 per step → 1 periodic)

**Problem**: Four `.to_vec1()` NaN checks (BUG #14) + one `.to_scalar()` force GPU queue flush every training step. GPU cannot pipeline.

**Fix**:
1. Replace per-step NaN detection with periodic check (every 100 steps)
2. Use GPU-side `Tensor::any()` + `Tensor::isnan()` to check NaN without CPU transfer
3. Only pull `.to_vec1()` on the periodic check or when GPU-side NaN detected
4. Keep the `.to_scalar()` in `estimate_avg_q_value` but call it every 100 steps instead of every step

**Files**: `dqn/dqn.rs` (`compute_loss_internal`), `trainers/dqn/trainer.rs` (Q-value monitoring frequency)

## Important Fixes

### Fix 4: Cache Loaded Data Across Trials

**Problem**: Every hyperopt trial re-loads 36 `.dbn.zst` files, decompresses, extracts 51 features. GPU idle for minutes per trial boundary.

**Fix**:
1. In hyperopt runner, load data once into `Arc<Vec<TrainingData>>` before trial loop
2. Pass shared reference to each `train_with_params()` call
3. Add `train_with_preloaded_data()` variant that accepts pre-loaded data

**Files**: `hyperopt/adapters/dqn.rs`, `hyperopt/adapters/ppo.rs`

### Fix 5: Batch Experience Storage (Remove Per-Sample Lock)

**Problem**: `agent.write().await` acquired per sample (100K+ per epoch). Serializes experience collection.

**Fix**:
1. Collect actions + experiences into `Vec` during batch processing
2. Single `agent.write().await` to store entire batch at once
3. Move `track_action()` into batched path

**Files**: `trainers/dqn/trainer.rs` (experience collection loop)

## Files Changed Summary

| File | Fix | Change |
|------|-----|--------|
| `dqn/dqn.rs` | 1,3 | Add `mixed_precision` to DQNConfig, periodic NaN check |
| `trainers/dqn/config.rs` | 1 | Add `mixed_precision` to DQNHyperparameters |
| `trainers/dqn/trainer.rs` | 1,5 | Wire mixed precision, batch experience storage |
| `dqn/agent.rs` | 1 | Pass `config.mixed_precision` to QNetworkConfig |
| `hyperopt/adapters/dqn.rs` | 1,4 | Auto-detect AMP, pre-loaded data path |
| `hyperopt/adapters/ppo.rs` | 4 | Pre-loaded data path |
| `examples/train_baseline_rl.rs` | 2 | Read `hidden_dim_base` from JSON |
| `examples/evaluate_baseline.rs` | 2 | Read `hidden_dim_base` from JSON |

## Expected Outcome

With all 5 fixes:
- TENSOR_ACTIVE: 0% → 40-60% (BF16 actually engaged)
- GPU_UTIL: 12% → 60-80% (fewer sync stalls, larger models)
- Trial startup: minutes → seconds (cached data)
- Training step throughput: ~5x (1 sync per 100 steps vs 5 per step)