feat(ml): GPU full saturation — remove artificial caps and enable VRAM-aware scaling

Phase 1: Remove artificial caps
- TFT benchmark: VRAM-scaled batch sizes (4/16/32/64) replacing hardcoded max_batch=4
- Liquid CUDA: VRAM-aware config defaults (batch 256-2048, pool 10% VRAM)
- DQN trainer: remove double-clamp between AutoBatchSizer and HardwareBudget

Phase 2: Mixed precision in training
- DQN agent: add BF16/FP16 dtype casting in forward_with_gradients and
  forward_without_gradients (training was bypassing forward_mixed)

Phase 3: Reduce CPU round-trips
- DQN trainer: flat buffer select_actions_batch (eliminate Vec<Vec<f32>>)
- DQN trainer: early-skip experience extraction (avoid .to_vec() on invalid)
- EpochPrefetcher: AtomicBool is_ready() so callers can detect completion

Phase 4: Adaptive scaling
- HardwareBudget: tiered safety factor (0.70-0.85 by GPU size)
- AutoBatchSizer: VRAM-proportional batch_overhead_mb (1.5% instead of fixed 250MB)

2451 tests pass, 0 clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-03-02 11:42:38 +01:00
parent a489e6455e
commit d63190b9b9
7 changed files with 193 additions and 84 deletions

View File

@@ -234,11 +234,14 @@ impl AutoBatchSizer {
// Add batch-level overhead (calculated before error checking)
// This accounts for intermediate buffers that don't scale linearly with batch size
let batch_overhead_mb = match config.model_precision {
ModelPrecision::FP32 => 250.0, // FP32: ~250MB per batch (attention, workspace)
ModelPrecision::INT8 => 75.0, // INT8: ~75MB per batch
ModelPrecision::QAT => 500.0, // QAT: ~500MB per batch (FP32 base + FakeQuantize intermediate tensors)
// Batch overhead scales with GPU VRAM (attention workspaces, cuDNN scratch)
// Base: ~1.5% of total VRAM for FP32, with per-precision multipliers
let base_overhead_pct = match config.model_precision {
ModelPrecision::FP32 => 0.015,
ModelPrecision::INT8 => 0.005,
ModelPrecision::QAT => 0.030,
};
let batch_overhead_mb = (self.total_memory_mb * base_overhead_pct).max(50.0);
debug!(
"Fixed overhead: Model={:.1}MB, Optimizer={:.1}MB, Gradients={:.1}MB, Activations={:.1}MB, Total={:.1}MB, Batch overhead={:.1}MB",
@@ -344,11 +347,13 @@ impl AutoBatchSizer {
};
let activation_mb = model_mb * activation_multiplier;
let fixed_overhead_mb = model_mb + optimizer_mb + gradient_mb + activation_mb;
let batch_overhead_mb = match config.model_precision {
ModelPrecision::FP32 => 250.0,
ModelPrecision::INT8 => 75.0,
ModelPrecision::QAT => 500.0,
// Batch overhead scales with GPU VRAM (attention workspaces, cuDNN scratch)
let base_overhead_pct = match config.model_precision {
ModelPrecision::FP32 => 0.015,
ModelPrecision::INT8 => 0.005,
ModelPrecision::QAT => 0.030,
};
let batch_overhead_mb = (self.total_memory_mb * base_overhead_pct).max(50.0);
let available = usable_memory_mb - fixed_overhead_mb - batch_overhead_mb;
if available <= 0.0 {