foxhunt/WAVE_8_10_QUICK_REFERENCE.md

Wave 8.10: TFT GPU Memory Profile - Quick Reference

Status: ❌ FAILED - TFT exceeds memory budget by 6x Date: 2025-10-15

---

Critical Findings

Memory Usage (F32, batch_size=32)

Component            Measured    Budget     Status
────────────────────────────────────────────────
Model Parameters     72MB        <300MB     ✅ PASS
Forward Activations  2,880MB     <200MB     ❌ FAIL (14x over)
Backward Gradients   0MB         <200MB     ✅ PASS
Optimizer State      144MB       <200MB     ✅ PASS
────────────────────────────────────────────────
PEAK TRAINING        3,096MB     <1000MB    ❌ FAIL (3.1x over)

Root Cause

• TFT architecture holds massive intermediate activations in GPU memory
• 615x overhead vs theoretical memory (4.8MB theoretical → 2,952MB measured)
• Hypothesis: Candle framework retains activation tensors for backpropagation

---

Immediate Actions Required

1. Enable FP16 Mixed Precision (50% reduction)

// In ml/tests/tft_e2e_training.rs
fn default_tft_config() -> TFTConfig {
    TFTConfig {
        mixed_precision: true,  // ← ADD THIS
        // ... rest of config
    }
}

Expected Result: 3,096MB → 1,548MB ✅ Under 2GB

2. Implement Gradient Checkpointing (75% reduction)

TFTConfig {
    memory_efficient: true,
    gradient_checkpointing: true,  // ← ADD THIS
}

Expected Result: 3,096MB → 774MB ✅ Under 1GB

3. Reduce Batch Size (fallback)

TFTConfig {
    batch_size: 8,  // Reduce from 32 → 8
}

Expected Result: 3,096MB → 774MB ✅ Under 1GB

---

Optimization Strategy Comparison

Strategy	Memory Reduction	Training Speed	Accuracy Impact	Difficulty
FP16 Mixed Precision	50%	+20% faster	<2% loss	Easy (1 line)
Gradient Checkpointing	75%	-40% slower	None	Medium (framework support)
Reduce Batch Size (32→8)	75%	-75% slower	None	Easy (1 line)
Shorter Sequence (60→30)	50%	No change	Model degradation	Medium (retraining)

---

Test Command

# Run GPU memory profiling
cargo test -p ml --test tft_e2e_training test_tft_gpu_memory_profiling -- --test-threads=1 --nocapture

# Expected output:
# ❌ Forward memory: 2952MB (should be <500MB)
# ❌ Training peak: 3096MB (should be <1GB)

---

Ensemble Impact

Current State (F32)

Model       Memory      Status
────────────────────────────────
DQN         6MB         ✅ OK
PPO         145MB       ✅ OK
MAMBA-2     164MB       ✅ OK
TFT         3,096MB     ❌ CRITICAL
────────────────────────────────
Total       3,411MB     ❌ 83% of 4GB GPU
Free        685MB       ❌ Insufficient headroom

Target State (FP16 + Checkpointing)

Model       Memory      Status
────────────────────────────────
DQN         3MB         ✅ OK
PPO         73MB        ✅ OK
MAMBA-2     82MB        ✅ OK
TFT         774MB       ✅ OK
────────────────────────────────
Total       932MB       ✅ 23% of 4GB GPU
Free        3,164MB     ✅ Ample headroom

---

Files Modified

1. ml/tests/tft_e2e_training.rs

   • Added test_tft_gpu_memory_profiling() test (line 592-697)
   • GPU memory measurement with nvidia-smi integration
   • Comprehensive validation checks

2. ml/src/tft/trainable_adapter.rs

   • Fixed optimizer API compatibility
   • Added GradStore management for backward pass
   • Fixed set_learning_rate() to use void return

---

Next Wave Tasks

Wave 8.11: FP16 Mixed Precision

• Enable mixed_precision = true in TFTConfig
• Validate accuracy degradation <5%
• Re-run memory profiling (expect 1,548MB)

Wave 8.12: Gradient Checkpointing

• Research Candle gradient checkpointing support
• Implement gradient_checkpointing = true config
• Benchmark training speed impact (<2x slowdown acceptable)

Wave 8.13: Production Validation

• Test ensemble training with optimized TFT
• Measure concurrent inference memory usage
• Update deployment documentation

---

Key Metrics

Memory Budget Violations

• Forward Activations: 2,880MB vs 200MB budget (14.4x over)
• Training Peak: 3,096MB vs 1,000MB budget (3.1x over)

Optimization Targets

• FP16: 50% reduction → 1,548MB (still 1.5x over)
• FP16 + Checkpointing: 75% reduction → 774MB ✅ MEETS BUDGET

---

Critical Path

Wave 8.10 (CURRENT)
    ↓
Wave 8.11: FP16 Mixed Precision (1 day)
    ↓
Wave 8.12: Gradient Checkpointing (2-3 days)
    ↓
Wave 8.13: Production Validation (1 day)
    ↓
Wave 8.14: Ensemble Deployment ✅

Total Timeline: 4-5 days to production-ready TFT

---

Documentation References

• Detailed Report: /home/jgrusewski/Work/foxhunt/WAVE_8_10_TFT_GPU_MEMORY_PROFILE.md
• Test File: /home/jgrusewski/Work/foxhunt/ml/tests/tft_e2e_training.rs (line 592-697)
• Optimizer Fix: /home/jgrusewski/Work/foxhunt/ml/src/tft/trainable_adapter.rs (line 299-322, 368-370)

---

Contact

Agent: Wave 8.10 Priority: CRITICAL (blocks ensemble production deployment) Action Owner: Wave 8.11 Agent (FP16 implementation) Due Date: 2025-10-16