foxhunt/QUICK_MEMORY_TEST_GUIDE.md

Quick Memory Test Guide (30 Second Version)

Date: 2025-10-23 Goal: Test memory issues WITHOUT full compilation

---

The Fast Path (30 seconds, NO GPU)

# Test 1: Quantization memory savings (validates INT8 works)
cargo test -p ml --test memory_optimization_tests \
  -- test_int8_quantization_basic --nocapture

# Test 2: Fake quantization forward pass (validates QAT works)
cargo test -p ml --test qat_test \
  -- test_fake_quantize_forward --nocapture

# Test 3: Observer statistics (validates device handling)
cargo test -p ml --test qat_test \
  -- test_qat_observer_statistics --nocapture

What you're looking for:

• ✅ test result: ok = All tests passed
• ❌ tensor device mismatch (CPU vs CUDA) = Device bug found!
• ❌ assertion failed: savings_percent >= 70 = Memory bug found!

---

The GPU Path (2-5 minutes, requires nvidia-smi)

# Test 1: Measure actual TFT memory usage
cargo test -p ml --test tft_int8_memory_benchmark_test \
  --features cuda -- test_int8_memory_reduction --nocapture

# Test 2: Validate all models fit in 4GB
cargo test -p ml --test gpu_memory_budget_validation \
  --features cuda -- test_all_models_fit_rtx3050ti --nocapture

What you're looking for:

• ✅ INT8 memory: 125 MB (target: <800 MB) = Memory reduction works!
• ✅ Total: 440 MB (89% headroom on 4GB) = Multi-model fit works!
• ❌ OOM: CUDA out of memory = Memory budget exceeded!

---

Memory Test Files (Priority Order)

High Priority (Run First)

1. memory_optimization_tests.rs (21KB)

   • 15 tests, 30s runtime, NO GPU
   • Validates INT8/INT4/FP16 memory savings
   • Tests 4GB GPU compatibility

2. qat_test.rs (23KB)

   • 16 tests, 10s runtime, NO GPU (CPU fallback)
   • Validates fake quantization
   • Tests observer statistics

3. tft_int8_memory_benchmark_test.rs (21KB)

   • 5 tests, 3 min runtime, GPU required
   • Measures ACTUAL VRAM usage via nvidia-smi
   • Validates 75% memory reduction

Medium Priority (Run After Tier 1)

4. qat_accuracy_validation_test.rs (23KB)

   • 8 tests, 30s runtime
   • Validates QAT vs PTQ accuracy

5. gpu_memory_budget_validation.rs (17KB)

   • 4 tests, 10 min runtime
   • Validates multi-model VRAM budget

Low Priority (Run if debugging specific issues)

6. qat_tft_integration_test.rs (15KB)

   • Full TFT QAT integration workflow

7. wave_d_memory_stress_test.rs (15KB)

   • 24-hour stress test (optional)

---

Memory Profiling Tools (3 Systems)

1. GPU Memory Profiler (nvidia-smi wrapper)

Location: ml/src/benchmark/memory_profiler.rs

Usage:

let mut profiler = MemoryProfiler::new(0);  // GPU 0
let baseline = profiler.take_snapshot()?;    // Before
// ... allocate memory ...
let after = profiler.take_snapshot()?;       // After
let vram_mb = after.vram_used_mb - baseline.vram_used_mb;
println!("VRAM used: {} MB", vram_mb);

Performance: <10ms per snapshot (with 100ms cache)

---

2. Safe Memory Manager (device-agnostic)

Location: ml/src/safety/memory_manager.rs

Usage:

let mut manager = SafeMemoryManager::new(&config);
manager.check_memory_availability(bytes, &device)?;  // Pre-check
manager.record_allocation(bytes, &device);           // Track
let stats = manager.get_memory_stats();              // Report

Performance: <1μs per operation (atomic counters)

---

3. CUDA Memory Pool (Liquid Networks)

Location: ml/src/liquid/cuda/memory.rs

Usage: Specialized for Liquid Networks only

---

Benchmarks (Use for deep profiling)

# Memory profiling benchmark (10 min, GPU required)
cargo bench --bench tft_int8_memory_bench --features cuda

# QAT vs PTQ comparison (15 min, CPU fallback available)
cargo bench --bench qat_vs_ptq_bench --features cuda

# Generate HTML reports
cargo bench --bench tft_int8_memory_bench --features cuda \
  -- --save-baseline main

---

Common Memory Issues

Issue 1: Device Mismatch

Symptom: tensor on CPU but operation expects CUDA

Test:

cargo test -p ml --test qat_test -- test_fake_quantize_forward --nocapture

Fix: Ensure tensor.to_device(&device) before observer calls

---

Issue 2: OOM During Model Creation

Symptom: CUDA out of memory

Test:

cargo test -p ml --test gpu_memory_budget_validation -- --nocapture

Fix: Enable gradient checkpointing or reduce batch size

---

Issue 3: Memory Leak

Symptom: VRAM grows over time (>50MB per 100 batches)

Test:

cargo test -p ml --test tft_int8_memory_benchmark_test \
  -- test_no_memory_leaks --nocapture

Fix: Check for unclosed CUDA streams or cached tensors

---

Issue 4: Quantization Not Working

Symptom: INT8 model uses same memory as FP32

Test:

cargo test -p ml --test memory_optimization_tests \
  -- test_int8_quantization_basic --nocapture

Fix: Verify quantization is enabled (not just FP32 with INT8 flag)

---

Expected Results (Sanity Check)

Memory Reduction Targets

Precision	Memory Savings	Test Threshold
INT8	75%	≥70%
INT4	87%	≥85%
FP16	50%	49-51%
BF16	50%	49-51%

GPU Memory Budget (RTX 3050 Ti, 4GB)

Model	Memory (INT8)	Memory (FP32)	Budget
DQN	6 MB	6 MB	<50 MB
PPO	145 MB	145 MB	<200 MB
MAMBA-2	164 MB	164 MB	<250 MB
TFT	125 MB	500 MB	<800 MB
Total	440 MB	815 MB	<3500 MB

Headroom: 89% available (3500 - 440 = 3060 MB free)

---

One-Liner Test Commands

Fast (30s, NO GPU)

cargo test -p ml --test memory_optimization_tests --test qat_test -- --nocapture

GPU (3 min, nvidia-smi)

cargo test -p ml --test tft_int8_memory_benchmark_test --features cuda -- --nocapture

Full Suite (10 min, GPU)

cargo test -p ml --tests --features cuda -- memory

Benchmarks (30 min, GPU)

cargo bench -p ml --benches --features cuda

---

Key Takeaways

1. Start with fast tests (30s, NO GPU) to catch most bugs
2. Memory tests don't require full model training (just creation + inference)
3. nvidia-smi integration gives REAL VRAM usage (no estimation)
4. Device mismatch likely NOT caught by existing tests (gap identified)
5. Test coverage is excellent except for device validation

Next Step: Run fast tests first, check for device errors, then run GPU tests if needed.

---

Emergency Debugging (When Tests Fail)

# Step 1: Check device mismatch
cargo test -p ml --test qat_test -- test_fake_quantize_forward --nocapture 2>&1 | grep -i "device\|cuda\|cpu"

# Step 2: Check memory savings
cargo test -p ml --test memory_optimization_tests -- test_int8 --nocapture 2>&1 | grep -i "savings\|reduction"

# Step 3: Check GPU memory
nvidia-smi  # Manual check before/after tests