foxhunt/ml/profiling_reports/TFT_INT8_MEMORY_PROFILING_GUIDE.md

TFT INT8 Memory Profiling Guide

Purpose: Comprehensive guide for profiling FP32 vs INT8 TFT memory usage to validate 75% memory reduction target.

Last Updated: 2025-10-21

---

Quick Start

Prerequisites

1. Hardware: NVIDIA GPU with CUDA support (tested on RTX 3050 Ti, 4GB VRAM)
2. Software: CUDA toolkit installed, nvidia-smi available in PATH
3. System: Linux/WSL2 with NVIDIA drivers

Basic Usage

# Run profiling with default settings
cargo run -p ml --example profile_tft_int8_memory --release --features cuda

# With verbose logging
cargo run -p ml --example profile_tft_int8_memory --release --features cuda -- --verbose

# Custom output directory
cargo run -p ml --example profile_tft_int8_memory --release --features cuda -- \
  --output-dir ml/profiling_reports/run_$(date +%Y%m%d_%H%M%S)

# More iterations for averaging
cargo run -p ml --example profile_tft_int8_memory --release --features cuda -- \
  --num-iterations 20

---

What Gets Measured

1. Parameter Memory (Static)

FP32: 4 bytes per weight tensor INT8: 1 byte per weight tensor (+ scale/zero-point metadata)

Expected Reduction: 75% (4x smaller)

Parameters include:
├── Variable Selection Networks (3x): input_dim × hidden_dim
├── LSTM Encoder: 4 × hidden_dim × hidden_dim
├── Temporal Attention: 4 × hidden_dim × hidden_dim (Q/K/V/O)
├── GRN Stacks: num_layers × hidden_dim × hidden_dim
└── Output Layer: hidden_dim × num_quantiles

2. Activation Memory (Dynamic)

Measured during forward pass: Intermediate tensors created during inference

Expected Reduction: 0-25% (FP32 activations retained for accuracy)

Activations include:
├── Variable selection outputs
├── Encoder outputs (static, historical, future)
├── LSTM hidden states
├── Attention scores & weights
└── Intermediate GRN activations

3. Optimizer Memory (Training Only)

Adam optimizer: 2x parameter count (momentum + variance buffers)

Expected Reduction: 75% (same as parameters)

Optimizer states:
├── Gradients: same shape as parameters
├── Momentum (1st moment): same shape as parameters
└── Variance (2nd moment): same shape as parameters

---

Expected Results

FP32 Baseline (Default Config: 225 features, 256 hidden_dim)

Component	Estimated Size	Notes
Parameters	500-800 MB	Depends on num_layers, hidden_dim
Activations	200-400 MB	Varies with batch_size, sequence_length
Optimizer	1000-1600 MB	Adam: 2x parameters
Total	1700-2800 MB	Peak during training

INT8 Quantized (Same Config)

Component	Estimated Size	Notes
Parameters	125-200 MB	75% reduction (4x smaller)
Activations	150-350 MB	Minimal reduction (FP32 retained)
Optimizer	250-400 MB	75% reduction (quantized grads)
Total	525-950 MB	~70-75% reduction

Memory Savings

FP32 Total:   ~2000 MB
INT8 Total:   ~500 MB
Reduction:    ~1500 MB (75%)
RTX 3050 Ti:  4096 MB total (12% utilization with INT8)

---

Report Outputs

1. Markdown Report (tft_int8_memory_profile.md)

Human-readable report with:

• Executive summary
• Memory breakdown comparison table
• Performance metrics (latency overhead)
• ASCII bar charts
• Notes and recommendations

Example:

# TFT INT8 Memory Profiling Report

**Timestamp**: 2025-10-21T10:00:00Z
**GPU Device**: RTX 3050 Ti
**Total VRAM**: 4096 MB

## Executive Summary

- **Memory Reduction**: 1500.0 MB (75.0%)
- **75% Target**: ✅ **ACHIEVED**
- **FP32 Peak**: 2000 MB
- **INT8 Peak**: 500 MB

## Memory Breakdown Comparison

| Component | FP32 (MB) | INT8 (MB) | Reduction (%) |
|-----------|-----------|-----------|---------------|
| Parameters | 700 | 175 | 75.0% |
| Activations | 400 | 350 | 12.5% |
| Optimizer | 900 | 225 | 75.0% |
| **Total** | **2000** | **500** | **75.0%** |

2. JSON Report (tft_int8_memory_profile.json)

Machine-readable report for programmatic analysis:

{
  "timestamp": "2025-10-21T10:00:00Z",
  "gpu_device": "RTX 3050 Ti",
  "gpu_total_vram_mb": 4096.0,
  "fp32_profile": {
    "variant": "FP32",
    "breakdown": {
      "parameters_mb": 700.0,
      "activations_mb": 400.0,
      "optimizer_mb": 900.0,
      "total_mb": 2000.0
    },
    "peak_memory_mb": 2000.0,
    "avg_memory_mb": 1950.0,
    "samples_collected": 10,
    "inference_latency_us": 5000
  },
  "int8_profile": { ... },
  "memory_reduction_mb": 1500.0,
  "memory_reduction_percent": 75.0,
  "meets_75_percent_target": true
}

---

Interpreting Results

✅ Success Criteria

1. Memory Reduction ≥ 75%: INT8 total memory is ≤25% of FP32 baseline
2. Parameter Reduction ≥ 70%: Weight tensors achieve near 4x compression
3. Latency Overhead ≤ 10%: INT8 dequantization doesn't slow inference
4. No Memory Leaks: Consistent memory across iterations

❌ Failure Scenarios

Issue	Likely Cause	Solution
Reduction < 75%	Activations not quantized	Expected - activations stay FP32 for accuracy
Reduction < 50%	Quantization not applied	Check new_from_fp32() call succeeded
Latency > +20%	Dequantization overhead	Use per-channel quantization, CUDA kernels
Memory leak	Intermediate tensors not freed	Check .detach() calls, garbage collection

🔍 Debugging Tips

1. Low reduction (<50%):

   # Check quantized weights are actually INT8
   cargo run --example profile_tft_int8_memory -- --verbose 2>&1 | grep "INT8"
   

2. High latency overhead (>20%):

   # Profile with smaller batch size
   cargo run --example profile_tft_int8_memory -- --batch-size 1
   

3. Memory leak suspected:

   # More iterations to confirm
   cargo run --example profile_tft_int8_memory -- --num-iterations 50
   

4. CUDA errors:

   # Check nvidia-smi availability
   nvidia-smi --query-gpu=memory.used,memory.total --format=csv
   

---

Advanced Configuration

Custom TFT Configurations

Edit profile_tft_int8_memory.rs to test different model sizes:

// Small model (testing)
let config = TFTConfig {
    input_dim: 64,
    hidden_dim: 128,
    num_layers: 2,
    ..Default::default()
};

// Production model (Wave C+D)
let config = TFTConfig::default(); // 225 features, 256 hidden_dim

// Large model (future scaling)
let config = TFTConfig {
    input_dim: 512,
    hidden_dim: 512,
    num_layers: 8,
    ..Default::default()
};

Profiling Only Specific Components

// Profile only attention memory
let _ = int8_model.forward_temporal_attention(&historical_features, false)?;

// Profile only quantile output
let _ = int8_model.forward_quantile_output(&decoder_output, &quantized_weights)?;

// Profile only future decoder
let _ = int8_model.forward_future_decoder(&future_features, &decoder_weights)?;

---

Benchmarking Methodology

Memory Measurement Strategy

1. Baseline Snapshot: Measure GPU memory before model creation
2. Post-Load Snapshot: Measure memory after model instantiation (parameters)
3. Warmup Inference: Run 1 inference to allocate activation buffers
4. Peak Measurement: Run N iterations, track max VRAM usage
5. Leak Detection: Compare first vs last iteration (should be stable)

Averaging Strategy

• Default: 10 iterations (balance speed vs accuracy)
• Quick: 3 iterations (fast validation)
• Thorough: 20-50 iterations (production validation)

Snapshot Timing

• 100ms cache: nvidia-smi calls reuse cached values within 100ms window
• 500ms stabilization: Wait after model load for GPU allocator to stabilize
• 100ms between iterations: Ensure CUDA operations complete

---

Integration with CI/CD

Automated Profiling

# .github/workflows/memory_profiling.yml
name: TFT Memory Profiling

on:
  push:
    paths:
      - 'ml/src/tft/**'
      - 'ml/src/memory_optimization/**'

jobs:
  profile:
    runs-on: self-hosted-gpu
    steps:
      - uses: actions/checkout@v3
      - name: Run profiling
        run: |
          cargo run -p ml --example profile_tft_int8_memory --release --features cuda
      - name: Upload report
        uses: actions/upload-artifact@v3
        with:
          name: memory-profile
          path: ml/profiling_reports/tft_int8_memory_profile.md

Regression Detection

# Compare against baseline
baseline_mb=$(jq '.int8_profile.peak_memory_mb' baseline.json)
current_mb=$(jq '.int8_profile.peak_memory_mb' tft_int8_memory_profile.json)

if (( $(echo "$current_mb > $baseline_mb * 1.1" | bc -l) )); then
  echo "❌ Memory regression detected: $current_mb MB > $baseline_mb MB"
  exit 1
fi

---

Troubleshooting

Common Errors

1. CUDA device not available

⚠️  CUDA not available, this profiling requires GPU

Solution: Ensure NVIDIA drivers installed, nvidia-smi works, CUDA feature enabled.

2. nvidia-smi not found

Failed to run nvidia-smi: No such file or directory

Solution: Install NVIDIA drivers, ensure nvidia-smi in PATH.

3. Out of Memory (OOM)

CUDA error: out of memory

Solution: Reduce batch_size, reduce sequence_length, or use smaller model config.

4. Profiling very slow

Profiling taking >5 minutes

Solution: Reduce --num-iterations, ensure release build (--release), check GPU isn't being used by other processes.

GPU Health Check

# Check GPU availability
nvidia-smi

# Check CUDA version
nvcc --version

# Check GPU memory usage
nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv -l 1

# Kill GPU processes
nvidia-smi --query-compute-apps=pid --format=csv,noheader | xargs kill -9

---

Performance Expectations

Profiling Runtime

Configuration	Expected Time	Notes
Default (10 iterations)	30-60 seconds	Standard validation
Quick (3 iterations)	10-20 seconds	Fast smoke test
Thorough (50 iterations)	2-5 minutes	Production validation

Memory Overhead

• nvidia-smi subprocess: ~5-10 MB system RAM
• Profiler snapshots: ~1 KB per snapshot (negligible)
• JSON report: ~5-10 KB
• Markdown report: ~10-20 KB

---

FAQ

Q1: Why is activation memory not reduced by 75%?

A: Activations remain FP32 for numerical stability. Quantizing activations can hurt model accuracy. Only static parameters (weights) are quantized to INT8.

Q2: Why is the reduction less than 75% in practice?

A: Total memory includes activations (FP32) + optimizer states (FP32). Only parameters achieve 75% reduction. Overall reduction is typically 60-70%.

Q3: Can I quantize activations too?

A: Yes, but this requires activation quantization (not implemented yet). Expected additional savings: 10-20%. See ml/src/memory_optimization/quantization.rs for future implementation.

Q4: How does this compare to model pruning?

A: Quantization (4x compression) is orthogonal to pruning (remove weights). You can combine both for 8-16x total compression.

Q5: What about inference-only deployment?

A: Remove optimizer states to save 2x parameter memory. Total INT8 inference-only: ~300-400 MB (vs 1200-1600 MB FP32).

---

References

• CLAUDE.md: Production ML model specifications (DQN, PPO, MAMBA-2, TFT memory budgets)
• ml/src/tft/quantized_tft.rs: INT8 quantized TFT implementation
• ml/src/memory_optimization/: Quantization utilities and configs
• ml/tests/tft_int8_memory_benchmark_test.rs: Memory benchmark test suite

---

Change Log

2025-10-21: Initial profiling guide created

• Comprehensive FP32 vs INT8 comparison
• Memory breakdown (parameters, activations, optimizer)
• Markdown + JSON report generation
• 75% reduction target validation

---

Generated by: AGENT-152 (Wave 152 Memory Profiling Initiative) Model: Claude Sonnet 4.5 (claude-sonnet-4-5-20250929)