foxhunt/docs/archive/ml_models/TFT_QUICK_FIX_GUIDE.md

TFT Trainer Quick Fix Guide

Quick Reference for Implementing Validation Loss and GPU Fixes

---

Issue Summary

1. Validation Loss Returns 0.0: Happens when validation_frequency > 1 or val_loader is empty
2. GPU Not Used: Silent fallback to CPU, no clear logging, user doesn't know if GPU is working

---

Root Causes

Validation Loss Issue

// Line 387: Validation only runs every 5th epoch
if epoch % self.training_config.validation_frequency == 0 {
    // Runs on epochs 0, 5, 10, 15...
} else {
    (0.0, ValidationMetrics::default())  // Epochs 1-4, 6-9, etc. = 0.0
}

GPU Issue

// Line 278: No distinction between GPU success and CPU fallback
let device = if config.use_gpu {
    Device::cuda_if_available(0)?  // Returns Cpu if CUDA fails - no warning!
}

---

Quick Fixes (Copy-Paste Ready)

Fix 1: GPU Device Selection (/home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs, line 275)

REPLACE THIS:

let device = if config.use_gpu {
    Device::cuda_if_available(0)
        .map_err(|e| MLError::ConfigError {
            reason: format!("GPU requested but not available: {}", e),
        })?
} else {
    Device::Cpu
};
info!("Using device: {:?}", device);

WITH THIS:

let device = if config.use_gpu {
    match Device::cuda_if_available(0) {
        Ok(Device::Cuda(cuda_device)) => {
            info!("✓ GPU enabled: CUDA device 0");
            info!("  Tensors will be allocated on GPU (5-10x speedup expected)");
            Device::Cuda(cuda_device)
        },
        Ok(Device::Cpu) => {
            warn!("⚠ GPU requested but CUDA unavailable - falling back to CPU");
            warn!("  Check: nvidia-smi, CUDA_HOME, LD_LIBRARY_PATH");
            Device::Cpu
        },
        Err(e) => {
            return Err(MLError::ConfigError {
                reason: format!("GPU init failed: {}. Check CUDA drivers.", e),
            });
        },
        _ => {
            warn!("⚠ Unexpected device type, using CPU");
            Device::Cpu
        },
    }
} else {
    info!("Using CPU (use_gpu=false)");
    Device::Cpu
};
info!("Device: {:?}", device);

---

Fix 2: Validation Loss Defensive Checks (/home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs, line 535)

ADD AT START OF validate_epoch():

async fn validate_epoch(
    &mut self,
    val_loader: &mut TFTDataLoader,
    epoch: usize,  // Changed from _epoch to epoch
) -> MLResult<(f64, ValidationMetrics)> {
    // NEW: Check for empty loader
    if val_loader.len() == 0 {
        warn!("Validation loader empty (epoch {}) - returning 0.0", epoch + 1);
        return Ok((0.0, ValidationMetrics::default()));
    }

    info!("Validation start (epoch {}): {} batches", epoch + 1, val_loader.len());

    let mut total_loss = 0.0;
    let mut total_quantile_loss = 0.0;
    let mut total_rmse = 0.0;
    let mut attention_entropies = Vec::new();
    let mut batch_count = 0;
    let mut skipped_batches = 0;  // NEW: Track skipped batches

    for batch in val_loader.iter() {
        // NEW: Skip empty batches
        if batch.targets.nrows() == 0 {
            skipped_batches += 1;
            continue;
        }

        // ... existing code ...

        batch_count += 1;
    }

    // NEW: Check for zero batches
    if batch_count == 0 {
        warn!(
            "Zero non-empty batches (epoch {}, skipped: {})",
            epoch + 1, skipped_batches
        );
        return Ok((0.0, ValidationMetrics::default()));
    }

    info!(
        "Validation complete (epoch {}): {} batches, {} skipped",
        epoch + 1, batch_count, skipped_batches
    );

    // ... rest of function unchanged ...
}

---

Fix 3: Validation Frequency (/home/jgrusewski/Work/foxhunt/ml/src/tft/training.rs, line 94)

CHANGE:

validation_frequency: 5,  // Only validates every 5th epoch

TO:

validation_frequency: 1,  // Validate every epoch

---

Verification Steps

1. Check GPU Detection

# Start training and watch logs
cargo run -p ml --example comprehensive_model_backtest -- --use-gpu

# Expected output:
# ✓ GPU enabled: CUDA device 0
#   Tensors will be allocated on GPU (5-10x speedup expected)
# Device: Cuda(CudaDevice(DeviceId(0)))

2. Monitor GPU Usage

# In separate terminal
watch -n 1 nvidia-smi

# Expected during training:
# GPU-Util: 30-80%
# Memory-Usage: 1500-3000 MB

3. Check Validation Loss

# Run 10 epochs
cargo run -- --epochs 10 --use-gpu

# Expected output:
# Epoch 1/10: Train Loss: 2.345678, Val Loss: 2.123456 (non-zero!)
# Epoch 2/10: Train Loss: 2.234567, Val Loss: 2.012345 (non-zero!)
# ... all 10 epochs show non-zero val_loss

---

Expected Results

Metric	Before Fix	After Fix
Val Loss (Epoch 2)	0.000000	1.234567 (non-zero)
GPU Message	"Using device: Cpu"	"✓ GPU enabled: CUDA device 0"
GPU Utilization	0%	30-80%
Epoch Duration	10-20 min (CPU)	60-120 sec (GPU)

---

Troubleshooting

GPU Still Shows 0% Utilization

# Check CUDA availability
python3 -c "import torch; print(torch.cuda.is_available())"

# Check environment
echo $CUDA_HOME
echo $LD_LIBRARY_PATH

# Verify nvidia-smi works
nvidia-smi

Validation Loss Still 0.0

# Check validation data size
# Add this to validate_epoch():
info!("Val loader size: {}, first batch rows: {}",
      val_loader.len(),
      val_loader.batches.first().map(|b| b.targets.nrows()).unwrap_or(0));

Out of Memory (OOM) on GPU

# Reduce batch size in config
# Change from 32 to 16:
batch_size: 16,  // Was 32, reduced for 4GB VRAM

---

Testing Checklist

• GPU detection logs show "✓ GPU enabled" (not CPU fallback warning)
• nvidia-smi shows >30% GPU utilization during training
• nvidia-smi shows 1.5-3.0 GB GPU memory usage
• Validation loss is non-zero for ALL epochs (not just epoch 0, 5, 10...)
• Epoch duration is 60-120 seconds (not 10-20 minutes)
• Training completes successfully without OOM errors

---

Files to Modify

1. /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs

   • Line 275: GPU device selection (Fix 1)
   • Line 535: validate_epoch() defensive checks (Fix 2)

2. /home/jgrusewski/Work/foxhunt/ml/src/tft/training.rs

   • Line 94: validation_frequency change (Fix 3)

---

Performance Targets

Configuration	Epoch Time (GPU)	GPU Utilization	Memory
Batch 16	30-60 sec	40-60%	1.2-1.8 GB
Batch 32	60-120 sec	50-70%	1.8-2.5 GB
Batch 64	120-240 sec	60-80%	2.5-3.5 GB

Speedup vs CPU: 5-10x faster

---

Implementation Time: 10-15 minutes Testing Time: 30-60 minutes (10-epoch training) Total: ~45-75 minutes

---

Status: ✅ READY TO IMPLEMENT