foxhunt/TFT_QUICK_FIX_GUIDE.md

# TFT Trainer Quick Fix Guide

**Quick Reference for Implementing Validation Loss and GPU Fixes**

---

## Issue Summary

1. **Validation Loss Returns 0.0**: Happens when validation_frequency > 1 or val_loader is empty
2. **GPU Not Used**: Silent fallback to CPU, no clear logging, user doesn't know if GPU is working

---

## Root Causes

### Validation Loss Issue
```rust
// Line 387: Validation only runs every 5th epoch
if epoch % self.training_config.validation_frequency == 0 {
    // Runs on epochs 0, 5, 10, 15...
} else {
    (0.0, ValidationMetrics::default())  // Epochs 1-4, 6-9, etc. = 0.0
}
```

### GPU Issue
```rust
// Line 278: No distinction between GPU success and CPU fallback
let device = if config.use_gpu {
    Device::cuda_if_available(0)?  // Returns Cpu if CUDA fails - no warning!
}
```

---

## Quick Fixes (Copy-Paste Ready)

### Fix 1: GPU Device Selection (/home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs, line 275)

**REPLACE THIS:**
```rust
let device = if config.use_gpu {
    Device::cuda_if_available(0)
        .map_err(|e| MLError::ConfigError {
            reason: format!("GPU requested but not available: {}", e),
        })?
} else {
    Device::Cpu
};
info!("Using device: {:?}", device);
```

**WITH THIS:**
```rust
let device = if config.use_gpu {
    match Device::cuda_if_available(0) {
        Ok(Device::Cuda(cuda_device)) => {
            info!("✓ GPU enabled: CUDA device 0");
            info!("  Tensors will be allocated on GPU (5-10x speedup expected)");
            Device::Cuda(cuda_device)
        },
        Ok(Device::Cpu) => {
            warn!("⚠ GPU requested but CUDA unavailable - falling back to CPU");
            warn!("  Check: nvidia-smi, CUDA_HOME, LD_LIBRARY_PATH");
            Device::Cpu
        },
        Err(e) => {
            return Err(MLError::ConfigError {
                reason: format!("GPU init failed: {}. Check CUDA drivers.", e),
            });
        },
        _ => {
            warn!("⚠ Unexpected device type, using CPU");
            Device::Cpu
        },
    }
} else {
    info!("Using CPU (use_gpu=false)");
    Device::Cpu
};
info!("Device: {:?}", device);
```

---

### Fix 2: Validation Loss Defensive Checks (/home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs, line 535)

**ADD AT START OF validate_epoch():**
```rust
async fn validate_epoch(
    &mut self,
    val_loader: &mut TFTDataLoader,
    epoch: usize,  // Changed from _epoch to epoch
) -> MLResult<(f64, ValidationMetrics)> {
    // NEW: Check for empty loader
    if val_loader.len() == 0 {
        warn!("Validation loader empty (epoch {}) - returning 0.0", epoch + 1);
        return Ok((0.0, ValidationMetrics::default()));
    }

    info!("Validation start (epoch {}): {} batches", epoch + 1, val_loader.len());

    let mut total_loss = 0.0;
    let mut total_quantile_loss = 0.0;
    let mut total_rmse = 0.0;
    let mut attention_entropies = Vec::new();
    let mut batch_count = 0;
    let mut skipped_batches = 0;  // NEW: Track skipped batches

    for batch in val_loader.iter() {
        // NEW: Skip empty batches
        if batch.targets.nrows() == 0 {
            skipped_batches += 1;
            continue;
        }

        // ... existing code ...

        batch_count += 1;
    }

    // NEW: Check for zero batches
    if batch_count == 0 {
        warn!(
            "Zero non-empty batches (epoch {}, skipped: {})",
            epoch + 1, skipped_batches
        );
        return Ok((0.0, ValidationMetrics::default()));
    }

    info!(
        "Validation complete (epoch {}): {} batches, {} skipped",
        epoch + 1, batch_count, skipped_batches
    );

    // ... rest of function unchanged ...
}
```

---

### Fix 3: Validation Frequency (/home/jgrusewski/Work/foxhunt/ml/src/tft/training.rs, line 94)

**CHANGE:**
```rust
validation_frequency: 5,  // Only validates every 5th epoch
```

**TO:**
```rust
validation_frequency: 1,  // Validate every epoch
```

---

## Verification Steps

### 1. Check GPU Detection
```bash
# Start training and watch logs
cargo run -p ml --example comprehensive_model_backtest -- --use-gpu

# Expected output:
# ✓ GPU enabled: CUDA device 0
#   Tensors will be allocated on GPU (5-10x speedup expected)
# Device: Cuda(CudaDevice(DeviceId(0)))
```

### 2. Monitor GPU Usage
```bash
# In separate terminal
watch -n 1 nvidia-smi

# Expected during training:
# GPU-Util: 30-80%
# Memory-Usage: 1500-3000 MB
```

### 3. Check Validation Loss
```bash
# Run 10 epochs
cargo run -- --epochs 10 --use-gpu

# Expected output:
# Epoch 1/10: Train Loss: 2.345678, Val Loss: 2.123456 (non-zero!)
# Epoch 2/10: Train Loss: 2.234567, Val Loss: 2.012345 (non-zero!)
# ... all 10 epochs show non-zero val_loss
```

---

## Expected Results

| Metric | Before Fix | After Fix |
|--------|-----------|-----------|
| **Val Loss (Epoch 2)** | 0.000000 | 1.234567 (non-zero) |
| **GPU Message** | "Using device: Cpu" | "✓ GPU enabled: CUDA device 0" |
| **GPU Utilization** | 0% | 30-80% |
| **Epoch Duration** | 10-20 min (CPU) | 60-120 sec (GPU) |

---

## Troubleshooting

### GPU Still Shows 0% Utilization
```bash
# Check CUDA availability
python3 -c "import torch; print(torch.cuda.is_available())"

# Check environment
echo $CUDA_HOME
echo $LD_LIBRARY_PATH

# Verify nvidia-smi works
nvidia-smi
```

### Validation Loss Still 0.0
```bash
# Check validation data size
# Add this to validate_epoch():
info!("Val loader size: {}, first batch rows: {}",
      val_loader.len(),
      val_loader.batches.first().map(|b| b.targets.nrows()).unwrap_or(0));
```

### Out of Memory (OOM) on GPU
```bash
# Reduce batch size in config
# Change from 32 to 16:
batch_size: 16,  // Was 32, reduced for 4GB VRAM
```

---

## Testing Checklist

- [ ] GPU detection logs show "✓ GPU enabled" (not CPU fallback warning)
- [ ] `nvidia-smi` shows >30% GPU utilization during training
- [ ] `nvidia-smi` shows 1.5-3.0 GB GPU memory usage
- [ ] Validation loss is non-zero for ALL epochs (not just epoch 0, 5, 10...)
- [ ] Epoch duration is 60-120 seconds (not 10-20 minutes)
- [ ] Training completes successfully without OOM errors

---

## Files to Modify

1. `/home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs`
   - Line 275: GPU device selection (Fix 1)
   - Line 535: validate_epoch() defensive checks (Fix 2)

2. `/home/jgrusewski/Work/foxhunt/ml/src/tft/training.rs`
   - Line 94: validation_frequency change (Fix 3)

---

## Performance Targets

| Configuration | Epoch Time (GPU) | GPU Utilization | Memory |
|---------------|------------------|-----------------|---------|
| Batch 16 | 30-60 sec | 40-60% | 1.2-1.8 GB |
| Batch 32 | 60-120 sec | 50-70% | 1.8-2.5 GB |
| Batch 64 | 120-240 sec | 60-80% | 2.5-3.5 GB |

**Speedup vs CPU**: 5-10x faster

---

**Implementation Time**: 10-15 minutes
**Testing Time**: 30-60 minutes (10-epoch training)
**Total**: ~45-75 minutes

---

**Status**: ✅ READY TO IMPLEMENT