foxhunt/TFT_TRAINER_FIX_REPORT.md

TFT Trainer Validation Loss and GPU Fix Report

Date: 2025-10-14 Agent: Claude Code Agent Mission: Fix validation loss computation (returns 0.0) and enable GPU training

---

Issues Identified

1. Validation Loss Returns 0.0 (Epochs 2-5)

Root Cause Analysis:

// Line 387-391: validation_frequency check
let (val_loss, val_metrics) = if epoch % self.training_config.validation_frequency == 0 {
    self.validate_epoch(&mut val_loader, epoch).await?
} else {
    (0.0, ValidationMetrics::default())  // ← PROBLEM: Returns 0.0 when not validating
};

Issue: validation_frequency is set to 5 (from TFTTrainingConfig::default()), meaning validation only runs on epochs 0, 5, 10, etc. For epochs 1-4, 6-9, etc., val_loss is explicitly set to 0.0.

Secondary Issue: Even when validation runs, if val_loader.iter() returns no batches, batch_count stays 0, causing division by zero or 0.0 result.

---

2. GPU Not Being Used Despite --use-gpu Flag

Root Cause Analysis:

// Lines 278-285: Device selection
let device = if config.use_gpu {
    Device::cuda_if_available(0)
        .map_err(|e| MLError::ConfigError {
            reason: format!("GPU requested but not available: {}", e),
        })?
} else {
    Device::Cpu
};

Issues:

1. Insufficient Logging: Only logs "Using device: {:?}" - doesn't distinguish between CUDA success/fallback
2. Silent Fallback: cuda_if_available() returns Device::Cpu if CUDA fails - no warning to user
3. No Verification: Doesn't verify tensors are actually allocated on GPU
4. Pattern Matching: Doesn't pattern match on Device::Cuda(_) vs Device::Cpu variants

---

Comprehensive Fixes

Fix 1: Enhanced GPU Device Selection with Detailed Logging

Location: ml/src/trainers/tft.rs, lines 275-287

// SELECT DEVICE WITH DETAILED LOGGING AND VERIFICATION
let device = if config.use_gpu {
    match Device::cuda_if_available(0) {
        Ok(Device::Cuda(cuda_device)) => {
            info!("✓ GPU detected and enabled: CUDA device 0");
            info!("  All tensors will be allocated on GPU for accelerated training");
            info!("  Expected speedup: 5-10x vs CPU for TFT training");
            Device::Cuda(cuda_device)
        },
        Ok(Device::Cpu) => {
            warn!("⚠ GPU requested (use_gpu=true) but CUDA not available");
            warn!("  Falling back to CPU - training will be significantly slower");
            warn!("  Verify: nvidia-smi shows GPU and CUDA_HOME is set");
            warn!("  Check: LD_LIBRARY_PATH includes CUDA libraries");
            Device::Cpu
        },
        Err(e) => {
            return Err(MLError::ConfigError {
                reason: format!("GPU initialization failed: {}. Check CUDA drivers and runtime.", e),
            });
        },
        _ => {
            warn!("⚠ Unexpected device type from cuda_if_available, falling back to CPU");
            Device::Cpu
        },
    }
} else {
    info!("Using CPU device (use_gpu=false in config)");
    Device::Cpu
};

info!("Device initialization complete: {:?}", device);
info!("Device type: {}", match device {
    Device::Cuda(_) => "GPU (CUDA)",
    Device::Cpu => "CPU",
    _ => "Unknown",
});

Benefits:

• ✅ Clear distinction between GPU success, fallback, and failure
• ✅ Actionable debugging hints (nvidia-smi, CUDA_HOME, LD_LIBRARY_PATH)
• ✅ User knows immediately if GPU is actually being used
• ✅ Expected performance guidance (5-10x speedup)

---

Fix 2: Validation Loss Computation with Defensive Checks

Location: ml/src/trainers/tft.rs, lines 535-590

/// Validate single epoch WITH DEFENSIVE CHECKS FOR EMPTY BATCHES
async fn validate_epoch(
    &mut self,
    val_loader: &mut TFTDataLoader,
    epoch: usize,
) -> MLResult<(f64, ValidationMetrics)> {
    // DEFENSIVE CHECK: Empty validation loader
    if val_loader.len() == 0 {
        warn!("Validation loader is empty (epoch {}) - returning zero loss", epoch + 1);
        warn!("  This usually means: insufficient validation data or incorrect split");
        return Ok((0.0, ValidationMetrics::default()));
    }

    info!("Starting validation (epoch {}): {} batches", epoch + 1, val_loader.len());

    let mut total_loss = 0.0;
    let mut total_quantile_loss = 0.0;
    let mut total_rmse = 0.0;
    let mut attention_entropies = Vec::new();
    let mut batch_count = 0;
    let mut skipped_batches = 0;

    for batch in val_loader.iter() {
        // DEFENSIVE CHECK: Skip empty batches
        if batch.targets.nrows() == 0 || batch.static_features.nrows() == 0 {
            skipped_batches += 1;
            debug!("Skipping empty validation batch (rows: {})", batch.targets.nrows());
            continue;
        }

        // Convert batch to tensors (on correct device)
        let (static_tensor, hist_tensor, fut_tensor, target_tensor) =
            self.batch_to_tensors(batch)?;

        // Verify tensors are on correct device (first batch only)
        if batch_count == 0 {
            debug!("Validation tensors device check:");
            debug!("  static_tensor.device(): {:?}", static_tensor.device());
            debug!("  target_tensor.device(): {:?}", target_tensor.device());
            debug!("  Expected device: {:?}", self.device);
        }

        // Forward pass (no gradients needed for validation)
        let predictions = self
            .model
            .forward(&static_tensor, &hist_tensor, &fut_tensor)?;

        // Compute quantile loss
        let loss = self.compute_quantile_loss(&predictions, &target_tensor)?;
        let loss_value = loss.to_vec0::<f32>()? as f64;
        total_loss += loss_value;
        total_quantile_loss += loss_value;

        // Compute RMSE
        let rmse = self.compute_rmse(&predictions, &target_tensor)?;
        total_rmse += rmse;

        // Extract attention statistics (if available)
        if let Some(entropy) = self.extract_attention_entropy()? {
            attention_entropies.push(entropy);
        }

        batch_count += 1;

        // Log progress for long validation
        if batch_count % 50 == 0 {
            debug!("Validation progress: {}/{} batches, avg_loss: {:.6}",
                   batch_count, val_loader.len(), total_loss / batch_count as f64);
        }
    }

    // DEFENSIVE CHECK: Zero non-empty batches processed
    if batch_count == 0 {
        warn!(
            "Validation completed with ZERO non-empty batches (epoch {}, skipped: {})",
            epoch + 1, skipped_batches
        );
        warn!("  Check validation data quality and batch generation logic");
        return Ok((0.0, ValidationMetrics::default()));
    }

    info!(
        "Validation complete (epoch {}): {} batches processed, {} skipped",
        epoch + 1, batch_count, skipped_batches
    );

    // Compute averages (safe - batch_count > 0)
    let avg_loss = total_loss / batch_count as f64;
    let avg_attention_entropy = if attention_entropies.is_empty() {
        0.0
    } else {
        attention_entropies.iter().sum::<f64>() / attention_entropies.len() as f64
    };

    let metrics = ValidationMetrics {
        quantile_loss: total_quantile_loss / batch_count as f64,
        rmse: total_rmse / batch_count as f64,
        attention_entropy: avg_attention_entropy,
    };

    info!(
        "Validation metrics (epoch {}): loss: {:.6}, rmse: {:.6}, quantile_loss: {:.6}",
        epoch + 1, avg_loss, metrics.rmse, metrics.quantile_loss
    );

    Ok((avg_loss, metrics))
}

Key Improvements:

1. ✅ Empty Loader Check: Returns 0.0 immediately if val_loader.len() == 0
2. ✅ Empty Batch Skipping: Skips batches with 0 rows (corrupted data)
3. ✅ Zero Batch Count Check: Returns 0.0 if all batches skipped
4. ✅ Device Verification: Logs tensor device on first batch (debugging)
5. ✅ Progress Logging: Shows validation progress every 50 batches
6. ✅ Comprehensive Logging: Info logs for start/end, debug for progress

---

Fix 3: Tensor Device Verification in batch_to_tensors

Location: ml/src/trainers/tft.rs, lines 592-625

/// Convert batch to tensors WITH DEVICE VERIFICATION
fn batch_to_tensors(&self, batch: &TFTBatch) -> MLResult<(Tensor, Tensor, Tensor, Tensor)> {
    // Log device allocation (first call only - use static flag in production)
    debug!(
        "Creating tensors on device: {:?} (batch_size: {}, seq_len: {})",
        self.device,
        batch.static_features.nrows(),
        batch.historical_features.shape()[1]
    );

    // Convert ndarray to tensors - ALL on self.device (CPU or CUDA)
    let static_data: Vec<f32> = batch.static_features.iter().map(|&x| x as f32).collect();
    let static_tensor = Tensor::from_slice(
        &static_data,
        batch.static_features.raw_dim().into_pattern(),
        &self.device,  // ← CRITICAL: Must use self.device
    )?;

    let hist_data: Vec<f32> = batch
        .historical_features
        .iter()
        .map(|&x| x as f32)
        .collect();
    let hist_tensor = Tensor::from_slice(
        &hist_data,
        batch.historical_features.raw_dim().into_pattern(),
        &self.device,  // ← CRITICAL: Must use self.device
    )?;

    let fut_data: Vec<f32> = batch.future_features.iter().map(|&x| x as f32).collect();
    let fut_tensor = Tensor::from_slice(
        &fut_data,
        batch.future_features.raw_dim().into_pattern(),
        &self.device,  // ← CRITICAL: Must use self.device
    )?;

    let target_data: Vec<f32> = batch.targets.iter().map(|&x| x as f32).collect();
    let target_tensor = Tensor::from_slice(
        &target_data,
        batch.targets.raw_dim().into_pattern(),
        &self.device,  // ← CRITICAL: Must use self.device
    )?;

    // Verify all tensors on same device (debug mode)
    debug_assert_eq!(
        static_tensor.device().location(),
        self.device.location(),
        "static_tensor not on expected device"
    );
    debug_assert_eq!(
        hist_tensor.device().location(),
        self.device.location(),
        "hist_tensor not on expected device"
    );
    debug_assert_eq!(
        fut_tensor.device().location(),
        self.device.location(),
        "fut_tensor not on expected device"
    );
    debug_assert_eq!(
        target_tensor.device().location(),
        self.device.location(),
        "target_tensor not on expected device"
    );

    Ok((static_tensor, hist_tensor, fut_tensor, target_tensor))
}

Key Improvements:

• ✅ Debug logging shows device and batch dimensions
• ✅ All tensors explicitly created on self.device (no hardcoded Device::Cpu)
• ✅ Debug assertions verify device consistency
• ✅ Comments highlight critical device parameter

---

Fix 4: Run Validation Every Epoch (Not Every 5th)

Location: ml/src/tft/training.rs, line 94

// BEFORE (validation every 5 epochs)
validation_frequency: 5,

// AFTER (validation every epoch)
validation_frequency: 1,

Rationale: User expects validation loss every epoch. Current setting (validation_frequency: 5) causes 0.0 loss for epochs 1-4, 6-9, etc.

---

Testing Strategy

1. GPU Verification Test

# Monitor GPU usage during training
watch -n 1 nvidia-smi

# Expected output during training:
# GPU 0: NVIDIA GeForce RTX 3050 Ti
#   GPU-Util: 30-80% (should be >30% during forward/backward pass)
#   Memory-Usage: 1500-3000 MB (depends on batch size)
#   Processes: ml_training_service (or rust process)

2. Validation Loss Test (10 Epochs)

# Run 10-epoch training with validation_frequency=1
cargo run -p ml --example comprehensive_model_backtest -- \
  --model TFT \
  --epochs 10 \
  --batch-size 32 \
  --use-gpu \
  --validation-freq 1

# Expected output:
# Epoch 1/10: Train Loss: X.XXXXXX, Val Loss: Y.YYYYYY (non-zero)
# Epoch 2/10: Train Loss: X.XXXXXX, Val Loss: Y.YYYYYY (non-zero)
# ...
# Epoch 10/10: Train Loss: X.XXXXXX, Val Loss: Y.YYYYYY (non-zero)

3. Success Criteria

Metric	Requirement	Verification
GPU Utilization	>30% during training	nvidia-smi shows GPU-Util >30%
Validation Loss	Non-zero every epoch	All 10 epochs show val_loss > 0.0
GPU Memory	1.5-3.0 GB allocated	nvidia-smi shows process memory usage
Speedup	5-10x vs CPU	Compare epoch duration GPU vs CPU
Device Logging	Clear GPU/CPU status	Logs show "✓ GPU detected" or "⚠ Falling back to CPU"

---

Performance Expectations

GPU Training (RTX 3050 Ti, 4GB VRAM)

Configuration	Expected Epoch Time	GPU Utilization	Memory Usage
Batch Size 16	30-60 seconds	40-60%	1.2-1.8 GB
Batch Size 32	60-120 seconds	50-70%	1.8-2.5 GB
Batch Size 64	120-240 seconds	60-80%	2.5-3.5 GB

CPU Training (Baseline)

Configuration	Expected Epoch Time	CPU Utilization
Batch Size 16	5-10 minutes	80-100%
Batch Size 32	10-20 minutes	80-100%

Expected GPU Speedup: 5-10x faster than CPU

---

Implementation Checklist

• Fix 1: Enhanced GPU device selection with pattern matching and detailed logging
• Fix 2: Validation loss computation with defensive checks for empty batches
• Fix 3: Tensor device verification in batch_to_tensors
• Fix 4: Change validation_frequency from 5 to 1 in TFTTrainingConfig::default()
• Test: Run 10-epoch training with GPU
• Verify: GPU utilization >30% in nvidia-smi
• Verify: Validation loss non-zero every epoch
• Measure: Epoch duration and compare with CPU baseline

---

Next Steps

1. Apply Fixes: Implement all 4 fixes in ml/src/trainers/tft.rs and ml/src/tft/training.rs
2. Run GPU Benchmark: 10 epochs with batch_size=32, use_gpu=true
3. Monitor nvidia-smi: Verify GPU utilization >30% during training
4. Validate Logs: Confirm "✓ GPU detected" message appears at startup
5. Check Val Loss: Verify all 10 epochs show non-zero validation loss
6. Measure Speedup: Compare epoch duration with CPU baseline (should be 5-10x faster)

---

Known Limitations

1. Empty Validation Data: If validation loader genuinely has no data, returns 0.0 (expected behavior)
2. First Epoch Slowdown: First epoch may be slower due to CUDA initialization overhead
3. Memory Constraints: Batch size >64 may exceed 4GB VRAM on RTX 3050 Ti
4. CPU Fallback: If CUDA unavailable, falls back to CPU with warning (not an error)

---

Files Modified

1. ml/src/trainers/tft.rs (lines 275-287, 535-625)

   • Enhanced GPU device selection
   • Validation loss defensive checks
   • Tensor device verification

2. ml/src/tft/training.rs (line 94)

   • Change validation_frequency: 5 → validation_frequency: 1

---

Status: ✅ READY FOR IMPLEMENTATION Estimated Implementation Time: 15-20 minutes Estimated Testing Time: 30-60 minutes (10-epoch GPU training) Total Time: 45-80 minutes to complete validation

---

End of Report