foxhunt/TFT_TARGET_NORMALIZATION_FIX.md

TFT Target Normalization Fix - Complete Implementation

Status: ✅ IMPLEMENTED Date: 2025-01-28 Priority: P0 CRITICAL

---

Problem Statement

TFT training suffered from massive loss values (1000-10000) due to a critical scale mismatch:

• Features: Z-score normalized via log returns (~-0.1 to 0.1)
• Targets: Raw ES futures prices (4500-5500)
• Scale mismatch: ~50,000x difference

This caused:

1. Gradient explosion
2. Training instability
3. Impossibly large loss values
4. Model unable to converge

---

Root Cause Analysis

Code Evidence

Features (NORMALIZED) - /home/jgrusewski/Work/foxhunt/ml/src/features/extraction.rs:261-266:

fn extract_ohlcv_features(&self, out: &mut [f64]) -> Result<()> {
    out[0] = safe_log_return(bar.open, prev_close);   // ~-0.1 to 0.1
    out[1] = safe_log_return(bar.high, prev_close);   // ~-0.1 to 0.1
    out[2] = safe_log_return(bar.low, prev_close);    // ~-0.1 to 0.1
    out[3] = safe_log_return(bar.close, prev_close);  // ~-0.1 to 0.1
}

Targets (RAW PRICES - BEFORE FIX) - /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft_parquet.rs:384-389:

let mut targets = Vec::new();
for j in (i + LOOKBACK)..(i + LOOKBACK + HORIZON) {
    targets.push(all_ohlcv_bars[j + 50].close); // 4500-5500 RAW!!!
}

---

Solution Implementation

1. Added Normalization Parameters Storage

File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft_parquet.rs

/// Normalization parameters for target denormalization
#[derive(Debug, Clone)]
pub struct NormalizationParams {
    pub mean: f64,
    pub std: f64,
}

2. Updated TFTTrainer Struct

File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs:250-256

pub struct TFTTrainer {
    // ... existing fields

    /// Target normalization parameters (for denormalizing predictions)
    pub target_mean: Option<f64>,
    pub target_std: Option<f64>,
}

3. Compute Normalization Parameters

File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft_parquet.rs:184-210

// Compute normalization parameters from close prices
info!("Computing target normalization parameters...");
let all_closes: Vec<f64> = all_ohlcv_bars.iter().map(|b| b.close).collect();

if all_closes.is_empty() {
    return Err(MLError::InsufficientData(
        "No close prices available for normalization".to_string()
    ));
}

let price_mean = all_closes.iter().sum::<f64>() / all_closes.len() as f64;
let price_variance = all_closes
    .iter()
    .map(|c| (c - price_mean).powi(2))
    .sum::<f64>() / all_closes.len() as f64;
let price_std = price_variance.sqrt();

// Validate normalization params
if price_std < 1e-8 {
    return Err(MLError::InvalidInput(
        format!("Price std_dev too small ({:.2e}), data may be constant", price_std)
    ));
}

info!(
    "Target normalization: mean={:.2}, std={:.2} (z-score will bring targets to ~[-3, 3] scale)",
    price_mean, price_std
);

// Store normalization params in trainer for denormalization during evaluation
self.target_mean = Some(price_mean);
self.target_std = Some(price_std);

4. Apply Z-Score Normalization to Targets

File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft_parquet.rs:254-261

// Targets: Next 10 close prices (Z-SCORE NORMALIZED)
let mut targets = Vec::new();
for j in (i + LOOKBACK)..(i + LOOKBACK + HORIZON) {
    let raw_price = all_ohlcv_bars[j + 50].close;
    // Apply z-score normalization: (price - mean) / std
    // This brings targets to ~[-3, 3] scale, matching log-return features
    let normalized = (raw_price - price_mean) / (price_std + 1e-8);
    targets.push(normalized);
}

5. Initialize Fields in Constructor

File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs:712-713

use_gradient_checkpointing: config.use_gradient_checkpointing,
target_mean: None,
target_std: None,

---

Expected Outcomes

Before Fix

• Training loss: 1,000 - 10,000
• Predictions: Meaningless (exploding gradients)
• Convergence: Impossible

After Fix

• Training loss: < 10.0 (normalized scale)
• Predictions: -3 to 3 (normalized) → 4500-5500 (denormalized)
• Convergence: Stable, rapid

---

Validation Checklist

• Normalization params computed from training data
• Z-score applied to targets: (price - mean) / std
• Params stored in TFTTrainer for denormalization
• Epsilon added for numerical stability (1e-8)
• Validation for zero std_dev
• Code compiles without errors
• Run 5-epoch training test
• Verify loss < 10.0
• Verify predictions in reasonable range
• Add denormalization for evaluation metrics

---

Next Steps

IMMEDIATE (30 min)

1. Test Training:

cargo run -p ml --example train_tft_parquet --release --features cuda -- \
  --parquet-file test_data/ES_FUT_180d.parquet --epochs 5

2. Verify Metrics:
   • Loss should be < 10.0 after epoch 1
   • Loss should decrease consistently
   • No NaN/Inf in gradients

SHORT-TERM (2-4 hours)

3. Add Denormalization for Evaluation:

   • Update validation loop in tft.rs
   • Denormalize predictions: pred * std + mean
   • Compute MAE/RMSE in original $ scale
   • Log both normalized and denormalized metrics

4. Add Checkpoint Persistence:

   • Save target_mean and target_std in checkpoint metadata
   • Load params when resuming training
   • Essential for inference in production

Example Denormalization Code:

// In validation loop (tft.rs)
if let (Some(mean), Some(std)) = (self.target_mean, self.target_std) {
    let denorm_pred = prediction * std + mean;
    let denorm_target = target * std + mean;

    // Compute metrics in original scale
    let mae_dollars = (denorm_pred - denorm_target).abs();
    info!("MAE (original scale): ${:.2}", mae_dollars);
}

---

Files Modified

1. /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft_parquet.rs

   • Added NormalizationParams struct
   • Compute normalization params from training data
   • Apply z-score to targets
   • Changed load_training_data_from_parquet to &mut self

2. /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs

   • Added target_mean: Option<f64> field
   • Added target_std: Option<f64> field
   • Initialize fields in constructor

---

Technical Details

Normalization Formula

z-score: normalized = (value - mean) / (std + epsilon)
denormalization: original = normalized * std + mean

Why Z-Score?

1. Scale Matching: Brings targets to same scale as features (-3 to 3)
2. Gradient Stability: Prevents explosion/vanishing
3. Order Preservation: Quantile loss still works (monotonic transformation)
4. Industry Standard: Common practice for time-series forecasting

Numerical Stability

• Epsilon (1e-8) prevents division by zero
• Validation for zero std_dev catches constant data
• Clipping not needed (z-score naturally bounded for normal distributions)

---

References

• Expert Analysis: Gemini 2.5 Pro validation (continuation_id: 5d01638f-de33-4c12-a250-e6dbab90565b)
• Original Issue: CLAUDE.md P0 priority
• Related Fixes: MAMBA-2 also needs same fix (separate ticket)

---

Success Criteria

✅ PASS: Loss < 10.0, stable training, reasonable predictions ❌ FAIL: Loss > 100, NaN/Inf, divergence

Status: Implementation complete, pending validation testing