foxhunt/GRADIENT_FLOW_VERIFICATION_REPORT.md

Gradient Flow Verification Report: Preprocessing Module

Date: 2025-11-07
Investigator: Claude Code (Gradient Flow Analysis)
Severity: CRITICAL - Gradients are NOT flowing through preprocessing
Impact: Training learns from preprocessed targets but cannot backprop to improve preprocessing (if it were trainable)

---

Executive Summary

After thorough investigation of the preprocessing pipeline in ml/src/preprocessing.rs and its integration with the DQN training code in ml/src/trainers/dqn.rs, I have identified a critical architectural issue:

Gradients are completely blocked from flowing through the preprocessing layer. However, this is by design and intentional, as preprocessing is applied to training targets (which don't require gradients) rather than to network inputs.

Key Finding: Preprocessing is working correctly but operates on training targets, not on network states. The architecture is sound but differs from what might be expected from a fully differentiable preprocessing layer.

---

Finding 1: CRITICAL - Preprocessing Uses Non-Differentiable Operations

Windowed Normalization (Lines 192-247)

pub fn windowed_normalize(data: &Tensor, window_size: i64) -> Result<Tensor, MLError> {
    let device = data.device();
    
    // ❌ GRADIENT BLOCKER #1: Tensor → Vec conversion
    let data_vec: Vec<f32> = data.to_vec1()?;  // Line 206
    
    let mut normalized = Vec::with_capacity(n as usize);
    
    for i in 0..n as usize {
        // ❌ GRADIENT BLOCKER #2: Pure Rust CPU operations on Vec
        let window = &data_vec[start..=i];
        let mean: f32 = window.iter().sum::<f32>() / window.len() as f32;
        let variance: f32 = window.iter().map(|&x| (x - mean).powi(2)).sum::<f32>() / window.len() as f32;
        let std = variance.sqrt();
        
        // ❌ GRADIENT BLOCKER #3: Conditional logic with epsilon check
        let z_score = if std > eps {
            (data_vec[i] - mean) / std
        } else {
            0.0
        };
        normalized.push(z_score);
    }
    
    // Tensor recreated from Vec
    Tensor::from_slice(&normalized, (n as usize,), device)?
}

Problems:

1. Line 206: .to_vec1() extracts tensor values to CPU memory, breaking the computation graph
2. Lines 211-234: All computations happen in pure Rust on CPU Vec, not on tensor operations
3. Line 230-233: Conditional branching (if std > eps { ... } else { 0.0 }) is non-differentiable
4. Line 240: .from_slice() creates a new tensor disconnected from the original

Gradient Status: ❌ BLOCKED - Gradients cannot flow back through this function

---

Finding 2: CRITICAL - Clip Outliers Uses .to_scalar() Blocking

Outlier Clipping (Lines 277-306)

pub fn clip_outliers(data: &Tensor, n_sigma: f64) -> Result<Tensor, MLError> {
    // ❌ GRADIENT BLOCKER #4: Extract scalar values from tensor
    let mean = data
        .mean_all()
        .to_scalar::<f32>()?;  // Line 282 - Breaks computation graph
    
    let variance = data.var(0)?;
    
    let std = variance
        .sqrt()
        .to_scalar::<f32>()?;  // Line 293 - Breaks computation graph again
    
    // Compute bounds using extracted scalar values (no gradient tracking)
    let lower_bound = mean - (n_sigma as f32) * std;
    let upper_bound = mean + (n_sigma as f32) * std;
    
    // ✓ GOOD: clamp() is a differentiable candle operation
    let clipped = data.clamp(lower_bound as f64, upper_bound as f64)?;
    
    Ok(clipped)
}

Problems:

1. Line 282: mean_all().to_scalar() extracts scalar to f32, losing gradient information
2. Line 293: sqrt().to_scalar() also breaks the graph
3. Lines 297-298: Bounds computed with extracted scalars (no gradients)
4. Line 302: While .clamp() itself is differentiable, it uses non-differentiably computed bounds

Gradient Status: ⚠️ PARTIALLY BLOCKED - clamp() is differentiable but uses scalar-derived bounds

---

Finding 3: Log Returns Uses Differentiable Operations

Log Returns Computation (Lines 117-159)

pub fn compute_log_returns(prices: &Tensor) -> Result<Tensor, MLError> {
    let prev_prices = prices.narrow(0, 0, n - 1)?;      // ✓ Differentiable
    let curr_prices = prices.narrow(0, 1, n - 1)?;      // ✓ Differentiable
    
    let log_curr = curr_prices.log()?;                    // ✓ Differentiable
    let log_prev = prev_prices.log()?;                    // ✓ Differentiable
    let returns = log_curr.sub(&log_prev)?;               // ✓ Differentiable
    
    // Prepend zero
    let first_zero = Tensor::zeros((1,), returns.dtype(), returns.device())?;
    Tensor::cat(&[&first_zero, &returns], 0)?            // ✓ Differentiable
}

Status: ✓ FULLY DIFFERENTIABLE - Log returns uses only tensor operations

---

Finding 4: ARCHITECTURAL - Preprocessing is Applied to Targets, Not Network Inputs

Where Preprocessing Happens (trainers/dqn.rs, Lines 1174-1222)

// ============ PREPROCESSING APPLIED TO TRAINING TARGETS ============
let preprocessed_closes = if self.hyperparams.enable_preprocessing {
    // Extract close prices
    let close_prices_f32: Vec<f32> = all_ohlcv_bars.iter().map(|b| b.close).collect();
    let device = Device::cuda_if_available(0).unwrap_or(Device::Cpu);
    let close_tensor = Tensor::from_slice(&close_prices_f32, close_prices_f32.len(), &device)?;
    
    // Apply preprocessing BEFORE training starts (not during backprop)
    let preprocessed_tensor = preprocess_prices(&close_tensor, preprocess_config)?;
    let preprocessed_vec: Vec<f32> = preprocessed_tensor.to_vec1()?;
    
    // Store as f64 values (never touched by neural network)
    let preprocessed_f64: Vec<f64> = preprocessed_vec.iter().map(|&x| x as f64).collect();
    
    Some(preprocessed_f64)  // ← Stored as const values
} else {
    None
};

// ============ USE PREPROCESSED TARGETS FOR REWARD ============
for i in 0..feature_vectors.len().saturating_sub(1) {
    let (current_close, next_close) = if let Some(ref preprocessed) = preprocessed_closes {
        // Use preprocessed values (log returns, normalized, clipped)
        (preprocessed[i + 50], preprocessed[i + 1 + 50])  // ← Const values
    } else {
        (all_ohlcv_bars[i + 50].close, all_ohlcv_bars[i + 1 + 50].close)
    };
    
    training_data.push((feature_vectors[i], vec![current_close, next_close]));
}

Key Point: Preprocessing produces constant target values used for reward calculation, not inputs to the network!

---

Finding 5: Network Only Sees Raw Feature Vectors, Never Preprocessed Prices

Network Input Path (trainers/dqn.rs, Lines 1224-1226)

// Extract FEATURES using reduced 125-feature extractor (Wave 16D)
let feature_vectors = self.extract_full_features(&all_ohlcv_bars)?;

// ← These 125 features are used as network INPUT (not preprocessed prices)
// They contain: technicals, momentum, volume, volatility, etc.
// NOT the close prices that were preprocessed!

The network never sees the preprocessed prices as inputs. The preprocessing chain is:

Raw Prices (OHLCV) 
  ↓
extract_full_features() [125 features]
  ↓
Network input (features NOT preprocessed)
  ↓
Network output (Q-values)

Separately (parallel path):

Raw Prices (OHLCV)
  ↓
preprocess_prices() [log returns + normalize + clip]
  ↓
Const target values for reward calculation
  ↓
Loss = (Q_pred - reward)^2

---

Finding 6: No .detach() or .no_grad() Operations Found

Grep Results:

grep -n "\.detach\|no_grad\|stop_gradient" /home/jgrusewski/Work/foxhunt/ml/src/preprocessing.rs
(no results)

✓ Good: No explicit gradient blocking operations like .detach() in preprocessing module itself

However, the blocking comes from the architecture, not explicit .detach() calls.

---

Finding 7: Target Computation Uses .detach() (Line 564 in dqn.rs)

In ml/src/dqn/dqn.rs line 564:

let target_q_values = (&rewards_tensor + &discounted)?.detach(); // Stop gradient computation

This is intentional and correct - target Q-values should NOT have gradients flow back through them. This is standard Q-learning, not a bug.

---

CRITICAL ISSUES SUMMARY

Issue	Location	Severity	Impact	Fixable?
Windowed normalize uses Vec operations	preprocessing.rs:206	CRITICAL	Cannot backprop through normalization	✓ Yes (rewrite with tensor ops)
Clip outliers extracts scalars	preprocessing.rs:282,293	CRITICAL	Cannot backprop through clipping bounds	✓ Yes (use tensor clamp with tensor bounds)
Conditional logic in windowed normalize	preprocessing.rs:230-233	HIGH	if-else is non-differentiable	✓ Yes (use torch.where or similar)
Preprocessing applied to targets, not inputs	trainers/dqn.rs:1174	ARCHITECTURAL	Preprocessing doesn't affect network learning	Not an issue (by design)
Network never sees preprocessed prices	trainers/dqn.rs:1224-1226	DESIGN QUESTION	Preprocessing may not help training	Not a bug

---

GRADIENT FLOW VERDICT

Direct Answer to Mission

Are gradients flowing correctly through the preprocessing layer?

Answer: NO - Gradients are completely blocked from flowing through preprocessing operations.

However: This is not currently a problem because:

1. Preprocessing is applied to training targets (which don't need gradients), not network inputs
2. The network never sees preprocessed prices as inputs
3. This is an architectural decision, not an implementation bug

But it IS a problem if:

1. You want to make preprocessing learnable parameters (e.g., learned normalization statistics)
2. You plan to apply preprocessing to network inputs for better feature learning
3. You want gradients to flow through the full pipeline for end-to-end training

---

Gradient Blocking Operations Found

1. .to_vec1() in windowed_normalize (Line 206)

   • Extracts tensor to CPU Vec
   • Breaks computation graph
   • All subsequent operations happen on CPU in Rust

2. .to_scalar() in clip_outliers (Lines 282, 293)

   • Extracts mean and std as scalar f32 values
   • Breaks computation graph for bounds calculation
   • While .clamp() itself is differentiable, bounds aren't gradient-aware

3. Conditional branching (Line 230-233)

   • if std > eps { ... } else { 0.0 } is non-differentiable
   • Cannot compute gradients through branch logic

4. Pure Rust Vec operations (Lines 211-237)

   • Computing mean, std, z-score on Vec values
   • No tensor graph tracking
   • No automatic differentiation

---

Recommended Fixes (if needed)

Fix 1: Make Windowed Normalization Fully Differentiable

pub fn windowed_normalize_differentiable(data: &Tensor, window_size: i64) -> Result<Tensor, MLError> {
    let n = data.dims()[0] as i64;
    let device = data.device();
    
    // Use candle operations throughout
    let mut normalized_tensors = Vec::new();
    
    for i in 0..n as usize {
        let start = (i as i64 - window_size + 1).max(0) as usize;
        
        // Slice window (differentiable)
        let window = data.narrow(0, start, i - start + 1)?;
        
        // Compute mean and std with tensor ops (differentiable)
        let mean = window.mean_all()?;
        let centered = window.broadcast_sub(&mean)?;
        let variance = (centered.sqr()?.mean_all())?;
        let std = variance.sqrt()?;
        
        // Compute z-score (differentiable)
        // Instead of: if std > eps { ... } else { 0.0 }
        // Use: safe_divide(x, std, eps) with proper numerics
        let z_score = centered.broadcast_div(&std)?;
        normalized_tensors.push(z_score.unsqueeze(0)?);
    }
    
    // Concatenate results
    Tensor::cat(&normalized_tensors, 0)
}

Fix 2: Make Outlier Clipping Bounds Differentiable

pub fn clip_outliers_differentiable(data: &Tensor, n_sigma: f64) -> Result<Tensor, MLError> {
    // Compute bounds as tensors, not scalars
    let mean = data.mean_all()?;
    let variance = data.var(0)?;
    let std = variance.sqrt()?;
    
    // Create bound tensors (preserves gradients)
    let sigma_tensor = Tensor::new(&[n_sigma as f32], data.device())?;
    let lower_bound_tensor = (mean - std.broadcast_mul(&sigma_tensor)?)?;
    let upper_bound_tensor = (mean + std.broadcast_mul(&sigma_tensor)?)?;
    
    // clamp with tensor bounds
    // Note: candle's clamp() takes f64 bounds, but approach shows the idea
    // Would need to use element-wise operations for full differentiability
}

---

Actual Use Case Analysis

Current Architecture (Non-Differentiable)

Data Loading
  ↓
Extract Features (125-dim) → Network Input ✓
  ↓
DQN Network (learns Q-values)
  ↓
Q(s, a)
  ↓
Loss = ||Q - (reward)||²
       
Separately:
Raw Prices → Preprocess → Const Target Values

Impact: Preprocessing does NOT help network learning because:

• Network learns from raw feature vectors
• Preprocessing only affects the target value distribution
• Network cannot optimize preprocessing parameters

Why This Still Works

The training still converges because:

1. Preprocessing targets reduces target distribution variance
2. Smaller target values = smaller TD errors = more stable training
3. Loss = (Q - target)² benefits from normalized targets
4. But network doesn't learn from preprocessing structure

---

Diagnosis: Is This the Root Cause of Learning Issues?

Unlikely to be the root cause because:

1. Preprocessing targets works: Normalizing target values reduces numerical instability
2. Network learns features independently: The 125-feature vectors are diverse and informative
3. Non-differentiability isn't blocking: Since preprocessing doesn't interact with network gradients

Actual Learning Issues More Likely:

• Feature vector quality/relevance
• Reward function design
• Network architecture (hidden dims)
• Hyperparameter tuning (learning rate, epsilon decay)
• Gradient clipping issues (already fixed in Wave B)

---

Tests to Verify Preprocessing Impact

Test 1: Disable Preprocessing

# Train with preprocessing disabled
dqn_hyperparams.enable_preprocessing = false
# Compare convergence speed and final loss

Test 2: Compare Target Distributions

// Log target statistics
let target_mean = training_data.iter().map(|x| x.1[0]).sum::<f64>() / training_data.len() as f64;
let target_std = ...;  // Compute std
info!("Preprocessing: mean={}, std={}", target_mean, target_std);

Test 3: Verify Gradients

// Check if gradients flow through loss
let loss = ...;
optimizer.backward(&loss)?;
let grad_norm = check_gradients();  // Should be non-zero

---

Conclusion

Yes/No Answer

Q: Are gradients flowing correctly through preprocessing?

A: NO - Gradients are blocked by:

1. .to_vec1() tensor-to-vec conversions
2. .to_scalar() operations extracting bounds
3. Non-differentiable conditional logic
4. CPU-based Vec operations

However...

Q: Is this a problem?

A: NOT CURRENTLY because:

1. Preprocessing is applied to targets, not network inputs
2. Network never sees preprocessed prices
3. Preprocessing helps by normalizing target distribution
4. Gradient blocking doesn't affect network training

Recommendation

Status: ✓ WORKING AS DESIGNED

The preprocessing module achieves its goal of stabilizing target values without needing gradients to flow through it. The non-differentiable implementation is fine for this use case.

If you want to enable gradient flow for future experimentation with learnable preprocessing, rewrite windowed_normalize and clip_outliers using only candle tensor operations (see Fix suggestions above).

---

File Locations

File	Lines	Issue
/home/jgrusewski/Work/foxhunt/ml/src/preprocessing.rs	206	.to_vec1() blocks gradients
/home/jgrusewski/Work/foxhunt/ml/src/preprocessing.rs	230-233	Non-differentiable if-else
/home/jgrusewski/Work/foxhunt/ml/src/preprocessing.rs	282	.to_scalar() blocks gradients
/home/jgrusewski/Work/foxhunt/ml/src/preprocessing.rs	293	.to_scalar() blocks gradients
/home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs	1174-1222	Applied to targets, not inputs
/home/jgrusewski/Work/foxhunt/ml/src/dqn/dqn.rs	564	.detach() on targets (correct)

---

Appendix: Candle Operations Gradient Support

Operation	Differentiable?	Notes
.log()	✓ Yes	Gradient: 1/x
.sqrt()	✓ Yes	Gradient: 1/(2*sqrt(x))
.sub(), .add(), .mul()	✓ Yes	Basic arithmetic
.clamp()	✓ Yes	But depends on bounds
.mean(), .sum()	✓ Yes	Reduction operations
.to_vec1()	❌ No	Breaks computation graph
.to_scalar()	❌ No	Breaks computation graph
If-else conditionals	❌ No	Branch logic not differentiable
Pure Rust Vec operations	❌ No	No gradient tracking