foxhunt/GRADIENT_FLOW_VERIFICATION_REPORT.md

# Gradient Flow Verification Report: Preprocessing Module

**Date**: 2025-11-07  
**Investigator**: Claude Code (Gradient Flow Analysis)  
**Severity**: CRITICAL - Gradients are NOT flowing through preprocessing  
**Impact**: Training learns from preprocessed targets but cannot backprop to improve preprocessing (if it were trainable)

---

## Executive Summary

After thorough investigation of the preprocessing pipeline in `ml/src/preprocessing.rs` and its integration with the DQN training code in `ml/src/trainers/dqn.rs`, I have identified a **critical architectural issue**: 

**Gradients are completely blocked from flowing through the preprocessing layer.** However, this is **by design and intentional**, as preprocessing is applied to training targets (which don't require gradients) rather than to network inputs.

**Key Finding**: Preprocessing **is working correctly** but operates on training targets, not on network states. The architecture is sound but differs from what might be expected from a fully differentiable preprocessing layer.

---

## Finding 1: CRITICAL - Preprocessing Uses Non-Differentiable Operations

### Windowed Normalization (Lines 192-247)

```rust
pub fn windowed_normalize(data: &Tensor, window_size: i64) -> Result<Tensor, MLError> {
    let device = data.device();
    
    // ❌ GRADIENT BLOCKER #1: Tensor → Vec conversion
    let data_vec: Vec<f32> = data.to_vec1()?;  // Line 206
    
    let mut normalized = Vec::with_capacity(n as usize);
    
    for i in 0..n as usize {
        // ❌ GRADIENT BLOCKER #2: Pure Rust CPU operations on Vec
        let window = &data_vec[start..=i];
        let mean: f32 = window.iter().sum::<f32>() / window.len() as f32;
        let variance: f32 = window.iter().map(|&x| (x - mean).powi(2)).sum::<f32>() / window.len() as f32;
        let std = variance.sqrt();
        
        // ❌ GRADIENT BLOCKER #3: Conditional logic with epsilon check
        let z_score = if std > eps {
            (data_vec[i] - mean) / std
        } else {
            0.0
        };
        normalized.push(z_score);
    }
    
    // Tensor recreated from Vec
    Tensor::from_slice(&normalized, (n as usize,), device)?
}
```

**Problems**:
1. **Line 206**: `.to_vec1()` extracts tensor values to CPU memory, breaking the computation graph
2. **Lines 211-234**: All computations happen in pure Rust on CPU Vec, not on tensor operations
3. **Line 230-233**: Conditional branching (`if std > eps { ... } else { 0.0 }`) is non-differentiable
4. **Line 240**: `.from_slice()` creates a new tensor disconnected from the original

**Gradient Status**: ❌ **BLOCKED** - Gradients cannot flow back through this function

---

## Finding 2: CRITICAL - Clip Outliers Uses `.to_scalar()` Blocking

### Outlier Clipping (Lines 277-306)

```rust
pub fn clip_outliers(data: &Tensor, n_sigma: f64) -> Result<Tensor, MLError> {
    // ❌ GRADIENT BLOCKER #4: Extract scalar values from tensor
    let mean = data
        .mean_all()
        .to_scalar::<f32>()?;  // Line 282 - Breaks computation graph
    
    let variance = data.var(0)?;
    
    let std = variance
        .sqrt()
        .to_scalar::<f32>()?;  // Line 293 - Breaks computation graph again
    
    // Compute bounds using extracted scalar values (no gradient tracking)
    let lower_bound = mean - (n_sigma as f32) * std;
    let upper_bound = mean + (n_sigma as f32) * std;
    
    // ✓ GOOD: clamp() is a differentiable candle operation
    let clipped = data.clamp(lower_bound as f64, upper_bound as f64)?;
    
    Ok(clipped)
}
```

**Problems**:
1. **Line 282**: `mean_all().to_scalar()` extracts scalar to f32, losing gradient information
2. **Line 293**: `sqrt().to_scalar()` also breaks the graph
3. **Lines 297-298**: Bounds computed with extracted scalars (no gradients)
4. **Line 302**: While `.clamp()` itself is differentiable, it uses non-differentiably computed bounds

**Gradient Status**: ⚠️ **PARTIALLY BLOCKED** - clamp() is differentiable but uses scalar-derived bounds

---

## Finding 3: Log Returns Uses Differentiable Operations

### Log Returns Computation (Lines 117-159)

```rust
pub fn compute_log_returns(prices: &Tensor) -> Result<Tensor, MLError> {
    let prev_prices = prices.narrow(0, 0, n - 1)?;      // ✓ Differentiable
    let curr_prices = prices.narrow(0, 1, n - 1)?;      // ✓ Differentiable
    
    let log_curr = curr_prices.log()?;                    // ✓ Differentiable
    let log_prev = prev_prices.log()?;                    // ✓ Differentiable
    let returns = log_curr.sub(&log_prev)?;               // ✓ Differentiable
    
    // Prepend zero
    let first_zero = Tensor::zeros((1,), returns.dtype(), returns.device())?;
    Tensor::cat(&[&first_zero, &returns], 0)?            // ✓ Differentiable
}
```

**Status**: ✓ **FULLY DIFFERENTIABLE** - Log returns uses only tensor operations

---

## Finding 4: ARCHITECTURAL - Preprocessing is Applied to Targets, Not Network Inputs

### Where Preprocessing Happens (trainers/dqn.rs, Lines 1174-1222)

```rust
// ============ PREPROCESSING APPLIED TO TRAINING TARGETS ============
let preprocessed_closes = if self.hyperparams.enable_preprocessing {
    // Extract close prices
    let close_prices_f32: Vec<f32> = all_ohlcv_bars.iter().map(|b| b.close).collect();
    let device = Device::cuda_if_available(0).unwrap_or(Device::Cpu);
    let close_tensor = Tensor::from_slice(&close_prices_f32, close_prices_f32.len(), &device)?;
    
    // Apply preprocessing BEFORE training starts (not during backprop)
    let preprocessed_tensor = preprocess_prices(&close_tensor, preprocess_config)?;
    let preprocessed_vec: Vec<f32> = preprocessed_tensor.to_vec1()?;
    
    // Store as f64 values (never touched by neural network)
    let preprocessed_f64: Vec<f64> = preprocessed_vec.iter().map(|&x| x as f64).collect();
    
    Some(preprocessed_f64)  // ← Stored as const values
} else {
    None
};

// ============ USE PREPROCESSED TARGETS FOR REWARD ============
for i in 0..feature_vectors.len().saturating_sub(1) {
    let (current_close, next_close) = if let Some(ref preprocessed) = preprocessed_closes {
        // Use preprocessed values (log returns, normalized, clipped)
        (preprocessed[i + 50], preprocessed[i + 1 + 50])  // ← Const values
    } else {
        (all_ohlcv_bars[i + 50].close, all_ohlcv_bars[i + 1 + 50].close)
    };
    
    training_data.push((feature_vectors[i], vec![current_close, next_close]));
}
```

**Key Point**: Preprocessing produces **constant target values** used for reward calculation, not inputs to the network!

---

## Finding 5: Network Only Sees Raw Feature Vectors, Never Preprocessed Prices

### Network Input Path (trainers/dqn.rs, Lines 1224-1226)

```rust
// Extract FEATURES using reduced 125-feature extractor (Wave 16D)
let feature_vectors = self.extract_full_features(&all_ohlcv_bars)?;

// ← These 125 features are used as network INPUT (not preprocessed prices)
// They contain: technicals, momentum, volume, volatility, etc.
// NOT the close prices that were preprocessed!
```

The **network never sees the preprocessed prices** as inputs. The preprocessing chain is:

```
Raw Prices (OHLCV) 
  ↓
extract_full_features() [125 features]
  ↓
Network input (features NOT preprocessed)
  ↓
Network output (Q-values)
```

Separately (parallel path):
```
Raw Prices (OHLCV)
  ↓
preprocess_prices() [log returns + normalize + clip]
  ↓
Const target values for reward calculation
  ↓
Loss = (Q_pred - reward)^2
```

---

## Finding 6: No `.detach()` or `.no_grad()` Operations Found

**Grep Results**:
```
grep -n "\.detach\|no_grad\|stop_gradient" /home/jgrusewski/Work/foxhunt/ml/src/preprocessing.rs
(no results)
```

✓ **Good**: No explicit gradient blocking operations like `.detach()` in preprocessing module itself

However, the blocking comes from the **architecture**, not explicit `.detach()` calls.

---

## Finding 7: Target Computation Uses `.detach()` (Line 564 in dqn.rs)

In `ml/src/dqn/dqn.rs` line 564:

```rust
let target_q_values = (&rewards_tensor + &discounted)?.detach(); // Stop gradient computation
```

This is **intentional and correct** - target Q-values should NOT have gradients flow back through them. This is standard Q-learning, not a bug.

---

## CRITICAL ISSUES SUMMARY

| Issue | Location | Severity | Impact | Fixable? |
|-------|----------|----------|--------|----------|
| **Windowed normalize uses Vec operations** | preprocessing.rs:206 | CRITICAL | Cannot backprop through normalization | ✓ Yes (rewrite with tensor ops) |
| **Clip outliers extracts scalars** | preprocessing.rs:282,293 | CRITICAL | Cannot backprop through clipping bounds | ✓ Yes (use tensor clamp with tensor bounds) |
| **Conditional logic in windowed normalize** | preprocessing.rs:230-233 | HIGH | if-else is non-differentiable | ✓ Yes (use torch.where or similar) |
| **Preprocessing applied to targets, not inputs** | trainers/dqn.rs:1174 | ARCHITECTURAL | Preprocessing doesn't affect network learning | Not an issue (by design) |
| **Network never sees preprocessed prices** | trainers/dqn.rs:1224-1226 | DESIGN QUESTION | Preprocessing may not help training | Not a bug |

---

## GRADIENT FLOW VERDICT

### Direct Answer to Mission

**Are gradients flowing correctly through the preprocessing layer?**

**Answer: NO** - Gradients are completely blocked from flowing through preprocessing operations.

**However**: This is **not currently a problem** because:
1. Preprocessing is applied to **training targets** (which don't need gradients), not network inputs
2. The network never sees preprocessed prices as inputs
3. This is an architectural decision, not an implementation bug

**But it IS a problem if**:
1. You want to make preprocessing learnable parameters (e.g., learned normalization statistics)
2. You plan to apply preprocessing to network inputs for better feature learning
3. You want gradients to flow through the full pipeline for end-to-end training

---

## Gradient Blocking Operations Found

1. **`.to_vec1()` in windowed_normalize (Line 206)**
   - Extracts tensor to CPU Vec
   - Breaks computation graph
   - All subsequent operations happen on CPU in Rust

2. **`.to_scalar()` in clip_outliers (Lines 282, 293)**
   - Extracts mean and std as scalar f32 values
   - Breaks computation graph for bounds calculation
   - While `.clamp()` itself is differentiable, bounds aren't gradient-aware

3. **Conditional branching (Line 230-233)**
   - `if std > eps { ... } else { 0.0 }` is non-differentiable
   - Cannot compute gradients through branch logic

4. **Pure Rust Vec operations (Lines 211-237)**
   - Computing mean, std, z-score on Vec values
   - No tensor graph tracking
   - No automatic differentiation

---

## Recommended Fixes (if needed)

### Fix 1: Make Windowed Normalization Fully Differentiable

```rust
pub fn windowed_normalize_differentiable(data: &Tensor, window_size: i64) -> Result<Tensor, MLError> {
    let n = data.dims()[0] as i64;
    let device = data.device();
    
    // Use candle operations throughout
    let mut normalized_tensors = Vec::new();
    
    for i in 0..n as usize {
        let start = (i as i64 - window_size + 1).max(0) as usize;
        
        // Slice window (differentiable)
        let window = data.narrow(0, start, i - start + 1)?;
        
        // Compute mean and std with tensor ops (differentiable)
        let mean = window.mean_all()?;
        let centered = window.broadcast_sub(&mean)?;
        let variance = (centered.sqr()?.mean_all())?;
        let std = variance.sqrt()?;
        
        // Compute z-score (differentiable)
        // Instead of: if std > eps { ... } else { 0.0 }
        // Use: safe_divide(x, std, eps) with proper numerics
        let z_score = centered.broadcast_div(&std)?;
        normalized_tensors.push(z_score.unsqueeze(0)?);
    }
    
    // Concatenate results
    Tensor::cat(&normalized_tensors, 0)
}
```

### Fix 2: Make Outlier Clipping Bounds Differentiable

```rust
pub fn clip_outliers_differentiable(data: &Tensor, n_sigma: f64) -> Result<Tensor, MLError> {
    // Compute bounds as tensors, not scalars
    let mean = data.mean_all()?;
    let variance = data.var(0)?;
    let std = variance.sqrt()?;
    
    // Create bound tensors (preserves gradients)
    let sigma_tensor = Tensor::new(&[n_sigma as f32], data.device())?;
    let lower_bound_tensor = (mean - std.broadcast_mul(&sigma_tensor)?)?;
    let upper_bound_tensor = (mean + std.broadcast_mul(&sigma_tensor)?)?;
    
    // clamp with tensor bounds
    // Note: candle's clamp() takes f64 bounds, but approach shows the idea
    // Would need to use element-wise operations for full differentiability
}
```

---

## Actual Use Case Analysis

### Current Architecture (Non-Differentiable)

```
Data Loading
  ↓
Extract Features (125-dim) → Network Input ✓
  ↓
DQN Network (learns Q-values)
  ↓
Q(s, a)
  ↓
Loss = ||Q - (reward)||²
       
Separately:
Raw Prices → Preprocess → Const Target Values
```

**Impact**: Preprocessing **does NOT help network learning** because:
- Network learns from raw feature vectors
- Preprocessing only affects the target value distribution
- Network cannot optimize preprocessing parameters

### Why This Still Works

The training **still converges** because:
1. Preprocessing targets reduces **target distribution variance**
2. Smaller target values = smaller TD errors = more stable training
3. Loss = (Q - target)² benefits from normalized targets
4. But network doesn't learn from preprocessing structure

---

## Diagnosis: Is This the Root Cause of Learning Issues?

**Unlikely to be the root cause** because:

1. **Preprocessing targets works**: Normalizing target values reduces numerical instability
2. **Network learns features independently**: The 125-feature vectors are diverse and informative
3. **Non-differentiability isn't blocking**: Since preprocessing doesn't interact with network gradients

**Actual Learning Issues More Likely**:
- Feature vector quality/relevance
- Reward function design
- Network architecture (hidden dims)
- Hyperparameter tuning (learning rate, epsilon decay)
- Gradient clipping issues (already fixed in Wave B)

---

## Tests to Verify Preprocessing Impact

### Test 1: Disable Preprocessing
```bash
# Train with preprocessing disabled
dqn_hyperparams.enable_preprocessing = false
# Compare convergence speed and final loss
```

### Test 2: Compare Target Distributions
```rust
// Log target statistics
let target_mean = training_data.iter().map(|x| x.1[0]).sum::<f64>() / training_data.len() as f64;
let target_std = ...;  // Compute std
info!("Preprocessing: mean={}, std={}", target_mean, target_std);
```

### Test 3: Verify Gradients
```rust
// Check if gradients flow through loss
let loss = ...;
optimizer.backward(&loss)?;
let grad_norm = check_gradients();  // Should be non-zero
```

---

## Conclusion

### Yes/No Answer

**Q: Are gradients flowing correctly through preprocessing?**

**A: NO** - Gradients are blocked by:
1. `.to_vec1()` tensor-to-vec conversions
2. `.to_scalar()` operations extracting bounds
3. Non-differentiable conditional logic
4. CPU-based Vec operations

### However...

**Q: Is this a problem?**

**A: NOT CURRENTLY** because:
1. Preprocessing is applied to targets, not network inputs
2. Network never sees preprocessed prices
3. Preprocessing helps by normalizing target distribution
4. Gradient blocking doesn't affect network training

### Recommendation

**Status**: ✓ **WORKING AS DESIGNED**

The preprocessing module achieves its goal of **stabilizing target values** without needing gradients to flow through it. The non-differentiable implementation is fine for this use case.

**If you want to enable gradient flow** for future experimentation with learnable preprocessing, rewrite `windowed_normalize` and `clip_outliers` using only candle tensor operations (see Fix suggestions above).

---

## File Locations

| File | Lines | Issue |
|------|-------|-------|
| `/home/jgrusewski/Work/foxhunt/ml/src/preprocessing.rs` | 206 | `.to_vec1()` blocks gradients |
| `/home/jgrusewski/Work/foxhunt/ml/src/preprocessing.rs` | 230-233 | Non-differentiable if-else |
| `/home/jgrusewski/Work/foxhunt/ml/src/preprocessing.rs` | 282 | `.to_scalar()` blocks gradients |
| `/home/jgrusewski/Work/foxhunt/ml/src/preprocessing.rs` | 293 | `.to_scalar()` blocks gradients |
| `/home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs` | 1174-1222 | Applied to targets, not inputs |
| `/home/jgrusewski/Work/foxhunt/ml/src/dqn/dqn.rs` | 564 | `.detach()` on targets (correct) |

---

## Appendix: Candle Operations Gradient Support

| Operation | Differentiable? | Notes |
|-----------|-----------------|-------|
| `.log()` | ✓ Yes | Gradient: 1/x |
| `.sqrt()` | ✓ Yes | Gradient: 1/(2*sqrt(x)) |
| `.sub()`, `.add()`, `.mul()` | ✓ Yes | Basic arithmetic |
| `.clamp()` | ✓ Yes | But depends on bounds |
| `.mean()`, `.sum()` | ✓ Yes | Reduction operations |
| `.to_vec1()` | ❌ No | Breaks computation graph |
| `.to_scalar()` | ❌ No | Breaks computation graph |
| If-else conditionals | ❌ No | Branch logic not differentiable |
| Pure Rust Vec operations | ❌ No | No gradient tracking |