foxhunt/docs/codebase-cleanup/agent20_reward_normalization_analysis.md

# Agent 20: Reward Normalization Analysis Report

**Date**: 2025-11-27
**Task**: Review reward normalization for potential overfitting issues
**Status**: ✅ COMPLETE - Analysis Finished

---

## Executive Summary

**VERDICT**: The reward normalization system is **well-designed** with appropriate parameters. However, there are **two potential concerns** related to the effective window size and normalization algorithm that warrant attention.

### Key Findings

1. ✅ **EMA Algorithm**: Correctly chosen over Welford's for non-stationary markets
2. ⚠️ **Window Size**: 100-step effective window may be too short for regime detection
3. ✅ **Normalization Range**: Proper clipping at ±3.0σ preserves economic signal
4. ⚠️ **Overfitting Risk**: EMA parameters (α=0.01, β=0.01) are fixed, not hyperparameter-tuned

---

## 1. RewardNormalizer Implementation Analysis

### Architecture (Lines 14-165 in `/ml/src/dqn/reward.rs`)

```rust
pub struct RewardNormalizer {
    mean: f64,              // Running mean via EMA
    variance: f64,          // Running variance via EMA
    alpha: 0.01,           // Mean decay rate (100-step window)
    beta: 0.01,            // Variance decay rate (100-step window)
    initialized: bool,
    epsilon: 1e-8,         // Numerical stability
}
```

### Algorithm: Exponential Moving Average (EMA)

**Update Rule** (Lines 101-114):
```rust
if !self.initialized {
    self.mean = value;
    self.variance = 1.0;
    self.initialized = true;
} else {
    // EMA update for mean
    self.mean = self.alpha * value + (1.0 - self.alpha) * self.mean;

    // EMA update for variance
    let diff = value - self.mean;
    self.variance = self.beta * diff.powi(2) + (1.0 - self.beta) * self.variance;
}
```

**Normalization** (Lines 121-135):
```rust
let std = self.variance.sqrt();
if std < self.epsilon {
    return value;
}
(value - self.mean) / std
```

---

## 2. Parameter Analysis

### 2.1 Window Size: α = β = 0.01 → Effective Window ≈ 100 Steps

**Effective Window Calculation**:
- For decay rate α, effective window = 1/α
- α = 0.01 → **100 steps**

**Decay Profile** (Lines 95-98):
```
After 100 steps: weight = (1-0.01)^100 = 0.366 (63.4% decay)
After 200 steps: weight = (1-0.01)^200 = 0.134 (86.6% decay)
After 500 steps: weight = (1-0.01)^500 = 0.007 (99.3% decay)
```

**Assessment**:
- ✅ **Reactive**: Adapts quickly to changing reward distributions
- ⚠️ **Too Short?**: May overfit to recent 100 episodes, missing longer-term trends
- ⚠️ **Regime Detection**: Trading regimes can last 500+ steps (hours/days)

**Recommendation**:
```rust
// Consider longer windows for regime stability
alpha: 0.001,  // 1000-step window (more stable)
beta: 0.005,   // 200-step window (variance adapts faster)
```

---

## 3. Normalization Range & Clipping

### 3.1 Clipping at ±3.0σ (Line 557)

```rust
let normalized_reward = if let Some(normalizer) = &mut self.normalizer {
    let norm = normalizer.normalize(final_reward_f64);
    normalizer.update(final_reward_f64);  // Update AFTER normalization
    norm.clamp(-3.0, 3.0)  // Clip to ±3σ
} else {
    final_reward_f64
};
```

**Assessment**:
- ✅ **Appropriate Range**: ±3σ covers 99.7% of normal distribution
- ✅ **Preserves Signal**: Wide enough to not distort economic information
- ✅ **Prevents Explosion**: Bounds extreme outliers (e.g., rare large P&L events)

**Expected Q-Values** (Line 556):
```
Q* = r / (1 - γ)
   ≈ 3.0 / (1 - 0.9626)
   ≈ 80
```

---

## 4. Overfitting Risk Assessment

### 4.1 Fixed vs. Tunable Parameters

**Current Implementation**:
```rust
alpha: 0.01,  // FIXED (hardcoded)
beta: 0.01,   // FIXED (hardcoded)
```

**Concern**: These are **not exposed as hyperparameters** in the reward config.

**Evidence from Codebase**:
- `RewardConfig` (Lines 167-213) does NOT include `alpha`/`beta` parameters
- Hyperopt tuning (checked via `ml/src/hyperopt/`) does NOT optimize EMA decay rates
- All trials use the same fixed 100-step window

### 4.2 Potential Overfitting Scenarios

#### Scenario A: Too Short Window (α = 0.01)
```
Market regime: Volatile (500 steps) → Calm (500 steps) → Volatile (500 steps)
EMA behavior: Quickly forgets old volatility, normalizes to recent 100 steps
Result: Rewards appear stable in volatile regime → agent underestimates risk
```

#### Scenario B: Too Long Window (α = 0.001)
```
Market regime: Calm (1000 steps) → Flash crash (10 steps) → Recovery (50 steps)
EMA behavior: Slow to react, uses stale calm-regime statistics
Result: Crash rewards seem extreme → agent avoids trading during recovery
```

**Current Risk**: **MODERATE**
- 100-step window is a reasonable middle ground
- But lack of hyperparameter tuning means we don't know if it's optimal

---

## 5. Normalization Correctness (BUG #41 Fix)

### 5.1 Why EMA Over Welford's Algorithm

**Documentation** (Lines 25-29):
> Welford's algorithm gives equal weight to all historical samples, causing mean drift in non-stationary markets. When the market produces losses, the mean drifts negative and stays there, inverting reward signs. EMA gives exponentially decaying weight to old samples, adapting to regime changes.

**Assessment**:
- ✅ **Correct Choice**: EMA is appropriate for non-stationary trading environments
- ✅ **Prevents Mean Drift**: Old regimes decay exponentially, not linearly
- ✅ **BUG #41 Fix**: Addresses the Welford's algorithm issue mentioned in prior audit reports

---

## 6. Integration with Reward Function

### 6.1 Update Order (CRITICAL FIX, Lines 544-552)

```rust
// CRITICAL: Normalize BEFORE update to avoid zeroing first reward
let norm = normalizer.normalize(final_reward_f64);  // Uses PREVIOUS stats
normalizer.update(final_reward_f64);                // Updates for NEXT call
```

**Comment** (Lines 545-548):
> Bug: If we update(x) then normalize(x), first call returns 0:
>   update(x): mean = x, variance = 1.0
>   normalize(x): returns (x - x) / 1.0 = 0
> Solution: normalize(x) using PREVIOUS stats, THEN update with new value

**Assessment**:
- ✅ **Order Correct**: Prevents first-reward zero bug
- ✅ **Proper Implementation**: Update happens AFTER normalization

---

## 7. Testing Coverage

### 7.1 Existing Tests

**Found in Codebase**:
- `ml/tests/dqn_reward_normalization_test.rs` (11 unit tests)
- `ml/tests/dqn_reward_calculation_test.rs` (12 tests)

**Coverage**:
- ✅ BUY/SELL symmetry
- ✅ Zero P&L normalization
- ✅ Negative P&L handling
- ✅ Large P&L normalization

### 7.2 Missing Tests (from TEST_COVERAGE_GAPS_AUDIT.md)

❌ **RewardNormalizer Update During Training**:
- No test verifies `normalizer.update()` is called after each reward
- **Risk**: Normalizer may use stale epoch-1 statistics for all epochs

❌ **Normalized Reward Range (-1.0 to 1.0)**:
- No test verifies rewards stay within expected bounds
- **Risk**: Rewards may explode if normalization fails

---

## 8. Recommendations

### Priority 1: Add Window Size Hyperparameter

**Current**:
```rust
impl RewardNormalizer {
    pub fn new() -> Self {
        Self {
            alpha: 0.01,  // FIXED
            beta: 0.01,   // FIXED
            // ...
        }
    }
}
```

**Proposed**:
```rust
pub struct RewardConfig {
    // ... existing fields ...

    /// EMA decay rate for mean (default: 0.01 = 100-step window)
    pub reward_normalizer_alpha: Decimal,
    /// EMA decay rate for variance (default: 0.01 = 100-step window)
    pub reward_normalizer_beta: Decimal,
}

impl Default for RewardConfig {
    fn default() -> Self {
        Self {
            // ... existing defaults ...
            reward_normalizer_alpha: Decimal::try_from(0.01).unwrap(),
            reward_normalizer_beta: Decimal::try_from(0.01).unwrap(),
        }
    }
}
```

**Hyperopt Search Space**:
```python
# Add to hyperopt configuration
"reward_normalizer_alpha": hp.loguniform("rn_alpha", -7, -2),  # 0.001 to 0.1
"reward_normalizer_beta": hp.loguniform("rn_beta", -7, -2),    # 0.001 to 0.1
```

### Priority 2: Test Normalization Update During Training

**Proposed Test** (add to `ml/tests/dqn_reward_normalization_test.rs`):
```rust
#[test]
fn test_normalizer_updates_during_training() {
    let config = RewardConfig::default();
    let mut reward_fn = RewardFunction::new(config);

    // Verify initial state
    assert_eq!(reward_fn.normalizer.as_ref().unwrap().count(), 0);

    // Run 100 reward calculations
    for i in 0..100 {
        let action = FactoredAction::new(/* ... */);
        let current_state = create_test_state(/* ... */);
        let next_state = create_test_state(/* ... */);

        reward_fn.calculate_reward(action, &current_state, &next_state, &[])?;
    }

    // Verify normalizer updated 100 times
    assert_eq!(reward_fn.normalizer.as_ref().unwrap().count(), 100);

    // Verify mean/variance changed from initialization
    let (mean, std) = reward_fn.normalizer.as_ref().unwrap().get_stats();
    assert_ne!(mean, 0.0);
    assert_ne!(std, 1.0);
}
```

### Priority 3: Add Regime-Aware Window Sizing

**Proposed Enhancement**:
```rust
pub struct RewardNormalizer {
    mean: f64,
    variance: f64,
    alpha: f64,
    beta: f64,

    // NEW: Adaptive window sizing
    regime_detector: Option<RegimeDetector>,
    alpha_fast: f64,   // 0.01 (100-step) for volatile regimes
    alpha_slow: f64,   // 0.001 (1000-step) for stable regimes
}

impl RewardNormalizer {
    pub fn update(&mut self, value: f64) {
        // Detect regime change
        let alpha = if let Some(detector) = &self.regime_detector {
            if detector.is_volatile() {
                self.alpha_fast  // React quickly to volatility
            } else {
                self.alpha_slow  // Use longer window for stability
            }
        } else {
            self.alpha  // Default fixed window
        };

        // EMA update with adaptive window
        self.mean = alpha * value + (1.0 - alpha) * self.mean;
        // ... variance update ...
    }
}
```

---

## 9. Comparison with Alternative Approaches

### 9.1 Welford's Algorithm (Original Implementation)

**Pros**:
- ✅ Exact online mean/variance calculation
- ✅ No hyperparameters to tune

**Cons**:
- ❌ Equal weight to all samples → mean drift in non-stationary markets (BUG #41)
- ❌ Cannot adapt to regime changes

**Verdict**: ❌ **Not suitable for trading** (already replaced with EMA)

### 9.2 Batch Normalization (Per-Episode)

**Pros**:
- ✅ No window size parameter
- ✅ Normalizes each episode independently

**Cons**:
- ❌ Cannot normalize online (needs full episode)
- ❌ Breaks temporal correlation across episodes

**Verdict**: ❌ **Not suitable for online RL**

### 9.3 Quantile Normalization

**Pros**:
- ✅ Robust to outliers
- ✅ No assumption of normal distribution

**Cons**:
- ❌ Requires storing large buffer of rewards
- ❌ More complex to implement

**Verdict**: ⚠️ **Worth exploring** for extreme market events

---

## 10. Final Assessment

### Current Parameters

| Parameter | Value | Effective Window | Assessment |
|-----------|-------|------------------|------------|
| `alpha` (mean decay) | 0.01 | 100 steps | ⚠️ May be too reactive |
| `beta` (variance decay) | 0.01 | 100 steps | ⚠️ May be too reactive |
| Clipping range | ±3.0σ | 99.7% coverage | ✅ Appropriate |
| Algorithm | EMA | O(1) memory | ✅ Correct choice |

### Risk Analysis

| Risk Type | Severity | Likelihood | Mitigation |
|-----------|----------|------------|------------|
| **Too short window** | MEDIUM | MEDIUM | Expose as hyperparameter, test 500-1000 step windows |
| **Reward hacking** | LOW | LOW | Clipping at ±3σ prevents extreme values |
| **Gradient issues** | LOW | LOW | Normalization to ~N(0,1) stabilizes gradients |
| **Overfitting to recent data** | MEDIUM | MEDIUM | Use regime-aware adaptive windows |

### Does Normalization Preserve Reward Structure?

**Yes**, with caveats:

1. ✅ **Relative Ordering**: Normalization preserves `r1 > r2 > r3` ordering
2. ✅ **Economic Signal**: ±3σ clipping is wide enough to not distort information
3. ⚠️ **Magnitude Information**: Raw reward magnitude is lost (by design)
4. ⚠️ **Temporal Correlation**: 100-step window may smooth out short-term patterns

---

## 11. Summary & Action Items

### What's Working Well

1. ✅ **EMA Algorithm**: Correct choice for non-stationary markets (BUG #41 fix)
2. ✅ **Update Order**: Normalize BEFORE update (prevents first-reward zero bug)
3. ✅ **Clipping Range**: ±3σ is appropriate for preserving signal
4. ✅ **Numerical Stability**: epsilon=1e-8 prevents division by zero

### What Needs Attention

1. ⚠️ **Window Size**: 100 steps may be too short for regime detection
2. ⚠️ **Fixed Parameters**: α/β not exposed as hyperparameters for tuning
3. ⚠️ **Missing Tests**: No integration test for normalizer update during training

### Recommended Actions

| Action | Priority | Effort | Impact |
|--------|----------|--------|--------|
| Add α/β as hyperparameters | HIGH | LOW | Enables hyperopt tuning |
| Test 500-1000 step windows | HIGH | MEDIUM | Reduces overfitting risk |
| Add normalizer update test | MEDIUM | LOW | Catches regression bugs |
| Implement regime-aware windows | LOW | HIGH | Advanced optimization |

---

## Appendix: Effective Window Size Cheat Sheet

| Alpha (α) | Effective Window | Use Case |
|-----------|------------------|----------|
| 0.05 | 20 steps | Very reactive (intraday trading) |
| 0.01 | **100 steps** | **Current default** (balanced) |
| 0.005 | 200 steps | Moderate stability |
| 0.001 | 1000 steps | Stable regimes (multi-day) |
| 0.0001 | 10000 steps | Very stable (trend-following) |

**Recommendation**: Test α ∈ {0.001, 0.005, 0.01} via hyperopt to find optimal balance.

---

**Agent 20 Analysis Complete**
**Next Steps**: Coordinate with Agent 21 (Hyperparameter Tuner) to add α/β to search space.