foxhunt/WAVE10_A10_BUG_REPORT.md

Wave 10-A10: HOLD Penalty Bug Report

Executive Summary

Status: ❌ BLOCKED - Cannot run Phase 1 trials due to zero price error
Root Cause: calculate_hold_reward incorrectly treats normalized log returns as raw prices
Severity: CRITICAL - Prevents all training runs after epoch 1
Impact: 100% training failure rate (2/2 trials failed with same error)

---

Error Details

Error Message

Error: Training from Parquet failed

Caused by:
    Invalid input: Current price is zero in calculate_hold_reward

Failure Pattern

• Trial 1 (penalty=0.5): Crashed after epoch 1 (4350 steps, 4.16s)
• Trial 2 (penalty=1.0): Crashed after epoch 1 (4350 steps, 4.04s)
• Consistency: 100% failure rate at validation phase

Stack Trace Location

ml/src/dqn/reward.rs:266-269
if current_price == Decimal::ZERO {
    return Err(MLError::InvalidInput(
        "Current price is zero in calculate_hold_reward".to_string(),
    ));
}

---

Root Cause Analysis

The Bug

File: ml/src/dqn/reward.rs lines 254-287
Function: calculate_hold_reward

fn calculate_hold_reward(
    &self,
    current_state: &TradingState,
    next_state: &TradingState,
) -> Result<Decimal, MLError> {
    // ❌ BUG: Treats normalized log returns as raw prices
    let current_price = Decimal::try_from(*current_state.price_features.get(0).unwrap_or(&0.0) as f64)
        .unwrap_or(Decimal::ZERO);
    let next_price = Decimal::try_from(*next_state.price_features.get(0).unwrap_or(&0.0) as f64)
        .unwrap_or(Decimal::ZERO);

    // ❌ Fails when log returns are zero (no price change)
    if current_price == Decimal::ZERO {
        return Err(MLError::InvalidInput(
            "Current price is zero in calculate_hold_reward".to_string(),
        ));
    }

    // ❌ Calculates price change from log returns (meaningless)
    let price_change = next_price - current_price;
    let price_change_pct = price_change / current_price;
    ...
}

What price_features[0] Actually Contains

Source: ml/src/trainers/dqn.rs lines 1642-1682

/// - Features 0-3: OHLC log returns → price_features (signed, normalized)
let price_features: Vec<f32> = vec![
    close_log_return,   // Feature 0: log(close_t / close_{t-1})
    high_log_return,    // Feature 1: log(high_t / close_{t-1})
    low_log_return,     // Feature 2: log(low_t / close_{t-1})
    open_log_return,    // Feature 3: log(open_t / close_{t-1})
];

Key Point: price_features[0] contains log returns (can be 0.0 for stable prices), NOT raw close prices.

Why This Causes Crashes

1. Log returns can be 0.0: When close price is stable (close_t == close_{t-1}), log(1.0) = 0.0
2. Zero check triggers error: if current_price == Decimal::ZERO → immediate crash
3. Validation batch processing: Error occurs during validation loop (after epoch 1 completes)
4. No fallback: No graceful handling, entire training run terminates

---

Impact Analysis

Training Metrics (Before Crash)

Trial 1 (penalty=0.5):

• Epoch 1 completed: 4350 steps, 4.16s
• Train loss: 685.82
• Avg Q-value: 200.74
• Q-spread (HOLD-BUY): 263.37 points ⚠️
• Action distribution: BUY=0%, SELL=0%, HOLD=100% ❌
• Entropy: 0.000 (total action bias)

Trial 2 (penalty=1.0):

• Epoch 1 completed: 4350 steps, 4.04s
• Train loss: 639.19
• Avg Q-value: 203.41
• Q-spread (HOLD-BUY): Similar to Trial 1
• Action distribution: Expected BUY=0%, SELL=0%, HOLD=100% (not measured due to crash)

Key Observations

1. Penalty ineffective: Despite penalty=0.5 vs 1.0, HOLD bias remains 100%
2. Gradient collapse: grad_norm=0.0000 at all steps (gradient clipping bug?)
3. Q-value divergence: HOLD advantage ~200-260 points (BUY/SELL collapsed)
4. Zero entropy: Perfect action bias (all HOLD), diversity penalty not working

---

Recommended Fix

Option 1: Use Log Returns Directly (Preferred)

Rationale: Log returns already measure price volatility

fn calculate_hold_reward(
    &self,
    current_state: &TradingState,
    next_state: &TradingState,
) -> Result<Decimal, MLError> {
    // ✅ Use log returns directly (already measures price change)
    let current_log_return = Decimal::try_from(*current_state.price_features.get(0).unwrap_or(&0.0) as f64)
        .unwrap_or(Decimal::ZERO);
    let next_log_return = Decimal::try_from(*next_state.price_features.get(0).unwrap_or(&0.0) as f64)
        .unwrap_or(Decimal::ZERO);

    // Calculate volatility as absolute log return change
    let volatility = (next_log_return - current_log_return).abs();

    // Apply threshold (e.g., 0.02 for 2% price movement)
    let hold_reward = if volatility < self.config.movement_threshold {
        self.config.hold_reward  // Reward stability
    } else {
        -self.config.hold_penalty_weight  // Penalize holding during volatility
    };

    Ok(hold_reward)
}

Pros:

• No additional data required (uses existing features)
• Mathematically sound (log returns = price volatility)
• Zero-safe (log returns can be zero without error)

Cons:

• Threshold interpretation changes (from % to log space)
• May need to recalibrate movement_threshold (current: 0.02)

Option 2: Pass Raw Prices Through TrainingState

Rationale: Preserve original intent (raw price volatility)

// Add to TradingState
pub struct TradingState {
    pub price_features: Vec<f32>,        // Normalized features
    pub technical_indicators: Vec<f32>,
    pub market_features: Vec<f32>,
    pub portfolio_features: Vec<f32>,
    pub raw_close_price: Option<f32>,    // ✅ NEW: Raw close price for reward calculation
}

// Update calculate_hold_reward
fn calculate_hold_reward(
    &self,
    current_state: &TradingState,
    next_state: &TradingState,
) -> Result<Decimal, MLError> {
    // ✅ Use raw prices if available, fallback to graceful handling
    let current_price = current_state.raw_close_price
        .ok_or_else(|| MLError::InvalidInput("Raw close price not available".to_string()))?;
    let next_price = next_state.raw_close_price
        .ok_or_else(|| MLError::InvalidInput("Raw close price not available".to_string()))?;

    if current_price <= 0.0 {
        return Ok(Decimal::ZERO);  // ✅ Graceful fallback instead of crash
    }

    let price_change_pct = (next_price - current_price) / current_price;
    
    let hold_reward = if price_change_pct.abs() < self.config.movement_threshold {
        self.config.hold_reward
    } else {
        -self.config.hold_penalty_weight
    };

    Ok(Decimal::try_from(hold_reward).unwrap_or(Decimal::ZERO))
}

Pros:

• Preserves original reward semantics (raw price volatility)
• Clearer separation of concerns (features vs raw data)
• Graceful fallback (return 0.0 instead of crash)

Cons:

• Requires modifying TradingState struct
• Need to populate raw_close_price in all call sites (13 locations)
• Larger memory footprint (extra f32 per state)

---

Additional Issues Discovered

1. Gradient Collapse (Bug #1 Regression?)

Evidence:

[2025-11-05T22:59:40.728164Z] WARN ml::dqn::dqn: ⚠️  GRADIENT COLLAPSE: norm=0.000000 at step 100
[2025-11-05T22:59:40.799971Z] WARN ml::dqn::dqn: ⚠️  GRADIENT COLLAPSE: norm=0.000000 at step 200
[2025-11-05T22:59:40.870041Z] WARN ml::dqn::dqn: ⚠️  GRADIENT COLLAPSE: norm=0.000000 at step 300

Hypothesis: Gradient clipping fix (Bug #1) may not be active, or Q-values have collapsed again.

Verification Needed: Check if max_norm=10.0 is correctly applied in ml/src/trainers/dqn.rs.

2. HOLD Penalty Not Working (Hypothesis)

Evidence:

• penalty=0.5: HOLD=100%, entropy=0.000
• penalty=1.0: HOLD=100% (expected, not measured due to crash)

Hypothesis:

1. Reward calculation crashes before penalty can take effect
2. OR penalty is being applied to wrong component (base_reward vs diversity_bonus)

Verification Needed:

• Check if diversity penalty (-0.1) is reaching action selection
• Verify entropy calculation includes recent_actions buffer

---

Recommendations

Immediate Actions (Block Phase 1)

1. Fix Option 1: Implement log return volatility fix (preferred, 30 min)
2. Test fix: Run 1-epoch smoke test to verify no crash (2 min)
3. Resume Phase 1: Re-run trials with penalty=[0.5, 1.0, 2.0]

Follow-Up Actions (Phase 2)

1. Investigate gradient collapse: Verify Bug #1 fix is active
2. Validate HOLD penalty: Add debug logging to reward calculation
3. Test diversity penalty: Verify entropy-based action regularization

---

Conclusion

Current Status: Phase 1 blocked by zero price error (100% failure rate)

Recommended Fix: Option 1 (log return volatility) - fastest implementation, mathematically sound

ETA: 30-45 minutes (implementation + smoke test + retry Phase 1)

Risk: Low - log returns already measure price volatility, no semantic change

---

Appendix: Log Excerpts

Trial 1 Final Steps (Before Crash)

Step 4340: BUY=-131.29, SELL=-18.64, HOLD=107.93, loss=544.32
Step 4350: BUY=-128.71, SELL=19.59, HOLD=123.28, loss=519.10
Epoch 1/5: train_loss=685.82, Q-value=200.74, grad_norm=0.0000, train_steps=4350
Error: Training from Parquet failed
Caused by: Invalid input: Current price is zero in calculate_hold_reward

Trial 2 Identical Pattern

Step 4340: BUY=-134.53, SELL=20.86, HOLD=131.49, loss=544.36
Step 4350: BUY=-128.10, SELL=19.73, HOLD=125.02, loss=537.51
Epoch 1/5: train_loss=639.19, Q-value=203.41, grad_norm=0.0000, train_steps=4350
Error: Training from Parquet failed
Caused by: Invalid input: Current price is zero in calculate_hold_reward

Pattern: Crash occurs during validation phase after epoch 1 completes successfully.