# Wave 10-A8 Phase 1: Coarse HOLD Penalty Search - Results

**Date**: 2025-11-05
**Duration**: 25 minutes
**Status**: ❌ **FAILED** - Critical bug discovered

---

## Executive Summary

**Phase 1 objective**: Test 3 HOLD penalty values [0.05, 0.10, 0.50] to reduce action bias from 96.4% HOLD to <75%.

**Result**: **ALL 3 TRIALS FAILED** with entropy < 0.05 and HOLD bias > 99%.

**Root Cause**: `hold_penalty_weight` parameter is **NOT CONNECTED** to the reward calculation logic. The CLI flag was successfully added, but the value is never used in `calculate_hold_reward()`.

---

## Trials Completed

| Trial | HOLD Penalty | Samples | Avg Q-values (BUY/SELL/HOLD) | Q-Spread | Action Distribution | Entropy | Status |
|-------|-------------|---------|------------------------------|----------|---------------------|---------|--------|
| **1**     | 0.05        | 2,175   | -37.67 / -129.72 / **142.70** | +180.37 pts | BUY=0.4%, SELL=0.0%, HOLD=**99.6%** | **0.039** | ❌ FAIL |
| **2**     | 0.10        | 2,175   | -27.21 / -193.75 / **166.74** | +193.94 pts | BUY=0.4%, SELL=0.0%, HOLD=**99.6%** | **0.035** | ❌ FAIL |
| **3**     | 0.50        | 2,175   | -72.35 / -95.74 / **141.97** | +214.32 pts | BUY=0.2%, SELL=0.0%, HOLD=**99.8%** | **0.023** | ❌ FAIL |

**Success Criteria**: Entropy > 0.5 AND HOLD < 75%
**Result**: **0/3 trials passed**

---

## Critical Bug Analysis

### Bug Location

**File**: `/home/jgrusewski/Work/foxhunt/ml/src/dqn/reward.rs`
**Function**: `calculate_hold_reward()` (lines 269-285)

### Current Implementation

```rust
fn calculate_hold_reward(
    &self,
    current_state: &TradingState,
    next_state: &TradingState,
) -> Result<Decimal, MLError> {
    // Extract close prices
    let current_price = ...;
    let next_price = ...;

    // Calculate price change
    let price_change_pct = ((next_price - current_price) / current_price).abs();

    // HARDCODED values - penalty weight is ignored!
    let hold_reward = if price_change_pct < self.config.movement_threshold {
        Decimal::try_from(0.002)  // ← Always +0.002
    } else {
        Decimal::try_from(-0.001) // ← Always -0.001
    };

    Ok(hold_reward)
}
```

### Problem

1. `hold_penalty_weight` is passed via CLI (✅ working)
2. `hold_penalty_weight` is set in `DQNHyperparameters` (✅ working)
3. `RewardConfig` struct has NO field for `hold_penalty_weight` (❌ missing)
4. `calculate_hold_reward()` returns hardcoded values (❌ bug)

### Evidence

**Trial 1 vs Trial 3**: Penalty increased 10x (0.05 → 0.50), but:
- HOLD bias **worsened**: 99.6% → 99.8% (+0.2%)
- Q-spread **increased**: 180 → 214 points (+18.8%)
- Entropy **decreased**: 0.039 → 0.023 (-41%)

This proves the penalty weight has **zero effect** on Q-values.

---

## Log Files

Training logs saved to:
```
/tmp/hold_penalty_0.05.log  (654 KB)
/tmp/hold_penalty_0.10.log  (654 KB)
/tmp/hold_penalty_0.50.log  (654 KB)
```

**Key Observations**:
1. ✅ Training completed successfully (4.15s per epoch, 5 epochs each)
2. ✅ Gradient warnings present (grad_norm=0.0000) but training converged
3. ❌ Q-values show HOLD bias in all trials (200-250 point advantage)
4. ❌ Action distribution shows 99%+ HOLD in all trials

---

## Root Cause Investigation

### Missing Connection

**`train_dqn.rs` (lines 298-310)**:
```rust
let hyperparams = DQNHyperparameters {
    // ... other parameters ...
    hold_penalty_weight: opts.hold_penalty_weight,  // ← Set correctly
    // ...
};
```

**`trainers/dqn.rs`** (missing):
```rust
// ❌ MISSING: Connection from hyperparameters to RewardConfig
// Should pass hold_penalty_weight to RewardConfig during initialization
```

**`reward.rs` (lines 13-23)**:
```rust
pub struct RewardConfig {
    pub pnl_weight: Decimal,
    pub risk_weight: Decimal,
    pub cost_weight: Decimal,
    pub hold_reward: Decimal,  // ← Renamed to "hold_reward" (misleading)
    pub movement_threshold: Decimal,
    pub diversity_weight: Decimal,
    // ❌ MISSING: hold_penalty_weight field
}
```

---

## Impact Analysis

### Current State
- **Wave 10-A6**: 96.4% HOLD bias (agent doesn't trade)
- **Phase 1 trials**: 99.6-99.8% HOLD bias (worse than baseline)
- **Entropy**: <0.05 (effectively zero diversity)
- **Q-spread**: 180-214 points (massive HOLD advantage)

### Why This Matters
1. **Manual tuning impossible**: Cannot fix HOLD bias without working penalty mechanism
2. **Wave 10-A7 research invalidated**: Assumed penalty was working (it wasn't)
3. **Production blocker**: Cannot deploy DQN with 99% HOLD bias

---

## Recommended Fix

### Step 1: Add `hold_penalty_weight` to `RewardConfig`

```rust
pub struct RewardConfig {
    pub pnl_weight: Decimal,
    pub risk_weight: Decimal,
    pub cost_weight: Decimal,
    pub hold_reward: Decimal,
    pub movement_threshold: Decimal,
    pub diversity_weight: Decimal,
    pub hold_penalty_weight: Decimal,  // ← ADD THIS
}
```

### Step 2: Update `Default` implementation

```rust
impl Default for RewardConfig {
    fn default() -> Self {
        Self {
            // ... existing fields ...
            hold_penalty_weight: Decimal::try_from(0.01).unwrap_or(Decimal::ZERO),
        }
    }
}
```

### Step 3: Fix `calculate_hold_reward()` to apply penalty

```rust
fn calculate_hold_reward(
    &self,
    current_state: &TradingState,
    next_state: &TradingState,
) -> Result<Decimal, MLError> {
    // Extract close prices
    let current_price = Decimal::try_from(*current_state.price_features.get(0).unwrap_or(&0.0) as f64)
        .unwrap_or(Decimal::ZERO);
    let next_price = Decimal::try_from(*next_state.price_features.get(0).unwrap_or(&0.0) as f64)
        .unwrap_or(Decimal::ZERO);

    if current_price == Decimal::ZERO {
        return Ok(Decimal::ZERO);  // Return zero instead of default reward
    }

    // Calculate price change
    let price_change_pct = ((next_price - current_price) / current_price).abs();

    // Dynamic HOLD reward based on price movement
    let base_reward = if price_change_pct < self.config.movement_threshold {
        Decimal::try_from(0.002).map_err(|e| MLError::InvalidInput(format!("Failed to create hold reward: {}", e)))?
    } else {
        Decimal::try_from(-0.001).map_err(|e| MLError::InvalidInput(format!("Failed to create hold penalty: {}", e)))?
    };

    // Apply HOLD penalty weight (negative value to penalize HOLD)
    let penalty = self.config.hold_penalty_weight;
    let final_reward = base_reward - penalty;  // Subtract penalty to discourage HOLD

    Ok(final_reward)
}
```

### Step 4: Connect hyperparameters to `RewardConfig`

In `trainers/dqn.rs`, when initializing `RewardFunction`:

```rust
let reward_config = RewardConfig {
    pnl_weight: Decimal::ONE,
    risk_weight: Decimal::try_from(self.hyperparams.risk_weight).unwrap_or(Decimal::ZERO),
    cost_weight: Decimal::try_from(0.05).unwrap_or(Decimal::ZERO),
    hold_reward: Decimal::try_from(0.001).unwrap_or(Decimal::ZERO),
    movement_threshold: Decimal::try_from(self.hyperparams.movement_threshold).unwrap_or(Decimal::ZERO),
    diversity_weight: Decimal::try_from(-0.1).unwrap_or(Decimal::ZERO),
    hold_penalty_weight: Decimal::try_from(self.hyperparams.hold_penalty_weight).unwrap_or(Decimal::ZERO),  // ← ADD THIS
};
```

---

## Next Steps

### Phase 1 (Revised): Fix & Verify

1. **Implement fix** (30 min):
   - Add `hold_penalty_weight` to `RewardConfig`
   - Update `calculate_hold_reward()` to apply penalty
   - Connect hyperparameters to reward config

2. **Verification test** (15 min):
   - Run single trial with penalty=1.0
   - Verify Q-value spread decreases
   - Confirm HOLD bias reduces

### Phase 2: Re-run Coarse Search

Once fix is verified:
- Re-run trials with penalties [0.5, 1.0, 2.0]
- Target: Entropy > 0.5, HOLD < 75%
- Duration: 15 minutes

### Phase 3: Fine-Tuning

If Phase 2 identifies working range:
- Test 3-5 values in optimal range
- Run 50-epoch validation
- Deploy to production

---

## Code Changes Made

### File: `ml/examples/train_dqn.rs`

**Lines 130-132** (added CLI flag):
```rust
/// HOLD penalty weight (higher = stronger penalty for holding)
#[arg(long, default_value = "0.01")]
hold_penalty_weight: f64,
```

**Line 307** (connected to hyperparameters):
```rust
hold_penalty_weight: opts.hold_penalty_weight,  // Configurable via CLI
```

**Status**: ✅ CLI flag working (verified via logs)

### Files NOT Modified (bug still present)

- `ml/src/dqn/reward.rs`: Missing `hold_penalty_weight` field and logic
- `ml/src/trainers/dqn.rs`: Missing connection to `RewardConfig`

---

## Lessons Learned

### What Went Wrong
1. **Incomplete feature**: CLI flag added without connecting to business logic
2. **No validation**: No test verified penalty actually affects Q-values
3. **Misleading naming**: `hold_reward` field name implies penalty (but it's hardcoded reward)
4. **Insufficient testing**: Wave 10-A7 research assumed penalty was working

### What Went Right
1. **Test-driven approach**: 5-epoch trials caught bug before expensive 50-epoch runs
2. **Systematic analysis**: Python script revealed Q-values unchanged across trials
3. **Root cause investigation**: Traced bug to exact code location
4. **Documentation**: This report provides actionable fix

---

## Cost Analysis

**Time spent**: 25 minutes (Phase 1 execution + analysis)
**Trials completed**: 3 (15 min total training time)
**Bug found**: YES (saved 2-3 hours of failed tuning)
**ROI**: **POSITIVE** - Early bug detection prevents wasted hyperopt runs

---

## Conclusion

**Phase 1 status**: ❌ **FAILED** (critical bug discovered)
**Blocker identified**: `hold_penalty_weight` not connected to reward calculation
**Fix complexity**: LOW (30 min implementation + 15 min verification)
**Next action**: Implement fix, verify with single trial, then re-run Phase 1

**Recommendation**: **DO NOT PROCEED** to Phase 2 until bug is fixed and verified.